Image processing method and device for removing artifacts
Through iterative training of adaptive convolutional dictionary network and the use of basic artifact dictionary, the complexity of artifact removal and data acquisition problems in image processing are solved, efficient artifact removal effect is achieved, and clear image processing is provided Results and strong generalization ability.
Patent Information
- Application Number
- CN202210408968.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-04-19
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2042-04-19
AI Technical Summary
When removing artifacts, the mathematical model is complex, the application scope is limited, and some data is difficult to obtain, making it difficult to effectively identify and remove artifacts in images.
Adaptive convolutional dictionary network is adopted, by establishing training data sets including artifact-free image samples, using the adaptive convolutional dictionary network for iterative training, optimizing network parameters to remove artifacts, and using basic artifact dictionary and weighting The coefficient determines the adaptive convolution kernel to remove artifacts from the image.
It realizes simple and effective removal of artifacts in images, obtains clearer images without being disturbed by artifacts, has strong generalization of the model, clear physical meaning, and is easy to understand and apply.
Smart Images

Figure CN115131452B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of artificial intelligence, and more specifically, to an image processing method, device, computer program product, and storage medium for artifact removal. Background Art
[0002] As one of the most important sources of information for humans, images account for a large proportion of various information databases. With the rapid development of computer technology, image processing has been widely used in all aspects of human social life, such as industrial inspection, medicine, intelligent robots, etc. Images are often used in various fields to describe and express the characteristics and logical relationships of things due to their vividness and intuitiveness. They have a wide range of applications. Therefore, the development of image processing technology and information processing in various fields are extremely important.
[0003] Image processing technology is a technology that uses computers to process image information. It mainly includes image digitization, image enhancement and restoration, image data encoding, image segmentation and image recognition. Among them, the purpose of image restoration technology is to restore the degraded image to its original true appearance as much as possible. For example, it is used in the field of removing artifacts in images, such as removing noise in images, removing raindrops in rainy images taken on rainy days, and removing metal artifacts in CT images. However, the current image processing methods face technical bottlenecks such as complex mathematical models, limited application scope, and difficulty in obtaining some data when removing artifacts.
[0004] Therefore, there is a need for an image processing method that can be widely used and effectively identify artifacts in images and remove them from images with artifacts, so as to help people obtain clearer images without being disturbed by artifacts. Summary of the invention
[0005] In order to solve the above problems, the present disclosure provides an image processing method, device, computer program product and storage medium for artifact removal. The image processing method for artifact removal provided by the present disclosure includes: establishing a training data set for training a neural network, wherein the training data set includes multiple groups of image samples, each group of image samples includes an image with artifacts (Y) and a corresponding image without artifacts (X) and an image mask (I); for at least one group of image samples among the multiple groups of image samples, using an adaptive convolutional dictionary network, performing artifact removal processing on the image with artifacts (Y) to obtain a processed image, iteratively training the adaptive convolutional dictionary network based on the artifact-free image (X) and the processed image, and the objective function processed by the image mask (I); The method comprises the steps of: training the image processing method and the processing method of the adaptive convolution dictionary network to optimize the network parameters of the adaptive convolution dictionary network, wherein the adaptive convolution dictionary network comprises a basic artifact dictionary, the basic artifact dictionary is a convolution dictionary that does not vary with samples and comprises a first number of artifact convolution kernels, and a second number of adaptive convolution kernels for the image sample are determined by combining a plurality of artifact convolution kernels in the basic artifact dictionary with a weighting coefficient that varies with samples, wherein an artifact image in the image with artifacts is determined by convolution of the second number of adaptive convolution kernels with image features of the image with artifacts, and the artifact image is removed from the image with artifacts to obtain the processed image.
[0006] The image processing method disclosed in the present invention can simply and effectively remove artifacts in an image to obtain a clearer image that is not disturbed by artifacts.
[0007] According to an embodiment of the present disclosure, the first number of artifact convolution kernels indicates an artifact pattern, and the image features indicate the location of the artifact pattern, wherein the adaptive convolution dictionary network includes a T-level network, wherein, in the t-th level network, the weighting coefficients and image features output by the t-1-th level network are updated using an iterative update rule based on proximal gradient descent to obtain the weighting coefficients and image features of the t-th level network, wherein t is an integer greater than 1 and less than or equal to T.
[0008] According to an embodiment of the present disclosure, each level of the network includes a weighted coefficient update network, an image feature update network, and an artifact removal image update network, wherein the weighted coefficient update network, the image feature update network, and the artifact removal image update network include a residual network structure and a normalization processing layer.
[0009] According to an embodiment of the present disclosure, the establishment of a training data set for training a neural network also includes at least one of the following: normalizing the pixel values of the image with artifacts (Y); and randomly cropping the image with artifacts (Y) to obtain image blocks, and randomly flipping the image blocks according to a predetermined probability.
[0010] According to an embodiment of the present disclosure, the objective function is a loss objective function constructed based on the artifact-free image (X) and the processed image, wherein the iterative training of the adaptive convolutional dictionary network based on the artifact-free image (X) and the processed image, and the objective function to optimize the network parameters of the adaptive convolutional dictionary network also includes: calculating the loss objective function and back-propagating its result to the adaptive convolutional dictionary network, and optimizing the network parameters of the adaptive convolutional dictionary network based on an adaptive moment estimation (Adam) algorithm.
[0011] According to an embodiment of the present disclosure, the image processing method further includes: after the training is completed, testing the adaptive convolutional dictionary network, wherein the testing of the adaptive convolutional dictionary network includes: preprocessing the image with artifacts to be tested and inputting it into the adaptive convolutional dictionary network; using the adaptive convolutional dictionary network, processing the image with artifacts to be tested to obtain a processed image with artifacts removed.
[0012] According to an embodiment of the present disclosure, the artifact is a metal artifact, the image with artifact is a CT image with metal artifacts, and the training data set includes multiple groups of CT image samples, each group of image samples includes a CT image with metal artifacts and its corresponding CT image without metal artifacts and a non-metal area mask.
[0013] According to an embodiment of the present disclosure, each level of the network includes a weighted coefficient update network, a metal artifact image feature update network, and a metal artifact removal image update network, wherein the weighted coefficient update network, the metal artifact image feature update network, and the metal artifact removal image update network include a residual network structure; and the weighted coefficient update network includes: a linear layer, a rectified linear unit (Rectified Linear Unit, ReLU) layer, a cross-link layer, and a batch normalization (BatchNormalization, BN) layer; the metal artifact image feature update network includes: a convolutional layer, a BN layer, a ReLU layer, and a cross-link layer; the metal artifact removal image update network includes: a convolutional layer, a BN layer, a ReLU layer, and a cross-link layer.
[0014] An embodiment of the present disclosure also provides an image processing method for artifact removal, comprising: obtaining an input image to be processed; using an adaptive convolutional dictionary network to process the input image to obtain a processed image with artifacts removed, wherein the adaptive convolutional dictionary network is trained based on an artifact database (D) and includes T-level networks, wherein a first-level network obtains first-level image features and a first-level artifact removed image output by the first-level network based on the input image; a t-th level network obtains and outputs the t-th level image features and the t-th level artifact removed image output by the t-th level network at least partially based on the t-1-th level image features and the t-1-th level artifact removed image output by the t-1-th level network, wherein t is greater than 1 and less than or equal to T; a T-th level network obtains and outputs the T-th level artifact removed image output by the T-th level network as the processed image with artifacts removed based on the T-1-th level image features and the T-1-th level artifact removed image output by the T-1-th level network.
[0015] According to an embodiment of the present disclosure, the adaptive convolution dictionary network includes a basic artifact dictionary, which is a convolution dictionary that does not change with the input image and includes a first number of artifact convolution kernels, determines the t-th level weighting coefficient of the t-th level network, determines a second number of adaptive convolution kernels for the t-th level network through multiple artifact convolution kernels in the basic artifact dictionary and the t-th level weighting coefficient; and determines a t-th level artifact removed image based on the second number of adaptive convolution kernels and the image features of the t-th level network.
[0016] The embodiment of the present disclosure provides an image processing device for artifact removal, comprising: a training data set establishment module, configured to: establish a training data set for training a neural network, wherein the training data set includes multiple groups of image samples, each group of image samples includes an image with artifacts (Y) and a corresponding image without artifacts (X) and an image mask (I); an adaptive convolution dictionary network, configured to: for at least one group of image samples among the multiple groups of image samples, perform artifact removal processing on the image with artifacts (Y) to obtain a processed image; a training module, configured to: based on the image without artifacts (X) and the processed image, and the target function processed by the image mask (I); The adaptive convolution dictionary network is iteratively trained to optimize the network parameters of the adaptive convolution dictionary network; wherein the adaptive convolution dictionary network includes a basic artifact dictionary, which is a convolution dictionary that does not vary with samples and includes a first number of artifact convolution kernels, and a second number of adaptive convolution kernels for the image sample are determined by using a plurality of artifact convolution kernels in the basic artifact dictionary and a weighting coefficient that varies with samples, wherein an artifact image in the image with artifacts is determined by convolution of the second number of adaptive convolution kernels with image features of the image with artifacts, and the artifact image is removed from the image with artifacts to obtain the processed image.
[0017] An embodiment of the present disclosure provides an image processing device for artifact removal, comprising: an image acquisition module, configured to: acquire an input image to be processed; an image processing module, configured to: use an adaptive convolutional dictionary network to process the input image to obtain a processed image with artifacts removed; wherein the adaptive convolutional dictionary network is trained based on an artifact database (D) and comprises a T-level network, wherein a first-level network obtains a first-level image feature and a first-level artifact removed image output by the first-level network based on the input image; a t-th-level network obtains and outputs a t-th-level image feature and a t-th-level artifact removed image output by the t-th-level network at least partially based on the t-1-th-level image feature and the t-1-th-level artifact removed image output by the t-th-level network, wherein t is greater than 1 and less than or equal to T; and a T-th-level network obtains and outputs a T-th-level artifact removed image output by the T-th-level network as the processed image with artifacts removed based on the T-1-th-level image feature and the T-1-th-level artifact removed image output by the T-th-level network.
[0018] An embodiment of the present disclosure provides a computer program product, which includes computer software code. When the computer software code is executed by a processor, the above method is provided.
[0019] An embodiment of the present disclosure provides a computer-readable storage medium having computer-executable instructions stored thereon, and when the instructions are executed by a processor, the above method is provided.
[0020] The image processing method disclosed in the present invention can perform artifact removal processing based only on the image to be processed without the need to obtain the chord diagram of the image additionally. The adaptive convolutional dictionary network used in the present invention makes full use of the prior structure of the artifact image, and the artifact removal effect is better and the model generalization is stronger. In addition, the mathematical model used in the image processing method disclosed in the present invention has a clear physical meaning and strong explanatory power, and the physical meaning of each network module is clearer, which facilitates the understanding and application of those skilled in the art. The image processing method disclosed in the present invention can simply and effectively remove artifacts in an image to obtain a clearer image that is not interfered with by artifacts. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some exemplary embodiments of the present disclosure, and a person of ordinary skill in the art can obtain other drawings based on these drawings without creative work.
[0022] Herein, in the accompanying drawings:
[0023] Figure 1A-1B are schematic diagrams respectively showing an image with artifacts and an image without artifacts according to an embodiment of the present disclosure;
[0024] Figure 2A-2B is a schematic diagram showing an image processing method based on a chord diagram according to an embodiment of the present disclosure;
[0025] Figure 3A-3B is a schematic flow chart showing an image processing method for artifact removal according to an embodiment of the present disclosure;
[0026] Figure 4 is an exemplary schematic diagram showing an image processing model based on a weighted adaptive convolution dictionary according to an embodiment of the present disclosure;
[0027] Figure 5A-Figure 5B is a schematic diagram showing an iterative update process of each level of the network in an adaptive convolutional dictionary network according to an embodiment of the present disclosure;
[0028] Figure 6 is a schematic flow chart showing an image processing process for artifact removal according to an embodiment of the present disclosure;
[0029] Figure 7A-7B is a schematic diagram showing the composition of an image processing device for artifact removal according to an embodiment of the present disclosure;
[0030] Figure 8 is an architecture showing a computing device according to an embodiment of the present disclosure; and
[0031] Fig. 9 is a schematic diagram illustrating a storage medium according to an embodiment of the present disclosure. DETAILED DESCRIPTION
[0032] In order to make the purpose, technical solution and advantages of the present disclosure more obvious, the exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described here.
[0033] Furthermore, in the present specification and the drawings, steps and elements having substantially the same or similar features are denoted by the same or similar reference numerals, and repeated description of these steps and elements will be omitted.
[0034] In addition, in this specification and the accompanying drawings, elements are described in singular or plural form, depending on the embodiment. However, the singular and plural forms are appropriately selected for the proposed situation only for the convenience of explanation and are not intended to limit the present disclosure thereto. Therefore, the singular form may include the plural form, and the plural form may also include the singular form, unless the context clearly indicates otherwise.
[0035] In this specification and the drawings, substantially the same or similar steps and elements are represented by the same or similar reference numerals, and repeated descriptions of these steps and elements will be omitted. Meanwhile, in the description of the present disclosure, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance or ranking.
[0036] To facilitate description of the present disclosure, concepts related to the present disclosure are introduced below.
[0037] The method disclosed herein may be based on artificial intelligence (AI). Artificial intelligence is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. For example, for methods based on artificial intelligence, it is possible to perform machine learning in a manner similar to human perception, such as extracting image information, performing image analysis and processing by training neural networks.
[0038] Image processing technology is a technology that uses computers to process image information. It mainly includes: image digitization, image enhancement and restoration, image data encoding, image segmentation and image recognition. At present, image processing technology is mainly based on computers. Artificial Neural Networks (ANN) is an algorithmic mathematical model that simulates the structure and behavior of biological nervous systems and performs distributed parallel information processing. ANN achieves the purpose of processing information by adjusting the weight relationship between internal neurons and neurons. When processing images, the computer uses the original image or the image that has been properly preprocessed as the input signal of the neural network, and obtains the processed image signal or classification result at the output of the neural network.
[0039] In summary, the solutions provided by the embodiments of the present disclosure involve computer technologies such as artificial intelligence, image processing, and neural networks. The embodiments of the present disclosure will be further described below in conjunction with the accompanying drawings.
[0040] Taking CT images as an example, Figure 1A is a schematic diagram showing an image with artifacts according to an embodiment of the present disclosure.
[0041] X-ray computed tomography (CT) has been widely used in clinical diagnosis. However, during the imaging process, the presence of metal implants in the patient's body, such as dental fillings and hip prostheses, usually leads to the loss of projection data, resulting in severe metal artifacts in the reconstructed CT images. Figure 1A As shown in FIG. 1 , the structure of the metal artifact is a group of stripe shadows with similar shapes. For different tissue structures and / or metal implants, the thickness and brightness of the stripe shadows will be different. Figure 1A It can be seen that the presence of metal artifacts makes the CT image blurred, seriously interfering with the clear presentation of the patient's body details in the CT image, making it difficult for doctors to make correct judgments based on the CT image.
[0042] Therefore, an image processing method is needed that can effectively identify artifacts in images and remove them from images with artifacts, so that the processed artifact-free images are Figure 1B As shown. Figure 1B The clear CT images without artifacts shown can help doctors make more accurate diagnoses.
[0043] Figure 2A A schematic diagram of an image processing method based on a chord diagram according to an embodiment of the present disclosure is shown.
[0044] like Figure 2AAs shown in the figure, DuDoNet++ is a joint learning scheme based on CT images and chord diagrams. The scheme uses two network modules, SE-Net and IE-Net, to jointly process CT images with metal artifacts. SE-Net is a network module for chord diagrams, and IE-Net is a network module for CT images. Figure 2A In, S ma The chord diagram representing metal contamination, S se represents the repaired and enhanced chord diagram, X se Indicated by S se The CT image obtained after the reverse projection transformation (RIL), M represents the shape feature of the metal in the CT image domain (i.e., mask), M p represents the shape characteristics of the metal in the chordal domain obtained after M is transformed by Radon (FP), X ma Represents a CT image with metal artifacts, X out Represents the output reconstructed image. Figure 2A It can be seen that CT images with metal artifacts need to be processed jointly in the chord domain (using SE-Net) and the CT image domain (using IE-Net) to achieve the purpose of image artifact removal, where the transformation between the chord domain and the CT image domain is converted through a differentiable Radon transform layer.
[0045] Similarly, Figure 2B A schematic diagram of another image processing method based on a chord diagram according to an embodiment of the present disclosure is shown.
[0046] like Figure 2B As shown in Figure 1, DSCMAR is also a joint processing scheme based on CT images and chord graphs. Unlike DuDoNet++, this scheme first uses PriorNet to obtain a relatively clean repaired image, then uses SinoNet to further correct and enhance the chord graph, and finally obtains the reconstructed CT image through the filtered back projection (FBP) layer. Figure 2B In, S ma The chord diagram representing metal contamination, T r Chord diagram representing metal characteristics, S LI represents the chord diagram after linear interpolation, X LI Indicated by S LI The CT image obtained after back-projection transformation, X ma Represents a CT image with metal artifacts, X prior represents the initial repaired CT image after PriorNet processing, S prior Indicated by X prior The initial repaired chord diagram obtained after forward projection transformation, S res Indicates that by SLI and S prior The chord diagram obtained after subtraction, S corr Represents the further modified and enhanced chord graph after SinoNet processing. corr Performing filtered back-projection transformation can obtain an image without metal artifacts.
[0047] Figure 2A and Figure 2B The chord diagram-based image processing method shown can achieve the following Figure 1A and Figure 1B The image processing for artifact removal is shown in the figure. However, the common limitation of DuDoNet++ and DSCMAR is that in these two schemes, the chord diagram information used is difficult to obtain in practice and usually needs to be provided by the equipment manufacturer. Moreover, the designed network does not embed the prior information specific to the Metal Artifacts Reduction (MAR) task well, and the generalization ability of the network model is limited. In addition, each network module contained in these two schemes has weak physical interpretability and is not easy for technicians in this field to understand and use.
[0048] In response to these problems, the present disclosure proposes an image processing method for removing artifacts, which can only perform reconstruction based on the image with artifacts, overcoming the problem of difficulty in obtaining chord diagram data. Moreover, the image processing method of the present disclosure uses a specific weighted convolution dictionary model to encode the prior structure of the artifacts, making the network more reliable and more generalizable. In addition, the adaptive convolution dictionary network model of the present disclosure has clear physical interpretability and is easy for technicians in this field to understand and use.
[0049] Figure 4 It is an exemplary schematic diagram of an image processing model based on a weighted adaptive convolutional dictionary according to an embodiment of the present disclosure.
[0050] For CT images with metal artifacts, the non-metal area can be decomposed into the following model:
[0051] I⊙Y=I⊙X+I⊙A, (1)
[0052] in, is a CT image with metal artifacts; H and W are the height and width of the image respectively; X is the clean CT image to be restored; I is a non-metal region mask, whose elements are {0, 1}, where 1 represents a non-metal region; A is a metal artifact. According to the embodiment of the present disclosure, the non-metal region mask I may be known, which is used to focus the mathematical model to be solved on the non-metal domain, and no longer pay attention to the solution of the metal domain.
[0053] It should be understood that for different CT images with metal artifacts, the metal artifacts A contained therein basically present a common or similar pattern, that is, a stripe structure with a similar shape. At the same time, due to the mutual influence between normal human tissue and metal artifacts, in different CT images with metal artifacts, the artifact patterns contained therein have some specific properties, such as pixel intensity.
[0054] Considering the above characteristics of CT images with metal artifacts, according to the embodiment of the present disclosure, for each image with metal artifacts to be processed (Y), a basic artifact dictionary pre-learned from existing samples is used to obtain the metal artifacts. To construct an adaptive convolution kernel for this image And extract the artifact features of the image Then the adaptive convolution kernel based on the image Artifact characteristics The metal artifact A of the image is obtained, thereby realizing the extraction of metal artifacts. It should be understood that the basic artifact dictionary It can be either an existing known basic artifact dictionary or a basic artifact dictionary obtained by training and synthesizing existing samples.
[0055] According to an embodiment of the present disclosure, a weighted convolution dictionary model may be used to encode and model the metal artifact A:
[0056]
[0057] in, is a sample-invariant dictionary consisting of d convolution kernels, representing the common patterns of different metal artifacts. In short, A common database representing different metal artifact types in all CT images with metal artifacts; is the weighting coefficient that varies with the sample; represents a specific convolution kernel, which represents the recurring pattern of a metal artifact, and p×p is the size of the convolution kernel; is the feature layer, representing the location where the local pattern recurs; N is the number of specific convolution kernels used to encode A; It is a two-dimensional planar convolution operation, where both p and d are positive integers.
[0058] also, and
[0059]
[0060] That is, the basic artifact dictionary is a convolution dictionary that does not vary with samples. The method includes a first number (d) of artifact convolution kernels. A second number of adaptive convolution kernels for the image sample can be determined by using a plurality of artifact convolution kernels in the basic artifact dictionary and a weighting coefficient (K) that varies with the sample.
[0061] Substituting equation (2) into equation (1), we can obtain the model of the non-metal area corresponding to the final CT image with metal artifacts:
[0062]
[0063] Figure 5A The schematic structure of the adaptive convolution dictionary network according to the embodiment of the present disclosure is shown, which includes T-level networks. In each level of the network, the weight coefficient K, the feature layer And the clean CT image X is updated. Figure 5B The schematic structure of each level of the network according to the embodiment of the present disclosure is shown.
[0064] The following will first combine Figure 4 , 5A 5B to describe the image processing method according to an embodiment of the present disclosure, and then describe Figure 5A and Figure 5B Mathematical model of the adaptive convolutional dictionary network.
[0065] Figure 3B 3 is a schematic flowchart 320 showing a model use process of an image processing method for artifact removal according to an embodiment of the present disclosure.
[0066] In step S321, an input image to be processed is obtained.
[0067] According to an embodiment of the present disclosure, the input image to be processed may be an image obtained by applying the corresponding image mask (I) to the original CT image, i.e., I⊙Y. Optionally, the input image to be processed may be an image with artifacts, or an image with artifacts that has been preprocessed (e.g., denoising, normalization). It should be noted that the image mask (I) in the present disclosure is a mask of the area to be studied or the area of interest.
[0068] In step S322, the input image is processed using an adaptive convolutional dictionary network to obtain a processed image with artifacts removed.
[0069] The adaptive convolution dictionary network is trained based on an artifact database (D) and includes a T-level network (e.g. Figure 5AAs shown), wherein the first-level network obtains the first-level image features and the first-level artifact-removed image output by the first-level network based on the input image; the t-th level network obtains and outputs the t-th level image features and the t-th level artifact-removed image output by the t-th level network at least partially based on the t-1-th level image features and the t-1-th level artifact-removed image output by the t-th level network, wherein t is greater than 1 and less than or equal to T; the T-th level network obtains and outputs the T-th level artifact-removed image output by the T-th level network as the processed image after artifact removal based on the T-1-th level image features and the T-1-th level artifact-removed image output by the T-th level network. The processed image after artifact removal can be output to the user through a display or through film, drawings, etc.
[0070] According to an embodiment of the present disclosure, the adaptive convolution dictionary network includes a basic artifact dictionary, which is a convolution dictionary that does not change with the input image and includes a first number of artifact convolution kernels. In the t-th level network, the t-th level weighting coefficient of the t-th level network is first determined, and then the second number of adaptive convolution kernels for the t-th level network is determined by the first number of artifact convolution kernels in the basic artifact dictionary and the t-th level weighting coefficient; then the t-th level artifact removed image is determined based on the second number of adaptive convolution kernels and the image features of the t-th level network.
[0071] According to an embodiment of the present disclosure, each level of the network may include a weighted coefficient update network, an image feature update network, and an artifact removal image update network, wherein the weighted coefficient update network, the image feature update network, and the artifact removal image update network include a residual network structure and a normalization processing layer. It should be understood that the network structure of the weighted coefficient update network, the image feature update network, and the artifact removal image update network can be diverse, and the network structure is usually related to the dimensional characteristics of the solution variable corresponding to the network. For example, the weighted coefficient update network solves the variable as a coefficient, so it usually includes a linear layer, while the image feature update network and the artifact removal image update network solve the variable as a two-dimensional image, so it usually includes a convolution layer.
[0072] According to an embodiment of the present disclosure, the artifact may be a metal artifact, the input image to be processed may be a CT image with metal artifacts, and the image mask (I) may be a non-metal area mask corresponding to the CT image with metal artifacts. In this embodiment, each level of the network may include a weight coefficient update network, a metal artifact image feature update network, and a metal artifact removal image update network, wherein the weight coefficient update network, the metal artifact image feature update network, and the metal artifact removal image update network include a residual network structure; and the weight coefficient update network includes: a linear layer, a rectified linear unit (Rectified Linear Unit, ReLU) layer, a cross-link layer, and a batch normalization (Batch Normalization, BN) layer; the metal artifact image feature update network includes: a convolution layer, a BN layer, a ReLU layer, and a cross-link layer; the metal artifact removal image update network includes: a convolution layer, a BN layer, a ReLU layer, and a cross-link layer.
[0073] Figure 3A 3 is a schematic flowchart 310 showing a neural network training process of an image processing method for artifact removal according to an embodiment of the present disclosure.
[0074] In step S311, a training data set for training a neural network is established, wherein the training data set includes multiple groups of image samples, each group of image samples includes an image with artifacts (Y) and a corresponding image without artifacts (X) and an image mask (I).
[0075] According to the embodiment of the present disclosure, a disclosed image library and different types of metal masks can be used to synthesize artifacts according to a data simulation process, and an image with artifacts and a metal mask (M mask) corresponding to the image with artifacts can be combined. e ) as training data. For example, in the application scenario of removing artifacts from CT images with metal artifacts, the public DeepLesion image library and different types of metal masks can be used to synthesize metal artifacts according to the data simulation process, and the CT images with metal artifacts and metal masks can be used as training data. It should be understood that the different types of masks or different types of metal masks are used to set the size and shape of the implant in the CT image, and the corresponding artifact shape can be simulated based on the size and shape of the implant. Metal mask (M e ) has a corresponding relationship with the non-metallic area mask (I), and can be deduced from each other, that is, M e +I=1.
[0076] According to an embodiment of the present disclosure, in order to process sample data in a larger data range, the establishment of a training data set for training a neural network may further include: normalizing the pixel values of the image with artifacts (Y).
[0077] In addition, in order to obtain diversity of sample data, the establishment of a training data set for training a neural network may also include: randomly cropping the image with artifacts (Y) to obtain image blocks, and randomly flipping the image blocks according to a predetermined probability.
[0078] For example, the numerical range of the image with artifacts in the training data can be clipped (for example, values that are too large or negative are removed so that the unnecessary numerical range is no longer retained, but the necessary tissue information is not lost), and then normalized so that the pixel value of the image is stuck in the threshold [0,1]. Optionally, the normalized data can be converted to the [0,255] range to facilitate computer processing.
[0079] According to an embodiment of the present disclosure, each training image and the corresponding mask can also be randomly cropped to form a smaller image block (for example, an image block of 64x64 pixels), and then randomly horizontally mirrored and vertically mirrored with a predetermined probability (for example, 0.5) to obtain more diverse training sample data.
[0080] In step S312, for at least one group of image samples among the multiple groups of image samples, an adaptive convolutional dictionary network is used to perform artifact removal processing on the image with artifacts (Y) to obtain a processed image.
[0081] According to the embodiment of the present disclosure, Figure 4 As shown, the adaptive convolution dictionary network may include a basic artifact dictionary, which is a convolution dictionary that does not vary with samples and includes a first number of artifact convolution kernels, and a second number of adaptive convolution kernels for the image sample may be determined by combining a plurality of artifact convolution kernels in the basic artifact dictionary with a weighting coefficient that varies with samples. Moreover, an artifact image in the image with artifacts may be determined by convolution of the second number of adaptive convolution kernels with the image features of the image with artifacts, and the artifact image may be removed from the image with artifacts to obtain the processed image.
[0082] In step S313, the adaptive convolutional dictionary network is iteratively trained based on the artifact-free image (X) and the processed image, and the objective function processed by the image mask (I) to optimize the network parameters of the adaptive convolutional dictionary network.
[0083] According to an embodiment of the present disclosure, the objective function may be a loss objective function constructed based on the artifact-free image (X) and the processed image, wherein the iterative training of the adaptive convolutional dictionary network based on the artifact-free image (X) and the processed image, and the objective function to optimize the network parameters of the adaptive convolutional dictionary network also includes: calculating the loss objective function and back-propagating the result thereof to the adaptive convolutional dictionary network, and optimizing the network parameters of the adaptive convolutional dictionary network based on an adaptive moment estimation (Adam) algorithm.
[0084] According to an embodiment of the present disclosure, the objective function may also be a loss objective function constructed based on the artifact-free image (X) and the processed image and processed by the image mask (I).
[0085] According to an embodiment of the present disclosure, the first number of artifact convolution kernels indicates an artifact pattern, and the image feature indicates a location of the artifact pattern, wherein the adaptive convolution dictionary network includes a T-level network, wherein, in the t-th level network, the weighted coefficients and image features output by the t-1-th level network are updated using an iterative update rule based on proximal gradient descent to obtain the weighted coefficients and image features of the t-th level network, wherein t is an integer greater than 1 and less than or equal to T
[0086] According to an embodiment of the present disclosure, each level of the network may include a weighted coefficient update network, an image feature update network, and an artifact removal image update network, wherein the weighted coefficient update network, the image feature update network, and the artifact removal image update network include a residual network structure and a normalization processing layer. It should be understood that the network structure of the weighted coefficient update network, the image feature update network, and the artifact removal image update network can be diverse, and the network structure is usually related to the dimensional characteristics of the solution variable corresponding to the network. For example, the weighted coefficient update network solves the variable as a coefficient, so it usually includes a linear layer, while the image feature update network and the artifact removal image update network solve the variable as a two-dimensional image, so it usually includes a convolution layer.
[0087] According to an embodiment of the present disclosure, the image processing method for removing artifacts may further include: after the training is completed, testing the adaptive convolutional dictionary network to evaluate the effect of image processing. The testing of the adaptive convolutional dictionary network includes: preprocessing the image with artifacts to be tested and inputting it into the adaptive convolutional dictionary network; using the adaptive convolutional dictionary network, processing the image with artifacts to be tested to obtain a processed image with artifacts removed.
[0088] According to an embodiment of the present disclosure, the artifact may be a metal artifact, and the image with the artifact may be a CT image with the metal artifact. Therefore, the training data set includes multiple groups of CT image samples, each group of image samples includes a CT image with metal artifacts and a corresponding CT image without metal artifacts; the adaptive convolution dictionary network includes a basic metal artifact dictionary, the basic metal artifact dictionary is a metal artifact convolution dictionary that does not change with the CT image sample and includes a first number of metal artifact convolution kernels, and a second number of adaptive convolution kernels for the CT image sample is determined by using multiple metal artifact convolution kernels in the basic metal artifact dictionary and weighting coefficients that change with the CT image sample, wherein the first number of metal artifact convolution kernels indicates a metal artifact pattern, and the metal artifact image feature indicates the location of the metal artifact pattern.
[0089] According to an embodiment of the present disclosure, in an application scenario for removing artifacts from CT images, each level of the network may include a weighted coefficient update network, a metal artifact image feature update network, and a metal artifact removal image update network, wherein the weighted coefficient update network, the metal artifact image feature update network, and the metal artifact removal image update network include a residual network structure; and the weighted coefficient update network includes: a linear layer, a rectified linear unit (Rectified Linear Unit, ReLU) layer, a cross-link layer, and a batch normalization (Batch Normalization, BN) layer; the metal artifact image feature update network includes: a convolutional layer, a BN layer, a ReLU layer, and a cross-link layer; the metal artifact removal image update network includes: a convolutional layer, a BN layer, a ReLU layer, and a cross-link layer.
[0090] Continue below Figure 4 The mathematical model for removing metal artifacts from CT images with metal artifacts is used as an example. Figure 4 , 5A and Figure 5B To describe the mathematical model of the image processing method for artifact removal described in the present disclosure and the structure of the adaptive convolutional dictionary network.
[0091] According to the embodiment of the present disclosure, a common database representing different types of metal artifacts in all CT images with metal artifacts can be It is constructed as a convolutional layer, based on which an adaptive convolutional dictionary network is constructed, and the optimized parameters of the adaptive convolutional dictionary network are obtained through end-to-end training of the training data set.
[0092] For the mathematical model shown in equation (4) above:
[0093]
[0094] The solution goal is to estimate K from Y. and X, the corresponding optimization problem is:
[0095]
[0096] subject to‖K n ‖2=1,n=1,2,…,N (5)
[0097] Among them, α, β and γ are trade-off parameters, f1(·), f2(·) and f3(·) are regularization terms, representing the weighting coefficient K, feature layer, and and the prior structure of the clean CT image X, which can be designed as a neural network module for solution.
[0098] To solve the optimization problem in (5), the proximal gradient technique can be used to alternately update the weight coefficient K and the feature layer And the clean CT image X. The details are as follows:
[0099] Update K:
[0100] In the (t+1)th iteration, K can be updated as:
[0101]
[0102] subject to‖K n ‖2=1,n=1,2,…,N (6)
[0103] The corresponding quadratic approximation is:
[0104]
[0105]
[0106] Where Ω={K|‖K n ‖2=1,n=1,2,…,N};
[0107] η1 is the update step size, which can be deduced as:
[0108]
[0109] in, It is a depth convolution operation; Indicates that the tensor is expanded in the third dimension; vec(·) indicates a vectorized operation.
[0110] Equation (7) can be equivalently written as:
[0111]
[0112] For the general prior term f1(·), equation (9) can be written as:
[0113]
[0114] in,
[0115]
[0116] is the proximal operator, which is related to the regularization term f1(·); Ω can be obtained by A normalization operation is introduced to achieve this.
[0117] renew
[0118] Similar to the update of K, in the (t+1)th iteration, Can be updated to:
[0119]
[0120] Among them, η2 is the update step size, For the general prior term f2(·), equation (11) can be written as:
[0121]
[0122] in, is the proximal operator, which is related to the regularization term f2(·);
[0123] is the transposed convolution operation.
[0124] Update X:
[0125] Given K (t+1) and Can be updated to:
[0126]
[0127] in, Furthermore, we can get the update rule of X as:
[0128]
[0129] in, is the proximal operator, which is related to the regularization term f3(·).
[0130] By expanding the update formulas (10)(12)(14), we can eventually build a complete adaptive convolutional dictionary network (ACDNet), in which each network has good physical interpretation.
[0131] exist Figure 5A FIG. 2 shows a schematic structure of an adaptive convolutional dictionary network according to an embodiment of the present disclosure, which includes T-level networks. In each level of the network, the weight coefficient K, the feature layer And the processed CT image X is updated. Figure 5B The schematic structure of each level of the network according to an embodiment of the present disclosure is shown in FIG.
[0132] According to the embodiment of the present disclosure, Figure 5A The ACDNet shown in the figure is composed of T stages (i.e., T-level networks). In each stage, the corresponding network structure is composed of K-net, and X-net, which are used to implement K, and iterative updates of X.
[0133] Below, explain Figure 5A and Figure 5B The correspondence between the network structure and the above mathematical model.
[0134] exist Figure 5A and Figure 5B in, specifically,
[0135] in It is a residual structure, specifically: linear layer, ReLU layer, linear layer, cross-link layer, and normalization operation layer at dimension d;
[0136] in It consists of 3 residual blocks, each of which includes: convolution layer, BN layer, ReLU layer, convolution layer, BN layer, and cross-link layer;
[0137] in It consists of 3 residual blocks, each of which includes: convolution layer, BN layer, ReLU layer, convolution layer, BN layer, and cross-link layer.
[0138] Depend on Figure 5A It can be seen that the first-level network obtains the first-level image feature M output by the first-level network based on the input image. (1) and the first level artifact removal image X (1)The t-th level network is at least partially based on the t-1th level image feature M output by the t-1th level network (t-1) and the t-1th level artifact-removed image X (t-1) , get and output the t-th level image feature M output by the t-th level network (t) and the t-th level artifact-removed image X (t) , where t is greater than 1 and less than or equal to T; the T-th level network is based on the T-1th level image feature M output by the T-1th level network (T-1) and the T-1th level artifact removal image X (T-1) , get and output the T-th level artifact removal image X output by the T-th level network (T) As the processed image after removing artifacts.
[0139] For each level of the network in the adaptive convolutional dictionary network, the iterative solution process is as follows: Figure 5B As shown. In the t-th level network, the weight coefficients and image features output by the t-1-th level network are updated using an iterative update rule based on proximal gradient descent to obtain the weight coefficients and image features of the t-th level network, where t is an integer greater than 1 and less than or equal to T. K-net, M-net, and X-net are iteratively solved in sequence.
[0140] The adaptive convolutional dictionary network can be iteratively trained based on the artifact-free image (X) and the processed image, and the objective function processed by the image mask (I) to optimize the network parameters of the adaptive convolutional dictionary network. The objective function can be a loss objective function constructed based on the artifact-free image (X) and the processed image, that is,
[0141]
[0142] Among them, μ t is a compromise parameter, and ω1 and ω2 are weights used to balance the losses. For example, in a simulation experiment, μ t =0.1(t=0,1,…,T-1),μ T =1,ω1=ω2=5×10 -4 , T=10.
[0143] According to an embodiment of the present disclosure, the optimization parameters can be updated and solved based on the Adam algorithm (Adaptive moment estimation), including: Convolution Kernel Step size η1, η2 and η3. In each iteration, the prediction result error is calculated and back-propagated to the convolutional neural network model, the gradient is calculated and the parameters of the convolutional neural network model are updated.
[0144] Figure 6 is a schematic flowchart illustrating an image processing process for artifact removal according to an embodiment of the present disclosure.
[0145] like Figure 6 As shown, the image processing process for artifact removal according to an embodiment of the present disclosure includes a neural network training phase and a testing phase.
[0146] In the neural network training stage, the image with artifacts can be preprocessed first to establish a training data set for training the neural network, wherein the training data set includes multiple groups of image samples, each group of image samples includes an image with artifacts (Y) and its corresponding image without artifacts (X) and an image mask (I). Then, based on the preprocessed image samples, the computer iteratively trains ACDNet according to the settings of the neural network training, wherein during the training process, the parameters of ACDNet are updated based on a predetermined objective function and an Adam optimization algorithm. If the predetermined number of iterations is reached during the training process, the trained model is saved, and if the predetermined number of iterations is not reached, the ACDNet training continues.
[0147] It should be understood that the purpose of using the predetermined number of iterations as the criterion for determining that the ACDNet training is complete is to prevent the network from overfitting. Optionally, the optimized image can be output in a visual manner, and the ACDNet training can be stopped after confirming that the optimized image meets the requirements.
[0148] In the testing phase, the input image to be processed and its corresponding image mask (I) are input to the computer, the computer loads the trained model, and obtains the image after removing artifacts through ACDNet forward calculation. The computer can output the image after removing artifacts for user reference.
[0149] Similarly, in actual use, the image processing process is similar to that in the testing phase, so it will not be repeated here.
[0150] Fig. 7A 710 is a schematic diagram showing the composition of an image processing device for artifact removal according to an embodiment of the present disclosure. The device 710 is used in a neural network training process during image processing.
[0151] According to an embodiment of the present disclosure, the image processing apparatus 710 for artifact removal may include: a training data set establishment module 711 , an adaptive convolution dictionary network 712 and a training module 713 .
[0152] Among them, the training data set establishment module 711 can be configured to: establish a training data set for training a neural network, wherein the training data set includes multiple groups of image samples, each group of image samples includes an artifact image (Y) and its corresponding artifact-free image (X) and an image mask (I).
[0153] Optionally, the establishment of a training data set for training a neural network may also include at least one of the following: normalizing pixel values of the image with artifacts (Y); and randomly cropping the image with artifacts (Y) to obtain image blocks, and randomly flipping the image blocks according to a predetermined probability.
[0154] The adaptive convolution dictionary network 712 may be configured to: for at least one group of image samples among the multiple groups of image samples, perform artifact removal processing on the image with artifacts (Y) to obtain a processed image.
[0155] The adaptive convolution dictionary network 712 may include a basic artifact dictionary, which is a convolution dictionary that does not vary with samples and includes a first number of artifact convolution kernels, and a second number of adaptive convolution kernels for the image sample may be determined by combining a plurality of artifact convolution kernels in the basic artifact dictionary with a weighting coefficient that varies with samples. Furthermore, an artifact image in the image with artifacts may be determined by convolution of the second number of adaptive convolution kernels with image features of the image with artifacts, and the artifact image may be removed from the image with artifacts to obtain the processed image.
[0156] The training module 713 may be configured to iteratively train the adaptive convolutional dictionary network based on the artifact-free image (X) and the processed image, and the objective function processed by the image mask (I) to optimize the network parameters of the adaptive convolutional dictionary network.
[0157] According to an embodiment of the present disclosure, the objective function may be a loss objective function constructed based on the artifact-free image (X) and the processed image, wherein the iterative training of the adaptive convolutional dictionary network based on the artifact-free image (X) and the processed image, and the objective function to optimize the network parameters of the adaptive convolutional dictionary network also includes: calculating the loss objective function and back-propagating the result thereof to the adaptive convolutional dictionary network, and optimizing the network parameters of the adaptive convolutional dictionary network based on an adaptive moment estimation (Adam) algorithm.
[0158] According to an embodiment of the present disclosure, the first number of artifact convolution kernels indicates an artifact pattern, and the image feature indicates a location of the artifact pattern, wherein the adaptive convolution dictionary network includes a T-level network, wherein, in the t-th level network, the weighted coefficients and image features output by the t-1-th level network are updated using an iterative update rule based on proximal gradient descent to obtain the weighted coefficients and image features of the t-th level network, wherein t is an integer greater than 1 and less than or equal to T
[0159] According to an embodiment of the present disclosure, each level of the network includes a weighted coefficient update network, an image feature update network, and an artifact removal image update network, wherein the weighted coefficient update network, the image feature update network, and the artifact removal image update network include a residual network structure and a normalization processing layer.
[0160] According to an embodiment of the present disclosure, the image processing device 710 for artifact removal may further include: a testing module 714, configured to: test the adaptive convolutional dictionary network after training is completed, wherein the testing of the adaptive convolutional dictionary network includes: preprocessing the image with artifacts to be tested and inputting it into the adaptive convolutional dictionary network; using the adaptive convolutional dictionary network, processing the image with artifacts to be tested to obtain a processed image with artifacts removed.
[0161] Figure 7B 7 is a schematic diagram showing the composition of an image processing device for artifact removal according to an embodiment of the present disclosure, wherein the device 720 is used for a model use process during image processing.
[0162] According to an embodiment of the present disclosure, the image processing device 720 for removing artifacts may include: an image acquisition module 721 and an image processing module 722 .
[0163] The image acquisition module 721 may be configured to: acquire an input image to be processed.
[0164] According to an embodiment of the present disclosure, the input image to be processed may be an image obtained by applying the original CT image to its corresponding image mask. For example, the image acquisition module 721 may receive the original CT image and the image mask corresponding thereto, and apply the original CT image to its corresponding image mask, thereby acquiring the input image to be processed.
[0165] According to the embodiment of the present disclosure, the input image to be processed may be an image with artifacts, or may be an image with artifacts that has been pre-processed (eg, denoising processing, normalization processing).
[0166] The image processing module 722 may be configured to: process the input image using an adaptive convolutional dictionary network to obtain a processed image with artifacts removed.
[0167] The adaptive convolution dictionary network is trained based on an artifact database (D) and includes T-level networks, wherein the first-level network obtains the first-level image features and the first-level artifact removed image output by the first-level network based on the input image; the t-th level network obtains and outputs the t-th level image features and the t-th level artifact removed image output by the t-th level network at least partially based on the t-1-th level image features and the t-1-th level artifact removed image output by the t-1-th level network, wherein t is greater than 1 and less than or equal to T; the T-th level network obtains and outputs the T-th level artifact removed image output by the T-th level network as the processed image with artifacts removed based on the T-1-th level image features and the T-1-th level artifact removed image output by the T-1-th level network.
[0168] According to an embodiment of the present disclosure, the adaptive convolution dictionary network includes a basic artifact dictionary, which is a convolution dictionary that does not change with the input image and includes a first number of artifact convolution kernels, determines the t-th level weighting coefficient of the t-th level network, determines a second number of adaptive convolution kernels for the t-th level network through multiple artifact convolution kernels in the basic artifact dictionary and the t-th level weighting coefficient; and determines a t-th level artifact removed image based on the second number of adaptive convolution kernels and the image features of the t-th level network.
[0169] In general, various example embodiments of the present disclosure may be implemented in hardware or dedicated circuits, software, firmware, logic, or any combination thereof. Certain aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device. When various aspects of the embodiments of the present disclosure are illustrated or described as block diagrams, flow charts, or using some other graphical representation, it will be understood that the blocks, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuits or logic, general purpose hardware or controllers or other computing devices, or some combination thereof as non-limiting examples.
[0170] For example, the method or device according to the embodiment of the present disclosure may also be implemented by Figure 8 The architecture of the computing device 3000 shown in FIG. Figure 8 As shown, the computing device 3000 may include a bus 3010, one or more CPUs 3020, a read-only memory (ROM) 3030, a random access memory (RAM) 3040, a communication port 3050 connected to a network, an input / output component 3060, a hard disk 3070, etc. The storage device in the computing device 3000, such as the ROM 3030 or the hard disk 3070, may store various data or files used for processing and / or communication of the method provided in the present disclosure and program instructions executed by the CPU. The computing device 3000 may also include a user interface 3080. Of course, Figure 8The architecture shown is only exemplary and can be omitted according to actual needs when implementing different devices. Figure 8 One or more components of a computing device are shown.
[0171] According to yet another aspect of the present disclosure, a computer-readable storage medium is provided. Fig. 9 A schematic diagram 4000 of a storage medium according to the present disclosure is shown.
[0172] like Fig. 9 As shown, a computer storage medium 4020 stores computer readable instructions 4010. When the computer readable instructions 4010 are executed by the processor, the method according to the embodiment of the present disclosure described with reference to the above figures can be executed. The computer readable storage medium in the embodiment of the present disclosure can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. The non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM) or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous connection dynamic random access memory (SLDRAM) and direct memory bus random access memory (DR RAM). It should be noted that memory of the methods described herein is intended to include, but is not limited to, these and any other suitable types of memory. It should be noted that memory of the methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.
[0173] The embodiments of the present disclosure also provide a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the method according to the embodiments of the present disclosure.
[0174] In summary, the embodiments of the present disclosure provide an image processing method, device, computer program product and storage medium for artifact removal. The image processing method for artifact removal provided by the present disclosure includes: establishing a training data set for training a neural network, wherein the training data set includes multiple groups of image samples, each group of image samples includes an image with artifacts (Y) and a corresponding image without artifacts (X) and an image mask (I); for at least one group of image samples among the multiple groups of image samples, using an adaptive convolutional dictionary network, performing artifact removal processing on the image with artifacts (Y) to obtain a processed image, iteratively training the adaptive convolutional dictionary network based on the artifact-free image (X) and the processed image, and the objective function processed by the image mask (I); The method comprises the steps of: training the image processing method and the processing method of the adaptive convolution dictionary network to optimize the network parameters of the adaptive convolution dictionary network, wherein the adaptive convolution dictionary network comprises a basic artifact dictionary, the basic artifact dictionary is a convolution dictionary that does not vary with samples and comprises a first number of artifact convolution kernels, and a second number of adaptive convolution kernels for the image sample are determined by combining a plurality of artifact convolution kernels in the basic artifact dictionary with a weighting coefficient that varies with samples, wherein an artifact image in the image with artifacts is determined by convolution of the second number of adaptive convolution kernels with image features of the image with artifacts, and the artifact image is removed from the image with artifacts to obtain the processed image.
[0175] The image processing method disclosed in the present invention can perform artifact removal processing based only on the image to be processed without the need to obtain the chord diagram of the image additionally. The adaptive convolutional dictionary network used in the present invention makes full use of the prior structure of the artifact image, and the artifact removal effect is better and the model generalization is stronger. In addition, the mathematical model used in the image processing method disclosed in the present invention has a clear physical meaning and strong explanatory power, and the physical meaning of each network module is clearer, which facilitates the understanding and application of those skilled in the art. The image processing method disclosed in the present invention can simply and effectively remove artifacts in an image to obtain a clearer image that is not interfered with by artifacts.
[0176] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, a program segment, or a part of a code, and the module, program segment, or a part of the code contains at least one executable instruction for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0177] The present disclosure uses specific words to describe the embodiments of the present disclosure. For example, "first / second embodiment", "one embodiment", and / or "some embodiments" refer to a certain feature, structure or characteristic related to at least one embodiment of the present disclosure. Therefore, it should be emphasized and noted that "one embodiment" or "an embodiment" or "an alternative embodiment" mentioned twice or more in different locations in this specification does not necessarily refer to the same embodiment. In addition, certain features, structures or characteristics in one or more embodiments of the present disclosure may be appropriately combined.
[0178] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the invention belongs. It should also be understood that terms such as those defined in common dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology and should not be interpreted in an idealized or extremely formal sense, unless explicitly defined as such herein.
[0179] The above is an explanation of the present invention and should not be considered as a limitation thereof. Although several exemplary embodiments of the present invention have been described, it will be readily appreciated by those skilled in the art that many modifications may be made to the exemplary embodiments without departing from the novel teachings and advantages of the present invention. Therefore, all such modifications are intended to be included within the scope of the present invention as defined in the claims. It should be understood that the above is an explanation of the present invention and should not be considered as being limited to the specific embodiments disclosed, and modifications to the disclosed embodiments and other embodiments are intended to be included within the scope of the appended claims. The present invention is defined by the claims and their equivalents.
Claims
1. An image processing method for artifact removal, comprising: Establishing a training data set for training a neural network, wherein the training data set includes multiple groups of image samples, each group of image samples includes an image with artifacts (Y) and a corresponding image without artifacts (X) and an image mask (I); For at least one set of image samples in the plurality of sets of image samples, Using an adaptive convolutional dictionary network, the artifact image (Y) is subjected to artifact removal processing to obtain a processed image. Iteratively training the adaptive convolutional dictionary network based on the artifact-free image (X) and the processed image, and the objective function processed by the image mask (I) to optimize the network parameters of the adaptive convolutional dictionary network, The adaptive convolution dictionary network includes a basic artifact dictionary, which is a convolution dictionary that does not vary with samples and includes a first number of artifact convolution kernels, and a second number of adaptive convolution kernels for the image sample is determined by using a plurality of artifact convolution kernels in the basic artifact dictionary and a weighting coefficient that varies with samples, The artifact image in the image with artifacts is determined by convolving the second number of adaptive convolution kernels with the image features of the image with artifacts, and the artifact image is removed from the image with artifacts to obtain the processed image.
2. The image processing method according to claim 1, wherein: The first number of artifact convolution kernels indicates an artifact pattern, and the image feature indicates a location of the artifact pattern. The adaptive convolutional dictionary network includes T-level networks, wherein, in the t-th level network, the weighting coefficients and image features output by the t-1-th level network are updated using an iterative update rule based on proximal gradient descent to obtain the weighting coefficients and image features of the t-th level network, wherein t is an integer greater than 1 and less than or equal to T.
3. The image processing method according to claim 2, wherein: Each level of the network includes a weighted coefficient update network, an image feature update network, and an artifact removal image update network, where: The weighted coefficient update network, the image feature update network, and the artifact removal image update network include a residual network structure and a normalization processing layer.
4. The image processing method according to claim 1, wherein: The step of establishing a training data set for training a neural network further includes at least one of the following: Normalizing the pixel values of the image with artifacts (Y); and The image with artifacts (Y) is randomly cropped to obtain image blocks, and the image blocks are randomly flipped according to a predetermined probability.
5. The image processing method according to claim 1, wherein: The objective function is a loss objective function constructed based on the artifact-free image (X) and the processed image, Wherein, the iterative training of the adaptive convolutional dictionary network based on the artifact-free image (X) and the processed image, and the objective function to optimize the network parameters of the adaptive convolutional dictionary network further includes: The loss objective function is calculated and its result is back-propagated to the adaptive convolutional dictionary network, and the network parameters of the adaptive convolutional dictionary network are optimized based on the adaptive moment estimation (Adam) algorithm.
6. The image processing method according to claim 1, further comprising: After the training is completed, the adaptive convolutional dictionary network is tested. Wherein, the testing of the adaptive convolutional dictionary network includes: Preprocessing the artifact image to be tested and inputting it into the adaptive convolution dictionary network; The image with artifacts to be tested is processed by using an adaptive convolutional dictionary network to obtain a processed image with artifacts removed.
7. The image processing method according to claim 1, wherein: The artifact is a metal artifact, and the image with artifact is a CT image with metal artifact. The training data set includes multiple groups of CT image samples, each group of image samples includes a CT image with metal artifacts and a corresponding CT image without metal artifacts and a non-metal area mask; The adaptive convolution dictionary network includes a basic metal artifact dictionary, which is a metal artifact convolution dictionary that does not change with CT image samples and includes a first number of metal artifact convolution kernels, and a second number of adaptive convolution kernels for the CT image samples are determined by using multiple metal artifact convolution kernels in the basic metal artifact dictionary and weighting coefficients that change with CT image samples, wherein the first number of metal artifact convolution kernels indicates a metal artifact pattern, and the metal artifact image feature indicates a location where the metal artifact pattern is located.
8. The image processing method according to claim 7, wherein: Each level of the network includes a weighted coefficient update network, a metal artifact image feature update network, and a metal artifact removal image update network, wherein: The weight coefficient update network, the metal artifact image feature update network, and the metal artifact removal image update network include a residual network structure; and The weighted coefficient update network includes: linear layer, rectified linear unit (ReLU) layer, cross-link layer, and batch normalization (BN) layer; The metal artifact image feature update network includes: convolution layer, BN layer, ReLU layer, and cross-link layer; The metal artifact removal image update network includes: convolution layer, BN layer, ReLU layer, and cross-link layer.
9. An image processing method for artifact removal, comprising: Get the input image to be processed; The input image is processed using an adaptive convolutional dictionary network to obtain a processed image with artifacts removed, The adaptive convolution dictionary network is trained based on an artifact database (D) and includes T-level networks, wherein the first-level network obtains the first-level image features and the first-level artifact removed image output by the first-level network based on the input image; the t-th level network obtains and outputs the t-th level image features and the t-th level artifact removed image output by the t-th level network at least partially based on the t-1-th level image features and the t-1-th level artifact removed image output by the t-1-th level network, wherein t is greater than 1 and less than or equal to T; the T-th level network obtains and outputs the T-th level artifact removed image output by the T-th level network as the processed image with artifacts removed based on the T-1-th level image features and the T-1-th level artifact removed image output by the T-1-th level network.
10. The image processing method according to claim 9, wherein: The adaptive convolution dictionary network includes a basic artifact dictionary, which is a convolution dictionary that does not vary with the input image and includes a first number of artifact convolution kernels, Determine a t-th level weighting coefficient of a t-th level network, determine a second number of adaptive convolution kernels for the t-th level network through a plurality of artifact convolution kernels in the basic artifact dictionary and the t-th level weighting coefficient; and determine a t-th level artifact removed image based on the second number of adaptive convolution kernels and image features of the t-th level network.
11. The image processing method according to claim 9, wherein: Each level of the network includes a weighted coefficient update network, an image feature update network, and an artifact removal image update network, where: The weighted coefficient update network, the image feature update network, and the artifact removal image update network include a residual network structure and a normalization processing layer.
12. The image processing method according to claim 9, wherein: The input image to be processed is an image to which an image mask (I) is applied.
13. The image processing method according to claim 12, wherein: The artifact is a metal artifact, the input image to be processed is a CT image with metal artifacts, and the image mask (I) is a non-metal area mask corresponding to the CT image with metal artifacts. Each level of the network includes a weighted coefficient update network, a metal artifact image feature update network, and a metal artifact removal image update network, wherein: The weight coefficient update network, the metal artifact image feature update network, and the metal artifact removal image update network include a residual network structure; and The weighted coefficient update network includes: linear layer, rectified linear unit (ReLU) layer, cross-link layer, and batch normalization (BN) layer; The metal artifact image feature update network includes: convolution layer, BN layer, ReLU layer, and cross-link layer; The metal artifact removal image update network includes: convolution layer, BN layer, ReLU layer, and cross-link layer.
14. An image processing device for removing artifacts, comprising: A training data set establishment module is configured to: establish a training data set for training a neural network, wherein the training data set includes a plurality of groups of image samples, each group of image samples includes an image with artifacts (Y) and a corresponding image without artifacts (X) and an image mask (I); The adaptive convolution dictionary network is configured to: for at least one group of image samples among the multiple groups of image samples, perform artifact removal processing on the image with artifacts (Y) to obtain a processed image; A training module is configured to: iteratively train the adaptive convolutional dictionary network based on the artifact-free image (X) and the processed image, and the objective function processed by the image mask (I), so as to optimize the network parameters of the adaptive convolutional dictionary network; The adaptive convolution dictionary network includes a basic artifact dictionary, which is a convolution dictionary that does not vary with samples and includes a first number of artifact convolution kernels, and a second number of adaptive convolution kernels for the image sample is determined by using a plurality of artifact convolution kernels in the basic artifact dictionary and a weighting coefficient that varies with samples, The artifact image in the image with artifacts is determined by convolving the second number of adaptive convolution kernels with the image features of the image with artifacts, and the artifact image is removed from the image with artifacts to obtain the processed image.
15. An image processing device for removing artifacts, comprising: The image acquisition module is configured to: acquire an input image to be processed; The image processing module is configured to: process the input image using an adaptive convolutional dictionary network to obtain a processed image with artifacts removed; The adaptive convolution dictionary network is trained based on an artifact database (D) and includes T-level networks, wherein the first-level network obtains the first-level image features and the first-level artifact removed image output by the first-level network based on the input image; the t-th level network obtains and outputs the t-th level image features and the t-th level artifact removed image output by the t-th level network at least partially based on the t-1-th level image features and the t-1-th level artifact removed image output by the t-1-th level network, wherein t is greater than 1 and less than or equal to T; the T-th level network obtains and outputs the T-th level artifact removed image output by the T-th level network as the processed image with artifacts removed based on the T-1-th level image features and the T-1-th level artifact removed image output by the T-1-th level network.
16. A computer program product, comprising computer software code, which is used to implement the method according to any one of claims 1 to 13 when executed by a processor.
17. A computer-readable storage medium having computer-executable instructions stored thereon, wherein the instructions are used to implement the method according to any one of claims 1 to 13 when executed by a processor.
