Rotation equivariant constraint relaxation method and device based on convolutional network framework

Abstract

The invention discloses a rotation equivariant constraint relaxation method and device based on a convolutional network framework, and the method comprises the steps: setting a hyper-parameter, processing an original image pair into a training image pair based on the hyper-parameter, and forming a training image set; designing a relaxed rotation equivariant constraint convolutional layer based on the rotation equivariant characteristic of the image; replacing a convolutional layer of the deep learning network model with a relaxed rotation isovariant constraint convolutional layer to construct a rotation isovariant relaxed network; iteratively training the rotation isovariant relaxation network by using the training image set until a stop condition is reached, and obtaining a trained rotation isovariant relaxation network; and inputting the degraded image into the trained rotation isovariant relaxation network to obtain a recovered image. The problem that an existing deep learning network model is not high in applicability to imperfect symmetric images is solved, and the method has excellent generalization ability and application potential.

Description

Technical Field

[0001] The present application relates to the field of image processing and deep learning technology, and in particular to a method and device for relaxing rotation equivariant constraints based on a convolutional network framework. Background Art

[0002] In many image processing tasks, there are various symmetries in the local features and global semantic representation of images, such as translation, rotation, and reflection symmetry. Symmetries in data can usually be embedded in the neural network architecture to provide inductive biases, so that the model can obtain better generalization capabilities. The key to the success of deep learning lies in the characterization of symmetries and invariances in data. For example, convolutional networks use window sliding to characterize the translation invariance in data; group equivariant convolutional networks further generalize convolutional networks and characterize more symmetric properties in data, such as rotation and reflection symmetry.

[0003] However, existing rotational equivariant networks assume that the symmetry in the data is perfect, and the network learns to approximate a function that is strictly invariant or equivariant under a given symmetry group, but real-world data is rarely completely symmetrical, so strictly equivariant or invariant networks may be too limited in some image processing tasks, for example, the numbers "6" and "9" become difficult to distinguish under rotational invariance. Therefore, appropriately relaxing the equivariant constraints and allowing the model to adaptively learn the symmetry and equivariance from the training data will further improve the performance.

[0004] Although existing technologies have explored how to construct relaxed equivariant networks, previous studies have shown that the translation invariance of convolutional neural networks is too strict and cannot encode absolute coordinate information. Existing methods break the equivariance of convolutional layers by explicitly adding absolute coordinate information to the network or relaxing the degree of weight sharing of convolution kernels in space. These methods have significantly improved the accuracy of high-level visual tasks such as image classification, but require many additional parameters and focus on relaxing translation symmetry constraints. Summary of the invention

[0005] The embodiments of the present application solve the problem that existing deep learning network models are not very applicable to non-perfectly symmetrical images by providing a method and device for relaxing rotation equivariant constraints based on a convolutional network framework.

[0006] In a first aspect, an embodiment of the present application provides a rotation equivariant constraint relaxation method based on a convolutional network framework, comprising: setting hyperparameters, and processing original image pairs into training image pairs based on the hyperparameters to form a training image set; designing a relaxed rotation equivariant constraint convolution layer based on the rotation equivariant characteristics of the image; replacing the convolution layer of a deep learning network model with the relaxed rotation equivariant constraint convolution layer to construct a rotation equivariant relaxation network; iteratively training the rotation equivariant relaxation network using the training image set until a stopping condition is reached to obtain the trained rotation equivariant relaxation network; and inputting a degraded image into the trained rotation equivariant relaxation network to obtain a restored image.

[0007] In combination with the first aspect, in a possible implementation method, processing the original image pair into a training image pair based on the hyperparameter includes: rotating the original images in the original image pair according to the hyperparameter; wherein the original image pair includes a degraded image and its corresponding true value image; and splicing the degraded image before and after rotation with the true value image to obtain a spliced ​​degraded image and a spliced ​​true value image to constitute the training image pair.

[0008] In combination with the first aspect, in a possible implementation manner, the relaxed rotation equivariant constrained convolutional layer is as follows:

[0009]

[0010] Where, L layer represents the regularization term calculated in the relaxed rotation equivariant constrained convolutional layer, φ(·) represents the convolutional mapping, Indicated in The rotation matrix of the randomly selected angle in is, F represents the feature space obtained after convolution mapping, Represents rotation and channel shift operations, X represents the input training image pair, and R represents the action space of the rotation operation, that is, the input image domain.

[0011] In combination with the first aspect, in a possible implementation method, the iterative training of the rotation equivariant relaxation network using the training image set until a stopping condition is reached includes: determining the current hyperparameters of this round of iteration, and iteratively performing a network training step based on the rotation equivariant relaxation network until the stopping condition is reached; the network training step includes: inputting the current hyperparameters and the training image pairs into the rotation equivariant relaxation network; calculating the relaxed equivariant regularization term of the rotation equivariant relaxation network; determining the total loss of the rotation equivariant relaxation network based on the relaxed equivariant regularization term; calculating the gradient between the total loss and the current hyperparameters; updating the current hyperparameters based on the gradient, and performing the network training step.

[0012] In combination with the first aspect, in a possible implementation manner, a method for calculating the relaxed equivariant regularization term is as follows:

[0013]

[0014] Where, L equi represents the relaxed equivariant regularization term, φ (1) represents the first l layers of the rotationally equivariantly relaxed network, Indicated in The rotation matrix of the randomly selected angle in , φ(·) represents the convolution mapping, and F represents the feature space obtained after the convolution mapping. represents the rotation and channel shift operation, l represents the first l layers of the network, X represents the input training image pair, and R represents the action space of the rotation operation, that is, the input image domain. Indicates taking the average.

[0015] In combination with the first aspect, in a possible implementation, the total loss is calculated as follows:

[0016] L=||softENet(I)-I target ||2+αL equi ;

[0017] Where L represents the total loss, softENet represents the rotation equivariant relaxation network, I represents the input training image pair, and I target represents the label corresponding to the input training image pair, α represents the hyperparameter controlling the strength of the equivariant constraint, and L equi represents the relaxed equivariant regularization term.

[0018] In combination with the first aspect, in a possible implementation, the hyperparameters include element parameters and rotation parameters; the element parameters are used to determine the rotation method of the original image pair; and the rotation parameters are used to determine the rotation angle of the original image pair.

[0019] In the second aspect, an embodiment of the present application provides a rotation equivariant constraint relaxation device based on a convolutional network framework, comprising: a pre-processing module, used to set hyperparameters, and based on the hyperparameters, process the original image pair into a training image pair to form a training image set; a design module, used to design a relaxed rotation equivariant constrained convolutional layer based on the rotation equivariant characteristics of the image; a construction module, used to replace the convolutional layer of the deep learning network model with the relaxed rotation equivariant constrained convolutional layer to construct a rotation equivariant relaxation network; a training module, used to iteratively train the rotation equivariant relaxation network using the training image set until the stopping condition is reached to obtain the trained rotation equivariant relaxation network; a processing module, used to input the degraded image into the trained rotation equivariant relaxation network to obtain a restored image.

[0020] In a third aspect, an embodiment of the present application provides a device, comprising: a processor; a memory for storing processor-executable instructions; when the processor executes the executable instructions, it implements the method described in the first aspect or any possible implementation method of the first aspect.

[0021] In a fourth aspect, an embodiment of the present application provides a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium includes a medium for storing a computer program or instructions, which, when executed, enables the method described in the first aspect or any possible implementation method of the first aspect to be implemented.

[0022] One or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages:

[0023] The embodiment of the present application, by designing a relaxed rotation equivariant constrained convolutional layer, can enable the network to adaptively learn equivariance that conforms to the data characteristics from the training data set; by replacing the convolutional layer to construct a rotation equivariant relaxed network, the application cost can be reduced, and the problem of low applicability of existing deep learning network models to non-perfectly symmetrical images can be effectively solved. Furthermore, on the basis of not actually changing the existing network structure, by only replacing the original convolutional layer, better performance can be achieved in tasks such as image deraining and medical image artifact removal, and it has excellent generalization ability and application potential. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0025] Figure 1 A flowchart of a rotation equivariant constraint relaxation method based on a convolutional network framework provided in an embodiment of the present application;

[0026] Figure 2 A schematic diagram of the structure of a rotation equivariant constraint relaxation device based on a convolutional network framework provided in an embodiment of the present application;

[0027] Figure 3 A schematic diagram of calculating the rotation equivariant regularization of the relaxed rotation equivariant constrained convolutional layer provided in an embodiment of the present application;

[0028] Figure 4 A comparison chart of rain removal effects of different rain removal models provided in the embodiments of the present application;

[0029] Figure 5 A comparison chart of the equivariant error visualization results of the relaxed rotation equivariant constrained convolution layer and the ordinary convolution provided in the embodiment of the present application. DETAILED DESCRIPTION

[0030] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0031] The following describes some of the techniques involved in the embodiments of the present application to facilitate understanding, and they should be considered as merely exemplary. Therefore, it should be appreciated by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for the sake of clarity and conciseness, some descriptions of well-known functions and structures are omitted in the following description.

[0032] Figure 1 1 is a flowchart of a rotation equivariant constraint relaxation method based on a convolutional network framework provided in an embodiment of the present application, including steps 101 to 105. Figure 1 This is only an execution order shown in the embodiment of the present application, and does not represent the only execution order of the rotation equivariant constraint relaxation method based on the convolutional network framework. In the case where the final result can be achieved, Figure 1 The steps shown may be performed in parallel or reversed.

[0033] Step 101: set hyperparameters, and process the original image pair into a training image pair based on the hyperparameters to form a training image set. In the embodiment of the present application, the hyperparameters include element parameters and rotation parameters. The element parameters are used to determine the rotation mode of the original image pair. The rotation parameters are used to determine the rotation angle of the original image pair.

[0034] Exemplarily, the element parameter and the rotation parameter are represented by tranNum and n, respectively. The element parameter is exemplarily set to 4 or 8. When the element parameter is set to 4, the rotation parameter is a randomly generated integer in the range of [1,4]. Among them, 1, 2, 3, and 4 represent the execution of 90 degrees, 180 degrees, 270 degrees, and 360 degrees of center rotation, respectively.

[0035] In the embodiment of the present application, the original images in the original image pair are rotated according to the hyperparameters. The original image pair includes a degraded image and its corresponding true value image. The degraded image and the true value image before and after the rotation are spliced ​​to obtain a spliced ​​degraded image and a spliced ​​true value image to form a training image pair.

[0036] Specifically, the degraded image I in the original image pair deg and its corresponding true value image I gt Rotate according to the rotation parameters in the hyperparameters to get the rotated backward image rot_I deg and the rotated true image rot_I gt , the degraded images before and after rotation are spliced ​​with the true value images before and after rotation in a specific data dimension, and the obtained spliced ​​degraded image I and the spliced ​​true value image I target The training image pairs are constructed as follows:

[0037] I=concat((I deg ,rot_I deg ), dim=0),

[0038] I target =concat((I gt ,rot_I gt ),dim=0).

[0039] in, dim=0 means that the degraded images before and after rotation are concatenated with the true images before and after rotation in the Batch dimension of the data. B represents Batch, that is, in each training iteration, the model randomly extracts Batch from the training image set for inference. C represents the number of channels of the image, H represents the height of the image, W represents the width of the image, dim represents the dimension of the concat operation, and concat represents the concatenation operation.

[0040] Multiple groups of original image pairs are rotated and spliced ​​according to the above steps to obtain multiple training image pairs to form a training image set.

[0041] Step 102: Design a relaxed rotation equivariant constrained convolution layer based on the rotation equivariant property of the image. In the embodiment of the present application, the relaxed rotation equivariant constrained convolution layer is different from the previous method of realizing rotation equivariance, which requires the design of a group equivariant convolution kernel, that is, a strict equivariant constraint is imposed on the convolution kernel. The relaxed rotation equivariant constrained convolution layer of the present application is achieved by (1) calculating the rotation of the training image pair after each convolution layer and performing channel shift. (2) The image is first rotated and then passed through the relaxed rotation equivariant constrained convolution layer. The difference between the two is used as a regularization term for network training, thereby imposing a rotation equivariant constraint on the feature layer plane, so that the constructed network has a more consistent with the symmetry characteristics existing in different images and a more relaxed rotation equivariant feature extraction capability.

[0042] Specifically, since existing equivariant networks are usually designed based on the framework of group convolution, the convolution kernel in the group convolution will show strict rotation and channel rotation in the corresponding group dimension. Due to this property, group convolution is strict in describing the rotational symmetry in the image. However, strictly equivariant networks are too limited in some image processing tasks (it is difficult to distinguish between "6" and "9"). This application does not adopt the group convolution framework, but uses an ordinary convolutional network and relaxes the strict equivariant constraints by adding a regularization term to the network. Based on this idea, a relaxed rotation equivariant constraint convolution layer is constructed. The specific implementation formula is as follows:

[0043]

[0044] Where, L layer represents the regularization term calculated in the relaxed rotation equivariant constrained convolutional layer, φ(·) represents the convolutional mapping, Indicated in The rotation matrix of the randomly selected angle in is, F represents the feature space obtained after convolution mapping, Represents rotation and channel shift operations, X represents the input training image pair, and R represents the action space of the rotation operation, that is, the input image domain.

[0045] Based on the relaxed rotation equivariant constrained convolution layer, the input training image is rotated after each convolution layer, and channel shift is performed (the first term on the right side of the equation). The image is rotated before passing through the convolution layer (the second term on the right side of the equation). The difference between the two is used as a regularization term for network training, thereby applying a rotation equivariant regularization on the feature layer plane, so that the constructed rotation equivariant relaxed network has more consistent with the symmetry characteristics existing in the data set and more relaxed rotation equivariant feature extraction capabilities.

[0046] Step 103: Replace the convolutional layer of the deep learning network model with the relaxed rotation equivariant constrained convolutional layer to construct a rotation equivariant relaxed network. In an embodiment of the present application, the deep learning network model can be a network RCDNet (image deraining network model based on deep expansion), DICDNet (Deep Interpretable Convolutional Dictionary Network), ACDNet (Adaptive Convolutional Dictionary Network, Adaptive Convolutional Dictionary Network suitable for CT image artifact removal), OSCNet (Orientation-Shared Convolutional Network, a direction-shared convolutional network for CT metal artifact learning) and other classic network models in image processing tasks such as image deraining and medical image artifact removal. The convolutional layers in these classic network models are replaced with the relaxed rotation equivariant constrained convolutional layers designed in this application to obtain a rotation equivariant relaxed network.

[0047] Step 104: Iteratively train the rotation equivariant relaxation network using the training image set until the stopping condition is reached to obtain the trained rotation equivariant relaxation network. In an embodiment of the present application, the current hyperparameters of this iteration are determined, and the network training step is iteratively performed based on the rotation equivariant relaxation network until the stopping condition is reached. The network training step includes: inputting the current hyperparameters and the training image pair into the rotation equivariant relaxation network. Calculate the relaxed equivariant regularization term of the rotation equivariant relaxation network. Determine the total loss of the rotation equivariant relaxation network based on the relaxed equivariant regularization term. Calculate the gradient between the total loss and the current hyperparameter. Update the current hyperparameter based on the gradient, and perform the network training step.

[0048] Specifically, a set of data is determined as the current hyperparameters in the hyperparameters set in step 101, a set of training image pairs are randomly selected from the training image set, and the current hyperparameters and the selected training image pairs are input into the rotation equivariant relaxation network.

[0049] Furthermore, the relaxed equivariant regularization term of the rotational equivariant relaxed network is calculated as follows:

[0050]

[0051] Where, L equi represents the relaxed equivariant regularization term, φ (l) represents the first l layers of the rotationally equivariantly relaxed network, Indicated in The rotation matrix of the randomly selected angle in , φ(·) represents the convolution mapping, and F represents the feature space obtained after the convolution mapping. represents the rotation and channel shift operation, l represents the first l layers of the network, X represents the input training image pair, and R represents the action space of the rotation operation, that is, the input image domain. Indicates taking the average.

[0052] The total loss is calculated as follows:

[0053] L=||softENet(I)-I target ||2+αL equi .

[0054] Where L represents the total loss, softENet represents the rotation equivariant relaxation network, I represents the input training image pair, and I target represents the label corresponding to the input training image pair (target image or real image, that is, the effect image that the model is expected to achieve), α represents the hyperparameter that controls the strength of the equivariant constraint, and L equi represents the relaxed equivariant regularization term.

[0055] This application uses the Adam optimization algorithm to update the network parameters of the rotational equivariant relaxation network to optimize the total loss, specifically as follows: Calculate the gradient between the total loss and the current hyperparameters. Update the current hyperparameters according to the gradient. Execute the network training step again based on the updated current hyperparameters.

[0056] Exemplarily, the stop condition of the present application is set to reach a preset number of training times (200 times). The conditions for stopping training or the number of iterative training can be set according to actual needs in this field, and no specific limitation is made here.

[0057] Step 105: Input the degraded image into the trained rotation equivariant relaxation network to obtain a restored image. In the embodiment of the present application, the degraded image is processed based on the rotation equivariant relaxation network trained in step 104 to output a processed restored image.

[0058] In one embodiment of the present application, image deraining training and testing are performed on four public datasets, namely Rain100L, Rain100H, Rain1400 and Rain12. The four public datasets contain multiple sets of image pairs consisting of clean background images and rainy images of their corresponding scenes, and the rainy images contain rain marks of various directions or densities.

[0059] The image resolution of these four public datasets is 481×321 pixels. The training samples and test samples in the four public datasets are divided into (training / testing) 200 / 100, 1800 / 100, 12600 / 1400, and 200 / 12. The optimizer uses the Adam optimizer to update the hyperparameters. The learning rate corresponding to each network / model is set to 0.001, the batch size is 16, and the iteration is 100 generations (training stop condition). During training, the images in the dataset are randomly cropped into 64×64 pixel image blocks for training. All experiments were completed on an NVIDIA V100 GPU and implemented using the Pytorch framework. The test indicators are PSNR (Peak Signal-to-Noise Ratio, an indicator used to evaluate the quality of image reconstruction. The higher the PSNR value, the better the quality of the processed image and the smaller the difference with the original image) and SSIM (Structural Similarity, an indicator used to measure the similarity between two images, calculated based on three factors: brightness, contrast and structure of the image components. The closer the SSIM is to 1, the better the image quality is), which includes the following steps in sequence.

[0060] Step S1: Select a dataset from the four public datasets Rain100L, Rain100H, Rain1400, and Rain12 to obtain the original image pairs, and determine the hyperparameters for this network training: the element parameter tranNum is set to 4, and the rotation parameter n is a randomly generated integer in the range of [1,4]. For example, 3 represents a 270-degree center rotation of the original image pair. The degraded image (rainy image) and the corresponding true value image (clean background image) in the original image pair are rotated according to the determined rotation parameter n to obtain the rotated image, and finally they are spliced ​​in the Batch dimension to obtain the spliced ​​degraded image and the spliced ​​true value image to form a training image pair.

[0061] Step S2: According to the rotation equivariance characteristics contained in natural images and medical images, a relaxed rotation equivariance constrained convolutional layer with relaxed rotation equivariance constraints is designed.

[0062] Step S3: Replace the convolutional layer in the existing image deraining network model RCDNet with the relaxed rotation equivariant constrained convolutional layer to obtain the rotation equivariant relaxed network RCDNet-Loss.

[0063] Step S4: Using the method recorded in the above step 104, use the training image processed in step S1 to train the rotation equivariant relaxation network RCDNet-Loss until it is iterated 100 times, and save the trained rotation equivariant relaxation network RCDNet-Loss at this time.

[0064] Under the condition that all experimental conditions are consistent, the comparative experimental results of the present invention and other 11 image deraining models on four public datasets Rain100L, Rain100H, Rain1400 and Rain12 are shown in Table 1.

[0065] Table 1 Comparative experimental results of different rain removal models / networks on four public datasets

[0066]

[0067]

[0068] The experiment of the present invention is based on the RCDNet image deraining network model based on deep expansion developed in the field of natural image deraining in recent years. The convolutional layers in the network are replaced by the relaxed rotation equivariant constrained convolutional layers designed in this application to obtain the model RCDNet-Loss, and compared with the strict rotation equivariant network RCDNet-p4 under the same setting. The results show that compared with the strict equivariant constraints, the present application can achieve better performance.

[0069] In Table 1, Input represents the input training image pair. GMM represents Gaussian Mixture Model. JCAS represents a deraining model for joint convolutional analysis and synthetic sparse representation for single image layer separation. Clear represents a deep network model for single image deraining. DNN represents Deep Neural Networks. RESCAN, PReNet, JORDER_E, SIRR and IDT are deep learning image deraining models proposed in recent years.

[0070] Some rain removal results are as follows Figure 4 As shown, the method of the present application can remove rain strips more completely and clearly while maintaining the detailed texture of the original image, further improving the rain removal effect. It can be seen that the rain removal results of the present application are better than other comparison methods, demonstrating the application potential of the present application.

[0071] In another embodiment of the present application, training and testing of CT image artifact removal are performed on the DeepLesion dataset (a large CT image dataset publicly released by the Clinical Center of the National Institutes of Health (NIH), containing more than 32,000 lesion annotations from more than 10,000 case studies, and involving various lesion types, such as lung nodules, liver tumors, lymphadenopathy, etc.). The image resolution is 416×416 pixels. The learning rate corresponding to each network / model is set to 0.0002, the batch size is 32, and the iteration is 200 generations (training stop condition). During training, the images in the dataset are randomly cropped into 64×64 pixel image blocks for training. All experiments were completed on an NVIDIA V100 GPU and implemented using the Pytorch framework. The test indicators are PSNR and SSIM, including the following steps.

[0072] Step S10: Obtain an original image pair from the DeepLesion dataset, which involves an image X without metal artifacts, an image Y damaged by metal artifacts, and an image of a non-metal region I. Determine hyperparameters, the element parameter tranNum is set to 4, and the rotation parameter n is a randomly generated integer in the range of 1-4, for example, 2 represents a 180-degree center rotation of the original image pair.

[0073] The original image pair is pre-processed, and the metal artifact-free image X′, the metal artifact-damaged image Y′, and the non-metal area image I′ in the original image pair are rotated according to the determined rotation parameter n to obtain the rotated metal artifact-free image X, the metal artifact-damaged image Y, and the non-metal area image I, and the two data are spliced ​​in the Batch dimension to obtain the training image pair, as follows:

[0074] I=concat((I′,rot_I),dim=0),

[0075] X=concat((X′,rot_X),dim=0),

[0076] Y=concat((Y′,rot_Y),dim=0).

[0077] Among them, dim=0 means splicing in the Batch dimension.

[0078] Step S20: According to the rotation equivariance characteristics contained in natural images and medical images, a relaxed rotation equivariance constrained convolutional layer with relaxed rotation equivariance constraints is designed.

[0079] Step S30: Replace the convolutional layers in the existing medical image artifact removal networks ACDNet, DICDNet and OSCNet with the above-mentioned relaxed rotation equivariant constrained convolutional layers to obtain rotation equivariant relaxed networks ACDNet-Loss, DICDNet-Loss and OSCNet-Loss.

[0080] Step S40: Using the method recorded in the above step 104, use the training image pairs processed in step S10 to respectively train the rotation equivariant relaxation networks ACDNet-Loss, DICDNet-Loss and OSCNet-Loss, until it is iterated 200 times, and save the trained rotation equivariant relaxation networks ACDNet-Loss, DICDNet-Loss and OSCNet-Loss at this time.

[0081] According to the physical imaging mechanism of common CT images damaged by metal artifacts, the relationship between the rotated image damaged by metal artifacts Y, the rotated image without metal artifacts X, and the rotated image of the non-metal region I is:

[0082] I⊙Y=I⊙X+I⊙M.

[0083] in, M is the metal artifact layer to be estimated, is a binary mask used to represent the area of ​​non-metallic artifacts, that is, the non-metallic area image. ⊙ is the pixel-by-pixel multiplication based on pixels. H and W represent the height and width of the image respectively, and X represents the input training image pair.

[0084] The optimization problem of CT image reconstruction damaged by metal artifacts is:

[0085]

[0086] in, and denote the artifact-free image to be estimated and the metal artifact layer to be estimated, λ1 and λ2 denote weight coefficients, and f1(·) and f2(·) denote the and The regularization term of Represents the square norm. Thus, by solving the optimization problem, the potential metal artifact-free image and metal artifact layer can be estimated based on the existing image damaged by metal artifacts, and X represents the input training image pair.

[0087] Then the relaxed equivariant regularization terms and total losses of the rotational equivariant relaxation networks ACDNet-Loss, DICDNet-Loss and OSCNet-Loss are calculated, and the hyperparameters are optimized to obtain the trained rotational equivariant relaxation networks ACDNet-Loss, DICDNet-Loss and OSCNet-Loss.

[0088] With all experimental conditions being consistent, the comparative experimental results of this application and other medical image artifact removal models on the DeepLesion public dataset are shown in Table 2.

[0089] Table 2 Comparative experimental results of different medical image artifact removal models / networks on the DeepLesion public dataset

[0090]

[0091]

[0092] The experiment of this application is based on the models ACDNet, DICDNet and OSCNet developed in the field of medical image artifact removal in recent years. The convolutional layers in the network are replaced with the relaxed rotation equivariant constrained convolutional layers designed by this application to obtain the models ACDNet-Loss, DICDNet-Loss and OSCNet-Loss. And under the same settings, they are compared with the strictly rotation equivariant networks ACDNet-p4, DICDNet-p4, OSCNet-p4 (the convolutional layers in the network are replaced with strictly rotation equivariant group convolutional layers). The results show that compared with strict equivariant constraints, relaxed equivariant constraints can further improve network performance, and it can also be seen from the experimental results that when the network attempts to characterize equivariance that does not exist in the data, it may cause the network's expression ability to decrease. The method of this application enables the network to adaptively learn equivariance that conforms to the symmetric characteristics of the data from the data set, which helps to improve performance.

[0093] In Table 2, Input represents an image damaged by metal artifacts. LI and NMAR are early CT image artifact removal methods. CNNMAR, DuDoNet, DSCMAR and InDuDoNet are newly proposed deep learning methods for CT image artifact removal in recent years. ACDNet, DICDNet and OSCNet are recently proposed CT deep learning reconstruction models based on deep unfolding.

[0094] In general, for the reconstruction models in multiple application problems, after using the rotation equivariant relaxation network constructed by the rotation equivariant constraint relaxation method based on the convolutional network framework proposed in this application, the results of these applications have achieved performance improvements, including numerical accuracy (Table 1 and Table 2) and visual effects ( Figure 4 ).also, Figure 5 (a) is the visualization result of the equivariant error when the image is input into a trained ResNet network using ordinary convolution, and (b) is the visualization result of the equivariant error when the image is input into a trained ResNet network using a relaxed rotation equivariant constraint convolution layer. Comparing the two results, it can be clearly seen that the method of the present application effectively imposes rotation equivariant constraints on the network, reduces the equivariant error, and proves the effectiveness of the present application.

[0095] Although the present application provides method operation steps as described in the embodiments or flowcharts, more or fewer operation steps may be included based on conventional or non-creative labor. The order of steps listed in this embodiment is only one way of executing the order of many steps and does not represent the only execution order. When the actual device or client product is executed, it can be executed in sequence or in parallel according to the method shown in this embodiment or the accompanying drawings (for example, in a parallel processor or multi-threaded processing environment).

[0096] like Figure 2 As shown, the embodiment of the present application also provides a rotation equivariant constraint relaxation device 200 based on a convolutional network framework. The device includes: a pre-processing module 201, a design module 202, a construction module 203, a training module 204 and a processing module 205, as follows.

[0097] The pre-processing module 201 is used to set hyper parameters, and process the original image pairs into training image pairs based on the hyper parameters to form a training image set.

[0098] The design module 202 is used to design a relaxed rotation equivariant constrained convolutional layer based on the rotation equivariant property of the image.

[0099] The construction module 203 is used to replace the convolutional layer of the deep learning network model with the relaxed rotation equivariant constrained convolutional layer to construct a rotation equivariant relaxed network.

[0100] The training module 204 is used to iteratively train the rotation equivariant relaxation network using the training image set until a stop condition is reached, thereby obtaining a trained rotation equivariant relaxation network.

[0101] The processing module 205 is used to input the degraded image into the trained rotational equivariant relaxation network to obtain a restored image.

[0102] Some modules in the apparatus described in the present application can be described in the general context of computer executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, classes, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0103] The devices or modules described in the above application embodiments can be implemented by computer chips or entities, or by products with certain functions. For the convenience of description, the above devices are described in various modules according to their functions. When implementing the embodiments of the present application, the functions of each module can be implemented in the same or multiple software and / or hardware. Of course, the module that implements a certain function can also be implemented by combining multiple sub-modules or sub-units.

[0104] The methods, devices or modules described in this application can be implemented in the form of computer-readable program codes. The controller can be implemented in any appropriate manner. For example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program codes (such as software or firmware) that can be executed by the (micro) processor, logic gates, switches, application-specific integrated circuits (English: Application Specific Integrated Circuit; Abbreviation: ASIC), programmable logic controllers and embedded microcontrollers. Examples of controllers include but are not limited to the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20 and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code, the controller can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers and embedded microcontrollers by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included in it for implementing various functions can also be regarded as structures within the hardware component. Or even, the means for realizing various functions may be regarded as both a software module for realizing the method and a structure within a hardware component.

[0105] An embodiment of the present application further provides a device, comprising: a processor; a memory for storing processor executable instructions; when the processor executes the executable instructions, the method described in the embodiment of the present application is implemented.

[0106] The embodiments of the present application also provide a non-volatile computer-readable storage medium on which a computer program or instruction is stored. When the computer program or instruction is executed, the method described in the embodiments of the present application is implemented.

[0107] In addition, each functional module in each embodiment of the present invention may be integrated into one processing module, or each module may exist independently, or two or more modules may be integrated into one module.

[0108] The above storage media include but are not limited to random access memory (RAM), read-only memory (ROM), cache, hard disk (HDD) or memory card. The memory can be used to store computer program instructions.

[0109] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present application can be implemented by means of software plus necessary hardware. Based on such an understanding, the technical solution of the present application can be essentially or partly contributed to the prior art in the form of a software product, or it can be reflected in the implementation process of data migration. The computer software product can be stored in a storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a mobile terminal, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.

[0110] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. All or part of this application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, mobile communication terminals, multi-processor systems, microprocessor-based systems, programmable electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc.

[0111] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit the present application. Although the present application has been described in detail with reference to the aforementioned embodiments, a person of ordinary skill in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some or all of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the present application.

Claims

1. A rotation equivariant constraint relaxation method based on a convolutional network framework, characterized in that: include: Setting hyperparameters, and processing the original image pairs into training image pairs based on the hyperparameters to form a training image set; Design a relaxed rotation equivariant constrained convolutional layer based on the rotation equivariant property of the image; Replacing the convolutional layer of the deep learning network model with the relaxed rotation equivariant constrained convolutional layer to construct a rotation equivariant relaxed network; Iteratively training the rotation equivariant relaxation network using the training image set until a stopping condition is reached to obtain the trained rotation equivariant relaxation network; The degraded image is input into the trained rotation equivariant relaxation network to obtain a restored image.

2. The method according to claim 1, characterized in that: The processing of the original image pair into a training image pair based on the hyperparameter comprises: Rotating the original images in the original image pair respectively according to the hyperparameters; wherein the original image pair includes a degraded image and its corresponding true value image; The degraded image before and after rotation and the true value image are respectively spliced ​​to obtain a spliced ​​degraded image and a spliced ​​true value image to constitute the training image pair.

3. The method according to claim 1, characterized in that The regularization term of the relaxed rotation equivariant constrained convolutional layer is as follows: Where, L layer represents the regularization term calculated in the relaxed rotation equivariant constrained convolutional layer, φ(·) represents the convolutional mapping, Indicated in The rotation matrix of the randomly selected angle in is, F represents the feature space obtained after convolution mapping, Represents rotation and channel shift operations, X represents the input training image pair, and R represents the action space of the rotation operation, that is, the input image domain.

4. The method according to claim 1, characterized in that: The iterative training of the rotation equivariant relaxation network using the training image set until a stopping condition is reached comprises: Determine the current hyperparameters of this round of iteration, and iteratively perform network training steps based on the rotational equivariant relaxation network until the stop condition is reached; The network training step comprises: Inputting the current hyperparameter and the training image pair into the rotation equivariant relaxation network; Calculating a relaxed equivariant regularization term of the rotationally equivariant relaxed network; Determining the total loss of the rotational equivariant relaxed network according to the relaxed equivariant regularization term; Calculate the gradient between the total loss and the current hyperparameter; The current hyperparameters are updated according to the gradient, and the network training step is performed.

5. The method according to claim 4, characterized in that The calculation method of the relaxed equivariant regularization term is as follows: Where, L equi represents the relaxed equivariant regularization term, φ (l) represents the first l layers of the rotationally equivariantly relaxed network, Indicated in The rotation matrix of the randomly selected angle in , φ(·) represents the convolution mapping, and F represents the feature space obtained after the convolution mapping. represents the rotation and channel shift operation, l represents the first l layers of the network, X represents the input training image pair, and R represents the action space of the rotation operation, that is, the input image domain. Indicates taking the average.

6. The method according to claim 4, characterized in that The calculation method of the total loss is as follows: L=||softENet(I)-I target || s +αL equi ; Where L represents the total loss, softENet represents the rotation equivariant relaxation network, I represents the input training image pair, and I target represents the label corresponding to the input training image pair, α represents the hyperparameter controlling the strength of the equivariant constraint, and L equi represents the relaxed equivariant regularization term.

7. The method according to claim 2, characterized in that The hyperparameters include element parameters and rotation parameters; The element parameters are used to determine the rotation mode of the original image pair; The rotation parameter is used to determine the rotation angle of the original image pair.

8. A rotation equivariant constraint relaxation device based on a convolutional network framework, characterized in that: include: A pre-processing module, used for setting hyperparameters, and processing the original image pairs into training image pairs based on the hyperparameters to form a training image set; A design module for designing a relaxed rotation equivariant constrained convolutional layer based on the rotation equivariant property of the image; A construction module, used to replace the convolutional layer of the deep learning network model with the relaxed rotation equivariant constrained convolutional layer to construct a rotation equivariant relaxed network; A training module, used for iteratively training the rotation equivariant relaxation network using the training image set until a stopping condition is reached, thereby obtaining the trained rotation equivariant relaxation network; The processing module is used to input the degraded image into the trained rotational equivariant relaxation network to obtain a restored image.

9. A device for performing a rotation equivariant constraint relaxation method based on a convolutional network framework, characterized in that: include: processor; a memory for storing processor-executable instructions; When the processor executes the executable instructions, the method according to any one of claims 1 to 7 is implemented.

10. A non-volatile computer-readable storage medium, characterized in that: The device comprises a computer program or an instruction for storing the computer program or the instruction, which, when executed, enables the method according to any one of claims 1 to 7 to be implemented.

Images