Parameterized rotation equivariant convolution method and device suitable for image processing task

Abstract

The invention discloses a parameterized rotation isovariant convolution method and device suitable for an image processing task, and relates to the technical field of deep learning. By using a parameterization technology, taking a bicubic interpolation function as a substrate of a combined expression, and constructing a parameterization convolution framework by performing linear combination with a learnable weight coefficient; based on the rotational symmetry characteristic of the image, constructing a rotational equivariant convolution layer by using a parameterized convolution framework; replacing a traditional convolutional layer in the image processing network with the constructed rotation equivariant convolutional layer to construct a rotation equivariant convolutional network with a rotation equivariant characteristic; and inputting a to-be-trained image in the training data set into the rotation isovariant convolutional network, calculating a loss function, iteratively updating network parameters and optimizing the loss function by adopting an optimization algorithm until a preset stop condition is reached, and obtaining an optimized rotation isovariant convolutional network. The problem that an existing method is insufficient in accuracy and stability when processing complex image data is solved.

Description

Technical Field

[0001] The present application relates to the field of deep learning technology, and in particular to a parameterized rotational equivariant convolution method and device suitable for image processing tasks. Background Art

[0002] In image processing tasks, image data often contains various transformation symmetries, such as translation, rotation, and reflection, which permeate both underlying local features and high-level semantic representations. Leveraging these symmetries can effectively reduce the number of model parameters and enhance the generalization capabilities of machine learning methods.

[0003] Convolutional neural networks (CNNs) have achieved tremendous success in tasks such as image recognition, object detection, and semantic segmentation, thanks to their translational equivariance. This property enables efficient utilization of network parameters and improves model generalization. However, traditional CNNs have limitations in preserving broader symmetries, such as those under rotation and reflection.

[0004] Although parametric methods based on harmonic basis and Fourier basis have achieved advanced performance in underlying visual tasks, the two-dimensional continuous filter kernel function in the constructed convolution framework still has a gap with the actual underlying function, which affects the accuracy and stability of rotational equivariant convolution when processing complex image data. Summary of the Invention

[0005] In an embodiment of the present application, a parameterized rotational equivariant convolution method suitable for image processing tasks is provided to solve the problem of insufficient accuracy and stability of existing methods when processing complex image data.

[0006] In the first aspect, an embodiment of the present application provides a parameterized rotation-equivariant convolution method suitable for image processing tasks, the method comprising: preprocessing the original input image of the image processing task to obtain a training data set and a test data set; using parameterization technology, taking the bicubic interpolation function as the basis of the combination expression, and constructing a parameterized convolution framework by linearly combining it with a learnable weight coefficient; based on the rotational symmetry characteristics of the image, using the parameterized convolution framework to construct a rotation-equivariant convolution layer for processing image data with rotational symmetry; replacing the traditional convolution layer in the image processing network with the constructed rotation-equivariant convolution layer to construct a rotation-equivariant convolution network with rotation-equivariant characteristics; inputting the image to be trained in the training data set into the rotation-equivariant convolution network, calculating the loss function according to the output of the rotation-equivariant convolution network, and iteratively updating the network parameters and optimizing the loss function using an optimization algorithm until a preset stopping condition is reached to obtain an optimized rotation-equivariant convolution network; inputting the data to be tested into the optimized rotation-equivariant convolution network, performing forward inference to obtain a restored image or conversion result, and using the test data set to evaluate the performance of the network.

[0007] In one possible implementation, the original input image of the image processing task is preprocessed to obtain a training data set and a test data set, including: the preprocessing includes a reshaping operation, an augmentation operation and a segmentation operation, and the preprocessed image data constitutes a training set and a test set.

[0008] In one possible implementation, the parameterization technique is used to take the bicubic interpolation function as the basis of the combination expression, and a parameterized convolution framework is constructed by linearly combining it with the learnable weight coefficients, including: the expression of the bicubic interpolation function is: Among them, φ Bic (y) is the bicubic interpolation function, y is the independent variable of the bicubic interpolation function, and represents the relative position of a pixel point under consideration and the target pixel point in the horizontal or vertical direction during the image processing process; a combined expression is constructed based on the expression of the bicubic interpolation function, and the combined expression is: Where [x1,x2] T is the two-dimensional coordinate of the image, x1 is the horizontal coordinate, x2 is the vertical coordinate, T is the transpose operation, p is the size of the convolution kernel, δ is the discretized grid size, a represents the horizontal position offset of the current pixel relative to the target pixel, and b represents the vertical position offset of the current pixel relative to the target pixel. The expression of the parameterized convolution framework constructed by linearly combining the combined expression with the learnable weight coefficient is: Among them, φ(x) is the two-dimensional implicit function to be learned of the two-dimensional convolution kernel in the parameterized convolution framework, K is the number of functions in the combined expression involved in the linear combination, ω k is the weight coefficient of the kth to be learned, φ k (x) is the kth pre-set function, x is the input variable, and represents the spatial position of the two-dimensional coordinate in the image.

[0009] In one possible implementation, the rotational equivariant convolution layer includes an input layer, an intermediate layer, and an output layer; the input layer is used to receive the input image and convert it into a tensor format; the intermediate layer gradually extracts high-order features through multi-layer stacking, including activation layers, pooling layers, and normalization layers to perform detailed processing of the features; the output layer is used to generate the final prediction result based on the type of task.

[0010] In a possible implementation, the rotational symmetry characteristics of the image are used to construct a rotational equivariant convolution layer using a parameterized convolution framework to process image data with rotational symmetry, including: defining the input layer convolution, the intermediate layer convolution and the output layer convolution in a continuous space to obtain a continuous domain convolution; the input layer convolution maps the input image to a feature map defined on E(2), and for any (y, A)∈E(2), the expression is: Among them, Ψ[r](y,A) is the feature map output by the input layer convolution, E(2) is the Euclidean group containing 2D translation and rotation transformations, Ψ is the input layer convolution, x, y are the coordinates in the two-dimensional space, A is the rotation matrix acting on the input layer convolution kernel, and this type of rotation matrix is ​​used as the index of the group direction channel in the feature map, r is the input image, is a two-dimensional space, is the parameterized convolution kernel of the input layer built based on the parameterized convolution framework, σ is the measure in two-dimensional space, and d is the integral symbol; the intermediate layer convolution maps the feature map to another feature map defined on E(2). For any (y, B)∈E(2), the expression is: Among them, Φ[e](y,B) is the feature map output by the intermediate layer convolution, Φ is the intermediate layer convolution, B is the rotation matrix acting on the intermediate layer convolution kernel, e is the continuous domain feature map, O(2) is the orthogonal group, is the convolution kernel corresponding to the group channel A constructed based on the parameterized convolution framework, and v is the Haar measure on the orthogonal group; the output layer convolution maps the feature map to a function on the smooth function space on the two-dimensional plane. For any The expression is: Among them, Υ[e](y) is the output result of the output layer convolution, Υ is the output layer convolution, e is the continuous domain feature map, It is the parameterized convolution kernel of the output layer built based on the parameterized convolution framework. The continuous domain convolution is discretized to obtain the equivariant convolution in the discrete domain; the image I∈R n×n is considered as a smooth function r defined at the center of a regular n×n grid cell: The two-dimensional discretization representation, for i, j = 1, 2, ..., n, the pixel value of the image at position (i, j) is: I ij =r(x ij );in, h is the grid size, I ij For each element in the image, I∈R n×n is an image of size n×n, where n is the number of pixels in the horizontal and vertical directions of the image, and r is a number defined in a two-dimensional real space. Smooth function on ; characteristic map is considered to be defined in The discretized form of the continuous function on ; where S is a subgroup of O(2), t is the number of elements in S, is a semidirect product group, from the real number space and the subgroup S of the orthogonal group; the characteristic graph is represented as a function from the smooth function e: A three-dimensional grid tensor obtained by sampling, for i, j = 1, 2, ..., n, the value of the feature map F is expressed as: in, A∈S is the rotation matrix, is the value of the feature map at the group channel index A and coordinate (i, j), x ij is the coordinate of the center point of the grid in the i-th row and j-th column of the regular n×n grid unit; the p×p filter kernel of a single channel is expressed as a discretized smooth function The obtained two-dimensional grid function satisfies in, is a two-dimensional grid function, is a smooth function defined on a two-dimensional plane, st represents all functions that satisfy The spatial position x, two-dimensional grid function The value of is zero, otherwise it is non-zero, and p is the size of the filter kernel. Through sampling and approximation, the continuous domain convolution operation is discretized. The input layer convolution kernel in the discrete domain is expressed as: in, is the discretized input layer convolution kernel The value at group channel index A and coordinate (i, j), p is the size of the input layer convolution kernel; the intermediate layer convolution kernel in the discrete domain is expressed as: in, is the discretized intermediate layer convolution kernel The values ​​at the two group channel indices A and B and the coordinates (i, j); the output layer convolution kernel in the discrete domain is expressed as: in, is the discretized output layer convolution kernel The value at the group channel index B and the coordinate (i, j); the input layer convolution expression of the rotation equivariant convolution layer is: Where I is the input image, * is the discrete two-dimensional convolution operation; the intermediate layer convolution expression of the rotation equivariant convolution layer is: Among them, F A is a slice of the discrete domain feature map at the group channel index A; the output layer convolution expression of the rotation equivariant convolution layer is: Among them, F B is a slice of the discrete domain feature map at the group channel index B.

[0011] In one possible implementation, the image to be trained in the training data set is input into a rotational equivariant convolutional network, a loss function is calculated according to the output of the rotational equivariant convolutional network, and an optimization algorithm is used to iteratively update the network parameters and optimize the loss function until a preset stopping condition is reached to obtain an optimized rotational equivariant convolutional network, including: the optimization algorithm is a gradient descent algorithm; the network parameters are updated according to the gradient information obtained by the gradient descent algorithm; the preset stopping condition includes reaching a preset iterative step.

[0012] In the second aspect, the embodiment of the present application provides a parameterized rotation equivariant convolution device suitable for image processing tasks, which includes: a preprocessing module for preprocessing the original input image of the image processing task to obtain a training data set and a test data set; a parameterized convolution framework module for constructing a parameterized convolution framework by using parameterization technology and taking the bicubic interpolation function as the basis of the combination expression by linearly combining it with the learnable weight coefficient; a processing module for constructing a rotation equivariant convolution layer based on the rotational symmetry characteristics of the image using the parameterized convolution framework to process image data with rotational symmetry; and a rotation equivariant convolution network module for constructing a rotation equivariant convolution layer. The block is used to replace the traditional convolution layer in the image processing network with the constructed rotation equivariant convolution layer to construct a rotation equivariant convolution network with rotation equivariant characteristics; the iteration module is used to input the image to be trained in the training data set into the rotation equivariant convolution network, calculate the loss function according to the output of the rotation equivariant convolution network, and use the optimization algorithm to iteratively update the network parameters and optimize the loss function until the preset stopping condition is reached to obtain the optimized rotation equivariant convolution network; the evaluation module is used to input the data to be tested into the optimized rotation equivariant convolution network, perform forward inference to obtain the restored image or conversion result, and use the test data set to evaluate the performance of the network.

[0013] In a third aspect, an embodiment of the present application provides a parameterized rotational equivariant convolution server suitable for image processing tasks, comprising a memory and a processor; the memory is used to store computer-executable instructions; the processor is used to execute the computer-executable instructions to implement the method described in the first aspect or any possible implementation method of the first aspect.

[0014] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores executable instructions. When a computer executes the executable instructions, it can implement the method described in the first aspect or any possible implementation method of the first aspect.

[0015] One or more technical solutions provided in the embodiments of this application have at least the following technical effects: The embodiments of this application provide a parameterized rotation-equivariant convolution method suitable for image processing tasks. This method preprocesses the original input image and constructs high-quality training and test datasets, providing a reliable data foundation for model training and evaluation, ensuring that the model can learn the essential characteristics and distribution patterns of image data and reducing the interference of noise and outliers on model performance. Utilizing parameterization technology, a parameterized convolution framework is constructed based on bicubic interpolation functions. Weight coefficients can be learned through linear combination, enabling flexible construction of convolution kernels. This improves the adaptability of the convolution kernel to local image features and enhances the generalization and robustness of the model. The rotation-equivariant convolution layer constructed based on the rotational symmetry of the image can accurately process image data with rotational symmetry. Replacing the traditional convolution operator with rotation-equivariant convolution to construct a rotation-equivariant network embeds the network with rotational symmetry prior information of the image, ensuring that the network can maintain the inherent symmetric structure of the image when extracting features, and the output results are consistent and stable, significantly improving the performance of the model in multiple visual tasks. The entire training process continuously optimizes the model by feeding training data into the rotation-equivariant convolutional network and employing an optimization algorithm to iteratively update network parameters and optimize the loss function. After reaching the preset stopping condition, the optimized rotation-equivariant convolutional network demonstrates higher accuracy and lower loss on the training dataset, demonstrating that the model has fully learned the useful information in the data. This overcomes the inadequate accuracy and stability of existing methods when processing complex image data. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments of the present application or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0017] Figure 1 A flowchart of a parameterized rotational equivariant convolution method suitable for image processing tasks provided in an embodiment of the present application;

[0018] Figure 2 A schematic diagram comparing the implicit functions of an 11×11 convolution kernel implemented using different convolution kernel parameterization methods provided in the embodiments of this application;

[0019] Figure 3 The network architecture of the rotational equivariant convolutional layer provided in the embodiment of the present application;

[0020] Figure 4A schematic diagram of a parameterized rotational equivariant convolution device suitable for image processing tasks provided in an embodiment of the present application;

[0021] Figure 5 A schematic diagram of a parameterized rotation equivariant convolution server suitable for image processing tasks provided in an embodiment of the present application. DETAILED DESCRIPTION

[0022] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0023] The following description of some of the technologies involved in the embodiments of this application is provided to facilitate understanding and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of this application. Similarly, for the sake of clarity and conciseness, some descriptions of well-known functions and structures are omitted from the following description.

[0024] The embodiment of the present application provides a parameterized rotation equivariant convolution method suitable for image processing tasks, such as Figure 1 As shown, the method includes steps S101 to S106. Figure 1 This is only an execution order shown in the embodiment of the present application, and does not represent the only execution order of a parameterized rotation equivariant convolution method suitable for image processing tasks. In the case that the final result can be achieved, Figure 1 The steps shown may be performed in parallel or reversed.

[0025] S101: Preprocess the original input image of the image processing task to obtain a training dataset and a test dataset.

[0026] Preprocessing includes reshaping, augmentation, and segmentation operations. The preprocessed image data constitute the training set and test set.

[0027] Specifically, this application can be applied to image processing tasks such as infinite super-resolution, style conversion, cross-modal conversion of medical images, natural image denoising, 2x super-resolution of natural images, and image deraining.

[0028] Furthermore, for different image processing tasks, specific preprocessing strategies can be adopted, and appropriate data sets are selected for training and testing. For the infinite super-resolution task, the DIV2K data set is used as training data. This data set contains 800 high-quality images with a resolution of 2K, which can provide the model with rich image details and texture information. During the training stage, in order to adapt to the super-resolution requirements of different scale factors, high-resolution image blocks of 48s×48s size are randomly cropped from the data samples, where the scale factor s is sampled from [2,4]. Subsequently, the paired low-resolution image blocks corresponding to the scale factor s are obtained through the bicubic downsampling method, thereby constructing the input-output pairs required for training. In addition, in order to enhance the supervision signal, 48s are sampled from the real-value image. 2 pixel points and their corresponding coordinates are used for supervised learning during the training process. In the testing phase, the performance of the method is evaluated on the DIV2K validation set and multiple standard benchmark datasets, including Urban100, B100, Set14 and Set5, to ensure the generalization ability and robustness of the model. For the unpaired image-to-image conversion task, three public datasets summer2winter_yosemite, old2young and BDD100K are selected as training and test data. These datasets cover different types of transformations in natural scenes, such as seasonal transformations, age transformations, etc., which can fully verify the performance of the model in the unpaired image-to-image conversion task. In order to unify the input size and reduce the amount of computation, all images are cropped to 256×256 size. For the MRI image modality conversion task, the BraTS2019 public dataset is selected as training and test data. This dataset contains four modalities: T1-weighted (T1w), T1-enhanced (T1Gd), T2-weighted (T2w), and fluid-attenuated inversion recovery (FLAIR) sequences. These provide the model with rich medical image information. Training is performed on 2D axial slices of original MRI images to fully utilize the spatial information within the images. Furthermore, to evaluate the model's performance, quantitative metrics are calculated between the 3D reconstruction results and the real data. For image denoising, the DIV2K dataset is used for training and tested on Urban100, B100, Set14, and Set5 to evaluate the model's denoising performance under varying noise levels. For image 2x super-resolution, the DIV2K dataset is used for training and tested on Urban100, B100, Set14, and Set5 to validate the model's super-resolution performance under a fixed scale factor. For single-image rain removal, the Rain100L dataset is used for training and testing. This dataset contains a large number of images with raindrops and their corresponding rain-free images, fully validating the model's effectiveness in rain removal.

[0029] Through the above preprocessing strategies and dataset selection, suitable training datasets and test datasets were constructed for different image processing tasks, laying a solid foundation for subsequent model training and performance evaluation.

[0030] S102: Using parameterization technology, the bicubic interpolation function is used as the basis of the combination expression, and a parameterized convolution framework is constructed by linearly combining it with learnable weight coefficients.

[0031] Specifically, in the fields of image processing and computer vision, the design of convolution kernels is crucial to model performance. To learn convolution kernels more flexibly and efficiently, this application proposes a parameterized convolution framework based on bicubic interpolation functions.

[0032] Using parameterization technology, the bicubic interpolation function is used as the basis of the combination expression. By linearly combining it with the learnable weight coefficients, a parameterized convolution framework is constructed, which includes the following steps.

[0033] The expression of the bicubic interpolation function is: Among them, φ Bic (y) is the bicubic interpolation function, y is the independent variable of the bicubic interpolation function, and represents the relative position of a pixel point under consideration and the target pixel point in the horizontal or vertical direction during the image processing process.

[0034] The combined expression is constructed based on the expression of the bicubic interpolation function. The combined expression is: Where [x1,x2] T is the two-dimensional coordinate of the image, x1 is the horizontal coordinate, x2 is the vertical coordinate, T is the transpose operation, p is the size of the convolution kernel, δ is the discretized grid size, a represents the horizontal position offset of the current pixel relative to the target pixel, and b represents the vertical position offset of the current pixel relative to the target pixel.

[0035] Specifically, the combined expression generates elements at different positions in the convolution kernel by adjusting the sizes of a and b.

[0036] The expression of the parameterized convolution framework constructed by linearly combining the combination expression with the learnable weight coefficient is: Among them, φ(x) is the two-dimensional implicit function to be learned of the two-dimensional convolution kernel in the parameterized convolution framework, K is the number of functions in the combined expression involved in the linear combination, ω k is the weight coefficient of the kth to be learned, φ k (x) is the kth pre-set function, x is the input variable, and represents the spatial position of the two-dimensional coordinate in the image.

[0037] Specifically, by learning the weight coefficients, the shape and characteristics of the convolution kernel can be adjusted to adapt to different image processing tasks.

[0038] Figure 2 Schematic diagram comparing the implicit functions of the 11×11 convolution kernel implemented by different convolution kernel parameterization methods provided in the embodiments of this application. Figure 2 (a) in the figure is a discretized convolution kernel of 11×11. Figure 2 (b) in the figure is the true two-dimensional implicit function of the discretized convolution kernel and its grayscale mapping. Figure 2 (c) in the figure is the two-dimensional implicit function and its grayscale mapping obtained by the parameterization method of the harmonic basis. Figure 2 (d) in the figure is the two-dimensional implicit function and its grayscale mapping obtained by the parameterization method of the Fourier basis. Figure 2 (e) in the figure is the two-dimensional implicit function and its grayscale mapping obtained by the parameterized convolution framework of the bicubic interpolation function constructed by this application. By comparison, it can be found that there is a significant gap between the continuous two-dimensional implicit function obtained by the parameterized method based on the harmonic basis and the Fourier basis and the real two-dimensional implicit function of the two-dimensional convolution kernel. This is because these filtering methods are highly dependent on high-frequency sine and cosine functions, resulting in a large number of unnecessary function fluctuations. In contrast, the parameterized convolution framework of the bicubic interpolation function constructed by this application is more natural and reasonable in representing the convolution kernel implicit function. Bicubic interpolation is one of the most commonly used interpolation methods in image transformation, and its effectiveness has been widely verified in practice. From Figure 2 In (e), we can intuitively see that the implicit function represented by the parameterized convolution framework using the bicubic interpolation function is closer to the actual two-dimensional continuous convolution kernel implicit function. This shows that the parameterized convolution framework of this application has higher accuracy and flexibility in representing the convolution kernel implicit function.

[0039] S103: Based on the rotational symmetry of the image, a parameterized convolution framework is used to construct a rotational equivariant convolution layer for processing image data with rotational symmetry.

[0040] The rotation equivariant convolution layer includes input layer, intermediate layer and output layer.

[0041] The input layer is used to receive the input image and convert it into tensor format.

[0042] Specifically, the input image can be a low-resolution image, an original style image, a noisy image, a rain image, etc., and it is converted into a tensor format that the network can process.

[0043] The middle layer gradually extracts high-order features through multi-layer stacking, including activation layer, pooling layer, and normalization layer to process the features in detail.

[0044] The output layer is used to generate the final prediction results based on the type of task.

[0045] Based on the rotational symmetry characteristics of the image, a parameterized convolution framework is used to construct a rotational equivariant convolution layer for processing image data with rotational symmetry, including the following steps.

[0046] The input layer convolution, intermediate layer convolution and output layer convolution are defined in continuous space to obtain continuous domain convolution.

[0047] Specifically, when exploring the intrinsic rotational symmetry of an image, the orthogonal group O(2) and the Euclidean group E(2) containing 2-dimensional translation and rotation transformations are introduced. The element is (x, A), based on which the rotation equivariant convolution layer is constructed. Consider the orthogonal group Equivariance on , where A∈S is a rotation matrix and the rotation group S is a subgroup of O(2).

[0048] The input layer convolution maps the input image to the feature map defined on E(2). For any (y, A)∈E(2), the expression is: Among them, Ψ[r](y,A) is the feature map output by the input layer convolution, Ψ is the input layer convolution, x, y are the coordinates in two-dimensional space, A is the rotation matrix acting on the input layer convolution kernel, and this type of rotation matrix is ​​used as the index of the group direction channel in the feature map, r is the input image, is a two-dimensional space, is the parameterized convolution kernel of the input layer built based on the parameterized convolution framework, σ is the measure in two-dimensional space, and d is the integral symbol.

[0049] Specifically, the input image is defined as is a set of infinitely differentiable functions defined in two-dimensional space.

[0050] The intermediate layer convolution maps the feature map to another feature map defined on E(2). For any (y, B)∈E(2), the expression is: Among them, Φ[e](y,B) is the feature map output by the intermediate layer convolution, Φ is the intermediate layer convolution, B is the rotation matrix acting on the intermediate layer convolution kernel, e is the continuous domain feature map, is the convolution kernel corresponding to the group channel A constructed based on the parameterized convolution framework, and v is the Haar measure on the orthogonal group.

[0051] Specifically, the continuous domain feature map is defined as e∈C ∞ (E(2)). C ∞ (E(2)) is the set of infinitely differentiable functions defined on the Euclidean group E(2).

[0052] The output layer convolution maps the feature map into a function on a smooth function space on a two-dimensional plane. For any The expression is: Among them, Υ[e](y) is the output result of the output layer convolution, Υ is the output layer convolution, e is the continuous domain feature map, Parameterized convolution kernel for the output layer built on the parameterized convolution framework.

[0053] The continuous domain convolution is discretized to obtain the equivariant convolution in the discrete domain.

[0054] Specifically, in image processing tasks, the continuous domain convolution needs to be discretized to obtain the equivariant convolution in the discrete domain that can be actually used.

[0055] Image I∈R n×n is considered as a smooth function r defined at the center of a regular n×n grid cell: The two-dimensional discretization representation, for i, j = 1, 2, ..., n, the pixel value of the image at position (i, j) is: I ij =r(x ij ).in, h is the grid size, I ij For each element in the image, I∈R n×n is an image of size n×n, where n is the number of pixels in the horizontal and vertical directions of the image, and r is a number defined in a two-dimensional real space. Smooth functions on .

[0056] Feature Map is considered to be defined in The discretized form of the continuous function on . Where S is a subgroup of O(2), t is the number of elements in S, is a semidirect product group, from the real number space and the subgroup S of the orthogonal group.

[0057] The feature map is represented as a function from a smooth function e: A three-dimensional grid tensor obtained by sampling, for i, j = 1, 2, ..., n, the value of the feature map F is expressed as: in, A∈S is the rotation matrix, is the value of the feature map at the group channel index A and coordinate (i, j), x ij is the coordinate of the center point of the grid in the i-th row and j-th column of the regular n×n grid cell.

[0058] The single-channel p×p filter kernel is represented by a discretized smooth function The obtained two-dimensional grid function satisfies in, is a two-dimensional grid function, is a smooth function defined on a two-dimensional plane, st represents all functions that satisfy The spatial position x, two-dimensional grid function The value of is zero, otherwise it is non-zero, and p is the size of the filter kernel.

[0059] Through sampling and approximation, the continuous domain convolution operation is converted into a convolution kernel in the discrete domain.

[0060] The input layer convolution kernel in the discrete domain is expressed as: in, Discretized input layer convolution kernel The value at group channel index A and coordinate (i, j), p is the size of the convolution kernel in the input layer.

[0061] The convolution kernel of the intermediate layer in the discrete domain is expressed as: in, is the discretized intermediate layer convolution kernel The values ​​at the two group channel indices A, B and coordinates (i, j).

[0062] The output layer convolution kernel in the discrete domain is expressed as: in, is the discretized output layer convolution kernel The value at group channel index B and coordinate (i, j).

[0063] The input layer convolution expression of the rotation equivariant convolution layer is: Where I is the input image and * is the discrete two-dimensional convolution operation.

[0064] The intermediate layer convolution expression of the rotation equivariant convolution layer is: Among them, F A is a slice of the discrete domain feature map at the group channel index A.

[0065] The output layer convolution expression of the rotation equivariant convolution layer is: Among them, F B is a slice of the discrete domain feature map at the group channel index B.

[0066] Specifically, the above steps successfully constructed a rotational equivariant convolutional layer, which can fully utilize the rotational symmetry prior information of the image and improve the feature extraction capability of the model.

[0067] S104: replacing the traditional convolutional layer in the image processing network with the constructed rotation equivariant convolutional layer to construct a rotation equivariant convolutional network with rotation equivariant properties.

[0068] Specifically, the image processing network is an existing image processing network. This replacement strategy aims to retain the basic architecture and functions of the original network while embedding rotation equivariance, enabling the network to better process image data with rotational symmetry.

[0069] The optimization algorithm is a gradient descent algorithm. The network parameters are updated based on the gradient information obtained by the gradient descent algorithm. The preset stopping condition includes reaching a preset number of iterations.

[0070] Specifically, when the network training reaches a preset number of iterations, the training process will stop and the current network parameters will be saved as the final model.

[0071] Furthermore, in the examples of this application, the replacement convolutional networks include a variety of classic image processing networks, such as: EDSR: used for infinite super-resolution, denoising, and 2x super-resolution tasks; CycleGAN, CUT, and santa: used for image-to-image conversion tasks; HyperGAN: used for MRI (magnetic resonance imaging) modality conversion tasks; and RCDNet: used for rain removal tasks.

[0072] Figure 3 The network architecture of the rotation equivariant convolution layer provided in the embodiment of the present application. The architecture includes three main parts: input layer convolution, intermediate layer convolution and output layer convolution. Each subgraph corresponds to the detailed structure of these three parts. In the example of the present application, the rotation group S(A i ,B i ∈S) is a 2πi / 3 rotation (i=1,2,3). i is the angle index, A i , B i is the rotation matrix of 2πi / 3 under the corresponding angle, and this rotation matrix is ​​used as the index of the group direction channel in the feature map. This means that the rotation equivariant convolution can fully embed the rotational symmetry prior information of the input image when performing feature extraction.

[0073] S105: Input the image to be trained in the training data set into the rotation equivariant convolutional network, calculate the loss function according to the output of the rotation equivariant convolutional network, and iteratively update the network parameters and optimize the loss function using an optimization algorithm until a preset stopping condition is reached to obtain an optimized rotation equivariant convolutional network.

[0074] Specifically, according to the specific visual task, a corresponding loss function is set to measure the difference between the network output and the true label. These loss functions can include mean square error, reconstruction loss, contrastive learning loss, adversarial loss, perceptual loss, etc.

[0075] S106: Input the data to be tested into the optimized rotation equivariant convolutional network for forward inference. The data passes through each layer of the network (such as convolution layer, pooling layer, activation function, etc.) in turn, and each layer transforms the data (such as feature extraction, nonlinear mapping, etc.), and finally obtains the restored image or conversion result. According to the specific task, the evaluation indicators are selected and the performance of the network is evaluated using the test data set. For example, in the infinite super-resolution task, the peak signal-to-noise ratio (PSNR) is used for quantitative evaluation; the style conversion task uses FID (Fréchet Inception Distance) to compare the distribution difference between the generated image and the real image in the feature space; the medical image modality conversion uses the mean absolute error (MAE), structural similarity index (SSIM) and peak signal-to-noise ratio (PSNR); the denoising and rain removal tasks are measured using the structural similarity index (SSIM) and peak signal-to-noise ratio (PSNR).

[0076] Table 1 shows the comparison results of this application (B-Conv) and the most advanced parameterized equivariant convolution F-Conv on the infinite super-resolution task. All experiments are based on the classic infinite super-resolution method, and the encoder is uniformly set to EDSR-baseline. The traditional convolution operator is replaced with equivariant convolution to obtain a rotational equivariant network. In the table, LIIF, OPE, and LTE are the names of the INR methods commonly used in infinite super-resolution tasks, CNN is an ordinary convolutional network, F-Conv is a rotational equivariant convolution based on the Fourier basis, and B-Conv is an equivariant convolution method based on the bicubic interpolation function. It can be clearly seen that this application shows superior performance in all benchmark tests (Urban100, BSD100, Set14, Set5).

[0077] Table 1

[0078]

[0079] Table 2 shows the image-to-image conversion results on the summer2winter_yosemite (natural scenery transitioning from winter to summer) and old2young (face rejuvenation) datasets. In the table, Method lists the different comparison methods. B-CycleGAN, B-CUT, and B-santa use the proposed equivariant convolution method based on the bicubic interpolation function. FID is the evaluation metric (smaller values ​​represent better generation results), and Param represents the number of model parameters. Clearly, replacing the standard convolutional network with the constructed parameterized convolution framework based on the bicubic interpolation function not only reduces the number of network parameters but also improves the fidelity of the image conversion results.

[0080] Table 2

[0081]

[0082] Table 3 shows the conversion results on BDD100K: converting road images from day to night. In the table, "Method" lists the different comparison methods. B-CycleGAN, B-CUT, and B-santa use the proposed equivariant convolution method based on the bicubic interpolation function. FID is the evaluation metric (smaller values ​​indicate better generation results), and Param is the number of model parameters. This method achieves a lightweight network and achieves higher-quality conversion results.

[0083] Table 3

[0084]

[0085] Table 4 shows the MRI modality conversion results. A total of four modalities, T1W, T1Gd, T2W and FLAIR, are included. The indicators shown are the results of taking the average of all 12 conversion tasks on the test set. In the table, Method is listed as different comparison methods, B-HyperGAN uses the proposed equivariant convolution method based on the bicubic interpolation function, MAE, PSNR, and SSIM are evaluation indicators (the smaller the MAE value, the better the generation effect, and the larger the PSNR and SSIM values, the better the generation effect), and Param is the model parameter quantity. Based on the baseline method HyperGAN, which performs well in this task, the traditional convolution operator is replaced with the proposed equivariant convolution (B-HyperGAN). It is found that the images generated by this application have more realistic contrast and can better retain local details, which is crucial for medical diagnosis.

[0086] Table 4

[0087]

[0088] Table 5 shows the results of image denoising. It compares the currently more advanced equivariant convolution methods (Method is listed as the method name, and B-Conv is the proposed equivariant convolution method based on the bicubic interpolation function). Peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) were used as evaluation indicators (larger values ​​indicate better generation effect). The method of this application showed advantages in peak signal-to-noise ratio on four test sets (Urban100, BSD100, Set14, and Set5).

[0089] Table 5

[0090]

[0091] Table 6 shows the results of 2x super-resolution of the image. Consistent with the above image denoising experiment, the more advanced equivariant convolution methods are compared (Method is the method name, B-Conv is the proposed equivariant convolution method based on the bicubic interpolation function). Peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) are used as evaluation indicators (the larger the value, the better the generation effect). It can be seen that this application has significant advantages in the underlying visual tasks.

[0092] Table 6

[0093]

[0094] Table 7 shows the results of single-image rain removal. We selected RCDNet, the most advanced method for this task, and constructed it into a rotational equivariant network based on the proposed equivariant convolution method for experimental testing. In the table, "Method" lists the different comparison methods. B-RCDNet uses the proposed equivariant convolution method based on the bicubic interpolation function. PSNR and SSIM are evaluation metrics (larger values ​​represent better generation results). Clearly, the improved network in this application has superior performance.

[0095] Table 7

[0096]

[0097] The embodiment of the present application also provides a parameterized rotation equivariant convolution device 400 suitable for image processing tasks, such as Figure 4 As shown, the device includes: a preprocessing module 401, a parameterized convolution framework construction module 402, a processing module 403, a rotation equivariant convolution network construction module 404, an iteration module 405 and an evaluation module 406.

[0098] The preprocessing module 401 is used to preprocess the original input image of the image processing task to obtain a training data set and a test data set.

[0099] The parameterized convolution framework construction module 402 is used to utilize parameterization technology, take the bicubic interpolation function as the basis of the combination expression, and construct the parameterized convolution framework by linearly combining it with the learnable weight coefficients.

[0100] The processing module 403 is used to construct a rotational equivariant convolution layer based on the rotational symmetry characteristics of the image using a parameterized convolution framework to process image data with rotational symmetry.

[0101] The rotation equivariant convolutional network construction module 404 is used to replace the traditional convolutional layer in the image processing network with the constructed rotation equivariant convolutional layer to construct a rotation equivariant convolutional network with rotation equivariant characteristics.

[0102] The iteration module 405 is used to input the images to be trained in the training data set into the rotation equivariant convolutional network, calculate the loss function according to the output of the rotation equivariant convolutional network, and iteratively update the network parameters and optimize the loss function using an optimization algorithm until a preset stopping condition is reached to obtain an optimized rotation equivariant convolutional network.

[0103] The evaluation module 406 is used to input the data to be tested into the optimized rotation equivariant convolutional network, perform forward inference to obtain a restored image or conversion result, and use the test data set to evaluate the performance of the network.

[0104] Some modules in the apparatus described herein may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, classes, etc. that perform specific tasks or implement specific abstract data types. The present application may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communications network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0105] The devices or modules described in the above application embodiments can be implemented by computer chips or physical devices, or by products with certain functions. For ease of description, the above devices are described separately by function in various modules. When implementing the embodiments of this application, the functions of each module can be implemented in the same or multiple software and / or hardware. Of course, a module that implements a certain function can also be implemented by combining multiple sub-modules or sub-units.

[0106] The methods, devices, or modules described in this application can be implemented in the form of computer-readable program code. The controller can be implemented in any appropriate manner. For example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also know that in addition to implementing the controller in the form of pure computer-readable program code, it is entirely possible to implement the same function of the controller in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, such a controller can be considered a hardware component, and the devices included therein for implementing various functions can also be considered as structures within the hardware component. Or even, the means for implementing various functions may be considered to be both a software module for implementing the method and a structure within a hardware component.

[0107] like Figure 5 As shown, an embodiment of the present application also provides a parameterized rotation equivariant convolution server suitable for image processing tasks, including a memory 501 and a processor 502; the memory 501 is used to store computer-executable instructions; the processor 502 is used to execute computer-executable instructions to implement a parameterized rotation equivariant convolution method suitable for image processing tasks described above in an embodiment of the present application.

[0108] An embodiment of the present application also provides a computer-readable storage medium, which stores executable instructions. When a computer executes the executable instructions, it can implement the parameterized rotation equivariant convolution method suitable for image processing tasks described above in the embodiment of the present application.

[0109] Through the description of the above implementation methods, it can be known that those skilled in the art can clearly understand that the present application can be implemented by means of software plus necessary hardware. Based on this understanding, the technical solution of the present application can be essentially or the part that contributes to the prior art can be embodied in the form of a software product, or it can be embodied in the implementation process of data migration. The computer software product can be stored in a storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a mobile terminal, a server, or a network device, etc.) to execute the method described in the embodiment of the present application.

[0110] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to in detail. Each embodiment focuses on the differences from other embodiments. All or part of this application can be used in many general or special computer system environments or configurations.

[0111] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit the present application. Although the present application has been described in detail with reference to the aforementioned embodiments, a person of ordinary skill in the art should understand that the technical solutions described in the aforementioned embodiments can still be modified, or some or all of the technical features therein can be replaced by equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the present application.

Claims

1. A parameterized rotation equivariant convolution method suitable for image processing tasks, characterized in that: include: Preprocess the original input images of the image processing task to obtain training data sets and test data sets; Using parameterization technology, the bicubic interpolation function is used as the basis of the combination expression, and a parameterized convolution framework is constructed by linearly combining it with learnable weight coefficients. Based on the rotational symmetry of the image, a parameterized convolution framework is used to construct a rotational equivariant convolution layer to process image data with rotational symmetry. The constructed rotation equivariant convolutional layer replaces the traditional convolutional layer in the image processing network to construct a rotation equivariant convolutional network with rotation equivariant properties; The training images in the training data set are input into the rotation equivariant convolutional network, the loss function is calculated according to the output of the rotation equivariant convolutional network, and the optimization algorithm is used to iteratively update the network parameters and optimize the loss function until the preset stopping condition is reached to obtain the optimized rotation equivariant convolutional network; The test data is input into the optimized rotation-equivariant convolutional network, forward inference is performed to obtain the restored image or conversion result, and the performance of the network is evaluated using the test dataset.

2. The parameterized rotation equivariant convolution method for image processing tasks according to claim 1, characterized in that: The preprocessing of the original input image of the image processing task to obtain a training dataset and a test dataset includes: The preprocessing includes reshaping operations, augmentation operations, and segmentation operations, and the preprocessed image data constitute a training set and a test set.

3. The parameterized rotation equivariant convolution method for image processing tasks according to claim 1, characterized in that: The parametric technology is used to construct a parameterized convolution framework by taking the bicubic interpolation function as the basis of the combination expression and linearly combining it with the learnable weight coefficients, including: The expression of the bicubic interpolation function is: Among them, φ Bic (y) is the bicubic interpolation function, y is the independent variable of the bicubic interpolation function, and represents the relative position of a pixel point under consideration and the target pixel point in the horizontal or vertical direction during image processing; The combined expression is constructed based on the expression of the bicubic interpolation function. The combined expression is: Where [x1,x2] T is the two-dimensional coordinate of the image, x1 is the horizontal coordinate, x2 is the vertical coordinate, T is the transpose operation, p is the size of the convolution kernel, δ is the discretized grid size, a represents the horizontal offset of the current pixel relative to the target pixel, and b represents the vertical offset of the current pixel relative to the target pixel; The expression of the parameterized convolution framework constructed by linearly combining the combination expression with the learnable weight coefficient is: Among them, φ(x) is the two-dimensional implicit function to be learned of the two-dimensional convolution kernel in the parameterized convolution framework, K is the number of functions in the combined expression participating in the linear combination, ω k is the weight coefficient of the kth to be learned, φ k (x) is the kth pre-set function, x is the input variable, and represents the spatial position of the two-dimensional coordinate in the image.

4. The parameterized rotation equivariant convolution method for image processing tasks according to claim 3, characterized in that: The rotational equivariant convolution layer includes input layer, intermediate layer and output layer; The input layer is used to receive the input image and convert it into tensor format; The middle layer gradually extracts high-level features through multi-layer stacking, including activation layer, pooling layer, and normalization layer to process the features in detail; The output layer is used to generate the final prediction results based on the type of task.

5. The parameterized rotation equivariant convolution method for image processing tasks according to claim 3, characterized in that: The method uses a parameterized convolution framework to construct a rotationally equivariant convolution layer based on the rotational symmetry of the image, and is used to process image data with rotational symmetry, including: Define the input layer convolution, intermediate layer convolution and output layer convolution into continuous space to obtain continuous domain convolution; The input layer convolution maps the input image to the feature map defined on E(2). For any (y, A)∈E(2), the expression is: Among them, Ψ[r](y,A) is the feature map output by the input layer convolution, E(2) is the Euclidean group containing 2D translation and rotation transformations, Ψ is the input layer convolution, x, y are the coordinates in the two-dimensional space, A is the rotation matrix acting on the input layer convolution kernel, and this type of rotation matrix is ​​used as the index of the group direction channel in the feature map, r is the input image, is a two-dimensional space, is the parameterized convolution kernel of the input layer built based on the parameterized convolution framework, σ is the measure in two-dimensional space, and d is the integral symbol; The intermediate layer convolution maps the feature map to another feature map defined on E(2). For any (y, B)∈E(2), the expression is: Among them, Φ[e](y,B) is the feature map output by the intermediate layer convolution, Φ is the intermediate layer convolution, x, y are the coordinates in the two-dimensional space, B is the rotation matrix acting on the convolution kernel of the intermediate layer, and such rotation matrix is ​​used as the index of the group direction channel in the feature map, e is the continuous domain feature map, O(2) is the orthogonal group, is the convolution kernel corresponding to the group channel A constructed based on the parameterized convolution framework, and v is the Haar measure on the orthogonal group; The output layer convolution maps the feature map into a function on a smooth function space on a two-dimensional plane. For any The expression is: Among them, Υ[e](y) is the output result of the output layer convolution, Υ is the output layer convolution, x, y are the coordinates in the two-dimensional space, e is the continuous domain feature map, Parameterized convolution kernel for the output layer built on the parameterized convolution framework; Discretize the continuous domain convolution to obtain the equivariant convolution in the discrete domain; Image I∈R n×n is considered as a smooth function defined at the center of a regular n×n grid cell The two-dimensional discretization representation, for i, j = 1, 2, ..., n, the pixel value of the image at position (i, j) is: I ij =r(x ij );in, h is the grid size, I ij For each element in the image, I∈R n×n is an image of size n×n, where n is the number of pixels in the horizontal and vertical directions of the image, and r is a number defined in a two-dimensional real space. Smooth functions on ; Feature Map is considered to be defined in The discretized form of the continuous function on ; where S is a subgroup of O(2), t is the number of elements in S, is a semidirect product group, from the real number space and the subgroup S of the orthogonal group; Represent the feature map as a function from a smooth function A three-dimensional grid tensor obtained by sampling, for i, j = 1, 2, ..., n, the value of the feature map F is expressed as: in, A∈S is the rotation matrix on the rotation transformation group, is the value of the feature map at the group channel index A and coordinate (i, j), x ij is the coordinate of the center point of the grid in the i-th row and j-th column of the regular n×n grid cell; The single-channel p×p filter kernel is represented by a discretized smooth function The obtained two-dimensional grid function satisfies in, is a two-dimensional grid function, is a smooth function defined on a two-dimensional plane, st represents all functions that satisfy The spatial position x, two-dimensional grid function The value of is zero, otherwise it is non-zero, and p is the size of the filter kernel; Through sampling and approximation, the continuous domain convolution operation is converted into a convolution kernel in the discrete domain; The input layer convolution kernel in the discrete domain is expressed as: in, is the discretized input layer convolution kernel The value at the group channel index A and coordinate (i, j), p is the size of the convolution kernel of the input layer, A -1 For rotation operation; The convolution kernel of the intermediate layer in the discrete domain is expressed as: in, is the discretized intermediate layer convolution kernel In the two group channel indices of B and A and the values ​​at coordinates (i, j), B -1 For rotation operation; The output layer convolution kernel in the discrete domain is expressed as: in, is the discretized output layer convolution kernel The value at the group channel index B and coordinate (i, j), B -1 For rotation operation; The input layer convolution expression of the rotation equivariant convolution layer is: Where I is the input image, * is the discrete two-dimensional convolution operation; The intermediate layer convolution expression of the rotation equivariant convolution layer is: Among them, F A is a slice of the discrete domain feature map F at the group channel index A; The output layer convolution expression of the rotation equivariant convolution layer is: Among them, F B is a slice of the discrete domain feature map F at the group channel index B.

6. The parameterized rotation equivariant convolution method for image processing tasks according to claim 5, characterized in that: The image to be trained in the training data set is input into the rotation equivariant convolutional network, the loss function is calculated according to the output of the rotation equivariant convolutional network, and the network parameters and the optimized loss function are iteratively updated using an optimization algorithm until a preset stopping condition is reached to obtain an optimized rotation equivariant convolutional network, including: The optimization algorithm is the gradient descent algorithm; Update network parameters based on the gradient information obtained by the gradient descent algorithm; The preset stopping condition includes reaching a preset number of iteration steps.

7. A parameterized rotational equivariant convolution device suitable for image processing tasks, characterized in that: include: The preprocessing module is used to preprocess the original input images of the image processing task to obtain training data sets and test data sets; Construct a parameterized convolution framework module, which uses parameterization technology to use the bicubic interpolation function as the basis of the combination expression and linearly combines it with the learnable weight coefficients to construct a parameterized convolution framework; A processing module, configured to construct a rotationally equivariant convolution layer using a parameterized convolution framework based on the rotational symmetry characteristics of the image, so as to process image data with rotational symmetry; Construct a rotation-equivariant convolutional network module, which is used to replace the traditional convolutional layer in the image processing network with the constructed rotation-equivariant convolutional layer to construct a rotation-equivariant convolutional network with rotation-equivariant properties; The iteration module is used to input the training images in the training data set into the rotation-equivariant convolutional network, calculate the loss function based on the output of the rotation-equivariant convolutional network, and iteratively update the network parameters and optimize the loss function using an optimization algorithm until the preset stopping condition is reached, thereby obtaining the optimized rotation-equivariant convolutional network; The evaluation module is used to input the test data into the optimized rotation-equivariant convolutional network, perform forward inference to obtain the restored image or conversion result, and use the test dataset to evaluate the performance of the network.

8. A parameterized rotation equivariant convolution server suitable for image processing tasks, characterized in that: including memory and processor; The memory is used to store computer-executable instructions; The processor is configured to execute the computer-executable instructions to implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores executable instructions, and when a computer executes the executable instructions, the method according to any one of claims 1 to 6 can be implemented.

Images