Image processing method and system based on multi-stage processing architecture

Abstract

The present invention discloses an image processing method and system based on a multi-stage processing architecture, which relates to the field of image processing technology. On the basis of the existing deep learning technology, the method is improved, and the environmental information is comprehensively considered in the feature extraction and classification process while taking into account the influence of ambient light and shooting angle. The acquired image is denoised by using transform domain denoising technology and convolutional neural network, and segmentation is performed by using a segmentation backbone network combined with parallel self-attention model optimization. The image is restored by using a multi-scale image restoration module, and the error caused by environmental factors is reduced by using a three-dimensional reconstruction module, so as to provide an image processing method suitable for images acquired in various environments.

Description

Technical Field

[0001] The present invention relates to image processing, and in particular to an image processing method and system based on a multi-stage processing architecture. Background Art

[0002] Image processing can cover a variety of scenarios and situations, and can be applied in biology, medicine, security, autonomous driving and identification of emergencies. When an emergency occurs, important information can be quickly conveyed to decision makers and handlers to ensure timely intervention and effective response; applied to medical imaging, it can quickly convey important information to medical personnel and doctors to ensure rapid diagnosis and effective treatment; applied to radar security, it can quickly highlight abnormal area images to convey important information to responsible personnel to ensure rapid response and effective response.

[0003] In the image processing system, the current existing technology is to process and analyze the image layer by layer through a multi-level convolutional network. In fact, due to the high-dimensional characteristics of the feature map after multiple convolutions and pooling, it is difficult to accurately retain all the details of the standard image at each stage. In fact, due to the high computational complexity when processing high-resolution images, it is difficult to achieve accurate segmentation and recognition. In the process of feature integration and detail reconstruction, the fusion of features at different levels will inevitably cause information loss and blurred details, which makes the final output image result have a certain degree of unclearness and error, and this error will seriously affect the overall performance of the system in the case of fine image analysis and processing.

[0004] In the application of information transmission, if there are errors in image processing, it will affect the discovery of abnormal situations and the accuracy of image information, thus affecting the accuracy and reliability of information transmission. In the fields of disasters, pests and diseases identification, if abnormal situations are not identified, serious threats may arise. Therefore, when processing images, it is very important to ensure that the image processing methods and systems have high accuracy and reliability. Summary of the invention

[0005] The purpose of the present invention is to overcome the shortcomings of the prior art and provide an image processing method based on a multi-stage processing architecture. The technical problem to be solved is: the image recognition method in the prior art mainly uses a multi-stage convolutional network to process and analyze images, which has high computational complexity when applied in actual scenarios, makes it difficult to retain details, and has large errors. The purpose of the present invention is to specifically target image processing and provide an image processing method based on a multi-stage processing architecture. Image processing is divided into multiple stages, and denoising, segmentation and restoration are performed in stages based on the multi-stage processing architecture, thereby reducing computational complexity and retaining details.

[0006] On the basis of the existing deep learning technology, and considering the influence of ambient lighting and shooting angle, the present invention adopts transform domain denoising technology and convolutional neural network to denoise the collected image in view of the problem that traditional standard denoising methods may cause loss of image details and single wavelet denoising may cause image blur; in view of the limitations of image enhancement, traditional image enhancement methods such as histogram equalization may cause over-enhancement and artifacts, while traditional image segmentation methods such as threshold segmentation and region growing have poor segmentation effects for complex backgrounds and low-contrast areas. Deep learning methods require a large amount of labeled data and computing resources and may not be suitable for resource-constrained environments. The present invention adopts a segmentation backbone network combined with a parallel self-attention model optimization for segmentation, combines the self-attention model to adaptively enhance image features, adopts a deep learning method, combines image information of different scales, and improves the segmentation ability of multi-scale objects; in view of the fact that image restoration may not be effective in processing severe blur and distortion, especially when the blur type is complex or unknown, the present invention adopts a multi-scale image restoration module to restore the image, uses a pyramid network architecture to extract features from multiple scales, captures information of different scales, and uses a three-dimensional reconstruction module to reduce errors caused by environmental factors, and provides an image processing method and system based on a multi-stage processing architecture.

[0007] The purpose of the present invention is mainly achieved through the following technical solutions:

[0008] An image processing method of a multi-stage processing architecture comprises the following steps:

[0009] Step S1, input an image to be processed, adjust the image to a set size and resolution to obtain a standard image, add random Gaussian noise to the standard image, and crop the image with Gaussian noise added to an image with the same size and resolution as the standard image to obtain a noisy image.

[0010] Step S2: using transform domain denoising technology to perform wavelet decomposition on the noisy image, increasing the receptive field to obtain a wavelet denoised image; then denoising the wavelet denoised image through a convolutional neural network, and outputting a feature image as a denoised image.

[0011] Step S3, performing image enhancement and segmentation on the denoised image through a segmentation backbone network; wherein the denoised image is input into the encoder of the segmentation backbone network to extract multiple image region nodes and feature maps; a region node level self-attention model and a channel level self-attention model are used to capture the spatial connection between region nodes and the semantic association across channels respectively, and the output fusion is represented as enhanced features; the enhanced features are sent to the decoder of the segmentation backbone network to output the segmented image.

[0012] Step S4, input the segmented image to restore the image; wherein the restored image includes: processing details of different scales through a multi-scale image restoration module to form a feature map of multi-scale information as an output of a first restored image; using a generative adversarial network adopted by a deep learning model to restore the first restored image to output a second restored image; processing the second restored image through a three-dimensional reconstruction module to obtain a final output completed processed image.

[0013] A further optimization scheme is that the segmentation backbone network in step S3 includes multiple layers of convolution operations, and the last layer output feature map is recorded as ,and , Each position of corresponds to a local area of ​​the denoised image, which is used as an image area node.

[0014] H represents the height of the feature map, which indicates the number of pixels in the vertical direction of the feature map;

[0015] W represents the width of the feature map, which indicates the number of pixels in the horizontal direction of the feature map;

[0016] D represents the depth of the feature map;

[0017] C represents the number of channels of the feature map, indicating how many eigenvalues ​​there are at each spatial position, including different feature information;

[0018] H and W describe the spatial size of the feature map, and D describes the depth of the feature map. In three-dimensional image processing, let Describe the three-dimensional size, Reshape into a matrix , reshape the matrix Each row in represents an image region node with C eigenvalues; and there is a reshaping matrix The transposed matrix of , , each row of the transposed matrix represents a certain feature map and includes all the eigenvalues ​​on the channel map.

[0019] In step S3, based on the self-attention model at the regional node level, a feature map after regional node association enhancement is calculated;

[0020] In step S3, based on the channel-level self-attention, the feature map after channel semantic enhancement is calculated;

[0021] In step S3, the feature map after regional node association enhancement and the feature map after channel semantic enhancement are fused and calculated, and input into the decoder of the segmentation backbone network to generate a segmentation image.

[0022] A further optimization scheme is that the feature map calculation after regional node association enhancement includes:

[0023] Input reshape matrix , using linear projection to obtain three feature maps with the same dimension , and ,in , for the feature map and The transpose of is used for matrix multiplication, and the Softmax function is applied to calculate the regional node attention map , ,and .

[0024] The regional node attention map The jth row and ith column in , represents the influence of the i-th regional node on the j-th regional node, where , the more similar the feature representations of the two positions are, the greater the correlation between them; the regional node attention map With feature map Multiply and combine the result with the reshaped matrix of the model input Add together to finally get the feature map of the model's output region node association enhancement , ,and .

[0025] Feature map after regional node association enhancement It is the weighted sum of the features at all positions and the original features, has global spatial information, and enhances the features according to the regional node attention map.

[0026] The further optimization scheme is that the feature map calculation after channel semantic enhancement includes:

[0027] Input transposed matrix , using linear projection to obtain three feature maps with the same dimension , and , ; For feature map and The transpose of is used for matrix multiplication, and the Softmax function is applied to calculate the channel attention map , ,and .

[0028] The channel attention map The i-th row and j-th column in , represents the influence of the jth channel on the ith channel, where , The closer the value is to 1, the greater the impact.

[0029] Channel Attention Map With feature map Multiply and add the result to the transposed matrix of the model Add together to get the feature map of cross-channel semantic enhancement , ,and .

[0030] Feature map after channel semantic enhancement It includes the semantic information of all feature maps, has global spatial information, and enhances features at the channel level.

[0031] The further optimization scheme is that the fusion computing includes:

[0032] The inputs of the regional node level self-attention model and the channel level self-attention model are both feature maps output by the segmentation backbone encoder , so the feature map after regional node association enhancement Feature map after channel semantic enhancement can be merged by directly adding them. Transpose, and then Add together to get the fused feature map , the fused feature map It includes both the spatial connections between nodes in long-distance regions and the semantic associations across channels.

[0033] The feature map Reshaped back The decoder of the segmentation backbone network outputs the segmentation result as a segmented image.

[0034] A further optimization scheme is that the multi-scale image restoration module adopts a pyramid network architecture, extracts features from multiple scales, performs pooling operations on the segmented image at each scale, generates feature maps of different sizes, and concatenates and weightedly fuses the feature maps to form a feature map including multi-scale information as the output for the first restored image.

[0035] A further optimization scheme is that the generative adversarial network includes a generator network and a discriminator network, and the generative adversarial network used by the deep learning model restores the first restored image and outputs a second restored image, including:

[0036] The first restored image is input into the generator network, and a generated image is output; wherein the generator network uses a convolutional neural network to map the first restored image to the generated image; the generator network includes 1 initialization model and 6 upsampling models to generate feature maps of different resolutions; the first restored image is used as a correction of the high-resolution image in the channel direction through category information and low-resolution feature maps; the generator network gradually generates a high-resolution image from the initial category information and noise vector by alternately stacking upsampling models, and outputs the generated image.

[0037] The generated image and the standard image are used as input to the discriminator network; the convolutional layer is used to extract image features and classify them, the similarity between the generated image and the standard image is calculated, and the probability that the output image is the original one is obtained.

[0038] The perceptual loss of the loss function is calculated by the convolutional network, which calculates the difference between the generated image and the standard image in the feature space, and then calculates the difference value between the features to measure the quality of the generated image.

[0039] Among them, after the first restored image is input into the generated image output by the generator network, the generator network repeatedly uses its output generated image as input to generate a new generated image again until the image quality difference value of the generated image of the generator network is the lowest, and then the final generated image is used as the second restored image, and the above steps are repeated until the difference value between the features of the generated image of the generator network is the lowest and the generated image with the highest image quality record is used as the second restored image.

[0040] A further optimization scheme is that the ray projection algorithm adopted by the 3D reconstruction module generates a ray along the direction of the captured sampling standard image, samples each point of the ray at the same interval for the second restored image, and performs visual mapping on each sampling point, assigning different color and opacity values ​​to each sampling point, and then accumulates the color and opacity values ​​of all sampling points in the specified direction according to the imaging rules, and processes the second restored image through the 3D reconstruction module as the final output completed processed image.

[0041] The present invention also includes an image processing system based on a multi-stage processing architecture, including a data pre-processing unit, a denoising processing unit, an image enhancement and segmentation unit, and an image restoration unit connected in sequence.

[0042] The data preprocessing unit inputs an image, adjusts the image to a set size and resolution, and obtains a standard image; adds random Gaussian noise and crops it into an image of the same size as the standard image, and obtains a noisy image; adjusts the image format of the standard image and the noisy image, performs normalization processing, and scales the pixel values ​​to the interval [-1, 1]; and outputs the standard image and the noisy image.

[0043] The denoising processing unit includes a wavelet denoising module and a convolutional neural network module; the standard image and the noise image are input and wavelet decomposition is performed through the wavelet denoising module to obtain a wavelet denoised image; the wavelet denoised image is denoised by using a convolutional neural network to output a feature image; and the network is trained to remove noise.

[0044] The wavelet denoising module performs wavelet decomposition on the image to obtain four sub-band images; the four sub-band images are input into the denoising network of the wavelet denoising module, and the denoising network is optimized using the loss function until the loss is stable; an image is selected from the standard image, and the same wavelet decomposition and network processing are performed to obtain the denoised component; the reconstructed image is synthesized using wavelet inverse transform to restore it to the standard image quality level, and the wavelet denoised image is output.

[0045] The convolutional neural network module includes convolution layer, pooling layer, ReLU activation function and deconvolution layer; the input wavelet denoised image is used to extract local features through the convolution layer, and the feature map is generated through the convolution operation; the pooling layer reduces the spatial dimension of the feature map and retains important features; the ReLU activation function is applied to introduce nonlinear features; the deconvolution layer performs upsampling, restores the image size, and outputs the denoised image.

[0046] The image enhancement and segmentation unit inputs the denoised image into the segmentation backbone encoder, uses the regional node level self-attention model and the channel level self-attention model to capture the spatial connection and semantic association respectively, enhances the features, and then sends it to the decoder of the segmentation backbone network for segmentation, and outputs the segmented image.

[0047] The image restoration unit includes a multi-scale image restoration module, a deep learning detail restoration module and a three-dimensional reconstruction module which are connected in sequence.

[0048] The multi-scale image restoration module includes input processing, feature extraction and feature fusion.

[0049] Input processing, inputting the segmented image into the multi-scale image restoration module to process details of different scales; the feature extraction, using the pyramid network architecture to perform multi-scale feature extraction; obtaining feature maps of different sizes through 1x1, 2x2, 4x4 and 8x8 pooling windows; the feature fusion, performing weighted fusion on feature maps of different scales to generate a first restored image, and calculating the mean square error between the first restored image and the standard image to adjust the pooling window size.

[0050] The deep learning detail restoration module adopts a generative adversarial network module, including a generator network and a discriminator network; the first restored image is input into the generator network, and the generator network adopts a convolutional neural network to generate a high-resolution image through classification information and low-resolution feature maps, and outputs the generated image; the discriminator network is used to evaluate the quality of the generated image, and the perceptual loss is introduced to calculate the quality of the discriminant image, and the second restored image is output as the generated image.

[0051] The image restoration unit adopts the ray projection algorithm of the 3D reconstruction module to generate rays along the shooting direction, samples the second restored image, accumulates color and opacity values ​​according to the imaging rules, and processes the second restored image through the 3D reconstruction module as the final output completed processed image.

[0052] In summary, the present invention has the following beneficial effects compared with the prior art:

[0053] The hardware platform of the system of the present invention can be implemented on a high-performance computing platform or an embedded device, supports large-scale parallel computing, and ensures real-time processing; and adopts a software framework, using a modular software framework to achieve independent calling and integration of different image processing functions, support flexible configuration and expansion, and enhance system flexibility, support the integration of multiple image processing tasks, and realize a wider range of application scenarios, such as medical imaging, satellite remote sensing, surveillance video, etc.

[0054] Multi-stage processing: A multi-stage processing architecture is adopted to provide a systematic image noise processing and detail method through Gaussian noise addition, wavelet denoising, convolutional neural network denoising and multi-scale restoration. It can also be expanded to apply to other irregular morphological image processing, including cancer CT image processing and radar image processing.

[0055] Adaptive denoising: Combining transform domain denoising and convolutional neural network, it effectively removes mixed noise in the image while retaining image details, improving image clarity and fidelity compared to single denoising methods.

[0056] Enhanced features: Through the regional node level self-attention model and the channel level self-attention model, it is possible to capture long-distance spatial relationships and cross-channel semantic associations, perform enhanced features, and improve the accuracy of image segmentation and the ability to restore details.

[0057] Details: Utilize the multi-scale image restoration module and the 3D reconstruction module to enhance image details, so that the final output image has higher quality and accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] The drawings described herein are used to provide a further understanding of the embodiments of the present invention, constitute a part of this application, and do not constitute a limitation of the embodiments of the present invention. In the drawings:

[0059] Figure 1 A schematic diagram of the steps of an image processing method based on a multi-stage processing architecture;

[0060] Figure 2 A schematic diagram of image enhancement and segmentation steps of an image processing method based on a multi-stage processing architecture;

[0061] Figure 3 A schematic diagram of the steps of outputting a second restored image using an image processing method based on a multi-stage processing architecture. DETAILED DESCRIPTION

[0062] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with embodiments and drawings. The exemplary embodiments of the present invention and their description are only used to explain the present invention and are not intended to limit the present invention.

[0063] The existing image recognition methods mainly use multi-level convolutional networks to process and analyze images. When applied in actual scenarios, the computational complexity is high, it is difficult to retain details, and there are large errors. In view of this, the present invention provides the following embodiments to solve the above technical problems.

[0064] like Figure 1 As shown, in an embodiment, an image processing method based on a multi-stage processing architecture is specifically as follows:

[0065] Step S1, input an image to be processed, adjust the image to a set size and resolution to obtain a standard image, add random Gaussian noise to the standard image, and crop the image with Gaussian noise added to an image with the same size and resolution as the standard image to obtain a noisy image.

[0066] Step S2: using transform domain denoising technology to perform wavelet decomposition on the noisy image, increasing the receptive field to obtain a wavelet denoised image; then denoising the wavelet denoised image through a convolutional neural network, and outputting a feature image as a denoised image.

[0067] Step S3, performing image enhancement and segmentation on the denoised image through a segmentation backbone network; wherein the denoised image is input into the encoder of the segmentation backbone network to extract multiple image region nodes and feature maps; a region node level self-attention model and a channel level self-attention model are used to capture the spatial connection between region nodes and the semantic association across channels respectively, and the output fusion representation is used as an enhanced feature; the enhanced feature is sent to the decoder of the segmentation backbone network to output a segmented image.

[0068] Step S4, input the segmented image to restore the image; wherein the restored image includes: processing details of different scales through a multi-scale image restoration module to form a feature map of multi-scale information as an output of a first restored image; using a generative adversarial network adopted by a deep learning model to restore the first restored image to output a second restored image; processing the second restored image through a three-dimensional reconstruction module to obtain a final output completed processed image.

[0069] In this embodiment, an image processing system based on a multi-stage processing architecture is also provided, including a data pre-processing unit, a denoising processing unit, an image enhancement and segmentation unit, and an image restoration unit connected in sequence.

[0070] The data preprocessing unit inputs an image, adjusts the image to a set size and resolution, and obtains a standard image; adds random Gaussian noise and crops it into an image of the same size as the standard image, and obtains a noisy image; adjusts the image format of the standard image and the noisy image, performs normalization processing, and scales the pixel values ​​to the interval [-1, 1]; and outputs the standard image and the noisy image.

[0071] The denoising processing unit includes a wavelet denoising module and a convolutional neural network module; the input standard image and the noise image are subjected to wavelet decomposition through the wavelet denoising module to obtain a wavelet denoised image; the wavelet denoised image is denoised using a convolutional neural network to output a feature image; and the network is trained to remove noise.

[0072] The wavelet denoising module performs wavelet decomposition on the image to obtain four sub-band images; the four sub-band images are input into the denoising network of the wavelet denoising module, and the denoising network is optimized using the loss function until the loss is stable; an image is selected from the standard image, and the same wavelet decomposition and network processing are performed to obtain the denoised component; the reconstructed image is synthesized using wavelet inverse transform to restore it to the standard image quality level, and the wavelet denoised image is output.

[0073] Based on the existing deep learning technology, methodological improvements are made, and the influence of ambient lighting and shooting angle is taken into consideration. Environmental information is comprehensively considered in the feature extraction and classification process. Transform domain denoising technology and convolutional neural network are used to denoise the acquired images, and a segmentation backbone network combined with a parallel self-attention model is used for segmentation. A multi-scale image restoration module is used to restore the image, and a three-dimensional reconstruction module is used to reduce the errors caused by environmental factors. A processing method for images acquired in various environments is provided.

[0074] The convolutional neural network module includes convolution layer, pooling layer, ReLU activation function and deconvolution layer; the input wavelet denoised image is used to extract local features through the convolution layer, and the feature map is generated through the convolution operation; the pooling layer reduces the spatial dimension of the feature map and retains important features; the ReLU activation function is applied to introduce nonlinear features; the deconvolution layer performs upsampling, restores the image size, and outputs the denoised image.

[0075] The image enhancement and segmentation unit inputs the denoised image into the segmentation backbone encoder, uses the regional node level self-attention model and the channel level self-attention model to capture the spatial connection and semantic association respectively, enhances the features, and then sends it to the decoder of the segmentation backbone network for segmentation, and outputs the segmented image.

[0076] The image restoration unit includes a multi-scale image restoration module, a deep learning detail restoration module and a three-dimensional reconstruction module which are connected in sequence.

[0077] The multi-scale image restoration module includes input processing, feature extraction and feature fusion.

[0078] Input processing: input the segmented image into the multi-scale image restoration module to process details of different scales; feature extraction: use the pyramid network architecture to perform multi-scale feature extraction; obtain feature maps of different sizes through 1x1, 2x2, 4x4 and 8x8 pooling windows; feature fusion: perform weighted fusion on feature maps of different scales to generate the first restored image, and calculate the mean square error between the first restored image and the standard image to adjust the pooling window size.

[0079] The deep learning detail restoration module adopts a generative adversarial network module, including a generator network and a discriminator network; the first restored image is input into the generator network, and the generator network adopts a convolutional neural network to generate a high-resolution image through classification information and low-resolution feature maps, and outputs the generated image; the discriminator network is used to evaluate the quality of the generated image, and the perceptual loss is introduced to calculate the quality of the discriminant image, and the second restored image is output as the generated image.

[0080] The image restoration unit adopts the ray projection algorithm of the 3D reconstruction module to generate rays along the shooting direction, samples the second restored image, accumulates color and opacity values ​​according to the imaging rules, outputs an image with obvious features, and processes the second restored image through the 3D reconstruction module as the final output completed processed image.

[0081] In this embodiment, specific method steps applicable to the image processing method and system based on the multi-stage processing architecture include:

[0082] Step S1: input the image to be processed, adjust the image to a set size and resolution, the set size and resolution can be set according to the specific situation, obtain a standard image, add random The image with Gaussian noise is cropped into an image with the same size and resolution as the standard image to obtain a noisy image.

[0083] If the image to be processed is a group of images, and there are several images in the group, it is necessary to correspond the standard image and the noise image of each image in the group, and the standard image and the noise image of each image are grouped together for subsequent processing.

[0084] We collected various images as original images, adjusted the original images, and adjusted all the images to the same size and resolution. This ensures that when they are input into the generative adversarial network, all images have a consistent format for easy network processing.

[0085] Normalize the adjusted image, normalize the pixel values ​​of the image, and scale the pixel values ​​to the interval [-1, 1] to obtain a standard image, which helps to speed up the training of the network and improve stability.

[0086] Add random The image with Gaussian noise is cropped into an image with the same size and resolution as the standard image to obtain a noisy image; the original clean image training set is set to , add Gaussian noise , we can get the noisy image .

[0087] Step S2, corresponding to the standard image and the noise image, use the transform domain denoising technology to perform wavelet decomposition on the noise image, increase the receptive field to obtain the wavelet denoised image; then use the convolutional neural network to denoise the wavelet denoised image, output the feature image as the denoised image, remove the noise in the image, and retain the image details. The transform domain denoising technology of the present invention uses wavelet transform to process the noise in a specific frequency range; the neural network uses a denoising network of a convolutional neural network to process different types of noise. Using a convolutional neural network with deep learning, the deep noise neural network can increase the receptive field but will lead to a high computational burden. Therefore, the multi-resolution characteristics of the wavelet transform and the sparsity of the wavelet coefficients in the transform domain denoising technology are used to expand the receptive field and reduce the computational burden.

[0088] Because the noise in the actual image is a mixture of multiple noises, it is not possible to use only a single adaptive weighted mean filter or wavelet threshold to denoise the image. Using only a single method to remove the mixed noise in the image will make the image blurred.

[0089] Step S21: firstly, the standard image and the noise image are decomposed by wavelet transform domain denoising technology to remove the noise in the image while retaining the image details.

[0090] Transform domain denoising technology is a method of reducing noise by converting a signal or image from a spatial domain to a transform domain. The noise is divided into high, medium and low frequency noise in terms of frequency, and noise of different frequencies is separated using a transform domain method. Then, an inverse transform is performed to convert the image from the transform domain to the original spatial domain, ultimately achieving the purpose of removing image noise. Compared with the method of directly processing the image in the spatial domain, the transform domain denoising technology used in the present invention can more effectively identify noise and more effectively remove unnecessary noise. Wavelet transform is a commonly used method in this type of technology. Wavelet transform is a method of reorganizing a signal into different frequency quantities, which can provide time and frequency information at the same time. Unlike Fourier transform, wavelet transform uses a wavelet basis function with finite support, so it has a greater advantage in processing unstable and non-stationary signals.

[0091] Step S211, the standard image and the noise image are matched, and the two images are decomposed by wavelet, and the images are converted from the spatial domain to the wavelet transform domain. The images are decomposed into different frequency bands in the wavelet domain, so that the noise can be effectively removed in a specific frequency band, and four corresponding sub-band images are obtained respectively, namely Four sub-band images.

[0092] In step S212, the four sub-band images of the standard image and the noise image are respectively input into the designed network for training in one-to-one correspondence, and the loss function is used to reversely adjust the network parameters to make the network converge. When the loss tends to be stable, the training is stopped.

[0093] Step S213: Select a picture from the standard image. There is no need to crop the picture. Just do the same operation as step S211, and then input it into the trained network to obtain four denoised components. .

[0094] Step S214, the denoised components are reconstructed by using inverse wavelet transform to convert the denoised image from the wavelet domain back to the spatial domain. There is little loss in the denoising process, and the image quality is roughly at the original level.

[0095] Step S22, design a convolutional neural network: design a network architecture using a convolutional layer, an activation function ReLU, a pooling layer, and a deconvolution layer.

[0096] Step S221, convolution layer: When the image passes through the convolution layer, the convolution kernel slides on the image to extract local features. Each convolution operation generates a new feature map, which is the feature representation of the image.

[0097] The convolution layer extracts features from the input data through convolution operations. The number of convolution kernels obtained by training the convolution network is used to effectively extract features from the input data. Each convolution kernel learns the input data with a set step size, and uses a matrix of a set size as the convolution kernel to slide on the input image according to the set step size.

[0098] At each location, the area covered by the convolution kernel With convolution kernel The weight matrix is ​​multiplied element by element and summed to get a convolution result. .

[0099] Perform convolution operations on the feature map, put the results of each convolution operation into the corresponding position of the output feature map, use the results of the convolution operation as the extracted features, and output feature maps with the same number of convolution kernels. For convolution operations involving the same convolution kernel, the weights of the convolution kernels are shared.

[0100] As hyperparameters in the network, the convolution kernel size, step size, and padding jointly determine the size of the feature map, where the step size refers to the position interval size of the convolution kernel sliding position, which can perform pooling operations on the feature map.

[0101] There is an input image area and a 3x3 convolution kernel. The specific calculation is as follows:

[0102] Enter an image region

[0103] Let the convolution kernel be:

[0104] The convolution operation at this time has a step size of 1 and a padding of 0, covering the convolution kernel in the upper left corner of the image.

[0105] Perform element-wise multiplication and sum:

[0106] Store the result -6 in the corresponding position of the feature map.

[0107] Repeat the above steps to perform convolution operation on the entire input image and generate a feature map.

[0108] Setting the step size and padding operation will change the feature map output size, which is The feature map of size is , the step length is , the padding size is After the convolution operation, the output feature map space size can be expressed as:

[0109]

[0110] In the convolution operation, the convolution layer can automatically learn and extract local features in the image, including edges and corners, and has spatial invariance. The convolution operation is robust to the spatial translation of the image, that is, the same features can be detected regardless of their position in the image.

[0111] And in the convolution operation, the parameters are shared, and a convolution kernel can be shared across the entire image, reducing the number of parameters and thus reducing computational complexity.

[0112] Step S222, pooling layer: The pooling layer reduces the spatial dimension of the feature map through pooling operations, reduces the computational complexity, and retains the most important features. The size of the image is reduced, but the key features are retained.

[0113] The calculation of the pooling layer includes maximum pooling or average pooling, determining the pooling operation within the pooling window, and the pooling operation includes selecting the maximum value or calculating the average value of the data in the window;

[0114] Determine the size and stride of the window. The window size determines the area covered by the pooling operation, while the stride determines the step size of the window movement on the image.

[0115] Slide the pooling window across the input feature map, perform a pooling operation on the features in each window area, and fill the results of the pooling operation into the corresponding positions of the output feature map.

[0116] Repeat the above steps until the entire feature map is processed.

[0117] The output feature map is obtained without the need for training parameters. The calculation results can be directly obtained to finally obtain the feature values ​​of each region.

[0118] Step S223, activation function ReLU: After applying the activation function ReLU, the convolutional neural network performs linear operations on the input information. The present invention takes into account that when the convolutional neural network expresses more complex features, it is necessary to apply the activation function ReLU with nonlinear characteristics, so that the network has nonlinear mapping capabilities. The nonlinear characteristics of the image are enhanced, which helps the network learn complex noise patterns.

[0119] The mathematical expression of the activation function ReLU is:

[0120] Input: Input value x is fed into the ReLU activation function.

[0121] Calculation: Calculate the larger value of the input value and 0; if x is greater than 0, ReLU outputs x; if x is less than or equal to 0, ReLU outputs 0.

[0122] The activation function ReLU can introduce nonlinearity, enabling the network to learn and model complex functions and enhance the nonlinear characteristics of the image. The calculation of ReLU is simple and efficient, faster and more efficient than other activation functions. Since ReLU sets the negative value to 0, it will produce sparse activation, thereby improving the efficiency of the model.

[0123] Step S224, deconvolution layer: The deconvolution layer uses the transposed convolution layer for upsampling to increase the spatial size of the feature map. The spatial resolution of the image, the deconvolution layer enlarges the spatial dimension of the feature map by learning a transposed convolution kernel, so that the denoised image is close to the size of the standard image.

[0124] Define the transposed convolution kernel and choose the size of the convolution kernel to be 3x3, which is similar to the convolution kernel used in the convolution layer.

[0125] Interpolation and convolution are performed. First, zeros are inserted between each pixel of the input feature map to increase the size of the feature map. 0 is inserted to generate a 4x4 feature map from a 2x2 feature map. A convolution operation is performed on the expanded feature map using a transposed convolution kernel. This step is similar to a regular convolution operation, but is performed in reverse to achieve higher spatial resolution.

[0126] Generate an output feature map, and use the result of the convolution operation as the value of the output feature map. Convolve the interpolated feature map with the convolution kernel to expand the spatial size and generate the final output feature map.

[0127] A 2x2 input feature map and a 3x3 convolution kernel are calculated as follows:

[0128] Input feature map

[0129] After interpolation, we get

[0130] Apply the transposed convolution kernel

[0131] The generated 4x4 output feature map is:

[0132] Through these steps, the image changes at each denoising stage, from the input standard image to the final output denoised image. Each step has its specific purpose and impact to ensure that the final image quality is at the best level possible.

[0133] like Figure 2 As shown, the specific steps of step S3 image enhancement and segmentation are as follows:

[0134] Traditional convolution encodes the features of the area covered by the convolution kernel when downsampling, which makes it difficult to capture the relationship between distant nodes in the image. In addition, the downsampling process often involves dimensionality compression, which can extract more representative features but also loses some detailed information.

[0135] This embodiment adopts a segmentation method based on a regional node level self-attention model and a channel level self-attention model, which is used to capture the spatial connection between long-distance regional nodes and the semantic association between channels respectively and fuse them.

[0136] It consists of three sub-steps. First, the encoder of the segmentation backbone network is used to extract feature maps, which include multiple image region nodes and feature maps. 3DnnU-Net is used as the segmentation backbone network, and the feature maps extracted by its encoder contain texture, morphology, and semantic features. Then, a parallel self-attention model including a regional node-level self-attention model and a channel-level self-attention model is proposed to capture the spatial connection between regional nodes and the semantic association across channels respectively. Finally, the two outputs of the parallel attention model are fused to further enhance the features and sent to the decoder of the segmentation backbone network to obtain the final segmentation result, which is used as the segmented image.

[0137] Step S31, the present invention adopts 3DnnU-Net including encoder-decoder structure as the segmentation backbone network, and the denoised image is used as the input of the network. The encoder in the network includes 6 encoding layers, each of which is composed of 3×3×3 convolution blocks, 3×3×3 strided convolution blocks, instance normalization and LeakyReLU activation. The decoder also contains 6 decoding layers, in each layer we perform transposed convolution, instance normalization and LeakyReLU activation to achieve the purpose of decoding. A jump connection is added between the encoding layer and the corresponding decoding layer to assist in decoding segmentation.

[0138] The segmentation backbone network uses multiple layers of convolution operations to convert the original pixel data of the image into gradually abstract feature maps. These feature maps capture the high-level features of the image, including edges, textures, and shapes.

[0139] Deep convolutional networks, using deep convolutional neural networks, can extract high-level features in images through multi-layer convolution operations. Deep convolutional networks build a network architecture that can capture and learn complex features of images.

[0140] The image is gradually extracted with high-level features through multiple convolutional layers, and each layer gradually learns deeper features of the image to form a feature map.

[0141] The segmentation backbone network in step S3 includes multiple layers of convolution operations, and the last layer output feature map is recorded as ,and , Each position of corresponds to a local area of ​​the denoised image, which is used as an image area node.

[0142] H represents the height of the feature map, which indicates the number of pixels in the vertical direction of the feature map;

[0143] W represents the width of the feature map, which indicates the number of pixels in the horizontal direction of the feature map;

[0144] D represents the depth of the feature map;

[0145] C represents the number of channels of the feature map, indicating how many eigenvalues ​​there are at each spatial position, including different feature information;

[0146] H and W describe the spatial size of the feature map, and D describes the depth of the feature map. In three-dimensional image processing, let Describe the three-dimensional size, Reshape into a matrix , reshape the matrix Each row in represents an image region node with C eigenvalues; and there is a reshaping matrix The transposed matrix of , , each row of the transposed matrix represents a certain feature map and includes all the eigenvalues ​​on the channel map.

[0147] Based on the regional node level self-attention model, the feature map after regional node association enhancement is calculated, which is used to capture the long-distance association between nodes.

[0148] For example: in the image processing scenario where the present invention is applied to analyze the lesions of leaves in nature, in order to emphasize the lesions of the leaves, an image with the lesion area enhanced is output. In the segmentation task, in the actual natural world, many diseased leaves have multiple lesions of different sizes on both sides, which makes there is a long distance between the lesions, and these lesions are the captured nodes; in the scenario where the present invention is applied to perform radar analysis of the sea surface and thus process images, in order to emphasize the situation of ships in the sea area, an image with the area where each ship is located is output. In the segmentation task, because in the actual sea surface situation, there are many ships on the sea surface, the ships are of different sizes and the distribution is also random, which makes there are different semantics and long distances between some ships, and these ships are the captured nodes.

[0149] Input reshape matrix , using linear projection to obtain three feature maps with the same dimension , and ,in , for the feature map and The transpose of is used for matrix multiplication, and the Softmax function is applied to calculate the regional node attention map , ,and .

[0150] Regional Node Attention Map It is essentially a matrix, where the regional node attention map The jth row and ith column in , represents the influence of the i-th regional node on the j-th regional node, where , the more similar the feature representations of the two positions are, the greater the correlation between them; the regional node attention map With feature map Multiply and combine the result with the reshaped matrix of the model input Add together to finally get the feature map of the model's output region node association enhancement , ,and .

[0151] Feature map after regional node association enhancement It is the weighted sum of the features at all positions and the original features, has global spatial information, and enhances the features according to the regional node attention map.

[0152] Step S33: Based on channel-level self-attention, calculate the feature map after channel semantic enhancement to capture the semantic relationship between channel maps.

[0153] In the feature maps extracted by the segmentation network encoder, each channel map can be regarded as a response to specific semantics, and there is a correlation between these responses. By capturing the semantic relationship between channel maps, the semantic information of each feature map can be made more comprehensive, thereby improving the segmentation accuracy of the model. Therefore, a self-attention mechanism at the channel level is adopted to enhance the semantic features of the channel.

[0154] The channel feature enhancement model is similar to the regional node feature enhancement model. The input transposed matrix , using linear projection to obtain three feature maps with the same dimension , and , ; For feature map and The transpose of is used for matrix multiplication, and the Softmax function is applied to calculate the channel attention map , ,and .

[0155] Channel Attention Map It is essentially a matrix, where the channel attention map The i-th row and j-th column in , represents the influence of the jth channel on the ith channel, where , The closer the value is to 1, the greater the impact.

[0156] Channel Attention Map With feature map Multiply and add the result to the transposed matrix of the model Add together to get the feature map of cross-channel semantic enhancement , ,and .

[0157] Feature map after channel semantic enhancement It includes the semantic information of all feature maps, has global spatial information, and enhances features at the channel level.

[0158] Step S34: perform fusion calculation on the enhanced feature map, fuse the feature map after regional node association enhancement and the feature map after channel semantic enhancement, and input them into the decoder of the segmentation backbone network to generate a segmented image.

[0159] The input of the region node level self-attention model and the channel level self-attention model are both feature maps output by the segmentation backbone encoder. , so the feature map after regional node association enhancement Feature map after channel semantic enhancement can be merged by directly adding them. Transpose, and then Add together to get the fused feature map , the fused feature map It includes both the spatial connections between nodes in long-distance regions and the semantic associations across channels.

[0160] The feature map Reshaped back The decoder of the segmentation backbone network outputs the segmentation result as a segmented image.

[0161] Step S4: restore the image.

[0162] The input segmented image is processed through the multi-scale image restoration module to process details of different scales, and the details and textures in the image are enhanced using the deep learning model. The processed image is output through the ray projection algorithm of the 3D reconstruction module.

[0163] Step S41, the multi-scale image restoration module uses a pyramid network architecture to extract features from multiple scales, performs pooling operations on the segmented image at each scale, generates feature maps of different sizes, and concatenates and weights the feature maps to form a feature map including multi-scale information as the output of the first restored image. The multi-scale image restoration module is used to process the details of different scales of the image, making the restoration effect more comprehensive and accurate.

[0164] Multi-scale feature extraction: The segmented image is input into the pyramid network architecture to extract features from multiple scales and capture information at different scales.

[0165] Pooling is performed on the segmented image at each scale to capture features at different scales. This can obtain richer contextual information and help the segmentation network to perform effective feature fusion at different scales.

[0166] The image is pooled at different scales to generate feature maps of different sizes, which are then concatenated and fused to form a feature map that includes multi-scale information. This process enhances the ability to capture features of different scales.

[0167] The input image is processed through hierarchical pooling operations of different sizes, such as 1x1, 2x2, 4x4, and 8x8 pooling windows.

[0168] Combine feature information from different scales to reconstruct details. A feature fusion strategy can be used, feature weighted fusion, to weightedly fuse the pooling results at different scales to form a feature map of multi-scale information as the output of the first restored image. This operation will merge information from different scales into a more expressive feature representation.

[0169] The mean square error of the pixel-level difference between the first restored image and the standard image is calculated to evaluate the image quality and adjust the window size of the hierarchical pooling operation.

[0170] Step S42: The generative adversarial network adopted by the deep learning model restores the first restored image and outputs a second restored image, using the details and textures in the deep learning model image, wherein the generative adversarial network includes a generator network and a discriminator network.

[0171] The first restored image is restored with high quality through a generative adversarial network.

[0172] Generative Adversarial Network: The generator network and the discriminator network process and evaluate the image respectively to generate a restored image with prominent details and textures. The generator network is responsible for generating images with prominent features, while the discriminator network is used to evaluate the quality of the generated image.

[0173] A generative adversarial network is introduced for image restoration. During the training process of the generative adversarial network, two network models are continuously trained in the adversarial process. The generator network can optimize the contrast and brightness of the image. The generator network fits the feature-enhanced image by learning the standard image and generates a series of generated images. The discriminator network is used to distinguish the difference between the original and generated data, and calculates the similarity between the original and generated data through the feature map to ensure the visual naturalness and originality of the restored image.

[0174] Step S421: The input image is passed through the generator network to generate a generated image. The generator network uses a convolutional neural network to map the first restored image to the generated image.

[0175] The first restored image is input into the generator network, and the generated image is output. The generator network uses a convolutional neural network to map the first restored image to the generated image. The generator network includes 1 initialization model and 6 upsampling models to generate feature maps of different resolutions. The first restored image is used as a correction of the high-resolution image in the channel direction through category information and low-resolution feature maps. The generator network gradually generates a high-resolution image from the initial category information and noise vector by alternately stacking upsampling models, and outputs the generated image.

[0176] Step S422: Use the generated image and the standard image as inputs of the discriminator network.

[0177] The generated image and the standard image are used as the input of the discriminator network. The convolution layer is used to extract image features and classify them. The similarity between the generated image and the standard image is calculated. The probability that the output image is the original one is used as a quality assessment. A threshold is set. If it is within the threshold, it is judged whether its visual effect is similar to the standard image.

[0178] Step S423: Calculate the loss function.

[0179] When the quality assessment reaches the threshold and the generated image is judged to be similar to the standard image, if they are similar, the perceptual loss of the loss function is calculated. The difference between the generated image and the standard image in the feature space is calculated through the convolutional network, and then the difference value between the features is calculated to measure the quality of the generated image.

[0180] Introducing perceptual loss can help improve the visual quality of generated images. Perceptual loss is calculated through a convolutional network to calculate the difference between the generated image and the standard image in the feature space. The generated image and the standard image are extracted through the trained network, and then the difference between these features is calculated to measure the quality of the generated image.

[0181] And in the specific embodiment, adversarial loss is introduced. For the generator network, it is to maximize the probability of misjudgment of the discriminator network on the generated image to improve the quality of the generated image; for the discriminator network, it is to correctly classify the standard image and the generated image, maximize the correct classification rate of the standard image, and minimize the misjudgment rate of the generated image.

[0182] Repeat step S421 and step S422,

[0183] The image input to the generator network for the first time is restored for the first time, and the generated image is named the first generated image, corresponding to the first image quality;

[0184] The image input to the generator network for the second time generates the image for the first time, and the generated image is named the second generated image, corresponding to the second image quality;

[0185] The image input to the generator network for the third time generates the image for the second time. The generated image is named the third generated image, corresponding to the third image quality;

[0186] The adversarial generation is continued until the image quality record of the generator-generated image is the lowest. After the first restored image is input into the generator network and the generated image is output, the generator network repeatedly uses its output generated image as input to generate a new generated image again until the image quality difference value of the generator network's generated image is the lowest, and then the final generated image is used as the second restored image.

[0187] Step S43, the ray projection algorithm of the three-dimensional reconstruction module can generate the point information of the image through the second restoration image, and output an image with obvious features. The principle is simple and easy to implement. The three-dimensional reconstruction module is a process of generating a three-dimensional model by extracting information from multiple two-dimensional images or other sensor data. The ray projection method of the three-dimensional reconstruction module is to generate a ray along the direction of the line of sight, sample each point of the ray at the same interval, and perform visual mapping on each sampled point.

[0188] In the embodiment, a ray is generated along the direction of shooting the sampled standard image, each point of the ray is sampled at the same interval for the second restored image, and visual mapping is performed on each sampling point, and a different color and opacity value is assigned to each sampling point. Then, the color and opacity values ​​of all sampling points are accumulated in the specified direction according to the imaging rules, and finally an image with obvious features is obtained. The second restored image is processed by a three-dimensional reconstruction module as a completed processed image as the final output.

[0189] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. An image processing method based on a multi-stage processing architecture, characterized in that: include: Step S1, inputting an image to be processed, adjusting the image to a set size and resolution to obtain a standard image, adding random Gaussian noise to the standard image, and cropping the image with Gaussian noise added to an image with the same size and resolution as the standard image to obtain a noise image; Step S2, using transform domain denoising technology to perform wavelet decomposition on the noisy image, increasing the receptive field to obtain a wavelet denoised image; then denoising the wavelet denoised image through a convolutional neural network, and outputting a feature image as a denoised image; Step S3, performing image enhancement and segmentation on the denoised image through a segmentation backbone network; The encoder of the denoising image input segmentation backbone network extracts multiple image region nodes and feature maps; the region node level self-attention model and the channel level self-attention model are used to capture the spatial connection between region nodes and the semantic association across channels respectively, and the output fusion representation is used as enhanced features; The enhanced features are fed into the decoder of the segmentation backbone network to output the segmented image; Step S4, input the segmented image to restore the image; wherein the restored image includes: processing details of different scales through a multi-scale image restoration module to form a feature map of multi-scale information as an output of a first restored image; using a generative adversarial network adopted by a deep learning model to restore the first restored image to output a second restored image; processing the second restored image through a three-dimensional reconstruction module to obtain a final output completed processed image.

2. The image processing method based on a multi-stage processing architecture according to claim 1, characterized in that: The segmentation backbone network in step S3 includes multiple layers of convolution operations, and the last layer output feature map is recorded as ,and , Each position of corresponds to a local area of ​​the denoised image, and the local area serves as an image area node; H represents the height of the feature map, which indicates the number of pixels in the vertical direction of the feature map; W represents the width of the feature map, which indicates the number of pixels in the horizontal direction of the feature map; D represents the depth of the feature map; C represents the number of channels of the feature map, indicating how many eigenvalues ​​there are at each spatial position, including different feature information; H and W describe the spatial size of the feature map, and D describes the depth of the feature map. In three-dimensional image processing, let Describe the three-dimensional size, Reshape into a matrix , reshape the matrix Each row in represents an image region node with C eigenvalues; and there is a reshaping matrix The transposed matrix of , , each row of the transposed matrix represents a certain feature map and includes all the eigenvalues ​​on the channel map; In the step S3, based on the self-attention model at the regional node level, a feature map after regional node association enhancement is calculated; In the step S3, based on the channel level self-attention, a feature map after channel semantic enhancement is calculated; In the step S3, the feature map after regional node association enhancement and the feature map after channel semantic enhancement are fused and calculated, and input into the decoder of the segmentation backbone network to generate a segmentation image.

3. The image processing method based on a multi-stage processing architecture according to claim 2, characterized in that: The feature map calculation after the regional node association enhancement includes: Input reshape matrix , using linear projection to obtain three feature maps with the same dimension , and ,in , for the feature map and The transpose of is used for matrix multiplication, and the Softmax function is applied to calculate the regional node attention map , ,and ; The regional node attention map The jth row and ith column in , represents the influence of the i-th regional node on the j-th regional node, where , the more similar the feature representations of the two positions are, the greater the correlation between them; the regional node attention map With feature map Multiply and combine the result with the reshaped matrix of the model input Add together to finally get the feature map of the model's output region node association enhancement , ,and ; Feature map after regional node association enhancement It is the weighted sum of the features at all positions and the original features, has global spatial information, and enhances the features according to the regional node attention map.

4. The image processing method based on a multi-stage processing architecture according to claim 2, characterized in that: The feature map calculation after the channel semantic enhancement includes: Input transposed matrix , using linear projection to obtain three feature maps with the same dimension , and , ; For feature map and The transpose of is used for matrix multiplication, and the Softmax function is applied to calculate the channel attention map , ,and ; The channel attention map The i-th row and j-th column in , represents the influence of the jth channel on the ith channel, where , The closer the value is to 1, the greater the impact; Channel Attention Map With feature map Multiply and add the result to the transposed matrix of the model Add together to get the feature map of cross-channel semantic enhancement , ,and ; Feature map after channel semantic enhancement It includes the semantic information of all feature maps, has global spatial information, and enhances features at the channel level.

5. The image processing method based on a multi-stage processing architecture according to claim 2, characterized in that: The fusion calculation includes: The inputs of the regional node level self-attention model and the channel level self-attention model are both feature maps output by the segmentation backbone encoder , so the feature map after regional node association enhancement Feature map after channel semantic enhancement can be merged by directly adding them. Transpose, and then Add together to get the fused feature map , the fused feature map It includes both the spatial connections of nodes in long-distance regions and the semantic associations across channels; The feature map Reshaped back The decoder of the segmentation backbone network outputs the segmentation result as a segmented image.

6. The image processing method based on a multi-stage processing architecture according to claim 1, characterized in that: The multi-scale image restoration module adopts a pyramid network architecture, extracts features from multiple scales, performs pooling operations on the segmented image at each scale, generates feature maps of different sizes, and concatenates and weightedly fuses the feature maps to form a feature map including multi-scale information as the output for the first restored image.

7. The image processing method based on a multi-stage processing architecture according to claim 1, characterized in that: The generative adversarial network includes a generator network and a discriminator network. The generative adversarial network used by the deep learning model restores the first restored image and outputs a second restored image, including: The first restored image is input into the generator network, and the generated image is output; wherein the generator network uses a convolutional neural network to map the first restored image to the generated image; the generator network includes 1 initialization model and 6 upsampling models to generate feature maps of different resolutions; the first restored image is used as a channel-wise correction of the high-resolution image through category information and low-resolution feature maps; the generator network gradually generates a high-resolution image from the initial category information and noise vector by alternately stacking upsampling models, and outputs the generated image; The generated image and the standard image are used as inputs to the discriminator network; the convolutional layer is used to extract image features and classify them, the similarity between the generated image and the standard image is calculated, and the probability that the output image is the original one is obtained; Calculate the perceptual loss of the loss function, calculate it through the convolutional network, calculate the difference between the generated image and the standard image in the feature space, and then calculate the difference value between the features to measure the quality of the generated image; Among them, after the first restored image is input into the generated image output by the generator network, the generator network repeatedly uses its output generated image as input to generate a new generated image again until the image quality difference value of the generated image of the generator network is the lowest, and then the final generated image is used as the second restored image, and the above steps are repeated until the difference value between the features of the generated image of the generator network is the lowest and the generated image with the highest image quality record is used as the second restored image.

8. The image processing method based on a multi-stage processing architecture according to claim 1, characterized in that: The ray projection algorithm adopted by the 3D reconstruction module generates a ray along the direction of shooting the sampled standard image, samples each point of the ray at the same interval for the second restored image, and performs visual mapping on each sampling point, assigning different color and opacity values ​​to each sampling point, and then accumulating the color and opacity values ​​of all sampling points in the specified direction according to the imaging rules, and processing the second restored image through the 3D reconstruction module as the final output completed processed image.

9. An image processing system based on a multi-stage processing architecture, characterized in that: It includes a data pre-processing unit, a denoising processing unit, an image enhancement and segmentation unit and an image restoration unit which are connected in sequence; The data preprocessing unit inputs an image, adjusts the image to a set size and resolution, and obtains a standard image; adds random Gaussian noise and cuts it into an image of the same size as the standard image, and obtains a noise image; adjusts the image format of the standard image and the noise image, performs normalization processing, and scales the pixel values ​​to the interval [-1, 1]; and outputs the standard image and the noise image; The denoising processing unit includes a wavelet denoising module and a convolutional neural network module; Input the standard image and the noise image, perform wavelet decomposition through a wavelet denoising module, and obtain a wavelet denoised image; use a convolutional neural network to denoise the wavelet denoised image, and output a feature image; Train the network to remove noise; The wavelet denoising module performs wavelet decomposition on the image to obtain four sub-band images; the four sub-band images are input into the denoising network of the wavelet denoising module, and the denoising network is optimized by using a loss function until the loss is stable; an image is selected from the standard image, and the same wavelet decomposition and network processing are performed to obtain a denoised component; an inverse wavelet transform is used to synthesize and reconstruct the image, and the image is restored to the standard image quality level, and a wavelet denoised image is output; The convolutional neural network module includes a convolution layer, a pooling layer, a ReLU activation function and a deconvolution layer; the input wavelet denoised image is used to extract local features through the convolution layer, and a feature map is generated through a convolution operation; the pooling layer reduces the spatial dimension of the feature map and retains important features; the ReLU activation function is applied to introduce nonlinear features; The deconvolution layer performs upsampling, restores the image size, and outputs a denoised image; The image enhancement and segmentation unit inputs the denoised image to the segmentation backbone encoder, uses the regional node level self-attention model and the channel level self-attention model to capture the spatial connection and semantic association respectively, enhances the features, and then sends it to the decoder of the segmentation backbone network for segmentation, and outputs the segmented image; The image restoration unit includes a multi-scale image restoration module, a deep learning detail restoration module and a three-dimensional reconstruction module which are connected in sequence; The multi-scale image restoration module includes input processing, feature extraction and feature fusion; The input processing inputs the segmented image into the multi-scale image restoration module to process details of different scales; the feature extraction uses a pyramid network architecture to perform multi-scale feature extraction; feature maps of different sizes are obtained through 1x1, 2x2, 4x4 and 8x8 pooling windows; the feature fusion performs weighted fusion on feature maps of different scales to generate a first restored image, and calculates the mean square error between the first restored image and the standard image to adjust the pooling window size; The deep learning detail restoration module adopts a generative adversarial network module, including a generator network and a discriminator network; the first restored image is input into the generator network, the generator network adopts a convolutional neural network, generates a high-resolution image through classification information and a low-resolution feature map, and outputs a generated image; Use the discriminator network to evaluate the quality of the generated image, introduce perceptual loss to calculate the discriminant image quality, and output the second restored image to generate the image; The image restoration unit adopts the ray projection algorithm of the three-dimensional reconstruction module, generates rays along the shooting direction, samples the second restored image, accumulates color and opacity values ​​according to the imaging rules, and processes the second restored image through the three-dimensional reconstruction module as the final output completed processed image.

Images