Image processing method, image processing device and storage medium

Abstract

An image processing method, an image processing device and a computer-readable storage medium. The image processing method includes: obtaining an input image; performing N reconstruction operations on the input image using an image reconstruction network to obtain a reconstructed image corresponding to the input image. The image reconstruction network includes an encoding subnetwork, a decoding subnetwork and a connection subnetwork, the encoding subnetwork includes at least one encoding block, each encoding block includes multiple self-attention operators, the decoding subnetwork includes at least one decoding block, each decoding block includes multiple self-attention operators, and the connection subnetwork includes a connection layer, and a coding block is connected to a decoding block through a connection layer mapping. Each reconstruction operation includes: performing feature extraction processing on the input of the encoding subnetwork through the encoding subnetwork to obtain a first feature map and the output of at least one encoding block; using the decoding subnetwork and the connection subnetwork to process the first feature map and the output of at least one encoding block to obtain the output of the decoding subnetwork.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to an image processing method, an image processing device, and a computer-readable storage medium. Background Art

[0002] Image reconstruction has very important applications in video, remote sensing, medicine, signal processing and other fields. For example, without changing the satellite image detection system, collecting high-definition meteorological remote sensing images is more conducive to observation; magnetic resonance imaging can provide better soft tissue contrast and safety than computer tomography (for example, to prevent patients from being exposed to nuclear radiation, etc.), however, the slow speed of magnetic resonance imaging will reduce patient comfort and lead to high costs; in monitoring systems such as banks and transportation, high-resolution images are often required to extract important clues. Summary of the invention

[0003] At least one embodiment of the present disclosure provides an image processing method, comprising: obtaining an input image; performing N reconstruction operations on the input image using an image reconstruction network to obtain a reconstructed image corresponding to the input image, wherein N is a positive integer greater than or equal to 1, and the image reconstruction network comprises an encoding subnetwork, a decoding subnetwork, and a connection subnetwork, the encoding subnetwork comprises at least one encoding block, each encoding block comprises a plurality of self-attention operators, the decoding subnetwork comprises at least one decoding block, each decoding block comprises a plurality of self-attention operators, the connection subnetwork comprises a connection layer, and an encoding block in the at least one encoding block is connected to a decoding block in the at least one decoding block through the connection layer mapping, wherein each reconstruction operation comprises: performing feature extraction processing on the input of the encoding subnetwork through the encoding subnetwork to obtain a first feature map and an output of the at least one encoding block; and processing the first feature map and the output of the at least one encoding block using the decoding subnetwork and the connection subnetwork to obtain the output of the decoding subnetwork.

[0004] For example, in the image processing method provided by at least one embodiment of the present disclosure, the image reconstruction network also includes a bottleneck layer, wherein, when the image reconstruction network includes the bottleneck layer, each reconstruction operation includes: performing feature extraction processing on the input of the encoding subnetwork through the encoding subnetwork to obtain the first feature map and the output of the at least one encoding block; using the bottleneck layer to process the first feature map to obtain a second feature map; using the decoding subnetwork and the connection subnetwork to process the second feature map and the output of the at least one encoding block to obtain the output of the decoding subnetwork.

[0005] For example, in the image processing method provided by at least one embodiment of the present disclosure, the bottleneck layer includes multiple self-attention operators, and the first feature map is processed using the bottleneck layer to obtain the second feature map, including: using the multiple self-attention operators of the bottleneck layer to perform representation learning on the first feature map to obtain the second feature map.

[0006] For example, in the image processing method provided by at least one embodiment of the present disclosure, the multiple self-attention operators of the bottleneck layer include window-based self-attention operators and moving window-based self-attention operators that are connected alternately in sequence, and the number of window-based self-attention operators in the bottleneck layer is 1 more than the number of moving window-based self-attention operators.

[0007] For example, in the image processing method provided by at least one embodiment of the present disclosure, when N is 1, for the N reconstruction operations, the input of the encoding subnetwork is the input image, and the output of the decoding subnetwork is the reconstructed image; when N is greater than 1, data fitting processing and data correction processing are performed on the output of the decoding subnetwork in the i-th reconstruction operation among the N reconstruction operations to obtain the input of the encoding subnetwork in the i+1-th reconstruction operation among the N reconstruction operations, i is a positive integer and less than N, when i is 1, the input of the encoding subnetwork in the i-th reconstruction operation is the input image, and when i is N-1, the output of the decoding subnetwork in the i+1-th reconstruction operation is the reconstructed image.

[0008] For example, in the image processing method provided by at least one embodiment of the present disclosure, the connection subnetwork includes at least one connection layer, and the at least one encoding block is connected to the at least one decoding block in a one-to-one mapping manner through the at least one connection layer, and each of the connection layers is used to map and connect the encoding blocks and decoding blocks having the same size features.

[0009] For example, in the image processing method provided by at least one embodiment of the present disclosure, the decoding subnetwork also includes a linear projection layer, the at least one encoding block includes a plurality of encoding blocks connected in sequence, the at least one decoding block includes a plurality of decoding blocks connected in sequence, the connection subnetwork includes a plurality of connection layers, the plurality of encoding blocks are connected to the plurality of decoding blocks in a one-to-one mapping through the plurality of connection layers, each of the connection layers is used to connect the output of the encoding block corresponding to the connection layer with the output of the decoding block corresponding to the connection layer, the output of the connection layer is used as the input of the next decoding block connected to the decoding block corresponding to the connection layer, or the output of the connection layer is used as the input of the linear projection layer.

[0010] For example, in the image processing method provided by at least one embodiment of the present disclosure, the encoding subnetwork also includes an output encoding block, and the at least one encoding block includes an input encoding block. The encoding subnetwork performs feature extraction processing on the input of the encoding subnetwork to obtain a first feature map and the output of the at least one encoding block, including: processing the input of the encoding subnetwork through the input encoding block to obtain the output of the input encoding block; and processing the output of the input encoding block using the output encoding block to obtain the first feature map.

[0011] For example, in the image processing method provided by at least one embodiment of the present disclosure, the output coding block includes multiple self-attention operators and patch merging layers, and the output coding block is used to process the output of the input coding block to obtain the first feature map, including: performing a downsampling operation and a first channel number adjustment operation on the output of the input coding block through the patch merging layer in the output coding block to obtain the first encoding feature map; performing representation learning on the first encoding feature map through multiple self-attention operators in the output coding block to obtain the first feature map.

[0012] For example, in the image processing method provided by at least one embodiment of the present disclosure, the patch merging layer includes a downsampling layer and a first linear layer, the downsampling layer is used to perform the downsampling operation, and the first linear layer is used to perform the first channel number adjustment operation; the downsampling operation and the first channel number adjustment operation are performed on the output of the input coding block through the patch merging layer in the output coding block to obtain a first coding feature map, including: performing the downsampling operation on the output of the input coding block through the downsampling layer to obtain the output of the downsampling layer; performing the first channel number adjustment operation on the output of the downsampling layer through the first linear layer to obtain the first coding feature map.

[0013] For example, in the image processing method provided by at least one embodiment of the present disclosure, the dimension of the first encoding feature map is the same as the dimension of the first feature map, and the size of the first encoding feature map is the same as the size of the first feature map.

[0014] For example, in the image processing method provided by at least one embodiment of the present disclosure, the at least one coding block includes an input coding block and at least one intermediate coding block connected in sequence, each intermediate coding block also includes a patch merging layer, and each intermediate coding block is used to use the patch merging layer of the intermediate coding block to perform a downsampling operation and a first channel number adjustment operation on the input of the intermediate coding block to obtain the output of the patch merging layer of the intermediate coding block; and use multiple self-attention operators in the intermediate coding block to perform representation learning on the output of the patch merging layer of the intermediate coding block to obtain the output of the intermediate coding block.

[0015] For example, in the image processing method provided by at least one embodiment of the present disclosure, the encoding subnetwork also includes an output encoding block, the at least one intermediate encoding block includes a first intermediate encoding block and a second intermediate encoding block, and the encoding subnetwork is used to perform feature extraction processing on the input of the encoding subnetwork to obtain a first feature map and the output of the at least one encoding block, including: processing the input of the encoding subnetwork through the input encoding block to obtain the output of the input encoding block; processing the output of the input encoding block using the first intermediate encoding block to obtain the output of the first intermediate encoding block; processing the output of the first intermediate encoding block using the second intermediate encoding block to obtain the output of the second intermediate encoding block; processing the output of the second intermediate encoding block using the output encoding block to obtain the first feature map.

[0016] For example, in the image processing method provided by at least one embodiment of the present disclosure, the input coding block also includes a linear embedding layer, and the input of the coding sub-network is processed by the input coding block to obtain the output of the input coding block, including: processing the input of the coding sub-network by the linear embedding layer in the input coding block to obtain a second coding feature map; and performing representation learning on the second coding feature map by multiple self-attention operators in the input coding block to obtain the output of the input coding block.

[0017] For example, in the image processing method provided by at least one embodiment of the present disclosure, the dimension of the second encoding feature map is the same as the dimension of the output of the input encoding block, and the size of the second encoding feature map is the same as the size of the output of the input encoding block.

[0018] For example, in the image processing method provided by at least one embodiment of the present disclosure, each decoding block also includes a patch expansion layer, and each decoding block is used to use the patch expansion layer of the decoding block to perform an upsampling operation and a second channel number adjustment operation on the input of the decoding block to obtain the output of the patch merging layer of the decoding block; and use multiple self-attention operators in the decoding block to perform representation learning on the output of the patch merging layer of the decoding block to obtain the output of the decoding block.

[0019] For example, in the image processing method provided by at least one embodiment of the present disclosure, the patch expansion layer includes an upsampling layer and a second linear layer, the upsampling layer is used to perform the upsampling operation, and the second linear layer is used to perform the second channel number adjustment operation.

[0020] For example, in the image processing method provided by at least one embodiment of the present disclosure, the decoding subnetwork also includes a linear projection layer, the at least one decoding block includes an input decoding block, the at least one connection layer includes a first connection layer, and the second feature map and the output of the at least one encoding block are processed by the decoding subnetwork and the connection subnetwork to obtain the output of the decoding subnetwork, including: processing the second feature map through the input decoding block to obtain the output of the input decoding block; connecting the output of the encoding block corresponding to the first connection layer and the output of the input decoding block through the first connection layer to obtain the output of the first connection layer, and performing a projection mapping operation on the output of the first connection layer through the linear projection layer to obtain the output of the decoding subnetwork.

[0021] For example, in the image processing method provided by at least one embodiment of the present disclosure, the decoding subnetwork also includes a linear projection layer, the at least one decoding block includes an input decoding block and at least one intermediate decoding block connected in sequence, the at least one intermediate decoding block includes a first intermediate decoding block and a second intermediate decoding block connected in sequence, the at least one connection layer includes a first connection layer, a second connection layer and a third connection layer, and the decoding subnetwork and the connection subnetwork are used to process the second feature map and the output of the at least one encoding block to obtain the output of the decoding subnetwork, including: processing the second feature map through the input decoding block to obtain the output of the input decoding block; processing the output of the encoding block corresponding to the first connection layer and the output of the input decoding block through the first connection layer The output of the first connection layer is processed by row connection to obtain the output of the first connection layer; the output of the first connection layer is processed by the first intermediate decoding block to obtain the output of the first intermediate decoding block; the output of the coding block corresponding to the second connection layer and the output of the first intermediate decoding block are connected through the second connection layer to obtain the output of the second connection layer; the output of the second connection layer is processed by the second intermediate decoding block to obtain the output of the second intermediate decoding block; the output of the coding block corresponding to the third connection layer and the output of the second intermediate decoding block are connected through the third connection layer to obtain the output of the third connection layer; the output of the third connection layer is projected through the linear projection layer to obtain the output of the decoding subnetwork.

[0022] For example, in the image processing method provided by at least one embodiment of the present disclosure, the multiple self-attention operators in each of the at least one encoding block and the at least one decoding block include at least one symmetric window self-attention operator block, each symmetric window self-attention operator block includes a window-based self-attention operator and a moving window-based self-attention operator, and the output of the window-based self-attention operator serves as the input of the moving window-based self-attention operator.

[0023] For example, in the image processing method provided by at least one embodiment of the present disclosure, obtaining an input image includes: determining sampling data, wherein the sampling data is data obtained by undersampling; initializing the sampling data to convert the sampling data from the frequency domain to the corresponding spatial domain to obtain an intermediate input image; and processing the intermediate input image to obtain the input image.

[0024] For example, in the image processing method provided by at least one embodiment of the present disclosure, the intermediate input image is processed to obtain the input image, including: performing data fitting processing and data correction processing on the intermediate input image to obtain the input image.

[0025] For example, in the image processing method provided by at least one embodiment of the present disclosure, the size of the input image is the same as the size of the reconstructed image, and the number of channels of the input image is the same as the number of channels of the reconstructed image.

[0026] For example, in the image processing method provided by at least one embodiment of the present disclosure, before obtaining the input image, the image processing method also includes: in a training stage, training the image reconstruction network using a sample image and a sample reconstructed image corresponding to the sample image, wherein in the training stage, the loss function corresponding to the image reconstruction network includes a mean absolute error loss function and a structural similarity loss function.

[0027] For example, in the image processing method provided by at least one embodiment of the present disclosure, the loss function corresponding to the image reconstruction network is expressed as:

[0028] L Mix =λ*L SSIM +(1-λ)*L MAE

[0029] Among them, L Mix represents the loss function corresponding to the image reconstruction network, L SSIM represents the structural similarity loss function, L MAE represents the mean absolute error loss function, and λ is a constant.

[0030] At least one embodiment of the present disclosure further provides an image processing device, comprising a memory and a processor. The memory stores computer executable instructions suitable for being executed by the processor, and when the computer executable instructions are executed by the processor, one or more steps of the image processing method according to any embodiment of the present disclosure are executed.

[0031] At least one embodiment of the present disclosure further provides a computer-readable storage medium that non-transitorily stores computer-executable instructions, which, when executed by a computer, can execute one or more steps of the image processing method according to any embodiment of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings of the embodiments will be briefly introduced below. Obviously, the drawings in the following description only relate to some embodiments of the present disclosure, but are not intended to limit the present disclosure.

[0033] Figure 1 A schematic flow chart of an image processing method provided for at least one embodiment of the present disclosure;

[0034] Figure 2 A schematic diagram of an undersampling process provided for at least one embodiment of the present disclosure;

[0035] Figure 3 A schematic diagram of the overall process of an image processing method provided by at least one embodiment of the present disclosure;

[0036] Figure 4A A schematic diagram of a structure of multiple self-attention operators in an encoding block or a decoding block provided by at least one embodiment of the present disclosure;

[0037] Figure 4B A schematic diagram of a structure of multiple self-attention operators in a bottleneck layer provided by at least one embodiment of the present disclosure;

[0038] Figure 5 A schematic structural diagram of an image reconstruction network provided for at least one embodiment of the present disclosure;

[0039] Figure 6 A schematic structural diagram of another image reconstruction network provided for at least one embodiment of the present disclosure;

[0040] Figure 7 An operation diagram of a patch merging layer provided for at least one embodiment of the present disclosure;

[0041] Figure 8 A schematic diagram of the structure of another image reconstruction network provided by at least one embodiment of the present disclosure;

[0042] Fig. 9 A schematic block diagram of an image processing device provided for at least one embodiment of the present disclosure;

[0043] Fig.10 A schematic diagram of a computer-readable storage medium provided for at least one embodiment of the present disclosure. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution of the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings of the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the described embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0045] Unless otherwise defined, the technical terms or scientific terms used in the present disclosure should be understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0046] In order to keep the following description of the embodiments of the present disclosure clear and concise, the present disclosure omits detailed descriptions of some known functions and known components.

[0047] The model of magnetic resonance imaging (MRI) is shown in the following formula (1):

[0048] f=Am+ε (1)

[0049] In formula (1), A:C N →C M is the discrete Fourier transform with undersampling (i.e., forward imaging model), f∈C M is the undersampled k-space (k-space is the dual space of the ordinary space under Fourier transform) observation data, m∈C Nis the magnetic resonance image to be reconstructed, ε is the additive noise, and ε is the system noise (Gaussian noise) caused by the imaging process. Inverting high-quality reconstructed images from undersampled observation data is essentially solving an underdetermined inversion problem, so the prior information about the magnetic resonance image should be appropriately selected as a solution. However, the traditional methods based on regularization and compressed sensing using prior information are not suitable for medical images because the details of the images generated by this method are blurred, but the detailed features in the magnetic resonance image are more important than the overall features. At present, the inversion method based on deep learning performs very well in solving underdetermined problems and can overcome the limitations of existing mathematical model-based methods. This is because the training data and prior knowledge can be pushed and accumulated as the training time grows. Most of the deep learning methods based on iterative expansion are derived from mathematical model algorithms with convergence or asymptotic convergence guarantees (for example, proximal gradient operators, gradient descent, etc.) to provide more information about the relationship between network topology and performance. The reconstruction method based on expansion has gradually evolved from learning only regularization parameters to learning learnable operators and functions in the compressed sensing reconstruction formula. Deep learning methods based on non-iterative unfolding initially rely on standard networks to learn feature mappings between undersampled k-space data or zero-filled images and reconstructed images.

[0050] At present, most of the image reconstruction methods based on deep learning rely on convolutional neural networks (CNN). The iterative expansion deep learning method based on CNN is limited by the inherent locality in capturing long-range dependencies, and is prone to smoothing the details in the image, with poor ability to retain details; the non-iterative expansion deep learning method based on CNN uses an end-to-end network to learn the feature mapping between low-resolution images and reconstructed images, but this type of network is like a black box and has the disadvantage of low interpretability.

[0051] The present disclosure relates to the technical field of image processing (i.e., image reconstruction) and image inversion based on deep learning. At least one embodiment of the present disclosure provides an image processing method, which includes: obtaining an input image; performing N reconstruction operations on the input image using an image reconstruction network to obtain a reconstructed image corresponding to the input image. N is a positive integer greater than or equal to 1. The image reconstruction network includes an encoding subnetwork, a decoding subnetwork, and a connection subnetwork. The encoding subnetwork includes at least one encoding block, each encoding block includes multiple self-attention operators, the decoding subnetwork includes at least one decoding block, each decoding block includes multiple self-attention operators, and the connection subnetwork includes a connection layer. One encoding block in at least one encoding block is connected to one decoding block in at least one decoding block through a connection layer mapping. Each reconstruction operation includes: performing feature extraction processing on the input of the encoding subnetwork through the encoding subnetwork to obtain a first feature map and an output of at least one encoding block; using the decoding subnetwork and the connection subnetwork to process the first feature map and the output of at least one encoding block to obtain the output of the decoding subnetwork.

[0052] The image processing method provided by the embodiments of the present disclosure is an image reconstruction method based on a learnable iterative attention network, which can invert a high-quality reconstructed image from undersampled observation data. Based on the image processing method provided by the embodiments of the present disclosure, the quality and clarity of the reconstructed image can be improved, so that the detailed features in the reconstructed image can be retained, and a higher quality reconstructed image can be obtained while reducing the network training time. It can also accelerate the generation of the reconstructed image, improve the processing speed, and reduce the operation time, and has better and wider application prospects.

[0053] At least one embodiment of the present disclosure also provides an image processing apparatus and a computer-readable storage medium.

[0054] The image processing method provided in the embodiment of the present disclosure may be applied to the image processing device provided in the embodiment of the present disclosure, and the image processing device may be configured on an electronic device. For example, in some embodiments, the electronic device may be a medical type electronic device.

[0055] The embodiments of the present disclosure are described in detail below, but the present disclosure is not limited to these specific embodiments.

[0056] Figure 1 A schematic flowchart of an image processing method provided for at least one embodiment of the present disclosure.

[0057] For example, in the image processing method provided in the embodiment of the present disclosure, the image reconstruction network may be a neural network, and the image reconstruction network may include an encoding subnetwork, a decoding subnetwork, and a connection subnetwork. The encoding subnetwork is used to extract a feature map, and the decoding subnetwork is used to reconstruct and generate a reconstructed image (e.g., an MRI image, etc.).

[0058] For example, the connection subnetwork is used to connect the encoding subnetwork and the decoding subnetwork, that is, to connect shallow features and deep features for compensation, thereby reducing the loss of spatial information caused by compression and downsampling. Here, the so-called connection, for example, can be a skip connection (i.e., a merged concatenate operation), that is, the feature maps of the same size in the layers connecting the encoding subnetwork and the decoding subnetwork are connected by memory mapping (so that the vectors corresponding to the features are merged and the number of channels in the layer where the features are located is doubled). The connection subnetwork includes a connection layer, which is used to fuse the multi-scale features of the downsampled features from the encoding subnetwork and the upsampled features from the decoding subnetwork.

[0059] For example, the encoding subnetwork includes at least one encoding block, each encoding block includes multiple self-attention operators, and the decoding subnetwork includes at least one decoding block, each decoding block includes multiple self-attention operators. One encoding block in at least one encoding block is connected to one decoding block in at least one decoding block through a connection layer mapping. For example, in some embodiments, the connection subnetwork includes at least one connection layer, and at least one encoding block is connected to at least one decoding block through at least one connection layer. In other words, each encoding block is mapped and connected to the corresponding decoding block through a connection layer. The size of the output of the encoding block connected to the connection layer is the same as the size of the output of the decoding block connected to the connection layer, and the dimension of the output of the encoding block connected to the connection layer is also the same as the dimension of the output of the decoding block connected to the connection layer. In other words, each connection layer is used to map and connect the encoding block and the decoding block of the feature map with the same size and dimension.

[0060] like Figure 1 As shown, the image processing method provided by the embodiment of the present disclosure may include, but is not limited to, the following steps:

[0061] Step S10: obtaining an input image;

[0062] Step S20: using the image reconstruction network to perform N reconstruction operations on the input image to obtain a reconstructed image corresponding to the input image.

[0063] For example, in step 10, the input image may be a grayscale image or a color image.

[0064] For example, the size of the input image may be 256 (pixels)*256 (pixels). However, the size of the input image may also be 512*512, etc., and the embodiments of the present disclosure do not limit this. In the following description of the present disclosure, the size of the input image is 256*256 as an example for detailed description.

[0065] For example, in some embodiments, step S10 may include: determining sampling data; initializing the sampling data to convert the sampling data from the frequency domain to the corresponding spatial domain to obtain an intermediate input image; and processing the intermediate input image to obtain an input image.

[0066] For example, the sampled data is data obtained by undersampling. In the embodiments of the present disclosure, undersampling data can be directly used, thereby reducing the time of the sampling process and improving the speed of data acquisition. When the sampled data is magnetic resonance data, the speed of magnetic resonance imaging can be accelerated, the patient's comfort can be improved, and the cost can be reduced.

[0067] For example, in step S10, the sampled data may be various types of data, and thus the input image may be various types of images. For example, the sampled data may be medical data, and in this case, the input image is a medical image. According to the classification of the equipment for acquiring medical data, the medical data may include ultrasound data, X-ray computed tomography (CT) data, magnetic resonance data (i.e., nuclear magnetic resonance data), digital subtraction angiography (DSA) data, and positron emission computed tomography (PET) data, etc. According to the classification of the content of the medical data, the medical data may include brain tissue nuclear magnetic resonance data, spinal cord nuclear magnetic resonance data, fundus data, blood vessel data, pancreatic CT data, and lung CT data, etc. For example, the sampled data may also be monitoring data, and in this case, the input image is a monitoring image, such as a monitoring image of a road, a monitoring image of a bank, etc.; for example, the sampled data may also be meteorological remote sensing data, and in this case, the input image may also be a meteorological remote sensing image, etc. It should be noted that the input image may also be a person image, an animal or plant image, or a landscape image, etc. The embodiments of the present disclosure do not limit the specific type of the input image.

[0068] For example, the sampled data may be acquired by a collection device. When the sampled data is medical data, the collection device may include, for example, ultrasound equipment, X-ray equipment, nuclear magnetic resonance equipment, nuclear medicine equipment, medical optical equipment, and thermal imaging equipment, etc., which are not limited in the present disclosure.

[0069] For example, the sampled data may be the original sampled data directly collected by the acquisition device, or may be the data obtained after preprocessing the original sampled data. For example, in order to avoid the influence of the data quality, data imbalance, etc. of the original sampled data on the image reconstruction accuracy, in step S10, the image processing method provided by the embodiment of the present disclosure may also include preprocessing the original sampled data to obtain the sampled data. The preprocessing may eliminate irrelevant information or noise information in the original sampled data to obtain the sampled data, so as to facilitate better reconstruction of the input image. The preprocessing may include, for example, augmenting (Data Augment) and scaling the original sampled data. Augmentation includes augmenting the data of the original sampled data by random cropping, rotation, flipping, skewing, adjusting brightness, contrast, etc. Scaling includes scaling the original sampled data in equal proportion and cutting it to a preset size (e.g., 256*256) to facilitate subsequent image reconstruction operations. It should be noted that the preprocessing may also include gamma correction, enhancement or noise reduction filtering, and other processing.

[0070] For example, in some embodiments, in step S10, determining the sampling data includes: obtaining measurement data, wherein the measurement data is full sampling data; undersampling the measurement data to obtain the sampling data. For example, full sampling may be performed to obtain the measurement data, and then the full sampling measurement data may be undersampled to obtain the sampling data, and at this time, the reconstructed image obtained based on the sampling data may be compared with the image determined based on the measurement data, so as to determine the quality of the reconstructed image. For example, the measurement data may be data that has been sampled in advance.

[0071] For example, the sampled data may be frequency domain data, and thus the above-mentioned undersampling means frequency domain undersampling. Figure 2 Figure 2 shows a schematic diagram of the undersampling process. Figure 2As shown, the measurement data 101 is K-space data, and the measurement data 101 is data obtained by full sampling. For example, the size of the measurement data 101 is 256*256. Then, the random mask is applied to the measurement data 101 to perform frequency domain undersampling on the measurement data 101 to obtain the sampled data 102, and the size of the sampled data 102 is also 256*256. The random mask can be in the form of random X*zero filling. The random X*zero filling means that the columns are undersampled, and the remaining part after undersampling is zero-filled (the pixel value is 0). If X is 4, 25% of the columns in the measurement data 101 are randomly sampled, and the remaining 75% of the columns in the measurement data 101 (that is, the columns other than the 25% of the columns obtained by sampling) are zero-filled (the pixel value is 0, black). Since the embodiment of the present disclosure only needs to process the sampled data, undersampling can be performed, that is, only sampling is performed to obtain the sampled data. If X is 4, the sampling speed can be accelerated by 4 times. It should be noted that the specific value of X can be set according to actual conditions, and X can also be 2, 6, 8, etc.

[0072] For example, in some embodiments, in step S10, the initialization process may include an inverse Fourier transform, thereby converting the sampled data from the frequency domain to the corresponding spatial domain to obtain an intermediate input image. Figure 2 As shown, the sampled data 102 (frequency domain) is inverse Fourier transformed to obtain an intermediate input image 103 (spatial domain).

[0073] Figure 3 A schematic diagram of the overall flow of an image processing method provided for some embodiments of the present disclosure.

[0074] Figure 3 Four sampling data are shown, namely, sampling data SD1, sampling data SD2, sampling data SD3 and sampling data SD4. The sampling data SD1 is initialized (i.e., inverse Fourier transform A * ) to obtain the intermediate input image MP1, initialize the sampling data SD2 to obtain the intermediate input image MP2, initialize the sampling data SD3 to obtain the intermediate input image MP3, and initialize the sampling data SD4 to obtain the intermediate input image MP4.

[0075] For example, in some embodiments, in step S10, the intermediate input image is processed to obtain the input image, including: performing data fitting processing and data correction processing on the intermediate input image to obtain the input image.

[0076] For example, in some embodiments, in step S20, the size of the input image is the same as the size of the reconstructed image, and the number of channels of the input image is the same as the number of channels of the reconstructed image.

[0077] For example, in step S20, each reconstruction operation includes: performing feature extraction processing on the input of the encoding subnetwork through the encoding subnetwork to obtain a first feature map and an output of at least one encoding block; using the decoding subnetwork and the connection subnetwork to process the first feature map and the output of at least one encoding block to obtain the output of the decoding subnetwork.

[0078] For example, in step S20, N is a positive integer greater than or equal to 1. For example, when N is 1, it means that only one reconstruction operation is performed on the input image to obtain a reconstructed image. At this time, for N reconstruction operations (i.e., the first reconstruction operation), the input of the encoding subnetwork is the input image, and the output of the decoding subnetwork is the reconstructed image. For example, when N is greater than 1, it means that multiple reconstruction operations are iteratively performed on the input image to obtain a reconstructed image. At this time, data fitting processing and data correction processing are performed on the output of the decoding subnetwork in the i-th reconstruction operation in the N reconstruction operations to obtain the input of the encoding subnetwork in the i+1-th reconstruction operation in the N reconstruction operations, i is a positive integer and less than N, when i is 1, the input of the encoding subnetwork in the i-th reconstruction operation (i.e., the first reconstruction operation) is the input image, and when i is N-1, the output of the decoding subnetwork in the i+1-th reconstruction operation (i.e., the N-th reconstruction operation) is the reconstructed image.

[0079] For example, Figure 3 As shown, when N is greater than 1, in the first reconstruction operation, any intermediate input image among the intermediate input images MP1 to MP4 is used as the current intermediate input image m0, and the current intermediate input image m0 is subjected to data fitting processing (I-ηA * A) and data correction processing (ηA * f), then the result of the data fitting process and the result of the data correction process are summed, that is, the result of the data fitting process and the result of the data correction process are added to obtain an input image, which can be the input of the image reconstruction network in the first reconstruction operation; then, the input image is reconstructed using the image reconstruction network to obtain a reconstructed image m1, which is the output of the image reconstruction network in the first reconstruction operation. When multiple iterative processes are required, a second reconstruction operation is performed on the reconstructed image m1. In the second reconstruction operation, first, the reconstructed image m1 is subjected to data fitting processing (I-ηA * A) and data correction processing (ηA *f), to obtain the fitting processing result of the reconstructed image m1 and the correction processing result of the reconstructed image m1, sum the fitting processing result of the reconstructed image m1 and the correction processing result of the reconstructed image m1 to obtain the input of the image reconstruction network in the second reconstruction operation, and then use the image reconstruction network to perform a reconstruction operation on the input of the image reconstruction network (that is, the result obtained by summing the fitting processing result of the reconstructed image m1 and the correction processing result of the reconstructed image m1) to obtain the reconstructed image corresponding to the second reconstruction operation. And so on, it iterates N times, and takes the reconstructed image mN corresponding to the Nth reconstruction operation as the final reconstruction result. For example, Figure 3 As shown, N iterative reconstruction operations are performed on the intermediate input image MP1 to obtain a reconstructed image RP1 corresponding to the intermediate input image MP1; N iterative reconstruction operations are performed on the intermediate input image MP2 to obtain a reconstructed image RP2 corresponding to the intermediate input image MP2; N iterative reconstruction operations are performed on the intermediate input image MP3 to obtain a reconstructed image RP3 corresponding to the intermediate input image MP3; N iterative reconstruction operations are performed on the intermediate input image MP4 to obtain a reconstructed image RP4 corresponding to the intermediate input image MP4.

[0080] For example, in some embodiments, in an image reconstruction network, the multiple self-attention operators in each encoding block (including the output encoding block, described later) include at least one symmetric window self-attention operator block, each symmetric window self-attention operator block includes a window-based self-attention operator and a moving window-based self-attention operator, and the output of the window-based self-attention operator serves as the input of the moving window-based self-attention operator. Similarly, the multiple self-attention operators in each decoding block include at least one symmetric window self-attention operator block, each symmetric window self-attention operator block includes a window-based self-attention operator and a moving window-based self-attention operator, and the output of the window-based self-attention operator serves as the input of the moving window-based self-attention operator. It should be noted that in the description of the present disclosure, it is taken that the multiple self-attention operators in each encoding block include a symmetric window self-attention operator block, and the multiple self-attention operators in each decoding block include a symmetric window self-attention operator block, but the present disclosure is not limited to this.

[0081] For example, the parameters of the window-based self-attention operators in different encoding blocks may be the same or different, and the parameters of the moving window-based self-attention operators in different encoding blocks may be the same or different; the parameters of the window-based self-attention operators in different decoding blocks may be the same or different, and the parameters of the moving window-based self-attention operators in different decoding blocks may be the same or different.

[0082] Figure 4AA schematic diagram of a structure of multiple self-attention operators in a coding block or a decoding block provided in some embodiments of the present disclosure. Taking multiple self-attention operators in a coding block as an example.

[0083] For example, in some embodiments, the multiple self-attention operators in each encoding block are self-attention operator blocks of symmetric windows (W-MSA and SW-MSA appear in pairs), such as Figure 4A As shown, the multiple self-attention operators in the coding block include a window-based self-attention operator W0 and a moving window-based self-attention operator SW0, and the output of the window-based self-attention operator W0 is used as the input of the moving window-based self-attention operator SW0. The window-based self-attention operator W0 includes a multilayer perceptron machine MLP (Multilayer Perceptron), two layer normalization layers LN (Layernorm), and a SWIN non-shifted window self-attention operator W-MSA (window based Multi-head self-attention). The moving window-based self-attention operator SW0 includes a multilayer perceptron machine MLP, two layer normalization layers LN, and a SWIN shifted window self-attention operator SW-MSA (shifted window based Multi-head self-attention).

[0084] It should be noted that the structure of the multiple self-attention operators in the decoding block is the same as the structure of the multiple self-attention operators in the encoding block, and the repeated parts will not be repeated.

[0085] Figure 4B A schematic diagram of the structure of multiple self-attention operators in the bottleneck layer provided for some embodiments of the present disclosure.

[0086] For example, in some embodiments, the image reconstruction network further includes a bottleneck layer, which is used to reduce the depth of the network and minimize the number of model parameters. The bottleneck layer can be used to extract features at a high level of semantics to facilitate image reconstruction in a subsequent decoding subnetwork. In the bottleneck layer, the feature dimension (i.e., the number of channels) and the resolution (i.e., the image size) remain unchanged.

[0087] In the case where the image reconstruction network includes a bottleneck layer, each reconstruction operation includes: performing feature extraction processing on the input of the encoding subnetwork through the encoding subnetwork to obtain a first feature map and an output of at least one encoding block; processing the first feature map using the bottleneck layer to obtain a second feature map; and processing the second feature map and the output of at least one encoding block using the decoding subnetwork and the connection subnetwork to obtain the output of the decoding subnetwork. It should be noted that in the description of the present disclosure, unless otherwise specified, the case where the image reconstruction network includes a bottleneck layer is used as an example for description.

[0088] For example, in some embodiments, the bottleneck layer includes multiple self-attention operators. The parameters of the multiple self-attention operators in the bottleneck layer may be the same or different.

[0089] For example, in some embodiments, the multiple self-attention operators in the bottleneck layer are self-attention operator blocks of asymmetric windows (W-MSA and SW-MSA do not appear in pairs), for example, the multiple self-attention operators in the bottleneck layer include window-based self-attention operators and moving window-based self-attention operators that are connected alternately in sequence, the number of window-based self-attention operators in the bottleneck layer is one more than the number of moving window-based self-attention operators, and the multiple self-attention operators in the bottleneck layer include multiple window-based self-attention operators and at least one moving window-based self-attention operator that are connected alternately in sequence. In the bottleneck layer, except for the last window-based self-attention operator, the output of each window-based self-attention operator serves as the input of a moving window-based self-attention operator, and the output of each moving window-based self-attention operator serves as the input of a window-based self-attention operator. The output of the last window-based self-attention operator serves as the output of the bottleneck layer.

[0090] It should be noted that, in an embodiment of the present disclosure, the asymmetric window self-attention operator block may include a window-based self-attention operator and at least one symmetric window self-attention operator block, so that in the asymmetric window self-attention operator block, the window-based self-attention operator and the moving window-based self-attention operator do not appear in pairs, and the number of window-based self-attention operators is 1 more than the number of moving window-based self-attention operators.

[0091] For example, Figure 4B As shown, in some embodiments, the multiple self-attention operators in the bottleneck layer include two window-based self-attention operators (i.e., window-based self-attention operator W1 and window-based self-attention operator W2) and one moving window-based self-attention operator SW1, the first feature map is used as the input of the window-based self-attention operator W1, the output of the window-based self-attention operator W1 is used as the input of the moving window-based self-attention operator SW1, the output of the moving window-based self-attention operator SW1 is used as the input of the window-based self-attention operator W2, and the output of the window-based self-attention operator W2 is used as the second feature map.

[0092] It should be noted that the bottleneck layer of the present invention is not limited to this. In other embodiments, the multiple self-attention operators in the bottleneck layer include three window-based self-attention operators and two moving window-based self-attention operators. The first feature map is used as the input of the first window-based self-attention operator among the three window-based self-attention operators, the output of the first window-based self-attention operator is used as the input of the first moving window-based self-attention operator among the two moving window-based self-attention operators, the output of the first moving window-based self-attention operator is used as the input of the second window-based self-attention operator among the three window-based self-attention operators, the output of the second window-based self-attention operator is used as the input of the second moving window-based self-attention operator among the two moving window-based self-attention operators, the output of the second moving window-based self-attention operator is used as the input of the third window-based self-attention operator among the three window-based self-attention operators, and the output of the third window-based self-attention operator is used as the second feature map.

[0093] like Figure 4B As shown, the window-based self-attention operator W1 and the window-based self-attention operator W2 have the same structure. Taking the window-based self-attention operator W1 as an example, the window-based self-attention operator W1 includes a multi-layer perceptron MLP, two-layer normalization layers LN, and a SWIN non-shifted window self-attention operator W-MSA. The moving window-based self-attention operator SW1 includes a multi-layer perceptron MLP, two-layer normalization layers LN, and a SWIN shifted window self-attention operator SW-MSA.

[0094] It should be noted that in Figure 4A and Figure 4B In the symbol represents a residual connection.

[0095] Figure 5 A schematic structural diagram of an image reconstruction network provided for some embodiments of the present disclosure, Figure 6 A schematic structural diagram of another image reconstruction network provided for some embodiments of the present disclosure.

[0096] For example, Figure 5 and Figure 6As shown, in some embodiments, the size of the input image 100a is H*W, and the size of the reconstructed image 200a is also H*W. For example, the number of channels of the input image 100a and the number of channels of the reconstructed image 200a are the same. For example, when the input image 100a and the reconstructed image 200a are grayscale images, the number of channels of the input image 100a and the number of channels of the reconstructed image 200a are both 1; when the input image 100a and the reconstructed image 200a are color images, the number of channels (i.e., dimensions) of the input image 100a and the number of channels of the reconstructed image 200a are both 3 (i.e., three color channels of R, G, and B).

[0097] For example, in some embodiments, Figure 5 and Figure 6 As shown, the image reconstruction network may include an encoding subnetwork 10a, a decoding subnetwork 20a, and a connection subnetwork 30a. The encoding subnetwork 10a is used to extract feature information of the input image 100a, and the decoding subnetwork 20a is used to perform image reconstruction according to the output of the encoding subnetwork 10a. After N reconstruction operations are performed through the image reconstruction network, the decoding subnetwork 20a outputs the corresponding reconstructed image 200a.

[0098] For example, Figure 5 and Figure 6 As shown, the encoding subnetwork 10a may include at least one encoding block and an output encoding block 102a, and at least one encoding block in the encoding subnetwork 10a includes an input encoding block 101a. The input encoding block 101a includes multiple self-attention operators Ma and a linear embedding layer (linear embedding) Ea, and the output encoding block 102a includes multiple self-attention operators Ma and a patch merging layer Da.

[0099] For example, the linear embedding layer Ea is used to map the input dimension (i.e., the number of channels) of the encoding subnetwork 10a to an arbitrary dimension C11 of the hidden space. In this process, the linear embedding layer Ea expands the input dimension of the encoding subnetwork 10a, and the specific expansion multiple can be set according to actual conditions.

[0100] For example, the patch merging layer Da in the output coding block 102a includes a downsampling layer and a first linear layer, the downsampling layer is used to perform a downsampling operation, and the first linear layer is used to perform a first channel number adjustment operation. For example, the downsampling factor in the downsampling operation can be 2*2, that is, the width and height of the image are reduced by 2 times, that is, the width of the downsampled image becomes 1 / 2 of the width of the original image, and the height of the downsampled image becomes 1 / 2 of the height of the original image.

[0101] Figure 7 A schematic diagram of the operation of a patch merging layer provided for some embodiments of the present disclosure. Figure 7Each rectangular grid in represents a pixel P.

[0102] For example, in some embodiments, Figure 7 As shown, a downsampling operation is performed on an image (B, H, W, C) through a downsampling layer in a patch merging layer, where B is 1, H is 8, W is 8, and C (i.e., the number of channels) is 1, i.e., the size of the image (B, H, W, C) is 8*8, and the number of channels of the image (B, H, W, C) is 1. In this downsampling operation, the downsampling size can be 2, and pixels are selected in the row direction X and the column direction Y of the image (B, H, W, C) with an interval of 2 pixels, so that four downsampling images can be obtained, and the four downsampling images are downsampling image X0, downsampling image X1, downsampling image X2, and downsampling image X3. The number of channels of each downsampling image is 1, and then the four downsampling images X0 to X3 are spliced ​​in the dimension direction, so that the number of channels of the spliced ​​image becomes 4 times that of the image (B, H, W, C), i.e., the number of channels of the spliced ​​image is 4. Finally, the number of channels of the spliced ​​image is adjusted to twice that of the image (B, H, W, C) through the first linear layer (fully connected layer) in the patch merging layer, and the output of the patch merging layer is finally obtained. The number of channels of the output of the patch merging layer is 2, and the size of the output of the patch merging layer is 4*4. At this time, the first channel number adjustment operation means that the number of channels will be reduced.

[0103] It should be noted that the present disclosure does not limit the specific structure of the patch merging layer, as long as it can achieve the corresponding functions. In the patch merging layer, the downsampling operation can also be implemented in a rearrange manner. At this time, first, a linear layer can be used to expand the feature dimension (number of channels C) of the image by 8 times, and then the number of channels is adjusted to twice that of the original image through the rearrange layer, thereby realizing the function of the patch merging layer. At this time, the first channel number adjustment operation indicates that the number of channels will be increased. The downsampling operation can also be implemented by pooling. The embodiments of the present disclosure do not limit the specific implementation methods of the downsampling operation and the first channel number adjustment operation.

[0104] For example, in some embodiments, in each reconstruction operation, the input of the encoding subnetwork is subjected to feature extraction processing by the encoding subnetwork to obtain a first feature map and an output of at least one encoding block, including: processing the input of the encoding subnetwork by the input encoding block to obtain an output of the input encoding block; and processing the output of the input encoding block by the output encoding block to obtain the first feature map. For example, the output of at least one encoding block includes the output of the input encoding block.

[0105] For example, in each reconstruction operation, the output of the input coding block is processed by the output coding block to obtain a first feature map, including: performing a downsampling operation and a first channel number adjustment operation on the output of the input coding block through a patch merging layer in the output coding block to obtain a first encoding feature map; performing representation learning on the first encoding feature map through multiple self-attention operators in the output coding block to obtain a first feature map.

[0106] For example, in some embodiments, a downsampling operation and a first channel number adjustment operation are performed on the output of the input coding block through a patch merging layer in the output coding block to obtain a first coding feature map, including: performing a downsampling operation on the output of the input coding block through a downsampling layer in the patch merging layer in the output coding block to obtain the output of the downsampling layer in the patch merging layer in the output coding block; performing a first channel number adjustment operation on the output of the downsampling layer in the patch merging layer in the output coding block through a first linear layer in the patch merging layer in the output coding block to obtain the first coding feature map.

[0107] For example, Figure 5 and Figure 6 As shown, the patch merging layer Da in the output coding block 102a performs a downsampling operation and a first channel number adjustment operation on the output F13 of the input coding block 101a to obtain a first coding feature map F13, which is the output of the patch merging layer Da. The multiple self-attention operators Ma in the output coding block 102a perform representation learning on the first coding feature map F13 to obtain a first feature map F14, which is the output of the coding sub-network 10a.

[0108] For example, the size of the first coding feature map F13 is the same as the size of the first feature map F14, for example, both are H / 8*W / 8. The dimension of the first coding feature map F13 is the same as the dimension of the first feature map F14, for example, both are C12.

[0109] For example, in each reconstruction operation, the input of the encoding subnetwork is processed by the input encoding block to obtain the output of the input encoding block, including: processing the input of the encoding subnetwork by the linear embedding layer in the input encoding block to obtain a second encoding feature map; performing representation learning on the second encoding feature map by multiple self-attention operators in the input encoding block to obtain the output of the input encoding block.

[0110] For example, Figure 5 and Figure 6As shown, the linear embedding layer Ea in the input coding block 101a processes the input of the coding subnetwork 10a (for example, the input image 100a) (for example, increasing the number of channels of the input image 100a) to obtain a second coding feature map F11, which is the output of the linear embedding layer Ea. The multiple self-attention operators Ma in the input coding block 101a perform representation learning on the second coding feature map F11 to obtain the output F12 of the input coding block 101a, and the output F12 of the input coding block 101a is a feature map.

[0111] For example, the size of the second coding feature map F11 is the same as the size of the output F12 of the input coding block 101a, for example, both are H / 4*W / 4. The dimension of the second coding feature map F11 is the same as the dimension of the output F12 of the input coding block, for example, both are C11.

[0112] For example, the size of the output F12 of the input coding block 101a is larger than the size of the first coding feature map F13. The dimension of the output F12 of the input coding block 101a is smaller than the dimension of the first coding feature map F13, that is, C11 is smaller than C12.

[0113] For example, each decoding block also includes a patch expansion layer, and each decoding block is used to use the patch expansion layer of the decoding block to perform an upsampling operation and a second channel number adjustment operation on the input of the decoding block to obtain the output of the patch merging layer of the decoding block; and use multiple self-attention operators in the decoding block to perform representation learning on the output of the patch merging layer of the decoding block to obtain the output of the decoding block.

[0114] For example, in some embodiments, the patch expansion layer includes an upsampling layer and a second linear layer, the upsampling layer is used to perform an upsampling operation, and the second linear layer is used to perform a second channel number adjustment operation.

[0115] For example, in some implementations, the upsampling layer in the patch extension layer can implement the upsampling operation by bilinear interpolation, cubic interpolation or PixelShuffle. When bilinear interpolation / cubic interpolation is used, the upsampling layer in the patch extension layer can reshape the current feature map into a higher resolution post-sampling feature map (2x upsampling), and accordingly, the second linear layer in the patch extension layer reduces the number of channels of the post-sampling feature map to half the number of channels of the current feature map. When PixelShuffle is used, the upsampling layer in the patch extension layer can reshape the current feature map into a higher resolution post-sampling feature map (2x upsampling), and at the same time, the number of channels of the post-sampling feature map is reduced to one-fourth of the number of channels of the current feature map, and accordingly, the second linear layer in the patch extension layer expands the number of channels of the post-sampling feature map to one-half of the number of channels of the current feature map.

[0116] For example, the upsampling factor in the upsampling operation can be 2*2, that is, the width and height of the image are both enlarged by 2 times, that is, the height of the upsampled image becomes twice the width of the original image, and the height of the upsampled image becomes twice the height of the original image.

[0117] It should be noted that the present disclosure does not limit the specific structure of the patch extension layer as long as it can achieve the corresponding functions. The embodiments of the present disclosure also do not limit the specific implementation of the upsampling operation and the second channel number adjustment operation.

[0118] For example, Figure 5 As shown, the decoding subnetwork 20a and the connection subnetwork 30a process the first feature map F14 and the output F12 of the input encoding block 101a to obtain the output of the decoding subnetwork 20a.

[0119] For example, Figure 5 and Figure 6 As shown, at least one decoding block in the decoding subnetwork 20a includes an input decoding block 103a, and the input decoding block 103a includes a patch expansion layer Ua and multiple self-attention operators Mb. The connection subnetwork 30a includes a connection layer, namely a first connection layer 301a. The decoding subnetwork 20a also includes a linear projection layer LP, which is used to perform a projection mapping operation on the output of the first connection layer 301a to obtain the output of the decoding subnetwork 20a.

[0120] For example, in Figure 5 In the example shown, in each reconstruction operation, the first feature map and the output of at least one encoding block are processed using a decoding subnetwork and a connection subnetwork to obtain the output of the decoding subnetwork, including: processing the first feature map through an input decoding block to obtain the output of the input decoding block; connecting the output of the encoding block corresponding to the first connection layer and the output of the input decoding block through a first connection layer to obtain the output of the first connection layer, and performing a projection mapping operation on the output of the first connection layer through a linear projection layer to obtain the output of the decoding subnetwork.

[0121] For example, in some embodiments, processing the first feature map through the input decoding block to obtain the output of the input decoding block includes: performing an upsampling operation and a second channel number adjustment operation on the first feature map through a patch expansion layer in the input decoding block to obtain the output of the patch expansion layer; performing representation learning on the output of the patch expansion layer through multiple self-attention operators in the input decoding block to obtain the output of the input decoding block.

[0122] For example, Figure 5As shown, the patch expansion layer Ua in the input decoding block 103a performs an upsampling operation and a second channel number adjustment operation on the first feature map F14 to obtain the output FD11 (feature map) of the patch expansion layer Ua, and the multiple self-attention operators Mb in the input decoding block 103a perform representation learning on the output FD11 of the patch expansion layer to obtain the output FD12 (feature map) of the input decoding block 103a; the first connection layer 301a maps and connects the output F12 of the input encoding block 101a and the output FD12 of the input decoding block 103a to obtain the output of the first connection layer 301a (not shown), and finally, the linear projection layer LP performs a projection mapping operation on the output of the first connection layer 301a to obtain the output of the decoding subnetwork 20a. When N is 1, the output of the decoding subnetwork 20a is the reconstructed image 200a; when N is greater than 1, the output of the decoding subnetwork 20a is subjected to data fitting processing and data correction processing to obtain the input of the encoding subnetwork 10a in the next reconstruction operation, and then the reconstruction operation is performed again. Finally, after iterating N reconstruction operations, the reconstructed image 200a is obtained.

[0123] For example, in the case where the image reconstruction network includes a bottleneck layer, i.e. Figure 6 In the example shown, in each reconstruction operation, the first feature map is processed using the bottleneck layer to obtain the second feature map, including: using multiple self-attention operators in the bottleneck layer to perform representation learning on the first feature map to obtain the second feature map.

[0124] For example, Figure 6 As shown, multiple self-attention operators of the bottleneck layer 40a perform representation learning on the first feature map F14 to obtain a second feature map F15. The second feature map F15 is the output of the bottleneck layer 40a. It should be noted that Figure 6 The image reconstruction network shown in FIG. 1 has the same structure as FIG. 2 except that it includes a bottleneck layer 40a. Figure 5 The image reconstruction network shown is the same.

[0125] For example, Figure 6 As shown, the decoding subnetwork 20a and the connection subnetwork 30a process the second feature map F15 and the output F12 of the input encoding block 101a to obtain the output of the decoding subnetwork 20a.

[0126] For example, Figure 6 As shown, the size of the second feature map F15 is the same as that of the first feature map F14, for example, both are H / 8*W / 8. The dimension of the second feature map F15 is also the same as that of the first feature map F14, for example, both are C12.

[0127] It should be noted that in Figure 6 In the example, the bottleneck layer 40a has Figure 4BThe structure shown, i.e., the bottleneck layer 40a, includes only three self-attention operators, but the embodiments of the present disclosure are not limited thereto. According to actual application requirements, the bottleneck layer 40a may include more self-attention operators.

[0128] For example, in Figure 6 In the example shown, in each reconstruction operation, the second feature map and the output of at least one encoding block are processed using the decoding subnetwork and the connection subnetwork to obtain the output of the decoding subnetwork, including: processing the second feature map through the input decoding block to obtain the output of the input decoding block; connecting the output of the encoding block corresponding to the first connection layer and the output of the input decoding block through the first connection layer to obtain the output of the first connection layer; performing a projection mapping operation on the output of the first connection layer through the linear projection layer to obtain the output of the decoding subnetwork.

[0129] For example, the second feature map is processed by the input decoding block to obtain the output of the input decoding block, including: performing an upsampling operation and a second channel number adjustment operation on the second feature map by the patch expansion layer in the input decoding block to obtain the output of the patch expansion layer in the input decoding block; and performing representation learning on the output of the patch expansion layer in the input decoding block by multiple self-attention operators in the input decoding block to obtain the output of the input decoding block.

[0130] For example, Figure 6 As shown, the patch expansion layer Ua in the input decoding block 103a performs an upsampling operation and a second channel number adjustment operation on the second feature map F15 to obtain the output FD11 (feature map) of the patch expansion layer Ua in the input decoding block 103a, and the multiple self-attention operators Mb in the input decoding block 103a perform representation learning on the output FD11 of the patch expansion layer Ua in the input decoding block 103a to obtain the output FD12 (feature map) of the input decoding block 103a; the first connection layer 301a connects the output F12 of the input encoding block 101a and the output FD12 of the input decoding block 103a to obtain the output of the first connection layer 301a (not shown), and finally, the linear projection layer LP performs a projection mapping operation on the output of the first connection layer 301a to obtain the output of the decoding subnetwork 20a. When N is 1, the output of the decoding subnetwork 20a is the reconstructed image 200a; when N is greater than 1, the output of the decoding subnetwork 20a is subjected to data fitting processing and data correction processing to obtain the input of the encoding subnetwork 10a in the next reconstruction operation, and then the reconstruction operation is performed again. Finally, after iterating N reconstruction operations, the reconstructed image 200a is obtained.

[0131] For example, the size of the output FD11 of the patch expansion layer Ua in the input decoding block 103a is the same as the size of the output FD12 of the input decoding block 103a, for example, both are H / 4*W / 4; the dimension of the output FD11 of the patch expansion layer Ua in the input decoding block 103a is the same as the dimension of the output FD12 of the input decoding block 103a, for example, both are D11.

[0132] For example, the size of the output F12 of the input coding block 101a and the size of the output FD12 of the input decoding block 103a can be the same, for example, both are H / 4*W / 4; the number of channels of the output F12 of the input coding block 101a and the number of channels of the output FD12 of the input decoding block 103a can be the same, that is, C11=D11.

[0133] For example, the size of the second feature map F15 is smaller than the size of the output FD11 of the patch extension layer Ua; the dimension of the second feature map F15 is larger than the dimension of the output FD11 of the patch extension layer Ua, that is, C12 is larger than D11.

[0134] Figure 8 A schematic diagram of the structure of another image reconstruction network provided for some embodiments of the present disclosure.

[0135] For example, Figure 8 As shown, in some embodiments, the image reconstruction network includes an encoding subnetwork 10b, a decoding subnetwork 20b, a connection subnetwork 30b and a bottleneck layer 40b. The image reconstruction network is used to perform a reconstruction operation on the input image 100b to obtain a reconstructed image 200b corresponding to the input image 100b. For example, the input image 100b can be an MRI image. The size of the input image 100b is the same as the size of the reconstructed image 200b, for example, both are H*W, and the number of channels (i.e., dimension) of the input image 100b is the same as the number of channels of the reconstructed image 200b, for example, both are 1.

[0136] For example, the decoding subnetwork also includes a linear projection layer, at least one encoding block includes multiple encoding blocks connected in sequence, at least one decoding block includes multiple decoding blocks connected in sequence, the connection subnetwork includes multiple connection layers, and the multiple encoding blocks are connected to the multiple decoding blocks through multiple connection layers in a one-to-one mapping manner. Each connection layer is used to connect the output of the encoding block corresponding to the connection layer and the output of the decoding block corresponding to the connection layer, and the output of the connection layer is used as the input of the next decoding block connected to the decoding block corresponding to the connection layer, or the output of the connection layer is used as the input of the linear projection layer.

[0137] For example, in some embodiments, Figure 8As shown, the decoding subnetwork 20b also includes a linear projection layer LP, at least one encoding block includes an input encoding block 101b and at least one intermediate encoding block connected in sequence, and at least one intermediate encoding block includes a first intermediate encoding block 102b and a second intermediate encoding block 103b connected in sequence. At least one decoding block includes an input decoding block 201b and at least one intermediate decoding block connected in sequence, and at least one intermediate decoding block includes a first intermediate decoding block 202b and a second intermediate decoding block 203b connected in sequence. The connection subnetwork 30b includes a plurality of connection layers, and the plurality of connection layers are respectively a first connection layer 301b, a first connection layer 302b, and a first connection layer 303b.

[0138] It should be noted that although Figure 8 Only two intermediate coding blocks and two intermediate decoding blocks are shown, but the embodiments of the present disclosure are not limited to this. The number of intermediate coding blocks and the number of intermediate decoding blocks can be set according to actual conditions. Accordingly, the number of connection layers can also be set according to actual conditions, as long as one connection layer is set for each group of intermediate coding blocks and intermediate decoding blocks. The embodiments of the present disclosure do not make specific limitations on this.

[0139] For example, Figure 8 As shown, the second intermediate coding block 103b is mapped and connected to the input decoding block 201b through the first connection layer 301b, that is, the first connection layer 301b is used to connect the output of the second intermediate coding block 103b and the output of the input decoding block 201b. Similarly, the first intermediate coding block 102b is mapped and connected to the first intermediate decoding block 202b through the second connection layer 302b, and the input coding block 101b is mapped and connected to the second intermediate decoding block 203b through the third connection layer 303b. For example, the output of the first connection layer 301b is used as the input of the first intermediate decoding block 202b, the output of the second connection layer 302b is used as the input of the second intermediate decoding block 203b, and the output of the third connection layer 303b is used as the input of the linear projection layer LP.

[0140] For example, in some embodiments, each intermediate coding block also includes a patch merging layer, and each intermediate coding block is used to use the patch merging layer of the intermediate coding block to perform a downsampling operation and a first channel number adjustment operation on the input of the intermediate coding block to obtain the output of the patch merging layer of the intermediate coding block; and use multiple self-attention operators in the intermediate coding block to perform representation learning on the output of the patch merging layer of the intermediate coding block to obtain the output of the intermediate coding block.

[0141] For example, the patch merging layer in each intermediate coding block may also include a downsampling layer and a first linear layer to respectively implement a downsampling operation and a first channel number adjustment operation. It should be noted that the detailed description of the patch merging layer can refer to the above combined Figure 7 Description.

[0142] For example, the coding subnetwork further includes an output coding block, and the at least one intermediate coding block includes a first intermediate coding block and a second intermediate coding block. In each reconstruction operation, the coding subnetwork performs feature extraction processing on the input of the coding subnetwork to obtain a first feature map and an output of at least one coding block, including: processing the input of the coding subnetwork through the input coding block to obtain the output of the input coding block; processing the output of the input coding block using the first intermediate coding block to obtain the output of the first intermediate coding block; processing the output of the first intermediate coding block using the second intermediate coding block to obtain the output of the second intermediate coding block; processing the output of the second intermediate coding block using the output coding block to obtain the first feature map.

[0143] For example, the output of at least one encoding block includes the output of the input encoding block, the output of the first intermediate encoding block, and the output of the second intermediate encoding block.

[0144] For example, Figure 8 As shown, in the encoding subnetwork 10b, the input encoding block 101b includes multiple self-attention operators Ma and a linear embedding layer Ea, the first intermediate encoding block 102b includes multiple self-attention operators Ma and a patch merging layer Da, the second intermediate encoding block 103b includes multiple self-attention operators Ma and a patch merging layer Da, and the output encoding block 104b includes multiple self-attention operators Ma and a patch merging layer Da.

[0145] For example, the input of the encoding subnetwork is processed through the input encoding block to obtain the output of the input encoding block, including: processing the input of the encoding subnetwork through the linear embedding layer of the input encoding block to obtain the output of the linear embedding layer of the input encoding block; and then performing representation learning on the output of the linear embedding layer through multiple self-attention operators in the input encoding block to obtain the output of the input encoding block.

[0146] For example, Figure 8 As shown, the linear embedding layer Ea of the input coding block 101b is used to map the input dimension (i.e., the number of channels) of the coding subnetwork 10b to an arbitrary dimension C21 of the latent space to obtain the output F21 (feature map) of the linear embedding layer Ea. In this process, the linear embedding layer Ea expands the input dimension of the coding subnetwork 10b, and the specific expansion multiple can be set according to the actual situation. Then, the multiple self-attention operators Ma in the input coding block 101b perform representation learning on the output F21 of the linear embedding layer Ea to obtain the output F22 (feature map) of the input coding block 101b.

[0147] For example, Figure 8As shown, the dimension of the output F21 of the linear embedding layer Ea and the dimension of the output F22 of the input coding block 101b are the same, for example, both are C21, and in some examples, C21 can be 48. The size of the output F21 of the linear embedding layer Ea and the size of the output F22 of the input coding block 101b are the same, for example, both are H / 4*W / 4.

[0148] For example, the output F22 of the input coding block 101b is transmitted to the first intermediate coding block 102b to serve as the input of the first intermediate coding block 102b.

[0149] For example, the output of the input coding block is processed using the first intermediate coding block to obtain the output of the first intermediate coding block, including: using the patch merging layer of the first intermediate coding block to perform a downsampling operation and a first channel number adjustment operation on the output of the input coding block to obtain the output of the patch merging layer of the first intermediate coding block; and then using multiple self-attention operators in the first intermediate coding block to perform representation learning on the output of the patch merging layer of the first intermediate coding block to obtain the output of the first intermediate coding block.

[0150] For example, Figure 8 As shown, the patch merging layer Da of the first intermediate coding block 102b performs a downsampling operation and a first channel number adjustment operation on the output F22 of the input coding block 101b to obtain the output F23 (feature map) of the patch merging layer Da of the first intermediate coding block 102b; the multiple self-attention operators Ma in the first intermediate coding block 102b perform representation learning on the output F23 of the patch merging layer Da of the first intermediate coding block 102b to obtain the output F24 (feature map) of the first intermediate coding block 102b.

[0151] For example, Figure 8 As shown, the dimension of the output F23 of the patch merging layer Da of the first intermediate coding block 102b and the dimension of the output F24 of the first intermediate coding block 102b are the same, for example, both are C22. In some examples, C22 may be 96. The size of the output F23 of the patch merging layer Da of the first intermediate coding block 102b and the size of the output F24 of the first intermediate coding block 102b are the same. For example, if the downsampling factor of the downsampling operation performed by the patch merging layer Da of the first intermediate coding block 102b is 2*2, the size of the output F23 of the patch merging layer Da of the first intermediate coding block 102b and the size of the output F24 of the first intermediate coding block 102b are both H / 8*W / 8.

[0152] For example, the output F24 of the first intermediate coding block 102b is transmitted to the second intermediate coding block 103b to serve as the input of the second intermediate coding block 103b.

[0153] For example, the output of the first intermediate coding block is processed by the second intermediate coding block to obtain the output of the second intermediate coding block, including: using the patch merging layer of the second intermediate coding block to perform a downsampling operation and a first channel number adjustment operation on the output of the first intermediate coding block to obtain the output of the patch merging layer of the second intermediate coding block; and then using multiple self-attention operators in the second intermediate coding block to perform representation learning on the output of the patch merging layer of the second intermediate coding block to obtain the output of the second intermediate coding block.

[0154] For example, Figure 8 As shown, the patch merging layer Da of the second intermediate coding block 103b performs a downsampling operation and a first channel number adjustment operation on the output F24 of the first intermediate coding block 102b to obtain the output F25 (feature map) of the patch merging layer Da of the second intermediate coding block 103b; the multiple self-attention operators Ma in the second intermediate coding block 103b perform representation learning on the output F25 of the patch merging layer Da of the second intermediate coding block 103b to obtain the output F26 (feature map) of the second intermediate coding block 103b.

[0155] For example, Figure 8 As shown, the dimension of the output F25 of the patch merging layer Da of the second intermediate coding block 103b is the same as the dimension of the output F26 of the second intermediate coding block 103b, for example, both are C23. In some examples, C23 may be 192. The size of the output F25 of the patch merging layer Da of the second intermediate coding block 103b is the same as the size of the output F26 of the second intermediate coding block 103b. For example, if the downsampling factor of the downsampling operation performed by the patch merging layer Da of the second intermediate coding block 103b is 2*2, the size of the output F25 of the patch merging layer Da of the second intermediate coding block 103b and the size of the output F26 of the second intermediate coding block 103b are both H / 16*W / 16.

[0156] For example, the output F26 of the second intermediate coding block 103b is transmitted to the output coding block 104b to serve as the input of the output coding block 104b.

[0157] For example, the output of the second intermediate coding block is processed by the output coding block to obtain the first feature map, including: using the patch merging layer of the output coding block to perform a downsampling operation and a first channel number adjustment operation on the output of the second intermediate coding block to obtain the output of the patch merging layer of the output coding block; and then using multiple self-attention operators in the output coding block to perform representation learning on the output of the patch merging layer of the output coding block to obtain the output of the output coding block. The output of the output coding block is the first feature map.

[0158] For example, Figure 8As shown, the patch merging layer Da of the output coding block 104b performs a downsampling operation and a first channel number adjustment operation on the output F26 of the second intermediate coding block 103b to obtain the output F27 (feature map) of the patch merging layer Da of the output coding block 104b; the multiple self-attention operators Ma in the coding block of the output coding block 104b perform representation learning on the output F27 of the patch merging layer Da of the output coding block 104b to obtain the output F28 (feature map) of the output coding block 104b. The output F28 of the output coding block 104b is the output of the coding subnetwork 10b, that is, the output F28 of the output coding block 104b is the first feature map.

[0159] For example, Figure 8 As shown, the dimension of the output F27 of the patch merging layer Da of the output coding block 104b and the dimension of the first feature map F28 are the same, for example, both are C24. In some examples, C24 can be 384. The size of the output F27 of the patch merging layer Da of the output coding block 104b and the size of the first feature map F28 are the same. For example, if the downsampling factor of the downsampling operation performed by the patch merging layer Da of the output coding block 104b is 2*2, the size of the output F27 of the patch merging layer Da of the output coding block 104b and the size of the first feature map F28 are both H / 32*W / 32.

[0160] For example, the first feature map F28 (ie, the output of the output encoding block 104 b ) is transmitted to the bottleneck layer 40 b as the input of the bottleneck layer 40 b .

[0161] For example, Figure 8 As shown, multiple self-attention operators Mc in the bottleneck layer 40b are used to perform representation learning on the first feature map F28 to obtain the second feature map F29.

[0162] For example, the size of the second feature map F29 is the same as that of the first feature map F28, for example, both are H / 32*W / 32; the dimension of the second feature map F29 is also the same as that of the first feature map F28, for example, both are C24.

[0163] For example, the second feature map F29 is transmitted to the decoding sub-network 20b as an input of the decoding sub-network 20b.

[0164] For example, each decoding block further includes a patch extension layer. The patch extension layer includes an upsampling layer and a second linear layer, the upsampling layer is used to perform an upsampling operation, and the second linear layer is used to perform a second channel number adjustment operation. It should be noted that the detailed description of the patch extension layer can refer to the above description of the patch extension layer, and the repeated parts will not be repeated.

[0165] For example, the decoding subnetwork also includes a linear projection layer, at least one decoding block includes an input decoding block and at least one intermediate decoding block connected in sequence, at least one intermediate decoding block includes a first intermediate decoding block and a second intermediate decoding block connected in sequence, and the connection subnetwork includes at least one connection layer, and at least one connection layer includes a first connection layer, a second connection layer, and a third connection layer.

[0166] For example, in some embodiments, in each reconstruction operation, the second feature map and the output of at least one encoding block are processed using a decoding subnetwork and a connection subnetwork to obtain the output of the decoding subnetwork, including: processing the second feature map through an input decoding block to obtain the output of the input decoding block; connecting the output of the encoding block corresponding to the first connection layer and the output of the input decoding block through a first connection layer to obtain the output of the first connection layer; processing the output of the first connection layer through a first intermediate decoding block to obtain the output of the first intermediate decoding block; connecting the output of the encoding block corresponding to the second connection layer and the output of the first intermediate decoding block through a second connection layer to obtain the output of the second connection layer; processing the output of the second connection layer through a second intermediate decoding block to obtain the output of the second intermediate decoding block; connecting the output of the encoding block corresponding to the third connection layer and the output of the second intermediate decoding block through a third connection layer to obtain the output of the third connection layer; performing a projection mapping operation on the output of the third connection layer through a linear projection layer to obtain the output of the decoding subnetwork.

[0167] For example, Figure 8 As shown, in some embodiments, in the decoding subnetwork 20b, the input decoding block 201b includes a patch expansion layer Ua and multiple self-attention operators Mb, the first intermediate decoding block 202b includes a patch expansion layer Ua and multiple self-attention operators Mb, and the second intermediate decoding block 203b includes a patch expansion layer Ua and multiple self-attention operators Mb.

[0168] For example, in some embodiments, the second feature map is processed by the input decoding block to obtain the output of the input decoding block, including: performing an upsampling operation and a second channel number adjustment operation on the second feature map through a patch expansion layer in the input decoding block to obtain the output of the patch expansion layer in the input decoding block; performing representation learning on the output of the patch expansion layer in the input decoding block through multiple self-attention operators in the input decoding block to obtain the output of the input decoding block.

[0169] For example, Figure 8As shown, in some embodiments, the patch expansion layer Ua in the input decoding block 201b performs an upsampling operation and a second channel number adjustment operation on the second feature map F29 to obtain the output FD21 (feature map) of the patch expansion layer Ua in the input decoding block 201b, and multiple self-attention operators Mb of the input decoding block 201b perform representation learning on the output FD21 of the patch expansion layer Ua in the input decoding block 201b to obtain the output FD22 (feature map) of the input decoding block 201b.

[0170] For example, Figure 8 As shown, the dimension of the output FD21 of the patch extension layer Ua in the input decoding block 201b and the dimension of the output FD22 of the input decoding block 201b are the same, for example, both are D21. In some examples, D21 may be 192. The size of the output FD21 of the patch extension layer Ua in the input decoding block 201b and the size of the output FD22 of the input decoding block 201b are the same. For example, if the upsampling factor of the upsampling operation performed by the patch extension layer Ua in the input decoding block 201b is 2*2, then the size of the output FD21 of the patch extension layer Ua in the input decoding block 201b and the size of the output FD22 of the input decoding block 201b are both H / 16*W / 16.

[0171] For example, the first connection layer 301b connects the output F26 of the second intermediate coding block 103b and the output FD22 of the input decoding block 201b to obtain the output FC11 of the first connection layer 301b. For example, the output F26 of the second intermediate coding block 103b and the output FD22 of the input decoding block 201b are connected in the dimension direction, and then the dimension of the connected feature map is adjusted to obtain the output FC11 of the first connection layer 301b.

[0172] For example, the size of the output F26 of the second intermediate coding block 103b is the same as the size of the output FD22 of the input decoding block 201b, and the dimension of the output F26 of the second intermediate coding block 103b is also the same as the dimension of the output FD22 of the input decoding block 201b.

[0173] For example, the size of the output FC11 of the first connection layer 301b is the same as the size of the output FD22 of the input decoding block 201b, and the dimension of the output FC11 of the first connection layer 301b is also the same as the dimension of the output FD22 of the input decoding block 201b.

[0174] For example, the output FC11 of the first connection layer 301b is transmitted to the first intermediate decoding block 202b to serve as the input of the first intermediate decoding block 202b.

[0175] For example, in some embodiments, the output of the first connection layer is processed by the first intermediate decoding block to obtain the output of the first intermediate decoding block, including: performing an upsampling operation and a second channel number adjustment operation on the output of the first connection layer by the patch expansion layer in the first intermediate decoding block to obtain the output of the patch expansion layer in the first intermediate decoding block; performing representation learning on the output of the patch expansion layer in the first intermediate decoding block by multiple self-attention operators in the first intermediate decoding block to obtain the output of the first intermediate decoding block.

[0176] For example, Figure 8 As shown, in some embodiments, the patch expansion layer Ua in the first intermediate decoding block 202b performs an upsampling operation and a second channel number adjustment operation on the output FC11 of the first connection layer 301b to obtain the output FD23 (feature map) of the patch expansion layer Ua in the first intermediate decoding block 202b, and multiple self-attention operators Mb of the first intermediate decoding block 202b perform representation learning on the output FD23 of the patch expansion layer Ua in the first intermediate decoding block 202b to obtain the output FD24 (feature map) of the first intermediate decoding block 202b.

[0177] For example, Figure 8 As shown, the dimension of the output FD23 of the patch extension layer Ua in the first intermediate decoding block 202b and the dimension of the output FD24 of the first intermediate decoding block 202b are the same, for example, both are D22. In some examples, D22 may be 96. The size of the output FD23 of the patch extension layer Ua in the first intermediate decoding block 202b and the size of the output FD24 of the first intermediate decoding block 202b are the same. For example, if the upsampling factor of the upsampling operation performed by the patch extension layer Ua in the first intermediate decoding block 202b is 2*2, then the size of the output FD23 of the patch extension layer Ua in the first intermediate decoding block 202b and the size of the output FD23 of the first intermediate decoding block 202b are both H / 8*W / 8.

[0178] For example, the second connection layer 302b connects the output F24 of the first intermediate coding block 102b and the output FD24 of the first intermediate decoding block 202b to obtain the output FC12 of the second connection layer 302b. For example, the output F24 of the first intermediate coding block 102b and the output FD24 of the first intermediate decoding block 202b are connected in the dimension direction, and then the dimension of the connected feature map is adjusted to obtain the output FC12 of the second connection layer 302b.

[0179] For example, the size of the output F24 of the first intermediate coding block 102b is the same as the size of the output FD24 of the first intermediate decoding block 202b, and the dimension of the output F24 of the first intermediate coding block 102b is also the same as the dimension of the output FD24 of the first intermediate decoding block 202b.

[0180] For example, the size of the output FC12 of the second connection layer 302b is the same as the size of the output FD24 of the first intermediate decoding block 202b, and the dimension of the output FC12 of the second connection layer 302b is also the same as the dimension of the output FD24 of the first intermediate decoding block 202b.

[0181] For example, the output FC12 of the second connection layer 302b is transmitted to the second intermediate decoding block 203b to serve as the input of the second intermediate decoding block 203b.

[0182] For example, in some embodiments, the output of the second connection layer is processed by the second intermediate decoding block to obtain the output of the second intermediate decoding block, including: performing an upsampling operation and a second channel number adjustment operation on the output of the second connection layer by the patch expansion layer in the second intermediate decoding block to obtain the output of the patch expansion layer in the second intermediate decoding block; performing representation learning on the output of the patch expansion layer in the second intermediate decoding block by multiple self-attention operators in the second intermediate decoding block to obtain the output of the second intermediate decoding block.

[0183] For example, Figure 8 As shown, in some embodiments, the patch expansion layer Ua in the second intermediate decoding block 203b performs an upsampling operation and a second channel number adjustment operation on the output FC12 of the second connection layer 302b to obtain the output FD25 (feature map) of the patch expansion layer Ua in the second intermediate decoding block 203b, and multiple self-attention operators Mb of the second intermediate decoding block 203b perform representation learning on the output FD25 of the patch expansion layer Ua in the second intermediate decoding block 203b to obtain the output FD26 (feature map) of the second intermediate decoding block 203b.

[0184] For example, Figure 8As shown, the dimension of the output FD25 of the patch extension layer Ua in the second intermediate decoding block 203b and the dimension of the output FD26 of the second intermediate decoding block 203b are the same, for example, both are D23. In some examples, D23 can be 48. The size of the output FD25 of the patch extension layer Ua in the second intermediate decoding block 203b and the size of the output FD26 of the second intermediate decoding block 203b are the same. For example, if the upsampling factor of the upsampling operation performed by the patch extension layer Ua in the second intermediate decoding block 203b is 2*2, the size of the output FD25 of the patch extension layer Ua in the second intermediate decoding block 203b and the size of the output FD26 of the second intermediate decoding block 203b are both H / 4*W / 4.

[0185] For example, the third connection layer 303b connects the output F22 of the input coding block 101b and the output FD26 of the second intermediate decoding block 203b to obtain the output FC13 of the third connection layer 303b. For example, the output F22 of the input coding block 101b and the output FD26 of the second intermediate decoding block 203b are connected in the dimension direction, and then the number of channels of the connected feature map is adjusted to obtain the output FC13 of the third connection layer 303b.

[0186] For example, the size of the output F22 of the input coding block 101b is the same as the size of the output FD26 of the second intermediate decoding block 203b, and the dimension of the output F22 of the input coding block 101b is also the same as the dimension of the output FD26 of the second intermediate decoding block 203b.

[0187] For example, the size of the output FC13 of the third connection layer 303b is the same as the size of the output FD26 of the second intermediate decoding block 203b, and the dimension of the output FC13 of the third connection layer 303b is also the same as the dimension of the output FD26 of the second intermediate decoding block 203b.

[0188] For example, the output FC13 of the third connection layer 303 b is transmitted to the linear projection layer LP to serve as the input of the linear projection layer LP.

[0189] For example, the linear projection layer LP performs a projection mapping operation on the output FC13 of the third connection layer 303b to obtain the output of the decoding sub-network 20b. The linear projection layer LP is used to keep mapping the output FC13 of the third connection layer 303b to an image with the same size and dimension as the input of the encoding sub-network 10b.

[0190] For example, when N is 1, the output of the decoding subnetwork 20b is the reconstructed image 200b; when N is greater than 1, the output of the decoding subnetwork 20b is subjected to data fitting processing and data correction processing to obtain the input of the encoding subnetwork 10b in the next reconstruction operation, and then the reconstruction operation is performed again. Finally, after iterating N reconstruction operations, the reconstructed image 200b is obtained.

[0191] It should be noted that in the embodiments of the present disclosure, the output of each encoding block is a feature map, the output of each decoding block is also a feature map, the output of the bottleneck layer is also a feature map, and the output of the connection layer is also a feature map. The dimension of the feature map (i.e., the feature dimension) represents the number of channels of the feature map.

[0192] The image processing method provided by the embodiment of the present disclosure is described in detail below using magnetic resonance imaging as an example. The specific values ​​of block size, characteristic parameters, etc. in the following description of the present disclosure can be adjusted according to actual applications and are not limited to the specific values ​​listed below (for illustration and example).

[0193] For example, in the reconstruction process of MRI images, the forward model and the noise are known.

[0194] In the embodiment of the present disclosure, the sampled data f is first initialized by passing through the initialization layer of the unfolded iterative network. The following formula (2) realizes the conversion of the sampled data f from (frequency domain) to the corresponding intermediate input image m (0) (spatial domain) operations.

[0195] m (0) =A * f (2)

[0196] Among them, A * represents the pseudo-inverse of the undersampled Fourier transform (MRI forward operator).

[0197] Then, the initialized intermediate input image m is processed through the data fitting layer and the data correction layer. (0) Data fitting processing and data correction processing are performed respectively to obtain an input image.

[0198] Then, the image reconstruction network is used to perform N reconstruction operations on the input image to obtain the reconstructed image corresponding to the input image. The specific structure of the image reconstruction network can be referred to above Figure 5 , 6 The structure shown in FIG8 is repeated in detail. Figure 8 The image reconstruction network shown illustrates an embodiment of the present disclosure.

[0199] In the first reconstruction operation, first, the input image 100b is blocked by a blocking module with a block size of 4*4 to obtain multiple non-overlapping block images, and the size of each block image is 4*4. For example, the dimension of the input image is 1. If the size of the input image is 256*256, then after blocking the input image, 64*64 block images can be obtained. Each block image serves as a "token", and the feature dimension of each block image is set to the concatenation of the grayscale values ​​of the pixels in the block image, so the feature dimension of each block image becomes 4*4*1=16. Multiple block images are combined to obtain a combined block image. The size of the combined block image is H / 4*W / 4, and the dimension is 16. The combined block image is used as the input of the linear embedding layer Ea. At this time, the dimension of the input of the linear embedding layer Ea is 16, and the size is H / 4*W / 4. As shown Figure 8 As shown, the linear embedding layer Ea in the input encoding block 101b maps the dimension of the input of the linear embedding layer Ea to an arbitrary dimension C21 of the latent space to obtain a feature map F21, for example, C21 is 48. The number of channels of the feature map F21 is C21, and the size is H / 4*W / 4. Then, the feature map F21 is fed into the multiple self-attention operators Ma in the input encoding block 101b, and the multiple self-attention operators Ma in the input encoding block 101b are used to perform representation learning on the output of the linear embedding layer Ea (i.e., the feature map F21) to obtain a feature map F22. The multiple self-attention operators Ma in the input encoding block 101b keep the feature dimension and resolution unchanged.

[0200] The feature map F22 is fed into the patch merging layer Da of the first intermediate coding block 102b, which is used to perform a downsampling operation and a first channel number adjustment operation on the feature map F22 to obtain a feature map F23; then, multiple self-attention operators Ma in the first intermediate coding block 102b perform representation learning on the feature map F23 to obtain a feature map F24. The dimensions of the feature map F23 and the feature map F24 are both C22, C22 can be 96, and the size of the feature map F23 and the size of the feature map F24 are both H / 8*W / 8.

[0201] Similar to the processing process of the first intermediate coding block 102b, the patch merging layer Da in the second intermediate coding block 103b performs a downsampling operation and a first channel number adjustment operation on the feature map F24 to obtain a feature map F25, and the multiple self-attention operators Ma in the second intermediate coding block 103b perform representation learning on the feature map F25 to obtain a feature map F26. The dimensions of the feature map F25 and the dimension of the feature map F26 are both C23, C23 can be 192, and the size of the feature map F25 and the size of the feature map F26 are both H / 16*W / 16.

[0202] Finally, similar to the processing process of the first intermediate coding block 102b, the patch merging layer Da in the output coding block 104b performs a downsampling operation and a first channel number adjustment operation on the feature map F26 to obtain a feature map F27, and the multiple self-attention operators Ma in the output coding block 104b perform representation learning on the feature map F27 to obtain a feature map F28. The number of channels of the feature map F27 and the number of channels of the feature map F28 are both C24, C24 can be 384, and the size of the feature map F27 and the size of the feature map F28 are both H / 32*W / 32.

[0203] In order to reduce the depth of the network and minimize the number of model parameters, such as Figure 8 As shown, use Figure 4B The two window-based self-attention operators (i.e., window-based self-attention operator W1 and window-based self-attention operator W2) and the moving window-based self-attention operator SW1 are used to construct the bottleneck layer 40b to learn deep feature representation. In the bottleneck layer 40b, the feature dimension and resolution remain unchanged.

[0204] For example, the bottleneck layer 40b performs representation learning on the feature map F28 to obtain the feature map F29. The feature map F29 is transmitted to the decoding subnetwork 20b. The size of the feature map F29 is H / 32*W / 32, and the dimension is C24.

[0205] The structure of the decoding subnetwork 20b is similar to that of the encoding subnetwork 10b, and forms a bilaterally symmetrical structure therewith. In the decoding subnetwork 20b, the patch expansion layer Ua is used to expand the extracted features. First, the feature map F29 is upsampled and the second channel number adjustment operation is performed through the patch expansion layer Ua in the input decoding block 201b to obtain the feature map FD21. Then, the feature map FD21 is sent to multiple self-attention operators in the input decoding block 201b for representation learning to obtain the feature map FD22. The size of the feature map FD21 and the size of the feature map FD22 are both H / 16*W / 16, and the dimension of the feature map FD21 and the dimension of the feature map FD22 are both D21, and D21 can be 192. The multiple self-attention operators in the input decoding block 201b keep the feature dimension and resolution unchanged.

[0206] Then, the first connection layer 301b performs mapping connection processing on the feature map F26 and the feature map FD22 to obtain the feature map FC11. The number of channels of the feature map FC11 is D21, and the size is H / 16*W / 16. The feature map FC11 is transmitted to the first intermediate decoding block 202b.

[0207] Similarly, the first intermediate decoding block 202b processes the feature map FC11 to obtain the feature map FD24; the second connection layer 302b maps and connects the feature map F24 and the feature map FD24 to obtain the feature map FC12. The dimension of the feature map FC12 is D22, D22 can be 96, and the size is H / 8*W / 8. The feature map FC12 is transmitted to the second intermediate decoding block 203b. The second intermediate decoding block 203b processes the feature map FC12 to obtain the feature map FD26; the third connection layer 303b maps and connects the feature map F22 and the feature map FD26 to obtain the feature map FC13. The dimension of the feature map FC13 is D23, D23 can be 48, and the size is H / 4*W / 4.

[0208] Finally, the linear projection layer LP is applied to the feature map FC13 to obtain the output of the decoding subnetwork 20b. When the input of the image reconstruction network is a grayscale image, the dimension of the output of the decoding subnetwork 20b is 1; when the input of the image reconstruction network is an RGB image, the dimension of the output of the decoding subnetwork 20b is 3; the size of the output of the decoding subnetwork 20b is H*W. The output of the decoding subnetwork 20b is the reconstruction result of the first reconstruction operation.

[0209] Then, the reconstruction result of the first reconstruction operation is subjected to data fitting processing and data correction processing to obtain the input of the encoding sub-network in the second reconstruction operation.

[0210] After N times of iterative reconstruction of the input image 100b, the reconstructed image m is obtained. (n) , the calculation process is shown in the following formula:

[0211] The image reconstruction network performs an iterative reconstruction operation on the input image to obtain the reconstructed image m. (n) , reconstruct the image m (n) It is the reconstruction result of the nth reconstruction operation of the image reconstruction network.

[0212] It should be noted that in the above description of the present disclosure, magnetic resonance imaging is used as an example for schematic illustration, but the present disclosure is not limited to this. Those skilled in the art should know that the image processing method provided by the embodiments of the present disclosure is not limited to reconstructing magnetic resonance images, but can also reconstruct meteorological remote sensing images, monitoring images, etc. The image processing method provided by the embodiments of the present disclosure can reconstruct various types of images, and the type of the reconstructed image is not limited herein.

[0213] For example, in some embodiments, before acquiring the input image, the image processing method further includes: in a training phase, training an image reconstruction network using the sample image and the sample reconstructed image corresponding to the sample image.

[0214] For example, the training stage may include: using the image reconstruction network to be trained to perform N reconstruction operations on the sample image to obtain a training reconstructed image corresponding to the sample image; calculating the loss value of the parameters corresponding to the image reconstruction network to be trained by the loss function corresponding to the image reconstruction network according to the training reconstructed image and the sample reconstructed image; correcting the parameters corresponding to the image reconstruction network to be trained according to the loss value, and obtaining a trained image reconstruction network when the loss function meets a predetermined condition, and continuing to input the sample image and the sample reconstructed image corresponding to the sample image to repeat the above training process when the loss function does not meet the predetermined condition;

[0215] For example, in the training phase, the initial parameters of the image reconstruction network to be trained can be set to random numbers.

[0216] For example, in one example, the sample images may be various types of medical images in a medical image library, so that the parameters of the image reconstruction network corresponding to different types of medical images can be trained. According to actual needs, the sample images may include different types of medical images such as magnetic resonance images and X-ray computed tomography (CT) images. For example, the sample images may include medical images obtained by sampling people of different ages and places, so as to improve the accuracy and clarity of the images reconstructed by the image reconstruction network.

[0217] For example, in one example, the predetermined condition may correspond to the minimum loss function of the image reconstruction network under a certain sample image input. In another example, the predetermined condition may be that the number of training times or training cycles of the image reconstruction network reaches a predetermined number.

[0218] For example, the loss function may represent the difference between the training reconstructed image and the sample reconstructed image. During the training phase, the loss function corresponding to the image reconstruction network may include the mean absolute error (MAE) loss function and the structural similarity index (SSIM) loss function.

[0219] For example, the loss function corresponding to the image reconstruction network is expressed as:

[0220] L Mix =λ*L SSIM +(1-λ)*L MAE

[0221] Among them, LMix Represents the loss function corresponding to the image reconstruction network, L SSIM represents the structural similarity loss function, L MAE represents the mean absolute error loss function, and λ is a constant. For example, in some examples, λ is 0.8.

[0222] For example, L Mix The larger the value of (i.e., closer to 1), the greater the difference between the training reconstructed image and the sample reconstructed image, that is, the greater the adjustment value of the parameter of the image reconstruction network to be trained. Mix The smaller the value of , the smaller the difference between the training reconstructed image and the sample reconstructed image.

[0223] For example, in some embodiments, the image processing method further includes: obtaining a sample image. Obtaining a sample image may include: obtaining training measurement data; undersampling the training measurement data to obtain training sampling data; initializing the training sampling data to convert the training sampling data from the frequency domain to the corresponding spatial domain to obtain an intermediate sample image; performing data fitting processing and data correction processing on the intermediate sample image to obtain a sample image. For example, the training measurement data is full sampling data that has been fully sampled, that is, during the training process, the sample image can be obtained by undersampling the existing full sampling data, thereby simplifying the method of obtaining the sample image, and flexibly and simply obtaining the sample image. At this time, the sample reconstructed image corresponding to the sample image can be obtained by processing the training measurement data.

[0224] In the embodiments of the present disclosure, the image reconstruction network has an effective reconstruction effect on image reconstruction. Taking the accelerated nuclear magnetic resonance imaging task as an example, the image reconstruction network helps to learn the detailed information of undersampled nuclear magnetic resonance imaging. The image reconstruction network can obtain higher quality reconstructed images under the condition of reducing the network training time. A large number of experiments were carried out on the fastMRI data set (knee-singlecoil) in the random mask (random 4*zero padding) undersampling mode. The experimental results show that the image processing method is superior to other image reconstruction methods in accelerating the reconstruction of nuclear magnetic resonance images. The peak signal-to-noise ratio (PSNR) of the reconstructed image obtained based on the image processing method can reach nearly 30 decibels (dB), and the structural similarity (SSIM) can reach nearly 0.625.

[0225] At least one embodiment of the present disclosure further provides an image processing device. Fig. 9 A schematic block diagram of an image processing device provided for at least one embodiment of the present disclosure.

[0226] For example, Fig. 9As shown, the image processing device 90 may include a memory 905 and a processor 910. The memory 905 is used to store computer executable instructions non-temporarily; the processor 910 is used to run the computer executable instructions, and when the computer executable instructions are run by the processor 910, one or more steps in the image processing method described in any embodiment of the present disclosure may be executed. The specific implementation of each step of the image processing method and the related explanation content can be referred to the embodiment of the above-mentioned image processing method, which will not be repeated here.

[0227] It should be noted that Fig. 9 The components of the image processing device 90 shown are merely exemplary and non-limiting. The image processing device 90 may also have other components according to actual application requirements.

[0228] For example, the processor 910 and the memory 905 may communicate with each other directly or indirectly.

[0229] For example, the processor 910 and the memory 905 can communicate via a network connection. The network can include a wireless network, a wired network, and / or any combination of a wireless network and a wired network, and the present disclosure does not limit the type and function of the network. For another example, the processor 910 and the memory 905 can also communicate via a bus connection. The bus can be a peripheral component interconnect standard (PCI) bus or an extended industrial standard architecture (EISA) bus, etc.

[0230] For example, the processor 910 and the memory 905 may be arranged on a server side (or a cloud), or on a client side (eg, a mobile device such as a mobile phone).

[0231] For example, the processor 910 may be a device having data processing capability and / or instruction execution capability, such as a central processing unit (CPU), a tensor processing unit (TPU), or a graphics processing unit (GPU), and may control other components in the image processing device 90 to perform desired functions. The central processing unit (CPU) may be an X86 or ARM architecture, etc.

[0232] For example, the memory 905 may include any combination of one or more computer program products, and the computer program products may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, erasable programmable read-only memory (EPROM), portable compact disk read-only memory (CD-ROM), USB memory, flash memory, etc. One or more computer-executable instructions may be stored on the computer-readable storage medium, and the processor 910 may execute the computer-executable instructions to implement various functions of the image processing device 90. Various applications and various data, as well as various data used and / or generated by the application, etc. may also be stored in the memory 905.

[0233] It should be noted that the image processing device 90 can achieve technical effects similar to those of the aforementioned image processing method, and the repeated parts will not be repeated.

[0234] At least one embodiment of the present disclosure also provides a computer-readable storage medium. Fig.10 A schematic diagram of a computer-readable storage medium provided for at least one embodiment of the present disclosure.

[0235] For example, Fig.10 As shown, one or more computer executable instructions 1001 may be non-transitorily stored on a computer readable storage medium 1000. For example, when the computer executable instructions 1001 are executed by a computer, one or more steps in the image processing method described in any embodiment of the present disclosure may be executed.

[0236] For another example, when the computer executable instruction 1001 is executed by a computer, one or more steps in the training phase of the image processing method described in any embodiment of the present disclosure may also be executed.

[0237] For example, the computer-readable storage medium 1000 may be applied to the above-mentioned image processing device, for example, it may be the memory 905 in the image processing device 90 .

[0238] For example, the description of the computer-readable storage medium 1000 may refer to the description of the memory 905 in the embodiment of the image processing apparatus, and the repeated parts will not be repeated.

[0239] There are a few points to note about this disclosure:

[0240] (1) The drawings of the embodiments of the present disclosure only relate to the structures related to the embodiments of the present disclosure, and other structures may refer to the general design.

[0241] (2) For the sake of clarity, the thickness and size of layers or structures are exaggerated in the drawings used to describe the embodiments of the present invention. It is understood that when an element such as a layer, film, region, or substrate is referred to as being "on" or "under" another element, the element may be "directly" "on" or "under" the other element, or there may be intervening elements.

[0242] (3) In the absence of conflict, the embodiments of the present disclosure and the features therein may be combined with each other to obtain new embodiments.

[0243] The above description is only a specific implementation of the present disclosure, but the protection scope of the present disclosure is not limited thereto. The protection scope of the present disclosure shall be based on the protection scope of the claims.

Claims

1. An image processing method, comprising: Get the input image; Performing N reconstruction operations on the input image using an image reconstruction network to obtain a reconstructed image corresponding to the input image, Wherein, N is a positive integer greater than or equal to 1, The image reconstruction network includes an encoding subnetwork, a decoding subnetwork, a connection subnetwork and a bottleneck layer. The encoding subnetwork includes at least one encoding block, each encoding block includes multiple self-attention operators, The decoding subnetwork includes at least one decoding block, each decoding block includes multiple self-attention operators, The connection subnetwork includes a connection layer, One of the at least one coding block and one of the at least one decoding block are connected via the connection layer mapping, Each reconstruction operation includes: Performing feature extraction processing on the input of the encoding subnetwork through the encoding subnetwork to obtain a first feature map and an output of the at least one encoding block; Processing the first feature map using the bottleneck layer to obtain a second feature map; Processing the second feature map and the output of the at least one encoding block using the decoding subnetwork and the connection subnetwork to obtain an output of the decoding subnetwork; Wherein, the bottleneck layer includes multiple self-attention operators; Processing the first feature map using the bottleneck layer to obtain a second feature map includes: The first feature map is represented by using multiple self-attention operators of the bottleneck layer to obtain the second feature map.

2. The image processing method according to claim 1, wherein: The multiple self-attention operators of the bottleneck layer include window-based self-attention operators and moving window-based self-attention operators that are alternately connected in sequence. The number of window-based self-attention operators in the bottleneck layer is one more than the number of moving window-based self-attention operators.

3. The image processing method according to claim 1 or 2, wherein: When N is 1, for the N reconstruction operations, the input of the encoding subnetwork is the input image, and the output of the decoding subnetwork is the reconstructed image. When N is greater than 1, data fitting processing and data correction processing are performed on the output of the decoding subnetwork in the i-th reconstruction operation in the N reconstruction operations to obtain the input of the encoding subnetwork in the i+1-th reconstruction operation in the N reconstruction operations, where i is a positive integer and less than N, When i is 1, the input of the encoding subnetwork in the i-th reconstruction operation is the input image, and when i is N-1, the output of the decoding subnetwork in the i+1-th reconstruction operation is the reconstructed image.

4. The image processing method according to claim 1 or 2, wherein: The connection subnetwork includes at least one connection layer, and the at least one encoding block is connected to the at least one decoding block through the at least one connection layer in a one-to-one mapping manner. Each connection layer is used to map and connect encoding blocks and decoding blocks with the same size characteristics.

5. The image processing method according to claim 1 or 2, wherein: The decoding subnetwork further includes a linear projection layer, the at least one encoding block includes a plurality of encoding blocks connected in sequence, the at least one decoding block includes a plurality of decoding blocks connected in sequence, the connection subnetwork includes a plurality of connection layers, The multiple encoding blocks are connected to the multiple decoding blocks through the multiple connection layers in a one-to-one mapping manner. Each of the connection layers is used to connect the output of the encoding block corresponding to the connection layer and the output of the decoding block corresponding to the connection layer. The output of the connection layer is used as the input of the next decoding block connected to the decoding block corresponding to the connection layer, or the output of the connection layer is used as the input of the linear projection layer.

6. The image processing method according to claim 1 or 2, wherein: The encoding subnetwork further includes an output encoding block, the at least one encoding block includes an input encoding block, Performing feature extraction processing on the input of the encoding subnetwork through the encoding subnetwork to obtain a first feature map and an output of the at least one encoding block, including: Processing the input of the encoding subnetwork through the input encoding block to obtain the output of the input encoding block; The output encoding block is used to process the output of the input encoding block to obtain the first feature map.

7. The image processing method according to claim 6, wherein: The output encoding block includes multiple self-attention operators and patch merging layers. Processing the output of the input coding block by using the output coding block to obtain the first feature map includes: Performing a downsampling operation and a first channel number adjustment operation on the output of the input coding block through a patch merging layer in the output coding block to obtain a first coding feature map; Representation learning is performed on the first encoding feature map through multiple self-attention operators in the output encoding block to obtain the first feature map.

8. The image processing method according to claim 7, wherein: The patch merging layer includes a downsampling layer and a first linear layer, the downsampling layer is used to perform the downsampling operation, and the first linear layer is used to perform the first channel number adjustment operation; Performing a downsampling operation and a first channel number adjustment operation on the output of the input coding block through a patch merging layer in the output coding block to obtain a first coding feature map, including: Performing the downsampling operation on the output of the input coding block through the downsampling layer to obtain the output of the downsampling layer; The first channel number adjustment operation is performed on the output of the downsampling layer through the first linear layer to obtain the first encoding feature map.

9. The image processing method according to claim 7, wherein: The dimension of the first encoding feature map is the same as the dimension of the first feature map, and the size of the first encoding feature map is the same as the size of the first feature map.

10. The image processing method according to claim 1 or 2, wherein: The at least one coding block comprises an input coding block and at least one intermediate coding block connected in sequence, Each intermediate coding block further includes a patch merging layer, and each intermediate coding block is used to perform a downsampling operation and a first channel number adjustment operation on an input of the intermediate coding block using the patch merging layer of the intermediate coding block to obtain an output of the patch merging layer of the intermediate coding block; The output of the patch merging layer of the intermediate coding block is represented by learning the output of the intermediate coding block using multiple self-attention operators in the intermediate coding block to obtain the output of the intermediate coding block.

11. The image processing method according to claim 10, wherein: The encoding subnetwork further includes an output encoding block, the at least one intermediate encoding block includes a first intermediate encoding block and a second intermediate encoding block, Performing feature extraction processing on the input of the encoding subnetwork through the encoding subnetwork to obtain a first feature map and an output of the at least one encoding block, including: Processing the input of the encoding subnetwork through the input encoding block to obtain the output of the input encoding block; Processing the output of the input coding block using the first intermediate coding block to obtain the output of the first intermediate coding block; Processing the output of the first intermediate coding block by using the second intermediate coding block to obtain the output of the second intermediate coding block; The output of the second intermediate coding block is processed using the output coding block to obtain the first feature map.

12. The image processing method according to claim 11, wherein: The input coding block also includes a linear embedding layer, Processing the input of the encoding subnetwork through the input encoding block to obtain the output of the input encoding block includes: Processing the input of the encoding subnetwork through a linear embedding layer in the input encoding block to obtain a second encoding feature map; Representation learning is performed on the second encoding feature map through multiple self-attention operators in the input encoding block to obtain the output of the input encoding block.

13. The image processing method according to claim 12, wherein: The dimension of the second encoding feature map is the same as the dimension of the output of the input encoding block, and the size of the second encoding feature map is the same as the size of the output of the input encoding block.

14. The image processing method according to claim 4, wherein: Each decoding block further includes a patch expansion layer, and each decoding block is used to perform an upsampling operation and a second channel number adjustment operation on an input of the decoding block using the patch expansion layer of the decoding block to obtain an output of the patch merging layer of the decoding block; The output of the patch merging layer of the decoding block is represented by learning using multiple self-attention operators in the decoding block to obtain the output of the decoding block.

15. The image processing method according to claim 14, wherein: The patch extension layer includes an upsampling layer and a second linear layer, the upsampling layer is used to perform the upsampling operation, and the second linear layer is used to perform the second channel number adjustment operation.

16. The image processing method according to claim 14, wherein: The decoding subnetwork further comprises a linear projection layer, the at least one decoding block comprises an input decoding block, the at least one connection layer comprises a first connection layer, Processing the second feature map and the output of the at least one encoding block using the decoding subnetwork and the connection subnetwork to obtain the output of the decoding subnetwork includes: Processing the second feature map by the input decoding block to obtain an output of the input decoding block; The output of the coding block corresponding to the first connection layer is connected with the output of the input decoding block through the first connection layer to obtain the output of the first connection layer, A projection mapping operation is performed on the output of the first connection layer through the linear projection layer to obtain the output of the decoding sub-network.

17. The image processing method according to claim 14, wherein: The decoding subnetwork further includes a linear projection layer, the at least one decoding block includes an input decoding block and at least one intermediate decoding block connected in sequence, the at least one intermediate decoding block includes a first intermediate decoding block and a second intermediate decoding block connected in sequence, The at least one connection layer comprises a first connection layer, a second connection layer and a third connection layer, Processing the second feature map and the output of the at least one encoding block using the decoding subnetwork and the connection subnetwork to obtain the output of the decoding subnetwork includes: Processing the second feature map by the input decoding block to obtain an output of the input decoding block; Connecting the output of the coding block corresponding to the first connection layer and the output of the input decoding block through the first connection layer to obtain the output of the first connection layer; Processing the output of the first connection layer by the first intermediate decoding block to obtain an output of the first intermediate decoding block; Connecting the output of the coding block corresponding to the second connection layer and the output of the first intermediate decoding block through the second connection layer to obtain the output of the second connection layer; Processing the output of the second connection layer by the second intermediate decoding block to obtain an output of the second intermediate decoding block; Connecting the output of the coding block corresponding to the third connection layer and the output of the second intermediate decoding block through the third connection layer to obtain the output of the third connection layer; A projection mapping operation is performed on the output of the third connection layer through the linear projection layer to obtain the output of the decoding sub-network.

18. The image processing method according to claim 1 or 2, wherein: The multiple self-attention operators in each of the at least one encoding block and the at least one decoding block include at least one symmetric window self-attention operator block, each symmetric window self-attention operator block includes a window-based self-attention operator and a moving window-based self-attention operator, and the output of the window-based self-attention operator serves as the input of the moving window-based self-attention operator.

19. The image processing method according to claim 1 or 2, wherein: Getting the input image includes: Determine sampling data, wherein the sampling data is data obtained by undersampling; Initializing the sampled data to convert the sampled data from a frequency domain to a corresponding spatial domain to obtain an intermediate input image; The intermediate input image is processed to obtain the input image.

20. The image processing method according to claim 19, wherein: Processing the intermediate input image to obtain the input image includes: The intermediate input image is subjected to data fitting processing and data correction processing to obtain the input image.

21. The image processing method according to claim 1 or 2, wherein: The size of the input image is the same as that of the reconstructed image, and the number of channels of the input image is the same as that of the reconstructed image.

22. The image processing method according to claim 1 or 2, wherein: Before acquiring the input image, the image processing method further includes: In the training phase, the image reconstruction network is trained using sample images and sample reconstructed images corresponding to the sample images. Among them, in the training stage, the loss function corresponding to the image reconstruction network includes a mean absolute error loss function and a structural similarity loss function.

23. The image processing method according to claim 22, wherein: The loss function corresponding to the image reconstruction network is expressed as: L Mix =λ*L SSIM +(1-λ)*L MAE Among them, L Mix represents the loss function corresponding to the image reconstruction network, L SSIM represents the structural similarity loss function, L MAE represents the mean absolute error loss function, and λ is a constant.

24. An image processing device comprising a memory and a processor, in, The memory stores computer executable instructions suitable for being executed by the processor, and when the computer executable instructions are executed by the processor, one or more steps in the image processing method according to any one of claims 1-23 are performed.

25. A computer-readable storage medium that non-transitorily stores computer-executable instructions, in, When the computer executable instructions are executed by a computer, one or more steps in the image processing method according to any one of claims 1 to 23 are executed.

Images