Super-resolution interpolation reconstruction method, device, equipment and medium

Abstract

The application discloses a kind of super-resolution interpolation reconstruction method, device, equipment and medium, it is related to medical image analysis technical field, the method can generate dense horizontal position interpolation result from sparse sampling image, increase interlayer resolution.The method uses frequency domain space domain collaborative learning module to realize the feature complementation of frequency domain and space domain, high-frequency characteristics can learn with low-frequency characteristics fairly, to better learn high-frequency information to make up for the deficiency of space domain.Horizontal position is mainly used as feature extraction direction, while the information of sagittal position and coronal position is further improved interpolation reconstruction effect.Multiple-view feature fusion module is used to fuse different view information, complementary information is extracted from three views by making features from different views interact in real time, without increasing too much computing amount.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical image analysis, in particular to a super-resolution interpolation reconstruction method and device for jointly learning spatial domain and frequency domain information, equipment and medium. BACKGROUND

[0002] Computed Tomography (CT) is one of the important tools for doctors to diagnose diseases and preoperative planning, which has the advantages of fast scanning speed and clear imaging of bone structure. In the imaging process, in order to reduce radiation to patients and ensure acquisition efficiency, CT volume data is usually reconstructed by relatively thick slices. This results in most CT images being anisotropic, specifically, the slices are sparser in the horizontal direction than in the sagittal and coronal directions. This difference in resolution can result in insufficient detail being displayed in the sagittal and coronal planes, thereby affecting the visibility of lesions or anatomical structures (especially small lesions or fine structures), and making it difficult to accurately assess the volume and shape of tumors or organs. Therefore, increasing the inter-slice resolution is necessary for clinical practice.

[0003] In the process of super-resolution reconstruction of medical images, information such as organs, blood vessels and bone structures often contains more anatomical information. In the spatial domain, this information is often contained in high-dimensional features. However, convolution-based network structures tend to extract low-frequency information first and then further fit high-frequency information during training, which can result in higher difficulty in learning high-frequency information and can cause fitting bias. SUMMARY

[0004] In view of the above problems, the present application provides a super-resolution interpolation reconstruction method, device, equipment and medium for overcoming the above problems or at least partially solving the above problems.

[0005] The present application provides the following solutions:

[0006] A super-resolution interpolation reconstruction method, comprising:

[0007] obtaining a to-be-interpolated horizontal CT image, a to-be-interpolated coronal CT image and a to-be-interpolated sagittal CT image generated by slicing from an axial angle;

[0008] performing feature extraction on the to-be-interpolated horizontal CT image, the to-be-interpolated coronal CT image and the to-be-interpolated sagittal CT image in a horizontal direction as a main feature extraction direction to obtain a first feature set, the first feature set comprising a horizontal view feature, a coronal view feature extracted along a channel and a height dimension, and a sagittal view feature extracted along a channel and a height dimension;

[0009] The first multi-view feature fusion module is used for fusing the first feature set to obtain a second feature set;

[0010] The second feature set is input into a frequency-space domain collaborative learning module, and a second multi-view feature fusion module is used for fusing features output by the frequency-space domain collaborative learning module to obtain a third feature set;

[0011] The first feature set, the second feature set and the third feature set are fused by using a feature fusion module to obtain a dense interpolation result corresponding to the to-be-interpolated horizontal CT image pair;

[0012] The frequency-space domain collaborative learning module includes a frequency domain branch and a spatial domain branch, the frequency domain branch is used for converting spatial domain features into frequency domain features, so that high-frequency features that are difficult to capture in the spatial domain are represented in the frequency domain; and the spatial domain branch is used for extracting spatial characteristic information of an image, so as to capture local features and global features at different scales through a SwinTrans layer.

[0013] Preferably, the frequency-space domain collaborative learning module includes a network structure based on a Transformer.

[0014] Preferably, the frequency domain branch is used for converting spatial domain features into frequency domain features by performing a discrete cosine transform on the features.

[0015] Preferably, the frequency domain branch is used for converting spatial domain features into frequency domain features by performing a discrete cosine transform on the features, including:

[0016] Two discrete cosine transform coefficient matrices C h and C w are generated along the height and width directions respectively, as follows:

[0017]

[0018] In the formula, k1 and k2 are frequency indexes in the height and width directions, p represents a row index of an image, and q represents a column index of the image.

[0019] The spatial domain features are respectively subjected to a discrete cosine transform along the height and width directions to obtain frequency domain features That is:

[0020]

[0021] In the formula, f represents a feature set, and T represents a transpose of a matrix.

[0022] Preferably, the spatial domain branch is used for extracting spatial characteristic information of an image, including:

[0023] The spatial features of the input image are extracted by a convolutional layer to obtain features D, H, and W are the resolution of the horizontal, coronal, and sagittal positions, respectively.

[0024] After linear embedding and transformation of the dimensions, the features become C represents the number of channels.

[0025] The Enter the SwinTrans layer, capture the local and global dependencies in the image through the self-attention mechanism of the sliding window and the multi-head self-attention mechanism, and realize the global feature representation of the CT image at different scales, as shown in the following formula:

[0026]

[0027] In the formula, is the i-th SwinTrans layer.

[0028] Preferably, the Before inputting the SwinTrans layer, the frequency domain features are subjected to layer normalization processing.

[0029] The sliding window of the SwinTrans layer divides the entire frequency band into non-overlapping MxM windows and performs self-attention operation.

[0030] The sliding window method allows features of different frequency bands to interact, realizing cross-frequency band and cross-channel feature learning.

[0031] The frequency domain features are transformed into spatial domain features by inverse discrete cosine transformation

[0032] Preferably, the first multi-view feature fusion module is represented by the following formula:

[0033]

[0034] In the formula: F axial ∈R D×H×W represents the given horizontal position view feature, F sag ∈R W×D×H represents the sagittal position view feature extracted along the channel and height dimensions, F cor ∈R H×D×W represents the coronal position view feature extracted along the channel and width dimensions.

[0035] An ultra-resolution interpolation reconstruction device for performing the ultra-resolution interpolation reconstruction method described above, the device comprising:

[0036] An image acquisition unit is configured to acquire a horizontal CT image, a coronal CT image and a sagittal CT image to be interpolated, which are generated by slicing from an axial angle;

[0037] A first feature set acquisition unit is configured to perform feature extraction on the horizontal CT image, the coronal CT image and the sagittal CT image to be interpolated respectively by taking the horizontal direction as the main feature extraction direction to obtain a first feature set, which includes a horizontal view feature, a coronal view feature extracted along the channel and height dimensions, and a sagittal view feature extracted along the channel and height dimensions;

[0038] A second feature set acquisition unit is configured to fuse the first feature set by using a first multi-view feature fusion module to obtain a second feature set;

[0039] A frequency-space collaborative learning unit is configured to input the second feature set into a frequency-space collaborative learning module, and fuse features output by the frequency-space collaborative learning module by using a second multi-view feature fusion module to obtain a third feature set;

[0040] A feature fusion unit is configured to fuse the first feature set, the second feature set and the third feature set by using a feature fusion module to obtain a dense interpolation result corresponding to the horizontal CT image to be interpolated;

[0041] The frequency-space collaborative learning module includes a frequency branch and a spatial branch, the frequency branch is configured to convert spatial domain features into frequency domain features, so that high-frequency features that are difficult to capture in the spatial domain can be represented in the frequency domain; and the spatial branch is configured to extract spatial characteristic information of an image, so that local features and global features at different scales can be captured by a SwinTrans layer.

[0042] An ultra-resolution interpolation reconstruction device, the device includes a processor and a memory:

[0043] The memory is configured to store program code and transmit the program code to the processor;

[0044] The processor is configured to execute the above-mentioned ultra-resolution interpolation reconstruction method according to instructions in the program code.

[0045] A computer readable storage medium is configured to store program code, and the program code is configured to execute the above-mentioned ultra-resolution interpolation reconstruction method.

[0046] According to the specific embodiments of the present application, the following technical effects are provided:

[0047] The super-resolution interpolation reconstruction method, device, equipment and medium provided by the embodiment of the application can generate a dense horizontal bit interpolation result from a sparse sampled image, and increase the interlayer resolution. The method uses a frequency domain and spatial domain collaborative learning module to realize feature complementation of the frequency domain and the spatial domain, and high-frequency features can learn with low-frequency features in a fair manner, so that high-frequency information is better learned to compensate for the deficiency of the spatial domain. The horizontal bit is used as the main feature extraction direction, and the information of the sagittal bit and the coronal bit is fused to further improve the interpolation reconstruction effect. A multi-view feature fusion module is used to fuse different view information, and complementary information is extracted from three views by making features from different views interact in real time, without increasing too much calculation amount.

[0048] Of course, implementing any product of the application does not necessarily need to achieve all the advantages described above at the same time. BRIEF DESCRIPTION OF DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the following will briefly introduce the drawings needed in the embodiments. Obviously, the drawings in the following description only constitute some embodiments of the application, and other drawings can also be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0050] Figure 1 is a flowchart of the super-resolution interpolation reconstruction method provided by the embodiment of the application;

[0051] Figure 2 is a network structure diagram of the super-resolution interpolation reconstruction method provided by the embodiment of the application;

[0052] Figure 3 is a frequency domain and spatial domain collaborative learning structure diagram provided by the embodiment of the application;

[0053] Figure 4 is a multi-view feature fusion structure diagram provided by the embodiment of the application;

[0054] Figure 5 is a quantitative comparison diagram of the method provided by the embodiment of the application and other advanced methods;

[0055] Figure 6 is a visual comparison diagram of the method provided by the embodiment of the application and other advanced methods;

[0056] Figure 7 is a schematic diagram of the super-resolution interpolation reconstruction device provided by the embodiment of the application;

[0057] Figure 8 is a schematic diagram of the super-resolution interpolation reconstruction device provided by the embodiment of the application. DETAILED DESCRIPTION

[0058] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application belong to the scope of protection of the present application.

[0059] Referring to Figure 1 An ultra-resolution interpolation reconstruction method provided by the embodiments of the present application is shown in Figure 1 The method can include:

[0060] S101: acquiring a to-be-interpolated horizontal CT image, a coronal CT image and a sagittal CT image generated by slicing processing from an axial angle;

[0061] S102: performing feature extraction on the to-be-interpolated horizontal CT image, the coronal CT image and the sagittal CT image respectively in a horizontal direction as a main feature extraction direction to obtain a first feature set, the first feature set including a horizontal view feature, a coronal view feature extracted along a channel and a height dimension, and a sagittal view feature extracted along a channel and a height dimension;

[0062] S103: fusing the first feature set by using a first multi-view feature fusion module to obtain a second feature set;

[0063] S104: inputting the second feature set into a frequency-space domain collaborative learning module, and fusing features output by the frequency-space domain collaborative learning module by using a second multi-view feature fusion module to obtain a third feature set;

[0064] S105: fusing the first feature set, the second feature set and the third feature set by using a feature fusion module to obtain a dense interpolation result corresponding to the to-be-interpolated horizontal CT image;

[0065] The frequency-space domain collaborative learning module includes a frequency domain branch and a spatial domain branch, the frequency domain branch is used to convert spatial domain features into frequency domain features, so that high-frequency features that are difficult to capture in the spatial domain can be represented in the frequency domain; and the spatial domain branch is used to extract spatial characteristic information of an image, so that local features and global features at different scales can be captured by a SwinTrans layer.

[0066] In a specific implementation, the frequency-space domain collaborative learning module can include a network structure based on a Transformer.

[0067] The frequency domain branch is used to convert spatial domain features into frequency domain features by performing discrete cosine transform on the features.

[0068] Generate two discrete cosine transform coefficient matrices C along the height and width directions respectively h and C w , as follows:

[0069]

[0070] Where k1 and k2 are the frequency indices in the height and width directions, p represents the row index of the image, and q represents the column index of the image;

[0071] The spatial domain features Perform discrete cosine transform in the height and width directions to obtain frequency domain features Right now:

[0072]

[0073] Where: f represents the feature set, and T represents the transpose of the matrix.

[0074] The spatial domain branch is used to extract spatial characteristic information of the image, including:

[0075] The convolution layer extracts the spatial features of the input image to obtain features D, H, and W are the resolutions of the horizontal, coronal, and sagittal planes, respectively;

[0076] After linear embedding and dimension transformation, the features become C represents the number of channels;

[0077] described Entering the SwinTrans layer, the local and global dependencies in the image are captured through the sliding window self-attention mechanism and the multi-head self-attention mechanism, realizing the global feature representation of CT images at different scales, as shown in the following formula:

[0078]

[0079] Where, is the i-th SwinTrans layer.

[0080] described Before inputting into the SwinTrans layer, the frequency domain features are normalized.

[0081] The sliding window of the SwinTrans layer divides the entire frequency band into non-overlapping M×M windows and performs self-attention operation;

[0082] The features of different frequency bands are interacted in a sliding window manner to realize cross-frequency band and cross-channel feature learning.

[0083] The frequency domain features are transformed into spatial domain features by inverse discrete cosine transform

[0084] The first multi-view feature fusion module is represented by the following formula:

[0085]

[0086] In the formula, Faxial∈R D×H×W represents a given horizontal view feature, Fsag∈R W×D×H represents a sagittal view feature extracted along the channel and height dimensions, and Fcor∈R H×D×W represents a coronal view feature extracted along the channel and width dimensions.

[0087] The super-resolution interpolation reconstruction method provided in the embodiments of the present application can generate a dense horizontal view interpolation result from a sparsely sampled image, and increase the interlayer resolution. The method uses a frequency domain and spatial domain collaborative learning module to realize feature complementation in the frequency domain and the spatial domain, so that high-frequency features can be learned fairly with low-frequency features, thereby better learning high-frequency information to compensate for the deficiency of the spatial domain. The horizontal view is taken as the main feature extraction direction, and the information of the sagittal view and the coronal view is fused to further improve the interpolation reconstruction effect. A multi-view feature fusion module is used to fuse different view information, and complementary information is extracted from the three views by making the features from different views interact in real time, without increasing too much computational amount.

[0088] The super-resolution interpolation reconstruction method provided in the embodiments of the present application will be described in detail below.

[0089] The embodiments of the present application realize a method for interpolating a CT image in a horizontal view by using an interpolation network that learns spatial domain and frequency domain information together, and the network structure is as shown in Figure 2 The method takes the horizontal view as the main feature extraction direction, and fuses the information of the sagittal view and the coronal view to further improve the interpolation reconstruction effect.

[0090] The anisotropic CT data input into the network is V∈R D×H×W , where D, H and W are the resolutions of the horizontal view, the coronal view and the sagittal view respectively. Generally, D

[0091] The 2D image sets of the horizontal view, the coronal view and the sagittal view are respectively represented as V axial ∈R D×W ,V cor ∈R D ×W and Vsag ∈R D×H .

[0092] The method aims to generate a dense interpolation result from a sparsely sampled image V where S is the up-sampling factor in the horizontal direction, and S-1 slices are interpolated between every two adjacent slices in the horizontal direction. The method mainly includes two core parts, which are the frequency-space collaborative learning module and the multi-view feature fusion module.

[0093] The frequency-space collaborative learning module is a network structure based on Transformer, which contains a frequency branch and a spatial branch.

[0094] The frequency branch converts the spatial domain to the frequency domain by performing a discrete cosine transform on the features, allowing high-frequency features that are difficult to capture in the spatial domain to be better represented in the frequency domain.

[0095] The spatial branch is used to extract the spatial characteristic information of the image. Through the proposed SwinTrans layer, it can efficiently capture local and global features at different scales. Finally, the features in the frequency domain and the spatial domain are complementary. SwinTrans is a hierarchical network that contains a sliding window operation. It restricts the attention mechanism within a window, taking into account local information on the one hand and saving computational resources on the other.

[0096] In the spatial domain, it includes convolution, linear embedding, transformation dimension, and global feature extraction steps, as shown in Figure 3 First, the spatial features of the input image are extracted through the convolution layer to obtain the feature After linear embedding and transformation dimension, the feature becomes where C is the number of channels. Then enter the SwinTrans layer, which captures local and global dependencies in the image through the sliding window self-attention mechanism and multi-head self-attention mechanism, improving the global feature representation of the CT image at different scales. It is shown by the following formula:

[0097]

[0098] where, is the i-th SwinTrans layer. Finally, the output is

[0099] In the frequency domain, since the discrete cosine transform has higher computational efficiency than the Fourier transform in processing image data and can reduce the problem of discontinuity at the boundary, the method uses the discrete cosine transform (DCT) to convert the spatial domain features to the frequency domain features. Specifically, the feature Transformed into frequency domain features by DCT Specifically, first, two DCT coefficient matrices C h and C w are generated along the height and width directions respectively

[0100]

[0101] where k1 and k2 are the frequency indices in the two directions. The spatial domain features are respectively transformed into frequency domain features by DCT in the two directions, i.e.

[0102]

[0103] In order to reduce the internal covariant shift and the stability of training, the layer normalization processing is performed on the frequency domain features before the input SwinTrans layer. In order to enable the features of different frequency bands to be equally learned, the sliding window of the SwinTrans layer first divides the entire frequency band into non-overlapping MxM windows and performs self-attention operation, ensuring that the frequency band of each window can be kept within a small difference range. Then, the features of different frequency bands are interacted in a sliding window manner to break the boundary of window division and realize cross-frequency band and cross-channel feature learning. Finally, the frequency domain features are transformed into spatial domain features

[0104] Since the axial view contains more information than other views, our interpolation network chooses to slice from the axial angle. However, sometimes even if the true complete organ tissue information can be obtained from the axial view, the organ morphological results obtained in the coronal or horizontal position may deviate from the correct direction. Therefore, it is necessary to combine the information of the sagittal and coronal positions and supplement the features learned from the horizontal view. The existing methods process the three views respectively, and then fuse them in the last stage, which will cause no real-time interaction between these features.

[0105] To this end, the present application provides two multi-view feature fusion modules that fuse information of different views, as shown in Figure 4 by making the features from different views interact in real time to extract complementary information from the three views without increasing too much computational cost. Given the horizontal view feature Faxial∈R D×H×W , the sagittal view feature F sag ∈R W×D×H is extracted along the channel and height dimensions, and the coronal view feature F cor ∈R H×D×W is extracted along the channel and width dimensions. The final output feature can be represented as:

[0106]

[0107] The CT images in the horizontal, coronal and sagittal directions to be interpolated are encoded by an encoder to obtain a first feature set The first feature set The first feature set is first input into a first multi-view feature fusion module for fusion to obtain a second feature set The second feature set The second feature set is input into a frequency-space domain collaborative learning module for frequency-space domain collaborative learning, and the obtained features are fused by a second multi-view feature fusion module to obtain a third feature set The first feature set, the second feature set and the third feature set are fused by a channel fusion module, and the fused features are decoded by a decoder to obtain a dense interpolation result. The second feature set The third feature set The first feature set, the second feature set and the third feature set are fused by a channel fusion module, and the fused features are decoded by a decoder to obtain a dense interpolation result.

[0108] In the network training stage, the network provided in the embodiments of the present application can use L1 norm as a loss function.

[0109] To verify the effectiveness of the method, experiments were performed on 120 CT data provided by Wenzhou Eye Optometry Hospital. The resolution of the CT is 512x512xL, where L∈[102, 179]. Figure 5 The quantitative and qualitative comparison results of the method and the current advanced method are presented. Figure 6 The visualization comparison of the method provided in the present application and other advanced methods is shown. The first row shows the results of the horizontal slice, and the second row shows the error map. The brighter, the greater the error.

[0110] Specifically, the method provided in the present application achieves the best performance under the up-sampling factors of x2, x4 and x6, and achieves PSNR values of 40.24 dB, 34.37 dB and 31.39 dB, respectively. The best results are achieved in SSIM.

[0111] It can be seen that the method provided in the embodiments of the present application can achieve good performance under different up-sampling factors. And with the increase of the up-sampling factor, the performance decreases at a smaller rate.

[0112] Referring to Figure 7 , the embodiments of the present application can also provide a super-resolution interpolation reconstruction device, as shown in Figure 7 , for performing the super-resolution interpolation reconstruction method described above. The device can include:

[0113] The image acquisition unit 701 is configured to acquire a horizontal CT image, a coronal CT image and a sagittal CT image to be interpolated generated by slicing from an axial angle;

[0114] The first feature set acquisition unit 702 is configured to perform feature extraction on the horizontal CT image, the coronal CT image and the sagittal CT image to be interpolated respectively in a horizontal direction as a main feature extraction direction to obtain a first feature set, which includes a horizontal view feature, a coronal view feature extracted along a channel and a height dimension, and a sagittal view feature extracted along a channel and a height dimension.

[0115] The second feature set acquisition unit 703 is configured to fuse the first feature set by using a first multi-view feature fusion module to obtain a second feature set.

[0116] The frequency-space domain collaborative learning unit 704 is configured to input the second feature set into a frequency-space domain collaborative learning module, and fuse features output by the frequency-space domain collaborative learning module by using a second multi-view feature fusion module to obtain a third feature set.

[0117] The feature fusion unit 705 is configured to fuse the first feature set, the second feature set and the third feature set by using a feature fusion module to obtain a dense interpolation result corresponding to the horizontal CT image to be interpolated.

[0118] The frequency-space domain collaborative learning module includes a frequency domain branch and a spatial domain branch. The frequency domain branch is configured to convert spatial domain features into frequency domain features, so that high-frequency features that are difficult to capture in the spatial domain can be represented in the frequency domain. The spatial domain branch is configured to extract spatial characteristic information of an image, so that local features and global features at different scales can be captured by a SwinTrans layer.

[0119] The embodiments of the present application can also provide a super-resolution interpolation reconstruction device. The device includes a processor and a memory:

[0120] The memory is configured to store program code and transmit the program code to the processor.

[0121] The processor is configured to execute the steps of the super-resolution interpolation reconstruction method according to the instructions in the program code.

[0122] As shown in Figure 8 The super-resolution interpolation reconstruction device provided by the embodiments of the present application can include a processor 10, a memory 11, a communication interface 12 and a communication bus 13. The processor 10, the memory 11 and the communication interface 12 can communicate with each other through the communication bus 13.

[0123] In the embodiments of the present application, the processor 10 can be a central processing unit (CPU), an application specific integrated circuit, a digital signal processor, a field programmable gate array, or other programmable logic device, etc.

[0124] The processor 10 can invoke a program stored in the memory 11, and specifically, the processor 10 can perform the operations in the embodiments of the super-resolution interpolation reconstruction method.

[0125] The memory 11 stores one or more programs, which can include program codes including computer operation instructions. In the embodiments of the present application, the memory 11 at least stores programs for implementing the following functions:

[0126] Obtaining a to-be-interpolated horizontal CT image, a coronal CT image and a sagittal CT image generated by slicing from an axial angle;

[0127] Performing feature extraction on the to-be-interpolated horizontal CT image, the coronal CT image and the sagittal CT image respectively in a horizontal direction as a main feature extraction direction to obtain a first feature set, the first feature set including horizontal view features, coronal view features extracted along the channel and height dimensions, and sagittal view features extracted along the channel and height dimensions;

[0128] Fusing the first feature set by using a first multi-view feature fusion module to obtain a second feature set;

[0129] Inputting the second feature set into a frequency-space domain collaborative learning module, and fusing features output by the frequency-space domain collaborative learning module by using a second multi-view feature fusion module to obtain a third feature set;

[0130] Fusing the first feature set, the second feature set and the third feature set by using a feature fusion module to obtain a dense interpolation result corresponding to the to-be-interpolated horizontal CT image;

[0131] The frequency-space domain collaborative learning module includes a frequency domain branch and a spatial domain branch. The frequency domain branch is used to convert spatial domain features into frequency domain features, so that high-frequency features that are difficult to capture in the spatial domain can be represented in the frequency domain. The spatial domain branch is used to extract spatial characteristic information of an image, so as to capture local features and global features at different scales through a SwinTrans layer.

[0132] In a possible implementation, the memory 11 can include a program storage area and a data storage area, where the program storage area can store an operating system and application programs required by at least one function (such as a file creation function, a data read-write function), and the like; and the data storage area can store data created during use, such as initialization data, and the like.

[0133] In addition, the memory 11 can include a high-speed random access memory, and can also include a non-volatile memory, for example, at least one magnetic disk storage device or other volatile solid-state storage device.

[0134] The communication interface 12 can be an interface of a communication module, used for connecting with other devices or systems.

[0135] Of course, it needs to be noted that, Figure 8 The structure shown does not constitute a limitation on the super-resolution interpolation reconstruction device in the embodiments of the present application, and in actual applications, the super-resolution interpolation reconstruction device can include more or fewer components than Figure 8 those shown, or combine certain components.

[0136] The embodiments of the present application can also provide a computer readable storage medium for storing program codes, where the program codes are used to perform the steps of the super-resolution interpolation reconstruction method described above.

[0137] It needs to be noted that, in this document, the relationship terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the terms “include”, “contain” or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the statement “including a…” does not exclude the presence of another identical element in the process, method, article or device including the element.

[0138] Those skilled in the art can clearly understand the application by the description of the above embodiments. The technical solutions of the application can be implemented by means of software plus necessary universal hardware platforms. Based on such an understanding, the technical solutions of the application can be embodied in the form of a software product. The computer software product can be stored in a storage medium, such as a ROM / RAM, a magnetic disk, or an optical disk, and includes a plurality of instructions to cause a computer device (such as a personal computer, a server, or a network device) to execute the methods described in various embodiments or some parts of the embodiments.

[0139] The various embodiments in the specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other. Each embodiment mainly describes the difference from other embodiments. In particular, the system or system embodiments are described more simply because they are basically similar to the method embodiments. The relevant parts can be referred to the part of the method embodiments. The above-described system and system embodiments are merely illustrative, and the units described as separate components can or can not be physically separated, and the components displayed as units can or can not be physical units, that is, they can be located in one place or distributed on multiple network units. According to actual needs, some or all of the modules can be selected to achieve the purpose of the embodiment. Those skilled in the art can understand and implement without creative labor.

[0140] The above only describes the preferred embodiments of the application and is not used to limit the protection scope of the application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the application is included in the protection scope of the application.

Claims

1. A super-resolution interpolation reconstruction method, characterized in that: include: Acquire a horizontal CT image to be interpolated, a coronal CT image, and a sagittal CT image generated by performing slice processing from an axial angle; performing feature extraction on the to-be-interpolated horizontal CT image, the coronal CT image, and the sagittal CT image, respectively, with the horizontal position being the primary feature extraction direction, to obtain a first feature set, wherein the first feature set includes horizontal view features, coronal view features extracted along channel and height dimensions, and sagittal view features extracted along channel and height dimensions; Using a first multi-view feature fusion module to fuse the first feature set to obtain a second feature set; Inputting the second feature set into the frequency-domain and space-domain collaborative learning module, and using the second multi-view feature fusion module to fuse the features output by the frequency-domain and space-domain collaborative learning module to obtain a third feature set; Using a feature fusion module to fuse the first feature set, the second feature set, and the third feature set to obtain a dense interpolation result corresponding to the horizontal CT image to be interpolated; Among them, the frequency domain and space domain collaborative learning module includes a frequency domain branch and a spatial domain branch. The frequency domain branch is used to convert spatial domain features into frequency domain features, so that high-frequency features that are difficult to capture in the spatial domain can be represented in the frequency domain; the spatial domain branch is used to extract the spatial characteristic information of the image, so as to capture local features and global features at different scales through the SwinTrans layer.

2. The super-resolution interpolation reconstruction method according to claim 1, characterized in that: The frequency-domain and space-domain collaborative learning module includes a Transformer-based network structure.

3. The super-resolution interpolation reconstruction method according to claim 2, characterized in that: The frequency domain branch is used to convert spatial domain features into frequency domain features by performing discrete cosine transform on the features.

4. The super-resolution interpolation reconstruction method according to claim 3, characterized in that: The frequency domain branch is used to convert the spatial domain features into frequency domain features by performing discrete cosine transform on the features, including: Generate two discrete cosine transform coefficient matrices C along the height and width directions respectively h and C w , as follows: Where k1 and k2 are the frequency indices in the height and width directions, p represents the row index of the image, and q represents the column index of the image; The spatial domain features Perform discrete cosine transform in the height and width directions to obtain frequency domain features Right now: Where: f represents the feature set, and T represents the transpose of the matrix.

5. The super-resolution interpolation reconstruction method according to claim 2, characterized in that: The spatial domain branch is used to extract spatial characteristic information of the image, including: The convolution layer extracts the spatial features of the input image to obtain features D, H, and W are the resolutions of the horizontal, coronal, and sagittal planes, respectively; After linear embedding and dimension transformation, the features become C represents the number of channels; described Entering the SwinTrans layer, the local and global dependencies in the image are captured through the sliding window self-attention mechanism and the multi-head self-attention mechanism, realizing the global feature representation of CT images at different scales, as shown in the following formula: Where, is the i-th SwinTrans layer.

6. The super-resolution interpolation reconstruction method according to claim 5, characterized in that: described Before inputting into the SwinTrans layer, the frequency domain features are normalized. The sliding window of the SwinTrans layer divides the entire frequency band into non-overlapping M×M windows and performs self-attention operation; A sliding window approach is used to allow features of different frequency bands to interact, enabling feature learning across frequency bands and channels. The frequency domain features are transformed into spatial domain features through inverse discrete cosine transform 7. The super-resolution interpolation reconstruction method according to claim 1, characterized in that: The first multi-view feature fusion module is expressed as follows: Where: F axial ∈R D×H×W Represents the given horizontal view feature, F sag ∈R W×D×H represents the sagittal view features extracted along the channel and height dimensions, F cor ∈R H×D×W Represents the coronal view features extracted along the channel and width dimensions.

8. A super-resolution interpolation reconstruction device, characterized in that: The apparatus is configured to perform the super-resolution interpolation reconstruction method according to any one of claims 1 to 7, comprising: An image acquisition unit is used to acquire a horizontal CT image to be interpolated, a coronal CT image, and a sagittal CT image generated by performing slice processing from an axial angle; a first feature set acquisition unit, configured to perform feature extraction on the horizontal CT image to be interpolated, the coronal CT image, and the sagittal CT image, respectively, with the horizontal position being a primary feature extraction direction, to obtain a first feature set, wherein the first feature set includes horizontal view features, coronal view features extracted along channel and height dimensions, and sagittal view features extracted along channel and height dimensions; A second feature set acquisition unit, configured to fuse the first feature set using a first multi-view feature fusion module to obtain a second feature set; a frequency-domain-spatial-domain collaborative learning unit, configured to input the second feature set into the frequency-domain-spatial-domain collaborative learning module, and fuse the features output by the frequency-domain-spatial-domain collaborative learning module using a second multi-view feature fusion module to obtain a third feature set; a feature fusion unit, configured to fuse the first feature set, the second feature set, and the third feature set using a feature fusion module to obtain a dense interpolation result corresponding to the horizontal CT image to be interpolated; Among them, the frequency domain and space domain collaborative learning module includes a frequency domain branch and a spatial domain branch. The frequency domain branch is used to convert spatial domain features into frequency domain features, so that high-frequency features that are difficult to capture in the spatial domain can be represented in the frequency domain; the spatial domain branch is used to extract the spatial characteristic information of the image, so as to capture local features and global features at different scales through the SwinTrans layer.

9. A super-resolution interpolation reconstruction device, characterized in that: The device includes a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is configured to execute the super-resolution interpolation reconstruction method according to any one of claims 1 to 7 according to instructions in the program code.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium is used to store program code, and the program code is used to execute the super-resolution interpolation reconstruction method according to any one of claims 1 to 7.

Images