Methods, devices, equipment, storage media, and products for the recognition of three-dimensional medical images.

Abstract

This application discloses a method, apparatus, device, storage medium, and product for recognizing three-dimensional medical images, relating to the field of artificial intelligence. The method includes: in the i-th round of feature extraction, rearranging the perspective of the (i-1)-th round of three-dimensional medical image features to obtain two-dimensional image features, where the (i-1)-th round of three-dimensional medical image features are features obtained from the (i-1)-th round of feature extraction of the three-dimensional medical image, and different two-dimensional image features are features of the (i-1)-th round of three-dimensional medical image features under different perspectives; extracting semantic features from each two-dimensional image feature to obtain image semantic features under different perspectives; fusing the image semantic features under different perspectives to obtain the i-th round of three-dimensional medical image features; and performing image recognition based on the k-th round of three-dimensional medical image features obtained from the k-th round of feature extraction to obtain the image recognition result of the three-dimensional medical image, where k is greater than or equal to i.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to a method, apparatus, device, storage medium, and product for recognizing three-dimensional medical images. Background Technology

[0002] In the medical field, using computer vision technology to identify three-dimensional medical images can help predict the state of a disease.

[0003] Currently, in the process of recognizing 3D medical images, dense prediction methods can be used for image analysis. Dense prediction methods refer to methods that predict each pixel in the image. In related technologies, dense prediction of 3D medical images involves image recognition based on the overall 3D medical image to obtain the image recognition result.

[0004] However, image recognition based directly on 3D medical images involves a large amount of computation, has low recognition efficiency, and requires a large amount of data for pre-training, making the method quite complex. Summary of the Invention

[0005] This application provides a method, apparatus, device, storage medium, and product for recognizing three-dimensional medical images, which can improve the recognition efficiency of three-dimensional medical images and reduce computational complexity. The technical solution is as follows:

[0006] On one hand, embodiments of this application provide a method for recognizing three-dimensional medical images, the method comprising:

[0007] In the i-th round of feature extraction, the three-dimensional medical image features of the (i-1)-th round are rearranged from the perspective to obtain two-dimensional image features. The three-dimensional medical image features of the (i-1)-th round are the features obtained by performing the (i-1)-th round of feature extraction on the three-dimensional medical image, and the different two-dimensional image features are the features of the three-dimensional medical image features of the (i-1)-th round under different perspectives.

[0008] Semantic features are extracted from each of the two-dimensional image features to obtain image semantic features from different perspectives;

[0009] The semantic features of the images from different perspectives are fused to obtain the i-th round of three-dimensional medical image features;

[0010] Image recognition is performed based on the three-dimensional medical image features obtained from the k-th round of feature extraction to obtain the image recognition result of the three-dimensional medical image, where k is greater than or equal to i.

[0011] On the other hand, embodiments of this application provide a three-dimensional medical image recognition device, the device comprising:

[0012] The perspective rearrangement module is used to rearrange the perspective of the three-dimensional medical image features in the (i-1)th round during the i-th round of feature extraction to obtain two-dimensional image features. The (i-1)th round of three-dimensional medical image features are the features obtained by performing the (i-1)th round of feature extraction on the three-dimensional medical image, and the different two-dimensional image features are the features of the (i-1)th round of three-dimensional medical image features under different perspectives.

[0013] The feature extraction module is used to extract semantic features from each of the two-dimensional image features to obtain image semantic features from different perspectives;

[0014] The feature fusion module is used to fuse the semantic features of the image from different perspectives to obtain the i-th round of three-dimensional medical image features.

[0015] The image recognition module is used to perform image recognition based on the three-dimensional medical image features obtained from the k-th round of feature extraction, and to obtain the image recognition result of the three-dimensional medical image, where k is greater than or equal to i.

[0016] On the other hand, embodiments of this application provide a computer device, the computer device including a processor and a memory, the memory storing at least one instruction, at least one program, code set or instruction set, the at least one instruction, the at least one program, the code set or instruction set being loaded and executed by the processor to implement the three-dimensional medical image recognition method as described above.

[0017] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction, at least one program, code set, or instruction set is stored therein, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the three-dimensional medical image recognition method as described above.

[0018] On the other hand, embodiments of this application provide a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the three-dimensional medical image recognition method provided above.

[0019] The beneficial effects of the technical solutions provided in this application include at least the following:

[0020] In this embodiment, in each feature extraction stage, the 3D medical image features are first rearranged by perspective, thus dividing them into 2D image features from different perspectives. Feature extraction is then performed on each of these 2D image features to obtain semantic features from different perspectives. These semantic features from different perspectives are then fused to obtain the extracted 3D image semantic features. In this process, because feature extraction is performed on 2D image features from different perspectives, compared to directly extracting 3D image features, the computational complexity is reduced. That is, feature extraction from different perspectives is performed using simplified local computational units, thereby improving the recognition efficiency of 3D medical images. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 A schematic diagram illustrating the principle of the three-dimensional medical image recognition method provided in the embodiments of this application is shown;

[0023] Figure 2 A schematic diagram of an implementation environment provided by an exemplary embodiment of this application is shown;

[0024] Figure 3 A flowchart illustrating a three-dimensional medical image recognition method provided in an exemplary embodiment of this application is shown;

[0025] Figure 4 A flowchart illustrating a method for recognizing three-dimensional medical images provided in another exemplary embodiment of this application is shown;

[0026] Figure 5 This illustration shows a schematic diagram of the overall image recognition structure provided in an exemplary embodiment of this application;

[0027] Figure 6 This invention provides a schematic diagram illustrating the spatial feature extraction process in an exemplary embodiment of this application.

[0028] Figure 7 This invention provides a schematic diagram illustrating the semantic feature extraction process in an exemplary embodiment of this application.

[0029] Figure 8 A schematic diagram illustrating the feature fusion process of an exemplary embodiment of this application is shown.

[0030] Figure 9A schematic diagram of the structure of a TR-MLP network illustrated in an exemplary embodiment of this application is shown;

[0031] Figure 10 A schematic diagram of the structure of a jump-connection fusion network illustrated in an exemplary embodiment of this application is shown;

[0032] Figure 11 This is a structural block diagram of a three-dimensional medical image recognition device provided in an exemplary embodiment of this application;

[0033] Figure 12 A schematic diagram of the structure of a computer device provided in an exemplary embodiment of this application is shown. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0035] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0036] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0037] Computer vision (CV) is a science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in recognizing and measuring targets, and further processes images to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image segmentation, image semantic understanding, image retrieval, video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), and common biometric recognition technologies such as facial recognition and fingerprint recognition. The 3D medical image recognition method involved in this application, i.e., the application of computer vision technology in the field of image recognition, reduces computational complexity and improves the efficiency of 3D medical image recognition by extracting features from the 2D image features corresponding to the 3D medical image features from different perspectives.

[0038] Indicative, such as Figure 1 As shown, in the i-th round of feature extraction, the i-1 round of three-dimensional medical image features 101 obtained from the (i-1)-th round of feature extraction are first rearranged by perspective to obtain the first two-dimensional image features 102 under the first perspective, the second two-dimensional image features 103 under the second perspective, and the third two-dimensional image features 104 under the third perspective. Semantic features are then extracted from the first two-dimensional image features 102, the second two-dimensional image features 103, and the third two-dimensional image features 104 under different perspectives to obtain the first image semantic features 105, the second image semantic features 106, and the third image semantic features 107. The three are then fused to obtain the i-th round of three-dimensional image semantic features 108.

[0039] By decomposing the features of a 3D medical image into 2D image features from different perspectives, and then extracting features from the 2D image features, the computational load is reduced, thereby improving the recognition efficiency of 3D medical images.

[0040] The method improved in this application can be applied to the image recognition process of any three-dimensional medical image. Illustratively, it can identify the category of each part in a three-dimensional medical image, thereby aiding in the analysis of lesions and organs.

[0041] Figure 2A schematic diagram of an implementation environment provided by an exemplary embodiment of this application is shown. This implementation environment includes a computer device 210 and a server 220. The computer device 210 and the server 220 communicate via a communication network. Optionally, the communication network can be a wired network or a wireless network, and the communication network can be at least one of a local area network (LAN), a metropolitan area network (MAN), and a wide area network (WAN).

[0042] Computer device 210 is an electronic device running a three-dimensional medical image recognition program. This electronic device can be a smartphone, tablet, or personal computer, etc., and this embodiment is not limited thereto. When it is necessary to recognize a three-dimensional medical image, the three-dimensional medical image can be input into the program of computer device 210. The computer device uploads the three-dimensional medical image to server 220, where server 220 performs image recognition and provides feedback on the recognition result.

[0043] Server 220 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0044] In some embodiments, server 220 is used to provide image recognition services for applications installed on computer device 210. Optionally, server 220 is equipped with an image recognition network for classifying three-dimensional medical images sent by computer device 210.

[0045] Of course, in other possible implementations, the image recognition network can also be deployed on the computer device 210 side, where image recognition is performed locally by the computer device 210 without the need for the server 220. Correspondingly, the image recognition network can be trained on the computer device 210 side, and this embodiment does not limit this. For ease of description, the following embodiments use the three-dimensional medical image recognition method executed by a computer device as an example for illustration.

[0046] Please refer to Figure 3 The diagram illustrates a flowchart of a three-dimensional medical imaging method provided in an exemplary embodiment of this application, the method comprising the following steps.

[0047] Step 301: In the i-th round of feature extraction, the perspective of the i-1 round of three-dimensional medical image features is rearranged to obtain two-dimensional image features. The i-1 round of three-dimensional medical image features are the features obtained by the i-1 round of feature extraction of the three-dimensional medical image, and different two-dimensional image features are the features of the i-1 round of three-dimensional medical image features under different perspectives.

[0048] Among them, three-dimensional medical image features are the features extracted from the three-dimensional medical image to be identified. The three-dimensional medical image to be identified can be a computed tomography (CT) image, a magnetic resonance imaging (MRI) image, or a positron emission tomography (PET) image, etc.

[0049] In this embodiment, three-dimensional medical image recognition is performed through a multi-round feature extraction process. Each round of feature extraction uses the same feature extraction network, and the input to the feature extraction network in each round is determined based on the output of the previous round. That is, in the i-th round of feature extraction, features are extracted based on the features of the (i-1)-th round of the three-dimensional medical image.

[0050] Since 3D medical image features are 3D data, directly extracting features from the entire 3D medical image involves a large computational load and a complex process. Therefore, in this embodiment, the 3D medical image features are first divided in each round of feature extraction. That is, in the i-th round of feature extraction, the features obtained in the (i-1)-th round of feature extraction are rearranged by perspective. Perspective rearrangement divides the 3D medical image features into 2D image features from different perspectives, thereby reducing computational complexity by performing feature extraction based on these 2D image features from different perspectives.

[0051] In one possible implementation, the (H, W, D) dimensions of the (i-1)th round of three-dimensional medical image features are rearranged by perspective to obtain two-dimensional image features containing three perspectives: (H, W), (H, D), and (W, D). Each perspective corresponds to a two-dimensional direction in the three-dimensional medical image features. Different two-dimensional image features are the image features corresponding to different two-dimensional image slices.

[0052] It should be noted that during the i-th round of feature extraction, when extracting features based on the (i-1)-th round of 3D medical image features, there may be a process of upsampling or downsampling the (i-1)-th round of 3D medical image features. At this time, the viewpoint of the (i-1)-th round of 3D medical image features after upsampling or downsampling will be rearranged to obtain 2D image features.

[0053] Step 302: Extract semantic features from each two-dimensional image feature to obtain image semantic features from different perspectives.

[0054] After obtaining the features of each two-dimensional image, semantic features will be extracted from these features to learn the image information in the corresponding two-dimensional image slices. The process of extracting semantic features from the two-dimensional image features includes learning the spatial information of the two-dimensional image transitions and learning the image semantics based on the corresponding viewpoint.

[0055] After extracting semantic features from each two-dimensional image feature, the corresponding semantic features of the image under different viewpoints can be obtained. That is, the semantic features of the image under the three viewpoints (H, W), (H, D) and (W, D) are obtained respectively.

[0056] Step 303: Perform feature fusion on the semantic features of the images from different perspectives to obtain the i-th round of three-dimensional medical image features.

[0057] In one possible implementation, after obtaining the semantic features of images from different perspectives, they can be fused to complete the feature extraction process of that round, obtaining the i-th round of three-dimensional medical image features, and then performing the feature extraction process of the (i+1)-th round of three-dimensional medical image features based on the i-th round of three-dimensional medical image features.

[0058] In this embodiment of the application, feature fusion is performed on the semantic features of images from different perspectives to achieve the aggregation of rich semantics from all perspectives, thereby completing the feature learning process of three-dimensional medical image features.

[0059] Step 304: Based on the three-dimensional medical image features obtained from the k-th round of feature extraction, perform image recognition to obtain the image recognition result of the three-dimensional medical image, where k is greater than or equal to i.

[0060] The feature extraction process will end after multiple rounds. Optionally, image recognition will be performed based on the 3D medical image features from the kth round after the kth round of feature extraction.

[0061] In summary, in this embodiment of the application, in each feature extraction stage, the 3D medical image features are first rearranged by perspective, thereby dividing them into 2D image features from different perspectives. Feature extraction is then performed on each of these 2D image features to obtain semantic features from different perspectives. These semantic features from different perspectives are then fused to obtain the extracted 3D image semantic features. In this process, because feature extraction is performed on 2D image features from different perspectives, compared to directly extracting 3D image features, the computational complexity is reduced. That is, feature extraction from different perspectives is performed using simplified local computational units, thereby improving the recognition efficiency of 3D medical images.

[0062] In one possible implementation, during the feature extraction process of two-dimensional image features from different perspectives, each two-dimensional image feature is divided to learn the features corresponding to local windows, and the context features of each two-dimensional image feature corresponding to the slice are learned to obtain the semantic features of the image from different perspectives. The following will be described with an exemplary embodiment.

[0063] Please refer to Figure 4 The diagram illustrates a flowchart of a three-dimensional medical imaging method provided in an exemplary embodiment of this application, the method comprising the following steps.

[0064] Step 401: In the i-th round of feature extraction, the three-dimensional medical image features of the (i-1)-th round are rearranged from the perspective to obtain two-dimensional image features.

[0065] After acquiring the 3D medical image, the first step is to perform initial embedding processing (pattern embedding). Optionally, a convolutional stem structure can be used for initial embedding to obtain initial 3D medical image features. Then, based on the initial 3D medical image features, multiple rounds of feature extraction are performed.

[0066] In this embodiment, the feature extraction process includes a feature encoding process and a feature decoding process. The feature encoding process includes downsampling of the 3D medical image features, while the feature decoding process includes upsampling of the 3D medical image features. The downsampling process uses a 3D convolution with a kernel size of 3 and a stride of 2, downsampling by a factor of two each time. The upsampling process uses a 3D transposed convolution with a kernel size of 2 and a stride of 2, upsampling by a factor of two each time. After multiple rounds of feature encoding and decoding, the obtained 3D medical image features are used for medical image recognition. Each round of feature extraction uses the same Transformer-MLP (TR-MLP) structure.

[0067] Indicative, such as Figure 5 As shown, the input size is C. iThe 3D medical image (×H×W×D) first undergoes initial embedding (patch embedding) 501, where the patch size is 2×2, resulting in 3D medical image features of C×H / 4×W / 4×D / 4. This is input into the first TR-MLP Block for the first round of feature extraction. After the first round of feature extraction, the resulting 3D medical image features are downsampled to 2C×H / 8×W / 8×D / 8, and this is input into the second TR-MLP Block for the second round of feature extraction. Then, the 3D medical image features of the second round are directly input into the third TR-MLP Block for the third round of feature extraction. After the third round, downsampling is performed again until it is downsampled to 8C×H / 32×W / 32×D / 32, and then the upsampling process continues. The feature encoding process in the downsampling process, namely the feature extraction process in TR-MLP Block 502 and the feature extraction process in the TR-MLP Block before it, is the feature encoding process, and the subsequent process is the feature decoding process.

[0068] Step 402: Extract spatial features from the two-dimensional image features to obtain the two-dimensional image spatial features.

[0069] After obtaining the two-dimensional image features corresponding to each viewpoint, spatial feature extraction is first performed on these features. This spatial feature extraction process involves learning the features of each corresponding two-dimensional image slice. During the spatial feature extraction based on the three viewpoints, the network parameters are shared, meaning the network parameters are identical. This process may include steps 402a-402c (not shown in the figure):

[0070] Step 402a: Divide the two-dimensional image features into windows to obtain local two-dimensional image features corresponding to N windows, and the N windows do not overlap.

[0071] In this process, a window-based multi-head self-attention (W-MSA) network structure is mainly used to model the long-range and local spatial semantic information in two-dimensional image slices. Specifically, when processing the two-dimensional image features using the W-MSA network structure, the two-dimensional image feature Z is first divided into N non-overlapping windows corresponding to local two-dimensional image features Z. i The partitioning process can be described as follows:

[0072] Z = {Z 1 Z 2 , ..., Z N}, N = HW / M 2

[0073] Where M is the window size set by W-MSA, and HW refers to the size of the two-dimensional image features, which is the size of the two-dimensional image obtained by segmentation under the (H, W) viewpoint.

[0074] Then, multi-head self-attention calculation is performed based on the window to obtain the output result.

[0075] Step 402b: Extract features from N local two-dimensional image features to obtain two-dimensional image window features.

[0076] When the local two-dimensional image features Z corresponding to N non-overlapping windows are obtained i Then, feature extraction is performed on each local two-dimensional image feature to obtain N two-dimensional image window features. This method includes the following steps:

[0077] Step 1: Perform self-attention processing on N local two-dimensional image features to obtain self-attention features of N local two-dimensional image features.

[0078] First, self-attention processing is performed on each local 2D image feature. This self-attention process is a multi-head self-attention process, with each local 2D image feature corresponding to multiple self-attention heads.

[0079] Optionally, self-attention processing is performed based on the query term Q, key term K, and value term V corresponding to the two-dimensional image features to obtain N self-attention features of the two-dimensional image window features.

[0080] Wherein, the query term (Query, Q), key term (Key, K), and value term (Value, V) corresponding to the k-th self-attention head are respectively and Then the local two-dimensional image feature Z corresponding to the i-th window i The k-th self-attention feature is calculated as follows:

[0081]

[0082]

[0083] RPE stands for Relative Position Encoding Information, which is the window position encoding and represents the spatial position information that the window can perceive.

[0084] Then the self-attention feature corresponding to the k-th self-attention head contains the features corresponding to N windows, that is:

[0085]

[0086] Step 2: Perform feature fusion on the self-attention features of N local two-dimensional image features to obtain the features inside the first image window.

[0087] After obtaining the self-attention features corresponding to each self-attention head for each window, all self-attention heads are merged and linearly mapped using a parameter matrix to obtain the internal features of the corresponding first image window. The method is as follows:

[0088] W-MSA(Z)=Concat[H1, H2,…,H k W H

[0089] Among them, W H That is, the parameter matrix.

[0090] Before performing self-attention processing based on the W-MSA structure, it is first necessary to process the features from view v. Normalization is performed, optionally using batch normalization (BN). Here, the viewpoint v is one of (H, W), (H, D), or (W, D). After normalization, it is input into the W-MSA structure for self-attention processing.

[0091] Indicative, such as Figure 6 As shown, firstly After performing Batch Normalization (BN), the data is then fed into W-MSA for self-attention processing. W-MSA includes a residual structure, meaning the W-MSA output is compared with the original input features. The images are then fused to obtain the features within the first image window. That is:

[0092]

[0093] v = {1, 2, 3}

[0094] Step 3: Perform convolution processing on the internal features of the first image window to obtain the interactive features of the first image window.

[0095] In this implementation, the W-MSA structure is used for feature learning of each segmented local 2D image. To further enhance the learning of 2D image features, in one possible implementation, a DWConv2D structure with a kernel size of 5 is used for convolution processing, thereby increasing the learning of locality between spatially adjacent windows. In another possible implementation, the features inside the first image window are input into the DWConv2D network for convolution processing to obtain the interaction features of the first image window.

[0096] DWConv2D also includes a residual structure, which fuses the features after convolution with the features inside the first image window to obtain the interactive features of the first image window. That is:

[0097]

[0098] Indicative, such as Figure 6 As shown, the features inside the first image window Input DWConv2D for convolution processing, and then combine the convolutional features with... The images are fused to obtain the interactive features of the first image window.

[0099] Step 4: Use a multilayer perceptron (MLP) to extract features from the interactive features of the first image window to obtain the two-dimensional image window features.

[0100] Subsequently, to further enhance the learning of 2D image switching under the corresponding viewpoint, the interaction features of the first image window after convolution were normalized using Batch Normalization (BN), and a Multilayer Perceptron (MLP) was used to learn the channel features, i.e. the slice features under the corresponding viewpoint, thus obtaining the 2D image window features.

[0101]

[0102] Step 402c involves rearranging the N windows and extracting features from the two-dimensional image windows corresponding to the rearranged N windows to obtain two-dimensional image spatial features. The window rearrangement is used to change the spatial position of the N windows.

[0103] After performing window self-attention learning using the W-MSA structure, it is also necessary to learn image feature information across windows. Therefore, in one possible implementation, the N windows are rearranged, thereby learning the window features of the rearranged two-dimensional image again.

[0104] Optionally, a shuffle operation can be used to rearrange the windows, thereby disrupting the spatial information and enhancing the interaction between cross-window information. After the windows are rearranged, the two-dimensional image window features corresponding to the N windows are learned to obtain the final two-dimensional image spatial features. This method may include the following steps:

[0105] Step 1: Perform self-attention processing on the N two-dimensional image window features after window rearrangement to obtain the self-attention features of the N two-dimensional image window features.

[0106] First, self-attention processing is performed on the two-dimensional image window features corresponding to each of the N rearranged windows to obtain self-attention features. The method is the same as the steps described above, and will not be repeated here.

[0107] Step 2: Perform feature fusion on N self-attention features to obtain the internal features of the second image window.

[0108] The process of fusing features to obtain the internal features of the second image window can be referred to the process of fusing features to obtain the internal features of the first image window, and will not be repeated here.

[0109] Step 3: Flip the internal features of the second image window and perform convolution on the flipped internal features of the second image window to obtain the interactive features of the second image window.

[0110] That is, the positions of the windows are shuffled again, and then the W-MSA structure is used to perform window self-attention learning again to enhance the learning of information across windows. After that, the internal features of the second image window are flipped to restore the position information of each window and restore them to their original positions.

[0111] Indicative, such as Figure 6 As shown, firstly, the features of the two-dimensional image window are normalized using BN. Then, a window repositioning operation (Transpose) is performed. Based on the W-MSA structure, feature learning is performed on the corresponding features of the N windows after the repositioning to obtain the second image window interaction features. Finally, the positions of the N windows are flipped again to restore the position information of each window. That is:

[0112]

[0113] in, This refers to the features obtained by flipping the position of the interactive features of the second image window.

[0114] After flipping the position, DWConv2D is used again for convolution processing to obtain the second image window interaction features. This process can be referred to as the process of obtaining the first image window interaction features through convolution processing in the above steps, and will not be repeated here.

[0115] Indicative, such as Figure 6 As shown, The input DWConv2D structure is convolved to obtain the interactive features of the second image window.

[0116]

[0117] Step 4: Use MLP to extract features from the interactive features of the second image window to obtain the two-dimensional image spatial features.

[0118] Correspondingly, after the convolution process, channel learning is performed again using MLP to obtain the final two-dimensional image spatial features.

[0119] Indicative, such as Figure 6 As shown, the interactive features of the second image window are first... The data is normalized and then input into an MLP for feature extraction to obtain the final two-dimensional image spatial features.

[0120]

[0121] Spatial feature extraction from two-dimensional images is a process of Full-View Slice Spatial Shuffle Block (FVSSSB), the overall process of which is as follows: Figure 6 As shown, this allows for the full learning of two-dimensional image features.

[0122] Step 403: Extract semantic features from the spatial features of the two-dimensional image based on the primary and secondary perspectives to obtain the semantic features of the image. The primary perspective is the perspective corresponding to the two-dimensional image features, and the secondary perspective is a perspective in the three-dimensional perspective that is different from the primary perspective.

[0123] Since two-dimensional image spatial features only represent the features of the corresponding two-dimensional viewpoint, after extracting spatial features from each two-dimensional image feature, the remaining semantic information from the third viewpoint is captured for supplementary learning. The process of extracting semantic features from the two-dimensional image spatial features to obtain image semantic features is a Slice-Aware Volume Context Mixing (SAVCM) process, in which the network parameters of the SAVCM network are shared across different viewpoints, meaning the network parameters are identical. This process may include the following steps:

[0124] Step 403a: Perform feature fusion on the two-dimensional image spatial features and the position coding features to obtain the first image semantic features. The position coding features are used to indicate the position information corresponding to the two-dimensional image features.

[0125] In one possible implementation, first, for each two-dimensional image spatial feature... Absolute Position Encoding (APE) is added to represent the spatial location information of corresponding 2D image features, i.e., the spatial location information of corresponding 2D image slices. This is an automatically learnable parameter, thereby injecting the location information of 2D image slices into the 2D image spatial features. Within, slice location-aware learning is implemented.

[0126] Indicative, such as Figure 7 As shown, the spatial features of the two-dimensional image are fused with the location-encoded features to obtain the first image semantic features. Right now:

[0127]

[0128] Among them, APE s That is Spatial location encoding corresponding to two-dimensional image features.

[0129] Step 403b: Using MLP, semantic features are extracted from the semantic features of the first image from the main viewpoint to obtain the semantic features of the main image.

[0130] In one possible implementation, semantic features are extracted from both the primary and secondary viewpoints. The primary viewpoint refers to the viewpoint corresponding to the two-dimensional image features, while the secondary viewpoints are the remaining viewpoints besides the primary viewpoint. For example, The two-dimensional image spatial features are obtained by extracting two-dimensional image features from the (H, W) perspective. The main perspective is (H, W), and the secondary perspective is the remaining D perspective.

[0131] Optionally, semantic features of the first image can be extracted using residual axial-MLP along the main viewpoint to obtain the semantic features of the main image. like Figure 7 As shown, the semantic features of the first image are first analyzed. The positions of (B, SP, C, TH) are rearranged to obtain (B, SP, TH, C). Then, MLP is used to extract along the channel C direction. First, the dimension is increased to 4C, and after extraction, it is restored to the original number of channels C. Then, the position of the extracted main image semantic features is restored to (B, SP, C, TH), where SP represents the spatial dimension under the main view.

[0132] Step 403c: Using MLP, semantic features of the first image are extracted from the secondary viewpoint to obtain the semantic features of the secondary image.

[0133] While extracting semantic features based on the primary perspective, semantic features of the first image are extracted using MLP based on the secondary perspective, resulting in secondary image semantic features. like Figure 7 As shown, the semantic features of the first image are extracted along the secondary viewpoint. The dimensions are first increased to 4TH, and after extraction, they are restored to the original dimension TH, where TH represents the spatial dimension under the secondary viewpoint.

[0134] Step 403d: Perform feature fusion on the semantic features of the main image and the semantic features of the auxiliary image to obtain the image semantic features.

[0135] After obtaining the semantic features of the main image and the semantic features of the auxiliary image, feature fusion is performed on the two to obtain the image semantic features. In one possible implementation, such as... Figure 7 As shown, as well as and original features The merged features are then mapped using an MLP to restore the original number of channels, thus obtaining the image semantic features. This process, by fusing image feature information from a third-person perspective, can perceive the contextual information of two-dimensional image slices, thereby improving the accuracy of feature learning. That is:

[0136]

[0137]

[0138] Step 404: The image semantic features and viewpoint features are fused to obtain the viewpoint image semantic features.

[0139] In the process of feature fusion, the features of each viewpoint are first processed. Adding APEs to the channels of the full-view learning features is beneficial because the aggregation of rich semantics is processed on a channel-by-channel basis. Therefore, APEs are added to the channels of the full-view features to achieve viewpoint awareness during aggregation. Here, APE is the encoding corresponding to a channel, indicating the corresponding viewpoint, such as the (H, W) viewpoint.

[0140]

[0141] Step 405: Perform feature fusion on the semantic features of the images from various perspectives to obtain the three-dimensional medical image features of the i-th round.

[0142] Next, we will examine the full-view features. The channels are merged, multiplied by 3, and then the merged features are normalized using LN. Finally, the MLP aggregator is used. va Mapping the features back to the original number of channels yields the medical volume feature output Z of the current block of the MLP-Transformer. l+1 That is, the 3D medical image features of the i-th round. That is:

[0143]

[0144] like Figure 8 As shown, the semantic features of each image are first fused with APE encoding, and then the three perspectives are stitched together to obtain the final three-dimensional medical image features.

[0145] Step 406: When the upsampling reaches the original size, the extracted three-dimensional medical image features are determined as the three-dimensional medical image features obtained in the k-th round of feature extraction.

[0146] In one possible implementation, when the upsampling reaches the original size of the 3D medical image, it is determined to be the k-th round of feature extraction. (Illustrative example follows.) Figure 5 As shown, when C′ is reached o When ×H×W×D, the corresponding result is determined as the k-th round of 3D medical image features. Target prediction (503) is then performed using these k-th round 3D medical image features to obtain the image recognition result. To further enhance the accuracy of image recognition, the target prediction result is fused with the corresponding features of the initial input 3D medical image, thereby performing image recognition based on the fused features. For example... Figure 5 As shown, firstly, the input three-dimensional medical image C... i Convolution is performed on ×H×W×D to obtain the feature C′ of the initial image. I ×H×W×D, C′ O ×H×W×D and C′ I The ×H×W×D values ​​are fused and then convolved to obtain the final output.

[0147] Step 407: Perform image recognition based on the features of the k-th round of three-dimensional medical images to obtain the image recognition result.

[0148] Finally, image recognition is performed based on the features of the k-th round of 3D medical images, which can then be used for image registration, classification, and other tasks on the 3D medical images.

[0149] In one possible implementation, the TR-MLP network structure can be as follows: Figure 9 As shown, firstly, the 3D medical image features Z input to the current block are... l The (H, W, D) dimensions are rearranged into (H, W), (H, D), and (W, D) 2D spatial slices containing three perspectives, each perspective corresponding to a 2D slice direction in 3D. The rearranged full-view 2D spatial slices are then fully learned using FVSSB. Next, slice-aware medical volume context mixing (SAVCM) is used to capture the remaining image semantic information along the third perspective. Finally, a perspective-aware aggregator is used to aggregate the rich semantics learned from the full perspective, resulting in the final 3D medical image feature output Z of this Transformer-MLP block. l+1The features are then used as input features for the next Transformer-MLP block. The three perspectives are computed in parallel, and the full-view slice space shuffling block network and the slice-aware medical body context hybrid block network share parameters across all perspectives. That is, semantic features are extracted from the 2D image features at each perspective using feature extraction networks with the same network parameters, resulting in image semantic features from different perspectives.

[0150] In this embodiment, by first learning the full-view 2D spatial information, then learning the third-dimensional semantics, and then fusing the full-view information, the context-aware capability of the 3D medical image features is realized, and the inductive bias capability is greatly enhanced, which can improve the accuracy of 3D medical image recognition. Furthermore, by replacing the computationally intensive 3D CNN and pure visual transformer with a simplified local visual Transformer-MLP computing unit, the computational complexity is reduced and the recognition efficiency is improved.

[0151] The feature extraction process includes feature encoding and feature decoding. Within the extraction process, a self-attention process is included, where self-attention is calculated based on Q, K, and V. In one possible implementation, to fuse multi-scale visual features, the features from the encoder and decoder are fused to obtain the Q, K, and V values ​​from the feature decoding process.

[0152] Optionally, the K and V values ​​in the t-th round of feature decoding are obtained by fusing the K and V values ​​in the (t-1)-th round of feature decoding with the K and V values ​​in the corresponding feature encoding process, and the Q value in the k-th round of decoding is the Q value in the (t-1)-th round of feature decoding.

[0153] In one possible implementation, the input features of the t-th round of feature decoding have the same resolution as the output features of its corresponding encoding process; that is, skip-connection fusion is performed on image features with the same resolution. (Illustrative example, such as...) Figure 5 As shown, the resolution corresponding to the second round of feature decoding is 4C×H / 16×W / 16×D / 16. The corresponding skip-connection fusion feature encoding process is the last round of encoding process with the same resolution of 4C×H / 16×W / 16×D / 16. When performing skip-connection fusion, the features input to the second round of feature decoding (i.e., the features after upsampling the output features of the first round of feature decoding) are fused with the output features of the last round of feature encoding process.

[0154] Let E be the output feature of the feature encoding process corresponding to the t-th round of feature decoding. v The input feature for the t-th round of feature decoding is D. v Let's take E as an example. Here, v refers to the feature corresponding to viewpoint v, that is, skip-connection fusion is performed under different viewpoints. First, let's look at E... vD v Convolution is performed using PWConv2D with a kernel size of 1. During feature decoding, the Q-value comes only from the previous feature decoding round, and for the skip-connection fusion between the encoder and decoder, only the K and V values ​​are fused. Therefore, as... Figure 10 As shown, PWConv2D is used to extract encoder features E v The original number of channels is divided into two parts to obtain encoder E. v K value and V value Right now:

[0155]

[0156] like Figure 10 As shown, PWConv2D is used to extract decoder features D v The original number of channels is divided into 3 parts to obtain the Q value. K value and V value Right now:

[0157]

[0158] Then, for the encoder With from the decoder To perform fusion, and to process the data from the encoder. With from the decoder To merge. That is:

[0159]

[0160] in, That is, the K value corresponding to the t-th round of feature decoding. That is, the D value corresponding to the t-th round of feature decoding, and the Q value corresponding to the t-th round of feature decoding. That is Of these, three are used for learning W-MSA in the t-th round of feature decoding. That is:

[0161]

[0162] In this embodiment, a skip-connection fusion network is introduced to perform skip-connection fusion of the features corresponding to the encoder and decoder, thereby fusing multi-scale information and enriching the semantic learning of image features.

[0163] Figure 11 This is a structural block diagram of a three-dimensional medical image recognition device provided in an exemplary embodiment of this application, such as... Figure 11 As shown, the device includes:

[0164] The perspective rearrangement module 1101 is used to rearrange the perspective of the three-dimensional medical image features in the (i-1)th round during the i-th round of feature extraction to obtain two-dimensional image features. The (i-1)th round of three-dimensional medical image features are features obtained by performing the (i-1)th round of feature extraction on the three-dimensional medical image, and different two-dimensional image features are features of the (i-1)th round of three-dimensional medical image features under different perspectives.

[0165] Feature extraction module 1102 is used to extract semantic features from each of the two-dimensional image features to obtain image semantic features from different perspectives;

[0166] Feature fusion module 1103 is used to fuse the semantic features of the image from different perspectives to obtain the i-th round of three-dimensional medical image features;

[0167] The image recognition module 1104 is used to perform image recognition based on the three-dimensional medical image features obtained from the k-th round of feature extraction, and to obtain the image recognition result of the three-dimensional medical image, where k is greater than or equal to i.

[0168] Optionally, the feature extraction module 1102 includes:

[0169] The first extraction unit is used to extract spatial features from the two-dimensional image features to obtain two-dimensional image spatial features;

[0170] The second extraction unit is used to extract semantic features from the spatial features of the two-dimensional image based on the main viewpoint and the auxiliary viewpoint to obtain the semantic features of the image. The main viewpoint is the viewpoint corresponding to the two-dimensional image features, and the auxiliary viewpoint is a viewpoint in the three-dimensional viewpoint that is different from the main viewpoint.

[0171] Optionally, the first extraction unit is further configured to:

[0172] The two-dimensional image features are divided into windows to obtain local two-dimensional image features corresponding to N windows, and the N windows do not overlap.

[0173] Feature extraction is performed on N local two-dimensional image features to obtain two-dimensional image window features;

[0174] The N windows are rearranged, and the features of the two-dimensional image windows corresponding to the rearranged N windows are extracted to obtain two-dimensional image spatial features. The window rearrangement is used to change the spatial position of the N windows.

[0175] Optionally, the first extraction unit is further configured to:

[0176] Self-attention processing is performed on the N local two-dimensional image features to obtain self-attention features of the N local two-dimensional image features;

[0177] The self-attention features of the N local two-dimensional image features are fused to obtain the features inside the first image window;

[0178] The features inside the first image window are convolved to obtain the interactive features of the first image window.

[0179] The interactive features of the first image window are extracted using a multilayer perceptron (MLP) to obtain the two-dimensional image window features.

[0180] Optionally, the first extraction unit is further configured to:

[0181] Self-attention processing is performed on the two-dimensional image window features corresponding to the N windows after window rearrangement to obtain the self-attention features corresponding to the N windows;

[0182] The N self-attention features are fused to obtain the internal features of the second image window;

[0183] The internal features of the second image window are flipped, and the flipped internal features of the second image window are convolved to obtain the interactive features of the second image window.

[0184] The two-dimensional image spatial features are obtained by using MLP to extract the interactive features of the second image window.

[0185] Optionally, the first extraction unit is further configured to:

[0186] Self-attention processing is performed based on the query term Q, key term K, and value term V corresponding to the two-dimensional image features to obtain N self-attention features of the two-dimensional image window features.

[0187] Optionally, the feature extraction process includes a feature encoding process and a feature decoding process. In the t-th round of feature decoding, the K and V values ​​are obtained by fusing the K and V values ​​in the (t-1)-th round of feature decoding with the K and V values ​​in the corresponding feature encoding process. In the k-th round of decoding, the Q value is the Q value in the (t-1)-th round of feature decoding.

[0188] Optionally, the second extraction unit is further configured to:

[0189] The spatial features and positional coding features of the two-dimensional image are fused to obtain the first semantic features of the image, wherein the positional coding features are used to indicate the positional information corresponding to the two-dimensional image features.

[0190] From the main viewpoint, semantic features of the first image are extracted using MLP to obtain the semantic features of the main image.

[0191] Under the secondary perspective, the MLP is used to extract semantic features from the semantic features of the first image to obtain the semantic features of the secondary image.

[0192] The semantic features of the main image and the semantic features of the auxiliary image are fused to obtain the semantic features of the image.

[0193] Optionally, the feature fusion module 1103 further includes:

[0194] The first fusion unit is used to fuse the image semantic features and viewpoint features to obtain viewpoint image semantic features;

[0195] The second fusion unit is used to perform feature fusion on the semantic features of the images from each of the aforementioned perspectives to obtain the i-th round of three-dimensional medical image features.

[0196] Optionally, the feature extraction module 1102 is further configured to:

[0197] Semantic features of the two-dimensional image under different perspectives are extracted by using feature extraction networks with the same network parameters respectively.

[0198] Optionally, the feature extraction process includes a feature encoding process and a feature decoding process. The feature encoding process includes a downsampling process of the three-dimensional medical image features, and the feature decoding process includes an upsampling process of the three-dimensional medical image features.

[0199] The image recognition module 1104 further includes:

[0200] The determining unit is configured to determine the extracted three-dimensional medical image features as the three-dimensional medical image features obtained in the k-th round of feature extraction when the upsampling reaches the original size;

[0201] The recognition unit is used to perform image recognition based on the features of the k-th round of three-dimensional medical images to obtain the image recognition result.

[0202] Optional, the three-dimensional medical image is a CT image, MRI image, or PET image.

[0203] In summary, in this embodiment, in each feature extraction stage, the 3D medical image features are first rearranged by perspective, thus dividing them into 2D image features from different perspectives. Feature extraction is then performed on each of these 2D image features to obtain semantic features from different perspectives. These semantic features from different perspectives are then fused to obtain the extracted 3D image semantic features. In this process, because feature extraction is performed on 2D image features from different perspectives, compared to directly extracting 3D image features, the computational complexity is reduced. That is, feature extraction from different perspectives is performed using simplified local computational units, thereby improving the recognition efficiency of 3D medical images.

[0204] It should be noted that the apparatus provided in the above embodiments is only an example of the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the apparatus can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and their implementation process can be found in the method embodiments, which will not be repeated here.

[0205] Please refer to Figure 12 This illustration shows a schematic diagram of a computer device provided in an exemplary embodiment of this application. Specifically, the computer device 1200 includes a Central Processing Unit (CPU) 1201, a system memory 1204 including a random access memory 1202 and a read-only memory 1203, and a system bus 1205 connecting the system memory 1204 and the CPU 1201. The computer device 1200 also includes a basic input / output system (I / O system) 1206 to facilitate the transfer of information between various devices within the computer, and a mass storage device 1207 for storing an operating system 1213, application programs 1214, and other program modules 1215.

[0206] The basic input / output system 1206 includes a display 1208 for displaying information and an input device 1209 for user input, such as a mouse or keyboard. Both the display 1208 and the input device 1209 are connected to the central processing unit 1201 via an input / output controller 1210 connected to the system bus 1205. The basic input / output system 1206 may also include the input / output controller 1210 for receiving and processing input from multiple other devices such as a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 1210 also provides output to a display screen, printer, or other types of output devices.

[0207] The mass storage device 1207 is connected to the central processing unit 1201 via a mass storage controller (not shown) connected to the system bus 1205. The mass storage device 1207 and its associated computer-readable media provide non-volatile storage for the computer device 1200. That is, the mass storage device 1207 may include computer-readable media (not shown) such as a hard disk or drive.

[0208] Without loss of generality, the computer-readable medium may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include random access memory (RAM), read-only memory (ROM), flash memory or other solid-state storage technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic tape cassettes, magnetic tape, disk storage, or other magnetic storage devices. Of course, those skilled in the art will recognize that the computer storage media are not limited to the above-mentioned types. The system memory 1204 and mass storage device 1207 described above can be collectively referred to as memory.

[0209] The memory stores one or more programs, which are configured to be executed by one or more central processing units 1201. The one or more programs contain instructions for implementing the methods described above, and the central processing unit 1201 executes the one or more programs to implement the methods provided in the various method embodiments described above.

[0210] According to various embodiments of this application, the computer device 1200 can also be connected to a remote computer on a network, such as the Internet. That is, the computer device 1200 can be connected to a network 1212 via a network interface unit 1211 connected to the system bus 1205, or the network interface unit 1211 can be used to connect to other types of networks or remote computer systems (not shown).

[0211] The memory further includes one or more programs stored in the memory, and the one or more programs include steps performed by a computer device in the methods provided in the embodiments of this application.

[0212] This application also provides a computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, at least one program, code set, or instruction set is loaded and executed by a processor to implement the three-dimensional medical image recognition method described in any of the above embodiments.

[0213] This application provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the three-dimensional medical image recognition method provided above.

[0214] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. This program can be stored in a computer-readable storage medium, which may be a computer-readable storage medium included in the memory described in the above embodiments; or it may be a standalone computer-readable storage medium not assembled into a terminal. The computer-readable storage medium stores at least one instruction, at least one program segment, a code set, or an instruction set. The at least one instruction, the at least one program segment, the code set, or the instruction set is loaded and executed by a processor to implement the three-dimensional medical image recognition method described in any of the above method embodiments.

[0215] Optionally, the computer-readable storage medium may include ROM, RAM, solid-state drives (SSDs), or optical discs, etc. The RAM may include resistive random access memory (ReRAM) and dynamic random access memory (DRAM). The sequence numbers of the embodiments described above are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0216] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0217] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for recognizing three-dimensional medical images, characterized in that, The method includes: In the i-th round of feature extraction during the k-round feature extraction process, the three-dimensional medical image features of the (i-1)-th round are rearranged from different perspectives to obtain two-dimensional image features. The three-dimensional medical image features of the (i-1)-th round are the features obtained by extracting features from the three-dimensional medical image in the (i-1)-th round, and the different two-dimensional image features are the features of the three-dimensional medical image features of the (i-1)-th round under different perspectives, where i is less than or equal to k. Spatial feature extraction is performed on the two-dimensional image features to obtain the two-dimensional image spatial features; Semantic features are extracted from the spatial features of the two-dimensional image based on the primary and secondary perspectives to obtain image semantic features under different perspectives. The primary perspective is the perspective corresponding to the two-dimensional image features, and the secondary perspective is a perspective in the three-dimensional perspective that is different from the primary perspective. The semantic features of the images from different perspectives are fused to obtain the i-th round of three-dimensional medical image features; Image recognition is performed based on the three-dimensional medical image features obtained from the k-th round of feature extraction to obtain the image recognition result of the three-dimensional medical image.

2. The method according to claim 1, characterized in that, The step of extracting spatial features from the two-dimensional image features to obtain two-dimensional image spatial features includes: The two-dimensional image features are divided into windows to obtain local two-dimensional image features corresponding to N windows, and the N windows do not overlap. Feature extraction is performed on N local two-dimensional image features to obtain two-dimensional image window features; The N windows are rearranged, and the features of the two-dimensional image windows corresponding to the rearranged N windows are extracted to obtain two-dimensional image spatial features. The window rearrangement is used to change the spatial position of the N windows.

3. The method according to claim 2, characterized in that, The step of extracting features from the N local two-dimensional image features to obtain two-dimensional image window features includes: Self-attention processing is performed on the N local two-dimensional image features to obtain self-attention features of the N local two-dimensional image features; The self-attention features of the N local two-dimensional image features are fused to obtain the features inside the first image window; The features inside the first image window are convolved to obtain the interactive features of the first image window. The interactive features of the first image window are extracted using a multilayer perceptron (MLP) to obtain the two-dimensional image window features.

4. The method according to claim 2, characterized in that, The step of extracting features from the two-dimensional image window features corresponding to the N rearranged windows to obtain two-dimensional image spatial features includes: Self-attention processing is performed on the two-dimensional image window features corresponding to the N windows after window rearrangement to obtain the self-attention features corresponding to the N windows; The N self-attention features are fused to obtain the internal features of the second image window; The internal features of the second image window are flipped, and the flipped internal features of the second image window are convolved to obtain the interactive features of the second image window. The two-dimensional image spatial features are obtained by using MLP to extract the interactive features of the second image window.

5. The method according to claim 3, characterized in that, The step of performing self-attention processing on N local two-dimensional image features to obtain self-attention features of N local two-dimensional image features includes: Self-attention processing is performed based on the query term Q, key term K, and value term V corresponding to the two-dimensional image features to obtain N self-attention features of the two-dimensional image window features.

6. The method according to claim 5, characterized in that, The feature extraction process includes a feature encoding process and a feature decoding process. In the t-th round of feature decoding, the K and V values ​​are obtained by fusing the K and V values ​​in the (t-1)-th round of feature decoding with the K and V values ​​in the corresponding feature encoding process. In the t-th round of decoding, the Q value is the Q value in the (t-1)-th round of feature decoding, where t is less than or equal to k.

7. The method according to claim 1, characterized in that, The semantic feature extraction of the two-dimensional image spatial features based on the primary and secondary viewpoints yields image semantic features under different viewpoints, including: The spatial features and positional coding features of the two-dimensional image are fused to obtain the first semantic features of the image, wherein the positional coding features are used to indicate the positional information corresponding to the two-dimensional image features. From the main viewpoint, semantic features of the first image are extracted using MLP to obtain the semantic features of the main image. Under the secondary perspective, the MLP is used to extract semantic features from the semantic features of the first image to obtain the semantic features of the secondary image. The semantic features of the main image and the semantic features of the auxiliary image are fused to obtain the semantic features of the image.

8. The method according to any one of claims 1 to 7, characterized in that, The feature fusion of the semantic features of the images from different perspectives to obtain the i-th round of three-dimensional medical image features includes: The image semantic features and viewpoint features are fused to obtain the viewpoint image semantic features; The semantic features of each of the aforementioned viewpoint images are fused to obtain the i-th round of three-dimensional medical image features.

9. The method according to any one of claims 1 to 7, characterized in that, The semantic features of the images from different perspectives are obtained by using a feature extraction network with the same network parameters to extract semantic features from the two-dimensional images from each perspective.

10. The method according to any one of claims 1 to 7, characterized in that, The feature extraction process includes a feature encoding process and a feature decoding process. The feature encoding process includes a downsampling process of the three-dimensional medical image features, and the feature decoding process includes an upsampling process of the three-dimensional medical image features. The image recognition result of the three-dimensional medical image obtained by the k-th round of feature extraction includes: When the upsampling reaches the original size, the extracted three-dimensional medical image features are determined as the three-dimensional medical image features obtained in the k-th round of feature extraction; Image recognition is performed based on the three-dimensional medical image features of the k-th round to obtain the image recognition result.

11. The method according to any one of claims 1 to 7, characterized in that, Three-dimensional medical images are computed tomography (CT), magnetic resonance imaging (MRI), or positron emission tomography (PET).

12. A three-dimensional medical image recognition device, characterized in that, The device includes: The perspective rearrangement module is used to rearrange the perspective of the three-dimensional medical image features in the (i-1)th round of feature extraction during the k-round feature extraction process to obtain two-dimensional image features. The (i-1)th round of three-dimensional medical image features are the features obtained by performing the (i-1)th round of feature extraction on the three-dimensional medical image, and the different two-dimensional image features are the features of the (i-1)th round of three-dimensional medical image features under different perspectives, where i is less than or equal to k. The feature extraction module is used to extract spatial features from the two-dimensional image features to obtain two-dimensional image spatial features; and to extract semantic features from the two-dimensional image spatial features based on the main viewpoint and the auxiliary viewpoint to obtain image semantic features under different viewpoints, wherein the main viewpoint is the viewpoint corresponding to the two-dimensional image features, and the auxiliary viewpoint is a viewpoint in three-dimensional viewpoint that is different from the main viewpoint. The feature fusion module is used to fuse the semantic features of the image from different perspectives to obtain the i-th round of three-dimensional medical image features. The image recognition module is used to perform image recognition based on the three-dimensional medical image features obtained from the k-th round of feature extraction, and to obtain the image recognition result of the three-dimensional medical image.

13. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing at least one instruction, at least one program, a code set, or an instruction set, the at least one instruction, the at least one program, the code set, or the instruction set being loaded and executed by the processor to implement the three-dimensional medical image recognition method as described in any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that, The readable storage medium stores at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the three-dimensional medical image recognition method as described in any one of claims 1 to 11.

15. A computer program product, characterized in that, The computer program product includes computer instructions stored in a computer-readable storage medium, a processor of a computer device reading the computer instructions from the computer-readable storage medium, and the processor executing the computer instructions to implement the method for three-dimensional medical imaging as described in any one of claims 1 to 11.

Images