Method, device and equipment for hierarchical segmentation of tissue structures in medical images and medium

Abstract

The application discloses a hierarchical segmentation method, device and equipment of a tissue structure in a medical image and a medium, and relates to the field of artificial intelligence. The method comprises the following steps: performing feature extraction on each two-dimensional medical image contained in a three-dimensional medical image to obtain image features corresponding to the two-dimensional medical images, the three-dimensional medical image being obtained by continuously scanning a target tissue structure; determining the offset of each two-dimensional medical image in a target direction based on the image features; performing feature alignment on the image features based on the offset to obtain aligned image features; and performing three-dimensional segmentation on the three-dimensional medical image based on the aligned image features to obtain a three-dimensional hierarchical distribution of the target tissue structure in the three-dimensional medical image. The three-dimensional hierarchical distribution of the tissue structure is segmented from the three-dimensional medical image, and the image features are aligned before segmentation, which can eliminate the image offset caused by the movement of the target tissue structure during the scanning process and improve the accuracy of hierarchical segmentation.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the field of artificial intelligence, and in particular to a method and apparatus for hierarchical segmentation of tissue structures in medical images, devices and media. BACKGROUND

[0002] Medical scanning is a technology that uses a scanning instrument to scan tissue structures to obtain three-dimensional images of the tissue structures. Common medical scanning technologies include Computed Tomography, Optical Coherence Tomography (OCT), and the like.

[0003] By observing medical images obtained by medical scanning, medical personnel can analyze whether the tissue structures have abnormalities or lesions. For example, medical personnel can diagnose the lesion condition of the retinal layer based on an OCT image of the eye. In related technologies, when a computer device is used to perform hierarchical segmentation on an OCT image, each two-dimensional OCT image is segmented as an independent individual to obtain the hierarchical position in each two-dimensional OCT image.

[0004] However, since the hierarchical distribution of tissue structures in a three-dimensional space has continuity, the accuracy of the segmentation result obtained by two-dimensional hierarchical segmentation is poor, and the diagnostic information that can be provided is limited, resulting in low utilization of medical images. SUMMARY

[0005] Embodiments of the present application provide a method and apparatus for hierarchical segmentation of tissue structures in medical images, devices and media, which can segment the three-dimensional hierarchical distribution of tissue structures from a three-dimensional medical image, improve the accuracy of hierarchical segmentation, provide more diagnostic information, and improve the utilization of medical images. The technical solution is as follows:

[0006] In one aspect, the present application provides a method for hierarchical segmentation of tissue structures in medical images, the method comprising:

[0007] extracting features from each two-dimensional medical image contained in a three-dimensional medical image to obtain image features corresponding to the two-dimensional medical images, the three-dimensional medical image being obtained by continuously scanning a target tissue structure;

[0008] determining the offset of each two-dimensional medical image in a target direction based on the image features;

[0009] aligning the image features based on the offset to obtain aligned image features;

[0010] perform three-dimensional segmentation on the three-dimensional medical image based on the aligned image features, to obtain a three-dimensional hierarchical distribution of the target tissue structure in the three-dimensional medical image.

[0011] In another aspect, an embodiment of the present application provides a hierarchical segmentation method for a tissue structure in a medical image, the method comprising:

[0012] extracting features of each sample two-dimensional medical image contained in a sample three-dimensional medical image by a feature extraction network, to obtain sample image features corresponding to the sample two-dimensional medical images, the sample three-dimensional medical image being obtained by continuously scanning a sample tissue structure;

[0013] inputting the sample image features into an alignment network, to obtain sample offset amounts of each of the sample two-dimensional medical images in a target direction;

[0014] aligning the sample image features based on the sample offset amounts, to obtain aligned sample image features;

[0015] inputting the aligned sample image features into a segmentation network, to obtain a sample three-dimensional segmentation result corresponding to the sample three-dimensional medical image, the sample three-dimensional segmentation result being used to represent a hierarchical distribution of the sample tissue structure;

[0016] training the feature extraction network, the alignment network, and the segmentation network based on the sample offset amounts, the sample three-dimensional segmentation result, and a sample annotation.

[0017] In another aspect, an embodiment of the present application provides a hierarchical segmentation device for a tissue structure in a medical image, the device comprising:

[0018] a first extraction module configured to extract features of each two-dimensional medical image contained in a three-dimensional medical image, to obtain image features corresponding to the two-dimensional medical images, the three-dimensional medical image being obtained by continuously scanning a target tissue structure;

[0019] an offset determination module configured to determine offset amounts of each of the two-dimensional medical images in a target direction based on the image features;

[0020] a first alignment module configured to align the image features based on the offset amounts, to obtain aligned image features;

[0021] a first segmentation module configured to perform three-dimensional segmentation on the three-dimensional medical image based on the aligned image features, to obtain a three-dimensional hierarchical distribution of the target tissue structure in the three-dimensional medical image.

[0022] In another aspect, an embodiment of the present application provides a hierarchical segmentation device for tissue structure in a medical image, the device comprising:

[0023] a second extraction module configured to perform feature extraction on each sample two-dimensional medical image contained in a sample three-dimensional medical image by using a feature extraction network to obtain sample image features corresponding to the sample two-dimensional medical images, the sample three-dimensional medical image being obtained by continuously scanning a sample tissue structure;

[0024] an offset prediction module configured to input the sample image features into an alignment network to obtain sample offset amounts of each of the sample two-dimensional medical images in a target direction;

[0025] a second alignment module configured to perform feature alignment on the sample image features based on the sample offset amounts to obtain aligned sample image features;

[0026] a second segmentation module configured to input the aligned sample image features into a segmentation network to obtain a sample three-dimensional segmentation result corresponding to the sample three-dimensional medical image, the sample three-dimensional segmentation result being used to represent a hierarchical distribution of the sample tissue structure;

[0027] a training module configured to train the feature extraction network, the alignment network, and the segmentation network based on the sample offset amounts, the sample three-dimensional segmentation result, and sample annotations.

[0028] In another aspect, an embodiment of the present application provides a computer device, the computer device comprising a processor and a memory, the memory storing at least one instruction, the at least one instruction being loaded and executed by the processor to implement the method according to the above aspect.

[0029] In another aspect, an embodiment of the present application provides a computer readable storage medium, the readable storage medium storing at least one instruction, the at least one instruction being loaded and executed by a processor to implement the method according to the above aspect.

[0030] In another aspect, an embodiment of the present application provides a computer program product or a computer program, the computer program product or the computer program comprising computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to cause the computer device to perform the method according to the above aspect.

[0031] In this embodiment, after feature extraction of the two-dimensional medical image in the three-dimensional medical image, the offset of the two-dimensional medical image caused by the movement of the target tissue structure during continuous scanning is first determined based on the image features. Then, the image features are aligned based on the offset, and the three-dimensional medical image is segmented based on the aligned image features to obtain the hierarchical distribution of the target tissue structure in the three-dimensional medical image. By adopting the scheme provided by this embodiment, hierarchical recognition at the three-dimensional level can be achieved, and the three-dimensional hierarchical distribution of the tissue structure can be segmented from the three-dimensional medical image, providing more effective information for subsequent diagnosis and improving the utilization rate of medical images. Furthermore, feature alignment of the image features before three-dimensional segmentation can eliminate the image offset caused by the movement of the target tissue structure during scanning, thereby improving the accuracy of the hierarchical distribution obtained by segmentation. Attached Figure Description

[0032] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0033] Figure 1 This invention provides a structural diagram of a hierarchical segmentation model according to an exemplary embodiment of the present application.

[0034] Figure 2 A schematic diagram of an implementation environment provided by an exemplary embodiment of this application is shown;

[0035] Figure 3 A flowchart illustrating a hierarchical segmentation method for tissue structures in medical images provided in an exemplary embodiment of this application is shown.

[0036] Figure 4 A flowchart illustrating a hierarchical segmentation method for tissue structures in medical images provided by another exemplary embodiment of this application is shown;

[0037] Figure 5 This is a schematic diagram illustrating an implementation of the hierarchical segmentation process in an exemplary embodiment of this application;

[0038] Figure 6 A flowchart illustrating a hierarchical segmentation method for tissue structures in medical images provided by another exemplary embodiment of this application is shown;

[0039] Figure 7 A flowchart illustrating a hierarchical segmentation method for tissue structures in medical images provided by another exemplary embodiment of this application is shown;

[0040] Figure 8This is a schematic diagram illustrating an exemplary embodiment of the model training process in this application;

[0041] Figure 9 This is a schematic diagram illustrating an implementation of the model training process, as shown in another exemplary embodiment of this application.

[0042] Figure 10 This is a comparison chart of the alignment effects of B-scan images under different schemes;

[0043] Figure 11 This is a structural block diagram of a hierarchical segmentation device for tissue structures in medical images provided in an exemplary embodiment of this application;

[0044] Figure 12 This is a structural block diagram of a hierarchical segmentation device for tissue structures in medical images provided in an exemplary embodiment of this application;

[0045] Figure 13 A schematic diagram of the structure of a computer device provided in an exemplary embodiment of this application is shown. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0047] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0048] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0049] Computer vision (CV) is a science that studies how to enable machines to "see." More specifically, it refers to using cameras and computers to replace human eyes in recognizing and measuring targets, and then performing image processing to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0050] The method described in this application involves using computer vision technology to perform hierarchical segmentation of tissue structures in medical images, thereby determining the three-dimensional hierarchical structure of the tissue. For example, using the solution provided in this application, the three-dimensional hierarchical location of the retinal layer can be segmented based on OCT images of the human eyeball.

[0051] In one implementation, the computer device utilizes a hierarchical segmentation model to achieve hierarchical segmentation of the organizational structure, wherein the hierarchical segmentation model is a 2D-3D hybrid model. For example... Figure 1 The diagram illustrates the structure of a hierarchical segmentation model provided in an exemplary embodiment of this application. The hierarchical segmentation model consists of a feature extraction network 101 (2D network), an alignment network 102 (3D network), and a segmentation network 103 (3D network).

[0052] During the hierarchical segmentation process, the three-dimensional medical image 104 (composed of continuously scanned two-dimensional medical images 1041) is first input into the feature extraction network 101. The feature extraction network 101 extracts features from each two-dimensional medical image 1041 to obtain image features 105. Since the target tissue structure may move during continuous scanning, there may be relative offsets between the scanned two-dimensional medical images 1041. Therefore, after the image features 105 are extracted by the feature extraction network 101, three-dimensional hierarchical segmentation is not performed directly based on the image features 105. Instead, based on the image features 105, the offset of the two-dimensional medical images 1041 is predicted using the alignment network 102 to obtain the offset 106 of each two-dimensional medical image 1041. Based on the offset 106, the image features 105 are aligned. Then, the aligned image features 105 are input into the segmentation network 103, which performs three-dimensional hierarchical segmentation to finally obtain the three-dimensional hierarchical distribution 107 of the target tissue structure in the three-dimensional medical image 104.

[0053] Compared to manual hierarchical recognition, the aforementioned hierarchical segmentation model enables three-dimensional hierarchical recognition of three-dimensional medical images, yielding the three-dimensional hierarchical distribution of tissue structures. This not only improves the efficiency of hierarchical segmentation but also extracts more effective information from medical images, increasing their utilization rate. Furthermore, by predicting the offsets between images before three-dimensional segmentation, feature alignment can be performed based on these offsets, eliminating the impact of tissue structure movement during scanning and improving the accuracy of the obtained three-dimensional hierarchical distribution. This, in turn, enhances the accuracy of subsequent medical diagnoses based on the three-dimensional hierarchical distribution.

[0054] The hierarchical segmentation method for tissue structures in medical images provided in this application can be used to complete the hierarchical segmentation task of tissue structures, which can be human tissue structures, such as eyeballs, hearts, blood vessels, etc.; and the medical images used can be CT images, OCT images, etc. This application does not limit the types of medical images and tissue structures.

[0055] In one possible application scenario, the method provided in this application embodiment can be implemented as all or part of a medical image processing program. When using the medical image processing program, medical personnel only need to input the scanned three-dimensional medical image into the program, which can automatically perform hierarchical segmentation of the three-dimensional medical image and output the hierarchical distribution of tissue structures in the image. Subsequently, medical personnel can analyze whether there are abnormalities or lesions in the tissue structure based on this hierarchical distribution. Of course, in addition to outputting the hierarchical distribution of tissue structures, the medical image processing program can also output aligned three-dimensional medical images so that medical personnel can manually identify the hierarchical distribution of tissue structures based on these images.

[0056] In other potential application scenarios, the output of a medical image processing program can serve as input for downstream systems, which then perform further processing based on the hierarchical distribution. For example, the hierarchical distribution output of a medical image processing program can be input into an AI consultation system, which can then provide reference consultation results based on the hierarchical distribution and feed them back to the user.

[0057] In addition to the above-mentioned application scenarios, the method provided in this application embodiment can also be applied to other scenarios that require hierarchical segmentation of tissue structures based on images. This application embodiment does not limit the specific application scenarios.

[0058] Figure 2A schematic diagram of an implementation environment provided by an exemplary embodiment of this application is shown. This implementation environment includes a terminal 210 and a server 220. The terminal 210 and the server 220 communicate via a communication network. Optionally, the communication network can be a wired network or a wireless network, and the communication network can be at least one of a local area network (LAN), a metropolitan area network (MAN), and a wide area network (WAN).

[0059] Terminal 210 is an electronic device with medical image hierarchical segmentation requirements. This electronic device can be a smartphone, tablet computer, or personal computer, etc., and this embodiment does not limit it. Figure 2 The following explanation uses Terminal 210, a computer used by medical staff, as an example.

[0060] In some embodiments, the terminal 210 is equipped with an application that has a medical image hierarchical segmentation function. When it is necessary to perform hierarchical segmentation on the scanned three-dimensional medical image, the user inputs the three-dimensional medical image to be segmented into the application, thereby uploading the three-dimensional medical image to the server 220, which then performs hierarchical segmentation on the three-dimensional medical image and returns the segmentation result.

[0061] Server 220 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0062] In some embodiments, server 220 provides medical image hierarchical segmentation services to applications installed on terminal 210. Optionally, server 220 is configured with a hierarchical segmentation model, which consists of a feature extraction network 221, an alignment network 222, and a segmentation network 223. During hierarchical segmentation, server 220 inputs the three-dimensional medical images to be segmented into feature extraction network 221, which extracts features from each two-dimensional medical image. The alignment network 222 then aligns the extracted image features, and the segmentation network 223 uses the aligned image features to perform three-dimensional segmentation. Finally, the obtained three-dimensional hierarchical distribution is fed back to terminal 210. After receiving the three-dimensional hierarchical distribution, terminal 210 parses and displays the distribution through the application.

[0063] Of course, in other possible implementations, the hierarchical segmentation model can also be deployed on the terminal 210 side, and the terminal 210 can perform three-dimensional hierarchical segmentation locally without the need for the server 220. This embodiment does not limit this. For ease of description, the following embodiments use the hierarchical segmentation method of tissue structure in medical images executed by a computer device as an example.

[0064] Figure 3 A flowchart illustrating a hierarchical segmentation method for tissue structures in medical images provided in an exemplary embodiment of this application is shown. This embodiment describes the method using a computer device as an example, and the method includes the following steps.

[0065] Step 301: Extract features from each two-dimensional medical image contained in the three-dimensional medical image to obtain the image features corresponding to the two-dimensional medical image. The three-dimensional medical image is obtained by continuously scanning the target tissue structure.

[0066] In this embodiment of the application, the three-dimensional medical image is a collection of images composed of several consecutively scanned two-dimensional medical images, that is, different two-dimensional medical images are scanned images of different slices in the target tissue structure.

[0067] The target tissue structure is a multi-layered structure. The purpose of this application embodiment is to perform three-dimensional hierarchical segmentation of the target tissue structure and determine the position of different layers in the three-dimensional medical image.

[0068] Optionally, the target tissue structure can be human tissue structures such as the eyeball, heart, and blood vessels, and the three-dimensional medical image can be a (three-dimensional) OCT image, while the two-dimensional medical image is an image obtained by B-scan (lateral scanning).

[0069] In a given In this case, a (3D) OCT image can be written as a real-valued function V(x,y,z): Ω→R, where the x and y axes are rows and columns in a B-scan image (i.e., a 2D medical image), and the z-axis is orthogonal to the B-scan image. V can be viewed as an ordered set of all B-scan images in an OCT image: V(b)={I b}, where I b Φ→R represents the b-th B-scan image, and

[0070] N B It is the total number of B-Scans images in a (3D) OCT image.

[0071] The following embodiments use three-dimensional medical images as ocular OCT images and require hierarchical segmentation of the retinal layer in the ocular OCT images as examples for illustration, but this does not constitute a limitation.

[0072] In one possible implementation, the computer device extracts features from each of the two-dimensional medical images in the three-dimensional medical image, obtaining the image features corresponding to each two-dimensional medical image, which are called feature maps. The image features corresponding to a single two-dimensional medical image are used to characterize the two-dimensional features of that two-dimensional medical image, while the stacked image features corresponding to multiple two-dimensional medical images can be used to characterize the three-dimensional features of the three-dimensional medical image.

[0073] Step 302: Determine the offset of each two-dimensional medical image in the target direction based on image features.

[0074] Because two-dimensional medical images in three-dimensional medical imaging are obtained through multiple scans (i.e., each two-dimensional medical image is acquired individually), and the target tissue structure may move during the scanning process, misalignment may occur between adjacent two-dimensional medical images. For example, during the scanning of the eye, the eyeball may move, causing misalignment of the retinal layer in adjacent two-dimensional medical images.

[0075] When misalignment occurs between two-dimensional medical images, directly using the extracted image features for three-dimensional segmentation will affect the accuracy of the final three-dimensional segmentation result (misalignment occurs between consecutive positions at the same level). To improve the accuracy of subsequent three-dimensional segmentation, in this embodiment, the computer device needs to determine the offset of each two-dimensional medical image based on the extracted image features. This offset is the offset of the two-dimensional medical image in the target direction, which can be the main direction of movement of the target tissue structure.

[0076] In one possible implementation, when the two-dimensional medical image is a B-scan image, the target direction is the A-scan (vertical scan) direction, i.e., the y-axis direction (column direction) of the B-scan image. Illustratively, when the two-dimensional medical image is a B-scan image in an ocular OCT image, the retinal layer can be represented as S = {r b,a}, where a∈[1,N A ], N A It is the number of A-scans, r b,a That is, the position of the retinal surface on the a-scan in the b-th B-scan image.

[0077] Step 303: Align the image features based on the offset to obtain the aligned image features.

[0078] Furthermore, based on the determined offset, the computer device performs feature alignment on the extracted image features in the target direction, correcting the image feature misalignment caused by the displacement of the target tissue structure during the scanning process, and obtaining aligned image features.

[0079] In one possible implementation, the computer device aligns image features in the A-scan direction based on the offset.

[0080] Step 304: Perform three-dimensional segmentation on the three-dimensional medical image based on the aligned image features to obtain the three-dimensional hierarchical distribution of the target tissue structure in the three-dimensional medical image.

[0081] Furthermore, the computer device performs 3D segmentation based on the aligned image features to obtain the 3D hierarchical distribution of the target tissue structure in the 3D medical image. In one possible implementation, the computer device performs 3D segmentation based on image features to obtain the coordinates of the hierarchy of the target tissue structure in each 2D medical image, and further obtains the 3D hierarchical distribution of the target tissue structure in the 3D medical image (e.g., a 3D smooth surface of a specific hierarchy in the target tissue structure) based on the coordinates in each 2D medical image.

[0082] In one possible application scenario, a computer device extracts features from each B-scan image in an OCT image of the eye, determines the offset of each B-scan image based on the image features, and aligns the image features based on this offset. After complete feature alignment, the computer device performs 3D segmentation based on the image features to obtain the 2D hierarchical distribution of the retinal layer in each B-scan image. Then, the 2D hierarchical distributions from each B-scan image are stitched together to obtain the 3D hierarchical distribution of the retinal layer.

[0083] In summary, in this embodiment, after feature extraction of the two-dimensional medical image in the three-dimensional medical image, the offset of the two-dimensional medical image caused by the movement of the target tissue structure during continuous scanning is first determined based on the image features. Then, the image features are aligned based on this offset, and feature segmentation is performed on the three-dimensional medical image based on the aligned image features to obtain the hierarchical distribution of the target tissue structure in the three-dimensional medical image. Using the scheme provided in this embodiment, hierarchical recognition at the three-dimensional level can be achieved, and the three-dimensional hierarchical distribution of the tissue structure can be segmented from the three-dimensional medical image, providing more effective information for subsequent diagnosis and improving the utilization rate of medical images. Furthermore, feature alignment of the image features before three-dimensional segmentation can eliminate the image offset caused by the movement of the target tissue structure during scanning, thereby improving the accuracy of the obtained hierarchical distribution.

[0084] Because complete medical images are large, directly performing hierarchical segmentation on them would be computationally too computationally demanding. Therefore, before performing hierarchical segmentation, computers typically divide the complete medical image into several smaller 3D medical images and then perform hierarchical segmentation on each of these smaller images. Correspondingly, after obtaining the 3D hierarchical distribution from each 3D medical image through hierarchical segmentation, the computer needs to further stitch together the corresponding 3D hierarchical distributions based on the position of each 3D medical image within the complete medical image to obtain the complete 3D hierarchical distribution corresponding to the complete medical image.

[0085] In addition, before segmentation, the computer device can perform other preprocessing on the complete medical image, such as flattening the retinal layer in the OCT image. This embodiment does not limit this.

[0086] In some embodiments, the feature extraction process described above is performed by a feature extraction network, the offset determination process is performed by an alignment network, and the 3D segmentation process is performed by a segmentation network. The processing flow of each network is described below through exemplary embodiments.

[0087] Figure 4 A flowchart illustrating a hierarchical segmentation method for tissue structures in medical images provided in another exemplary embodiment of this application is shown. This embodiment describes the method using a computer device as an example, and the method includes the following steps.

[0088] Step 401: The two-dimensional medical image is subjected to feature extraction through a feature extraction network to obtain at least two layers of image features corresponding to the two-dimensional medical image. The feature extraction network is a two-dimensional convolutional neural network, and the image features of different layers are obtained by feature extraction from different two-dimensional convolutional layers in the feature extraction network.

[0089] In this embodiment, the computer device uses a pre-trained feature extraction network to extract features from each two-dimensional medical image (the feature extraction network is equivalent to an encoder, used to downsample and encode the two-dimensional medical image). This feature extraction network is a two-dimensional convolutional neural network consisting of at least two two-dimensional convolutional layers. That is, the two-dimensional medical image undergoes feature extraction step-by-step through at least two two-dimensional convolutional layers (processed with convolutional kernels, pooling, activation, etc.), progressively extracting features from lower-level to higher-level images. Furthermore, as feature extraction deepens, the extracted image features become increasingly abstract (i.e., higher-level image features are more abstract than lower-level image features).

[0090] Optionally, the feature extraction network can be obtained by improving an existing feature extraction network or by redesigning it. The embodiments of this application do not limit the specific network structure of the feature extraction network. Furthermore, the number of network parameters can be reduced by decreasing the number of channels in each convolutional layer of the feature extraction network.

[0091] Since the high-level image features output by the feature extraction network are too abstract and not conducive to subsequent feature decoding based on the high-level image features, in this embodiment of the application, the computer device needs to obtain the image features extracted by each two-dimensional convolutional layer. That is, in addition to obtaining the high-level image features output by the last two-dimensional convolutional layer, it is also necessary to obtain the low-level image features output by other two-dimensional convolutional layers.

[0092] Indicative, such as Figure 5 As shown, the feature extraction network 51 has four two-dimensional convolutional layers: a first feature extraction layer 511, a second feature extraction layer 512, a third feature extraction layer 513, and a fourth feature extraction layer 514. After the eye OCT image 52 is input into the feature extraction network 51, each B-scan image undergoes step-by-step feature extraction. Correspondingly, the computer device acquires the image features output by each two-dimensional convolutional layer, namely the first image feature output by the first feature extraction layer 511, the second image feature output by the second feature extraction layer 512, the third image feature output by the third feature extraction layer 513, and the fourth image feature output by the fourth feature extraction layer 514.

[0093] Step 402: Input the image features of each layer into the alignment network to obtain the offset vector output by the alignment network. The offset vector contains the offset of each two-dimensional medical image in the target direction. The alignment network is a three-dimensional convolutional neural network composed of three-dimensional convolutional layers, and the alignment network is skip-connected to the convolutional layers in the feature extraction network.

[0094] After feature extraction is completed, in order to further determine the misalignment between consecutive two-dimensional medical images in the three-dimensional medical image, the computer device inputs the extracted image features of each layer into a pre-trained alignment network, and the alignment network outputs an offset vector indicating the offset of each two-dimensional medical image in the target direction.

[0095] In some embodiments, the offset vector is a one-dimensional vector, and the number of data in the one-dimensional vector is consistent with the number of two-dimensional medical images in the three-dimensional medical image, that is, each data in the offset vector represents the offset of the corresponding two-dimensional medical image in the target direction.

[0096] Because the image features of continuous two-dimensional medical images are spatially continuous, unlike feature extraction networks which use two-dimensional convolutional layers, alignment networks use three-dimensional convolutional layers to perform three-dimensional convolution processing on image features, thereby improving the accuracy of the determined offset.

[0097] The alignment network acts as a decoder, used to decode image features. In some embodiments, the alignment network consists of at least two three-dimensional convolutional layers (which perform convolutional pooling activation processing on image features through three-dimensional convolutional kernels), and the image features are decoded step by step through at least two three-dimensional convolutional layers.

[0098] Furthermore, to avoid inaccurate offset prediction due to alignment using only high-level image features, in this embodiment, the convolutional layers in the alignment network and the feature extraction network use a skip-connection. That is, the low-level image features extracted by the two-dimensional convolutional layer in the feature extraction network are passed to the three-dimensional convolutional layer in the alignment network, thereby incorporating more low-level image semantic information into the feature decoding process and improving the feature decoding quality.

[0099] Indicative, such as Figure 5 As shown, the alignment network 53 comprises three 3D convolutional layers and a first output layer 534, namely a first alignment layer 531, a second alignment layer 532, and a third feature extraction layer 533. The first alignment layer 531 and the third feature extraction layer 513 are connected by a skip connection (the third image features are directly passed to the first alignment layer 531), the second alignment layer 532 and the second feature extraction layer 512 are connected by a skip connection (the second image features are directly passed to the second alignment layer 532), and the third alignment layer 533 and the first feature extraction layer 511 are connected by a skip connection (the first image features are directly passed to the third alignment layer 533). After a series of feature decoding steps, the computer device performs fully connected processing on the image features through the first output layer 534 to obtain the offset vector 535 corresponding to the 3D medical image.

[0100] Step 403: Align the features of each layer of the image based on the offset to obtain the aligned image features.

[0101] To avoid the problem of low accuracy in 3D segmentation due to using only high-level image features, in this embodiment, the image features extracted by the feature extraction network from each layer are used for 3D segmentation. Therefore, the computer device needs to use the determined offset to perform feature alignment on the image features of each layer, that is, to perform feature alignment on both high-level and low-level image features.

[0102] In one possible implementation, the hierarchical segmentation model includes a Spatial Transformer Module (STM). The STM generates a corresponding spatial transformation parameter based on the input image or image features (i.e., feature map), and then performs a global spatial transformation on the image or feature map according to the spatial transformation parameter. In this embodiment, after the computer device inputs the image features and offset vectors into the STM, the STM aligns the image features based on the offsets in the offset vectors, thereby eliminating misalignments between image features.

[0103] Since the offset determined by the alignment network is for the original two-dimensional medical image, and a downsampling process occurs during feature extraction (i.e., the feature scale of the image features differs from that of the two-dimensional medical image), the computer device needs to adjust the offset to match the feature scale of the image features during feature alignment. Optionally, this step may include the following steps:

[0104] First, based on the feature scale of the image features, the offset is adjusted to obtain the adjusted offset.

[0105] In one possible implementation, for each layer of image features, the computer device determines the scaling ratio of the offset based on the feature scale of the image feature and the image size of the two-dimensional medical image, thereby adjusting the offset based on the scaling ratio.

[0106] In an illustrative example, when the image size of a two-dimensional medical image is 32×32 and the corresponding offset is d, for a 16×16 image feature (i.e., the size of the feature map is 16×16), the offset corresponding to the image feature is d / 2; for an 8×8 image feature, the offset corresponding to the image feature is d / 4.

[0107] Second, based on the adjusted offset, the image features are aligned to obtain the aligned image features.

[0108] Furthermore, the computer device aligns the image features in the target direction based on the offset (adjusted) corresponding to each layer of image features, thereby obtaining aligned image features and eliminating image feature misalignment.

[0109] Indicative, such as Figure 5 As shown, the computer device uses STM 55 to perform feature alignment on the image features extracted by the feature extraction network 51.

[0110] Step 404: Input the aligned image features of each layer into the segmentation network to obtain the layer distribution probability output by the segmentation network. The layer distribution probability is used to represent the probability of each layer in the target tissue structure being located in the three-dimensional medical image. The segmentation network is a three-dimensional convolutional neural network composed of three-dimensional convolutional layers, and the segmentation network and the convolutional layers in the feature extraction network are skip connections.

[0111] Because the target tissue structure exhibits continuity in the position of two-dimensional medical images (e.g., the retinal layer forms a smooth three-dimensional surface in a two-dimensional medical image), two-dimensional segmentation based on image features will lose the continuity information of the target tissue structure between two-dimensional medical images, affecting the accuracy of hierarchical segmentation. To improve the accuracy of hierarchical segmentation, in this embodiment, the computer device performs three-dimensional segmentation based on the image features of each layer. That is, the hierarchical continuity of the target tissue structure in different two-dimensional medical images is considered during the segmentation process, thereby improving the accuracy of the obtained three-dimensional hierarchical distribution.

[0112] The segmentation network acts as a decoder, used to decode image features to obtain 3D segmentation results. In some embodiments, the segmentation network consists of at least two 3D convolutional layers (which perform convolutional pooling activation processing on image features through 3D convolutional kernels), and the image features are decoded step by step through at least two 3D convolutional layers.

[0113] Furthermore, to avoid inaccurate segmentation results due to using only high-level image features for segmentation, in this embodiment, the convolutional layers in the segmentation network and the feature extraction network adopt a skip-connection. That is, the low-level image features extracted by the two-dimensional convolutional layer in the feature extraction network are passed to the three-dimensional convolutional layer in the segmentation network after feature alignment, thereby incorporating more low-level image semantic information into the feature decoding process and improving the feature decoding quality.

[0114] Optionally, the segmentation network also includes an output layer, which can be a fully connected layer. After the three-dimensional convolutional layer processes the image features, the processing result is input into the output layer to obtain the hierarchical distribution probability.

[0115] Schematic, when the two-dimensional medical image is a B-scan image, the probability distribution of this hierarchy can be expressed as q. b,a (r|V;θ), where V represents an ordered set of two-dimensional medical images (i.e., three-dimensional medical images), θ is the network parameter of the segmentation network, b represents the b-th two-dimensional medical image, a represents the a-th A-scan column in the two-dimensional medical image, and r represents the pixel in the r-th row of the A-scan column. A higher probability value for the hierarchical distribution probability indicates a higher probability that the pixel in the r-th row of the A-scan is at the target hierarchical level.

[0116] Indicative, such asFigure 5 As shown, the segmentation network 54 comprises three 3D convolutional layers and a second output layer 544, namely a first segmentation layer 541, a second segmentation layer 542, and a third segmentation layer 543. The first segmentation layer 541 is skipped to the third feature extraction layer 513 (the third image features are directly passed to the first segmentation layer 541 after feature alignment), the second segmentation layer 542 is skipped to the second feature extraction layer 512 (the second image features are directly passed to the second segmentation layer 542 after feature alignment), and the third segmentation layer 543 is skipped to the first feature extraction layer 511 (the first image features are directly passed to the third segmentation layer 543 after feature alignment). After a series of feature decoding steps, the computer performs fully connected processing on the image features through the second output layer 544 to obtain the hierarchical distribution probability 545.

[0117] Step 405: Generate a three-dimensional hierarchical distribution of the target organizational structure based on the hierarchical distribution probability.

[0118] In one possible implementation, the computer device obtains the three-dimensional hierarchical distribution of the target organizational structure based on the hierarchical distribution probability through soft-argmax (combining the softmax function to achieve argmax). Referring to the example in the above steps, this three-dimensional hierarchical distribution can be represented as: Where R is the height of A-scan (i.e., the number of pixels in A-scan).

[0119] Indicative, such as Figure 5 As shown, based on the hierarchical distribution probability 545, the computer device finally generates the retinal hierarchical distribution 546 corresponding to the OCT image 52 of the eye.

[0120] In this embodiment, the computer device uses a 2D-3D hybrid model (i.e., a hierarchical segmentation model composed of a 2D feature extraction network, a 3D alignment network, and a 3D segmentation network) to perform three-dimensional hierarchical segmentation. This incorporates the continuity information of the target tissue structure in the two-dimensional medical image into the hierarchical segmentation process, which helps to improve the accuracy of three-dimensional hierarchical segmentation. Furthermore, before performing three-dimensional hierarchical segmentation, feature alignment is performed on the image features to eliminate the impact of image feature misalignment on the accuracy of hierarchical segmentation, further improving the accuracy of three-dimensional segmentation.

[0121] The above embodiments illustrate the application process of the hierarchical segmentation model. The following exemplary embodiments illustrate the model training process.

[0122] Figure 6 A flowchart illustrating a hierarchical segmentation method for tissue structures in medical images provided in another exemplary embodiment of this application is shown. This embodiment describes the method using a computer device as an example, and the method includes the following steps.

[0123] Step 601: The feature extraction network is used to extract features from each sample two-dimensional medical image contained in the sample three-dimensional medical image to obtain the sample image features corresponding to the sample two-dimensional medical image. The sample three-dimensional medical image is obtained by continuously scanning the sample tissue structure.

[0124] Optionally, the sample 3D medical image is obtained from a training dataset, which can be a publicly available dataset or a custom dataset based on requirements, and each sample 3D medical image corresponds to its own sample annotation (ground truth). In this embodiment, the sample annotation is used to indicate the hierarchical position of the sample tissue structure in the sample 3D medical image.

[0125] Taking the SD-OCT training dataset as an example, this training dataset includes OCT images of the eyes of 256 normal individuals and OCT images of the eyes of 115 patients with age-related macular degeneration. The OCT images are composed of several consecutive B-scan images, and the OCT images correspond to three manually labeled retinal layers: the interior of the inner boundary membrane (ILM), the interior of the retinal pigment epithelium drusen complex (IRPE), and the exterior of Bruchs' membrane (OBM).

[0126] The network structure of the feature extraction network and the process of feature extraction by the feature extraction network can be referred to the above application-side embodiment, and will not be repeated here.

[0127] Step 602: Input the sample image features into the alignment network to obtain the sample offset of each sample two-dimensional medical image in the target direction.

[0128] In one possible implementation, the computer device inputs sample image features into an alignment network to obtain a sample offset vector, which contains the sample offset of each sample two-dimensional medical image in the target direction.

[0129] Optionally, a skip connection is used between the convolutional layers of the alignment network and the feature extraction network. The sample image features output by the intermediate convolutional layers of the feature extraction network can be passed to the intermediate convolutional layers of the alignment network, thereby incorporating more low-level image semantic information and improving the accuracy of offset prediction.

[0130] The network structure of the alignment network and the process of offset prediction by the alignment network can be referred to the above application-side embodiment, and will not be repeated here.

[0131] Step 603: Align the features of the sample image based on the sample offset to obtain the aligned sample image features.

[0132] In one possible implementation, the computer device performs feature alignment on the sample image features output by each convolutional layer of the feature extraction network based on the sample offset. During feature alignment, the computer device needs to adjust the sample offset based on the feature scale of the sample image features, thereby using the adjusted sample offset for feature alignment. The feature alignment process can be referred to the application-side embodiment described above, and will not be repeated here.

[0133] Step 604: Input the aligned sample image features into the segmentation network to obtain the sample 3D segmentation result corresponding to the sample 3D medical image. The sample 3D segmentation result is used to characterize the hierarchical distribution of the sample tissue structure.

[0134] Optionally, skip connections are used between the convolutional layers of the segmentation network and the feature extraction network. The sample image features output by the intermediate convolutional layers of the feature extraction network can be passed to the intermediate convolutional layers of the segmentation network after feature pair alignment, thereby incorporating more low-level image semantic information into the hierarchical segmentation process and improving the accuracy of hierarchical segmentation.

[0135] The network structure of the segmentation network and the process of the segmentation network performing three-dimensional hierarchical segmentation can be referred to the above application-side embodiment, and will not be repeated here.

[0136] In some embodiments, the three-dimensional segmentation result of the sample is represented as a sample hierarchical distribution probability, which is used to represent the probability of each level in the sample tissue structure being located in the three-dimensional medical image of the sample.

[0137] Schematic illustration: When the sample two-dimensional medical image is a B-scan image, the hierarchical distribution probability of this sample can be expressed as q. b,a (r|V;θ), where V represents the ordered set of sample 2D medical images (i.e., sample 3D medical images), θ is the network parameter of the segmentation network, b represents the b-th sample 2D medical image, a represents the a-th A-scan column in the sample 2D medical image, and r represents the pixel in the r-th row of the A-scan column. A higher probability value for the sample hierarchical distribution probability indicates a higher probability that the pixel in the r-th row of the A-scan is at the target hierarchical level.

[0138] Step 605: Based on the sample offset, the sample 3D segmentation results, and the sample annotation, train the feature extraction network, the alignment network, and the segmentation network.

[0139] After obtaining the sample offset and the sample 3D segmentation result through the above steps, the computer device uses the sample annotation corresponding to the sample 3D medical image as supervision to determine the total loss of the hierarchical segmentation model. Then, the backpropagation algorithm is used to train the feature extraction network, alignment network, and segmentation network (i.e., adjust the network parameters) until the training completion condition is met.

[0140] In one possible implementation, the above training process is performed on the PyTorch (1.4.0) neural network framework. Furthermore, the Adam algorithm is used as the optimizer during network training, with 120 training epochs and an initial learning rate of 1e-3. During training, if the loss function does not decrease within ten epochs, the learning rate is halved.

[0141] In summary, in this embodiment, after feature extraction of the two-dimensional medical image in the three-dimensional medical image, the offset of the two-dimensional medical image caused by the movement of the target tissue structure during continuous scanning is first determined based on the image features. Then, the image features are aligned based on this offset, and feature segmentation is performed on the three-dimensional medical image based on the aligned image features to obtain the hierarchical distribution of the target tissue structure in the three-dimensional medical image. Using the scheme provided in this embodiment, hierarchical recognition at the three-dimensional level can be achieved, and the three-dimensional hierarchical distribution of the tissue structure can be segmented from the three-dimensional medical image, providing more effective information for subsequent diagnosis and improving the utilization rate of medical images. Furthermore, feature alignment of the image features before three-dimensional segmentation can eliminate the image offset caused by the movement of the target tissue structure during scanning, thereby improving the accuracy of the obtained hierarchical distribution.

[0142] In one possible implementation, the loss of the hierarchical segmentation model mainly consists of two parts: the alignment loss of the alignment network and the segmentation loss of the segmentation network. For example... Figure 7 As shown, step 605 above may include the following steps:

[0143] Step 605A: Determine the alignment loss of the alignment network based on the sample offset and sample annotation.

[0144] After image alignment, adjacent two-dimensional medical images possess similarity. Therefore, a computer device can determine the image alignment effect by performing image matching on adjacent two-dimensional medical images. Furthermore, for well-aligned two-dimensional medical images, the hierarchical positions of tissue structures in adjacent images are close. Therefore, the computer device can also determine the image alignment effect based on the labeled hierarchical positions in the aligned two-dimensional medical images. In one possible implementation, this step may include the following steps:

[0145] I. Image alignment of sample two-dimensional medical images based on sample offset.

[0146] Computer equipment adjusts each sample two-dimensional medical image in the target direction based on the sample offset predicted by the alignment network, thereby achieving image alignment.

[0147] II. Determine the standardized cross-correlation loss based on the aligned two-dimensional medical images of adjacent samples.

[0148] After image alignment, the computer extracts adjacent sample two-dimensional medical images and determines the local normalized cross-correlation (NCC) loss between the two images. Illustratively, the NCC loss can be expressed as:

[0149]

[0150] Among them, (b) i ,b j () represents adjacent two-dimensional medical images of samples. Let φ be a set of adjacent two-dimensional medical images, where φ represents the image space and p is a pixel in the image space. p represents the pixels in the aligned image. k n represents the area around the pixel. 2 Each pixel.

[0151] 3. The first smoothing loss is determined based on the distance between the same sample annotation points in the aligned two-dimensional medical images of adjacent samples.

[0152] In one possible implementation, when the sample two-dimensional medical image is a B-scan image, the computer device adjusts the sample annotation points on the same A-scan in two adjacent B-scan images based on the sample offset, thereby determining the distance between the two sample annotation points and obtaining a first smoothing loss. The smaller the first smoothing loss, the smoother the hierarchical structure of the target tissue in the aligned image, and the better the alignment effect.

[0153] Schematic, the first smoothing loss can be expressed as:

[0154]

[0155] Among them, (b) i ,b j () represents adjacent two-dimensional medical images of samples. It is a set of two-dimensional medical images composed of adjacent samples. Indicates the bth i The annotation of the hierarchical position in column a-scan of the sample two-dimensional medical image. Indicates the bth i The offset of a sample two-dimensional medical image. Indicates the bth j The annotation of the hierarchical position in column a-scan of the sample two-dimensional medical image. Indicates the bth j The offset of the sample two-dimensional medical image, N A This is the column number of the A-scan.

[0156] Fourth, the standardized cross-correlation loss and the first smoothing loss are determined as alignment loss.

[0157] Furthermore, the computer device uses the NCC loss and the first smoothing loss as the alignment loss of the alignment network, which can be expressed as:

[0158] Indicative, such as Figure 8 As shown, after the feature extraction network 81 extracts features from the sample three-dimensional medical image 82, the alignment network 83 predicts the sample offset 831 based on the sample image features, and the computer device determines the alignment loss based on the sample offset 831 and the sample annotation 86.

[0159] Step 605B: Based on the 3D segmentation results of the samples and the sample annotations, determine the first segmentation loss of the segmentation network.

[0160] In one possible implementation, the 3D segmentation result of the samples is the sample hierarchical distribution probability output by the segmentation network, and the computer device determines the first segmentation loss, which may include the following steps:

[0161] I. Based on the sample hierarchical distribution probability indicated by the sample 3D segmentation results, determine the cross-entropy loss. The sample hierarchical distribution probability is used to represent the probability of each level in the sample tissue structure being located in the sample 3D medical image.

[0162] Schematic, when the sample two-dimensional medical image is a B-scan image, the cross-entropy loss can be expressed as:

[0163]

[0164] in, The probability of the sample hierarchy. This represents the layer label in the a-th column of the A-scan within the b-th B-scan image, where r is the number of pixels in the A-scan and R is the row number of the A-scan. This is an indicator function; the function value is 1 when x is true and 0 when x is false.

[0165] 2. Generate sample hierarchical distribution based on sample hierarchical distribution probability; determine the first norm loss based on sample hierarchical distribution and the label hierarchical distribution indicated by sample labels.

[0166] In one possible implementation, the computer device generates a sample hierarchical distribution based on the probability of the sample hierarchical distribution, thereby determining the distribution difference between the sample hierarchical distribution and the labeled hierarchical distribution indicated by the sample labels, and obtaining the first norm loss (L1) between the two. This first norm loss is used to guide the predicted hierarchy to approach the true value (i.e., the labeled hierarchy).

[0167] Schematic, the first norm loss can be expressed as:

[0168]

[0169] in, Indicates the hierarchical distribution of the samples. Obtained via soft-argmax: This is an indicator function; the function value is 1 when x is true and 0 when x is false.

[0170] 3. Determine the second smoothing loss based on the positional difference between adjacent points in the sample hierarchical distribution.

[0171] In one possible implementation, by utilizing the prior information that the hierarchical structure of the sample has a smooth surface, the computer device can determine the positional difference between adjacent points in the hierarchical distribution of the sample, and determine a second smoothing loss by using the two-dimensional gradient of the approximate surface as a constraint on the positional difference. This second smoothing loss is used to improve the smoothness of the predicted hierarchy.

[0172] Schematic, the second smoothing loss can be expressed as:

[0173]

[0174] in, and denoted by and , respectively, the pixel spacing along the z-axis and x-axis in the sample 3D medical image, and S represents the sample hierarchical distribution.

[0175] Fourth, the cross-entropy loss, the first norm loss, and the second smoothing loss are determined as the first segmentation loss.

[0176] Furthermore, the computer device uses the cross-entropy loss, the first norm loss, and the second smoothing loss as the first segmentation loss of the segmentation network, which can be expressed as:

[0177] λ is a hyperparameter used to control the weights of the second smoothing loss.

[0178] In one possible implementation, for ocular OCT images, the λ values ​​corresponding to the three retinal layers ILM, IRPE, and OBM are set to 0, 0.3, and 0.5, respectively.

[0179] Indicative, such as Figure 8 As shown, the STM 85 inputs the feature-aligned sample image features into the segmentation network 84, obtaining the sample hierarchical distribution probability 841 output by the segmentation network 84. The computer device determines the cross-entropy loss based on the sample hierarchical distribution probability 841 and the sample annotation 86; further, the computer device generates a sample hierarchical distribution 842 based on the sample hierarchical distribution probability 841, and then determines the first norm loss and the second smoothing loss based on the sample hierarchical distribution 842 and the sample annotation 86.

[0180] Step 605C: Based on the alignment loss and the first segmentation loss, train the feature extraction network, the alignment network, and the segmentation network.

[0181] Furthermore, the computer device determines the total loss based on the alignment loss and the first segmentation loss, and then uses the total loss to train the extraction network, the alignment network, and the segmentation network.

[0182] In one possible implementation, the segmentation network includes a first network output head and a second network output head. The first network output head outputs the 3D segmentation results of the samples, and the second network output head outputs the hierarchical label of the level to which each pixel in the image belongs. The addition of the second network output head can provide the network with additional training tasks, thereby achieving better training results by utilizing pixel-level supervision information (the second network output head is not needed in the application phase).

[0183] Optionally, after the computer device inputs the aligned sample image features into the segmentation network, in addition to obtaining the sample 3D segmentation result, it will also obtain the sample pixel annotation result corresponding to the sample 3D medical image. Accordingly, during the training process, the computer device trains the feature extraction network, the alignment network, and the segmentation network based on the sample offset, the sample 3D segmentation result, the sample pixel annotation result, and the sample annotation.

[0184] The pixel annotation results of this sample can be represented using a layer map.

[0185] In determining the loss, in addition to determining the alignment loss and the first segmentation loss in the above embodiments, the computer device also determines the second segmentation loss of the segmentation network based on the sample pixel annotation results and sample annotations. In one possible implementation, the computer device uses the Dice+ cross-entropy loss. As the second segmentation loss.

[0186] Indicative, in Figure 8 On the basis of, such as Figure 9 As shown, the second network output head of the segmentation network 84 outputs sample pixel labeling results 843, and the computer device determines the second segmentation loss based on the sample pixel labeling results 843 and sample labeling 86.

[0187] The hierarchical segmentation model trained using the scheme provided in the above embodiments can be applied to the three-dimensional segmentation of the retinal layer in OCT images of the eye, thereby improving the smoothness of the segmented retinal layer. For example... Figure 10 As shown, it presents a comparison chart of the B-scan image alignment effects between the relevant technical solutions and the solutions provided in this application. Table 1 provides a quantitative analysis of the alignment results from two dimensions: MAD (Mean Absolute Distance) and NCC (Normalized Cross-Correlation) (the lower the value, the better the alignment effect).

[0188] Table 1

[0189]

[0190] from Figure 10 As can be clearly seen from Table 1, the solution provided in the embodiments of this application can significantly improve the alignment effect of B-scan images.

[0191] Table 2 shows a comparison of segmentation results for various schemes on the SD-OCT training dataset. The evaluation metric chosen is the mean absolute distance (MSD), and the variance is also given.

[0192] Table 2

[0193]

[0194] As shown in Table 2, the experiment without any alignment algorithm yielded the worst results, indicating that misalignment between B-scans negatively impacts the 3D segmentation of OCT data. Furthermore, the addition of the alignment network improves the performance of the proposed solution compared to the pre-aligned approach. This is because the alignment results of the alignment network vary slightly in each round during network training, which acts as data augmentation for the segmentation network, thus achieving better results. Simultaneously, the removal of the smoothing loss reduces the effectiveness of the proposed solution, demonstrating the effectiveness of utilizing the prior information of retinal surface smoothing.

[0195] Figure 11 This is a structural block diagram of a hierarchical segmentation device for tissue structures in medical images provided in an exemplary embodiment of this application. The device includes:

[0196] The first extraction module 1101 is used to extract features from each two-dimensional medical image contained in the three-dimensional medical image to obtain the image features corresponding to the two-dimensional medical image. The three-dimensional medical image is obtained by continuously scanning the target tissue structure.

[0197] The offset determination module 1102 is used to determine the offset of each of the two-dimensional medical images in the target direction based on the image features;

[0198] The first alignment module 1103 is used to perform feature alignment on the image features based on the offset to obtain the aligned image features;

[0199] The first segmentation module 1104 is used to perform three-dimensional segmentation on the three-dimensional medical image based on the aligned image features to obtain the three-dimensional hierarchical distribution of the target tissue structure in the three-dimensional medical image.

[0200] Optionally, the first extraction module 1101 is used for:

[0201] The two-dimensional medical image is subjected to feature extraction by a feature extraction network to obtain at least two layers of image features corresponding to the two-dimensional medical image. The feature extraction network is a two-dimensional convolutional neural network, and the image features of different layers are obtained by feature extraction from different two-dimensional convolutional layers in the feature extraction network.

[0202] The first alignment module 1103 is used for:

[0203] Based on the offset, the image features of each layer are aligned to obtain the aligned image features.

[0204] Optionally, the first alignment module 1103 is specifically used for:

[0205] Based on the feature scale of the image features, the offset is adjusted to obtain the adjusted offset;

[0206] Based on the adjusted offset, the image features are aligned to obtain the aligned image features.

[0207] Optionally, the offset determination module 1102 is used for:

[0208] The image features of each layer are input into the alignment network to obtain the offset vector output by the alignment network. The offset vector contains the offset of each two-dimensional medical image in the target direction. The alignment network is a three-dimensional convolutional neural network composed of three-dimensional convolutional layers, and the alignment network is skip-connected to the convolutional layers in the feature extraction network.

[0209] Optionally, the first segmentation module 1104 is used for:

[0210] The aligned image features of each layer are input into the segmentation network to obtain the hierarchical distribution probability output by the segmentation network. The hierarchical distribution probability is used to represent the probability of each layer in the target tissue structure being located in the three-dimensional medical image. The segmentation network is a three-dimensional convolutional neural network composed of three-dimensional convolutional layers, and the segmentation network is skip-connected to the convolutional layers in the feature extraction network.

[0211] The three-dimensional hierarchical distribution of the target organizational structure is generated based on the hierarchical distribution probability.

[0212] Optionally, the three-dimensional medical image is obtained by segmenting a complete medical image;

[0213] The device further includes:

[0214] The stitching module is used to stitch together the three-dimensional hierarchical distributions corresponding to each of the three-dimensional medical images based on the position of the three-dimensional medical image in the complete medical image, so as to obtain the complete three-dimensional hierarchical distribution corresponding to the complete medical image.

[0215] Optionally, the three-dimensional medical image is an optical coherence tomography (OCT) image, the two-dimensional medical image is obtained by transverse scanning (B-scan), and the target direction is the longitudinal scanning (A-scan) direction.

[0216] In summary, in this embodiment, after feature extraction of the two-dimensional medical image in the three-dimensional medical image, the offset of the two-dimensional medical image caused by the movement of the target tissue structure during continuous scanning is first determined based on the image features. Then, the image features are aligned based on this offset, and feature segmentation is performed on the three-dimensional medical image based on the aligned image features to obtain the hierarchical distribution of the target tissue structure in the three-dimensional medical image. Using the scheme provided in this embodiment, hierarchical recognition at the three-dimensional level can be achieved, and the three-dimensional hierarchical distribution of the tissue structure can be segmented from the three-dimensional medical image, providing more effective information for subsequent diagnosis and improving the utilization rate of medical images. Furthermore, feature alignment of the image features before three-dimensional segmentation can eliminate the image offset caused by the movement of the target tissue structure during scanning, thereby improving the accuracy of the obtained hierarchical distribution.

[0217] Figure 12 This is a structural block diagram of a hierarchical segmentation device for tissue structures in medical images provided in an exemplary embodiment of this application. The device includes:

[0218] The second extraction module 1201 is used to extract features from each sample two-dimensional medical image contained in the sample three-dimensional medical image through a feature extraction network to obtain the sample image features corresponding to the sample two-dimensional medical image. The sample three-dimensional medical image is obtained by continuously scanning the sample tissue structure.

[0219] The offset prediction module 1202 is used to input the sample image features into the alignment network to obtain the sample offset of each sample two-dimensional medical image in the target direction;

[0220] The second alignment module 1203 is used to perform feature alignment on the sample image features based on the sample offset to obtain the aligned sample image features.

[0221] The second segmentation module 1204 is used to input the aligned sample image features into the segmentation network to obtain the sample three-dimensional segmentation result corresponding to the sample three-dimensional medical image. The sample three-dimensional segmentation result is used to characterize the hierarchical distribution of the sample tissue structure.

[0222] The training module 1205 is used to train the feature extraction network, the alignment network, and the segmentation network based on the sample offset, the sample 3D segmentation result, and the sample annotation.

[0223] Optionally, the training module 1205 includes:

[0224] The first loss determination unit is used to determine the alignment loss of the alignment network based on the sample offset and the sample label.

[0225] The second loss determination unit is used to determine the first segmentation loss of the segmentation network based on the sample 3D segmentation result and the sample annotation.

[0226] The training unit is used to train the feature extraction network, the alignment network, and the segmentation network based on the alignment loss and the first segmentation loss.

[0227] Optionally, the first loss determination unit is configured to:

[0228] Image alignment is performed on the two-dimensional medical image of the sample based on the sample offset;

[0229] Based on the aligned adjacent two-dimensional medical images of the samples, the standardized cross-correlation loss is determined;

[0230] The first smoothing loss is determined based on the distance between the same sample annotation points in the aligned two-dimensional medical images of adjacent samples.

[0231] The standardized cross-correlation loss and the first smoothing loss are determined as the alignment loss.

[0232] Optionally, the second loss determination unit is used for:

[0233] Based on the sample hierarchical distribution probability indicated by the three-dimensional segmentation result of the sample, the cross-entropy loss is determined. The sample hierarchical distribution probability is used to represent the probability of each level in the tissue structure of the sample being located in the three-dimensional medical image of the sample.

[0234] A sample hierarchical distribution is generated based on the probability of the sample hierarchical distribution; a first norm loss is determined based on the sample hierarchical distribution and the label hierarchical distribution indicated by the sample labels.

[0235] The second smoothing loss is determined based on the positional difference between adjacent points in the sample hierarchical distribution;

[0236] The cross-entropy loss, the first norm loss, and the second smoothing loss are determined as the first segmentation loss.

[0237] Optionally, the segmentation network includes a first network output head and a second network output head. The first network output head is used to output the three-dimensional segmentation result of the sample, and the second network output head is used to output the hierarchical label of the level to which each pixel in the image belongs.

[0238] The device further includes:

[0239] The third segmentation model is used to input the aligned features of the sample image into the segmentation network to obtain the sample pixel annotation results corresponding to the three-dimensional medical image of the sample.

[0240] The training module 1205 is also used for:

[0241] The feature extraction network, the alignment network, and the segmentation network are trained based on the sample offset, the sample 3D segmentation result, the sample pixel annotation result, and the sample annotation.

[0242] Optionally, the training module 1205 includes:

[0243] The first loss determination unit is used to determine the alignment loss of the alignment network based on the sample offset and the sample label.

[0244] The second loss determination unit is used to determine the first segmentation loss of the segmentation network based on the sample 3D segmentation result and the sample annotation.

[0245] The third loss determination unit is used to determine the second segmentation loss of the segmentation network based on the sample pixel annotation results and the sample annotations.

[0246] The training unit is used to train the feature extraction network, the alignment network, and the segmentation network based on the alignment loss, the first segmentation loss, and the second segmentation loss.

[0247] In summary, in this embodiment, after feature extraction of the two-dimensional medical image in the three-dimensional medical image, the offset of the two-dimensional medical image caused by the movement of the target tissue structure during continuous scanning is first determined based on the image features. Then, the image features are aligned based on this offset, and feature segmentation is performed on the three-dimensional medical image based on the aligned image features to obtain the hierarchical distribution of the target tissue structure in the three-dimensional medical image. Using the scheme provided in this embodiment, hierarchical recognition at the three-dimensional level can be achieved, and the three-dimensional hierarchical distribution of the tissue structure can be segmented from the three-dimensional medical image, providing more effective information for subsequent diagnosis and improving the utilization rate of medical images. Furthermore, feature alignment of the image features before three-dimensional segmentation can eliminate the image offset caused by the movement of the target tissue structure during scanning, thereby improving the accuracy of the obtained hierarchical distribution.

[0248] It should be noted that the apparatus provided in the above embodiments is only an example of the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the apparatus can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and their implementation process can be found in the method embodiments, which will not be repeated here.

[0249] Please refer to Figure 13 This illustration shows a schematic diagram of the structure of a computer device provided in an exemplary embodiment of this application. Specifically, the computer device 1300 includes a Central Processing Unit (CPU) 1301, a system memory 1304 including a random access memory 1302 and a read-only memory 1303, and a system bus 1305 connecting the system memory 1304 and the CPU 1301. The computer device 1300 also includes a basic input / output system (I / O system) 1306 that facilitates the transfer of information between various devices within the computer, and a mass storage device 1307 for storing the operating system 1313, application programs 1314, and other program modules 1315.

[0250] The basic input / output system 1306 includes a display 1308 for displaying information and an input device 1309 for user input, such as a mouse or keyboard. Both the display 1308 and the input device 1309 are connected to the central processing unit 1301 via an input / output controller 1310 connected to the system bus 1305. The basic input / output system 1306 may also include the input / output controller 1310 for receiving and processing input from multiple other devices such as a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 1310 also provides output to a display screen, printer, or other types of output devices.

[0251] The mass storage device 1307 is connected to the central processing unit 1301 via a mass storage controller (not shown) connected to the system bus 1305. The mass storage device 1307 and its associated computer-readable media provide non-volatile storage for the computer device 1300. That is, the mass storage device 1307 may include computer-readable media (not shown) such as a hard disk or drive.

[0252] Without loss of generality, the computer-readable medium may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include random access memory (RAM), read-only memory (ROM), flash memory or other solid-state storage technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic tape cassettes, magnetic tape, disk storage, or other magnetic storage devices. Of course, those skilled in the art will recognize that the computer storage media are not limited to the above-mentioned types. The system memory 1304 and mass storage device 1307 described above can be collectively referred to as memory.

[0253] The memory stores one or more programs, which are configured to be executed by one or more central processing units 1301. The one or more programs contain instructions for implementing the methods described above, and the central processing unit 1301 executes the one or more programs to implement the methods provided in the various method embodiments described above.

[0254] According to various embodiments of this application, the computer device 1300 can also be connected to a remote computer on a network, such as the Internet. That is, the computer device 1300 can be connected to a network 1312 via a network interface unit 1311 connected to the system bus 1305, or the network interface unit 1311 can be used to connect to other types of networks or remote computer systems (not shown).

[0255] The memory further includes one or more programs stored in the memory, and the one or more programs include steps performed by a computer device in the methods provided in the embodiments of this application.

[0256] This application also provides a computer-readable storage medium storing at least one instruction, which is loaded and executed by a processor to implement the hierarchical segmentation method for tissue structures in medical images as described in any of the above embodiments.

[0257] Optionally, the computer-readable storage medium may include ROM, RAM, solid-state drives (SSDs), or optical discs, etc. The RAM may include resistive random access memory (ReRAM) and dynamic random access memory (DRAM).

[0258] This application provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the hierarchical segmentation method for tissue structures in medical images described in the above embodiments.

[0259] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0260] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A hierarchical segmentation method for tissue structures in medical images, characterized in that, The method includes: The feature extraction network extracts features from each two-dimensional medical image contained in the three-dimensional medical image to obtain the image features corresponding to the two-dimensional medical image. The three-dimensional medical image is obtained by continuously scanning the target tissue structure. The image features are input into an alignment network to obtain the offset of each of the two-dimensional medical images in the target direction; Based on the offset, the image features are aligned to obtain the aligned image features; The aligned image features are input into a segmentation network for 3D segmentation to obtain the 3D hierarchical distribution of the target tissue structure in the 3D medical image. The feature extraction network, the alignment network, and the segmentation network are trained based on sample offsets, sample 3D segmentation results, and sample annotations. The sample offsets are obtained by the alignment network based on sample image features. The sample image features are obtained by the feature extraction network from each sample 2D medical image contained in the sample 3D medical image. The sample 3D segmentation results are obtained by the segmentation network based on the aligned sample image features.

2. The method according to claim 1, characterized in that, The step of extracting features from each two-dimensional medical image contained in the three-dimensional medical image using a feature extraction network to obtain the image features corresponding to the two-dimensional medical image includes: The feature extraction network is used to extract features from the two-dimensional medical image to obtain at least two layers of image features corresponding to the two-dimensional medical image. The feature extraction network is a two-dimensional convolutional neural network, and the image features of different layers are obtained by feature extraction from different two-dimensional convolutional layers in the feature extraction network. The step of aligning the image features based on the offset to obtain the aligned image features includes: Based on the offset, the image features of each layer are aligned to obtain the aligned image features.

3. The method according to claim 2, characterized in that, The step of aligning the image features of each layer based on the offset to obtain the aligned image features includes: Based on the feature scale of the image features, the offset is adjusted to obtain the adjusted offset; Based on the adjusted offset, the image features are aligned to obtain the aligned image features.

4. The method according to claim 2, characterized in that, The step of inputting the image features into the alignment network to obtain the offset of each two-dimensional medical image in the target direction includes: The image features of each layer are input into the alignment network to obtain the offset vector output by the alignment network. The offset vector contains the offset of each two-dimensional medical image in the target direction. The alignment network is a three-dimensional convolutional neural network composed of three-dimensional convolutional layers, and the alignment network is skip-connected to the convolutional layers in the feature extraction network.

5. The method according to claim 2, characterized in that, The step of inputting the aligned image features into a segmentation network for 3D segmentation to obtain the 3D hierarchical distribution of the target tissue structure in the 3D medical image includes: The aligned image features of each layer are input into the segmentation network to obtain the hierarchical distribution probability output by the segmentation network. The hierarchical distribution probability is used to represent the probability of each layer in the target tissue structure being located in the three-dimensional medical image. The segmentation network is a three-dimensional convolutional neural network composed of three-dimensional convolutional layers, and the segmentation network is skip-connected to the convolutional layers in the feature extraction network. The three-dimensional hierarchical distribution of the target organizational structure is generated based on the hierarchical distribution probability.

6. The method according to any one of claims 1 to 5, characterized in that, The three-dimensional medical image is obtained by dividing a complete medical image; After inputting the aligned image features into a segmentation network for 3D segmentation to obtain the 3D hierarchical distribution of the target tissue structure in the 3D medical image, the method further includes: Based on the position of the three-dimensional medical image in the complete medical image, the three-dimensional hierarchical distributions corresponding to each of the three-dimensional medical images are stitched together to obtain the complete three-dimensional hierarchical distribution corresponding to the complete medical image.

7. The method according to any one of claims 1 to 5, characterized in that, The three-dimensional medical image is an optical coherence tomography (OCT) image, the two-dimensional medical image is obtained by transverse scanning, and the target direction is the longitudinal scanning direction.

8. A hierarchical segmentation method for tissue structures in medical images, characterized in that, The method includes: The feature extraction network is used to extract features from each sample two-dimensional medical image contained in the sample three-dimensional medical image to obtain the sample image features corresponding to the sample two-dimensional medical image. The sample three-dimensional medical image is obtained by continuously scanning the sample tissue structure. The sample image features are input into an alignment network to obtain the sample offset of each sample two-dimensional medical image in the target direction; Based on the sample offset, the sample image features are aligned to obtain the aligned sample image features; The aligned sample image features are input into a segmentation network to obtain the sample three-dimensional segmentation result corresponding to the sample three-dimensional medical image. The sample three-dimensional segmentation result is used to characterize the hierarchical distribution of the sample tissue structure. Based on the sample offset, the sample 3D segmentation result, and the sample annotation, the feature extraction network, the alignment network, and the segmentation network are trained.

9. The method according to claim 8, characterized in that, The step of training the feature extraction network, the alignment network, and the segmentation network based on the sample offset, the sample 3D segmentation result, and the sample annotation includes: Based on the sample offset and the sample label, the alignment loss of the alignment network is determined; Based on the 3D segmentation results of the samples and the sample annotations, the first segmentation loss of the segmentation network is determined; The feature extraction network, the alignment network, and the segmentation network are trained based on the alignment loss and the first segmentation loss.

10. The method according to claim 9, characterized in that, The step of determining the alignment loss of the alignment network based on the sample offset and the sample label includes: Image alignment is performed on the two-dimensional medical image of the sample based on the sample offset; Based on the aligned adjacent two-dimensional medical images of the samples, the standardized cross-correlation loss is determined; The first smoothing loss is determined based on the distance between the same sample annotation points in the aligned two-dimensional medical images of adjacent samples. The standardized cross-correlation loss and the first smoothing loss are determined as the alignment loss.

11. The method according to claim 9, characterized in that, The step of determining the first segmentation loss of the segmentation network based on the sample 3D segmentation results and the sample annotations includes: Based on the sample hierarchical distribution probability indicated by the three-dimensional segmentation result of the sample, the cross-entropy loss is determined. The sample hierarchical distribution probability is used to represent the probability of each level in the tissue structure of the sample being located in the three-dimensional medical image of the sample. A sample hierarchical distribution is generated based on the probability of the sample hierarchical distribution; a first norm loss is determined based on the sample hierarchical distribution and the label hierarchical distribution indicated by the sample labels. The second smoothing loss is determined based on the positional difference between adjacent points in the sample hierarchical distribution; The cross-entropy loss, the first norm loss, and the second smoothing loss are determined as the first segmentation loss.

12. The method according to claim 8, characterized in that, The segmentation network includes a first network output head and a second network output head. The first network output head is used to output the three-dimensional segmentation result of the sample, and the second network output head is used to output the layer label of each pixel in the image. The method further includes: The aligned sample image features are input into the segmentation network to obtain the sample pixel annotation results corresponding to the sample three-dimensional medical image; The step of training the feature extraction network, the alignment network, and the segmentation network based on the sample offset, the sample 3D segmentation result, and the sample annotation includes: The feature extraction network, the alignment network, and the segmentation network are trained based on the sample offset, the sample 3D segmentation result, the sample pixel annotation result, and the sample annotation.

13. The method according to claim 12, characterized in that, The step of training the feature extraction network, the alignment network, and the segmentation network based on the sample offset, the sample 3D segmentation result, the sample pixel annotation result, and the sample annotation includes: Based on the sample offset and the sample label, the alignment loss of the alignment network is determined; Based on the 3D segmentation results of the samples and the sample annotations, the first segmentation loss of the segmentation network is determined; Based on the sample pixel annotation results and the sample annotations, the second segmentation loss of the segmentation network is determined; The feature extraction network, the alignment network, and the segmentation network are trained based on the alignment loss, the first segmentation loss, and the second segmentation loss.

14. A hierarchical segmentation device for tissue structures in medical images, characterized in that, The device includes: The first extraction module is used to extract features from each two-dimensional medical image contained in the three-dimensional medical image through a feature extraction network to obtain the image features corresponding to the two-dimensional medical image. The three-dimensional medical image is obtained by continuously scanning the target tissue structure. An offset determination module is used to input the image features into an alignment network to obtain the offset of each of the two-dimensional medical images in the target direction; The first alignment module is used to perform feature alignment on the image features based on the offset to obtain the aligned image features; The first segmentation module is used to input the aligned image features into a segmentation network for three-dimensional segmentation to obtain the three-dimensional hierarchical distribution of the target tissue structure in the three-dimensional medical image. The feature extraction network, the alignment network, and the segmentation network are trained based on sample offsets, sample three-dimensional segmentation results, and sample annotations. The sample offsets are obtained by the alignment network based on sample image features. The sample image features are obtained by the feature extraction network from each sample two-dimensional medical image contained in the sample three-dimensional medical image. The sample three-dimensional segmentation results are obtained by the segmentation network based on the aligned sample image features.

15. A hierarchical segmentation device for tissue structures in medical images, characterized in that, The device includes: The second extraction module is used to extract features from each sample two-dimensional medical image contained in the sample three-dimensional medical image through a feature extraction network to obtain the sample image features corresponding to the sample two-dimensional medical image. The sample three-dimensional medical image is obtained by continuously scanning the sample tissue structure. The offset prediction module is used to input the sample image features into the alignment network to obtain the sample offset of each sample two-dimensional medical image in the target direction; The second alignment module is used to perform feature alignment on the sample image features based on the sample offset to obtain the aligned sample image features. The second segmentation module is used to input the aligned features of the sample image into the segmentation network to obtain the sample three-dimensional segmentation result corresponding to the sample three-dimensional medical image. The sample three-dimensional segmentation result is used to characterize the hierarchical distribution of the sample tissue structure. The training module is used to train the feature extraction network, the alignment network, and the segmentation network based on the sample offset, the sample 3D segmentation result, and the sample annotation.

16. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing at least one instruction, the at least one instruction being loaded and executed by the processor to implement the method as claimed in any one of claims 1 to 7, or to implement the method as claimed in any one of claims 8 to 13.

17. A computer-readable storage medium, characterized in that, The readable storage medium stores at least one instruction, which is loaded and executed by a processor to implement the method as claimed in any one of claims 1 to 7, or to implement the method as claimed in any one of claims 8 to 13.

Images