Image segmentation method and device based on deep learning, equipment and storage medium

Abstract

The invention discloses an image segmentation method and device based on deep learning, equipment and a storage medium. The method comprises the steps of obtaining a target spine image; performing preliminary segmentation on the target spine image based on the coarse segmentation network to obtain a first mask; scaling and cutting the target spine image and the first mask based on a multi-scale scaling and cutting strategy to obtain a plurality of first intermediate images and a plurality of second intermediate images; inputting the first intermediate image and the second intermediate image into a fine segmentation network of a 3D attention mechanism to obtain an intermediate mask; and repairing the intermediate mask based on a preset mask repairing strategy to obtain a target segmentation result. According to the method, accurate segmentation can be carried out on fine structures such as lumbosacral nerve roots corresponding to the target spine image, and the defect that small target region features are lost in the process of carrying out forward segmentation on the spine image in an existing deep learning technology is overcome.

Description

Technical Field

[0001] The present application relates to but is not limited to the field of image processing technology, and in particular to a deep learning-based image segmentation method, device, equipment, and storage medium. Background Art

[0002] The automatic image segmentation method using deep learning can efficiently segment the lumbar sacral vertebrae, iliac bone structure, intervertebral disc, nerve root, dura mater and other structures in the spinal image, and present the lumbar sacral intervertebral foramen area in 3D visualization, which has important clinical significance for assisting doctors in intervertebral foramen puncture or surgical navigation puncture. However, the spinal structure in the lumbar sacral intervertebral foramen area not only involves large structures such as lumbar sacral vertebrae, iliac bone structure, intervertebral disc and dura mater, but also involves the small target segmentation of fine structures such as lumbar sacral nerve root and intervertebral disc herniation target. In the process of forward segmentation of spinal images, the existing deep learning technology easily loses the features of small target areas, and the performance of small target segmentation is not good, and it is impossible to accurately segment the fine structures in the spinal image. Summary of the invention

[0003] The embodiments of the present application provide a deep learning-based image segmentation method, apparatus, device, and storage medium, which can accurately segment small target areas in an image to be segmented.

[0004] In a first aspect, an embodiment of the present application provides an image segmentation method based on deep learning, comprising:

[0005] Acquire a target spinal column image, wherein the target spinal column image is an MR image of a lumbar sacral region of a target subject;

[0006] Performing preliminary segmentation on the target spine image based on a preset coarse segmentation network to obtain a first mask, wherein the first mask includes a plurality of target position information, each of which is used to characterize the position information of each human body structure in the lumbar sacral region in the first mask;

[0007] Based on a preset multi-scale scaling and cropping strategy, scaling and cropping are performed on the target spine image and the first mask respectively to obtain a plurality of first intermediate images and a plurality of second intermediate images;

[0008] Inputting the first intermediate image and the second intermediate image into a fine segmentation network of a 3D attention mechanism to obtain an intermediate mask;

[0009] The intermediate mask is repaired based on a preset mask repair strategy to obtain a target segmentation result.

[0010] In some embodiments, the human body structure includes lumbar sacral nerve roots, and the target spinal column image and the first mask are scaled and cropped based on a preset multi-scale scaling and cropping strategy to obtain a plurality of first intermediate images and a plurality of second intermediate images, including:

[0011] Performing image filling on each image slice of the target spinal column image to obtain a first intermediate spinal column image, wherein the image feature corresponding to the lumbar sacral nerve root is located at the center of the first intermediate spinal column image;

[0012] and performing image filling on each image slice of the first mask to obtain a second mask, wherein the image feature corresponding to the lumbar sacral nerve root is located at the center of the second mask;

[0013] Using a bilinear interpolation algorithm to perform multi-scale scaling processing on the target spine image to obtain a plurality of second intermediate spine images with different image scales;

[0014] Performing multi-scale scaling processing on the first mask using a nearest neighbor interpolation algorithm to obtain a plurality of intermediate masks with different image scales;

[0015] Cropping all the second intermediate spine images to obtain the first intermediate image, wherein all the first intermediate images have the same image size;

[0016] All of the intermediate masks are cropped to obtain the second intermediate images, wherein all of the second intermediate images have the same image size.

[0017] In some embodiments, the expression of the second intermediate spine image is:

[0018] (N x，s , N y，s )=(N x，0 +s×r,N y，0 +s×r);

[0019] Among them, (N x,s ,N y,s ) is the second intermediate spine image, (N x,0 ,N y,0 ) is the target spine image before scaling, s is the scale, and r is the scaling factor.

[0020] In some embodiments, the first intermediate image is obtained by cropping all of the second intermediate spine images according to the following formula:

[0021]

[0022] The second intermediate image is obtained by cropping all the intermediate masks according to the following formula:

[0023]

[0024] in, is the first intermediate image, is the second intermediate image, X *,i,s For the second intermediate spine image, Y *,i,s is the intermediate mask, i is the number of the first intermediate image and the second intermediate image, s is the scale, s∈(1,...,S), C x,s is the image center coordinate of the second intermediate spine image at the sth scale, C y,s is the image center coordinate of the intermediate mask at the sth scale, n x is the radius of the first image patch, n y is the radius of the second image patch.

[0025] In some embodiments, the first intermediate image and the second intermediate image are input into a fine segmentation network of a 3D attention mechanism to obtain an intermediate mask, including:

[0026] stacking all of the first intermediate images using the first image patches to obtain a pseudo 3D image;

[0027] stacking all the second intermediate images using the second image patches to obtain a pseudo 3D mask;

[0028] The pseudo 3D image and the pseudo 3D mask are input into a fine segmentation network of a 3D attention mechanism to obtain the intermediate mask.

[0029] In some embodiments, the fine segmentation network of the 3D attention mechanism is trained based on the following loss function:

[0030]

[0031] Among them, class is the segmentation category index, and class has two values ​​0 and 1, 1 represents the segmentation target and 0 represents the background.

[0032] In some embodiments, the first intermediate image and the second intermediate image are input into a fine segmentation network of a 3D attention mechanism to obtain an intermediate mask, which is obtained according to the following formula:

[0033] z=x×Soft max(x×y T );

[0034] Where x,y,z∈RC×(H×W×S) , x is the image feature with C channels, S scale and H×W spatial size, y T is the transpose of x, and z is the calculated 3D attention.

[0035] In a second aspect, an embodiment of the present application provides a control device, comprising at least one control processor and a memory for communicating with the at least one control processor; the memory stores instructions executable by the at least one control processor, and the instructions are executed by the at least one control processor so that the at least one control processor can execute the deep learning-based image segmentation method as described in the first aspect.

[0036] In a third aspect, an embodiment of the present application further provides an electronic device, comprising the control device of the second aspect.

[0037] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to execute the deep learning-based image segmentation method as described in the first aspect.

[0038] The embodiment of the present application provides an image segmentation method, device, equipment, and storage medium based on deep learning, and the method includes: obtaining a target spinal image, wherein the target spinal image is an MR image of the lumbar sacral region of the target object; performing preliminary segmentation on the target spinal image based on a preset coarse segmentation network to obtain a first mask, wherein the first mask includes multiple target position information, each of which is used to characterize the position information of each human structure in the lumbar sacral region in the first mask; performing scaling and cropping processing on the target spinal image and the first mask based on a preset multi-scale scaling and cropping strategy to obtain multiple first intermediate images and multiple second intermediate images; inputting the first intermediate image and the second intermediate image into a fine segmentation network of a 3D attention mechanism to obtain an intermediate mask; repairing the intermediate mask based on a preset mask repair strategy to obtain a target segmentation result. The present application can accurately segment fine structures such as lumbar sacral nerve roots corresponding to the target spinal image, making up for the defect that the existing deep learning technology loses the features of small target areas during the forward segmentation of spinal images. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 is a flowchart of a deep learning-based image segmentation method provided by an embodiment of the present application;

[0040] Figure 2 is a flowchart of steps for obtaining a first intermediate image and a second intermediate image provided by another embodiment of the present application;

[0041] Figure 3 is a flowchart of steps for obtaining an intermediate mask provided by another embodiment of the present application;

[0042] Figure 4 It is a structural diagram of a control device provided in another embodiment of the present application. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0044] It is understood that although the functional modules are divided in the device schematic diagram and the logical order is shown in the flow chart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flow chart. The terms "first", "second", etc. in the specification, claims or the above drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0045] The automatic image segmentation method using deep learning can efficiently segment the lumbar sacral vertebrae, iliac bone structure, intervertebral disc, nerve root, dura mater and other structures in the spinal image, and present the lumbar sacral intervertebral foramen area in 3D visualization, which has important clinical significance for assisting doctors in intervertebral foramen puncture or surgical navigation puncture. However, the spinal structure in the lumbar sacral intervertebral foramen area not only involves large structures such as lumbar sacral vertebrae, iliac bone structure, intervertebral disc and dura mater, but also involves the small target segmentation of fine structures such as lumbar sacral nerve root and intervertebral disc herniation target. In the process of forward segmentation of spinal images, the existing deep learning technology easily loses the features of small target areas, and the performance of small target segmentation is not good, and it is impossible to accurately segment the fine structures in the spinal image.

[0046] To solve the above-mentioned problems, the embodiment of the present application provides an image segmentation method, device, equipment, and storage medium based on deep learning, the method comprising: obtaining a target spinal image, the target spinal image being an MR image of the lumbar sacral region of the target object; performing preliminary segmentation on the target spinal image based on a preset coarse segmentation network to obtain a first mask, wherein the first mask includes a plurality of target position information, each of which is used to characterize the position information of each human structure in the lumbar sacral region in the first mask; performing scaling and cropping processing on the target spinal image and the first mask based on a preset multi-scale scaling and cropping strategy to obtain a plurality of first intermediate images and a plurality of second intermediate images; inputting the first intermediate image and the second intermediate image into a fine segmentation network of a 3D attention mechanism to obtain an intermediate mask; and repairing the intermediate mask based on a preset mask repair strategy to obtain a target segmentation result. The present application can accurately segment fine structures such as lumbar sacral nerve roots corresponding to the target spinal image, making up for the defect that the existing deep learning technology loses the features of small target areas during the forward segmentation of spinal images.

[0047] The embodiments of the present application are further described below in conjunction with the accompanying drawings.

[0048] refer to Figure 1 , Figure 1 This is a flowchart of a method for image segmentation based on deep learning provided by an embodiment of the present application. The embodiment of the present application provides a method, apparatus, device, and storage medium for image segmentation based on deep learning, and the method includes but is not limited to the following steps:

[0049] Step S110, acquiring a target spine image, where the target spine image is an MR image of the lumbar sacral region of the target subject;

[0050] Step S120, performing preliminary segmentation on the target spine image based on a preset coarse segmentation network to obtain a first mask, wherein the first mask includes a plurality of target position information, each target position information being used to characterize position information of each human body structure in the lumbar sacral region in the first mask;

[0051] Step S130, scaling and cropping the target spine image and the first mask respectively based on a preset multi-scale scaling and cropping strategy to obtain a plurality of first intermediate images and a plurality of second intermediate images;

[0052] Step S140, inputting the first intermediate image and the second intermediate image into a fine segmentation network of a 3D attention mechanism to obtain an intermediate mask;

[0053] Step S150: repairing the intermediate mask based on a preset mask repair strategy to obtain a target segmentation result.

[0054] It should be noted that the target spinal image in the embodiment of the present application can be the lumbar sacral segment imaging data of the target object obtained by using the 3D-SPACE sequence of a superconducting MR machine and exported in DICOM format, or it can be a multimodal image obtained after alignment of the 3D-C-arm image and the MR image. Technical personnel in this field can choose according to actual conditions.

[0055] It can be understood that since the target spinal image is an image of the lumbar sacral region of the corresponding target object, the lumbar sacral region is composed of spinal structures such as the lumbar sacral bone structure, iliac bone structure, dura mater and nerve roots. Therefore, in order to better assist doctors in seeing the various structures of the lumbar sacral region of the entire target object more clearly during surgery, the target spinal image not only involves the segmentation of large spinal target structures, but also involves the segmentation of small targets of fine structures such as lumbar sacral nerve roots and intervertebral disc herniation targets. In order to better perform small target segmentation on the target spinal image, this embodiment first performs a preliminary segmentation on the target spinal image based on a coarse segmentation network to obtain a first mask, which can achieve preliminary positioning of the target position information of each small target (i.e., lumbar sacral nerve roots, intervertebral disc herniation targets) in the target spinal image, providing an effective data basis for the subsequent precise segmentation of small targets.

[0056] It should be noted that the embodiment of the present application is not limited to a specific coarse segmentation network, and the coarse segmentation network of this embodiment may be a 3D-UNet network.

[0057] It can be understood that performing multi-scale scaling processing on the target spine image and the first mask respectively based on the preset multi-scale scaling and cropping strategy can more effectively extract image features and obtain more image details, and after scaling, further performing multi-scale cropping processing on the image can remove unnecessary parts of the image and improve image quality, thereby providing an effective data basis for subsequent accurate segmentation of target features.

[0058] It should be noted that the image segmentation method of the present embodiment is essentially to perform slice-by-slice segmentation along the z-axis of the first mask. The MR image corresponding to the first mask, that is, the slice of the target spine image has a layer thickness problem. In the image segmentation (coarse segmentation, fine segmentation) process, cutting discontinuity is prone to occur, resulting in small target features in certain slices being difficult to be correctly located in the coarse segmentation. Repairing the intermediate mask based on the preset mask repair strategy can effectively solve the problem of discontinuity of small target features and obtain accurate target segmentation results. The embodiment of the present application does not limit the specific mask repair strategy. It can be to expand the intermediate mask and then perform an erosion operation on the intermediate mask while keeping the slice size of the current intermediate mask unchanged to repair the slices that may be lost along the z-axis.

[0059] It can be understood that, in this embodiment, a target spinal image is obtained, which is an MR image of the lumbar sacral region of the target object; the target spinal image is preliminarily segmented based on a preset coarse segmentation network to obtain a first mask, wherein the first mask includes multiple target position information, each target position information is used to characterize the position information of each human structure in the lumbar sacral region in the first mask; the target spinal image and the first mask are scaled and cropped based on a preset multi-scale scaling and cropping strategy to obtain multiple first intermediate images and multiple second intermediate images; the first intermediate image and the second intermediate image are input into the fine segmentation network of the 3D attention mechanism, which can not only further finely segment small target features such as nerve roots, but also capture contextual information from receptive fields of different scales to obtain an intermediate mask; the intermediate mask is repaired based on a preset mask repair strategy, which can effectively solve the problem of discontinuous small target features and obtain accurate target segmentation results. The present application can accurately segment fine structures such as lumbar sacral nerve roots corresponding to the target spinal image, making up for the defect that the existing deep learning technology loses small target area features in the process of forward segmentation of spinal images.

[0060] It can be understood that the present application realizes the accurate segmentation of the features of the target spinal image based on the multimodal image lumbar intervertebral foramen multi-structure automatic segmentation model composed of a coarse segmentation network and a fine segmentation network of a 3D attention mechanism connected in sequence. The embodiment of the present application also provides a method for verifying the multimodal image lumbar intervertebral foramen multi-structure automatic segmentation model to provide support for training to obtain a better model, specifically including: using 5-fold cross-validation to evaluate the segmentation performance of the model. The 120 registered multimodal image data were randomly divided into 5 groups, each with 24 samples. In each experiment, 84 samples and 12 samples were selected from 4 groups to establish training and verification data sets, and the remaining 24 samples in 1 group were used to establish a test data set. During the training process, the model with the best segmentation performance on the verification data set was saved. Repeat the iterative training 5 times until the segmentation results of all samples were obtained, and the automatic segmentation performance of the multimodal image lumbar intervertebral foramen multi-structure automatic segmentation model was evaluated using indicators such as the Dice coefficient, precision and recall. The Dice coefficient is a set similarity measurement function, usually used to calculate the similarity of two samples, with a value range of [0,1]: when the Dice coefficient result is 0≤x<0.7, it will be considered as "poor" performance, 0.7≤x<0.8 "average" performance, 0.8≤x<0.9 "good" performance, and x≥0.9 "excellent" performance.

[0061] Among them, the Dice coefficient, precision and recall can be calculated according to the following formula:

[0062]

[0063] Among them, TP is the sample judged as a positive sample by the model, which is actually a positive sample; FP is the sample judged as a positive sample by the model, which is actually a negative sample; FN is the sample judged as a negative sample by the model, which is actually a positive sample; TN is the sample judged as a negative sample by the model, which is actually a negative sample.

[0064] In addition, refer to Figure 2 , in some embodiments, the human structure comprises a lumbar sacral nerve root, Figure 1 Step S130 includes but is not limited to the following steps:

[0065] Step S210, performing image filling on each image slice of the target spinal column image to obtain a first intermediate spinal column image, wherein the image feature corresponding to the lumbar sacral nerve root is located at the center of the first intermediate spinal column image;

[0066] Step S220, performing image filling on each image slice of the first mask to obtain a second mask, wherein the image feature corresponding to the lumbar sacral nerve root is located at the center of the second mask;

[0067] Step S230, performing multi-scale scaling processing on the first intermediate spine image using a bilinear interpolation algorithm to obtain a plurality of second intermediate spine images with different image scales;

[0068] Step S240, performing multi-scale scaling processing on the second mask using a nearest neighbor interpolation algorithm to obtain a plurality of intermediate masks with different image scales;

[0069] Step S250, cropping all the second intermediate spine images to obtain first intermediate images, wherein all the first intermediate images have the same image size;

[0070] Step S260: cropping all the intermediate masks to obtain second intermediate images, wherein all the second intermediate images have the same image size.

[0071] It can be understood that the fine structure of the lumbar sacral region may include lumbar sacral nerve roots, intervertebral disc herniation targets, etc. This embodiment is described by taking the segmentation of lumbar sacral nerve roots as an example: after the target spinal image is roughly segmented, each image slice of the target spinal image is image filled to obtain a first intermediate spinal image, and each image slice of the first mask is image filled to obtain a second mask, so as to ensure that the image features corresponding to the lumbar sacral nerve roots corresponding to the first intermediate spinal image and the second mask are located in the middle position, that is, the lumbar sacral nerve roots are centered. This embodiment does not limit the specific size of the image filling, and those skilled in the art can adjust it according to the size of the first mask; the lumbar sacral nerve roots may include left nerve roots and right nerve roots, let X L ,i, and Y L , i represent the first intermediate spinal image and the second mask corresponding to the left nerve root after centering. Let X R,i , and Y R,i They represent the first intermediate spine image and the second mask corresponding to the right nerve root after being centered. The first intermediate spine image is scaled by bilinear interpolation algorithm to obtain multiple second intermediate spine images X with different image scales. *,i , the nearest neighbor interpolation algorithm is used to perform multi-scale scaling on the second mask to obtain multiple intermediate masks Y with different image scales *,i Then, all the second intermediate spine images are cropped to obtain the first intermediate images, wherein all the first intermediate images have the same image size, and all the intermediate masks are cropped to obtain the second intermediate images, wherein all the second intermediate images have the same image size, providing effective data for the subsequent 3D attention mechanism's fine segmentation network.

[0072] It should be noted that the second intermediate spine image can be calculated according to the following formula:

[0073] (N x,s , N y,s )=(N x,0 +s×r,N y,0 +s×r);

[0074] Among them, (N x,s ,N y,s ) is the second intermediate spine image, (N x,0 ,N y,0 ) is the target spine image before scaling, s is the scale, and r is the scaling factor.

[0075] It should be noted that the first intermediate image is obtained by cropping all the second intermediate spine images according to the following formula:

[0076]

[0077] The second intermediate image is obtained by cropping all the intermediate masks according to the following formula:

[0078]

[0079] in, is the first intermediate image, is the second intermediate image, X *,i,s is the second intermediate spine image, Y *,i,s is the intermediate mask, i is the number of the first intermediate image and the second intermediate image, s is the scale, s∈(1,...,S), C x,s is the image center coordinate of the second intermediate spine image at the sth scale, C y,s is the image center coordinate of the middle mask at the sth scale, n x is the radius of the first image patch, n y is the radius of the second image patch, and * represents the attributes of the lumbar sacral nerve roots, including the left nerve root L and the right nerve root R.

[0080] In addition, refer to Figure 3 , in some embodiments, Figure 1 Step S140 includes but is not limited to the following steps:

[0081] Step S310, stacking all first intermediate images using the first image patch to obtain a pseudo 3D image;

[0082] Step S320, stacking all the second intermediate images using the second image patch to obtain a pseudo 3D mask;

[0083] In step S330, the pseudo 3D image and the pseudo 3D mask are input into the fine segmentation network of the 3D attention mechanism to obtain an intermediate mask.

[0084] It can be understood that this embodiment uses the first image patch to stack all the first intermediate images to obtain a pseudo 3D image, and uses the second image patch to stack all the second intermediate images to obtain a pseudo 3D mask, and inputs the pseudo 3D image and the pseudo 3D mask into the fine segmentation network of the 3D attention mechanism to obtain an intermediate mask. By stacking multiple images, unnecessary content or noise can be eliminated, and noise and distortion in the image can be reduced, thereby significantly improving the overall quality of the image, and can effectively improve the image quality of the intermediate mask, providing an effective data basis for subsequent image fine segmentation.

[0085] It should be noted that the specific structure of the fine segmentation network of the 3D attention mechanism of this embodiment includes a plurality of first convolution blocks, a 3D attention module, a plurality of second convolution blocks and a third convolution block connected in sequence, wherein the first convolution block includes three first convolution layers, the first convolution layer is connected to the next first convolution layer through a convolution unit, a batch normalization unit and a corrected linear unit, the first convolution block is connected to the next first convolution block through a maximum pooling unit, and the convolution parameters of the first convolution layers in each first convolution block are different, and the 3D attention module is used to perform the following steps: convolve, reshape and transform the feature image obtained by the adjacent first convolution block to obtain a first image; convolve and reshape the feature image obtained by the adjacent first convolution block to obtain a second image; Cartesian product the first image with the second image to obtain a third image; and soft The second convolution block includes three second convolutional layers, the second convolutional layer is connected to the next second convolutional layer through the convolution unit, the batch normalization unit and the correction linear unit, and the second convolutional block is connected to the next second convolutional block through the upper convolution unit. The convolution parameters of the second convolutional layers in each second convolutional block are different from each other. The second convolutional block is connected to the third convolutional block through the upper convolution unit. The third convolutional block includes four third convolutional layers connected in sequence. Between the third convolutional layers numbered 1, 2, and 3, the third convolutional layer is connected to the next third convolutional layer through the convolution unit, the batch normalization unit and the correction linear unit. The third convolutional layer numbered 3 is only connected to the third convolutional layer numbered 4 through the convolutional layer unit. This embodiment does not limit the specific number of the first convolutional block and the second convolutional block, and those skilled in the art can adjust it according to actual conditions.

[0086] It should be noted that the fine segmentation network of the 3D attention mechanism is trained based on the following loss function:

[0087]

[0088] Among them, class is the segmentation category index, and class has two values ​​0 and 1, 1 represents the segmentation target and 0 represents the background.

[0089] In addition, in the process of fine segmentation of nerve roots in this embodiment, the network generates S pseudo 3D masks, but the corresponding 2D slice only needs one of them. Therefore, in the test phase, for a given pseudo 3D mask, only the slice with the highest Dice coefficient in the validation data set is selected for subsequent processing. The selected mask is scaled back to its original scale and stacked into the mask of the real 3D image, and the final target segmentation result is obtained through mask repair.

[0090] It should be noted that the first intermediate image and the second intermediate image are input into the fine segmentation network of the 3D attention mechanism to obtain the intermediate mask, which is obtained according to the following formula:

[0091] z=x×Soft max(x×y T );

[0092] Where x,y,z∈R ×(H×W×S) , x is the image feature with C channels, S scale and H×W spatial size, y T is the transpose of x, and z is the calculated 3D attention.

[0093] like Figure 4 As shown, Figure 4 400 is a structural diagram of a control device provided by an embodiment of the present application. The present invention also provides a control device 400, including:

[0094] The processor 410 may be implemented by a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0095] The memory 420 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 420 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 420, and the processor 410 calls and executes the image segmentation method based on deep learning in the embodiment of this application;

[0096] Input / output interface 430, used to implement information input and output;

[0097] Communication interface 440, used to realize communication interaction between the present apparatus and other devices, which can be realized by wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.);

[0098] bus 450 , which transmits information between the various components of the device (e.g., processor 410 , memory 420 , input / output interface 430 , and communication interface 440 );

[0099] The processor 410 , the memory 420 , the input / output interface 430 , and the communication interface 440 are connected to each other in communication within the device via the bus 450 .

[0100] In addition, an embodiment of the present application further provides an electronic device, including the control device 400 of the above embodiment.

[0101] In addition, an embodiment of the present application further provides a storage medium, which is a computer-readable storage medium and stores a computer program. When the computer program is executed by a processor, the above-mentioned deep learning-based image segmentation method is implemented.

[0102] As a non-transient computer-readable storage medium, the memory can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage devices. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof. The device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and are implemented to be located in one place, or may also be distributed to multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.

[0103] It will be appreciated by those skilled in the art that all or some of the steps and systems in the disclosed method above may be implemented as software, firmware, hardware and appropriate combinations thereof. Some physical components or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor or a microprocessor, or may be implemented as hardware, or may be implemented as an integrated circuit, such as an application specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or a non-transitory medium) and a communication medium (or a temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, disk storage or other magnetic storage devices, or any other medium that may be used to store desired information and may be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically include computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0104] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the above-mentioned implementation mode. Technical personnel familiar with the field can also make various equivalent deformations or substitutions under the shared conditions without violating the spirit of the present invention. These equivalent deformations or substitutions are all included in the scope defined by the claims of the present invention.

Claims

1. A deep learning-based image segmentation method, characterized in that: include: Acquire a target spinal column image, wherein the target spinal column image is an MR image of a lumbar sacral region of a target subject; Performing preliminary segmentation on the target spine image based on a preset coarse segmentation network to obtain a first mask, wherein the first mask includes a plurality of target position information, each of which is used to characterize the position information of each human body structure in the lumbar sacral region in the first mask; Based on a preset multi-scale scaling and cropping strategy, scaling and cropping are performed on the target spine image and the first mask respectively to obtain a plurality of first intermediate images and a plurality of second intermediate images; Inputting the first intermediate image and the second intermediate image into a fine segmentation network of a 3D attention mechanism to obtain an intermediate mask; The intermediate mask is repaired based on a preset mask repair strategy to obtain a target segmentation result.

2. The image segmentation method based on deep learning according to claim 1, characterized in that: The human body structure includes lumbar sacral nerve roots, and the target spine image and the first mask are scaled and cropped based on the preset multi-scale scaling and cropping strategy to obtain a plurality of first intermediate images and a plurality of second intermediate images, including: Performing image filling on each image slice of the target spinal column image to obtain a first intermediate spinal column image, wherein the image feature corresponding to the lumbar sacral nerve root is located at the center of the first intermediate spinal column image; and performing image filling on each image slice of the first mask to obtain a second mask, wherein the image feature corresponding to the lumbar sacral nerve root is located at the center of the second mask; Using a bilinear interpolation algorithm to perform multi-scale scaling on the first intermediate spine image to obtain a plurality of second intermediate spine images with different image scales; Performing multi-scale scaling processing on the second mask using a nearest neighbor interpolation algorithm to obtain a plurality of intermediate masks with different image scales; Cropping all the second intermediate spine images to obtain the first intermediate image, wherein all the first intermediate images have the same image size; All of the intermediate masks are cropped to obtain the second intermediate images, wherein all of the second intermediate images have the same image size.

3. The image segmentation method based on deep learning according to claim 2, characterized in that: The expression of the second intermediate spine image is: (N x，s ,N y，s )=(N x，0 +s×r,N y，0 +s×r); Among them, (N x,s ,N y,s ) is the second intermediate spine image, (N x,0 ,N y,0 ) is the target spine image before scaling, s is the scale, and r is the scaling factor.

4. The image segmentation method based on deep learning according to claim 2, characterized in that: The first intermediate image is obtained by cropping all the second intermediate spine images according to the following formula: The second intermediate image is obtained by cropping all the intermediate masks according to the following formula: in, is the first intermediate image, is the second intermediate image, X *,i,s For the second intermediate spine image, Y *,i,s is the intermediate mask, i is the number of the first intermediate image and the second intermediate image, s is the scale, s∈(1,...,S), C x,s is the image center coordinate of the second intermediate spine image at the sth scale, C y,s is the image center coordinate of the intermediate mask at the sth scale, n x is the radius of the first image patch, n y is the radius of the second image patch.

5. The image segmentation method based on deep learning according to claim 4, characterized in that: The first intermediate image and the second intermediate image are input into the fine segmentation network of the 3D attention mechanism to obtain an intermediate mask, including: stacking all of the first intermediate images using the first image patches to obtain a pseudo 3D image; stacking all the second intermediate images using the second image patches to obtain a pseudo 3D mask; The pseudo 3D image and the pseudo 3D mask are input into a fine segmentation network of a 3D attention mechanism to obtain the intermediate mask.

6. The image segmentation method based on deep learning according to claim 5, characterized in that: The fine segmentation network of the 3D attention mechanism is trained based on the following loss function: Among them, class is the segmentation category index, and class has two values ​​0 and 1, 1 represents the segmentation target and 0 represents the background.

7. The image segmentation method based on deep learning according to claim 1, characterized in that: The first intermediate image and the second intermediate image are input into the fine segmentation network of the 3D attention mechanism to obtain the intermediate mask, which is obtained according to the following formula: z=x×Softmax(x×y T ); Where z,y,z∈R C×(H×W×S) , x is the image feature with C channels, S scale and H×W spatial size, y T is the transpose of x, and z is the calculated 3D attention.

8. A control device, characterized in that: It includes at least one control processor and a memory for communicating with the at least one control processor; the memory stores instructions that can be executed by the at least one control processor, and the instructions are executed by the at least one control processor so that the at least one control processor can execute the deep learning-based image segmentation method as described in any one of claims 1 to 7.

9. An electronic device, characterized in that: Comprising the control device as claimed in claim 8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the deep learning-based image segmentation method as described in any one of claims 1 to 7.

Images