Deep learning-based spine segmentation method, apparatus and device, and medium

Abstract

The invention discloses a spine segmentation method and device based on deep learning, equipment and a medium, and belongs to the technical field of artificial intelligence. According to the embodiment of the invention, two deep learning models are used to carry out two-stage image segmentation on the original image containing the spine, the spine in the medical image is automatically identified, each conical node is automatically segmented, and the position of each conical node is identified. A new method is provided for clinicians during spinal disease and operation planning, the method is quicker and more accurate, a certain part of global information or local information can be considered, and the method is more in line with the radiograph reading habit of the clinicians, so that a more accurate segmentation result is achieved, and the efficiency of the clinicians is improved. And the arrangement sequence of the cone nodes between the spine and the cervical vertebra is emphasized in the local segmentation process, so that the position of each cone node is obtained through fine segmentation identification, the model accuracy is further improved, and the image segmentation precision is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a spine segmentation method, device, equipment and medium based on deep learning. Background Art

[0002] Spinal diseases are one of the main causes of chronic pain and dysfunction worldwide, including lumbar disc herniation, scoliosis and fractures. Degenerative disc disease (DDD) such as lumbar spinal stenosis, lumbar disc herniation and cervical spondylosis are common and frequently occurring diseases that can seriously affect the quality of life of patients [1,2]. With the aging of the population, the incidence rate is gradually increasing, and the disease burden is becoming increasingly serious. Because the structure of the spine is complex and adjacent to important nerves and vascular tissues, in addition to the developmental variation, deformity or degeneration of the spine, it is also common. Therefore, how to improve the safety and accuracy of spinal surgery is crucial. In recent years, "precision medicine" has emerged, and studies [3] have shown that computer-guided spinal surgery can significantly improve the safety and efficiency of surgery. As the first step in diagnosis and computer guidance, an accurate and efficient spine segmentation and labeling method will help to carry out subsequent work more efficiently.

[0003] Spinal imaging examinations (such as CT and MRI) are important tools for diagnosing these diseases, and accurate spinal segmentation and cone identification are crucial for disease assessment and treatment planning. Traditionally, imaging examinations rely on manual labeling and analysis by doctors, which is not only time-consuming and labor-intensive, but also easily affected by subjective factors, resulting in inconsistency in labeling. In addition, with the increase in the amount of imaging data, the efficiency of manual analysis can no longer meet clinical needs. The application of automatic spinal segmentation and cone classification technology can significantly improve the efficiency and accuracy of image analysis, thereby facilitating the formulation of individualized treatment plans. For example, in scoliosis surgery, accurate segmentation and labeling help identify affected vertebrae, thereby formulating a more accurate surgical path.

[0004] In the prior art, a two-dimensional deep learning network is constructed based on patch (dividing the image into a certain number of small image blocks in advance) and a fully convolutional network to identify the spinal area on each image, but it does not identify which specific cone segment it is. Therefore, the segmentation accuracy of this spinal segmentation method is low and the auxiliary effect on spinal imaging examination is not good. Summary of the invention

[0005] The embodiment of the present application provides a spine segmentation method, device, equipment and medium based on deep learning, which can achieve the effect of improving segmentation accuracy, fast and accurate. The technical solution is as follows:

[0006] In one aspect, a spine segmentation method based on deep learning is provided, the method comprising:

[0007] Based on the original image including the spine, a plurality of slice images are acquired;

[0008] For a first image among the multiple slice images, a three-dimensional image block composed of the first image and its adjacent slice images is input into a first deep learning model, and the first deep learning model segments the first image based on the three-dimensional image block to obtain a first prediction result of the first image, where the first prediction result is used to indicate whether each pixel in the first image is a spinal region and a corresponding cone position;

[0009] In order from one end to the other end of the spine, the first prediction result of each target number of adjacent cone segments in the first prediction result and the second image are input into a three-dimensional second deep learning model. The second deep learning model segments the second image based on the first prediction result of each target number of cone segments, and outputs a second prediction result of the second image. The second image is a local image area corresponding to each target number of cone segments in the first image, and the second prediction result is used to indicate whether each pixel in the second image is a spinal region and the corresponding cone segment position.

[0010] In some embodiments, the step of acquiring a plurality of slice images based on an original image containing the spine includes:

[0011] The original image is preprocessed to obtain a first image, wherein the preprocessing includes cropping and / or standardization, wherein the cropping is to crop the background area other than the body in the original image according to grayscale information and / or position information.

[0012] In some embodiments, the process of obtaining the second prediction result of the second image based on the first prediction result and the second image is performed serially in an order from one end to the other end of the spine; or, the process of obtaining the second prediction result of the second image based on the first prediction result and the second image is performed in parallel through multiple processes.

[0013] In some embodiments, the training process of the first deep learning model is implemented based on a first loss function, the first loss function is obtained by integrating multiple loss functions using multiple scales, and the weights of multiple loss functions in the first loss function are adaptively adjusted according to the training situation.

[0014] In some embodiments, the training process of the second deep learning model is based on at least one of the false positive rate of the segmentation result and the boundary loss function, and a first loss function, wherein the first loss function is obtained based on the integration of multiple loss functions using multi-scales, and the weights of multiple loss functions in the first loss function are adaptively adjusted according to the training situation.

[0015] In some embodiments, the method further comprises:

[0016] The second prediction result of the second image is added to the original image according to the position information of the pixel points to obtain a third prediction result of the original image, wherein the third prediction result of the original image includes a segmentation result and a labeling result of the original image.

[0017] In some embodiments, adding the second prediction result of the second image to the original image according to the position information of the pixel points to obtain the third prediction result of the original image includes:

[0018] In response to the presence of overlapping pixels in the second prediction result of the second image, and the second prediction results of the overlapping pixels in different second images are different, the pixels around the overlapping pixels are clustered according to the second prediction results of the pixels around the overlapping pixels to obtain a third prediction result for the overlapping pixels.

[0019] In one aspect, a deep learning-based spinal segmentation device is provided, the device comprising:

[0020] An acquisition module, used for acquiring a plurality of slice images based on an original image including a spine;

[0021] A first segmentation module is used for inputting, for a first image among the multiple slice images, a three-dimensional image block composed of the first image and its adjacent slice images into a first deep learning model, and the first deep learning model segments the first image based on the three-dimensional image block to obtain a first prediction result of the first image, wherein the first prediction result is used to indicate whether each pixel in the first image is a spinal region and a corresponding cone position;

[0022] A second segmentation module is used to input the first prediction results of each target number of adjacent cone segments in the first prediction results and the second image into a three-dimensional second deep learning model in order from one end to the other end of the spine. The second deep learning model segments the second image based on the first prediction results of each target number of cone segments, and outputs a second prediction result of the second image, where the second image is a local image area corresponding to each target number of cone segments in the first image, and the second prediction result is used to indicate whether each pixel in the second image is a spinal region and the corresponding cone segment position.

[0023] In some embodiments, the acquisition module is used to preprocess the original image to obtain a first image, and the preprocessing includes cropping and / or standardization, and the cropping is to crop the background area outside the body in the original image according to grayscale information and / or position information.

[0024] In some embodiments, the process of obtaining the second prediction result of the second image based on the first prediction result and the second image is performed serially in an order from one end to the other end of the spine; or, the process of obtaining the second prediction result of the second image based on the first prediction result and the second image is performed in parallel through multiple processes.

[0025] In some embodiments, the training process of the first deep learning model is implemented based on a first loss function, the first loss function is obtained by integrating multiple loss functions using multiple scales, and the weights of multiple loss functions in the first loss function are adaptively adjusted according to the training situation.

[0026] In some embodiments, the training process of the second deep learning model is based on at least one of the false positive rate of the segmentation result and the boundary loss function, and a first loss function, wherein the first loss function is obtained based on the integration of multiple loss functions using multi-scales, and the weights of multiple loss functions in the first loss function are adaptively adjusted according to the training situation.

[0027] In some embodiments, the apparatus further comprises:

[0028] An adding module is used to add the second prediction result of the second image to the original image according to the position information of the pixel points to obtain a third prediction result of the original image, and the third prediction result of the original image includes a segmentation result and a marking result of the original image.

[0029] In some embodiments, the adding module is used to:

[0030] In response to the presence of overlapping pixels in the second prediction result of the second image, and the second prediction results of the overlapping pixels in different second images are different, the pixels around the overlapping pixels are clustered according to the second prediction results of the pixels around the overlapping pixels to obtain a third prediction result for the overlapping pixels.

[0031] On the one hand, an electronic device is provided, which includes one or more processors and one or more memories, wherein at least one computer program is stored in the one or more memories, and the at least one computer program is loaded and executed by the one or more processors to implement various optional implementations of the above-mentioned deep learning-based spine segmentation method.

[0032] On the one hand, a computer-readable storage medium is provided, wherein at least one computer program is stored in the storage medium, and the at least one computer program is loaded and executed by a processor to implement various optional implementations of the above-mentioned deep learning-based spine segmentation method.

[0033] In one aspect, a computer program product or a computer program is provided, wherein the computer program product or the computer program comprises one or more program codes, wherein the one or more program codes are stored in a computer-readable storage medium. One or more processors of an electronic device can read the one or more program codes from the computer-readable storage medium, and the one or more processors execute the one or more program codes, so that the electronic device can perform the spine segmentation method based on deep learning in any possible implementation manner described above.

[0034] In an embodiment of the present application, two deep learning models are used to perform two-stage image segmentation on the original image containing the spine, and the spine in the medical image is fully automatically identified. Each cone segment is automatically segmented and the position of each cone segment is identified, providing clinicians with a new method for spinal disease and surgical planning, which is more accurate and faster. The first stage is based on global information segmentation, and the second stage is based on local information segmentation. It can take into account global information or a part of local information, which is more in line with the clinician's reading habits, thereby achieving a more accurate segmentation result. In the local segmentation process, the order of cone segment arrangement between the spine and the cervical vertebra is emphasized, so that the position of each cone segment is obtained by fine segmentation and identification, further improving the model accuracy and image segmentation accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0036] Figure 1 is a schematic diagram of an implementation environment of a spine segmentation method based on deep learning provided in an embodiment of the present application;

[0037] Figure 2 is a flowchart of a spine segmentation method based on deep learning provided in an embodiment of the present application;

[0038] Figure 3 is a flowchart of a spine segmentation method based on deep learning provided in an embodiment of the present application;

[0039] Figure 4 is a structural schematic diagram of a spinal segmentation device based on deep learning provided in an embodiment of the present application;

[0040] Figure 5 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0041] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0042] In the present application, the terms "first", "second", etc. are used to distinguish between identical or similar items having substantially the same effects and functions. It should be understood that there is no logical or temporal dependency between "first", "second", and "nth", nor is the quantity and execution order limited. It should also be understood that although the following description uses the terms first, second, etc. to describe various elements, these elements should not be limited by the terms. These terms are only used to distinguish one element from another. For example, without departing from the scope of the various described examples, a first image can be referred to as a second image, and similarly, a second image can be referred to as a first image. Both the first image and the second image can be images, and in some cases, can be separate and different images.

[0043] The term "at least one" in this application means one or more, and the term "plurality" in this application means two or more, for example, a plurality of data packets means two or more data packets.

[0044] It should be understood that the terms used in the description of the various examples herein are only for describing specific examples and are not intended to be limiting. As used in the description of the various examples and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0045] It should also be understood that the term "and / or" used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items. The term "and / or" is a description of the association relationship of associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this application generally indicates that the associated objects before and after are in an "or" relationship.

[0046] It should also be understood that in the various embodiments of the present application, the size of the serial number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0047] It should also be understood that determining B based on A does not mean determining B only based on A. B can also be determined based on A and / or other information.

[0048] It should also be understood that the term “comprising” (also known as “inCludes”, “inCluding”, “Comprises” and / or “Comprising”) when used in this specification specifies the presence of stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0049] It should also be understood that the term "if" may be interpreted to mean "when" or "upon" or "in response to determining" or "in response to detecting." Similarly, the phrase "if it is determined that ..." or "if [a stated condition or event] is detected" may be interpreted to mean "upon determining that ..." or "in response to determining that ..." or "upon detecting [a stated condition or event]" or "in response to detecting [a stated condition or event]," depending on the context.

[0050] At least some functions of the device or electronic device provided in the embodiments of the present application can be implemented by an AI model, such as at least one module among multiple modules of the device or electronic device can be implemented by an AI model. Functions associated with AI can be performed by non-volatile memory, volatile memory and processor.

[0051] The processor may include one or more processors. In this case, the one or more processors may be general-purpose processors, such as a central processing unit (CPU), an application processor (AP), etc., or pure graphics processing units, such as a graphics processing unit (GPU), a visual processing unit (VPU), and / or an AI-specific processor, such as a neural processing unit (NPU).

[0052] The one or more processors control the processing of input data according to predefined operating rules or artificial intelligence (AI) models stored in non-volatile memory and volatile memory. The predefined operating rules or artificial intelligence models are provided by training or learning.

[0053] Here, providing by learning means obtaining a predefined operating rule or an AI model with desired characteristics by applying a learning algorithm to a plurality of learning data. The learning can be performed in the device or electronic device itself in which the AI ​​according to the embodiment is executed, and / or can be implemented by a separate server / system.

[0054] The AI ​​model may include multiple neural network layers. Each layer has multiple weight values, and each layer performs neural network calculations by calculating between the input data of the layer (such as the calculation results of the previous layer and / or the input data of the AI ​​model) and the multiple weight values ​​of the current layer. Examples of neural networks include, but are not limited to, convolutional neural networks (CNNs), deep neural networks (DNNs), recurrent neural networks (RNNs), restricted Boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), generative adversarial networks (GANs), and deep Q networks.

[0055] A learning algorithm is a method of using a plurality of learning data to train a predetermined target device (e.g., a robot) to enable, allow, or control the target device to make a determination or prediction. Examples of the learning algorithm include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0056] According to the present application, at least one step in the method performed in the electronic device, such as identifying architectural elements, classifying floor plan areas, etc., can be implemented using an artificial intelligence model. The processor of the electronic device can perform preprocessing operations on the data to convert it into a form suitable for use as an input to the artificial intelligence model. The artificial intelligence model can be obtained through training. Here, "obtained through training" means obtaining a predefined operating rule or artificial intelligence model configured to perform a desired feature (or purpose) by training a basic artificial intelligence model with multiple training data through a training algorithm.

[0057] The nouns involved in this application are explained below.

[0058] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0059] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0060] Computer Vision Technology (CV) Computer vision is a science that studies how to make machines "see". To put it more specifically, it refers to machine vision that uses cameras and computers to replace human eyes to identify, track and measure targets, and further performs graphic processing so that computer processing becomes an image that is more suitable for human eye observation or transmission to instrument detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multi-dimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and map construction, and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.

[0061] Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory, and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence.

[0062] Machine learning and deep learning usually include technologies such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0063] With the research and advancement of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, automatic driving, drones, robots, smart medical care, smart customer service, etc. I believe that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0064] The solutions provided in the embodiments of the present application involve artificial intelligence computer vision technology, machine learning and other technologies, which are specifically explained through the following embodiments.

[0065] The implementation environment of this application is described below.

[0066] Figure 1 1 is a schematic diagram of an implementation environment of a deep learning-based spine segmentation method provided in an embodiment of the present application. The implementation environment includes a terminal 101, or the implementation environment includes a terminal 101 and a deep learning-based spine segmentation platform 102. The terminal 101 is connected to the deep learning-based spine segmentation platform 102 via a wireless network or a wired network.

[0067] The terminal 101 is at least one of a smart phone, a game console, a desktop computer, a tablet computer, an e-book reader, an MP3 (Moving Picture Experts Group Audio Layer III) player or an MP4 (Moving Picture Experts Group Audio Layer IV) player, and a laptop computer. The terminal 101 is installed and runs an application that supports deep learning-based spine segmentation.

[0068] Exemplarily, the terminal 101 has an image segmentation function. After acquiring the original image containing the spine, two deep learning models can be used to perform two-stage image segmentation on it to obtain the spine area and the position of each cone segment in the image. The terminal 101 completes this work independently, or after the terminal 101 acquires the original image, it can transmit it to the spine segmentation platform 102 based on deep learning to provide image segmentation services for it. This embodiment of the application is not limited to this.

[0069] The spine segmentation platform 102 based on deep learning includes at least one of a server, multiple servers, a cloud computing platform, and a virtualization center. The spine segmentation platform 102 based on deep learning is used to provide background services for applications that support spine segmentation based on deep learning. Optionally, the spine segmentation platform 102 based on deep learning undertakes the main processing work, and the terminal 101 undertakes the secondary processing work; or, the spine segmentation platform 102 based on deep learning undertakes the secondary processing work, and the terminal 101 undertakes the main processing work; or, the spine segmentation platform 102 based on deep learning or the terminal 101 undertakes the processing work separately. Alternatively, the spine segmentation platform 102 based on deep learning and the terminal 101 use a distributed computing architecture for collaborative computing.

[0070] Optionally, the deep learning-based spinal segmentation platform 102 includes at least one server 1021 and a database 1022. The database 1022 is used to store data. In an embodiment of the present application, the database 1022 stores sample input data and sample annotation data, and can also store user input data to provide data services for at least one server 1021.

[0071] A server is an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. Terminals are smartphones, tablets, laptops, desktop computers, smart speakers, smart watches, etc., but are not limited to these.

[0072] Those skilled in the art know that the number of the above-mentioned terminals 101 and servers 1021 is more or less. For example, the above-mentioned terminal 101 and server 1021 are only one, or the above-mentioned terminals 101 and servers 1021 are dozens or hundreds, or more. The embodiment of the present application does not limit the number and device type of the terminals or servers.

[0073] In the related technologies, most of them segment the spinal region in the image, but do not accurately identify each spinal cone and which cone segment it belongs to. In a precise surgical environment, doctors need to clarify which cone segments have problems or need surgery. Therefore, it is not enough to simply segment the spine. It is necessary to automatically mark which cone segment each segmented cone segment belongs to. From the perspective of neural network training, it is possible to segment and identify each cone segment end-to-end, but in the face of 7 cervical vertebrae, 12 thoracic vertebrae, 5 lumbar vertebrae, 5 sacral vertebrae and 4 coccygeal vertebrae, and the complex situation that adjacent cone segments are very similar, it is difficult to accurately identify each cone segment through end-to-end prediction. And there is another problem that the actual spinal images scanned in clinical practice often have unclear boundaries on the spinal region and a lot of noise due to the adjacent structural characteristics of the spine and the imaging principle. This causes many spinal regions segmented by algorithms to be adhered together, making it difficult to distinguish each cone segment. Because of the above reasons, it is difficult for a simple 2D neural network to clearly identify which part of the adjacent area of ​​different cone segments belongs to which cone segment. A 3D neural network can make judgments based on the upper and lower adjacent image information. In addition, most of the data faced in clinical practice is data of patients with diseases, not conventional 3D morphology of the spine. In order to make the algorithm results effectively used in clinical practice, a simple end-to-end neural network is difficult to achieve this goal.

[0074] In response to the above problems, the present application proposes a two-stage automatic spine segmentation and labeling method based on deep learning. In the first stage, the spine is segmented out of the cervical, thoracic, lumbar, sacrum, and coccyx regions according to the global information of the image. This stage mainly distinguishes different spinal regions based on the different body tissues and organs around them. Because it relies on global information, some segmentation and recognition errors may occur locally. The second stage is based on the results of the first stage and performs local 3D neural network segmentation, aiming to obtain complete and independent segmentation results and correct labeling for each cone segment in the local area. Although the running time will be slower than the end-to-end one-stage neural network model, it can reduce the time for subsequent modifications by the doctor, and the segmentation results are more in line with clinical needs, which improves the doctor's work efficiency overall. The following is through Figure 2 The illustrated embodiment provides a detailed description of the above-mentioned two-stage image segmentation and labeling method based on deep learning.

[0075] Figure 2 is a flowchart of a spine segmentation method based on deep learning provided in an embodiment of the present application. The method is applied to an electronic device, which is a server or a terminal. Figure 2 , the method comprises the following steps.

[0076] 201. The electronic device obtains a plurality of slice images based on an original image including a spine.

[0077] The present invention proposes a two-stage automatic spine segmentation and labeling method based on deep learning, hoping to provide an automatic spine segmentation and labeling method, and provide a new method for clinicians to perform spinal diseases and surgical planning, which is more accurate and faster. The embodiment of the present application aims to fully automatically identify the spine in medical images, and automatically segment each cone segment, and automatically identify which part each cone segment belongs to, and can generate a three-dimensional structure based on this, providing important image information support for subsequent diagnosis and other work.

[0078] In the embodiment of the present application, the original image containing the spine may be a medical image containing the spine. When a patient visits a doctor, a medical image containing the spine may be taken so that the doctor can judge the patient's condition based on the medical image. The embodiment of the present application does not limit the specific shooting instrument used to shoot the original image containing the spine.

[0079] In some embodiments, the original image containing the spine is generally a three-dimensional (3D) image. Therefore, after the electronic device obtains the original image containing the spine, it can first process it to obtain multiple slice images, and then perform image segmentation on the slice images. It should be noted that the original image containing the spine is generally a 3D image. Each image is generally called a slice in medicine.

[0080] In some embodiments, the electronic device may preprocess the original image to obtain a first image, wherein the preprocessing includes cropping and / or standardization, wherein the cropping is to crop the background area outside the body in the original image according to grayscale information and / or position information.

[0081] Regarding cropping, the original image containing the spine may contain parts other than the user's body, such as the environment when the user took the photo. Considering that the purpose of image segmentation is to identify the user's spine region and the positions of each cone segment in the spine region, the parts other than the user's body are noise interference for image segmentation. Therefore, in the embodiment of the present application, the electronic device can first crop the original image, cropping the parts other than the user's body, leaving only the image region of the user's body.

[0082] Specifically, the electronic device obtains the original image data containing the patient's spine and cuts out the background area outside the body. This is because during the image acquisition stage, the background area outside the body is basically air and is not imaged; in addition, the scanning bed that may appear in the image is a fixed mechanical device, which can be removed through simple grayscale information and position information.

[0083] For standardization processing, image data containing only body areas are standardized. Image acquisition may also include some boundary points, or the difference between pixel points corresponding to different targets in the image is not too large. In this case, standardization can improve the accuracy of subsequent image segmentation.

[0084] 202. For a first image among multiple slice images, the electronic device inputs a three-dimensional image block consisting of the first image and its adjacent slice images into a first deep learning model, and the first deep learning model segments the first image based on the three-dimensional image block to obtain a first prediction result of the first image, where the first prediction result is used to indicate whether each pixel in the first image is a spinal region and the corresponding cone position.

[0085] After obtaining multiple slice images, for each first image, the electronic device can input the image block composed of the first image and the adjacent slice images into the first deep learning model. In order to fully extract the characteristics of the tissues and organs around the spine to assist in locating the cone position information, the first stage model (that is, the first deep learning model) inputs the processed slice image and its adjacent slices to form a three-dimensional image block. In this way, on the one hand, global image information can be obtained, and on the other hand, missed judgments caused by the lack of important information at certain levels can be avoided.

[0086] In step 202, not only can each pixel point be identified as being in the spinal region, but also the cone segment to which the pixel point belongs can be identified. The first deep learning model performs segmentation based on the slice image obtained from the original image, taking into account the global information of the image and avoiding missed judgments due to the lack of important information at certain levels.

[0087] In some embodiments, the first deep learning model can segment the first image and output a first prediction result of the first image. In other embodiments, the first deep learning model can segment the first image and output whether each pixel in the first image is a spinal region, and then obtain the cone position corresponding to each pixel in the first image through an image post-processing method. The following description takes the first prediction result obtained by direct segmentation of the first deep learning model as an example.

[0088] The first deep learning model may be a pre-trained model for image segmentation, and the first deep learning model may be trained based on a large number of sample original images and target prediction results of the sample original images.

[0089] In some embodiments, the training process of the first deep learning model can be: obtaining a sample original image and a target prediction result of the sample original image; obtaining multiple sample slice images based on the sample original image, and then inputting each sample slice image and an image block consisting of adjacent sample slice images into an initial first deep learning model, and the initial first deep learning model segments the sample slice image based on the input image block, outputs the prediction result of the sample slice image, and then obtains a first loss function based on the prediction result of the sample slice image and its target prediction result, and then adjusts the model parameters of the initial first deep learning model based on the first loss function, and then repeats the above steps of inputting the sample original image, image segmentation, and adjusting the model parameters until the first target condition is reached and the first deep learning model is obtained.

[0090] Among them, the first target condition can be set by relevant technical personnel based on experience or needs. For example, the first target condition can be that the number of iterations reaches a first number. The specific value of the first number can be set by relevant technical personnel based on needs or experience, or the first target condition can be that the first loss function converges. The embodiments of the present application are not limited to this.

[0091] For the first deep learning model, in step 202, the electronic device inputs the standardized data into the model of the first stage. The neural network of the first deep learning model can select various suitable deep neural network models, such as Unet, TransUnet, SwinUNETR, etc., and the model structure can also be designed by relevant technical personnel, or any model with image segmentation function can be directly used. Of course, it is also possible to further increase or reduce the intermediate operation layer based on personal experience, such as introducing other advanced technologies such as attention mechanisms to improve the prediction accuracy and generalization ability of the model. When the model architecture design is large enough, the learning ability of the model can learn accurate spinal segmentation and labeling, so the embodiment of the present application does not constrain the first deep learning model to a certain model architecture.

[0092] For the first loss function, the first loss function can also be called an accuracy evaluation function, and the first loss function can be designed by relevant technical personnel according to needs or experience. In some embodiments, the training process of the first deep learning model is implemented based on the first loss function, and the first loss function can be obtained based on the integration of multiple loss functions using multiple scales, and the weights of multiple loss functions in the first loss function are adaptively adjusted according to the training situation. That is, the first loss function can use a multi-scale, multiple loss function integration structure, and adaptively adjust the weights of multiple loss functions according to the training situation. Exemplarily, the multiple loss functions can consider but not only MSE (mean-square error), Dice coefficient, boundary loss and other loss functions.

[0093] Among them, mean square error, Dice coefficient is a measurement method for comparing the similarity of two samples. It can measure the degree of overlap between two sets and is often used in text similarity calculation, image processing, bioinformatics and other fields. The value range of the Dice coefficient is, the closer the value is to 1, the more similar the two sets are. Boundary loss aims to improve the model's detection and prediction accuracy of target edges. In this way, the mean square error can ensure the kernel loss, and the boundary loss can ensure the accuracy of the target edge. The accuracy of the model can be effectively improved through multiple loss functions.

[0094] 203. The electronic device inputs the first prediction result of each target number of adjacent cone segments in the first prediction result and the second image into a three-dimensional second deep learning model in order from one end to the other end of the spine. The second deep learning model segments the second image based on the first prediction result of each target number of cone segments, and outputs a second prediction result of the second image, where the second image is a local image area corresponding to each target number of cone segments in the first image, and the second prediction result is used to indicate whether each pixel in the second image is a spinal region and the corresponding cone segment position.

[0095] After the image segmentation in the previous stage, considering that the image segmentation in one stage is based on global information segmentation, the local segmentation may not be fine enough. For a simple end-to-end neural network, and the first stage is based on the whole slice segmentation, the accuracy of the segmentation, especially at the boundary, may not be accurate enough. Therefore, in an embodiment of the present application, a second deep learning model is provided, and the result obtained based on the image segmentation in the first stage can be input into the second deep learning model together with the original image, and the local image input is segmented in a certain order, so that both global information and local information can be taken into account to obtain a more accurate image segmentation result.

[0096] The input of the second-stage model includes both the original image and the predicted results of the first stage. The advantage is that the model can correct the results of the first stage based on local image details. This optimization method is also in line with the clinician's own reading habits, from overall to local analysis.

[0097] In some embodiments, the two ends of the spine are the cervical vertebrae and the coccyx, respectively. The order from one end of the spine to the other end can be from the cervical vertebrae to the coccyx (that is, from top to bottom), or from the coccyx to the cervical vertebrae (that is, from bottom to top). This embodiment of the present application is not limited to this.

[0098] In some embodiments, the target number can be set by relevant technical personnel according to needs or experience, and the embodiments of the present application do not limit this. For example, the target number can be 3 or 4, that is, in step 203, the results of the first stage prediction are sequentially (this order can be from the cervical vertebra to the coccyx, or from the coccyx to the cervical vertebra, that is, from top to bottom or from bottom to top) and the adjacent 2 cone sections or 3 cone section areas predicted in the first stage are taken as the input of the second stage each time. The input here includes the output of the first stage and the original image information of the corresponding local area. The output of the second stage model is the segmentation result of the new adjacent 2 cone sections or 3 cone sections and the corresponding cone section labels.

[0099] In some embodiments, the process of obtaining the second prediction result of the second image based on the first prediction result and the second image is performed serially in the order from one end to the other end of the spine; or, the process of obtaining the second prediction result of the second image based on the first prediction result and the second image is performed in parallel through multiple processes. That is, the process of image segmentation of the data of the target number of cone segments each time in step 203 can be performed serially in order or in parallel to improve processing efficiency. That is, the reasoning of the second stage model can be performed serially in order or in parallel to speed up the speed.

[0100] In some embodiments, the training process of the second deep learning model is based on at least one of the false positive rate of the segmentation result and the boundary loss function, and the first loss function is implemented, the first loss function is obtained based on the integration of multiple loss functions using multiple scales, and the weights of multiple loss functions in the first loss function are adaptively adjusted according to the training situation. That is, in addition to the loss function consistent with the first stage, the loss function (or accuracy evaluation function) of the second stage also needs to increase the constraints of the false positive rate and boundary loss of the classification label. Thereby strengthening the model's constraints on the correctness of boundaries and classifications, and obtaining an accurate segmentation result.

[0101] In some embodiments, the training process of the second deep learning model can be: obtaining multiple sample slice images of the sample original image, the prediction results of the sample original image obtained by the first deep learning model, and the target prediction results of the sample original image; then, in order from one end of the two ends of the spine to the other end, the prediction results of each target number of cone segments in the prediction results and the image area corresponding to the prediction results in the sample slice image are input into the three-dimensional initial second deep learning model, and the initial second deep learning model segments the image area based on the prediction results of each target number of cone segments, and outputs the prediction results of the image area. Then, based on the prediction results of the image area and its target prediction results, a second loss function is obtained, and then the model parameters of the initial second deep learning model are adjusted based on the second loss function, and then the above steps of inputting the sample original image, image segmentation, and adjusting the model parameters are repeated until the second target condition is reached, and the second deep learning model is obtained.

[0102] Among them, the second target condition can be set by relevant technical personnel based on experience or needs. For example, the second target condition can be that the number of iterations reaches a second number. The specific value of the second number can be set by relevant technical personnel based on needs or experience, or the second target condition can be that the second loss function converges. The embodiments of the present application are not limited to this.

[0103] In some embodiments, the first deep learning model and the second deep learning model can be jointly trained, so that the joint training of the two-stage models can combine the two models to segment the image more accurately. Specifically, the prediction results output by the initial first deep learning model can be input into the initial second deep learning model together with the sample slice image in the above order, and this is performed in each iteration. After the second deep learning model outputs the prediction results, the second loss function is obtained by the prediction results, and the first loss function and the second loss function are integrated into the target loss function. For example, the integration method can be weighted summation or other combination methods, which is not limited in the embodiments of the present application.

[0104] In some embodiments, after step 203, the electronic device may also add the second prediction result of the second image to the original image according to the position information of the pixel points to obtain a third prediction result of the original image, and the third prediction result of the original image includes the segmentation result and the marking result of the original image.

[0105] In some embodiments, the results predicted by the second stage model are placed back into the original image according to the position information to obtain the final segmentation and labeling results. In this process, if there is an inconsistency in the classification of overlapping pixels, local clustering of the surrounding pixels can be used to correct it (because the cone nodes are arranged in a top-down order). Therefore, in response to the presence of overlapping pixels in the second prediction result of the second image, and the second prediction results of the overlapping pixels in different second images are different, the electronic device can cluster the pixels around the overlapping pixels according to the second prediction results of the pixels around the overlapping pixels to obtain a third prediction result for the overlapping pixels.

[0106] In a specific example, if Figure 3 As shown, the original image data containing the patient's spine can be obtained, and then the background of the original image can be removed and standardized. The standardized data is then input into the first-stage neural network to preliminarily obtain the segmentation results and labels of each cone segment of the spine. Then, according to the order of the spine, the local image information and segmentation results of the cone segmented in the first stage can be input into the second-stage neural network to obtain more accurate segmentation results and labels. Then the second-stage results are placed back to the corresponding position of the original image, and the spine segmentation and labeling results corresponding to the final original image are output.

[0107] Through the above embodiments, the embodiments of the present application clearly provide an automatic method for automatic segmentation and labeling of the spine, that is, to segment the spinal region of the image obtained by clinical scanning and mark which part of the cone segment it is. It can save the time of clinicians to judge which cone segment a certain cone segment is, thereby improving work efficiency. Furthermore, we propose a two-stage method based on deep learning, because it is very easy to judge the position of the cone segment with the help of surrounding tissues and organs, but it is not conducive to the refined segmentation of the cone segment; if you only focus on local spinal segmentation, it is easy to misclassify the cone segment. Therefore, the two-stage segmentation and labeling method proposed here can take advantage of each other and obtain the best result. Furthermore, according to the prior information that the spine is arranged in order, the corresponding loss function in the model training is strengthened, so that the model can more accurately segment each independent cone segment with the correct upper and lower order.

[0108] The advantages of the embodiments of the present application are as follows:

[0109] 1) This method can automatically segment and mark each cone segment of the spine. Compared with many existing methods that only segment the spine area in the image, this method can segment the spine and accurately mark which cone segment each cone segment corresponds to;

[0110] 2) Two-stage more accurate segmentation: Compared with the end-to-end algorithms proposed by most algorithms, which can only take into account a part of the global information or the local information, this two-stage method is more in line with the reading habits of clinicians. First, the approximate area and category of each cone segment are determined, and then the boundary of each cone segment is locally refined, so as to achieve a more accurate segmentation result;

[0111] 3) Emphasizes the prior knowledge of the order of the spine from the cervical vertebrae to the thoracic vertebrae, lumbar vertebrae, sacrum and coccyx. In the model training stage, the prior information is used to constrain the loss function in a targeted manner, thereby further improving the model accuracy. Compared with other methods that use general model training loss functions, this method introduces prior knowledge.

[0112] In an embodiment of the present application, two deep learning models are used to perform two-stage image segmentation on the original image containing the spine, and the spine in the medical image is fully automatically identified. Each cone segment is automatically segmented and the position of each cone segment is identified, providing clinicians with a new method for spinal disease and surgical planning, which is more accurate and faster. The first stage is based on global information segmentation, and the second stage is based on local information segmentation. It can take into account global information or a part of local information, which is more in line with the clinician's reading habits, thereby achieving a more accurate segmentation result. In the local segmentation process, the order of cone segment arrangement between the spine and the cervical vertebra is emphasized, so that the position of each cone segment is obtained by fine segmentation and identification, further improving the model accuracy and image segmentation accuracy.

[0113] All the above optional technical solutions can be combined in any way to form optional embodiments of the present application, and will not be described in detail here.

[0114] Figure 4 is a structural diagram of a spinal segmentation device based on deep learning provided in an embodiment of the present application, see Figure 4 , the device comprises:

[0115] An acquisition module 401 is used to acquire multiple slice images based on the original image containing the spine;

[0116] A first segmentation module 402 is used for inputting, for a first image among the multiple slice images, a three-dimensional image block composed of the first image and its adjacent slice images into a first deep learning model, and the first deep learning model segments the first image based on the three-dimensional image block to obtain a first prediction result of the first image, wherein the first prediction result is used to indicate whether each pixel in the first image is a spinal region and a corresponding cone position;

[0117] The second segmentation module 403 is used to input the first prediction result of each target number of adjacent cone segments in the first prediction result and the second image into a three-dimensional second deep learning model in order from one end to the other end of the spine, and the second deep learning model segments the second image based on the first prediction result of each target number of cone segments, and outputs a second prediction result of the second image, where the second image is a local image area corresponding to each target number of cone segments in the first image, and the second prediction result is used to indicate whether each pixel in the second image is a spinal region and the corresponding cone segment position.

[0118] In some embodiments, the acquisition module 401 is used to preprocess the original image to obtain a first image, and the preprocessing includes cropping and / or standardization, and the cropping is to crop the background area outside the body in the original image according to grayscale information and / or position information.

[0119] In some embodiments, the process of obtaining the second prediction result of the second image based on the first prediction result and the second image is performed serially in an order from one end to the other end of the spine; or, the process of obtaining the second prediction result of the second image based on the first prediction result and the second image is performed in parallel through multiple processes.

[0120] In some embodiments, the training process of the first deep learning model is implemented based on a first loss function, the first loss function is obtained by integrating multiple loss functions using multiple scales, and the weights of multiple loss functions in the first loss function are adaptively adjusted according to the training situation.

[0121] In some embodiments, the training process of the second deep learning model is based on at least one of the false positive rate of the segmentation result and the boundary loss function, and a first loss function, wherein the first loss function is obtained based on the integration of multiple loss functions using multi-scales, and the weights of multiple loss functions in the first loss function are adaptively adjusted according to the training situation.

[0122] In some embodiments, the apparatus further comprises:

[0123] An adding module is used to add the second prediction result of the second image to the original image according to the position information of the pixel points to obtain a third prediction result of the original image, and the third prediction result of the original image includes a segmentation result and a marking result of the original image.

[0124] In some embodiments, the adding module is used to:

[0125] In response to the presence of overlapping pixels in the second prediction result of the second image, and the second prediction results of the overlapping pixels in different second images are different, the pixels around the overlapping pixels are clustered according to the second prediction results of the pixels around the overlapping pixels to obtain a third prediction result for the overlapping pixels.

[0126] The device provided in the embodiment of the present application uses two deep learning models to perform two-stage image segmentation on the original image containing the spine, fully automatically identifies the spine in the medical image, and automatically segments each cone segment and identifies the position of each cone segment, providing a new method for clinicians when performing spinal diseases and surgical planning, which is more accurate and faster. The first stage is based on global information segmentation, and the second stage is based on local information segmentation. It can take into account a part of the global information or the local information, which is more in line with the clinician's reading habits, thereby achieving a more accurate segmentation result. In the local segmentation process, the order of the cone segments between the spine and the cervical vertebrae is emphasized, so that the position of each cone segment is obtained by fine segmentation and identification, further improving the model accuracy and image segmentation accuracy.

[0127] It should be noted that: the spinal segmentation device based on deep learning provided in the above embodiment is only illustrated by the division of the above functional modules when trading stocks based on deep neural networks. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the spinal segmentation device based on deep learning is divided into different functional modules to complete all or part of the functions described above. In addition, the spinal segmentation device based on deep learning provided in the above embodiment and the spinal segmentation method embodiment based on deep learning belong to the same concept. The specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0128] Figure 5 It is a structural diagram of an electronic device provided in an embodiment of the present application. The electronic device 500 may have relatively large differences due to different configurations or performances, and may include one or more processors (Central Processing Units, CPU) 501 and one or more memories 502, wherein the memory 502 stores at least one computer program, and the at least one computer program is loaded and executed by the processor 501 to implement the deep learning-based spine segmentation method provided in the above-mentioned various method embodiments. The electronic device may also include other components for realizing the functions of the device, for example, the electronic device may also have components such as a wired or wireless network interface and an input and output interface for input and output. The embodiments of the present application are not described in detail here.

[0129] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including at least one computer program, wherein the at least one computer program is executable by a processor to perform the deep learning-based spine segmentation method in the above embodiment. For example, the computer-readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a compact disc (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, etc.

[0130] In an exemplary embodiment, a computer program product or a computer program is also provided, the computer program product or the computer program includes one or more program codes, and the one or more program codes are stored in a computer-readable storage medium. One or more processors of an electronic device can read the one or more program codes from the computer-readable storage medium, and the one or more processors execute the one or more program codes, so that the electronic device can perform the above-mentioned spine segmentation method based on deep learning.

[0131] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0132] It should be understood that determining B based on A does not mean determining B only based on A. B can also be determined based on A and / or other information.

[0133] A person of ordinary skill in the art will understand that all or part of the steps to implement the above embodiments can be completed by hardware, or can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. The above-mentioned storage medium can be a read-only memory, a disk or an optical disk, etc.

[0134] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A spine segmentation method based on deep learning, characterized in that: The method comprises: Based on the original image including the spine, a plurality of slice images are acquired; For a first image among the multiple slice images, a three-dimensional image block composed of the first image and its adjacent slice images is input into a first deep learning model, and the first deep learning model segments the first image based on the three-dimensional image block to obtain a first prediction result of the first image, where the first prediction result is used to indicate whether each pixel in the first image is a spinal region and a corresponding cone position; In order from one end to the other end of the spine, the first prediction result of each target number of adjacent cone segments in the first prediction result and the second image are input into a three-dimensional second deep learning model. The second deep learning model segments the second image based on the first prediction result of each target number of cone segments, and outputs a second prediction result of the second image. The second image is a local image area corresponding to each target number of cone segments in the first image, and the second prediction result is used to indicate whether each pixel in the second image is a spinal region and the corresponding cone segment position.

2. The spine segmentation method based on deep learning according to claim 1, characterized in that: The method of acquiring a plurality of slice images based on the original image containing the spine includes: The original image is preprocessed to obtain a first image, wherein the preprocessing includes cropping and / or standardization, wherein the cropping is to crop the background area other than the body in the original image according to grayscale information and / or position information.

3. The spine segmentation method based on deep learning according to claim 1, characterized in that: The process of obtaining the second prediction result of the second image based on the first prediction result and the second image is performed serially in an order from one end to the other end of the spine; or, the process of obtaining the second prediction result of the second image based on the first prediction result and the second image is performed in parallel through multiple processes.

4. The spine segmentation method based on deep learning according to claim 1, characterized in that: The training process of the first deep learning model is implemented based on a first loss function, and the first loss function is obtained by integrating multiple loss functions using multiple scales. The weights of multiple loss functions in the first loss function are adaptively adjusted according to the training situation.

5. The spine segmentation method based on deep learning according to claim 1, characterized in that: The training process of the second deep learning model is based on at least one of the false positive rate of the segmentation result and the boundary loss function, and the first loss function is implemented. The first loss function is obtained by integrating multiple loss functions using multiple scales, and the weights of multiple loss functions in the first loss function are adaptively adjusted according to the training situation.

6. The spine segmentation method based on deep learning according to claim 1, characterized in that: The method further comprises: The second prediction result of the second image is added to the original image according to the position information of the pixel points to obtain a third prediction result of the original image, wherein the third prediction result of the original image includes a segmentation result and a labeling result of the original image.

7. The spine segmentation method based on deep learning according to claim 6, characterized in that: The step of adding the second prediction result of the second image to the original image according to the position information of the pixel points to obtain the third prediction result of the original image includes: In response to the presence of overlapping pixels in the second prediction result of the second image, and the second prediction results of the overlapping pixels are different in different second images, the pixels with the same second prediction results around the overlapping pixels are clustered according to the second prediction results of the pixels around the overlapping pixels to obtain a third prediction result for the overlapping pixels.

8. A spinal segmentation device based on deep learning, characterized in that: The device comprises: An acquisition module, used for acquiring a plurality of slice images based on an original image including a spine; A first segmentation module is used for inputting, for a first image among the multiple slice images, a three-dimensional image block composed of the first image and its adjacent slice images into a first deep learning model, and the first deep learning model segments the first image based on the three-dimensional image block to obtain a first prediction result of the first image, wherein the first prediction result is used to indicate whether each pixel in the first image is a spinal region and a corresponding cone position; A second segmentation module is used to input the first prediction results of each target number of adjacent cone segments in the first prediction results and the second image into a three-dimensional second deep learning model in order from one end to the other end of the spine. The second deep learning model segments the second image based on the first prediction results of each target number of cone segments, and outputs a second prediction result of the second image, where the second image is a local image area corresponding to each target number of cone segments in the first image, and the second prediction result is used to indicate whether each pixel in the second image is a spinal region and the corresponding cone segment position.

9. An electronic device, characterized in that: The electronic device includes one or more processors and one or more memories, wherein at least one computer program is stored in the one or more memories, and the at least one computer program is loaded and executed by the one or more processors to implement the deep learning-based spinal segmentation method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The storage medium stores at least one computer program, and the at least one computer program is loaded and executed by the processor to implement the deep learning-based spine segmentation method as described in any one of claims 1 to 7.

Images