A bone density measurement method and device based on deep learning

Abstract

The present application provides a bone density measurement method and device based on deep learning, the method comprising: inputting a medical image to be measured into a spine recognition model to obtain a lumbar image to be measured; inputting a lumbar image to be measured into a bone density measurement model to obtain the bone density of the lumbar vertebra; the spine recognition model is obtained by training based on a first sample image, the first sample image is at least annotated with a recognition result of the lumbar vertebra; the bone density measurement model is obtained by training based on a second sample image, the second sample image is at least annotated with a bone density of the lumbar vertebra. In the present application, bone density measurement is achieved through a pre-trained spine recognition model and a bone density measurement model, thereby eliminating the need for professional instruments and saving unnecessary costs; and through two independent deep convolution models, data reuse and linkage with corresponding software are achieved.

Description

Technical Field

[0001] The present application relates to the technical field of medical image processing, and in particular to a bone density measurement method and device based on deep learning. Background Art

[0002] BMD (Bone mineral density) refers to the amount of bone mineral per unit area of ​​bone. Bone density is an important indicator of human bone metabolism. It can reflect the changes in bone mass and the degree of osteoporosis in the human body. It is also an important basis for predicting the risk of fractures. At the same time, bone density is also one of the important planning data for joint replacement, spinal implantation and disease diagnosis.

[0003] Existing bone density measurements are performed using bone density measuring instruments, but this detection method requires dedicated instruments, which not only adds additional costs, but the test results cannot be linked to existing preoperative planning software. Summary of the invention

[0004] The problem solved by this application is that currently bone density testing requires dedicated instruments.

[0005] To solve the above problems, the first aspect of the present application provides a bone density measurement method based on deep learning, including:

[0006] Inputting the medical image to be measured into the spine recognition model to obtain the lumbar image to be measured;

[0007] Inputting the lumbar vertebra image to be measured into the bone density measurement model to obtain the lumbar vertebra bone density;

[0008] The spine recognition model is obtained by training based on a first sample image, and the first sample image is at least marked with a recognition result of the lumbar vertebra; the bone density measurement model is obtained by training based on a second sample image, and the second sample image is at least marked with a bone density of the lumbar vertebra.

[0009] The second aspect of the present application provides a bone density measurement device based on deep learning, which includes:

[0010] A lumbar vertebra recognition module, which is used to input the medical image to be measured into the spine recognition model to obtain the lumbar vertebra image to be measured;

[0011] A bone density measurement module, which is used to input the lumbar vertebra image to be measured into the bone density measurement model to obtain the lumbar vertebra bone density;

[0012] The spine recognition model is obtained by training based on a first sample image, and the first sample image is at least marked with a recognition result of the lumbar vertebra; the bone density measurement model is obtained by training based on a second sample image, and the second sample image is at least marked with a bone density of the lumbar vertebra.

[0013] A third aspect of the present application provides an electronic device, comprising: a memory and a processor;

[0014] The memory is used to store programs;

[0015] The processor, coupled to the memory, is configured to execute the program to:

[0016] Inputting the medical image to be measured into the spine recognition model to obtain the lumbar image to be measured;

[0017] Inputting the lumbar vertebra image to be measured into the bone density measurement model to obtain the lumbar vertebra bone density;

[0018] The spine recognition model is obtained by training based on a first sample image, and the first sample image is at least marked with a recognition result of the lumbar vertebra; the bone density measurement model is obtained by training based on a second sample image, and the second sample image is at least marked with a bone density of the lumbar vertebra.

[0019] A fourth aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the program is executed by a processor to implement the above-mentioned deep learning-based bone density measurement method.

[0020] In this application, bone density measurement is achieved through pre-trained spine recognition model and bone density measurement model, which eliminates the need for professional instruments and saves unnecessary costs; and through two independent deep convolution models, data reuse and linkage with corresponding software are achieved.

[0021] In this application, the segmented lumbar image obtained by the spine recognition model is an important required image in the preoperative planning software, thereby realizing data reuse and completing the linkage with the preoperative planning software.

[0022] In the present application, the bone density of different local bones can be measured by using a spine identification model and a bone density measurement model that are independent of each other. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 is a model architecture diagram of a bone density measurement method according to an embodiment of the present application;

[0024] Figure 2 is a flow chart of a bone density measurement method according to an embodiment of the present application;

[0025] Figure 3 A flowchart of a bone density measurement method model training according to an embodiment of the present application;

[0026] Figure 4 A flowchart of lumbar vertebra identification according to the bone density measurement method of an embodiment of the present application;

[0027] Figure 5 It is a schematic diagram of edge operator processing of the bone density measurement method according to an embodiment of the present application;

[0028] Figure 6 is a structural block diagram of a bone density measuring device according to an embodiment of the present application;

[0029] Figure 7 is a structural block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0030] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application are described in detail below in conjunction with the accompanying drawings. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0031] It should be noted that, unless otherwise specified, the technical terms or scientific terms used in this application should have the common meanings understood by technicians in the field to which this application belongs.

[0032] BMD (Bone mineral density) refers to the amount of bone mineral per unit area of ​​bone. Bone density is an important indicator of human bone metabolism. It can reflect the changes in bone mass and the degree of osteoporosis in the human body. It is also an important basis for predicting the risk of fractures. At the same time, bone density is also one of the important planning data for joint replacement, spinal implantation and disease diagnosis.

[0033] Existing bone density measurements are performed using bone density measuring instruments, but this detection method requires dedicated instruments, which adds additional costs.

[0034] Moreover, for the existing preoperative planning software, bone density can only be obtained by reading bone density from a dedicated instrument. It is impossible to achieve linkage methods such as data reuse in the bone density processing process. The bone density results obtained are also overall bone density results, and it is impossible to obtain local bone density results, such as the bone density results of the thoracic spine.

[0035] In response to the above problems, the present application provides a new bone density measurement solution based on deep learning, which can solve the problem that the current bone density measurement requires special instruments and cannot be linked with preoperative planning software by first performing bone segmentation and then measuring the bone density of the segmented thoracic spine.

[0036] The present application embodiment provides a bone density measurement method based on deep learning, and the specific scheme of the method is as follows: Figure 1-Figure 5 As shown, the method can be performed by a bone density measurement device based on deep learning, and the bone density measurement device based on deep learning can be integrated into electronic devices such as computers, servers, computers, server clusters, and data centers. Figure 1 , Figure 2 As shown, it is a flow chart of a bone density measurement method based on deep learning according to an embodiment of the present application; wherein the bone density measurement method based on deep learning includes:

[0037] S200, inputting the medical image to be measured into a spine recognition model to obtain a lumbar vertebra image to be measured;

[0038] In the present application, the medical image to be measured is a medical image containing the lumbar spine, and an image of the lumbar spine can be obtained based on the medical image. The medical image includes, for example, an X-ray film, a computed tomography (CT) image, a magnetic resonance imaging (MRI) image, or a composite image of the above multiple images.

[0039] In this application, the spine recognition model is pre-trained.

[0040] S300, inputting the lumbar vertebra image to be measured into a bone density measurement model to obtain the lumbar vertebra bone density;

[0041] The spine recognition model is obtained by training based on a first sample image, and the first sample image is at least marked with a recognition result of the lumbar vertebra; the bone density measurement model is obtained by training based on a second sample image, and the second sample image is at least marked with a bone density of the lumbar vertebra.

[0042] In the present application, the bone density measurement model is also obtained by pre-training; by pre-training the spine recognition model and the bone density measurement model, the two models are combined to perform bone density measurement. Compared with the method of training the spine recognition model-bone density measurement model as a whole model, this method greatly reduces the difficulty of model training and improves the accuracy of model training.

[0043] In the present application, the first sample image and the second sample image are used to indicate the lumbar vertebrae of any object. Similarly, the medical image to be measured and the lumbar vertebrae image to be measured are also used to indicate the lumbar vertebrae of any object.

[0044] In the present application, the object is used to indicate a patient, and the first sample image, the second sample image, the medical image to be measured, and the lumbar vertebra image to be measured may be medical images obtained by scanning the patient.

[0045] In the present application, the first sample image and the medical image to be measured are images of the same type, which can specifically be a full-body image, a half-body image, a partial body image, etc., as long as they include an image of the lumbar spine; similarly, the second sample image and the lumbar spine image to be measured are images of the same type, which can specifically be a full-body image, a half-body image, a partial body image, etc., as long as they include an image of the lumbar spine.

[0046] Preferably, the second sample image and the lumbar vertebra image to be measured are partial images of the lumbar vertebra after image segmentation, or full-body images, half-body images, partial human body images, etc., with the lumbar vertebra parts marked.

[0047] In this application, bone density measurement is achieved through pre-trained spine recognition model and bone density measurement model, which eliminates the need for professional instruments and saves unnecessary costs; and through two independent deep convolution models, data reuse and linkage with corresponding software are achieved.

[0048] It should be noted that the segmented lumbar spine image obtained through the spine recognition model is an important required image in the preoperative planning software, thereby realizing data reuse and completing the linkage with the preoperative planning software.

[0049] In the present application, the bone density of different local bones can be measured by using a spine identification model and a bone density measurement model that are independent of each other.

[0050] Combination Figure 3 As shown, in one embodiment, the S200, before inputting the medical image to be measured into the spine recognition model to obtain the lumbar image to be measured, further includes:

[0051] S101, acquiring a first sample image and a second sample image, wherein the first sample image is annotated with the second sample image, and the second sample image is annotated with the bone density of the lumbar vertebrae;

[0052] S102, iteratively training the spine recognition model based on the first sample image until the overall loss of the spine recognition model converges;

[0053] The first sample image and the annotated second sample image are used to train a spine recognition model, and the annotated second sample image is a supervisory signal.

[0054] The spine recognition model is iteratively trained based on the first sample image until the overall loss of the spine recognition model converges, thereby obtaining a trained spine recognition model.

[0055] In this application, the specific training process of the spine recognition model may be:

[0056] Input a first sample image into a spine recognition model to obtain a predicted first sample result; calculate the overall loss of the spine recognition model based on the first sample result and a second sample image marked on the first sample image; and iterate the spine recognition model based on the overall loss until the overall loss converges.

[0057] S103: Iteratively train the bone density measurement model based on the second sample image until the overall loss of the bone density measurement model converges.

[0058] The second sample image annotated on the first sample image and the bone density of the lumbar vertebra corresponding to the second sample image are used to train the bone density measurement model. The second sample image and the corresponding bone density of the lumbar vertebra are used as input data, and the bone density of the lumbar vertebra is used as a supervisory signal.

[0059] The spine recognition model is iteratively trained based on the second sample image until the overall loss of the spine recognition model converges, thereby obtaining a trained bone density measurement model.

[0060] In this application, the specific training process of the bone density measurement model may be:

[0061] The second sample image is input into the bone density measurement model to obtain a predicted second sample result; based on the second sample result and the bone density of the lumbar vertebra marked on the second sample image, the overall loss of the bone density measurement model is calculated; based on the overall loss, the bone density measurement model is iterated until the overall loss converges.

[0062] It should be noted that the training data of the bone density measurement model only needs to be the second sample image and the corresponding lumbar bone density, and is not limited to the second sample image marked on the first sample image. The second sample image marked on the first sample image is used in this application to achieve the effect of reusing samples, so as to improve the coordination of training and reduce the collection of sample data.

[0063] In the present application, the first sample image is reused and directly used as a sample to independently train the spine recognition model and the bone density measurement model, thereby avoiding the error propagation and error increase caused by the linkage training.

[0064] In the present application, the spine recognition model and the bone density measurement model are trained separately by using the same set of first sample images, and the training samples are reused to increase the homology performance and compatibility between the trained spine recognition model and the bone density measurement model.

[0065] Combination Figure 4As shown, in one embodiment, the step S200, inputting the medical image to be measured into the spine recognition model to obtain the lumbar image to be measured, includes:

[0066] S201, inputting the medical image to be measured into the downsampling structure of the spine recognition model to obtain a plurality of downsampling feature maps with gradually decreasing sizes;

[0067] In one implementation, the medical image to be measured is input into the downsampling structure of the spine recognition model to obtain a plurality of downsampling feature maps with gradually decreasing sizes, which may specifically include:

[0068] The medical image to be measured is subjected to convolution processing and then to pooling processing to obtain a first down-sampled feature map; the first down-sampled feature map is subjected to convolution processing and then to pooling processing to obtain a second down-sampled feature map; and the convolution processing and pooling processing are performed in sequence to obtain multiple down-sampled feature maps with gradually decreasing sizes.

[0069] In the present application, the convolution processing in the downsampling structure can be Conv3x3+BN+Relu processing to obtain the corresponding feature extraction; the pooling processing in the downsampling structure can be maximum pooling processing, through which the number of channels is changed (the number of channels is doubled) to achieve the purpose of reducing the size.

[0070] In this application, Conv3x3+BN+Relu is a 3x3 convolutional layer, a normalization layer (BN, BatchNormalization) and a Relu function connected in sequence.

[0071] It should be noted that after the medical image to be measured is convolved, it is pooled to obtain the first down-sampled feature map. In the pooling process in this step, the size reduced in the specific pooling process is determined according to the preset size of the first down-sampled feature map and the medical image to be measured. In other words, the size of the first down-sampled feature map is fixed. If the size of the medical image to be measured changes, the size reduced by the pooling process in this step needs to be adjusted.

[0072] Preferably, after the medical image to be measured is convolved, 1x1 convolution is performed to obtain a first down-sampled feature map, thereby reducing the size range requirement for the medical image to be measured and ensuring the size of the first down-sampled feature map.

[0073] S202, inputting the downsampled feature map with the smallest size into a Transformer structure to obtain a global feature map;

[0074] In this application, combined Figure 1 As shown, after downsampling the third downsampled feature map, the hidden unit layer ( Figure 1Hidden Feature shown in ), and then perform linear projection processing ( Figure 1 After the LinearProjection described in , it is input into the Transformer structure to obtain the global feature map.

[0075] It should be noted that the Transformer structure, as a standard for natural language processing tasks, has certain differences in processing image data. In this application, the downsampled feature map of image properties is processed into a natural language-like form after hidden unit layers and linear projection processing, so that it can be directly input into the Transformer structure for processing.

[0076] In the present application, the processing process of the Transformer structure after receiving the corresponding information is to embed the position information through the embedding layer, so as to form a front-to-back order relationship for the natural language-like form (convert the position relationship into a front-to-back order relationship). The specific processing process of the Transformer structure for the natural language-like form with a front-to-back order relationship can refer to the prior art and will not be repeated in this application.

[0077] It should be noted that embedding the position information through the embedding layer can be completed in the hidden unit layer or after inputting the Transformer structure, which is not limited in this application.

[0078] It should also be noted that the Transformer structure is the standard for natural language processing tasks, and its output is also natural language. In this application, the output result of the Transformer structure is directly processed into the hidden unit layer ( Figure 1 The Hidden Feature shown in ) is then reshaped ( Figure 1 After reshape described in , we get the global feature map ( Figure 1 (D, H / 16, W / 16) shown in .

[0079] In this application, the output result of the Transformer structure is processed by hidden unit layer and reshaping, so as to convert the content similar to natural language form into feature map form.

[0080] In this application, by setting the hidden unit layer, linear projection layer and reshaping layer, the Transformer structure is perfectly applied to the field of image processing, greatly improving the globality of the global feature map and the accuracy of model training.

[0081] In one embodiment, after the third downsampled feature map is downsampled, the hidden unit layer is directly performed, and then linear projection processing is performed before inputting into the Transformer structure. The hidden unit layer splits the downsampled feature map (the fourth downsampled feature map) into multiple patches (image blocks), and each patch (image block) is linearly projected and dimensionally reduced before inputting into the Transformer structure. In this way, on the one hand, pixel-level attention operations can be avoided, and on the other hand, the vector after dimension reduction is directly input into the Transformer structure, which retains the processing characteristics of the Transformer structure to the maximum extent.

[0082] In one implementation, each patch is subjected to linear projection dimensionality reduction and then input into the Transformer structure. One linear projection may be shared by all patches, or a corresponding linear projection may be set for each patch, thereby achieving further accuracy.

[0083] In one implementation, each patch is assigned a corresponding linear projection, and different linear projections are partially shared / soft-shared to reduce parameter freedom.

[0084] In one implementation, the output result of the Transformer structure is directly processed through a hidden unit layer and then reshaped to obtain a global feature map. The output result of the Transformer structure is a plurality of patches, and the plurality of patches are combined into a feature map through a hidden unit layer.

[0085] It should be noted that the output result of the Transformer structure contains position information, based on which the combination order of multiple patches can be determined to form the corresponding feature map.

[0086] In the present application, the combined feature map is converted into a feature map of a corresponding format through reshaping processing, and the feature map is the global feature map.

[0087] S203, performing edge operator processing on the multiple downsampled feature maps to obtain multiple edge operator feature maps;

[0088] In the present application, edge features of the downsampled feature map can be highlighted through edge operator processing, thereby increasing the accuracy of segmentation.

[0089] In this application, combined Figure 5 As shown, edge operator processing is performed on the multiple downsampled feature maps to obtain multiple edge operator feature maps, including:

[0090] Based on the edge operator feature map, the downsampled feature map is convolved to obtain the edge convolution feature map; the downsampled feature map and the edge convolution feature map are added to obtain the edge operator feature map, thereby achieving the highlighting and strengthening of edge features.

[0091] In the present application, edge operator processing is performed on the multiple downsampled feature maps, that is, edge operator processing is performed on the multiple downsampled feature maps to obtain corresponding edge operator feature maps.

[0092] In one implementation, the edge operator is a Frei-Chen operator.

[0093] Among them, the Frei-Chen operator is a multi-scale filter used for edge detection, which filters the image horizontally, vertically and diagonally.

[0094] In one implementation, the Frei-Chen operator is:

[0095] -1 / 9 -2 / 9 -3 / 9 -2 / 9 -1 / 9 -2 / 9 -6 / 9 -12 / 9 -6 / 9 -2 / 9 -3 / 9 -12 / 9 88 / 9 -12 / 9 -3 / 9 -2 / 9 -6 / 9 -12 / 9 -6 / 9 -2 / 9 -1 / 9 -2 / 9 -3 / 9 -2 / 9 -1 / 9

[0096] In the present application, the Frei-Chen operator is set as a 5x5 convolution kernel, the central element of which is 88 / 9, and adjacent elements have different weights according to their positions.

[0097] In the present application, by performing a convolution operation on the convolution kernel and the image, the gradient information of the image in different directions can be directly obtained simultaneously, thereby achieving a better edge detection effect.

[0098] S204, inputting the plurality of edge operator feature maps and the global feature map into an upsampling structure to obtain a segmentation mask map, and determining the lumbar vertebra image to be measured based on the segmentation mask map.

[0099] Combination Figure 1 As shown, a plurality of edge operator feature maps and the global feature map are input into an upsampling structure to obtain a segmentation mask map, including:

[0100] Perform convolution processing and upsampling processing on the global feature map to obtain a third upsampled feature map;

[0101] The edge operator feature map corresponding to the third down-sampled feature map is concatenated with the three up-sampled feature maps to obtain a third concatenated feature map;

[0102] Performing convolution processing and upsampling processing on the third concatenated feature map to obtain a second upsampled feature map;

[0103] Concatenate the edge operator feature map corresponding to the second down-sampled feature map with the second up-sampled feature map to obtain a second concatenated feature map;

[0104] Performing convolution processing and upsampling processing on the second concatenated feature map to obtain a first upsampled feature map;

[0105] Concatenate the edge operator feature map corresponding to the first down-sampled feature map with the first up-sampled feature map to obtain a first concatenated feature map;

[0106] The first concatenated feature map is subjected to convolution processing and upsampling processing to obtain the segmentation mask map.

[0107] Combination Figure 1 As shown in the figure, the global feature map is (D, H / 16, W / 16), the global feature map after convolution processing is (512, H / 16, W / 16), the third spliced ​​feature map after convolution processing is (256, H / 8, W / 8), the second spliced ​​feature map after convolution processing is (128, H / 4, W / 4), the first spliced ​​feature map after convolution processing is (64, H / 2, W / 2), and the segmentation mask map is (16, H, W).

[0108] In the present application, the segmentation mask image segments all the bone parts in the medical image to be measured, and the lumbar vertebrae part is read from the segmentation mask image as the lumbar vertebrae image to be measured.

[0109] In one embodiment, the lumbar vertebrae image to be measured includes a first lumbar vertebra, a second lumbar vertebra, a third lumbar vertebra, and a fourth lumbar vertebra.

[0110] It should be noted that the lumbar vertebrae of the human body are usually large, with spinous processes extending horizontally backwards in a plate-like manner, and the gaps between adjacent spinous processes are wide, which can be used for lumbar puncture. The articular processes and articular surfaces are generally in the sagittal plane. Clinically, the human body has 5 lumbar vertebrae, namely the first lumbar vertebra, the second lumbar vertebra, the third lumbar vertebra, the fourth lumbar vertebra and the fifth lumbar vertebra. Each lumbar vertebra is generally composed of a vertebral body in the front and an appendage in the back. The morphology of these 5 lumbar vertebrae is different. The morphology of the first lumbar vertebra is similar to that of the thoracic vertebra, while the fourth and fifth lumbar vertebrae are the largest lumbar vertebrae.

[0111] However, due to developmental abnormalities, some people may have only 4 lumbar vertebrae, clinically known as sacralization, while some people may have 6 lumbar vertebrae, known as sacralization.

[0112] In the present application, by setting the lumbar vertebrae images to be measured to include images of the first lumbar vertebra, the second lumbar vertebra, the third lumbar vertebra and the fourth lumbar vertebra, detection errors caused by human body abnormalities can be avoided while ensuring the accuracy of bone density measurement.

[0113] In one embodiment, the bone density measurement model includes multiple convolutional layers, fully connected layers and SoftMax-like layers arranged in sequence; in S300, the lumbar vertebra image to be measured is input into the bone density measurement model to obtain the bone density of the lumbar vertebra, and the lumbar vertebra image to be measured is sequentially convolved through multiple convolutional layers, and the lumbar vertebra bone density value is output after processing through the fully connected layer and the SoftMax-like layer.

[0114] It should be noted that in the present application, the bone density measurement model is a continuous value, which is different from the output of the classification network which is a discrete value.

[0115] In one embodiment, the activation function of the SoftMax-like layer is:

[0116]

[0117] Among them, SoftMax1(x) is the activation function, x i 、x j is the element in the input vector, i and j are the element numbers.

[0118] In this application, SoftMax is a mathematical function that is usually used to convert a set of arbitrary real numbers into real numbers representing probability distributions. It is essentially a normalization function that can convert a set of arbitrary real values ​​into probability values ​​between [0,1], because SoftMax converts them to values ​​between 0 and 1, so they can be interpreted as probabilities. If one of the inputs is small or negative, SoftMax turns it into a small probability, and if the input is large, it turns it into a large probability, but it will always remain between 0 and 1.

[0119] However, for the standard SoftMax function, since the input is mapped between 0 and 1, and the sum of all output values ​​is 1, this means that even if some input values ​​are very small, they will have a non-zero output value after being processed by the SoftMax function. This will also cause the noise to be amplified, resulting in the final output result being affected by greater noise.

[0120] In this application, a 1 is added to the denominator of the SoftMax-like function; this change means that when the input value is very small, their output value can be closer to zero. This allows the corresponding output to tend to zero when there is no valuable information to add, thus greatly reducing unnecessary noise.

[0121] An embodiment of the present application provides a bone density measurement device based on deep learning, which is used to execute the bone density measurement method based on deep learning described in the above content of the present application. The bone density measurement device based on deep learning is described in detail below.

[0122] like Figure 6 As shown, the bone density measurement device based on deep learning includes:

[0123] A lumbar vertebra recognition module 101, which is used to input the medical image to be measured into a spine recognition model to obtain a lumbar vertebra image to be measured;

[0124] A bone density measurement module 102, which is used to input the lumbar vertebra image to be measured into a bone density measurement model to obtain the bone density of the lumbar vertebra;

[0125] The spine recognition model is obtained by training based on a first sample image, and the first sample image is at least marked with a recognition result of the lumbar vertebra; the bone density measurement model is obtained by training based on a second sample image, and the second sample image is at least marked with a bone density of the lumbar vertebra.

[0126] In one embodiment, the device further comprises a model training module, which is used to:

[0127] Acquire a first sample image and a second sample image, wherein the first sample image is annotated with the second sample image, and the second sample image is annotated with the bone density of the lumbar vertebra; iteratively train the spine recognition model based on the first sample image until the overall loss of the spine recognition model converges; iteratively train the bone density measurement model based on the second sample image until the overall loss of the bone density measurement model converges.

[0128] In one embodiment, the lumbar vertebra identification module 101 is further used for:

[0129] The medical image to be measured is input into the downsampling structure of the spine recognition model to obtain multiple downsampling feature maps with gradually decreasing sizes; the downsampling feature map with the smallest size is input into the Transformer structure to obtain a global feature map; edge operator processing is performed on the multiple downsampling feature maps to obtain multiple edge operator feature maps; the multiple edge operator feature maps and the global feature map are input into the upsampling structure to obtain a segmentation mask map, and the lumbar image to be measured is determined based on the segmentation mask map.

[0130] In one implementation, the edge operator is a Frei-Chen operator.

[0131] In one implementation, the Frei-Chen operator is:

[0132] -1 / 9 -2 / 9 -3 / 9 -2 / 9 -1 / 9 -2 / 9 -6 / 9 -12 / 9 -6 / 9 -2 / 9 -3 / 9 -12 / 9 88 / 9 -12 / 9 -3 / 9 -2 / 9 -6 / 9 -12 / 9 -6 / 9 -2 / 9 -1 / 9 -2 / 9 -3 / 9 -2 / 9 -1 / 9

[0133] In one embodiment, the lumbar vertebrae image to be measured includes a first lumbar vertebra, a second lumbar vertebra, a third lumbar vertebra, and a fourth lumbar vertebra.

[0134] In one embodiment, the bone density measurement model includes a plurality of convolutional layers, fully connected layers and SoftMax-like layers arranged in sequence; the bone density measurement module 102 is further used for:

[0135] The lumbar vertebra images to be measured are sequentially convolved through multiple convolutional layers, and the lumbar vertebrae bone density values ​​are output after being processed through a fully connected layer and a SoftMax-like layer.

[0136] The deep learning-based bone density measurement device provided in the above-mentioned embodiment of the present application has a corresponding relationship with the deep learning-based bone density measurement method provided in the embodiment of the present application. Therefore, the specific content in the device has a corresponding relationship with the bone density measurement method. The specific content can refer to the records in the bone density measurement method, and will not be repeated in this application.

[0137] The deep learning-based bone density measurement device provided in the above-mentioned embodiments of the present application and the deep learning-based bone density measurement method provided in the embodiments of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the application programs stored therein.

[0138] The above describes the internal functions and structure of the deep learning-based bone density measurement device, such as Figure 7 As shown, in practice, the bone density measurement device based on deep learning can be implemented as an electronic device, including: a memory 301 and a processor 303.

[0139] The memory 301 may be configured to store programs.

[0140] In addition, the memory 301 may also be configured to store various other data to support operations on the electronic device. Examples of such data include instructions for any application or method operating on the electronic device, contact data, phone book data, messages, pictures, videos, etc.

[0141] The memory 301 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0142] The processor 303 is coupled to the memory 301 and is configured to execute a program in the memory 301 to:

[0143] Inputting the medical image to be measured into the spine recognition model to obtain the lumbar image to be measured;

[0144] Inputting the lumbar vertebra image to be measured into the bone density measurement model to obtain the lumbar vertebra bone density;

[0145] The spine recognition model is obtained by training based on a first sample image, and the first sample image is at least marked with a recognition result of the lumbar vertebra; the bone density measurement model is obtained by training based on a second sample image, and the second sample image is at least marked with a bone density of the lumbar vertebra.

[0146] In one implementation, the processor 303 is configured to:

[0147] Acquire a first sample image and a second sample image, wherein the first sample image is annotated with the second sample image, and the second sample image is annotated with the bone density of the lumbar vertebra; iteratively train the spine recognition model based on the first sample image until the overall loss of the spine recognition model converges; iteratively train the bone density measurement model based on the second sample image until the overall loss of the bone density measurement model converges.

[0148] In one implementation, the processor 303 is further configured to:

[0149] The medical image to be measured is input into the downsampling structure of the spine recognition model to obtain multiple downsampling feature maps with gradually decreasing sizes; the downsampling feature map with the smallest size is input into the Transformer structure to obtain a global feature map; edge operator processing is performed on the multiple downsampling feature maps to obtain multiple edge operator feature maps; the multiple edge operator feature maps and the global feature map are input into the upsampling structure to obtain a segmentation mask map, and the lumbar image to be measured is determined based on the segmentation mask map.

[0150] In one implementation, the edge operator is a Frei-Chen operator.

[0151] In one implementation, the Frei-Chen operator is:

[0152] -1 / 9 -2 / 9 -3 / 9 -2 / 9 -1 / 9 -2 / 9 -6 / 9 -12 / 9 -6 / 9 -2 / 9 -3 / 9 -12 / 9 88 / 9 -12 / 9 -3 / 9 -2 / 9 -6 / 9 -12 / 9 -6 / 9 -2 / 9 -1 / 9 -2 / 9 -3 / 9 -2 / 9 -1 / 9

[0153] In one embodiment, the lumbar vertebrae image to be measured includes a first lumbar vertebra, a second lumbar vertebra, a third lumbar vertebra, and a fourth lumbar vertebra.

[0154] In one embodiment, the bone density measurement model includes a plurality of convolutional layers, fully connected layers and SoftMax-like layers arranged in sequence; the processor 303 is further used for:

[0155] The lumbar vertebra images to be measured are sequentially convolved through multiple convolutional layers, and the lumbar vertebrae bone density values ​​are output after being processed through a fully connected layer and a SoftMax-like layer.

[0156] In the present application, the processor is also specifically used to execute all processes and steps of the above-mentioned deep learning-based bone density measurement method. The specific content can be referred to the records in the bone density measurement method, and will not be repeated in this application.

[0157] In this application, Figure 7 Only some components are shown schematically, which does not mean that the electronic device only includes Figure 7 Components shown.

[0158] The electronic device provided in this embodiment is based on the same inventive concept as the deep learning-based bone density measurement method provided in the embodiment of the present application, and has the same beneficial effects as the method adopted, run or implemented by the application stored therein.

[0159] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0160] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0161] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0162] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0163] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0164] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0165] The present application also provides a computer-readable storage medium corresponding to the deep learning-based bone density measurement method provided in the aforementioned embodiment, on which a computer program (i.e., a program product) is stored. When the computer program is run by a processor, it will execute the deep learning-based bone density measurement method provided in any of the aforementioned embodiments.

[0166] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0167] The computer-readable storage medium provided in the above-mentioned embodiments of the present application and the deep learning-based bone density measurement method provided in the embodiments of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the application program stored therein.

[0168] It should be noted that in the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known structures and technologies are not shown in detail so as not to obscure the understanding of this description.

[0169] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0170] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A bone density measurement method based on deep learning, characterized in that: include: Inputting the medical image to be measured into the spine recognition model to obtain the lumbar image to be measured; Inputting the lumbar vertebra image to be measured into the bone density measurement model to obtain the lumbar vertebra bone density; The spine recognition model is obtained by training based on a first sample image, and the first sample image is at least annotated with a recognition result of the lumbar vertebra; the bone density measurement model is obtained by training based on a second sample image, and the second sample image is at least annotated with a bone density of the lumbar vertebra; The bone density measurement model includes a plurality of convolutional layers, a fully connected layer and a SoftMax-like layer arranged in sequence; the lumbar vertebra image to be measured is input into the bone density measurement model to obtain the lumbar vertebra bone density, the lumbar vertebra image to be measured is sequentially convoluted through a plurality of convolutional layers, and the lumbar vertebra bone density value is output after being processed through the fully connected layer and the SoftMax-like layer; The activation function of the SoftMax-like layer is: Among them, SoftMax1(x) is the activation function, x i 、x j is the element in the input vector, i and j are the element numbers; The step of inputting the medical image to be measured into the spine recognition model to obtain the lumbar vertebra image to be measured comprises: Inputting the medical image to be measured into the downsampling structure of the spine recognition model to obtain a plurality of downsampling feature maps with gradually decreasing sizes; Input the downsampled feature map with the smallest size into the Transformer structure to obtain a global feature map; Performing edge operator processing on the multiple downsampled feature maps to obtain multiple edge operator feature maps; Inputting the plurality of edge operator feature maps and the global feature map into an upsampling structure to obtain a segmentation mask map, and determining the lumbar vertebra image to be measured based on the segmentation mask map; The step of inputting the plurality of edge operator feature maps and the global feature map into an upsampling structure to obtain a segmentation mask map comprises: Perform convolution processing and upsampling processing on the global feature map to obtain a third upsampled feature map; The edge operator feature map corresponding to the third down-sampled feature map is concatenated with the three up-sampled feature maps to obtain a third concatenated feature map; Performing convolution processing and upsampling processing on the third concatenated feature map to obtain a second upsampled feature map; Concatenate the edge operator feature map corresponding to the second down-sampled feature map with the second up-sampled feature map to obtain a second concatenated feature map; Performing convolution processing and upsampling processing on the second concatenated feature map to obtain a first upsampled feature map; Concatenate the edge operator feature map corresponding to the first down-sampled feature map with the first up-sampled feature map to obtain a first concatenated feature map; The first concatenated feature map is subjected to convolution processing and upsampling processing to obtain the segmentation mask map.

2. The bone density measurement method according to claim 1, characterized in that: Before inputting the medical image to be measured into the spine recognition model to obtain the lumbar image to be measured, the method further includes: Acquire a first sample image and a second sample image, wherein the first sample image is annotated with the second sample image, and the second sample image is annotated with the bone density of the lumbar vertebrae; Iteratively training the spine recognition model based on the first sample image until the overall loss of the spine recognition model converges; The bone density measurement model is iteratively trained based on the second sample image until the overall loss of the bone density measurement model converges.

3. The bone density measurement method according to claim 1, characterized in that: The edge operator is a Frei-Chen operator.

4. The bone density measurement method according to any one of claims 1 to 3, characterized in that: The lumbar vertebra image to be measured includes a first lumbar vertebra, a second lumbar vertebra, a third lumbar vertebra and a fourth lumbar vertebra.

5. A bone density measurement device based on deep learning, characterized in that: include: A lumbar vertebra recognition module, which is used to input the medical image to be measured into the spine recognition model to obtain the lumbar vertebra image to be measured; A bone density measurement module, which is used to input the lumbar vertebra image to be measured into the bone density measurement model to obtain the lumbar vertebra bone density; The spine recognition model is obtained by training based on a first sample image, and the first sample image is at least annotated with a recognition result of the lumbar vertebra; the bone density measurement model is obtained by training based on a second sample image, and the second sample image is at least annotated with a bone density of the lumbar vertebra; The bone density measurement model includes a plurality of convolutional layers, a fully connected layer and a SoftMax-like layer arranged in sequence; the lumbar vertebra image to be measured is input into the bone density measurement model to obtain the lumbar vertebra bone density, the lumbar vertebra image to be measured is sequentially convoluted through a plurality of convolutional layers, and the lumbar vertebra bone density value is output after being processed through the fully connected layer and the SoftMax-like layer; The activation function of the SoftMax-like layer is: Among them, SoftMax1(x) is the activation function, x i 、x j is the element in the input vector, i and j are the element numbers; The step of inputting the medical image to be measured into the spine recognition model to obtain the lumbar vertebra image to be measured comprises: Inputting the medical image to be measured into the downsampling structure of the spine recognition model to obtain a plurality of downsampling feature maps with gradually decreasing sizes; Input the downsampled feature map with the smallest size into the Transformer structure to obtain a global feature map; Performing edge operator processing on the multiple downsampled feature maps to obtain multiple edge operator feature maps; Inputting the plurality of edge operator feature maps and the global feature map into an upsampling structure to obtain a segmentation mask map, and determining the lumbar vertebra image to be measured based on the segmentation mask map; The step of inputting the plurality of edge operator feature maps and the global feature map into an upsampling structure to obtain a segmentation mask map comprises: Perform convolution processing and upsampling processing on the global feature map to obtain a third upsampled feature map; The edge operator feature map corresponding to the third down-sampled feature map is concatenated with the three up-sampled feature maps to obtain a third concatenated feature map; Performing convolution processing and upsampling processing on the third concatenated feature map to obtain a second upsampled feature map; Concatenate the edge operator feature map corresponding to the second down-sampled feature map with the second up-sampled feature map to obtain a second concatenated feature map; Performing convolution processing and upsampling processing on the second concatenated feature map to obtain a first upsampled feature map; Concatenate the edge operator feature map corresponding to the first down-sampled feature map with the first up-sampled feature map to obtain a first concatenated feature map; The first concatenated feature map is subjected to convolution processing and upsampling processing to obtain the segmentation mask map.

6. An electronic device, characterized in that: include: Memory and processor; The memory is used to store programs; The processor, coupled to the memory, is configured to execute the program to: Inputting the medical image to be measured into the spine recognition model to obtain the lumbar image to be measured; Inputting the lumbar vertebra image to be measured into the bone density measurement model to obtain the lumbar vertebra bone density; The spine recognition model is obtained by training based on a first sample image, and the first sample image is at least annotated with a recognition result of the lumbar vertebra; the bone density measurement model is obtained by training based on a second sample image, and the second sample image is at least annotated with a bone density of the lumbar vertebra; The bone density measurement model includes a plurality of convolutional layers, a fully connected layer and a SoftMax-like layer arranged in sequence; the lumbar vertebra image to be measured is input into the bone density measurement model to obtain the lumbar vertebra bone density, the lumbar vertebra image to be measured is sequentially convoluted through a plurality of convolutional layers, and the lumbar vertebra bone density value is output after being processed through the fully connected layer and the SoftMax-like layer; The activation function of the SoftMax-like layer is: Among them, SoftMax1(x) is the activation function, x i 、x j is the element in the input vector, i and j are the element numbers; The step of inputting the medical image to be measured into the spine recognition model to obtain the lumbar vertebra image to be measured comprises: Inputting the medical image to be measured into the downsampling structure of the spine recognition model to obtain a plurality of downsampling feature maps with gradually decreasing sizes; Input the downsampled feature map with the smallest size into the Transformer structure to obtain a global feature map; Performing edge operator processing on the multiple downsampled feature maps to obtain multiple edge operator feature maps; Inputting the plurality of edge operator feature maps and the global feature map into an upsampling structure to obtain a segmentation mask map, and determining the lumbar vertebra image to be measured based on the segmentation mask map; The step of inputting the plurality of edge operator feature maps and the global feature map into an upsampling structure to obtain a segmentation mask map comprises: Perform convolution processing and upsampling processing on the global feature map to obtain a third upsampled feature map; The edge operator feature map corresponding to the third down-sampled feature map is concatenated with the three up-sampled feature maps to obtain a third concatenated feature map; Performing convolution processing and upsampling processing on the third concatenated feature map to obtain a second upsampled feature map; Concatenate the edge operator feature map corresponding to the second down-sampled feature map with the second up-sampled feature map to obtain a second concatenated feature map; Performing convolution processing and upsampling processing on the second concatenated feature map to obtain a first upsampled feature map; Concatenate the edge operator feature map corresponding to the first down-sampled feature map with the first up-sampled feature map to obtain a first concatenated feature map; The first concatenated feature map is subjected to convolution processing and upsampling processing to obtain the segmentation mask map.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the bone density measurement method based on deep learning as described in any one of claims 1 to 4.

Images