Spine detection method, device, equipment and storage medium based on multimodal images

Abstract

The present application relates to a spine detection method, apparatus, equipment and storage medium based on multimodal images, which obtains visible light image features and X-ray image spine key point features through a multimodal model; calculates a similarity measurement matrix between the two to determine similar pixel points of each X-ray image spine key point on the visible light image, and outputs the predicted coordinates of all spine key points on the visible light image; determines a loss function based on the predicted coordinates of the spine key points and their corresponding true coordinates of the spine key points, and performs reverse iteration on the predicted coordinates of the spine key points to obtain an optimized multimodal model; inputs any labeled X-ray image and the visible light image to be predicted into the optimized multimodal model, outputs and calculates the Cobb angle based on the spine key points of the visible light image to be predicted; compared with the prior art, the present application determines spine key points based on visible light images, which can reduce the risk of radiation exposure and improve the efficiency of spine detection.

Description

Technical Field

[0001] The present application relates to the technical field of image processing, and in particular to a spine detection method, apparatus, device and storage medium based on multimodal images. Background Art

[0002] Idiopathic scoliosis is a complex three-dimensional spinal deformity that causes abnormal curvature of the spine, which may lead to cardiac and respiratory dysfunction, as well as psychosocial problems. Clinically, imaging evaluation is the key to diagnosing scoliosis, among which X-ray imaging technology is widely used because it can reveal the morphological parameters of the spine.

[0003] Although deep learning-based spinal cord keypoint detection technology has made significant progress in X-ray images, the technology still faces challenges in practical application. The main problem is that the vertebrae in X-ray images are often obscured by soft tissues and organs in the chest cavity. This occlusion reduces the accuracy of single-view automatic keypoint positioning. To overcome this limitation, patients are usually required to undergo multiple X-ray imaging to obtain more comprehensive spinal information from different angles, thereby improving detection accuracy.

[0004] However, this multi-view imaging approach brings additional costs and risks. First, patients are exposed to more X-rays, which increases radiation exposure. In particular, long-term or repeated X-ray examinations may have negative effects on health. Second, additional X-ray imaging requires more specialized equipment and technicians, which not only occupies the resources of medical institutions but may also cause delays in the diagnosis and treatment process. Summary of the Invention

[0005] The present application provides a spine detection method, apparatus, device and storage medium based on multimodal images, which determines the key points of the spine based only on visible light images, can reduce the risk of radiation exposure and improve the efficiency of spine detection.

[0006] In a first aspect, the present application provides a spine detection method based on multimodal images, comprising: obtaining visible light image features in a visible light image and X-ray image features of an X-ray image based on a pre-built multimodal model, performing inner product processing on the X-ray image features based on masked spine key point features of the X-ray image to obtain X-ray image spine key point features; calculating a similarity measurement matrix between the visible light image features and the X-ray image spine key point features, and determining similar pixels of each X-ray image spine key point in the X-ray image spine key point features on the visible light image based on the similarity measurement matrix. points, based on the similar pixel points, determine and output the predicted coordinates of all spine key points on the visible light image based on the multimodal model; based on the predicted coordinates of all spine key points and the real coordinates of the spine key points corresponding to the predicted coordinates of all spine key points, determine the loss function, and perform reverse iterative training on the multimodal model based on the loss function to obtain an optimized multimodal model; input any labeled X-ray image and the visible light image to be predicted into the optimized multimodal model, so that the optimized multimodal model outputs the spine key points of the visible light image to be predicted, and calculates the Cobb angle based on the spine key points.

[0007] In one possible implementation, inner product processing is performed on the X-ray image features based on the masked spine key point features of the X-ray image to obtain the spine key point features of the X-ray image, specifically including: obtaining the coordinates of the manually labeled spine key points on the X-ray image, masking the coordinates of the manually labeled spine key points to obtain masked spine key point features; performing dimensionality conversion processing on the masked spine key point features to obtain target dimension masked spine key point features; performing inner product processing on the target dimension masked spine key point features and the X-ray image features to obtain X-ray image spine key point features.

[0008] In one possible implementation, calculating the similarity measurement matrix of the visible light image features and the spine key point features of the X-ray image specifically includes: obtaining the conjugate transpose of the visible light image features, performing vector outer product calculation on the conjugate transpose of the visible light image features and the spine key point features of the X-ray image, and obtaining a similarity measurement matrix.

[0009] In one possible implementation, based on the similarity measurement matrix, similar pixel points of each X-ray image spine key point in the X-ray image spine key point feature on the visible light image are determined, specifically including: based on the similarity measurement matrix, respectively determining the similarity measurement vectors of each X-ray image spine key point in the X-ray image spine key point feature and all pixel points in the visible light image; obtaining the maximum value of the similarity measurement vector in the similarity measurement vector corresponding to each X-ray image spine key point, and taking the pixel points corresponding to the maximum value of the similarity measurement vector as the similar pixel points of each X-ray image spine key point on the visible light image.

[0010] In one possible implementation, a loss function is determined based on the predicted coordinates of all the spinal key points and the real coordinates of the spinal key points corresponding to all the predicted coordinates of the spinal key points, specifically including: calculating the L1 norm between each spinal key point predicted coordinate and the real coordinates of the spinal key points corresponding to each spinal key point predicted coordinate; integrating the L1 norms corresponding to all the predicted coordinates of the spinal key points to obtain a total L1 norm, and using the total L1 norm as the loss function.

[0011] In one possible implementation, the Cobb angle is calculated based on the spinal key points, specifically including: obtaining adjacent spinal key points to obtain multiple spinal key point pairs, and calculating the direction vector corresponding to each spinal key point pair; obtaining the radian value of the direction vector, and performing angle conversion processing on the radian value to obtain the angle value corresponding to each spinal key point pair, and determining the Cobb angle based on the angle value.

[0012] In one possible implementation, obtaining visible light image features from a visible light image and X-ray image features from an X-ray image based on a pre-built multimodal model specifically includes: constructing a multimodal model comprising a first backbone network and a second backbone network; inputting the visible light image into the first backbone network so that the first backbone network outputs visible light image features corresponding to the visible light image; and inputting the X-ray image into the second backbone network so that the second backbone network outputs X-ray image features corresponding to the X-ray image.

[0013] In the second aspect, the present application provides a spine detection device based on multimodal images, including: an image feature acquisition module, a spine key point coordinate prediction module, a loss function determination model and a spine key point detection module; wherein the image feature acquisition module is used to acquire visible light image features in visible light images and X-ray image features of X-ray images based on a pre-built multimodal model, and perform inner product processing on the X-ray image features based on the masked spine key point features of the X-ray image to obtain the X-ray image spine key point features; the spine key point coordinate prediction module is used to calculate the similarity measurement matrix of the visible light image features and the X-ray image spine key point features, and determine the similarity of each X-ray image spine key point feature in the X-ray image based on the similarity measurement matrix. The line image spine key points are similar pixel points on the visible light image, and based on the similar pixel points, the predicted coordinates of all spine key points on the visible light image are determined and output based on the multimodal model; the loss function determination model is used to determine the loss function based on the predicted coordinates of all spine key points and the real coordinates of the spine key points corresponding to the predicted coordinates of all spine key points, and reverse iterative training is performed on the multimodal model based on the loss function to obtain an optimized multimodal model; the spine key point detection module is used to input any labeled X-ray image and the visible light image to be predicted into the optimized multimodal model, so that the optimized multimodal model outputs the spine key points of the visible light image to be predicted, and calculates the Cobb angle based on the spine key points.

[0014] In a third aspect, an embodiment of the present application further provides a computer device, which includes a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the above method when executing the computer program.

[0015] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program can implement the above method when executed by a processor.

[0016] The present invention provides a method that has the following advantages over the prior art:

[0017] Through the pre-built multimodal model, the visible light image features in the visible light image and the X-ray image features of the X-ray image are automatically acquired, and then a mask is applied to the X-ray image features to obtain the spine key point features of the X-ray image; the similarity measurement matrix of the visible light image features and the X-ray image spine key point features is calculated, and based on the similarity measurement matrix, the similar pixel points of each X-ray image spine key point in the X-ray image spine key point features on the visible light image are determined; based on the similar pixel points, the predicted coordinates of all the spine key points on the visible light image are determined and output based on the multimodal model, so as to achieve modeling of the correspondence between the visible light image and the spine key points in the X-ray image during the model training stage; and subsequently based on the predicted coordinates of all the spine key points, and the predicted coordinates of all the spine key points are respectively The corresponding real coordinates of the spinal key points are obtained, the loss function is determined, and the multimodal model is optimized through reverse iterative training to further improve the performance of the model and the accuracy of prediction; any labeled X-ray image and the visible light image to be predicted are input into the optimized multimodal model, so that the optimized multimodal model outputs the spinal key points of the visible light image to be predicted, and based on the spinal key points, the Cobb angle is calculated, so that in the model reasoning stage, any labeled X-ray image with existing labeled spinal key points can be used to assist in locating the spinal key points on the visible light image to be predicted. In the actual prediction process, the spinal key points can be detected on the visible light image to be predicted without the patient taking an X-ray image, thereby reducing the harm of radiation to the patient and saving medical costs. In this process, the fully automatic prediction of the spinal canal key points can be achieved, thereby improving the efficiency of spinal detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0019] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0020] One or more embodiments are exemplarily illustrated by pictures in the corresponding drawings. These exemplifications do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings are represented as similar elements. Unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0021] Figure 1This is a flowchart of an embodiment of a spine detection method based on multimodal images provided by the present application;

[0022] Figure 2 This is a schematic structural diagram of an embodiment of a multimodal image-based spinal column detection device provided by the present application;

[0023] Figure 3 This is a schematic diagram of an embodiment of the present application providing an application of a backbone network to extract image features;

[0024] Figure 4 This is a schematic diagram of mask processing according to an embodiment of the present application;

[0025] Figure 5 This is a schematic diagram of a similarity measurement matrix calculation process according to an embodiment of the present application;

[0026] Figure 6 This is a schematic diagram of a process for locating similar pixels on a visible light image that are most similar to spine key points on an X-ray image according to an embodiment of the present application;

[0027] Figure 7 This is a structural diagram of an electronic device provided by this application. DETAILED DESCRIPTION

[0028] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0029] The disclosure below provides many different embodiments or examples for implementing different structures of the present application. In order to simplify the disclosure of the present application, the components and settings of specific examples are described below. Of course, these are merely examples and are not intended to limit the present application. In addition, the present application may repeat reference numbers and / or letters in different examples. Such repetition is for the purpose of simplicity and clarity and does not in itself indicate the relationship between the various embodiments and / or settings discussed.

[0030] It will be understood that when used in this specification and the appended claims, the terms “comprises” and “comprising” indicate the presence of described features, integers, steps, operations, elements and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0031] It should also be understood that the terms used in this specification are for the purpose of describing specific embodiments only and are not intended to limit the present application. As used in this specification and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0032] It should be further understood that the term "and / or" used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0033] As used in this specification and the appended claims, the term "if" can be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0034] Example 1, see Figure 1 , Figure 1 This is a flowchart of an embodiment of a spine detection method based on multimodal images provided by the present application. Figure 1 As shown, the method includes steps 101 to 104, which are specifically as follows:

[0035] Step 101: Based on a pre-built multimodal model, obtain visible light image features in the visible light image and X-ray image features of the X-ray image, perform inner product processing on the X-ray image features based on the masked spine key point features of the X-ray image, and obtain the X-ray image spine key point features.

[0036] In one embodiment, a multimodal model is constructed, wherein the multimodal model includes a first backbone network and a second backbone network.

[0037] Specifically, a pre-trained ResNet model is selected as the first backbone network and the second backbone network, wherein the first backbone network and the second backbone network have been pre-trained on a large-scale image dataset and have learned rich feature representations.

[0038] In one embodiment, a visible light image is input into a pre-trained first backbone network, so that the first backbone network outputs visible light image features corresponding to the visible light image.

[0039] Specifically, the visible light image I q Input to the pre-trained first backbone network f q(·) so that the first backbone network f q (·) Based on its multi-layer neural convolutional network structure, the input visible light image is extracted separately to obtain the features of different levels extracted by each layer of neural convolutional network. After multi-layer processing, the first backbone network will output a visible light image feature x q =f q (I q ), where I q ∈R h×w , h represents the height of the image, w represents the width of the image, x q ∈R n×d , n represents the number of pixels, and d represents the feature dimension.

[0040] In one embodiment, the X-ray image is input into a pre-trained second backbone network so that the second backbone network outputs X-ray image features corresponding to the X-ray image; Figure 3 As shown, Figure 3 This is a schematic diagram of an embodiment provided by the present application for extracting image features using a backbone network.

[0041] Specifically, the X-ray image I s Input to the pre-trained second backbone network f s (·) so that the second backbone network f s (·) Based on its multi-layer neural convolutional network structure, the input X-ray image is extracted separately to obtain the features of different levels extracted by each layer of neural convolutional network. After multi-layer processing, the second backbone network will output an X-ray image feature x s =f s (I s ), where I s ∈R h×w , h represents the height of the image, w represents the width of the image, I s ∈R n×d , n represents the number of pixels, and d represents the feature dimension.

[0042] In one embodiment, the visible light image may be an image of any user, and the X-ray image may also be an image of any user.

[0043] Preferably, the visible light image and the X-ray image do not need to be paired according to users, that is, the visible light image corresponds to the first user and the X-ray image corresponds to the second user, wherein the first user and the second user are not the same user.

[0044] Specifically, during the model training stage, the number of the X-ray images can be one or more; when the number of the X-ray images is one, the corresponding relationship between the key points of the spine is subsequently constructed based on one X-ray image and one visible light image; when the number of the X-ray images is multiple, the corresponding relationship between the key points is subsequently constructed based on multiple X-ray images and one visible light image.

[0045] In one embodiment, after the visible light image features and the X-ray image features are extracted, the visible light image features and the X-ray image features are fused based on the pre-built multimodal model for subsequent prediction of spinal key points.

[0046] like Figure 4 As shown, Figure 4 This is a mask processing diagram of an embodiment provided by this application.

[0047] In one embodiment, the coordinates of the manually annotated spinal key points on the X-ray image are obtained, and the manually annotated spinal key point coordinates are masked to obtain masked spinal key point features; the masked spinal key point features are dimensionally converted to obtain target dimension masked spinal key point features; the target dimension masked spinal key point features and the X-ray image features are inner-product processed to obtain X-ray image spinal key point features.

[0048] Specifically, the spinal key points in the X-ray image are manually annotated to obtain manually annotated spinal key point coordinates, wherein the manually annotated spinal key point coordinates can be determined by a medical expert or by using an image annotation tool.

[0049] Specifically, when masking the manually annotated spine key point coordinates, a mask image is first created based on the X-ray image, and based on the manually annotated spine key point coordinates, the pixel positions corresponding to the manually annotated spine key point coordinates in the mask image are set to 1, and the pixel positions not corresponding to the manually annotated spine key point coordinates in the mask image are set to 0, and the masked spine key point feature m is obtained. s ∈R h×w .

[0050] Specifically, the mask image has the same image height and image width as the X-ray image.

[0051] Specifically, the two-dimensional mask spine key point feature m s ∈R h×w Convert to one-dimensional masked spine keypoint features Where n is the number of pixels in the X-ray image.

[0052] Preferably, the target dimension masked spine key point feature is a one-dimensional masked spine key point feature.

[0053] Specifically, when performing inner product processing on the target dimension masked spine key point feature and the X-ray image feature, the target dimension masked spine key point feature and the X-ray image feature are input into the inner product calculation formula to obtain the X-ray image spine key point feature; wherein, the inner product calculation formula is as follows:

[0054]

[0055] Where ⊙ represents the vector inner product, x k ∈R k×d represents the key point features of the spine in the X-ray image, k represents the number of key points of the spine in the X-ray image, represents the target dimension mask spine keypoint feature, x s Represents X-ray image features.

[0056] Step 102: Calculate a similarity measurement matrix between the visible light image features and the X-ray image spine key point features, and based on the similarity measurement matrix, determine similar pixel points of each X-ray image spine key point in the X-ray image spine key point features on the visible light image; based on the similar pixel points, determine and output the predicted coordinates of all spine key points on the visible light image based on the multimodal model.

[0057] In one embodiment, the conjugate transpose of the visible light image feature is obtained, and the vector outer product calculation is performed on the conjugate transpose of the visible light image feature and the spine key point feature of the X-ray image to obtain a similarity measurement matrix, such as Figure 5 As shown, Figure 5 This is a schematic diagram of the process of similarity measurement matrix calculation in an embodiment provided by this application.

[0058] Specifically, the conjugate transpose of the visible light image feature and the spine key point feature of the X-ray image are input into a preset vector outer product formula to obtain a similarity measurement matrix, wherein the vector outer product formula is as follows:

[0059]

[0060] Where S∈R n×k is the similarity measurement matrix, represents the vector outer product, is the conjugate transpose of the visible light image feature, x k ∈R k×drepresents the key point features of the spine in the X-ray image, k is the number of key points of the spine in the X-ray image, and d is the dimension of each key point feature of the spine in the X-ray image.

[0061] In one embodiment, based on the similarity measurement matrix, similarity measurement vectors between each X-ray image spine key point in the X-ray image spine key point features and all pixel points in the visible light image are determined respectively.

[0062] Specifically, since the vector outer product can be used to measure the similarity between two vectors, each element S in the similarity measurement matrix S is ij It can be used to represent the similarity between the i-th pixel in the visible light image feature and the j-th X-ray image spine keypoint in the X-ray image keypoint feature.

[0063] Specifically, the similarity measurement vector s between each X-ray image spine key point in the X-ray image spine key point feature and all pixels in the visible light image is i ∈S, where s i ∈R n×1 , i∈k, i represents the i-th X-ray image spine key point in the X-ray image spine key point feature, and the dimension of this similarity measurement vector is n×1, where n is the number of pixels in the visible light image.

[0064] In one embodiment, the maximum value of the similarity measurement vector in the similarity measurement vector corresponding to each X-ray image spine key point is obtained, and the pixel point corresponding to the maximum value of the similarity measurement vector is used as the similar pixel point of each X-ray image spine key point on the visible light image. Figure 6 As shown, Figure 6 This is a schematic diagram of a process for locating similar pixel points on a visible light image that are most similar to spine key points in an X-ray image according to an embodiment provided by the present application.

[0065] Specifically, the similarity measurement vector s i ∈R n×1 Perform vector dimension conversion to obtain the similarity measurement vector s of the target dimension i ∈R h×w , where h and w are the image height and image width of the visible light image, respectively; based on the above vector dimension conversion, the one-dimensional similarity measure vector can be rearranged into a two-dimensional matrix that matches the size of the visible light image.

[0066] Specifically, the n×1 dimensional similarity measure vector s iAfter being converted into a two-dimensional matrix of h×w dimensions, the maximum value of the similarity measurement vector in the similarity measurement vector corresponding to each spine key point of each X-ray image is determined based on the maximum value in the two-dimensional matrix.

[0067] Specifically, since the two-dimensional matrix matches the size of the visible light image, after determining the maximum value of the similarity measurement vector, the pixel points corresponding to the maximum value of the similarity measurement vector correspond to the pixel points on the visible light image. Therefore, the pixel points corresponding to the maximum value of the similarity measurement vector can be directly used as similar pixel points of each X-ray image spine key point on the visible light image.

[0068] In one embodiment, after each spine key point in the X-ray image has similar pixels on the visible light image, the coordinates (x i ,y i ), where x i and y i They respectively represent the column index and row index of similar pixel points in the two-dimensional matrix. Based on the row index and the column index, the predicted coordinates of the spine key point corresponding to the i-th spine key point on the visible light image can be directly obtained.

[0069] Specifically, the similar pixel points corresponding to the spine key points of each X-ray image are obtained, and the similar pixel point coordinates of each similar pixel point are used as the predicted coordinates of the spine key points on the visible light image to obtain the predicted coordinates of all spine key points on the visible light image, and the predicted coordinates of all spine key points on the visible light image are used as the output of the multimodal model.

[0070] Step 103: Based on the predicted coordinates of all the spinal key points and the true coordinates of the spinal key points corresponding to the predicted coordinates of all the spinal key points, a loss function is determined, and the multimodal model is reversely iterated trained based on the loss function to obtain an optimized multimodal model.

[0071] In one embodiment, the L1 norm between the predicted coordinates of each spinal key point and the real coordinates of the spinal key point corresponding to each spinal key point predicted coordinate is calculated separately, the L1 norms corresponding to all the spinal key point predicted coordinates are integrated to obtain the total L1 norm, and the total L1 norm is used as the loss function.

[0072] Specifically, for the visible light image, the spinal key points in the visible light image are also manually labeled to obtain the true coordinates of the manually labeled spinal key points, wherein the true coordinates of the manually labeled spinal key points can be determined by medical experts or using image annotation tools.

[0073] Specifically, the predicted coordinates of each spine key point and the real coordinates of the spine key point corresponding to the predicted coordinates of each spine key point are input into a preset L1 norm calculation formula, the total L1 norm is calculated, and the total L1 norm is used as the loss function, wherein the L1 norm calculation formula is as follows:

[0074]

[0075] Where L is the loss function, i is the i-th spine keypoint, k is the number of spine keypoints, (x i ,y i ) are the predicted coordinates of the spine key points, The predicted coordinates of each spine key point correspond to the true coordinates of the spine key point.

[0076] In one embodiment, when the multimodal model is reversely iteratively trained based on the loss function to obtain an optimized multimodal model, the back propagation algorithm can be used to calculate the gradient of the loss function with respect to the model parameters of the multimodal model, and the weights and biases of the model are updated according to the calculated gradient and the preset learning rate through the gradient descent method, and the above steps 101-103 are repeated until the loss function meets the preset threshold or reaches a predetermined number of iterations. That is, this process involves calculating the loss function in each iteration, back propagating the gradient, and updating the model parameters according to the gradient.

[0077] In one embodiment, after sufficient model training iterations, the multimodal model can learn how to predict the spinal key points on the visible light image based on the spinal key points in the X-ray image, that is, model the correspondence between the spinal key points in the X-ray image and the spinal key points in the visible light image, thereby obtaining an optimized multimodal model.

[0078] Step 104: Input any labeled X-ray image and the visible light image to be predicted into the optimized multimodal model, so that the optimized multimodal model outputs the spine key points of the visible light image to be predicted, and calculates the Cobb angle based on the spine key points.

[0079] In one embodiment, the arbitrary labeled X-ray image may be one or more X-ray images with manually labeled spine key points; the arbitrary labeled X-ray image and the visible light image to be predicted also do not need to be images of the same user.

[0080] In one embodiment, an arbitrarily labeled X-ray image with manually labeled spinal key points and a visible light image to be predicted without labeled spinal key points are input into the trained optimized multimodal model. Since the optimized multimodal model has modeled the similarity between the spinal key points in the X-ray image and the spinal key points in the visible light image, the optimized multimodal model can directly use the arbitrarily labeled X-ray image with manually labeled spinal key points to predict the spinal key points on the visible light image.

[0081] In one embodiment, after obtaining the spine key points of the visible light image to be predicted, adjacent spine key points are obtained to obtain multiple spine key point pairs, and the direction vector corresponding to each spine key point pair is calculated.

[0082] Specifically, all spine key points of the visible light image to be predicted are traversed, and based on the position distance between each spine key point, adjacent spine key points are determined to construct multiple spine key point pairs.

[0083] Preferably, adjacent spinal key points may be selected based on the coordinate positions corresponding to the spinal key points in order from top to bottom or from bottom to top to obtain a plurality of spinal key point pairs.

[0084] Specifically, when calculating the direction vector corresponding to each pair of spine key points, the coordinates of the first spine key point and the second spine key point in each pair of spine key points are obtained, and the first spine key point coordinates and the second spine key point coordinates are substituted into a preset direction vector calculation formula to obtain the direction vector corresponding to each pair of spine key points; wherein the preset direction vector calculation formula is as follows:

[0085] (Δ x ,Δ y )=(x j -x i ,y j -y i );

[0086] Where, (Δ x ,Δ y ) is the direction vector corresponding to each spine key point pair, (x i ,y i ) is the coordinate of the first spine key point, (x j ,y j ) is the coordinate of the second spine key point.

[0087] In one embodiment, the radian value of the direction vector is obtained, and the radian value is converted into an angle to obtain the angle value corresponding to each pair of spine key points, and the Cobb angle is determined based on the angle value.

[0088] Specifically, when obtaining the radian value of the direction vector, the atan2 function is used to calculate the direction vector (Δ x ,Δ y ) in radians a=atan2(Δ y ,Δ x ) Among them, the atan2 function has a range of The inverse tangent function is defined as follows:

[0089]

[0090] Specifically, the radian value is input into a preset angle conversion formula to obtain the angle value θ corresponding to each pair of spine key points, wherein the angle conversion formula is as follows:

[0091] θ=a*180 / π。

[0092] In one embodiment, the angle values ​​corresponding to each pair of spine key points are traversed to determine a monotonically increasing region and a monotonically decreasing region; wherein the angle values ​​in the monotonically increasing region are in an increasing state, and the angle values ​​in the monotonically decreasing region are in a decreasing state.

[0093] In one embodiment, in the monotonically increasing region, a first top vertebra and a first bottom vertebra are determined, and the maximum angle difference between the first top vertebra and the first bottom vertebra is used as the first Cobb angle.

[0094] In one embodiment, in the monotonically decreasing region, a second top vertebra and a second bottom vertebra are determined, and the maximum angle difference between the second top vertebra and the second bottom vertebra is used as the second Cobb angle.

[0095] Specifically, in the monotonically decreasing region or the monotonically increasing region, the bottom vertebra refers to the vertebra with the largest inclination toward the concave side of scoliosis, and the top vertebra is the vertebra that deviates the farthest from the midline and is in a horizontal position in scoliosis.

[0096] In summary, the present application provides a multimodal image-based spinal detection method, which can model the correspondence between the visible light image of any patient and the spinal key points in the X-ray image of any patient during the model training stage, wherein the visible light image and the X-ray image do not need to be paired according to the user, which can improve the rapid acquisition of model training data, and in the model reasoning stage, any labeled X-ray image with existing labeled spinal key points can be used to assist in locating the spinal key points on the visible light image to be predicted. In the actual prediction process, the spinal key points can be detected on the visible light image to be predicted without the patient taking an X-ray image, thereby reducing the harm of radiation to the patient and saving medical costs. In this process, fully automatic prediction of the spinal canal key points can be achieved, thereby improving the efficiency of spinal detection.

[0097] Example 2, see Figure 2 , Figure 2 It is a structural diagram of an embodiment of a multimodal image-based spine detection device provided by the present application. Corresponding to the above-mentioned multimodal image-based spine detection method, the present application also provides a multimodal image-based spine detection device. The multimodal image-based spine detection device includes a unit for executing the above-mentioned multimodal image-based spine detection method, and the multimodal image-based spine detection device can be configured in a desktop computer, tablet computer, laptop computer, and other terminals. Specifically, the multimodal image-based spine detection device includes an image feature acquisition module 201, a spine key point coordinate prediction module 202, a loss function determination model 203, and a spine key point detection module 204.

[0098] The image feature acquisition module 201 is used to obtain visible light image features in the visible light image and X-ray image features of the X-ray image based on a pre-built multimodal model, and perform inner product processing on the X-ray image features based on the masked spine key point features of the X-ray image to obtain the X-ray image spine key point features.

[0099] The spine key point coordinate prediction module 202 is used to calculate the similarity measurement matrix of the visible light image features and the X-ray image spine key point features, and based on the similarity measurement matrix, determine the similar pixel points of each X-ray image spine key point in the X-ray image spine key point features on the visible light image, and based on the similar pixel points, determine and output the predicted coordinates of all spine key points on the visible light image based on the multimodal model.

[0100] The loss function determination model 203 is used to determine the loss function based on the predicted coordinates of all the spinal key points and the real coordinates of the spinal key points corresponding to the predicted coordinates of all the spinal key points, and to perform reverse iterative training on the multimodal model based on the loss function to obtain an optimized multimodal model.

[0101] The spinal key point detection module 204 is used to input any labeled X-ray image and the visible light image to be predicted into the optimized multimodal model, so that the optimized multimodal model outputs the spinal key points of the visible light image to be predicted, and calculates the Cobb angle based on the spinal key points.

[0102] In one embodiment, the image feature acquisition module 201 is used to perform inner product processing on the X-ray image features based on the masked spine key point features of the X-ray image to obtain the spine key point features of the X-ray image, specifically including: obtaining the coordinates of the manually labeled spine key points on the X-ray image, performing mask processing on the manually labeled spine key point coordinates to obtain the masked spine key point features; performing dimensionality conversion processing on the masked spine key point features to obtain the target dimension masked spine key point features; performing inner product processing on the target dimension masked spine key point features and the X-ray image features to obtain the X-ray image spine key point features.

[0103] In one embodiment, the spine key point coordinate prediction module 202 is used to calculate the similarity measurement matrix of the visible light image features and the X-ray image spine key point features, specifically including: obtaining the conjugate transpose of the visible light image features, performing vector outer product calculation on the conjugate transpose of the visible light image features and the X-ray image spine key point features, and obtaining a similarity measurement matrix.

[0104] In one embodiment, the spine key point coordinate prediction module 202 is used to determine the similar pixel points of each X-ray image spine key point in the X-ray image spine key point feature on the visible light image based on the similarity measurement matrix, specifically including: based on the similarity measurement matrix, respectively determining the similarity measurement vectors of each X-ray image spine key point in the X-ray image spine key point feature and all the pixel points in the visible light image; obtaining the maximum value of the similarity measurement vector in the similarity measurement vector corresponding to each X-ray image spine key point, and taking the pixel points corresponding to the maximum value of the similarity measurement vector as the similar pixel points of each X-ray image spine key point on the visible light image.

[0105] In one embodiment, the loss function determination model 203 is used to determine the loss function based on the predicted coordinates of all the spinal key points and the real coordinates of the spinal key points corresponding to all the predicted coordinates of the spinal key points, specifically including: calculating the L1 norm between each spinal key point predicted coordinate and the real coordinates of the spinal key points corresponding to each spinal key point predicted coordinate; integrating the L1 norms corresponding to the predicted coordinates of all the spinal key points to obtain the total L1 norm, and using the total L1 norm as the loss function.

[0106] In one embodiment, the spinal key point detection module 204 is used to calculate the Cobb angle based on the spinal key points, specifically including: obtaining adjacent spinal key points to obtain multiple spinal key point pairs, and calculating the direction vector corresponding to each spinal key point pair; obtaining the radian value of the direction vector, and performing angle conversion processing on the radian value to obtain the angle value corresponding to each spinal key point pair, and determining the Cobb angle based on the angle value.

[0107] In one embodiment, the image feature acquisition module 201 is used to acquire visible light image features in a visible light image and X-ray image features of an X-ray image based on a pre-built multimodal model, specifically including: constructing a multimodal model, wherein the multimodal model includes a first backbone network and a second backbone network; inputting a visible light image into the first backbone network so that the first backbone network outputs visible light image features corresponding to the visible light image; and inputting an X-ray image into the second backbone network so that the second backbone network outputs X-ray image features corresponding to the X-ray image.

[0108] The multimodal image-based spine detection device can implement the multimodal image-based spine detection method of the method embodiment. The options in the method embodiment are also applicable to this embodiment and will not be described in detail here.

[0109] like Figure 7 As shown, Figure 7 This is a structural diagram of an electronic device provided by the present application; it includes a processor 111, a communication interface 112, a memory 113 and a communication bus 114, wherein the processor 111, the communication interface 112, and the memory 113 communicate with each other through the communication bus 114, and the memory 113 is used to store computer programs.

[0110] In one embodiment of the present application, the processor 111 is configured to implement the multimodal image-based spine detection method provided by any one of the aforementioned method embodiments when executing the program stored in the memory 113 .

[0111] Those skilled in the art will appreciate that all or part of the steps in the method of the above-described embodiment can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. The computer program is executed by at least one processor in the computer system to implement the steps in the method of the above-described embodiment.

[0112] Therefore, an embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the multimodal image-based spine detection method provided in any of the aforementioned method embodiments.

[0113] The storage medium is a physical, non-transient storage medium, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a magnetic disk, or an optical disk, etc. Any physical storage medium capable of storing program code can be non-volatile or volatile.

[0114] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0115] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of each unit is merely a logical functional division, and other division methods may be used in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not implemented.

[0116] The steps in the method of the embodiment of the present application can be adjusted in order, combined, and deleted according to actual needs. The units in the device of the embodiment of the present application can be combined, divided, and deleted according to actual needs. In addition, the functional units in the various embodiments of the present application can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into a single unit.

[0117] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the existing technology, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, terminal, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application.

[0118] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0119] Obviously, those skilled in the art may make various modifications and variations to this application without departing from the spirit and scope of this application. Thus, as long as these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.

[0120] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present application, and such modifications or substitutions should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A spine detection method based on multimodal images, characterized in that: include: Based on a pre-built multimodal model, obtain visible light image features in the visible light image and X-ray image features of the X-ray image, and perform inner product processing on the X-ray image features based on masked spine key point features of the X-ray image to obtain X-ray image spine key point features; Calculating a similarity measurement matrix between the visible light image features and the X-ray image spine key point features, and determining similar pixels on the visible light image for each X-ray image spine key point in the X-ray image spine key point features based on the similarity measurement matrix; and determining and outputting predicted coordinates of all spine key points on the visible light image based on the similar pixels and based on the multimodal model; Determining a loss function based on the predicted coordinates of all the spine key points and the true coordinates of the spine key points corresponding to the predicted coordinates of all the spine key points, and performing reverse iterative training on the multimodal model based on the loss function to obtain an optimized multimodal model; Any labeled X-ray image and the visible light image to be predicted are input into the optimized multimodal model so that the optimized multimodal model outputs the spine key points of the visible light image to be predicted, and the Cobb angle is calculated based on the spine key points.

2. The method according to claim 1, wherein: Performing inner product processing on the X-ray image features based on the masked spine key point features of the X-ray image to obtain the X-ray image spine key point features specifically includes: Obtaining manually annotated spine key point coordinates on the X-ray image, and performing masking processing on the manually annotated spine key point coordinates to obtain masked spine key point features; Performing dimension conversion processing on the masked spine key point features to obtain target dimension masked spine key point features; An inner product process is performed on the target dimension mask spine key point feature and the X-ray image feature to obtain the X-ray image spine key point feature.

3. The method according to claim 1, wherein Calculating a similarity measurement matrix between the visible light image features and the spine key point features of the X-ray image specifically includes: The conjugate transpose of the visible light image feature is obtained, and a vector outer product calculation is performed on the conjugate transpose of the visible light image feature and the spine key point feature of the X-ray image to obtain a similarity measurement matrix.

4. The method according to claim 1, wherein Determining similar pixel points on the visible light image for each X-ray image spine key point in the X-ray image spine key point features based on the similarity measurement matrix specifically includes: Based on the similarity measurement matrix, respectively determining a similarity measurement vector between each X-ray image spine key point in the X-ray image spine key point feature and all pixel points in the visible light image; The maximum value of the similarity measurement vector in the similarity measurement vector corresponding to each X-ray image spine key point is obtained, and the pixel points corresponding to the maximum value of the similarity measurement vector are respectively used as similar pixel points of each X-ray image spine key point on the visible light image.

5. The method according to claim 1, wherein: Based on the predicted coordinates of all the spine key points and the true coordinates of the spine key points corresponding to the predicted coordinates of all the spine key points, a loss function is determined, specifically including: Calculate the L1 norm between the predicted coordinates of each spine key point and the true coordinates of the spine key point corresponding to each predicted coordinate of the spine key point; The L1 norms corresponding to the predicted coordinates of all the spine key points are integrated to obtain a total L1 norm, and the total L1 norm is used as the loss function.

6. The method according to claim 1, wherein: Based on the spine key points, the Cobb angle is calculated, specifically including: Obtain adjacent spine key points to obtain multiple spine key point pairs, and calculate the direction vector corresponding to each spine key point pair; The radian value of the direction vector is obtained, and the radian value is converted into an angle to obtain the angle value corresponding to each pair of spine key points, and the Cobb angle is determined based on the angle value.

7. The method according to claim 1, wherein: Based on the pre-built multimodal model, visible light image features in visible light images and X-ray image features in X-ray images are obtained, including: Constructing a multimodal model, wherein the multimodal model includes a first backbone network and a second backbone network; Inputting a visible light image into the first backbone network so that the first backbone network outputs visible light image features corresponding to the visible light image; The X-ray image is input into the second backbone network so that the second backbone network outputs X-ray image features corresponding to the X-ray image.

8. A spinal column detection device based on multimodal images, characterized in that: include: Image feature acquisition module, spine key point coordinate prediction module, loss function determination model and spine key point detection module; The image feature acquisition module is configured to acquire visible light image features in the visible light image and X-ray image features of the X-ray image based on a pre-built multimodal model, and perform inner product processing on the X-ray image features based on the masked spine key point features of the X-ray image to obtain the spine key point features of the X-ray image; The spine key point coordinate prediction module is used to calculate a similarity measurement matrix between the visible light image features and the X-ray image spine key point features, and based on the similarity measurement matrix, determine similar pixels on the visible light image for each X-ray image spine key point in the X-ray image spine key point features, and based on the similar pixels, determine and output the predicted coordinates of all spine key points on the visible light image based on the multimodal model; The loss function determination model is used to determine the loss function based on the predicted coordinates of all the spine key points and the true coordinates of the spine key points corresponding to each of the predicted coordinates of all the spine key points, and perform reverse iterative training on the multimodal model based on the loss function to obtain an optimized multimodal model; The spine key point detection module is used to input any labeled X-ray image and the visible light image to be predicted into the optimized multimodal model, so that the optimized multimodal model outputs the spine key points of the visible light image to be predicted, and calculates the Cobb angle based on the spine key points.

9. A computer device, characterized in that: The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that The storage medium stores a computer program, and when the computer program is executed by a processor, the computer program can implement the method according to any one of claims 1 to 7.

Images