Spine detection method, device and equipment based on multi-modal image and storage medium

Abstract

The invention relates to a spine detection method and device based on a multi-modal image, equipment and a storage medium. Visible light image features and X-ray image spine key point features are obtained through a multi-modal model; calculating a similarity measurement matrix of the two to determine similar pixel points of each X-ray image spine key point on the visible light image, outputting and determining all spine key point prediction coordinates on the visible light image, determining a loss function based on the spine key point prediction coordinates and the corresponding spine key point real coordinates, and determining the loss function according to the loss function. The reverse iteration module is used for carrying out reverse iteration on the spine key point prediction coordinates to obtain an optimized multi-modal model; inputting any marked X-ray image and a to-be-predicted visible light image into the optimized multi-modal model, outputting and calculating a Cobb angle based on the spine key point of the to-be-predicted visible light image; compared with the prior art, the spine key point is determined based on the visible light image, the risk of radiation exposure can be reduced, and the spine detection efficiency is improved.

Description

Technical Field

[0001] The present application relates to the technical field of image processing, and in particular to a spine detection method, device, equipment and storage medium based on multimodal images. Background Art

[0002] Idiopathic scoliosis is a complex three-dimensional spinal deformity that causes abnormal curvature of the spine, which may lead to cardiac and respiratory dysfunction, as well as psychosocial problems. Clinically, imaging evaluation is the key to diagnosing scoliosis, among which X-ray imaging technology is widely used because it can reveal the morphological parameters of the spine.

[0003] Although deep learning-based spinal cord key point detection technology has made significant progress in X-ray images, the technology still faces challenges in practical applications. The main problem is that the vertebrae in X-ray images are often obscured by soft tissues and organs in the chest cavity. This occlusion reduces the accuracy of single-view automatic key point positioning. To overcome this limitation, patients are usually required to undergo multiple X-ray imaging to obtain more comprehensive spinal information from different angles, thereby improving detection accuracy.

[0004] However, this multi-view imaging approach brings additional costs and risks. First, patients are subjected to more X-ray exposures, which increases radiation exposure. In particular, long-term or repeated X-ray examinations may have negative effects on health. Second, additional X-ray imaging requires more professional equipment and technicians, which not only takes up the resources of medical institutions, but may also cause delays in the diagnosis and treatment process. Summary of the invention

[0005] The present application provides a spine detection method, apparatus, device and storage medium based on multimodal images, which can determine the key points of the spine based only on visible light images, reduce the risk of radiation exposure, and improve the efficiency of spine detection.

[0006] In a first aspect, the present application provides a spine detection method based on multimodal images, comprising: based on a pre-built multimodal model, obtaining visible light image features in a visible light image and X-ray image features of an X-ray image, performing inner product processing on the X-ray image features based on masked spine key point features of the X-ray image to obtain X-ray image spine key point features; calculating a similarity measurement matrix between the visible light image features and the X-ray image spine key point features, and based on the similarity measurement matrix, determining similar pixels of each X-ray image spine key point in the X-ray image spine key point features on the visible light image Points, based on the similar pixel points, determine and output the predicted coordinates of all spine key points on the visible light image based on the multimodal model; based on the predicted coordinates of all spine key points and the real coordinates of the spine key points corresponding to the predicted coordinates of all spine key points, determine the loss function, and perform reverse iterative training on the multimodal model based on the loss function to obtain an optimized multimodal model; input any labeled X-ray image and the visible light image to be predicted into the optimized multimodal model, so that the optimized multimodal model outputs the spine key points of the visible light image to be predicted, and calculates the Cobb angle based on the spine key points.

[0007] In a possible implementation, inner product processing is performed on the X-ray image features based on the masked spine key point features of the X-ray image to obtain the X-ray image spine key point features, specifically including: obtaining the coordinates of the manually annotated spine key points on the X-ray image, and performing mask processing on the manually annotated spine key point coordinates to obtain masked spine key point features; performing dimensionality conversion processing on the masked spine key point features to obtain target dimensional masked spine key point features; performing inner product processing on the target dimensional masked spine key point features and the X-ray image features to obtain X-ray image spine key point features.

[0008] In a possible implementation, calculating the similarity measurement matrix of the visible light image features and the X-ray image spine key point features specifically includes: obtaining the conjugate transpose of the visible light image features, performing vector outer product calculation on the conjugate transpose of the visible light image features and the X-ray image spine key point features, and obtaining a similarity measurement matrix.

[0009] In a possible implementation, based on the similarity measurement matrix, similar pixel points of each X-ray image spine key point in the X-ray image spine key point feature on the visible light image are determined, specifically including: based on the similarity measurement matrix, respectively determining the similarity measurement vectors of each X-ray image spine key point in the X-ray image spine key point feature and all pixel points in the visible light image; obtaining the maximum value of the similarity measurement vectors in the similarity measurement vectors corresponding to each X-ray image spine key point, and taking the pixel points corresponding to the maximum value of the similarity measurement vector as the similar pixel points of each X-ray image spine key point on the visible light image.

[0010] In one possible implementation, a loss function is determined based on the predicted coordinates of all the spinal key points and the real coordinates of the spinal key points corresponding to each of the predicted coordinates of all the spinal key points, specifically including: calculating the L1 norm between each spinal key point predicted coordinate and the real coordinates of the spinal key points corresponding to each spinal key point predicted coordinate; integrating the L1 norms corresponding to the predicted coordinates of all the spinal key points to obtain a total L1 norm, and using the total L1 norm as the loss function.

[0011] In a possible implementation, the Cobb angle is calculated based on the spinal key points, specifically including: obtaining adjacent spinal key points to obtain multiple spinal key point pairs, and calculating the direction vector corresponding to each spinal key point pair; obtaining the radian value of the direction vector, and performing angle conversion processing on the radian value to obtain the angle value corresponding to each spinal key point pair, and determining the Cobb angle based on the angle value.

[0012] In a possible implementation, based on a pre-built multimodal model, obtaining visible light image features in a visible light image and X-ray image features of an X-ray image specifically includes: building a multimodal model, wherein the multimodal model includes a first backbone network and a second backbone network; inputting a visible light image into the first backbone network so that the first backbone network outputs visible light image features corresponding to the visible light image; and inputting an X-ray image into the second backbone network so that the second backbone network outputs X-ray image features corresponding to the X-ray image. .

[0013] In a second aspect, the present application provides a spine detection device based on multimodal images, comprising: an image feature acquisition module, a spine key point coordinate prediction module, a loss function determination model and a spine key point detection module; wherein the image feature acquisition module is used to acquire visible light image features in visible light images and X-ray image features of X-ray images based on a pre-built multimodal model, and to perform inner product processing on the X-ray image features based on masked spine key point features of the X-ray image to obtain X-ray image spine key point features; the spine key point coordinate prediction module is used to calculate a similarity measurement matrix between the visible light image features and the X-ray image spine key point features, and to determine each X-ray image spine key point feature in the X-ray image based on the similarity measurement matrix. The method comprises the following steps: determining the predicted coordinates of all the spinal key points on the visible light image based on the similar pixel points of the line image spinal key points on the visible light image, and outputting the predicted coordinates of all the spinal key points on the visible light image based on the similar pixel points; determining the loss function based on the predicted coordinates of all the spinal key points and the real coordinates of the spinal key points corresponding to the predicted coordinates of all the spinal key points, and performing reverse iterative training on the multimodal model based on the loss function to obtain an optimized multimodal model; and the spinal key point detection module is used to input any labeled X-ray image and the visible light image to be predicted into the optimized multimodal model, so that the optimized multimodal model outputs the spinal key points of the visible light image to be predicted, and calculates the Cobb angle based on the spinal key points.

[0014] In a third aspect, an embodiment of the present application further provides a computer device, which includes a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the above method when executing the computer program.

[0015] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program can implement the above method when executed by a processor.

[0016] The present invention provides a method which has the following advantages compared with the prior art:

[0017] Through the pre-constructed multimodal model, the visible light image features in the visible light image and the X-ray image features of the X-ray image are automatically acquired, and then a mask is applied to the X-ray image features to obtain the spine key point features of the X-ray image; a similarity measurement matrix of the visible light image features and the X-ray image spine key point features is calculated, and based on the similarity measurement matrix, similar pixels of each X-ray image spine key point in the X-ray image spine key point features on the visible light image are determined, and based on the similar pixels, the predicted coordinates of all the spine key points on the visible light image are determined and output based on the multimodal model, so as to achieve modeling of the correspondence between the visible light image and the spine key points in the X-ray image during the model training stage; and then based on the predicted coordinates of all the spine key points, and the predicted coordinates of all the spine key points are respectively The corresponding real coordinates of the spinal key points are obtained, the loss function is determined, and the multimodal model is optimized through reverse iterative training to further improve the performance of the model and the accuracy of prediction; any labeled X-ray image and the visible light image to be predicted are input into the optimized multimodal model, so that the optimized multimodal model outputs the spinal key points of the visible light image to be predicted, and based on the spinal key points, the Cobb angle is calculated, so that in the model reasoning stage, any labeled X-ray image with existing labeled spinal key points can be used to assist in locating the spinal key points on the visible light image to be predicted. In the actual prediction process, the spinal key points can be detected on the visible light image to be predicted without the patient taking an X-ray image, thereby reducing the harm of radiation to the patient and saving medical costs. In this process, the fully automatic prediction of the spinal canal key points can be achieved, thereby improving the efficiency of spinal detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0019] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0020] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0021] Figure 1It is a flowchart of an embodiment of a spine detection method based on multimodal images provided by the present application;

[0022] Figure 2 It is a structural schematic diagram of an embodiment of a spine detection device based on multimodal images provided by the present application;

[0023] Figure 3 It is a schematic diagram of applying a backbone network to extract image features according to an embodiment provided by the present application;

[0024] Figure 4 is a mask processing schematic diagram of an embodiment provided by the present application;

[0025] Figure 5 It is a schematic diagram of a process of similarity measurement matrix calculation in an embodiment provided by the present application;

[0026] Figure 6 This is a schematic diagram of a process of locating similar pixel points on a visible light image that are most similar to spine key points on an X-ray image according to an embodiment of the present application;

[0027] Figure 7 It is a structural schematic diagram of an electronic device provided by this application. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0029] The disclosure below provides many different embodiments or examples to realize the different structures of the present application. In order to simplify the disclosure of the present application, the parts and settings of specific examples are described below. Of course, they are only examples, and the purpose is not to limit the present application. In addition, the present application can repeat reference numbers and / or letters in different examples. This repetition is for the purpose of simplification and clarity, and does not itself indicate the relationship between the various embodiments and / or settings discussed.

[0030] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0031] It should also be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0032] It should be further understood that the term “and / or” used in the specification and appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0033] As used in this specification and the appended claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0034] Example 1, see Figure 1 , Figure 1 is a flowchart of an embodiment of a spine detection method based on multimodal images provided by the present application, such as Figure 1 As shown, the method includes steps 101 to 104, which are specifically as follows:

[0035] Step 101: Based on a pre-built multimodal model, obtain visible light image features in the visible light image and X-ray image features of the X-ray image, perform inner product processing on the X-ray image features based on the masked spine key point features of the X-ray image, and obtain the X-ray image spine key point features.

[0036] In one embodiment, a multimodal model is constructed, wherein the multimodal model includes a first backbone network and a second backbone network.

[0037] Specifically, a pre-trained ResNet model is selected as the first backbone network and the second backbone network, wherein the first backbone network and the second backbone network have been pre-trained on a large-scale image dataset and have learned rich feature representations.

[0038] In one embodiment, a visible light image is input into a pre-trained first backbone network, so that the first backbone network outputs visible light image features corresponding to the visible light image.

[0039] Specifically, the visible light image I q Input to the pre-trained first backbone network f q(·) so that the first backbone network f q (·) Based on its multi-layer neural convolutional network structure, the input visible light image is extracted separately to obtain different levels of features extracted by each layer of neural convolutional network. After multi-layer processing, the first backbone network will output a visible light image feature x q =f q (I q ), where I q ∈R h×w , h represents the height of the image, w represents the width of the image, x q ∈R n×d , n represents the number of pixels, and d represents the feature dimension.

[0040] In one embodiment, the X-ray image is input into a pre-trained second backbone network so that the second backbone network outputs X-ray image features corresponding to the X-ray image; Figure 3 As shown, Figure 3 It is a schematic diagram of applying a backbone network to extract image features according to an embodiment provided by the present application.

[0041] Specifically, the X-ray image I s Input to the pre-trained second backbone network f s (·) so that the second backbone network f s (·) Based on its multi-layer neural convolutional network structure, the input X-ray image is extracted separately to obtain different levels of features extracted by each layer of neural convolutional network. After multi-layer processing, the second backbone network will output an X-ray image feature x s =f s (I s ), where I s ∈R h×w , h represents the height of the image, w represents the width of the image, I s ∈R n×d , n represents the number of pixels, and d represents the feature dimension.

[0042] In one embodiment, the visible light image may be an image of any user, and the X-ray image may also be an image of any user.

[0043] Preferably, the visible light image and the X-ray image do not need to be paired according to users, that is, the visible light image corresponds to the first user, and the X-ray image corresponds to the second user, wherein the first user and the second user are not the same user.

[0044] Specifically, in the model training stage, the number of the X-ray images can be one or more; when the number of the X-ray images is one, a correspondence relationship between spinal key points is subsequently constructed based on one X-ray image and one visible light image; when the number of the X-ray images is multiple, a correspondence relationship between key points is subsequently constructed based on multiple X-ray images and one visible light image.

[0045] In one embodiment, after the visible light image features and the X-ray image features are extracted, the visible light image features and the X-ray image features are fused based on the pre-built multimodal model for subsequent prediction of spinal key points.

[0046] like Figure 4 As shown, Figure 4 It is a mask processing schematic diagram of an embodiment provided by the present application.

[0047] In one embodiment, the coordinates of the manually annotated spinal key points on the X-ray image are obtained, and the manually annotated spinal key point coordinates are masked to obtain masked spinal key point features; the masked spinal key point features are dimensionally converted to obtain target dimension masked spinal key point features; the target dimension masked spinal key point features and the X-ray image features are inner-product processed to obtain X-ray image spinal key point features.

[0048] Specifically, the key points of the spine in the X-ray image are manually annotated to obtain the coordinates of the manually annotated key points of the spine, wherein the coordinates of the manually annotated key points of the spine can be determined by a medical expert or by using an image annotation tool.

[0049] Specifically, when masking the manually annotated spine key point coordinates, a mask image is first created based on the X-ray image, and based on the manually annotated spine key point coordinates, the pixel positions corresponding to the manually annotated spine key point coordinates in the mask image are set to 1, and the pixel positions not corresponding to the manually annotated spine key point coordinates in the mask image are set to 0, so as to obtain the mask spine key point feature m s ∈R h×w .

[0050] Specifically, the mask image has the same image height and image width as the X-ray image.

[0051] Specifically, the two-dimensional mask spine key point feature m s ∈R h×w Convert to 1D masked spine keypoint features Where n is the number of pixels in the X-ray image.

[0052] Preferably, the target dimension masked spine key point feature is a one-dimensional masked spine key point feature.

[0053] Specifically, when the inner product processing is performed on the target dimension mask spine key point feature and the X-ray image feature, the target dimension mask spine key point feature and the X-ray image feature are input into the inner product calculation formula to obtain the X-ray image spine key point feature; wherein the inner product calculation formula is as follows:

[0054]

[0055] In the formula, ⊙ represents the vector inner product, x k ∈R k×d represents the key point features of the spine in the X-ray image, k represents the number of key points of the spine in the X-ray image, represents the target dimension mask spine keypoint feature, x s Represents X-ray image features.

[0056] Step 102: Calculate a similarity measurement matrix between the visible light image features and the X-ray image spine key point features, and based on the similarity measurement matrix, determine similar pixel points on the visible light image for each X-ray image spine key point in the X-ray image spine key point features, and based on the similar pixel points, determine and output the predicted coordinates of all spine key points on the visible light image based on the multimodal model.

[0057] In one embodiment, the conjugate transpose of the visible light image feature is obtained, and the conjugate transpose of the visible light image feature and the spine key point feature of the X-ray image are vector-derived to obtain a similarity measurement matrix, such as: Figure 5 As shown, Figure 5 It is a schematic diagram of the process of similarity measurement matrix calculation in an embodiment provided by the present application.

[0058] Specifically, the conjugate transpose of the visible light image feature and the spine key point feature of the X-ray image are input into a preset vector outer product formula to obtain a similarity measurement matrix, wherein the vector outer product formula is as follows:

[0059]

[0060] Where S∈R n×k is the similarity measurement matrix, represents the vector outer product, is the conjugate transpose of the visible light image feature, x k ∈R k×drepresents the key point features of the spine of the X-ray image, k is the number of the spine key points of the X-ray image on the X-ray image, and d is the dimension of each key point feature of the spine key points of the X-ray image.

[0061] In one embodiment, based on the similarity measurement matrix, similarity measurement vectors between each X-ray image spine key point in the X-ray image spine key point features and all pixel points in the visible light image are determined respectively.

[0062] Specifically, since the vector outer product can be used to measure the similarity between two vectors, each element S in the similarity measurement matrix S is ij It can be used to represent the similarity between the i-th pixel in the visible light image feature and the j-th X-ray image spine key point in the X-ray image key point feature.

[0063] Specifically, the similarity measurement vector s between each X-ray image spine key point in the X-ray image spine key point feature and all pixel points in the visible light image is i ∈S, where s i ∈R n×1 , i∈k, i represents the i-th X-ray image spine key point in the X-ray image spine key point feature, and the dimension of this similarity measurement vector is n×1, where n is the number of pixels in the visible light image.

[0064] In one embodiment, the maximum value of the similarity measurement vector in the similarity measurement vector corresponding to each X-ray image spine key point is obtained, and the pixel point corresponding to the maximum value of the similarity measurement vector is used as the similar pixel point of each X-ray image spine key point on the visible light image. Figure 6 As shown, Figure 6 It is a schematic diagram of a process of locating similar pixel points on a visible light image that are most similar to spine key points in an X-ray image according to an embodiment provided by the present application.

[0065] Specifically, for the similarity measurement vector s i ∈R n×1 Perform vector dimension conversion to obtain the similarity measurement vector s of the target dimension i ∈R h×w , where h and w are the image height and image width of the visible light image, respectively; based on the above vector dimension conversion, the one-dimensional similarity measurement vector can be rearranged into a two-dimensional matrix that matches the size of the visible light image.

[0066] Specifically, the n×1-dimensional similarity metric vector s iAfter being converted into a two-dimensional matrix of h×w dimensions, the maximum value of the similarity measurement vector in the similarity measurement vector corresponding to each spine key point of each X-ray image is determined based on the maximum value in the two-dimensional matrix.

[0067] Specifically, since the two-dimensional matrix matches the size of the visible light image, after determining the maximum value of the similarity measurement vector, the pixel points corresponding to the maximum value of the similarity measurement vector correspond to the pixel points on the visible light image. Therefore, the pixel points corresponding to the maximum value of the similarity measurement vector can be directly used as similar pixel points of each X-ray image spine key point on the visible light image.

[0068] In one embodiment, after each spine key point of the X-ray image has a similar pixel point on the visible light image, the coordinates (x i ,y i ), where x i and i They respectively represent the column index and row index of similar pixel points in the two-dimensional matrix. Based on the row index and the column index, the predicted coordinates of the spine key point corresponding to the i-th spine key point on the visible light image can be directly obtained.

[0069] Specifically, the similar pixel points corresponding to the spinal key points of each X-ray image are obtained, and the similar pixel point coordinates of each of the similar pixel points are used as the predicted coordinates of the spinal key points on the visible light image to obtain the predicted coordinates of all the spinal key points on the visible light image, and the predicted coordinates of all the spinal key points on the visible light image are used as the output of the multimodal model.

[0070] Step 103: Based on the predicted coordinates of all the spinal key points and the real coordinates of the spinal key points corresponding to the predicted coordinates of all the spinal key points, a loss function is determined, and the multimodal model is reversely iterated trained based on the loss function to obtain an optimized multimodal model.

[0071] In one embodiment, the L1 norm between the predicted coordinates of each spinal key point and the real coordinates of the spinal key point corresponding to each spinal key point predicted coordinate is calculated respectively, the L1 norms corresponding to all the spinal key point predicted coordinates are integrated to obtain the total L1 norm, and the total L1 norm is used as the loss function.

[0072] Specifically, for the visible light image, the spinal key points in the visible light image are also manually labeled to obtain the real coordinates of the manually labeled spinal key points, wherein the real coordinates of the manually labeled spinal key points can be determined by medical experts or using image annotation tools.

[0073] Specifically, the predicted coordinates of each spine key point and the real coordinates of the spine key point corresponding to the predicted coordinates of each spine key point are input into a preset L1 norm calculation formula, the total L1 norm is calculated, and the total L1 norm is used as the loss function, wherein the L1 norm calculation formula is as follows:

[0074]

[0075] Where L is the loss function, i is the i-th spine keypoint, k is the number of spine keypoints, (x i ,y i ) are the predicted coordinates of the spine key points, The true coordinates of the spine key points corresponding to the predicted coordinates of each spine key point.

[0076] In one embodiment, when the multimodal model is reversely iterated trained based on the loss function to obtain an optimized multimodal model, the back propagation algorithm can be used to calculate the gradient of the loss function with respect to the model parameters of the multimodal model, and the weights and biases of the model are updated according to the calculated gradient and the preset learning rate through the gradient descent method, and the above steps 101-103 are repeated until the loss function meets the preset threshold or reaches a predetermined number of iterations, that is, this process involves calculating the loss function in each iteration, back propagating the gradient, and updating the model parameters according to the gradient.

[0077] In one embodiment, after sufficient model training iterations, the multimodal model can learn how to predict the spinal key points on the visible light image based on the spinal key points in the X-ray image, that is, to model the correspondence between the spinal key points in the X-ray image and the spinal key points in the visible light image, thereby obtaining an optimized multimodal model.

[0078] Step 104: Input any labeled X-ray image and the visible light image to be predicted into the optimized multimodal model, so that the optimized multimodal model outputs the spine key points of the visible light image to be predicted, and calculates the Cobb angle based on the spine key points.

[0079] In one embodiment, the arbitrary labeled X-ray image may be one or more X-ray images with manually labeled spine key points; the arbitrary labeled X-ray image and the visible light image to be predicted also do not need to be images of the same user.

[0080] In one embodiment, an arbitrarily labeled X-ray image with manually labeled spinal key points and an arbitrarily labeled visible light image to be predicted without labeled spinal key points are input into a trained optimized multimodal model. Since the optimized multimodal model has modeled the similarity between the spinal key points in the X-ray image and the spinal key points in the visible light image, the optimized multimodal model can directly use the arbitrarily labeled X-ray image with manually labeled spinal key points to predict the spinal key points on the visible light image.

[0081] In one embodiment, after obtaining the spine key points of the visible light image to be predicted, adjacent spine key points are obtained to obtain multiple spine key point pairs, and the direction vector corresponding to each spine key point pair is calculated.

[0082] Specifically, the spine key points of all visible light images to be predicted are traversed, and based on the position distance between each spine key point, adjacent spine key points are determined to construct multiple spine key point pairs.

[0083] Preferably, adjacent spine key points may be selected in a top-to-bottom or bottom-to-top order based on the coordinate positions corresponding to the spine key points to obtain a plurality of spine key point pairs.

[0084] Specifically, when calculating the direction vector corresponding to each pair of spine key points, the coordinates of the first spine key point and the second spine key point in each pair of spine key points are obtained, and the first spine key point coordinates and the second spine key point coordinates are substituted into a preset direction vector calculation formula to obtain the direction vector corresponding to each pair of spine key points; wherein the preset direction vector calculation formula is as follows:

[0085] (Δ x ,Δ y )=(x j -x i ,y j -y i );

[0086] In the formula, (Δ x ,Δ y ) is the direction vector corresponding to each spine key point pair, (x i ,y i ) is the coordinate of the first spine key point, (x j ,y j ) are the coordinates of the second spine key point.

[0087] In one embodiment, the radian value of the direction vector is obtained, and the radian value is converted into an angle to obtain the angle value corresponding to each pair of spine key points, and the Cobb angle is determined based on the angle value.

[0088] Specifically, when obtaining the radian value of the direction vector, the atan2 function is used to calculate the direction vector (Δ x ,Δ y ) in radians a=atan2(Δ y ,Δ x ) Among them, the atan2 function has a range of The inverse tangent function is defined as follows:

[0089]

[0090] Specifically, the radian value is input into a preset angle conversion formula to obtain the angle value θ corresponding to each pair of spine key points, wherein the angle conversion formula is as follows:

[0091] θ=a*180 / π.

[0092] In one embodiment, the angle values ​​corresponding to each pair of spine key points are traversed to determine a monotonically increasing region and a monotonically decreasing region; wherein the angle values ​​in the monotonically increasing region are increasing, and the angle values ​​in the monotonically decreasing region are decreasing.

[0093] In one embodiment, in the monotonically increasing region, a first top vertebra and a first bottom vertebra are determined, and the maximum angle difference between the first top vertebra and the first bottom vertebra is taken as a first Cobb angle.

[0094] In one embodiment, in the monotonically decreasing region, a second top vertebra and a second bottom vertebra are determined, and the maximum angle difference between the second top vertebra and the second bottom vertebra is taken as the second Cobb angle.

[0095] Specifically, in the monotonically decreasing region or the monotonically increasing region, the bottom vertebra refers to the vertebra with the largest inclination toward the concave side of the scoliosis, and the top vertebra is the vertebra that deviates the farthest from the midline and is in a horizontal position in the scoliosis.

[0096] In summary, the present application provides a spinal detection method based on multimodal images, which can model the correspondence between the visible light image of any patient and the spinal key points in the X-ray image of any patient in the model training stage, wherein the visible light image and the X-ray image do not need to be paired according to the user, which can improve the rapid acquisition of model training data, and in the model reasoning stage, any labeled X-ray image with existing labeled spinal key points can be used to assist in locating the spinal key points on the visible light image to be predicted. In the actual prediction process, the spinal key points can be detected on the visible light image to be predicted without the patient taking X-ray images, thereby reducing the harm of radiation to the patient and saving medical costs. In this process, the fully automatic prediction of the spinal canal key points can be achieved, thereby improving the efficiency of spinal detection.

[0097] Example 2, see Figure 2 , Figure 2 It is a structural schematic diagram of an embodiment of a spine detection device based on multimodal images provided by the present application. Corresponding to the above-mentioned spine detection method based on multimodal images, the present application also provides a spine detection device based on multimodal images. The spine detection device based on multimodal images includes a unit for executing the above-mentioned spine detection method based on multimodal images, and the spine detection device based on multimodal images can be configured in desktop computers, tablet computers, laptops, and other terminals. Specifically, the spine detection device based on multimodal images includes an image feature acquisition module 201, a spine key point coordinate prediction module 202, a loss function determination model 203 and a spine key point detection module 204.

[0098] The image feature acquisition module 201 is used to acquire visible light image features in the visible light image and X-ray image features of the X-ray image based on a pre-built multimodal model, and perform inner product processing on the X-ray image features based on the masked spine key point features of the X-ray image to obtain the X-ray image spine key point features.

[0099] The spine key point coordinate prediction module 202 is used to calculate the similarity measurement matrix of the visible light image features and the X-ray image spine key point features, and based on the similarity measurement matrix, determine the similar pixel points of each X-ray image spine key point in the X-ray image spine key point features on the visible light image, and based on the similar pixel points, determine and output the predicted coordinates of all spine key points on the visible light image based on the multimodal model.

[0100] The loss function determination model 203 is used to determine the loss function based on the predicted coordinates of all the spinal key points and the real coordinates of the spinal key points corresponding to each of the predicted coordinates of all the spinal key points, and to perform reverse iterative training on the multimodal model based on the loss function to obtain an optimized multimodal model.

[0101] The spine key point detection module 204 is used to input any labeled X-ray image and the visible light image to be predicted into the optimized multimodal model, so that the optimized multimodal model outputs the spine key points of the visible light image to be predicted, and calculates the Cobb angle based on the spine key points.

[0102] In one embodiment, the image feature acquisition module 201 is used to perform inner product processing on the X-ray image features based on the masked spine key point features of the X-ray image to obtain the spine key point features of the X-ray image, specifically including: obtaining the coordinates of the manually annotated spine key points on the X-ray image, and performing mask processing on the manually annotated spine key point coordinates to obtain the masked spine key point features; performing dimensionality conversion processing on the masked spine key point features to obtain the masked spine key point features of the target dimension; performing inner product processing on the masked spine key point features of the target dimension and the X-ray image features to obtain the spine key point features of the X-ray image.

[0103] In one embodiment, the spine key point coordinate prediction module 202 is used to calculate the similarity measurement matrix of the visible light image features and the X-ray image spine key point features, specifically including: obtaining the conjugate transpose of the visible light image features, performing vector outer product calculation on the conjugate transpose of the visible light image features and the X-ray image spine key point features, and obtaining a similarity measurement matrix.

[0104] In one embodiment, the spine key point coordinate prediction module 202 is used to determine the similar pixel points of each X-ray image spine key point in the X-ray image spine key point feature on the visible light image based on the similarity measurement matrix, specifically including: based on the similarity measurement matrix, respectively determining the similarity measurement vectors of each X-ray image spine key point in the X-ray image spine key point feature and all the pixel points in the visible light image; obtaining the maximum value of the similarity measurement vector in the similarity measurement vector corresponding to each X-ray image spine key point, and using the pixel points corresponding to the maximum value of the similarity measurement vector as the similar pixel points of each X-ray image spine key point on the visible light image.

[0105] In one embodiment, the loss function determination model 203 is used to determine the loss function based on the predicted coordinates of all the spinal key points and the real coordinates of the spinal key points corresponding to each of the predicted coordinates of all the spinal key points, specifically including: calculating the L1 norm between each spinal key point predicted coordinate and the real coordinates of the spinal key points corresponding to each spinal key point predicted coordinate; integrating the L1 norms corresponding to the predicted coordinates of all the spinal key points to obtain the total L1 norm, and using the total L1 norm as the loss function.

[0106] In one embodiment, the spinal key point detection module 204 is used to calculate the Cobb angle based on the spinal key points, specifically including: obtaining adjacent spinal key points to obtain multiple spinal key point pairs, and calculating the direction vector corresponding to each spinal key point pair; obtaining the radian value of the direction vector, and performing angle conversion processing on the radian value to obtain the angle value corresponding to each spinal key point pair, and determining the Cobb angle based on the angle value.

[0107] In one embodiment, the image feature acquisition module 201 is used to acquire visible light image features in a visible light image and X-ray image features of an X-ray image based on a pre-built multimodal model, specifically including: constructing a multimodal model, wherein the multimodal model includes a first backbone network and a second backbone network; inputting a visible light image into the first backbone network so that the first backbone network outputs visible light image features corresponding to the visible light image; and inputting an X-ray image into the second backbone network so that the second backbone network outputs X-ray image features corresponding to the X-ray image.

[0108] The above-mentioned multimodal image-based spine detection device can implement the multimodal image-based spine detection method of the above-mentioned method embodiment. The options in the above-mentioned method embodiment are also applicable to this embodiment and will not be described in detail here.

[0109] like Figure 7 As shown, Figure 7 It is a structural diagram of an electronic device provided by the present application; it includes a processor 111, a communication interface 112, a memory 113 and a communication bus 114, wherein the processor 111, the communication interface 112, and the memory 113 communicate with each other through the communication bus 114, and the memory 113 is used to store computer programs.

[0110] In one embodiment of the present application, the processor 111 is used to implement the multimodal image-based spine detection method provided by any one of the aforementioned method embodiments when executing the program stored in the memory 113 .

[0111] It is understood by those skilled in the art that all or part of the processes in the method for implementing the above embodiment can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a storage medium, which is a computer-readable storage medium. The computer program is executed by at least one processor in the computer system to implement the process steps of the embodiment of the above method.

[0112] Therefore, an embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the multimodal image-based spine detection method provided in any of the aforementioned method embodiments are implemented.

[0113] The storage medium is a physical, non-transient storage medium, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a magnetic disk, or an optical disk, etc., which can store program codes. The computer-readable storage medium can be non-volatile or volatile.

[0114] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0115] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of each unit is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0116] The steps in the method of the embodiment of the present application can be adjusted in order, combined and deleted according to actual needs. The units in the device of the embodiment of the present application can be combined, divided and deleted according to actual needs. In addition, the functional units in the various embodiments of the present application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0117] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, terminal, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application.

[0118] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0119] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

[0120] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A spine detection method based on multimodal images, characterized in that: include: Based on the pre-built multimodal model, a visible light image feature in the visible light image and an X-ray image feature of the X-ray image are obtained, and an inner product process is performed on the X-ray image feature based on the masked spine key point feature of the X-ray image to obtain the spine key point feature of the X-ray image; Calculating a similarity measurement matrix between the visible light image features and the X-ray image spine key point features, and based on the similarity measurement matrix, determining similar pixel points of each X-ray image spine key point in the X-ray image spine key point features on the visible light image, and based on the similar pixel points, determining and outputting predicted coordinates of all spine key points on the visible light image based on the multimodal model; Based on the predicted coordinates of all the spine key points and the real coordinates of the spine key points corresponding to the predicted coordinates of all the spine key points, a loss function is determined, and based on the loss function, reverse iterative training is performed on the multimodal model to obtain an optimized multimodal model; Any labeled X-ray image and the visible light image to be predicted are input into the optimized multimodal model, so that the optimized multimodal model outputs the spine key points of the visible light image to be predicted, and the Cobb angle is calculated based on the spine key points.

2. The method according to claim 1, characterized in that Performing inner product processing on the X-ray image features based on the masked spine key point features of the X-ray image to obtain the X-ray image spine key point features specifically includes: Acquire the coordinates of the manually annotated spine key points on the X-ray image, and perform mask processing on the manually annotated spine key point coordinates to obtain masked spine key point features; Performing dimension conversion processing on the masked spine key point features to obtain target dimension masked spine key point features; The target dimension mask spine key point feature and the X-ray image feature are subjected to inner product processing to obtain the X-ray image spine key point feature.

3. The method according to claim 1, characterized in that: Calculating the similarity measurement matrix of the visible light image features and the spine key point features of the X-ray image specifically includes: The conjugate transpose of the visible light image feature is obtained, and a vector outer product calculation is performed on the conjugate transpose of the visible light image feature and the spine key point feature of the X-ray image to obtain a similarity measurement matrix.

4. The method according to claim 1, characterized in that: Determining similar pixel points of each X-ray image spine key point in the X-ray image spine key point feature on the visible light image based on the similarity measurement matrix specifically includes: Based on the similarity measurement matrix, respectively determine the similarity measurement vectors between each X-ray image spine key point in the X-ray image spine key point features and all pixel points in the visible light image; The maximum value of the similarity measurement vector in the similarity measurement vector corresponding to each X-ray image spine key point is obtained, and the pixel points corresponding to the maximum value of the similarity measurement vector are respectively used as similar pixel points of each X-ray image spine key point on the visible light image.

5. The method according to claim 1, characterized in that: Based on the predicted coordinates of all the spine key points and the real coordinates of the spine key points corresponding to the predicted coordinates of all the spine key points, a loss function is determined, which specifically includes: Calculate the L1 norm between the predicted coordinates of each spine key point and the real coordinates of the spine key point corresponding to each spine key point predicted coordinate; The L1 norms corresponding to the predicted coordinates of all the spine key points are integrated to obtain a total L1 norm, and the total L1 norm is used as the loss function.

6. The method according to claim 1, characterized in that: Based on the spine key points, the Cobb angle is calculated, specifically including: Acquire adjacent spine key points to obtain multiple spine key point pairs, and calculate the direction vector corresponding to each spine key point pair; The radian value of the direction vector is obtained, and the radian value is converted into an angle to obtain the angle value corresponding to each pair of spine key points, and the Cobb angle is determined based on the angle value.

7. The method according to claim 1, characterized in that: Based on the pre-built multimodal model, the visible light image features in the visible light image and the X-ray image features in the X-ray image are obtained, including: Constructing a multimodal model, wherein the multimodal model includes a first backbone network and a second backbone network; Inputting a visible light image into the first backbone network so that the first backbone network outputs visible light image features corresponding to the visible light image; The X-ray image is input into the second backbone network so that the second backbone network outputs X-ray image features corresponding to the X-ray image.

8. A spine detection device based on multimodal images, characterized in that: include: Image feature acquisition module, spine key point coordinate prediction module, loss function determination model and spine key point detection module; The image feature acquisition module is used to acquire visible light image features in the visible light image and X-ray image features of the X-ray image based on the pre-built multimodal model, and perform inner product processing on the X-ray image features based on the masked spine key point features of the X-ray image to obtain the spine key point features of the X-ray image; The spine key point coordinate prediction module is used to calculate the similarity measurement matrix of the visible light image features and the X-ray image spine key point features, and determine similar pixel points of each X-ray image spine key point in the X-ray image spine key point features on the visible light image based on the similarity measurement matrix, and determine and output the predicted coordinates of all spine key points on the visible light image based on the similar pixel points and based on the multimodal model; The loss function determination model is used to determine the loss function based on the predicted coordinates of all the spine key points and the real coordinates of the spine key points corresponding to the predicted coordinates of all the spine key points, and perform reverse iterative training on the multimodal model based on the loss function to obtain an optimized multimodal model; The spine key point detection module is used to input any labeled X-ray image and the visible light image to be predicted into the optimized multimodal model, so that the optimized multimodal model outputs the spine key points of the visible light image to be predicted, and calculates the Cobb angle based on the spine key points.

9. A computer device, characterized in that: The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 can be implemented.

Images