Deep learning-based spinal osteoporotic fracture assessment method and system

Abstract

The invention provides a spinal osteoporotic fracture assessment method and system based on deep learning. The comprehensiveness and interpretability of spinal brittle fracture risk assessment can be improved. According to the invention, based on the hierarchical feature extraction module, a spine space geometric structure, iconography texture features and clinical data are deeply fused, and the limitation of traditional single-mode analysis is broken through; a dual-mode graph neural network structure combining a rigid edge and a flexible edge is innovatively adopted for establishment, the rigid edge forcibly keeps anatomical connection of adjacent vertebral bodies, and the flexible edge quantifies the fracture risk cascade effect between cross-segment vertebral bodies; a multi-head image attention network is further combined to adaptively capture a long-distance global interaction relationship between centrums, so that risk signals of high-risk centrums are diffused to an associated region along a high-weight edge; and the key risk vertebral body and the conduction path thereof are visually revealed through visual mapping (for example, node colors represent risk levels, and edge connection strength reflects mutual influence coefficients among vertebral bodies).

Description

Technical Field

[0001] The present invention relates to the field of medical image analysis and artificial intelligence technology, and in particular to a spinal osteoporotic fracture assessment method and system based on deep learning. Background Art

[0002] The existing clinical evaluation system has significant technical bottlenecks: the traditional diagnostic method mainly relies on dual energy X-ray Absorptiometry (DXA) to assess bone mineral density (BMD), and combines risk assessment tools such as FRAX to predict the risk of spinal fractures. However, it can only assess the overall risk of the spine and cannot locate high-risk vertebrae, resulting in a high rate of missed diagnosis.

[0003] In recent years, artificial intelligence technology has attempted to break through the above limitations, but existing solutions still have the following technical defects: 1) Ignoring the fracture cascade effect: Traditional machine learning methods extract features such as single vertebral texture through radiomics, but do not consider the "fracture cascade effect" between vertebrae, that is, when a vertebra is abnormal, it will increase the fracture risk of the remaining vertebrae, with the greatest impact on the adjacent vertebrae. Therefore, ignoring the correlation between the risk dependence of adjacent vertebral fractures can lead to a decrease in prediction accuracy; 2) Insufficient global perception: Deep learning methods require stacking multiple layers of convolution to achieve cross-vertebral interaction, which leads to a surge in model parameters and reduced computational efficiency. In addition, the cubic receptive field of the standard three-dimensional convolution kernel (such as 3×3×3) is difficult to cover the narrow and long anatomical structure of the spine (which is significantly extended in the length direction), resulting in the failure of modeling long-distance dependencies across vertebrae. 3) The multimodal fusion mechanism is crude: Although existing methods integrate imaging and clinical data, they use a simple splicing strategy and ignore the differences in weights of different modal features among vertebral segments. At the same time, the black-box feature extraction process leads to insufficient decision interpretability, making it difficult for doctors to trace the basis for their decisions.

[0004] Therefore, it is necessary to provide a new method to solve the above technical problems. Summary of the invention

[0005] In order to achieve the above-mentioned purpose and other advantages of the present invention, the first purpose of the present invention is to provide a method for evaluating spinal osteoporotic fractures based on deep learning, comprising the following steps: Obtain patient spinal images and clinical data, and extract multimodal features; The multimodal features are spliced ​​and fused to construct a spinal graph structure based on anatomical constraints according to the anatomical characteristics of the spine; A multi-head graph attention neural network is used for global interaction of multimodal features and spinal fracture risk assessment.

[0006] Furthermore, the multimodal features include geometric features, imaging features, and clinical features.

[0007] Furthermore, the step of extracting multimodal features includes extracting geometric features: The center point of the minimum circumscribed rectangle of each vertebra is taken as the spatial position of the vertebra ; The geometrical characteristics are calculated using the Cobb measurement method.

[0008] Furthermore, the step of calculating the geometric features using the Cobb measurement method includes: The intersection angle of the normal lines of the extended lines of the adjacent vertebral outer edges As the intervertebral angle, the distance between the end points of the inner edge tangent as intervertebral space; The characteristic value of each vertebra is the average value of the calculation results of two adjacent vertebrae; The spatial coordinates, intervertebral angles and intervertebral spaces are integrated into the geometric features of each vertebra to achieve quantitative characterization of the spatial morphology of the spinal vertebrae.

[0009] Furthermore, the step of extracting multimodal features includes extracting image features: The smallest circumscribed cuboid of each vertebra is used as the input size of the 3D convolutional neural network, and the size of the remaining vertebrae is uniformly filled by zero padding. The image features of each vertebra are extracted in turn through a 3D convolutional neural network.

[0010] Furthermore, the 3D convolutional neural network includes three 3D convolutional layers and three global pooling layers, each convolutional layer has batch normalization and activation function, the convolutional layer is used to perform convolution operation on the input image to extract its image features, and the global pooling layer is used to reduce the dimension of the feature map obtained by convolution, and finally output the image features of each vertebra.

[0011] Furthermore, the step of extracting multimodal features includes extracting clinical features: For the patients’ clinical data, data preprocessing is used to achieve heterogeneous feature fusion; The preprocessed data were concatenated into multidimensional vectors, which were used as the clinical characteristics of each vertebra.

[0012] Furthermore, the step of implementing heterogeneous feature fusion by data preprocessing for the patient's clinical data includes: Continuous variables were standardized to eliminate dimensional differences; Use one-hot encoding or label encoding for categorical variables to avoid the model misjudging the order of categories; The step of splicing the preprocessed data into a multidimensional vector comprises: The standardized continuous variables and the encoded categorical variables are concatenated into a multidimensional vector.

[0013] Furthermore, the step of splicing and fusing the multimodal features to construct a spinal graph structure based on anatomical constraints according to the anatomical characteristics of the spine includes: Each vertebra of the spine is defined as a graph node, and the feature vector of each node is spliced ​​and fused by three types of modality data: imaging, geometry, and clinical. Two types of edge connections are constructed based on the anatomical characteristics of the spine.

[0014] Furthermore, the step of constructing two types of edge connections based on the anatomical characteristics of the spine includes: Adjacent vertebrae are connected by rigid edges and weights are fixed to maintain the anatomical continuity of the spinal column sequence. For inter-segment vertebrae, flexible edge connections are used based on the spatial Euclidean distance. Bone density gradient Calculate the connection weights through the Gaussian kernel function Quantify the strength of mechanical connection between vertebrae. >threshold, connections are established to screen out vertebral pairs with significant biomechanical conduction effects, thereby achieving cross-scale structural establishment from a rigid anatomical framework to a flexible connection network.

[0015] Furthermore, the step of using a multi-head graph attention neural network to perform multimodal feature global interaction and spinal fracture risk assessment includes: Through the learnable parameter matrix Map the node features to the latent space, for each vertebral node and its neighbors , calculate the normalized attention weight: ; in, is the attention vector; according to Iteratively update node features so that the multimodal features of high-risk vertebrae diffuse to associated vertebrae through high-weight edges, where the high-weight edges are configured as Preset value, is anatomical connection strength × dynamic attention weight; The probability of each vertebral fracture is output through the Sigmoid function.

[0016] Furthermore, the method further comprises the steps of: The risk heat map was generated by bicubic interpolation and superimposed on the sagittal reconstructed view of the original image; The node color indicates the risk level, and the edge transparency and proportional to the vertebral body, and shows the connection between any vertebral body node and its associated vertebral body.

[0017] The second object of the present invention is to provide a spinal osteoporotic fracture assessment system based on deep learning, which applies the above method and includes a multimodal feature extraction module, a spinal graph structure construction module, and a spinal fracture risk assessment module; wherein, The multimodal feature extraction module is used to obtain the patient's spinal images and clinical data and extract multimodal features; The spinal graph structure construction module is used to splice and fuse the multimodal features and construct a spinal graph structure based on anatomical constraints according to the anatomical characteristics of the spine; The spinal fracture risk assessment module is used to use a multi-head graph attention neural network to perform global interaction of multimodal features and spinal fracture risk assessment.

[0018] Compared with the prior art, the present invention has the following beneficial effects: The present invention can improve the comprehensiveness and interpretability of spinal fragility fracture risk assessment by constructing a spinal osteoporotic fracture assessment method and system based on anatomical constraints and graph neural networks. Compared with the existing technology, its core advantages are: based on the hierarchical feature extraction module, the spatial geometric structure of the spine, the imaging texture features and clinical data are deeply integrated, breaking through the limitations of traditional single-modal analysis; innovatively adopting a dual-mode graph neural network structure combining rigid edges and flexible edges, in which the rigid edges force the anatomical connection of adjacent vertebrae to be maintained, and the flexible edges quantify the fracture risk cascade effect between intersegmental vertebrae; further combined with a multi-head graph attention network, the long-distance global interaction relationship between vertebrae is adaptively captured, so that the risk signal of high-risk vertebrae is diffused to the associated area along the high-weight edge, and the key risk vertebrae and their conduction paths are intuitively revealed through visual mapping (such as node color represents the risk level, and edge connection strength reflects the mutual influence coefficient between vertebrae). While reducing the computational complexity of traditional finite element analysis, this method avoids the interpretability defects of the black box model, and provides an intelligent solution for spinal fragility fracture assessment that takes into account anatomical constraints and clinical operability.

[0019] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention and implement it according to the contents of the specification, the following is a detailed description of the preferred embodiments of the present invention in conjunction with the accompanying drawings. The specific implementation of the present invention is given in detail by the following embodiments and their accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings: Figure 1 Process of evaluating spinal osteoporotic fractures based on deep learning Figure 1 ; Figure 2 Process of evaluating spinal osteoporotic fractures based on deep learning Figure 2 ; Figure 3 Schematic diagram of the spinal structure for anatomical constraints; Figure 4 Schematic diagram of the first layer of the multi-head graph attention network; Figure 5 Extract flow chart for geometric features; Figure 6 Flowchart for calculating geometric features using the Cobb measurement method; Figure 7 It is the flow chart of image feature extraction; Figure 8 Flowcharts were extracted for clinical features; Fig. 9 Flowchart for clinical data preprocessing; Fig.10 Construct a flow chart for the structure of the spine graph based on anatomical constraints; Fig.11 To construct two types of edge connection flow graphs based on the anatomical characteristics of the spine; Fig.12 This is a flowchart of the global interaction of multi-modal features of multi-head graph attention neural network and spinal fracture risk assessment; Fig.13 Visualize a flow chart for the model; Fig.14 Schematic diagram of the spinal osteoporotic fracture assessment system based on deep learning; Fig.15 It is a schematic diagram of computer equipment; Fig.16 A schematic diagram of a computer-readable storage medium. DETAILED DESCRIPTION

[0021] The present invention is further described below in conjunction with the accompanying drawings and specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. It should be noted that, under the premise of no conflict, the embodiments or technical features described below can be arbitrarily combined to form a new embodiment.

[0022] Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in the field without making any creative work shall fall within the scope of protection of the present invention.

[0023] The figure numbers in this application are only used to distinguish the various steps in the scheme, and are not used to limit the execution order of the various steps. The specific execution order is subject to the description in the specification.

[0024] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention. Example 1

[0025] A deep learning-based method for evaluating spinal osteoporotic fractures. Figure 1-Figure 2 As shown, the following steps are included: S100, obtaining patient spinal images and clinical data, and extracting multimodal features; In this embodiment, the patient's spine CT images and clinical data (such as age, bone density value, fracture history, etc.) are obtained, and multimodal features are extracted through the following parallel processing flow.

[0026] Furthermore, the multimodal features include geometric features, imaging features, and clinical features.

[0027] In some embodiments, Figure 5 As shown, the step of extracting multimodal features includes extracting geometric features: S101, taking the center point position of the minimum circumscribed rectangle of each vertebra as the spatial position of the vertebra ; S102. Calculate geometric features using the Cobb measurement method.

[0028] Furthermore, if Figure 6 As shown, the step of calculating geometric features using the Cobb measurement method includes: S1021, the intersection angle of the normal line of the extended line of the adjacent vertebral outer edge As the intervertebral angle, the distance between the end points of the inner edge tangent as intervertebral space; S1022. The characteristic value of each vertebra is the average value of the calculation results of the two adjacent vertebrae. The first and last vertebrae directly use the single-side adjacent value. S1023. The spatial coordinates, intervertebral angles and intervertebral spaces are integrated into the geometric features of each vertebra to achieve quantitative characterization of the spatial morphology of the spinal vertebrae.

[0029] In some embodiments, Figure 7 As shown, the step of extracting multimodal features includes extracting image features: S103, taking the minimum circumscribed cuboid of each vertebra as the input size of the 3D convolutional neural network, and using zero-filling to uniformly fill the sizes of the remaining vertebrae; S104, extracting image features of each vertebra in turn through a 3D convolutional neural network.

[0030] Furthermore, the 3D convolutional neural network includes three 3D convolutional layers and three global pooling layers, each convolutional layer has batch normalization (BN) and an activation function (ReLU), the convolutional layer is used to perform a convolution operation on the input image to extract its image features, and the global pooling layer is used to reduce the dimension of the feature map obtained by the convolution, and finally output the image features of each vertebra.

[0031] In some embodiments, Figure 8 As shown, the step of extracting multimodal features includes extracting clinical features: S105. For the patient's clinical data, data preprocessing is used to achieve heterogeneous feature fusion; Specifically, data preprocessing is used to achieve heterogeneous feature fusion based on the clinical information of patients, such as bone density (continuous variable), age (continuous variable), gender (categorical variable), and fracture history (categorical variable).

[0032] Furthermore, if Fig. 9 As shown, the steps of implementing heterogeneous feature fusion by data preprocessing for the patient's clinical data include: S1051. Standardize continuous variables, such as bone density using Z-score standardization and age using Min-Max normalization to eliminate dimensional differences; S1052. Use one-hot encoding (e.g., gender converted to binary vector) or label encoding (e.g., fracture history converted to 0 / 1) for categorical variables (e.g., gender, fracture history, etc.) to avoid the model misjudging the order of categories; S106. Concatenate the preprocessed data into a multidimensional vector, which is used as the clinical feature of each vertebra.

[0033] Specifically, the step of splicing the preprocessed data into a multidimensional vector includes: The standardized continuous variables and the encoded categorical variables were concatenated into a multidimensional vector, which was used as the clinical characteristics of each vertebra.

[0034] S110, splicing and fusing the multimodal features, and constructing a spinal graph structure based on anatomical constraints according to spinal anatomical characteristics; In some embodiments, Fig.10 As shown, the steps of splicing and fusing the multimodal features and constructing a spinal graph structure based on anatomical constraints according to the anatomical characteristics of the spine include: S111. Each vertebra of the spine (T1-L5) is defined as a graph node. The feature vector of each node is spliced ​​and fused by three types of modal data: imaging, geometry, and clinical, to ensure efficient representation of multimodal information.

[0035] S112. Construct two types of edge connections based on the anatomical characteristics of the spine.

[0036] Furthermore, if Fig.11 As shown, the steps of constructing two types of edge connections based on the anatomical characteristics of the spine include: S1121. Adjacent vertebrae (such as T12-L1, L1-L2) are connected by rigid edges with a fixed weight of 1.0 to maintain the anatomical continuity of the spinal sequence. S1122: Flexible edge connection is used for intersegmental vertebrae, based on the spatial Euclidean distance. Bone density gradient Calculate the connection weights through the Gaussian kernel function ( ) quantifies the strength of mechanical connection between vertebrae. > threshold (e.g., ), establish connections, thereby screening out vertebral pairs with significant biomechanical conduction effects, and realizing the cross-scale structure establishment from a rigid anatomical framework to a flexible connection network, such as Figure 3 shown.

[0037] S120. Using multi-head graph attention neural network for global interaction of multimodal features and spinal fracture risk assessment.

[0038] Optionally, a 3-layer multi-head graph attention network (Multi-head GAT, head number = 4) is used for global interaction of multimodal features and fracture risk assessment. Figure 4 Schematic diagram of the first layer of the 3-layer multi-head graph attention network.

[0039] In some embodiments, Fig.12 As shown, the steps of using a multi-head graph attention neural network to perform global interaction of multimodal features and spinal fracture risk assessment include: S121, through the learnable parameter matrix Map the node features to the latent space, for each vertebral node and its neighbors , calculate the normalized attention weight: ; in, is the attention vector; S122, press Iteratively update node features so that the multimodal features of high-risk vertebrae diffuse to associated vertebrae through high-weight edges, where the high-weight edges are configured as Preset values, such as , It can be interpreted as anatomical connection strength × dynamic attention weight; S123. Output the probability of each vertebral fracture through the Sigmoid function.

[0040] In some embodiments, Fig.13 As shown, the steps include: S130, after mapping the final node features to risk probabilities (0-1) through a Sigmoid function, a risk heat map is generated through bicubic interpolation, and the heat map is superimposed on the sagittal plane reconstruction view of the original CT image; S140, the node color indicates the risk level, such as red for risk>0.7, orange for risk 0.4-0.7, green for risk<0.4, and edge transparency proportional to the vertebral body, and displays any vertebral body node and its associated vertebral body ( )connect.

[0041] The present invention provides a method for evaluating spinal osteoporotic fractures based on anatomical constraints and graph neural networks. First, the multimodal features of the spine are obtained in parallel through a hierarchical feature extraction module: geometric features reflecting the complex structure of the spine, image features reflecting the vertebral morphology and internal texture, and clinical features reflecting individual information are extracted. On this basis, a spinal vertebral graph structure is constructed based on anatomical constraints, each vertebra is defined as a graph node, and the node features are generated by multimodal data fusion. Two types of edge connections are established based on spinal anatomical constraints - rigid edges are used to force connections between adjacent vertebrae to maintain anatomical continuity, and flexible edge weights are calculated between intersegmental vertebrae through spatial distance and bone density gradient to achieve graph neural network structure modeling of spinal anatomical structural characteristics. A multi-head graph attention network is further used to capture the influence coefficient between vertebrae, realize the global interaction of multimodal features across vertebrae, and drive the high-risk vertebral risk signal to diffuse along the high-weight edge to the associated vertebrae through a learnable attention weight matrix, so as to realize the quantification of the fracture cascade effect. Finally, a visualization method was used to map the node color to the fracture risk level (red - high risk, green - low risk), the edge connection showed the attention connection between each vertebra and the other vertebrae, and the attention weight was used to intuitively reflect the mutual influence relationship between vertebrae. Example 2

[0042] A spinal osteoporotic fracture assessment system based on deep learning, using the above method, for a detailed description of the method, please refer to the corresponding description in the above method embodiment, which will not be repeated here. Fig.14 As shown, the system 200 includes a multimodal feature extraction module 210, a spinal graph structure construction module 220, and a spinal fracture risk assessment module 230; wherein, The multimodal feature extraction module is used to obtain the patient's spinal images and clinical data and extract multimodal features; The spinal graph structure construction module is used to splice and fuse the multimodal features and construct a spinal graph structure based on anatomical constraints according to the anatomical characteristics of the spine; The spinal fracture risk assessment module is used to use a multi-head graph attention neural network to perform global interaction of multimodal features and spinal fracture risk assessment.

[0043] Based on the technical solution of the above embodiment, optionally, the multimodal features include geometric features, imaging features, and clinical features.

[0044] Based on the technical solution of the above embodiment, optionally, the step of extracting multimodal features includes extracting geometric features: The center point of the minimum circumscribed rectangle of each vertebra is taken as the spatial position of the vertebra ; The geometrical characteristics are calculated using the Cobb measurement method.

[0045] Based on the technical solution of the above embodiment, optionally, the step of calculating the geometric features by using the Cobb measurement method includes: The intersection angle of the normal lines of the extended lines of the adjacent vertebral outer edges As the intervertebral angle, the distance between the end points of the inner edge tangent as intervertebral space; The characteristic value of each vertebra is the average value of the calculation results of two adjacent vertebrae; The spatial coordinates, intervertebral angles and intervertebral spaces are integrated into the geometric features of each vertebra to achieve quantitative characterization of the spatial morphology of the spinal vertebrae.

[0046] Based on the technical solution of the above embodiment, optionally, the step of extracting multimodal features includes extracting image features: The smallest circumscribed cuboid of each vertebra is used as the input size of the 3D convolutional neural network, and the size of the remaining vertebrae is uniformly filled by zero padding. The image features of each vertebra are extracted in turn through a 3D convolutional neural network.

[0047] Based on the technical solution of the above embodiment, optionally, the 3D convolutional neural network includes three 3D convolutional layers and three global pooling layers, each convolutional layer has batch normalization and activation function, the convolutional layer is used to perform convolution operation on the input image to extract its image features, and the global pooling layer is used to reduce the dimension of the feature map obtained by convolution, and finally output the image features of each vertebra.

[0048] Based on the technical solution of the above embodiment, optionally, the step of extracting multimodal features includes extracting clinical features: For the patients’ clinical data, data preprocessing is used to achieve heterogeneous feature fusion; The preprocessed data were concatenated into multidimensional vectors, which were used as the clinical characteristics of each vertebra.

[0049] Based on the technical solution of the above embodiment, optionally, the step of implementing heterogeneous feature fusion by data preprocessing for the patient's clinical data includes: Continuous variables were standardized to eliminate dimensional differences; Use one-hot encoding or label encoding for categorical variables to avoid the model misjudging the order of categories; The step of splicing the preprocessed data into a multidimensional vector comprises: The standardized continuous variables and the encoded categorical variables are concatenated into a multidimensional vector.

[0050] Based on the technical solution of the above embodiment, optionally, the step of splicing and fusing the multimodal features to construct a spinal graph structure based on anatomical constraints according to spinal anatomical characteristics includes: Each vertebra of the spine is defined as a graph node, and the feature vector of each node is spliced ​​and fused by three types of modality data: imaging, geometry, and clinical. Two types of edge connections are constructed based on the anatomical characteristics of the spine.

[0051] Based on the technical solution of the above embodiment, optionally, the step of constructing two types of edge connections according to the anatomical characteristics of the spine includes: Adjacent vertebrae are connected by rigid edges and weights are fixed to maintain the anatomical continuity of the spinal column sequence. For inter-segment vertebrae, flexible edge connections are used based on the spatial Euclidean distance. Bone density gradient Calculate the connection weights through the Gaussian kernel function Quantify the strength of mechanical connection between vertebrae. >threshold, connections are established to screen out vertebral pairs with significant biomechanical conduction effects, thereby achieving cross-scale structural establishment from a rigid anatomical framework to a flexible connection network.

[0052] Based on the technical solution of the above embodiment, optionally, the step of using a multi-head graph attention neural network to perform global interaction of multimodal features and spinal fracture risk assessment includes: Through the learnable parameter matrix Map the node features to the latent space, for each vertebral node and its neighbors , calculate the normalized attention weight: ; in, is the attention vector; according to Iteratively update node features so that the multimodal features of high-risk vertebrae diffuse to associated vertebrae through high-weight edges, where the high-weight edges are configured as Preset value, is anatomical connection strength × dynamic attention weight; The probability of each vertebral fracture is output through the Sigmoid function.

[0053] Based on the technical solution of the above embodiment, optionally, the method further includes the following steps: The risk heat map was generated by bicubic interpolation and superimposed on the sagittal reconstructed view of the original image; The node color indicates the risk level, and the edge transparency and proportional to the vertebral body, and shows the connection between any vertebral body node and its associated vertebral body.

[0054] The present invention provides a spinal osteoporotic fracture assessment system based on anatomical constraints and graph neural networks. First, the multimodal features of the spine are obtained in parallel through a hierarchical feature extraction module: geometric features reflecting the complex structure of the spine, image features reflecting the vertebral morphology and internal texture, and clinical features reflecting individual information are extracted. On this basis, a spinal vertebral graph structure is constructed based on anatomical constraints, each vertebra is defined as a graph node, the node features are generated by multimodal data fusion, and two types of edge connections are established based on spinal anatomical constraints - rigid edges are used to force connections between adjacent vertebrae to maintain anatomical continuity, and flexible edge weights are calculated between inter-segmental vertebrae through spatial distance and bone density gradient, so as to realize graph neural network structure modeling of spinal anatomical structure characteristics. A multi-head graph attention network is further used to capture the influence coefficient between vertebrae, realize the global interaction of multimodal features across vertebrae, and drive the high-risk vertebral risk signal to diffuse along the high-weight edge to the associated vertebrae through a learnable attention weight matrix, so as to realize the quantification of the fracture cascade effect. Finally, a visualization method was used to map the node color to the fracture risk level (red - high risk, green - low risk), the edge connection showed the attention connection between each vertebra and the other vertebrae, and the attention weight was used to intuitively reflect the mutual influence relationship between vertebrae. Example 3

[0055] A computer device 300, such as Fig.15 As shown, it includes a memory 310, a processor 320, and a computer program 330 stored in the memory and executable on the processor. When the processor executes the computer program, the steps of a method for evaluating spinal osteoporotic fractures based on deep learning are implemented. For a detailed description of the method, reference may be made to the corresponding description in the above method embodiment, which will not be repeated here. Example 4

[0056] A computer readable storage medium such as Fig.16As shown, a computer program is stored thereon, and when the computer program is executed by the processor, the steps of a method for evaluating spinal osteoporotic fractures based on deep learning are implemented. For a detailed description of the method, reference may be made to the corresponding description in the above method embodiment, and no further description is given here.

[0057] The number of devices and processing scales described here are used to simplify the description of the present invention. Applications, modifications and variations of the present invention will be obvious to those skilled in the art.

[0058] Although the embodiments of the present invention have been disclosed as above, they are not limited to the applications listed in the specification and implementation modes. They can be fully applied to various fields suitable for the present invention. For those familiar with the art, additional modifications can be easily implemented. Therefore, without departing from the general concept defined by the claims and the scope of equivalents, the present invention is not limited to the specific details and the illustrations shown and described herein.

[0059] The apparatus, computer device, non-volatile computer storage medium and method provided in the embodiments of this specification correspond to each other, and therefore, the apparatus, computer device and non-volatile computer storage medium also have similar beneficial technical effects as the corresponding method. Since the beneficial technical effects of the method have been described in detail above, the beneficial technical effects of the corresponding apparatus, computer device and non-volatile computer storage medium will not be repeated here.

[0060] Those skilled in the art also know that, in addition to implementing the controller in a purely computer-readable program code, the controller can be made to implement the same function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, such a controller can be considered as a hardware component, and the devices for implementing various functions included therein can also be considered as structures within the hardware component. Or even, the devices for implementing various functions can be considered as both software units for implementing the method and structures within the hardware component.

[0061] The systems, devices or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. For the convenience of description, the above devices are described separately by functions in various units. Of course, when implementing one or more embodiments of this specification, the functions of each unit can be implemented in the same or more software and / or hardware.

[0062] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, the embodiments of this specification may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0063] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0064] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0065] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0066] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0067] The specification may be described in the general context of computer-executable instructions executed by a computer, such as program units. Generally, program units include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communications network. In a distributed computing environment, program units may be located in local and remote computer storage media, including storage devices.

[0068] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0069] The above description is only an embodiment of this specification and is not intended to limit one or more embodiments of this specification. For those skilled in the art, one or more embodiments of this specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of one or more embodiments of this specification shall be included in the scope of the claims of one or more embodiments of this specification.

Claims

1. A method for evaluating spinal osteoporotic fractures based on deep learning, characterized in that: The following steps are involved: Obtain patient spinal images and clinical data, and extract multimodal features; The multimodal features are spliced ​​and fused to construct a spinal graph structure based on anatomical constraints according to the anatomical characteristics of the spine; A multi-head graph attention neural network is used for global interaction of multimodal features and spinal fracture risk assessment.

2. The method for evaluating spinal osteoporotic fractures based on deep learning according to claim 1, characterized in that: The multimodal features include geometric features, imaging features, and clinical features.

3. A method for evaluating spinal osteoporotic fractures based on deep learning as claimed in claim 2, characterized in that: The step of extracting multimodal features includes extracting geometric features: The center point of the minimum circumscribed rectangle of each vertebra is taken as the spatial position of the vertebra ; The geometrical characteristics are calculated using the Cobb measurement method.

4. A method for evaluating spinal osteoporotic fractures based on deep learning as claimed in claim 3, characterized in that: The step of calculating geometric features using the Cobb measurement method comprises: The intersection angle of the normal lines of the extended lines of the adjacent vertebral outer edges As the intervertebral angle, the distance between the end points of the inner edge tangent as intervertebral space; The characteristic value of each vertebra is the average value of the calculation results of two adjacent vertebrae; The spatial coordinates, intervertebral angles and intervertebral spaces are integrated into the geometric features of each vertebra to achieve quantitative characterization of the spatial morphology of the spinal vertebrae.

5. A method for evaluating spinal osteoporotic fractures based on deep learning as claimed in claim 2, characterized in that: The step of extracting multimodal features includes extracting image features: The smallest circumscribed cuboid of each vertebra is used as the input size of the 3D convolutional neural network, and the size of the remaining vertebrae is uniformly filled by zero padding. The image features of each vertebra are extracted in turn through a 3D convolutional neural network.

6. A method for evaluating spinal osteoporotic fractures based on deep learning as claimed in claim 5, characterized in that: The 3D convolutional neural network includes three 3D convolutional layers and three global pooling layers. Each convolutional layer has batch normalization and activation functions. The convolutional layer is used to perform convolution operations on the input image to extract its image features. The global pooling layer is used to reduce the dimension of the feature map obtained by convolution, and finally output the image features of each vertebra.

7. A method for evaluating spinal osteoporotic fractures based on deep learning as claimed in claim 2, characterized in that: The step of extracting multimodal features includes extracting clinical features: For the patients’ clinical data, data preprocessing is used to achieve heterogeneous feature fusion; The preprocessed data were concatenated into multidimensional vectors, which were used as the clinical characteristics of each vertebra.

8. A method for evaluating spinal osteoporotic fractures based on deep learning as claimed in claim 7, characterized in that: The steps of implementing heterogeneous feature fusion by data preprocessing for the patient's clinical data include: Continuous variables were standardized to eliminate dimensional differences; Use one-hot encoding or label encoding for categorical variables to avoid the model misjudging the order of categories; The step of splicing the preprocessed data into a multidimensional vector comprises: The standardized continuous variables and the encoded categorical variables are concatenated into a multidimensional vector.

9. A method for evaluating spinal osteoporotic fractures based on deep learning as claimed in claim 2, characterized in that: The step of splicing and fusing the multimodal features and constructing a spinal graph structure based on anatomical constraints according to the anatomical characteristics of the spine includes: Each vertebra of the spine is defined as a graph node, and the feature vector of each node is spliced ​​and fused by three types of modality data: imaging, geometry, and clinical. Two types of edge connections are constructed based on the anatomical characteristics of the spine.

10. A method for evaluating spinal osteoporotic fractures based on deep learning as claimed in claim 9, characterized in that: The steps of constructing two types of edge connections according to the anatomical characteristics of the spine include: Adjacent vertebrae are connected by rigid edges and weights are fixed to maintain the anatomical continuity of the spinal column sequence. For inter-segment vertebrae, flexible edge connections are used based on the spatial Euclidean distance. Bone density gradient Calculate the connection weights through the Gaussian kernel function Quantify the strength of mechanical connection between vertebrae. >threshold, connections are established to screen out vertebral pairs with significant biomechanical conduction effects, thereby achieving cross-scale structural establishment from a rigid anatomical framework to a flexible connection network.

11. The method for evaluating spinal osteoporotic fractures based on deep learning according to claim 9, characterized in that: The steps of using a multi-head graph attention neural network to perform global interaction of multimodal features and spinal fracture risk assessment include: Through the learnable parameter matrix Map the node features to the latent space, for each vertebral node and its neighbors , calculate the normalized attention weight: ； in, is the attention vector; according to Iteratively update node features so that the multimodal features of high-risk vertebrae diffuse to associated vertebrae through high-weight edges, where the high-weight edges are configured as Preset value, is anatomical connection strength × dynamic attention weight; The probability of each vertebral fracture is output through the Sigmoid function.

12. A method for evaluating spinal osteoporotic fractures based on deep learning as claimed in claim 11, characterized in that: Also includes the steps: The risk heat map was generated by bicubic interpolation and superimposed on the sagittal reconstructed view of the original image; The node color indicates the risk level, and the edge transparency and proportional to the vertebral body, and shows the connection between any vertebral body node and its associated vertebral body.

13. A spinal osteoporotic fracture assessment system based on deep learning, using the method according to any one of claims 1 to 12, characterized in that: It includes multimodal feature extraction module, spinal graph structure construction module, and spinal fracture risk assessment module; among them, The multimodal feature extraction module is used to obtain the patient's spinal images and clinical data and extract multimodal features; The spinal graph structure construction module is used to splice and fuse the multimodal features and construct a spinal graph structure based on anatomical constraints according to the anatomical characteristics of the spine; The spinal fracture risk assessment module is used to use a multi-head graph attention neural network to perform global interaction of multimodal features and spinal fracture risk assessment.

Images