Scoliosis identification method and system based on deep learning

Abstract

The invention relates to the technical field of artificial intelligence, and discloses a scoliosis identification method and system based on deep learning, and the method comprises the steps: converting high-precision point cloud data into a three-dimensional grid model; utilizing a 3D sparse convolution sub-network in a preset double-branch deep network to extract spinal column three-dimensional geometric features of the three-dimensional grid model, and utilizing a U-Net sub-network in the double-branch deep network to extract spinal column two-dimensional structural features of the X-ray image; generating a cross-modal joint feature map of the target patient, establishing a cascade attention module of the cross-modal joint feature map, outputting a feature vector sequence of the cross-modal joint feature map, and analyzing a spinal column long-term dependency relationship of the target patient; and analyzing the Cobb angle value, scoliosis type and severity grade of the target patient to construct a scoliosis report of the target patient. According to the invention, the scoliosis identification accuracy can be improved.

Description

A Deep Learning-Based Method and System for Scoliosis Recognition Technical Field

[0001] This invention relates to a deep learning-based method and system for identifying scoliosis, belonging to the field of artificial intelligence technology. Background Technology

[0002] Scoliosis identification refers to the use of specific methods or techniques to detect and diagnose abnormal lateral curvature of the human spine. Scoliosis identification is a crucial first step in maintaining spinal health, ensuring normal growth and development in adolescents, and preventing serious complications. It enables effective prevention and treatment, and has a profound impact on the long-term health and quality of life of patients.

[0003] Traditional methods of scoliosis identification primarily rely on clinical examinations by physicians (such as observing posture and performing the Adams flexion test) and X-ray measurements. Physicians visually observe asymmetry in the patient's appearance and combine this with indicators such as the Cobb angle measured on X-rays to determine the presence and severity of scoliosis. However, relying solely on two-dimensional X-rays cannot fully reflect the three-dimensional morphology of the spine, potentially leading to incomplete assessments. Furthermore, these methods fail to automatically analyze the complex spatial relationships and long-term dependencies between different spinal segments, resulting in poor accuracy in scoliosis identification. Summary of the Invention

[0004] This invention provides a deep learning-based method and system for scoliosis identification, the main purpose of which is to improve the accuracy of scoliosis identification.

[0005] To achieve the above objectives, this invention provides a deep learning-based scoliosis identification method, comprising: collecting spinal data of a target patient, wherein the spinal data includes: high-precision point cloud data of the patient's spinal surface and X-ray images; converting the high-precision point cloud data into a three-dimensional mesh model; extracting the three-dimensional geometric features of the spine from the three-dimensional mesh model using a 3D sparse convolutional sub-network in a pre-defined dual-branch deep network, and extracting the two-dimensional structural features of the spine from the X-ray images using a U-Net sub-network in the dual-branch deep network; combining the three-dimensional geometric features and the two-dimensional structural features of the spine to generate a cross-modal joint feature map of the target patient; establishing a cascaded attention module for the cross-modal joint feature map, wherein the cascaded attention module includes a spatial attention sub-module and a channel attention sub-module; outputting a feature vector sequence of the cross-modal joint feature map based on the cascaded attention module, and analyzing the long-term spinal dependency of the target patient based on the feature vector sequence; and analyzing the Cobb angle, scoliosis type, and severity classification of the target patient based on the long-term spinal dependency and the feature vector sequence to construct a scoliosis report for the target patient.

[0006] Optionally, converting the high-precision point cloud data into a three-dimensional mesh model includes: identifying outliers in the high-precision point cloud data; filtering the high-precision point cloud data based on the outliers to obtain filtered high-precision point cloud data; registering the filtered high-precision point cloud data to obtain registered high-precision point cloud data; calculating the surface normal vectors of the registered high-precision point cloud data to perform surface reconstruction on the registered high-precision point cloud data to obtain a triangular mesh; and performing post-mesh processing on the triangular mesh to obtain the three-dimensional mesh model.

[0007] Optionally, calculating the surface normal vector of the registered high-precision point cloud data includes: marking the target point of the registered high-precision point cloud data; analyzing the local curvature weight and spine topological weight of the target point; calculating the neighborhood centroid of the target point; and calculating the surface normal vector of the registered high-precision point cloud data based on the local curvature weight, the spine topological weight, and the neighborhood centroid.

[0008] Optionally, the step of extracting the three-dimensional geometric features of the spine of the three-dimensional mesh model using a 3D sparse convolutional subnetwork in a preset dual-branch deep network includes: converting the three-dimensional mesh model into a sparse voxel mesh; extracting the sparse tensor of the sparse voxel mesh using sparse convolutional layers in the 3D sparse convolutional subnetwork; downsampling the sparse tensor using submanifold downsampling layers in the 3D sparse convolutional subnetwork to obtain a downsampled sparse tensor; transposing the downsampled sparse tensor using sparse transpose convolutional layers in the 3D sparse convolutional subnetwork to obtain a transposed sparse tensor; and combining the sparse tensor, the downsampled sparse tensor, and the transposed sparse tensor to extract the three-dimensional geometric features of the spine of the three-dimensional mesh model.

[0009] Optionally, the step of extracting the two-dimensional spinal structure features of the X-ray image using the U-Net sub-network in the dual-branch deep network includes: normalizing the grayscale of the X-ray image to obtain a normalized X-ray image; extracting the deep feature map of the normalized X-ray image using the encoder in the U-Net sub-network; and outputting the two-dimensional spinal structure features of the X-ray image using the decoder in the U-Net sub-network.

[0010] Optionally, generating the cross-modal joint feature map of the target patient by combining the three-dimensional geometric features and the two-dimensional structural features of the spine includes: spatializing the three-dimensional geometric features of the spine to obtain spatialized three-dimensional spinal features; reducing the dimensionality of the two-dimensional structural features of the spine to obtain reduced-dimensional two-dimensional spinal features; calculating the cross-modal interaction matrix of the spatialized three-dimensional spinal features and the reduced-dimensional two-dimensional spinal features; defining anatomical constraints on the spatialized three-dimensional spinal features and the reduced-dimensional two-dimensional spinal features; and fusing the spatialized three-dimensional spinal features and the reduced-dimensional two-dimensional spinal features based on the anatomical constraints and the cross-modal interaction matrix to obtain the cross-modal joint feature map of the target patient.

[0011] Optionally, the step of outputting the feature vector sequence of the cross-modal joint feature map based on the cascaded attention module includes: generating a max-pooling spatial feature map and an average-pooling spatial feature map of the cross-modal joint feature map; calculating the spatial weights of the cross-modal joint feature map using the spatial attention submodule in the cascaded attention module based on the max-pooling spatial feature map and the average-pooling spatial feature map; enhancing the cross-modal joint feature map based on the spatial weights to obtain an enhanced joint feature map; calculating the channel weights of the enhanced joint feature map using the channel attention submodule in the cascaded attention module; and outputting the feature vector sequence of the enhanced joint feature map based on the channel weights.

[0012] Optionally, the step of analyzing the long-term spinal dependence of the target patient based on the feature vector sequence includes: analyzing the biomechanical conduction path and modeling inverse mechanical feedback of the target patient; determining the bidirectional vertebral body hiding state of the target patient based on the biomechanical conduction path and modeling inverse mechanical feedback; establishing a spinal topology dependency graph of the target patient based on the bidirectional vertebral body hiding state; and analyzing the long-term spinal dependence of the target patient through the spinal topology dependency graph.

[0013] Optionally, the step of analyzing the Cobb angle, scoliosis type, and severity classification of the target patient based on the long-term spinal dependence relationship and the feature vector sequence includes: identifying the upper and lower scoliosis bodies of the target patient based on the feature vector sequence; calculating the upper and lower scoliosis body normal vectors; calculating the Cobb angle of the target patient based on the upper and lower scoliosis body normal vectors; analyzing the curvature pattern of the target patient based on the long-term spinal dependence relationship and the feature vector sequence to determine the scoliosis type of the target patient; and analyzing the severity classification of the target patient based on the Cobb angle value and the scoliosis type.

[0014] To address the aforementioned problems, this invention also provides a deep learning-based scoliosis recognition system. The system includes: a spinal data processing module for collecting spinal data from a target patient, wherein the spinal data includes high-precision point cloud data of the patient's spinal surface and X-ray images, and converting the high-precision point cloud data into a three-dimensional mesh model; a spinal feature extraction module for extracting three-dimensional geometric features of the spine from the three-dimensional mesh model using a 3D sparse convolutional sub-network in a preset dual-branch deep network, and extracting two-dimensional structural features of the spine from the X-ray images using a U-Net sub-network in the dual-branch deep network; and a spinal feature fusion module for combining the three-dimensional geometric features of the spine with the spinal features from the X-ray images. The system utilizes two-dimensional column structure features to generate a cross-modal joint feature map of the target patient. A cascaded attention module for the cross-modal joint feature map is established, comprising a spatial attention submodule and a channel attention submodule. A spinal dependency analysis module is used to output a feature vector sequence of the cross-modal joint feature map based on the cascaded attention module, and to analyze the long-term spinal dependency of the target patient based on the feature vector sequence. A scoliosis report construction module is used to analyze the Cobb angle, scoliosis type, and severity classification of the target patient based on the long-term spinal dependency and the feature vector sequence, in order to construct a scoliosis report for the target patient.

[0015] First, by acquiring high-precision point cloud data and constructing a 3D mesh model, and then using a 3D sparse convolutional sub-network to extract geometric features, the precise spatial morphology of the spine, subtle deviations in vertebral alignment, and complex changes in the overall curve can be captured. This 3D perspective overcomes the problem of insufficient information dimension that may exist in traditional X-ray image analysis, making the assessment of scoliosis more comprehensive and in-depth. At the same time, the parallel processing U-Net sub-network extracts 2D structural features from X-ray images, retaining important visual cues relied upon by doctors in traditional medical imaging diagnosis, such as bone density and projection morphology. This dual-branch structure design allows the model to utilize both 3D fine geometric information and 2D macroscopic structural information, achieving data complementarity and improving the robustness and accuracy of feature representation. More importantly, by fusing 3D geometric and 2D structural features to generate cross-modal joint feature maps and introducing cascaded attention, the model achieves this goal. The module (including spatial and channel attention) intelligently focuses on the regions and feature dimensions most relevant to scoliosis in images. Spatial attention ensures focus on key spinal anatomical structures (such as vertebral bodies and spinous processes), while channel attention optimizes the weight allocation of different feature channels (such as curvature, density, and edges). This self-supervised learning approach allows the model to dynamically adapt to the image characteristics of different patients, effectively suppressing noise and irrelevant information, thereby extracting more diagnostically valuable feature vector sequences. Based on these attention-optimized feature vector sequences, the model can analyze long-term spinal dependencies, i.e., the correlations between different vertebral bodies and different segments. This is crucial for understanding the overall pattern and progression trend of scoliosis. Ultimately, it accurately analyzes Cobb angle values, identifies scoliosis types, and assesses severity grading, providing strong technical support for constructing detailed and objective scoliosis reports. Therefore, this invention can improve the accuracy of scoliosis identification. Attached Figure Description

[0016] Figure 1 is a flowchart illustrating a deep learning-based scoliosis recognition method according to an embodiment of the present invention; Figure 2 is a module diagram illustrating the implementation of the deep learning-based scoliosis recognition method according to an embodiment of the present invention.

[0017] The objectives, features, and advantages of this invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0018] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0019] This application provides a deep learning-based method for scoliosis recognition. The execution entity of the deep learning-based scoliosis recognition method includes, but is not limited to, at least one of the following electronic devices that can be configured to execute the method provided in this application: a server, a terminal, etc. In other words, the deep learning-based scoliosis recognition method can be executed by software or hardware installed on a terminal device or a server device. The server includes, but is not limited to, a single server, a server cluster, a cloud server, or a cloud server cluster.

[0020] Referring to Figure 1, a flowchart illustrating a deep learning-based scoliosis identification method according to an embodiment of the present invention is shown. In this embodiment, the deep learning-based scoliosis identification method includes: S1, acquiring spinal data of a target patient, wherein the spinal data includes: high-precision point cloud data and X-ray images of the patient's spinal surface, and converting the high-precision point cloud data into a three-dimensional mesh model.

[0021] It should be explained that the target patient refers to the specific object that needs to be identified and analyzed for scoliosis; the high-precision point cloud data of the patient's spinal surface refers to the three-dimensional data representation method of the exposed part of the skin and subcutaneous tissue of the patient's back spine; and the X-ray image refers to a highly penetrating electromagnetic wave that can pass through human soft tissue and form an image on film or digital detector.

[0022] This invention converts the high-precision point cloud data into a three-dimensional mesh model, providing a foundation for subsequent three-dimensional feature extraction.

[0023] In detail, the process of converting the high-precision point cloud data into a three-dimensional mesh model includes: identifying outliers in the high-precision point cloud data; filtering the high-precision point cloud data based on the outliers to obtain filtered high-precision point cloud data; registering the filtered high-precision point cloud data to obtain registered high-precision point cloud data; calculating the surface normal vectors of the registered high-precision point cloud data to perform surface reconstruction on the registered high-precision point cloud data to obtain a triangular mesh; and performing post-mesh processing on the triangular mesh to obtain the three-dimensional mesh model.

[0024] Here, outliers refer to those points that are significantly deviated from their surrounding points in spatial location; filtered high-precision point cloud data refers to point cloud data after denoising processing; registered high-precision point cloud data refers to aligning data from different viewpoints or different scans to the same global coordinate system so that they can form a continuous and complete representation of the spine surface; surface normal vector refers to a vector perpendicular to a point on the object surface; triangular mesh refers to a triangular mesh output form composed of a series of interconnected triangular facets; and the three-dimensional mesh model refers to the final cleaned and optimized three-dimensional spine surface model.

[0025] Optionally, the identification of outliers in the high-precision point cloud data can be analyzed by calculating the average distance and standard deviation of each point to its k nearest neighbors (k=50, adjusted according to the point cloud density).

[0026] Further, calculating the surface normal vector of the registered high-precision point cloud data includes: marking the target point of the registered high-precision point cloud data; analyzing the local curvature weight and spine topological weight of the target point; calculating the neighborhood centroid of the target point; and calculating the surface normal vector of the registered high-precision point cloud data based on the local curvature weight, the spine topological weight, and the neighborhood centroid. Further, as another embodiment of the present invention, the surface normal vector is calculated using the following formula:

[0027] in, Indicates the target points in the registered high-precision point cloud data Surface normal vector, Indicates the target points in the registered high-precision point cloud data The three-dimensional coordinates of the neighborhood centroid. Indicates the target points in the registered high-precision point cloud data The neighborhood set, Indicates the target points in the registered high-precision point cloud data Neighborhood target points The three-dimensional coordinates Represents local curvature weights. Represents the topological weights of the spine. Indicates the target points in the registered high-precision point cloud data The transpose of the surface normal vector, arg min denotes the minimum value function.

[0028] It is understandable that the formula for calculating the surface normal vector incorporates local curvature weights. and spinal topological weights This makes the normal vector stable in flat areas of the cone and noise-resistant in edge or cross-structure areas, balancing the contributions of geometric features and anatomical structures to improve the accuracy of surface normal vector calculation.

[0029] Wherein, the target point refers to the point on which the surface normal vector is calculated; the local curvature weight refers to the degree of curvature of the surface in the region where the target point is located; the spinal topology weight refers to the position of the target point on the spine (e.g., which vertebra or part of the vertebra) and its relative relationship with the center lines of other vertebrae; the neighborhood centroid refers to the average value of the coordinates of the neighborhood points; and the transpose of the surface normal vector refers to the transpose of the surface normal vector. For example, if the surface normal vector is a row vector, then the transpose of the surface normal vector is a column vector.

[0030] S2. Extract the three-dimensional geometric features of the spine from the three-dimensional mesh model using the 3D sparse convolutional sub-network in the preset dual-branch deep network, and extract the two-dimensional structural features of the spine from the X-ray image using the U-Net sub-network in the dual-branch deep network.

[0031] This invention utilizes a 3D sparse convolutional subnetwork in a pre-defined dual-branch deep network to extract the three-dimensional geometric features of the spine from the three-dimensional mesh model, enabling efficient and effective extraction of fine three-dimensional geometric features for spine analysis from the three-dimensional mesh model.

[0032] In detail, the step of extracting the three-dimensional geometric features of the spine of the three-dimensional mesh model using a 3D sparse convolutional subnetwork in a preset dual-branch deep network includes: converting the three-dimensional mesh model into a sparse voxel mesh; extracting the sparse tensor of the sparse voxel mesh using sparse convolutional layers in the 3D sparse convolutional subnetwork; downsampling the sparse tensor using submanifold downsampling layers in the 3D sparse convolutional subnetwork to obtain a downsampled sparse tensor; transposing the downsampled sparse tensor using sparse transpose convolutional layers in the 3D sparse convolutional subnetwork to obtain a transposed sparse tensor; and combining the sparse tensor, the downsampled sparse tensor, and the transposed sparse tensor to extract the three-dimensional geometric features of the spine of the three-dimensional mesh model.

[0033] The dual-branch deep network refers to a deep learning network structure containing two parallel processing paths. The 3D sparse convolutional sub-network refers to a sub-network used to process 3D sparse data (such as sparse voxel meshes) and extract features using sparse convolution operations. The sparse voxel mesh refers to a representation of the original 3D mesh model. The sparse voxel mesh only stores voxels that contain geometric information of the model (e.g., voxels that intersect with or are very close to the surface of the model) and records their positions and attributes (such as geometric features or original data). The sparse convolutional layer refers to the basic computational unit in the 3D sparse convolutional sub-network, which extracts local spatial features from the input sparse voxel mesh (or the output of the previous sparse convolutional layer). The sparse tensor refers to the data structure of the sparse voxel mesh after processing by the sparse convolutional layer. The submanifold downsampling layer refers to a downsampling layer designed for sparse data. The downsampled sparse tensor refers to the result obtained after the submanifold downsampling layer processes the sparse tensor. It has a lower spatial resolution than the input sparse tensor. The sparse transposed convolutional layer refers to a "deconvolution" operation specifically designed for sparse data. The transposed sparse tensor is the result obtained after the sparse transposed convolutional layer processes and downsamples the sparse tensor. The three-dimensional geometric features of the spine refer to the quantitative representation extracted from the three-dimensional mesh model that can describe the geometric attributes of the spine, such as shape, structure, position, and orientation.

[0034] Optionally, the extraction of the sparse tensor of the sparse voxel grid using the sparse convolutional layer in the 3D sparse convolutional subnetwork can be obtained by calculating the dot product of the features and corresponding kernel weights of each active voxel in the sparse voxel grid, and then summing the results.

[0035] This invention utilizes the U-Net subnetwork in the dual-branch deep network to extract the two-dimensional structural features of the spine from the X-ray image, resulting in a feature map with the same size as the input image and containing rich two-dimensional structural information.

[0036] In detail, the step of extracting the two-dimensional spinal structure features of the X-ray image using the U-Net sub-network in the dual-branch deep network includes: normalizing the grayscale of the X-ray image to obtain a normalized X-ray image; extracting the deep feature map of the normalized X-ray image using the encoder in the U-Net sub-network; and outputting the two-dimensional spinal structure features of the X-ray image using the decoder in the U-Net sub-network.

[0037] The U-Net sub-network refers to a convolutional neural network (CNN) architecture used to extract two-dimensional structural features of the spine from X-ray images. The normalized X-ray image refers to the X-ray image after preprocessing (grayscale normalization). The encoder refers to the left part of the "U" shape in the U-Net structure. It consists of a series of convolutional layers, activation functions (such as ReLU), and pooling layers (such as max pooling). The deep feature maps refer to the feature representations generated by the encoder after multiple convolutional and pooling operations. These feature maps typically have a small spatial size (low resolution) and a large number of channels. The decoder refers to the right part of the "U" shape in the U-Net structure. It consists of a series of upsampling operations (such as transposed convolution, interpolation upsampling followed by convolution) and convolutional layers, and is connected to the encoder through skip connections. The two-dimensional structural features of the spine refer to the representations extracted from X-ray images by the U-Net sub-network (mainly through the collaborative work of the encoder and decoder) that characterize the geometric structure, position, and morphological information of the spine in a two-dimensional plane.

[0038] Optionally, the method for normalizing the X-ray image to obtain a normalized X-ray image is Min-Max normalization.

[0039] S3. Combining the three-dimensional geometric features of the spine and the two-dimensional structural features of the spine, a cross-modal joint feature map of the target patient is generated, and a cascaded attention module of the cross-modal joint feature map is established, wherein the cascaded attention module includes a spatial attention submodule and a channel attention submodule.

[0040] The present invention combines the three-dimensional geometric features and the two-dimensional structural features of the spine to generate a cross-modal joint feature map of the target patient, which can improve the reliability of subsequent spinal analysis.

[0041] In detail, the step of generating a cross-modal joint feature map of the target patient by combining the three-dimensional geometric features and the two-dimensional structural features of the spine includes: spatializing the three-dimensional geometric features of the spine to obtain spatialized three-dimensional spinal features; reducing the dimensionality of the two-dimensional structural features of the spine to obtain dimensionality-reduced two-dimensional spinal features; calculating the cross-modal interaction matrix of the spatialized three-dimensional spinal features and the dimensionality-reduced two-dimensional spinal features; defining anatomical constraints on the spatialized three-dimensional spinal features and the dimensionality-reduced two-dimensional spinal features; and fusing the spatialized three-dimensional spinal features and the dimensionality-reduced two-dimensional spinal features based on the anatomical constraints and the cross-modal interaction matrix to obtain the cross-modal joint feature map of the target patient.

[0042] The spatialized 3D spinal features refer to expanding the 128-dimensional vector of the spatialized 3D spinal features into a spatial dimension that matches the 2D feature map through a fully connected layer. The dimensionality-reduced 2D spinal features refer to the representation of the 32×32×512 feature map of the deepest layer of U-Net after compression into 32×32×128 using 1×1 convolution. The cross-modal interaction matrix refers to the relationship between the features of the spatialized 3D spinal features and the dimensionality-reduced 2D spinal features. The anatomical constraints refer to prior knowledge based on the known biological, mechanical, and spatial relationships of the human spine. In this invention, it means forcing the weight of the midline region of the spine (defined by a pre-segmentation mask) in the attention map to be ≥0.7 to ensure that key anatomical structures are not suppressed. The cross-modal joint feature map refers to the final feature representation obtained by fusing the spatialized 3D spinal features and the dimensionality-reduced 2D spinal features and utilizing the cross-modal interaction matrix and anatomical constraints.

[0043] Optionally, the cross-modal interaction matrix of the spatialized three-dimensional spine features and the reduced two-dimensional spine features can be calculated using an attention mechanism, such as an attention mechanism: allowing a feature of one modality (e.g., a three-dimensional feature) to "attention" to different parts of a feature of another modality (e.g., a two-dimensional feature), and learning a weight matrix to represent the degree of attention. This weight matrix is ​​the interaction matrix.

[0044] It should be explained that the spatial attention submodule refers to the module that enhances the spatial weight of key anatomical regions of the spine (such as vertebral endplates and lateral curvature apex) and suppresses irrelevant background, while the channel attention submodule refers to the module that dynamically adjusts the weights of different channels to highlight effective information in multimodal features (such as 3D geometric curvature channel vs. 2D texture channel).

[0045] S4. Based on the cascaded attention module, output the feature vector sequence of the cross-modal joint feature map, and analyze the long-term spinal dependence of the target patient according to the feature vector sequence.

[0046] The present invention, based on the cascaded attention module, outputs a feature vector sequence of the cross-modal joint feature map, which can intelligently focus on the regions and feature dimensions most relevant to scoliosis in the image, thereby extracting a feature vector sequence with greater diagnostic value.

[0047] In detail, the step of outputting the feature vector sequence of the cross-modal joint feature map based on the cascaded attention module includes: generating a max-pooling spatial feature map and an average-pooling spatial feature map of the cross-modal joint feature map; calculating the spatial weights of the cross-modal joint feature map using the spatial attention submodule in the cascaded attention module based on the max-pooling spatial feature map and the average-pooling spatial feature map; enhancing the cross-modal joint feature map based on the spatial weights to obtain an enhanced joint feature map; calculating the channel weights of the enhanced joint feature map using the channel attention submodule in the cascaded attention module; and outputting the feature vector sequence of the enhanced joint feature map based on the channel weights.

[0048] Wherein, the max-pooling spatial feature map refers to the feature map generated by applying max pooling to the input cross-modal joint feature map; the average-pooling spatial feature map refers to the feature map generated by applying average pooling to the input cross-modal joint feature map; the spatial weights refer to the weight map calculated by the spatial attention sub-module; the enhanced joint feature map refers to the result obtained after applying the spatial attention mechanism to the cross-modal joint feature map; the channel weights refer to the weight vectors calculated by the channel attention sub-module; and the feature vector sequence refers to the final output feature representation used for subsequent tasks (such as Cobb angle prediction, side bending classification, etc.).

[0049] Optionally, in generating the max-pooling spatial feature map and the average-pooling spatial feature map of the cross-modal joint feature map, the max-pooling spatial feature map refers to taking the maximum value along the channel dimension at each spatial location (e.g., each pixel or voxel) of the feature map, and the average-pooling spatial feature map refers to taking the average value along the channel dimension at each spatial location (e.g., each pixel or voxel) of the feature map.

[0050] Optionally, the enhancement of the cross-modal joint feature map based on the spatial weights can be obtained by multiplying the calculated spatial weights (which usually need to be normalized to the 0-1 range by a Sigmoid function) element-wise with the original "cross-modal joint feature map".

[0051] Based on the feature vector sequence, this invention analyzes the long-term spinal dependence of the target patient, which can analyze the correlation between different vertebrae and different segments. This is crucial for understanding the overall pattern and progression trend of scoliosis.

[0052] In detail, the step of analyzing the long-term spinal dependence of the target patient based on the feature vector sequence includes: analyzing the biomechanical conduction path and modeling inverse mechanical feedback of the target patient; determining the bidirectional vertebral body hiding state of the target patient based on the biomechanical conduction path and modeling inverse mechanical feedback; establishing a spinal topology dependency graph of the target patient based on the bidirectional vertebral body hiding state; and analyzing the long-term spinal dependence of the target patient through the spinal topology dependency graph.

[0053] The biomechanical conduction path refers to the progressive transmission process of abnormal mechanical effects (such as vertebral rotation and increased curvature) along the spinal sequence from the proximal (cephalic) to the distal (caudal) side in scoliosis. The modeling of reverse mechanical feedback refers to describing the mechanical reaction of distal vertebrae (such as lumbar vertebrae and pelvis) to proximal vertebrae (such as thoracic vertebrae). The bidirectional hidden state of vertebrae refers to the fused feature vector of each vertebra, which contains both forward conduction and reverse feedback information. The spinal topological dependency graph refers to a graph structure with vertebrae as nodes and biomechanical dependencies as edges. The long-term spinal dependency relationship refers to a non-local dependency relationship that spans multiple vertebrae or a long distance in the spinal sequence or structure. For example, severe scoliosis of the upper thoracic vertebrae may affect the alignment and curvature of the lumbar vertebrae through mechanical conduction and compensation mechanisms, which is a long-term dependency relationship.

[0054] Optionally, the determination of the bidirectional hidden state of the vertebral body of the target patient based on the biomechanical conduction path and modeled inverse mechanical feedback can be achieved by processing sequences or models of sequential dependencies (such as bidirectional LSTM / GRU), and calculating a hidden state for each vertebral body based on the analyzed biomechanical conduction path and modeled inverse mechanical feedback.

[0055] S5. Based on the long-term spinal dependence and the feature vector sequence, analyze the Cobb angle value, scoliosis type and severity grade of the target patient to construct a scoliosis report for the target patient.

[0056] Based on the long-term spinal dependence and the feature vector sequence, this invention analyzes the Cobb angle, scoliosis type, and severity grading of the target patient, providing strong technical support for constructing a detailed and objective scoliosis report.

[0057] In detail, the step of analyzing the Cobb angle, scoliosis type, and severity classification of the target patient based on the long-term spinal dependence relationship and the feature vector sequence includes: identifying the upper and lower scoliosis bodies of the target patient based on the feature vector sequence; calculating the normal vectors of the upper and lower scoliosis bodies; calculating the Cobb angle of the target patient based on the normal vectors of the upper and lower scoliosis bodies; analyzing the curvature pattern of the target patient based on the long-term spinal dependence relationship and the feature vector sequence to determine the scoliosis type of the target patient; and analyzing the severity classification of the target patient based on the Cobb angle and the scoliosis type.

[0058] In this context, the upper vertebra of the scoliosis refers to the uppermost vertebra along the scoliosis curve, and the lower vertebra refers to the lowermost vertebra along the scoliosis curve. The normal vector of the upper vertebra is the normal vector of the principal plane of the upper vertebra, and the normal vector of the lower vertebra is the normal vector of the principal plane of the lower vertebra. The Cobb angle is the angle between the planes of the upper and lower vertebrae along the scoliosis curve, used to quantify the severity of the scoliosis. The curvature pattern refers to the deformation characteristics of the scoliosis in three-dimensional space, including the curvature direction (left / right), curvature location (thoracic / lumbar), and curvature shape (single / double curvature). The scoliosis type refers to the subtypes of scoliosis classified according to the curvature pattern and location, including single-curvature, double-curvature, and triple-curvature types. The severity grading refers to the severity of the condition classified according to the Cobb angle and scoliosis type. For example, mild: Cobb angle < 20°, generally no intervention is required. Moderate: 20° ≤ Cobb angle < 40°, observation or bracing may be required. Severe: Cobb angle value ≥ 40°, usually requiring surgical intervention.

[0059] Optionally, based on the feature vector sequence, the identification of the upper and lower scoliosis bodies of the target patient can be achieved by edge detection to detect abrupt changes or edges in the feature vector sequence, thereby identifying the upper and lower scoliosis bodies.

[0060] Furthermore, as another embodiment of the present invention, the Cobb angle value is calculated using the following formula:

[0061] in, This represents the Cobb angle value of the target patient. This represents the body normal vector at the upper end of the lateral bend. This represents the body normal vector at the lower end of the lateral bend. This represents the inverse cosine function.

[0062] Yes, it is understandable that the Cobb angle calculation formula uses a vector method to calculate the Cobb angle, which can more accurately reflect the degree of scoliosis and reduce human error. The normal vector of the upper end of the scoliosis... and the normal vector of the lower end of the side bend These were extracted separately from the upper and lower ends of the scoliosis body to determine the direction of the spinal curvature. These are the body normal vectors of the upper end of the side bend. and the normal vector of the lower end of the side bend The modulus is used to normalize the vector, ensuring that the calculation is not affected by the vector length. The inverse cosine function is used to calculate the angle between two unit vectors.

[0063] Finally, this invention constructs a scoliosis report for the target patient to determine the degree of scoliosis. The scoliosis report is a detailed written document or data structure regarding the target patient's scoliosis condition, including: the target patient's Cobb angle value, scoliosis type, severity classification, overall spinal balance, and an estimate of vertebral rotation.

[0064] First, by acquiring high-precision point cloud data and constructing a 3D mesh model, and then using a 3D sparse convolutional sub-network to extract geometric features, the precise spatial morphology of the spine, subtle deviations in vertebral alignment, and complex changes in the overall curve can be captured. This 3D perspective overcomes the problem of insufficient information dimension that may exist in traditional X-ray image analysis, making the assessment of scoliosis more comprehensive and in-depth. At the same time, the parallel processing U-Net sub-network extracts 2D structural features from X-ray images, retaining important visual cues relied upon by doctors in traditional medical imaging diagnosis, such as bone density and projection morphology. This dual-branch structure design allows the model to utilize both 3D fine geometric information and 2D macroscopic structural information, achieving data complementarity and improving the robustness and accuracy of feature representation. More importantly, by fusing 3D geometric and 2D structural features to generate cross-modal joint feature maps and introducing cascaded attention, the model achieves this goal. The module (including spatial and channel attention) intelligently focuses on the regions and feature dimensions most relevant to scoliosis in images. Spatial attention ensures focus on key spinal anatomical structures (such as vertebral bodies and spinous processes), while channel attention optimizes the weight allocation of different feature channels (such as curvature, density, and edges). This self-supervised learning approach allows the model to dynamically adapt to the image characteristics of different patients, effectively suppressing noise and irrelevant information, thereby extracting more diagnostically valuable feature vector sequences. Based on these attention-optimized feature vector sequences, the model can analyze long-term spinal dependencies, i.e., the correlations between different vertebral bodies and different segments. This is crucial for understanding the overall pattern and progression trend of scoliosis. Ultimately, it accurately analyzes Cobb angle values, identifies scoliosis types, and assesses severity grading, providing strong technical support for constructing detailed and objective scoliosis reports. Therefore, this invention can improve the accuracy of scoliosis identification.

[0065] Figure 2 shows a functional block diagram of a deep learning-based scoliosis recognition system according to the present invention.

[0066] The deep learning-based scoliosis recognition system 200 described in this invention can be installed in an electronic device. Depending on the functions implemented, the deep learning-based scoliosis recognition system may include a spinal data processing module 201, a spinal feature extraction module 202, a spinal feature fusion module 203, a spinal dependency analysis module 204, and a scoliosis report construction module 205. The modules described in this invention can also be referred to as units, which are a series of computer program segments that can be executed by the processor of an electronic device and perform a fixed function, and are stored in the memory of the electronic device.

[0067] In this embodiment of the invention, the functions of each module / unit are as follows: The spinal data processing module 201 is used to collect spinal data of the target patient, wherein the spinal data includes: high-precision point cloud data and X-ray images of the patient's spinal surface, and converts the high-precision point cloud data into a three-dimensional mesh model; The spinal feature extraction module 202 is used to extract the three-dimensional geometric features of the spine from the three-dimensional mesh model using a 3D sparse convolutional sub-network in a preset dual-branch deep network, and to extract the two-dimensional structural features of the spine from the X-ray images using a U-Net sub-network in the dual-branch deep network; The spinal feature fusion module 203 is used to combine the three-dimensional geometric features of the spine and the two-dimensional structural features of the spine. The system generates a cross-modal joint feature map of the target patient and establishes a cascaded attention module for the cross-modal joint feature map, wherein the cascaded attention module includes a spatial attention submodule and a channel attention submodule; the spinal dependency analysis module 204 is used to output a feature vector sequence of the cross-modal joint feature map based on the cascaded attention module, and analyze the long-term spinal dependency of the target patient based on the feature vector sequence; the scoliosis report construction module 205 is used to analyze the Cobb angle value, scoliosis type and severity grade of the target patient based on the long-term spinal dependency and the feature vector sequence, so as to construct a scoliosis report for the target patient.

[0068] In detail, each module in the deep learning-based scoliosis recognition system 200 described in this embodiment of the invention uses the same technical means as the deep learning-based scoliosis recognition method described in Figure 1 above, and can produce the same technical effect, which will not be repeated here.

[0069] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0070] Finally, it should be noted that in the above embodiments, each embodiment can be combined with each other or independent. Deleting any one of them will not affect the technical implementation of other embodiments. The above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A deep learning-based method for scoliosis identification, characterized in that, The method includes: acquiring spinal data of a target patient, wherein the spinal data includes: high-precision point cloud data and X-ray images of the patient's spinal surface; converting the high-precision point cloud data into a three-dimensional mesh model; extracting the three-dimensional geometric features of the spine from the three-dimensional mesh model using a 3D sparse convolutional sub-network in a pre-defined dual-branch deep network; extracting the two-dimensional structural features of the spine from the X-ray images using a U-Net sub-network in the dual-branch deep network; combining the three-dimensional geometric features and the two-dimensional structural features of the spine to generate a cross-modal joint feature map of the target patient; establishing a cascaded attention module for the cross-modal joint feature map, wherein the cascaded attention module includes a spatial attention sub-module and a channel attention sub-module; outputting a feature vector sequence of the cross-modal joint feature map based on the cascaded attention module; analyzing the long-term spinal dependence of the target patient based on the feature vector sequence; and analyzing the Cobb angle, scoliosis type, and severity classification of the target patient based on the long-term spinal dependence and the feature vector sequence to construct a scoliosis report for the target patient.

2. The deep learning-based scoliosis identification method as described in claim 1, characterized in that, The step of converting the high-precision point cloud data into a three-dimensional mesh model includes: identifying outliers in the high-precision point cloud data; filtering the high-precision point cloud data based on the outliers to obtain filtered high-precision point cloud data; registering the filtered high-precision point cloud data to obtain registered high-precision point cloud data; calculating the surface normal vectors of the registered high-precision point cloud data to perform surface reconstruction on the registered high-precision point cloud data to obtain a triangular mesh; and performing post-mesh processing on the triangular mesh to obtain the three-dimensional mesh model.

3. The deep learning-based scoliosis identification method as described in claim 2, characterized in that, The calculation of the surface normal vector of the registered high-precision point cloud data includes: marking the target point of the registered high-precision point cloud data; analyzing the local curvature weight and spine topological weight of the target point; calculating the neighborhood centroid of the target point; and calculating the surface normal vector of the registered high-precision point cloud data using the following formula based on the local curvature weight, the spine topological weight, and the neighborhood centroid.

4. The deep learning-based scoliosis identification method as described in claim 3, characterized in that, The step of extracting the three-dimensional geometric features of the spine of the three-dimensional mesh model using a 3D sparse convolutional subnetwork in a preset dual-branch deep network includes: converting the three-dimensional mesh model into a sparse voxel mesh; extracting the sparse tensor of the sparse voxel mesh using sparse convolutional layers in the 3D sparse convolutional subnetwork; downsampling the sparse tensor using submanifold downsampling layers in the 3D sparse convolutional subnetwork to obtain a downsampled sparse tensor; transposing the downsampled sparse tensor using sparse transpose convolutional layers in the 3D sparse convolutional subnetwork to obtain a transposed sparse tensor; and combining the sparse tensor, the downsampled sparse tensor, and the transposed sparse tensor to extract the three-dimensional geometric features of the spine of the three-dimensional mesh model.

5. The deep learning-based scoliosis identification method as described in claim 4, characterized in that, The step of extracting the two-dimensional spinal structure features of the X-ray image using the U-Net subnetwork in the dual-branch deep network includes: normalizing the grayscale of the X-ray image to obtain a normalized X-ray image; extracting the deep feature map of the normalized X-ray image using the encoder in the U-Net subnetwork; and outputting the two-dimensional spinal structure features of the X-ray image using the decoder in the U-Net subnetwork.

6. The deep learning-based scoliosis identification method as described in claim 5, characterized in that, The step of generating a cross-modal joint feature map of the target patient by combining the three-dimensional geometric features and the two-dimensional structural features of the spine includes: spatializing the three-dimensional geometric features of the spine to obtain spatialized three-dimensional spinal features; reducing the dimensionality of the two-dimensional structural features of the spine to obtain dimensionality-reduced two-dimensional spinal features; calculating the cross-modal interaction matrix of the spatialized three-dimensional spinal features and the dimensionality-reduced two-dimensional spinal features; defining anatomical constraints on the spatialized three-dimensional spinal features and the dimensionality-reduced two-dimensional spinal features; and fusing the spatialized three-dimensional spinal features and the dimensionality-reduced two-dimensional spinal features based on the anatomical constraints and the cross-modal interaction matrix to obtain the cross-modal joint feature map of the target patient.

7. The deep learning-based scoliosis identification method as described in claim 6, characterized in that, The step of outputting the feature vector sequence of the cross-modal joint feature map based on the cascaded attention module includes: generating a max-pooling spatial feature map and an average-pooling spatial feature map of the cross-modal joint feature map; calculating the spatial weights of the cross-modal joint feature map using the spatial attention submodule in the cascaded attention module based on the max-pooling spatial feature map and the average-pooling spatial feature map; enhancing the cross-modal joint feature map based on the spatial weights to obtain an enhanced joint feature map; calculating the channel weights of the enhanced joint feature map using the channel attention submodule in the cascaded attention module; and outputting the feature vector sequence of the enhanced joint feature map based on the channel weights.

8. The deep learning-based scoliosis identification method as described in claim 7, characterized in that, The step of analyzing the long-term spinal dependence of the target patient based on the feature vector sequence includes: analyzing the biomechanical conduction path and modeling inverse mechanical feedback of the target patient; determining the bidirectional vertebral body hiding state of the target patient based on the biomechanical conduction path and modeling inverse mechanical feedback; establishing a spinal topology dependency graph of the target patient based on the bidirectional vertebral body hiding state; and analyzing the long-term spinal dependence of the target patient through the spinal topology dependency graph.

9. The deep learning-based scoliosis identification method as described in claim 8, characterized in that, The step of analyzing the Cobb angle, scoliosis type, and severity grading of the target patient based on the long-term spinal dependence relationship and the feature vector sequence includes: identifying the upper and lower scoliosis bodies of the target patient based on the feature vector sequence; calculating the normal vectors of the upper and lower scoliosis bodies; calculating the Cobb angle of the target patient based on the normal vectors of the upper and lower scoliosis bodies; analyzing the curvature pattern of the target patient based on the long-term spinal dependence relationship and the feature vector sequence to determine the scoliosis type of the target patient; and analyzing the severity grading of the target patient based on the Cobb angle and the scoliosis type.

10. A scoliosis recognition system based on deep learning, characterized in that, The system includes: a spinal data processing module for acquiring spinal data of a target patient, wherein the spinal data includes high-precision point cloud data and X-ray images of the patient's spinal surface, and converting the high-precision point cloud data into a three-dimensional mesh model; a spinal feature extraction module for extracting three-dimensional geometric features of the spine from the three-dimensional mesh model using a 3D sparse convolutional sub-network in a preset dual-branch deep network, and extracting two-dimensional structural features of the spine from the X-ray images using a U-Net sub-network in the dual-branch deep network; and a spinal feature fusion module for combining the three-dimensional geometric features and the two-dimensional structural features of the spine to generate a model of the target patient's spinal features. A cross-modal joint feature map is constructed, and a cascaded attention module is established for the cross-modal joint feature map, wherein the cascaded attention module includes a spatial attention submodule and a channel attention submodule; a spinal dependency analysis module is used to output the feature vector sequence of the cross-modal joint feature map based on the cascaded attention module, and to analyze the long-term spinal dependency of the target patient based on the feature vector sequence; a scoliosis report construction module is used to analyze the Cobb angle value, scoliosis type and severity grade of the target patient based on the long-term spinal dependency and the feature vector sequence, so as to construct a scoliosis report for the target patient.

Images