Spine three-dimensional evaluation and intelligent grading system based on multi-view learning

Abstract

The invention provides a spine three-dimensional evaluation and intelligent grading system based on multi-view learning, and relates to the technical field of intelligent evaluation of scoliosis, the spine three-dimensional evaluation and intelligent grading system comprises a multi-view image acquisition module for acquiring spine images of a patient through camera equipment, and the spine images of the patient comprise a back image, a side image and a back bending position image; the image preprocessing module is used for performing human body segmentation processing, cutting processing and pixel normalization processing on the spine image of the patient, and the human body segmentation processing uses a U-Net model to segment the spine image of the patient into a human body mask. The spine three-dimensional multi-view evaluation and intelligent grading system for multi-view learning is provided.

Description

Technical Field

[0001] This invention relates to the field of intelligent assessment technology for scoliosis, specifically a three-dimensional assessment and intelligent grading system for the spine based on multi-view learning. Background Technology

[0002] Scoliosis is a common spinal disorder, especially prevalent among adolescents. Traditional scoliosis assessment relies primarily on X-rays, CT scans, and physical examinations, but these methods have limitations. While imaging methods provide a wealth of data for assessment, they still depend on expert judgment, and the results are subject to uncertainty and error due to factors such as image quality, shooting angle, and physician experience. Currently, several medical imaging-based 3D modeling systems have been developed and applied to scoliosis assessment, such as ScolioStudio and technologies like structured light and RGB-D cameras. While these technologies improve assessment accuracy to some extent, most focus on single tasks, such as measuring the curvature angle or assessing posture, failing to perform multi-task, multi-label joint modeling.

[0003] Existing scoliosis assessment technologies fail to address the challenge of multi-task, multi-label joint modeling. Traditional assessment methods typically process only a single task, such as outputting only the curvature angle or posture assessment of different spinal segments, and cannot simultaneously make a comprehensive judgment on multiple assessment indicators. This makes existing technologies unable to effectively meet the multi-dimensional information needs of clinical practice when dealing with scoliosis. Especially when dealing with adolescent patients, existing technologies cannot simultaneously assess multiple aspects of information such as the severity of scoliosis, Schroth type, and direction of curvature, resulting in a lack of comprehensiveness and personalization in the final report. Furthermore, the report output of existing assessment systems often only includes partial professional data, which is difficult for ordinary patients and their families to understand, thus affecting triage decisions and subsequent rehabilitation. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a three-dimensional assessment and intelligent grading system for the spine based on multi-view learning. The technical problem this invention aims to solve is: how to achieve accurate, multi-dimensional intelligent assessment of scoliosis and automatic generation of personalized rehabilitation grading plans through three-dimensional assessment technology based on multi-view learning.

[0005] To achieve the above objectives, the present invention provides the following technical solution: A three-dimensional spine assessment and intelligent grading system based on multi-view learning, comprising: The multi-view image acquisition module acquires images of the patient's spine using a camera device. These images include back images, side images, and images of the back in a prone position. The image preprocessing module is used to perform human body segmentation, cropping and pixel normalization on the patient's spinal image. The human body segmentation uses the U-Net model to segment the patient's spinal image into a human body mask. The cropping process crops the human body mask to generate a human body region image. The feature extraction module extracts a 1024-dimensional feature vector from the human body region image through an autoencoder architecture. The feature extraction module combines the 1024-dimensional feature vectors into a shared feature matrix, which has 3072 dimensions. The multi-label evaluation module transforms the shared feature matrix into graph node feature representations based on a graph neural network model. The multi-label evaluation module outputs evaluation results through a multi-task learning framework. The evaluation results include the severity of scoliosis, Schroth type, and direction of curvature.

[0006] The graph neural network model processes the shared feature matrix through a graph convolutional network. The processing formula of the graph convolutional network is as follows: ; in, The first part of the graph represents the second part. Layer node feature matrix, Let be the adjacency matrix of the graph. Let the node degree matrix be... For the first The trainable weights of the layer It is the ReLU activation function; The autoencoder architecture includes an encoder and a decoder. The autoencoder architecture processes the back image, side image, and back-facing image in a prone position. The specific processing method is as follows: The encoder extracts the shared feature matrix using the ReLU activation function, the formula for which the ReLU activation function is calculated is: ; in, This indicates the input extracted from an image taken from the back, side, or prone position. This represents the convolution operation. This represents the feature matrix extracted from images from three perspectives; The decoder recovers the structural features of the patient's spinal image through deconvolution layers and a reconstruction loss function. The training model for the reconstruction loss function is as follows: ; in, For the original image, For the image reconstructed by the decoder, This represents the L2 norm.

[0007] The multi-task learning framework employs an uncertainty-weighted loss model, the calculation formula of which is: ; in, For the total loss, For mission losses, Indicates task The learnable uncertainty parameter.

[0008] The image preprocessing module further includes image data enhancement functions, which include horizontal flipping, ±10-degree rotation, ±5% translation, brightness perturbation, and contrast perturbation.

[0009] The camera device is an RGB camera, and the 3072-dimensional feature matrix is ​​obtained by concatenating three 1024-dimensional vectors.

[0010] The human body region image is scaled to a fixed size of 256 pixels × 256 pixels using a bicubic interpolation algorithm and then normalized.

[0011] This invention provides a three-dimensional assessment and intelligent grading system for the spine based on multi-view learning. It has the following beneficial effects: This invention employs a multi-view learning-based three-dimensional spinal assessment and intelligent grading system. It can acquire images of the patient's spine from different angles and perform efficient feature extraction and assessment using a deep learning model, overcoming the limitation of traditional assessment methods that focus only on a single task. This system can not only accurately measure the curvature angle of each spinal segment but also assess multiple dimensions of information, including the severity of scoliosis, Schroth type, and direction of curvature, providing comprehensive and accurate assessment results. Attached Figure Description

[0012] Figure 1 This is a schematic diagram of the architecture of a three-dimensional spine assessment system based on multi-view learning; Figure 2 This is a schematic diagram illustrating the working principle of the spinal image acquisition module. Detailed Implementation

[0013] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Example 1

[0014] like Figure 1-2As shown, this embodiment of the invention provides a three-dimensional assessment and intelligent grading system for the spine based on multi-view learning, including a multi-view image acquisition module that acquires images of the patient's spine through a camera device. The patient's spine images include back images, side images, and back images in a prone position. The camera device is an RGB camera, and the 3072-dimensional feature matrix is ​​obtained by concatenating three 1024-dimensional vectors.

[0015] The image preprocessing module performs human segmentation, cropping, and pixel normalization on patient spine images. Human segmentation uses the U-Net model to segment the patient spine image into a human mask. Cropping generates human region images by cropping the human mask. The human region images are scaled to a fixed size of 256 pixels × 256 pixels using bicubic interpolation and then normalized. Image cropping and normalization involve applying a trained human segmentation model to the acquired original image to segment the human mask, and automatically cropping based on the mask to retain the foreground image region. The human region is set to 1, and the background to 0. ; Converted to a binary mask after thresholding: ; Let the coordinates of all foreground pixels form a set: ; The bounding box coordinates are obtained as follows: ; ; The final foreground cropping region in R is: ; Use the bounding box described above to crop the image: ; The cropped foreground image contains only the main human body area, removing redundant background information. The cropped image is then proportionally scaled to a fixed size of 256×256 pixels.

[0016] ; The color space of the image is unified to adapt to the input data format requirements of the subsequent network structure.

[0017] The image preprocessing module further includes image data augmentation functions, such as horizontal flipping, ±10-degree rotation, ±5% translation, brightness perturbation, and contrast perturbation. To improve the model's robustness and generalization ability, various data augmentation techniques are applied to the cropped image data during the training phase, including: Random horizontal flip: ; Random rotation: ; The center of the image.

[0018] Simultaneously adjust lighting changes: brightness perturbation Contrast perturbation saturation perturbation Color disturbance : ; The affine transformation involves a slight translation of ±5%. Translation vector: , ; Scaling factor: ; The affine matrix is ​​represented as: ; Random cropping: The starting point for random cropping is: ; The feature extraction module extracts 1024-dimensional feature vectors from the human body region image using an autoencoder architecture. These 1024-dimensional feature vectors are then combined into a shared feature matrix, which has 3072 dimensions. The autoencoder architecture includes an encoder and a decoder. It processes back images, side images, and back images in a prone position. The specific processing method is as follows: The encoder extracts the shared feature matrix using the ReLU activation function. The formula for calculating the ReLU activation function is: ; in, This indicates the input extracted from an image taken from the back, side, or prone position. This represents the convolution operation. This represents the feature matrix extracted from images from three perspectives.

[0019] The decoder recovers the structural features of the patient's spine image through deconvolution layers and a reconstruction loss function. The training model for the reconstruction loss function is as follows: ; in, For the original image, For the image reconstructed by the decoder, This represents the L2 norm.

[0020] Input image for each view Each is fed into its corresponding encoder. and decoder The encoder uses stacked convolutional layers, batch normalization, ReLU activation, and max pooling to construct a feature extraction network that maps images to a low-dimensional latent space. ; The decoder uses a symmetrically designed deconvolutional network or upsampling network to attempt to recover the original image from the encoded features, preserving the semantic information of the original image. ; To ensure that the encoder captures the key semantic features of the image, a reconstruction loss MSE is introduced during training: ; This loss function drives the network to preserve key structural and texture features of the image, ensuring that the encoded representation... It can fully convey the meaning.

[0021] Encoding vectors of images from different perspectives By splicing and merging, a shared feature matrix is ​​formed. : ; Where N is the number of viewpoints and d is the dimension of the feature vector.

[0022] The multi-view fusion representation uses each row to represent a deep semantic meaning from a single perspective, and the matrix as a whole encompasses comprehensive features of spatial structure and external texture. Secondly, it supports graph neural network input. It can be directly used as a feature for graph structure nodes in subsequent graph convolution processing. It is adaptable to multi-task classifiers, and downstream networks can extract the feature subspace most sensitive to each task from this matrix.

[0023] The multi-label evaluation module, based on a graph neural network model, transforms the shared feature matrix into graph node feature representations. The module outputs evaluation results through a multi-task learning framework, including the severity of scoliosis, Schroth type, and curvature direction. The graph neural network model processes the shared feature matrix through a graph convolutional network. The processing formula for the graph convolutional network is as follows: ; in, The first part of the graph represents the second part. Layer node feature matrix, Let be the adjacency matrix of the graph. Let the node degree matrix be... For the first The trainable weights of the layer The ReLU activation function is used. The shared high-order feature representation is obtained based on the graph neural network. Multiple parallel task-specific classification branches were designed to simultaneously complete several medical assessment tasks related to scoliosis. A multi-task learning framework was used to maintain the consistency of the overall model structure and improve performance across various tasks.

[0024] Multiple tags: Scoliosis severity grading: The categories are normal, mild, moderate and severe, and the range of Cobb angle can be predicted based on the severity.

[0025] Schroth type classification: The category labels are normal, 3C, and 4C. Effective Schroth type identification can guide the design of rehabilitation training exercises for patients in the later stages.

[0026] Lateral bending direction determination: The category labels are normal, left bend, and right bend. In the later rehabilitation training of patients, the movement design for different lateral bending directions is mirror symmetrical.

[0027] Whether rehabilitation training is possible: The category is labeled as yes or no. There are corresponding grading schemes for patients at different stages. For example, surgical grading or stent grading may be more beneficial for some severely ill patients.

[0028] When the high-order feature representation matrix of each node is obtained Then, a learnable importance weight is assigned to each node. To achieve weighted aggregation: ; in , which are learnable parameters. Indicates bias. For the first The normalized attention scores of each node are weighted and aggregated into image-level features, suitable for judging viewpoint importance. ; Suppose there are T tasks in total, each task It has its own independent classification head and weight parameters. The prediction output for each task is: ; in ,Depend on get; For the task The classifier weights; Indicates task The number of categories; each task independently outputs its corresponding predicted probability distribution. .

[0029] Each task defines its cross-entropy loss function as follows: ; The multi-task learning framework employs an uncertainty-weighted loss model, the calculation formula of which is as follows: ; in, For the total loss, For mission losses, Indicates task The learnable uncertainty parameter.

[0030] As a practical application of this invention, the assessment result generation module generates a scoliosis assessment report based on the assessment results. The scoliosis assessment report includes the assessment conclusions of the scoliosis, the angles of each spinal segment, the type of scoliosis, the Schroth type, the rehabilitation grading plan, and grading recommendations. It also provides image visualization results and PDF report export functionality. Based on the scoliosis assessment report, a rehabilitation training plan is provided, including a right-side muscle strengthening plan for patients with left scoliosis and a left-side muscle strengthening plan for patients with right scoliosis.

[0031] The report should include the following main parts: Scoliosis assessment conclusion: The results of multi-task learning are combined to generate assessment conclusions. For example: "The scoliosis is moderate, the Schroth type is 3C, and the direction of curvature is right curvature." Schross type and recommendations: Based on the Schroth type, corresponding grading recommendations are generated. For example: "Schroth type 3C, personalized rehabilitation training is recommended based on the patient's condition." Grading recommendations: Based on the assessment results, rehabilitation grading recommendations are generated. For example: "Since the scoliosis is moderate, regular physical grading and muscle strengthening training are recommended, especially for patients with right-sided curvature, the left-side muscle groups should be strengthened." Image visualization results: Based on image data, the patient's spine is visualized, and key curvature angles and Schroth types are marked. The visualized images of the patient's spine are included in the report to help doctors and patients intuitively understand the actual condition of the spine.

[0032] Based on the generated evaluation results, a standardized PDF report is produced using a template rendering system.

[0033] For example, the report will include the following parts: Patient basic information: name, gender, age, examination date, etc.

[0034] Assessment conclusions: Scoliosis severity, Schroth type, and direction of curvature based on multi-task learning.

[0035] Tiered recommendations: Based on the assessment results, rehabilitation training plans and tiered recommendations are provided.

[0036] Image visualization: Annotations and diagrams of curvature angles of a spinal image.

[0037] The report can be provided to patients or doctors through a user interface. Patients can view assessment results and rehabilitation suggestions on mobile devices or computers, and download or print the PDF report. Doctors can then use the report content, combined with the patient's personalized triage plan, to conduct subsequent triage.

[0038] Example 2 This embodiment is based on a multi-label evaluation module. By designing a graph neural network model, an adversarial network framework is generated to enhance the performance of the graph neural network in scoliosis evaluation.

[0039] 1. Generate an adversarial network framework A generative adversarial network (GAN) framework is introduced to optimize graph structure learning through a generator and a discriminator, enhancing the robustness of graph neural networks in anomaly detection and graph structure generation. The generator attempts to generate a pseudo-graph structure, while the discriminator determines its authenticity. The training processes of the generator and discriminator are adversarial: the generator continuously improves to produce increasingly "realistic" pseudo-graph structures, while the discriminator continuously enhances its judgment ability to distinguish between real and pseudo-graph structures.

[0040] 2. Pseudograph structure generation Based on the shared feature matrix generated in the preceding steps, a generative adversarial network is used to generate a pseudo-graph structure. Each node in the graph structure represents image features from a viewpoint, while the edges of the graph represent the similarity or correlation between different viewpoints.

[0041] The structural modeling module before the graph neural network generates a similarity matrix to produce a realistic graph structure. : ; in, Used for graph convolution and adversarial training. This is the shared feature matrix obtained during the feature extraction process.

[0042] generator Received feature matrix Output generated graph structure The feature mapping process is as follows: ; The features of the mapping are calculated using similarity: ; Finally, normalization yields the pseudograph structure. : ; 3. Discriminator Training The discriminator's task is to distinguish between real and pseudo-graph structures. Real graph structures are calculated by the graph neural network based on actual image features, while pseudo-graph structures are generated by the generator.

[0043] The input to the discriminator is a graph structure A, which may be a real graph structure or a pseudo-graph structure output by the generator. The discriminator classifies graph structures using a neural network, and the output probability value indicates whether the graph structure is a true graph structure.

[0044] Discriminator Receiver diagram structure Output the predicted probability: ; Output values ​​close to 1 indicate "real image" and values ​​close to 0 indicate "fake image".

[0045] 4. Adversarial training between generator and discriminator The generator and discriminator are optimized through adversarial training. The generator aims to maximize the discriminator's error rate, even if it misclassifies a fake image as a real one. The discriminator, on the other hand, aims to correctly determine the authenticity of the graph structure, maximizing the probability of classifying it as a real image and minimizing the probability of classifying it as a fake image.

[0046] In generator and self-supervised adversarial optimization, the total loss function includes adversarial loss and reconstruction consistency loss: ; The discriminator loss above is used to distinguish the true graph structure. and pseudograph structure The generator loss is used to deceive the discriminator: ; In the reconstruction consistency loss, the Frobenius norm is used to measure the consistency loss. and Structural differences: ; The optimization process involves minimizing the total generator loss: ; in As a hyperparameter, it balances generation consistency and adversarial behavior. This module uses a generator. Discriminator An adversarial mechanism between graph learning modules is introduced to optimize the graph structure learning process, thereby improving the feature representation ability and robustness to abnormal samples of the generated graph structure B1.

[0047] 5. Optimization of Pseudograph Structures and Training of Graph Neural Networks The generator and discriminator are gradually optimized through adversarial training. The generator can produce increasingly realistic pseudo-graph structures, while the discriminator can better distinguish between real and pseudo-graphs. During training, the pseudo-graph structures are continuously optimized and combined with the feature learning part of the graph neural network. Finally, the generated pseudo-graph structures are input into the graph neural network to perform graph convolution operations, extract high-order features of each node, and update the node feature representations.

[0048] 6. Graph Neural Networks and Multi-Task Learning The generated pseudo-graph structure is combined with other parts of the graph neural network to complete high-order feature extraction. The graph neural network performs contextual aggregation on the features of nodes in the graph through convolution operations to obtain high-order semantic representations. The graph neural network model processes the shared feature matrix through a graph convolutional network, and the processing formula of the graph convolutional network is as follows: ; in, The first part of the graph represents the second part. Layer node feature matrix, Let be the adjacency matrix of the graph. Let the node degree matrix be... For the first The trainable weights of the layer This is the ReLU activation function.

[0049] Normalized adjacency matrix: This calculates the normalized adjacency matrix of the graph, which helps to avoid the influence of node degree on information propagation when graph convolution propagates information between nodes of different degrees. This step helps to control the influence between different nodes.

[0050] Feature propagation: The first part of the graph represents the second part. The feature matrix of each layer node is propagated through a weighted adjacency matrix, and each node updates its features based on its relationship with its neighbors.

[0051] Weighted transformation: For the first The trainable weights of the layer linearly transform the feature vector of each node, and the learned feature weights reflect the importance of different features in the learning process.

[0052] Nonlinear activation: σ is an activation function that introduces nonlinear changes, enhancing the network's expressive power and enabling the model to learn more complex features.

[0053] After graph convolution operations, the output of the last graph convolutional network layer is... ,in This is the updated higher-order semantic dimension; It includes information on the contextual structure between images from different perspectives, preserving the interaction patterns between global and local data; The data is fed into the subsequent multi-task classification branch to improve the model's accuracy in judging multi-label tasks that are sensitive to structural information.

[0054] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A spinal three-dimensional evaluation and intelligent grading system based on multi-view learning, characterized in that, The method comprises the steps of: a multi-view image acquisition module acquires patient spine images through a camera device, wherein the patient spine images include back images, side images and back leaning position images; an image preprocessing module is configured to perform human body segmentation processing, cropping processing and pixel normalization processing on the patient spine images, wherein the human body segmentation processing uses a U-Net model to segment the patient spine images into a human body mask, and the cropping processing crops the human body mask to generate a human body region image; a feature extraction module extracts a 1024-dimensional feature vector of the human body region image through a self-encoder architecture, wherein the feature extraction module combines the 1024-dimensional feature vector into a shared feature matrix, and the shared feature matrix is 3072-dimensional; a multi-label evaluation module converts the shared feature matrix into a graph node feature representation based on a graph neural network model, wherein the multi-label evaluation module outputs an evaluation result through a multi-task learning framework, and the evaluation result includes the severity of scoliosis, the Scoliosis Research Society (SRS) type and the direction of scoliosis.

2. The multi-view learning based spine 3D assessment and intelligent grading system according to claim 1, wherein: The graph neural network model processes the shared feature matrix through a graph convolution network, and the processing formula of the graph convolution network is: ； wherein, represents the first layer node feature matrix, is an adjacency matrix of the graph, is a node degree matrix, is the trainable weight of the first layer, is a ReLU activation function; The self-encoder architecture includes an encoder and a decoder, and the self-encoder architecture processes the back images, the side images and the back leaning position images in the following manner: The encoder extracts the shared feature matrix through a ReLU activation function, and the calculation formula of the ReLU activation function is: ； wherein, represents an input extracted from a back, side, or back leaning position image, represents a convolution operation, represents a feature matrix extracted from three view images; The decoder recovers the structural features of the patient spine images through a deconvolution layer and a reconstruction loss function, and the training model of the reconstruction loss function is: ； wherein, is the original image, is the image reconstructed by the decoder, denotes the L2 norm.

3. The multi-view learning based spine 3D assessment and intelligent grading system of claim 1, wherein: The multi-task learning framework adopts an uncertainty weighted loss model, and the calculation formula of the uncertainty weighted loss model is: ； wherein, is the total loss, is the task loss, denotes the task learnable uncertainty parameter.

4. The multi-view learning based spine 3D assessment and intelligent grading system of claim 1, wherein: The image preprocessing module further includes an image data enhancement function, and the image data enhancement function includes horizontal flipping, ±10 degree rotation, ±5% translation, brightness disturbance and contrast disturbance.

5. The multi-view learning based spine 3D assessment and intelligent grading system of claim 1, wherein: The camera device is an RGB camera, and the 3072-dimensional feature matrix is obtained by splicing three 1024-dimensional vectors.

6. The multi-view learning based spine 3D assessment and intelligent grading system of claim 1, wherein: The human body region image is scaled to a fixed size of 256 pixels x 256 pixels through a bicubic interpolation algorithm for pixel normalization processing.

Images