Training method, device, terminal, medium and analysis method of lightweight multi-mode spine model with vertebral segment perception

Abstract

The invention provides a training method and device of a lightweight multi-mode spine model with centrum segment perception, a terminal, a medium and an analysis method. According to the invention, low-cost localization deployment is realized by constructing the lightweight multi-mode architecture. Meanwhile, through specific data input and a progressive three-stage training strategy, a lightweight model with the spine specialized ability is constructed and trained, and then a more accurate spine image analysis result is obtained.

Description

Technical Field

[0001] This application relates to the field of multimodal large model technology, and in particular to a training method, device, terminal, medium and analysis method for a lightweight multimodal spinal model with vertebral segment perception. Background Technology

[0002] In recent years, with the rapid development of deep learning technology, general multimodal large language models (LVLMs) have achieved significant breakthroughs in the field of artificial intelligence. Closed-source large models such as GPT-4V and Gemini, as well as open-source medical-specific models such as LLaVA-Med and PMC-LLaMA, have demonstrated powerful image understanding and text generation capabilities. Currently, the conventional application mode of these models is to directly input medical images into the model and use its pre-trained general medical knowledge to perform zero-shot or few-shot inference.

[0003] While existing technologies perform reasonably well in general medical image description tasks, they have gradually revealed a series of insurmountable shortcomings when dealing with spinal surgery, which demands extremely high anatomical precision. These shortcomings are mainly reflected in the following three aspects:

[0004] First, general-purpose models generally lack the ability to sense vertebral segments, making it difficult to meet the precision requirements of surgical planning. Clinical decisions in spinal surgery heavily rely on the precise localization of specific anatomical segments. For example, the clinical symptoms and surgical approaches for a disc herniation in the L4-L5 segment are drastically different from those in the L5-S1 segment. However, existing general-purpose multimodal models lack this fine-grained anatomical prior knowledge. When faced with spinal images, they often only provide general qualitative conclusions such as the presence of a disc herniation or spinal stenosis, failing to pinpoint the specific vertebral level. This lack of localization results in diagnostic recommendations from the models lacking practical clinical guidance.

[0005] Secondly, existing models exhibit weak cross-modal inference capabilities when processing multimodal images. Diagnosing spinal diseases typically requires constructing a complete chain of imaging evidence: physicians need to examine X-rays to assess the overall spinal sequence (e.g., scoliosis angle), observe bone details via CT scans (e.g., osteophyte formation, fractures), and assess soft tissue condition and nerve compression using MRI. Existing model architectures often process single images in isolation, lacking mechanisms for establishing complementary and validation relationships between multiple modalities. Experiments show that existing models suffer from significant cross-modal alignment deficiencies when handling complex tasks requiring the integration of X-ray, CT, and MRI information, resulting in a significant performance degradation and difficulty in forming a complete diagnostic evidence chain.

[0006] Finally, ultra-large parameter models face the dual challenges of deployment difficulties and data privacy risks. Currently, the general-purpose large models performing well in the medical field (such as Gemini-Pro) typically have over 100 billion parameters (100B+), placing extremely high demands on computing resources (GPU memory, computing power), making low-cost local deployment difficult. If cloud API calls are used, sensitive patient image data must be uploaded to third-party servers, posing serious risks of medical data privacy breaches and compliance legal issues, significantly limiting their practical application in clinical settings. Summary of the Invention

[0007] In view of the shortcomings of the prior art described above, the purpose of this application is to provide a training method, device, terminal, medium and analysis method for a lightweight multimodal spinal model with vertebral segment perception, in order to solve at least one of the problems in the prior art.

[0008] To achieve the above and other related objectives, the first aspect of this application provides a training method for a lightweight multimodal spine model with vertebral segment awareness, comprising: constructing a lightweight multimodal architecture; performing three-stage training on the lightweight multimodal architecture to obtain a trained lightweight spine specialty multimodal large model with vertebral segment awareness; wherein the three-stage training includes: general medical and orthopedic basic training, spine specialty reinforcement and segment awareness logic injection training, and clinical workflow and long text alignment training.

[0009] In some embodiments of the first aspect of this application, the lightweight multimodal architecture includes: a visual encoder and a text encoder; wherein the visual encoder is connected to a multilayer perceptron; the output of the multilayer perceptron and the output of the text encoder are concatenated and used as input to a pre-trained lightweight large language model.

[0010] In some embodiments of the first aspect of this application, the parameter size of the lightweight large language model is 7B.

[0011] In some embodiments of the first aspect of this application, the medical and orthopedic basic training includes: acquiring a general medical multimodal dataset and a non-spinal pan-orthopedic dataset to construct a medical and orthopedic basic training set; using the medical and orthopedic basic training set and based on the DeepSpeed ​​Zero2 optimization strategy, training the lightweight multimodal architecture to obtain an initial training model.

[0012] In some embodiments of the first aspect of this application, the spinal specialty enhancement and segmental perception logic injection training includes: acquiring a spinal specialty instruction dataset; wherein, a portion of the spinal specialty instruction data in the spinal specialty instruction dataset includes long thought chains; the long thought chains include: anatomical localization, image analysis, neural connections, and diagnostic conclusions; using the spinal specialty instruction dataset and based on the DeepSpeed ​​Zero2 optimization strategy, training the initial training model to obtain an intermediate training model.

[0013] In some embodiments of the first aspect of this application, the clinical workflow and long text alignment training includes: acquiring multi-round consultation dialogue data and full-process medical report data to construct a clinical data training set; using the clinical data training set and based on the DeepSpeed ​​Zero3 Offloading optimization strategy, training the intermediate training model to obtain a trained lightweight multimodal large model of spine specialty with vertebral segment awareness.

[0014] To achieve the above and other related objectives, a second aspect of this application provides a training device for a lightweight multimodal spine model with vertebral segment awareness, comprising: a construction module for constructing a lightweight multimodal architecture; and a three-stage training module for performing three-stage training on the lightweight multimodal architecture to obtain a trained lightweight spine specialty multimodal large model with vertebral segment awareness; wherein the three-stage training includes: general medical and orthopedic basic training, spine specialty reinforcement and segment awareness logic injection training, and clinical workflow and long text alignment training.

[0015] To achieve the above and other related objectives, a third aspect of this application provides a spinal image analysis method, the method comprising: acquiring a spinal image to be analyzed and preprocessing it; inputting the spinal image into a lightweight multimodal spinal model with vertebral segment awareness to obtain corresponding analysis results; wherein the lightweight multimodal spinal model with vertebral segment awareness is trained by the training method for a lightweight multimodal spinal model with vertebral segment awareness as described above.

[0016] To achieve the above and other related objectives, a fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the training method for the lightweight multimodal spinal model with vertebral segment awareness or the spinal image analysis method.

[0017] To achieve the above and other related objectives, a fifth aspect of this application provides an electronic terminal, including a memory, a processor, and a computer program stored in the memory; the processor executes the computer program to implement the training method for the lightweight multimodal spinal model with vertebral segment perception or the spinal image analysis method.

[0018] As described above, the training method, device, terminal, medium, and analysis method of the lightweight multimodal spinal model with vertebral segment awareness of this application have the following beneficial effects: This application achieves low-cost localized deployment by constructing a lightweight multimodal architecture. Simultaneously, through specific data input and a progressive three-stage training strategy, a lightweight model with spinal specialty capabilities is constructed and trained, thereby obtaining more accurate spinal image analysis results. Attached Figure Description

[0019] Figure 1 The diagram shown is a schematic flowchart of a training method for a lightweight multimodal spinal model with vertebral segment awareness in one embodiment of this application.

[0020] Figure 2 The diagram shown is a schematic representation of a lightweight multimodal architecture in one embodiment of this application.

[0021] Figure 3 The diagram shown is a schematic representation of a three-stage training process in one embodiment of this application.

[0022] Figure 4 The diagram shown is a schematic block diagram of a training device for a lightweight multimodal spinal model with vertebral segment perception, as described in one embodiment of this application.

[0023] Figure 5 The diagram shown is a structural schematic of an electronic terminal according to an embodiment of this application. Detailed Implementation

[0024] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, unless otherwise specified, the following embodiments and features in the embodiments can be combined with each other.

[0025] In the embodiments of this application, the terms "first" and "second" are used to distinguish identical or similar items with essentially the same function and effect. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and that the terms "first" and "second" do not necessarily imply that they are different.

[0026] It should be noted that, in the embodiments of this application, the words "exemplary" or "for example" indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0027] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0028] To facilitate understanding of the embodiments of this application, firstly, in conjunction with Figure 1 Detailed explanation. Figure 1 This document illustrates a flowchart of a training method for a lightweight multimodal spinal model with vertebral segment awareness, according to an embodiment of the present invention. The training method for this lightweight multimodal spinal model with vertebral segment awareness mainly includes the following steps:

[0029] Step S101: Build a lightweight multimodal architecture.

[0030] In one embodiment, such as Figure 2 As shown, the lightweight multimodal architecture includes a visual encoder, a text encoder, a multilayer perceptron (MLP), and a lightweight large language model. The visual encoder is connected to the MLP, and the outputs of the visual encoder and the text encoder are concatenated as input to the lightweight large language model.

[0031] In one specific embodiment, the visual encoder employs a Vision Transformer (ViT) architecture, responsible for receiving high-resolution image data such as X-ray images, CT slices, and MRI sequences, and converting them into high-dimensional visual feature vectors. A Multilayer Perceptron (MLP) acts as a projection layer, serving as a bridge between vision and language, mapping visual features to a text embedding space aligned with a lightweight inference model. The lightweight large language model has a parameter scale of 7B (7 billion). This lightweight architecture, after quantization, can be adapted to consumer or enterprise-grade graphics cards with 24GB or 48GB of dedicated memory, enabling low-cost localized deployment. It is suitable for scenarios with various hardware limitations and limited computing power, such as private deployments within intranets. It should be understood that the quantization of the architecture is essentially an optimization process of compressing the numerical precision of core data such as weights and activation values ​​of artificial intelligence models (especially deep learning models). The aim is to significantly reduce the computational load, memory consumption, and inference latency of the model while sacrificing minimal model precision, allowing the model to be efficiently deployed on devices with limited computing power (such as mobile phones, embedded chips, and edge servers). In one specific embodiment, the lightweight multimodal architecture can be implemented using the Qwen2.5-VL-7B-Instruct model.

[0032] Step S102: Perform three-stage training on the lightweight multimodal architecture to obtain a trained lightweight spinal specialty multimodal large model with vertebral segment awareness; wherein, the three-stage training includes: general medical and orthopedic basic training, spinal specialty enhancement and segment awareness logic injection training, and clinical workflow and long text alignment training.

[0033] The core of this invention lies in gradually injecting and reshaping the parameter distribution of a lightweight multimodal architecture through three-stage training, thereby progressively infusing specialized capabilities. Specifically, for example... Figure 3 As shown,

[0034] The first phase of training focuses on general medical and orthopedic foundational training. To eliminate the domain gap between natural images and medical images, this phase utilizes a general medical multimodal dataset (e.g., PubMedVision, approximately 150,000 entries) and non-spinal pan-orthopedic data (covering trauma, joint, hand surgery, and other medical data, approximately 230,000 entries) to construct a basic medical and orthopedic training set. Using this training set, a lightweight multimodal architecture is trained according to a pre-set learning rate and maximum sequence length for full parameter fine-tuning. The first phase of training aims to adapt the visual encoder to the unique grayscale distribution and texture features of medical images, establishing basic anatomical concepts such as bones, soft tissues, and joint spaces. During training, the DeepSpeed ​​Zero2 optimization strategy is employed to ensure training stability while maintaining the ability to "understand" medical images. It should be understood that the DeepSpeed ​​Zero2 optimization strategy is DeepSpeed's memory optimization strategy for distributed training of large models. The core logic is: based on data parallelism, the optimizer state (such as Adam's m / v value) and gradients are split into multiple parts and stored on different GPUs (no longer storing a complete copy on each GPU); model parameters are still fully preserved on each GPU, avoiding excessive communication overhead; ultimately, this can reduce single-GPU memory usage by about 50%, with a small decrease in training speed (usually < 10%), representing a compromise between memory optimization and training efficiency. The learning rate and maximum sequence length can be set according to requirements and are not limited here. For example, the learning rate can be set to 1e-5, and the maximum sequence length to 16,384 tokens.

[0035] The second stage of training involves strengthening the spinal specialty and injecting segmental perception logic. After the model acquires basic medical vision capabilities, all non-spinal data is stripped away, and only a spinal specialty instruction dataset is input. This dataset primarily includes spinal image data and corresponding question-and-answer instructions. High-quality spinal specialty instruction data from the SpineMed-450k dataset can be used. Long chain-of-thought (CoT) is established in a portion of the spinal specialty instruction data within the dataset. Unlike the traditional simple image-diagnosis mapping, the spinal specialty instruction data constructed in this invention follows a logical path of first localization, then analysis, and finally diagnosis. A long chain-of-thought (CoT) is an extended reasoning method that includes multi-step continuous deduction and possesses deep reasoning, reflective correction, and multi-path exploration capabilities. The long chain-of-thought includes: anatomical localization, image analysis, neural connections, and diagnostic conclusions. For example, a long thought chain can be set as follows: <Anatomy Location>: Identify the L1-L5 vertebral body sequence...Locate the L4-L5 intervertebral space... <Imaging Analysis>: Observe decreased signal in the L4-L5 intervertebral disc, with a posterior protrusion of approximately 5mm... <Neural Correlation>: Compression of the right L5 nerve root... <Diagnosis Conclusion>: Lumbar disc herniation (L4-L5). The initial training model is trained using a spinal specialist instruction dataset to obtain an intermediate training model. Similarly, the DeepSpeed ​​Zero2 optimization strategy is used during training. This invention, through this mandatory text generation training mode, successfully distills the anatomical location thinking of a human physician into the model parameters, thereby achieving segmental perception capabilities.

[0036] The third stage of training involves aligning the clinical workflow with long text. To adapt to real-world clinical interaction scenarios, this stage acquires multi-round consultation dialogue data (simulating doctor-patient communication) and full-process medical report data (including six dimensions: image analysis, diagnosis, planning, and risk), constructing a clinical data training set. Using this clinical data training set, the intermediate training model is trained to obtain a lightweight, multimodal spinal specialist model with vertebral segment awareness. Since the medical report data is extremely long text, which can easily lead to GPU memory overflow, a DeepSpeed ​​Zero3 Offloading optimization strategy is adopted to offload some optimizer states to CPU memory, thereby supporting ultra-long context windows of up to 49,152 tokens. Simultaneously, the learning rate is reduced for fine-tuning to ensure the model's output is professional in tone and format, and can comprehensively process complementary information from multiple image sequences (such as simultaneous input of X-rays and MRIs). The learning rate for the third stage of training can also be set according to needs and is not limited here. For example, the learning rate for the third stage of training can be set to 1e-6.

[0037] It should be understood that ZeRO-3 (Zero Redundancy Optimizer Stage 3) is DeepSpeed's core optimization technology. It distributes the model's parameters, gradients, and optimizer states across different GPUs, with each GPU storing only 1 / N (where N is the number of GPUs) of the model data, completely eliminating data redundancy. This is the most memory-optimized stage in the ZeRO series. Offloading: When GPU memory is still insufficient, some data (parameters / gradients / optimizer states) is offloaded from the GPU to CPU memory or even SSD to gain greater model training capacity. The ZeRO-3 offloading optimization strategy is mainly divided into CPU Offloading and SSD Offloading.

[0038] It should be noted that the datasets used in all three training phases include both image and text data.

[0039] To better illustrate the training method of the lightweight multimodal spinal model with vertebral segment awareness of the present invention, a specific embodiment is provided.

[0040] Example 1: A training method for a lightweight multimodal spinal model with vertebral segment awareness.

[0041] Hardware environment: The training method is executed on a high-performance computing node equipped with eight NVIDIA A100 GPUs. This configuration represents the standard computing power level that hospitals or research institutions can afford.

[0042] Qwen2.5-VL-7B-Instruct was selected as the general vision-language model base. The model contains a Vision Transformer (ViT) encoder for extracting visual features and a large language model decoder with 7 billion parameters, which are connected through a multilayer perceptron.

[0043] Phase One Training: General Medicine and Orthopedics Basic Warm-up

[0044] Data preparation: Select the publicly available PubMedVision dataset (approximately 150,000 records) and non-spinal orthopedic data (including trauma, joint, hand surgery, etc., approximately 230,000 records).

[0045] Training strategy: The DeepSpeed ​​Zero2 memory optimization strategy is adopted, and the global batch size is set to maximize GPU utilization. The learning rate is set to 1e-5, and the maximum sequence length is limited to 16,384 tokens. Training lasts for 1 epoch.

[0046] Implementation results: Through this stage, the model learned to recognize basic medical anatomical structures (such as bones and joint spaces) and medical terms, eliminating the feature distribution differences between natural images and medical images.

[0047] The second stage of training: Specialized Spinal Learning, is the key to giving the model "expert thinking".

[0048] Data Preparation: All non-spine data were stripped away, and only high-quality spine specialty instruction data (including the Spine-choice and Spine-open subsets) from the SpineMed-450k dataset were input. Mandatory long thought chains (CoTs) were constructed from the high-quality spine specialty instruction data. For example, for a lumbar MRI, the training objective was not merely to output "disc herniation," but rather to output "Location: L4-L5 segment; Feature: T2-weighted signal reduction; Conclusion: disc herniation."

[0049] Training strategy: Maintain the DeepSpeed ​​Zero2 strategy and a learning rate of 1e-5. Through this reinforcement training, the model establishes an anatomical coordinate system for the spine at the parameter level, enabling it to accurately distinguish adjacent vertebrae such as L4 and L5.

[0050] Phase 3 Training: Workflow Alignment with Clinical Workflows. This phase aims to standardize the model's output format to adapt it to real-world clinical workflows.

[0051] Data preparation: Input multi-turn consultation dialogue data (Spine-chat) and full-process medical report data (Spine-report) containing six dimensions. Considering the characteristics of medical report generation tasks involving extremely long texts (such as those containing multi-sequence image descriptions and detailed surgical plans), the optimization strategy was switched to DeepSpeed ​​Zero3 Offloading. This strategy offloads the optimizer state to CPU memory, thereby supporting ultra-long context windows of up to 49,152 tokens and resolving the GPU memory overflow issue.

[0052] Fine-tuning strategy: Reduce the learning rate to 1e-6 for fine-tuning to ensure that the model output has a professional tone, rigorous structure, and can accurately handle the comprehensive analysis of multimodal data.

[0053] Figure 4 This is a schematic block diagram of a training device for a lightweight multimodal spinal model with vertebral segment sensing provided in an embodiment of this application. Figure 4 As shown, the training device 400 for the lightweight multimodal spinal model with vertebral segment awareness includes:

[0054] Module 401 is used to build lightweight multimodal architectures;

[0055] The three-stage training module 402 is used to perform three-stage training on the lightweight multimodal architecture to obtain a trained lightweight spinal specialty multimodal large model with vertebral segment awareness; wherein, the three-stage training includes: general medical and orthopedic basic training, spinal specialty reinforcement and segment awareness logic injection training, and clinical workflow and long text alignment training.

[0056] It should be understood that the specific process of each module performing the above-mentioned steps has been described in detail in the above method embodiments, and will not be repeated here for the sake of brevity.

[0057] It should also be understood that the module division in the embodiments of this application is illustrative and only represents a logical functional division; in actual implementation, there may be other division methods. Furthermore, the functional modules in the various embodiments of this application can be integrated into a single processor, exist as separate physical entities, or be integrated into a single module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0058] The present invention also provides a method for spinal image analysis, the method comprising:

[0059] Acquire the spinal images to be analyzed and perform preprocessing;

[0060] The spinal image is input into a lightweight multimodal spinal model with vertebral segment awareness to obtain the corresponding analysis results; wherein, the lightweight multimodal spinal model with vertebral segment awareness is trained by the training method described above.

[0061] Specifically, the visual encoder extracts image features, and the multilayer perceptron converts them into a vector sequence that the language model can understand. According to the training logic of the second stage, the model first generates anatomical localization information internally (such as confirming the location of T12 and L1), then retrieves lesion features based on the localization information, and finally generates analysis results.

[0062] In one embodiment, preprocessing involves converting the spinal image to be analyzed into an image format that the model can process.

[0063] In one embodiment, the spinal images to be analyzed are one or more sets of spinal images.

[0064] In one embodiment, the analysis results are presented as a structured text stream, including: image measurements, analysis conclusions, and treatment recommendations.

[0065] In one embodiment, the spinal image analysis method further includes: parsing the analysis results into a standard PDF report.

[0066] To better illustrate the spinal image analysis method, a specific embodiment is provided below.

[0067] Example 2: An auxiliary diagnosis and report generation method for adolescent idiopathic scoliosis (AIS).

[0068] This embodiment demonstrates how the trained model can assist doctors in completing the entire process from image analysis to report generation in a real clinical setting.

[0069] 1. Input Phase: Clinicians upload a set of full-spine X-ray images (anteroposterior and lateral views) of the patient via a local hospital terminal. The image data (DICOM format) is converted into an image format that the model can process, accompanied by a standard prompt: "Please analyze this patient's images and generate a diagnostic report."

[0070] 2. Model Inference Stage: The model first identifies the image as a full spine image and automatically detects key anatomical landmarks (such as the T1, L1, and S1 vertebrae). Next, the model calculates the Cobb angle (scoliosis angle) to be 45 degrees and accurately locates the main curve in the thoracic segment, with the apex vertebra being T8. Then, combining its built-in clinical guideline knowledge graph, the model determines that the Cobb angle exceeds the surgical indication (usually > 40 degrees). Based on the diagnostic results, the model generates a personalized treatment recommendation: "Posterior spinal fusion is recommended to correct the deformity, with fixation covering T4-L1." Finally, the model's analysis results are parsed and populated into the hospital's standard report template, ultimately outputting a structured PDF report containing the following:

[0071] Imaging findings: "Full spine anteroposterior radiograph shows rightward curvature of the thoracic spine, Cobb angle 45 degrees..."

[0072] AI-assisted diagnosis: "Adolescent idiopathic scoliosis (AIS), Lenke type 1".

[0073] Treatment recommendations: Includes a patient version (explaining the necessity of surgery in layman's terms) and a doctor version (specific surgical plan).

[0074] Risk assessment: "Low risk of nerve damage, good postoperative recovery is expected."

[0075] Disclaimer: This report is marked "This report was generated with AI assistance and is for reference only".

[0076] As can be seen from the above embodiments, the present invention can achieve high-precision, full-process auxiliary diagnosis of spinal diseases locally in hospitals without relying on ultra-large cloud models.

[0077] The advantages of this invention are:

[0078] 1. This invention overcomes the technical bottleneck of ambiguous anatomical localization in spinal surgery using general-purpose models, achieving precise segmental identification with surgical guidance significance. While existing general-purpose multimodal models possess basic visual recognition capabilities, they often lack spatial geometric perception, only providing general qualitative conclusions such as disc herniation. This invention, through a second phase of specialized intensive training, successfully incorporates an anatomical coordinate system into the model parameters. Experimental data shows that the system can accurately identify and distinguish subtle pathological differences between adjacent segments such as L4-L5 and L5-S1, moving beyond simple image interpretation to possessing anatomical localization thinking similar to that of a specialist physician. This acquisition of level-awareness allows the model's diagnostic suggestions to directly connect with surgical planning (such as determining the surgical segment), significantly improving the clinical practical value of assisted diagnosis and filling the gap in refined diagnosis and treatment using general-purpose models.

[0079] 2. This invention achieves high-performance local deployment with a lightweight architecture using 7B parameters, solving the privacy compliance challenges of medical data. Addressing the privacy risks associated with the excessive computational demands of large-scale general-purpose models (hundreds of billions of parameters) and their reliance on cloud deployment, this invention leverages the streamlined model architecture and efficient memory optimization strategies of its lightweight multimodal architecture. This system can run smoothly on a single enterprise-grade graphics card (such as the A100) or even a high-end consumer graphics card (such as the RTX 4090), supporting fully private local deployment. This not only significantly reduces hardware procurement and maintenance costs but also fundamentally avoids the legal and compliance risks of leaking sensitive patient image data.

[0080] 3. This invention significantly improves the model's performance in complex cross-modal reasoning tasks through thought chain injection technology. Spinal disease diagnosis often requires integrating sequential information from X-rays and soft tissue information from MRI. Existing models often struggle to handle this multimodal complementary information. This invention constructs a mandatory localization-analysis-diagnosis long thought chain data, forcing the model to learn to synthesize multi-source information and perform logical deductions like a human doctor. Test results show that the lightweight multimodal spinal model with vertebral segment awareness (SpineGPT) of this invention achieves an average accuracy of 87.44% in spinal diagnosis tasks. This performance significantly outperforms general-purpose open-source models with large parameter sets (such as Qwen-72B), greatly improving the model's parameter efficiency.

[0081] 4. This invention achieves a leap from fragmented question-and-answer to fully structured reporting, seamlessly integrating into clinical workflows. Traditional medical AI often can only answer single questions or generate non-standardized text, making it difficult to directly use for clinical documentation. This invention, through a third-stage clinical alignment training, particularly deep optimization for long text generation, enables the model to generate standardized reports encompassing six dimensions: image measurements, diagnostic conclusions, treatment recommendations, and risk warnings. This structured output can be directly exported to a format compatible with electronic medical record (EMR) systems, significantly reducing repetitive work for doctors in writing medical records and truly enabling AI to improve efficiency throughout the entire clinical diagnosis and treatment process.

[0082] Figure 5 This is a schematic block diagram of the electronic terminal provided in an embodiment of this application. Figure 5 As shown, the electronic terminal 500 includes at least one processor 501, a memory 502, at least one network interface 503, and a user interface 505. The various components in the device are coupled together via a bus system 504. It is understood that the bus system 504 is used to implement communication between these components. In addition to a data bus, the bus system 504 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in… Figure 5 The general will label all buses as bus systems.

[0083] The user interface 505 may include a monitor, keyboard, mouse, trackball, clicker, button, touchpad, or touch screen.

[0084] It is understood that memory 502 can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM) or programmable read-only memory (PROM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM) and synchronous static random access memory (SSRAM). The memories described in the embodiments of this invention are intended to include, but are not limited to, these and any other suitable categories of memory.

[0085] In this embodiment of the invention, the memory 502 is used to store various types of data to support the operation of the electronic terminal 500. Examples of this data include: any executable program for operation on the electronic terminal 500, such as the operating system 5021 and application programs 5022; the operating system 5021 contains various system programs, such as the framework layer, core library layer, driver layer, etc., for implementing various basic services and handling hardware-based tasks. The application program 5022 may contain various applications, such as a media player, browser, etc., for implementing various application services. The training method or spinal image analysis method for the lightweight multimodal spinal model with vertebral segment awareness provided in this embodiment of the invention can be included in the application program 5022.

[0086] The methods disclosed in the above embodiments of the present invention can be applied to processor 501, or implemented by processor 501. Processor 501 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in processor 501 or by instructions in the form of software. The processor 501 may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 501 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. General-purpose processor 501 may be a microprocessor or any conventional processor, etc. The steps of the accessory optimization method provided in the embodiments of the present invention can be directly reflected as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software module may be located in a storage medium, which is located in memory. The processor reads the information in the memory and combines it with its hardware to complete the steps of the aforementioned method.

[0087] In an exemplary embodiment, the electronic terminal 500 may be used by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), or complex programmable logic devices (CPLDs) to execute the aforementioned method.

[0088] According to the method provided in the embodiments of this application, this application also provides a computer program product, which includes: computer program code, which, when run on a computer, causes the computer to execute... Figure 1 The training method for a lightweight multimodal spine model with vertebral segment awareness, or a spine image analysis method, is shown in the embodiments described.

[0089] According to the method provided in the embodiments of this application, this application also provides a computer-readable storage medium storing program code, which, when executed on a computer, causes the computer to perform... Figure 1 The training method for a lightweight multimodal spine model with vertebral segment awareness, or a spine image analysis method, in any of the embodiments shown.

[0090] As used in this specification, the terms "component," "module," "system," etc., are used to refer to computer-related entities, hardware, firmware, combinations of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program, and / or a computer. As illustrated, applications running on computing devices and computing devices can both be components. One or more components may reside in a process and / or an execution thread, and components may be located on a single computer and / or distributed among two or more computers. Furthermore, these components can be executed from various computer-readable media on which various data structures are stored. Components can communicate, for example, via local and / or remote processes based on signals having one or more data packets (e.g., data from two components interacting with another component between a local system, a distributed system, and / or a network, such as the Internet interacting with other systems via signals).

[0091] Those skilled in the art will recognize that the various illustrative logical blocks and steps described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0092] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0093] In the embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0094] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0095] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0096] In the above embodiments, the functions of each functional unit can be implemented entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. A computer program product includes one or more computer instructions (programs). When the computer program instructions (programs) are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., high-density digital video discs, DVDs), or semiconductor media (e.g., solid-state disks, SSDs, etc.).

[0097] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0098] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0099] In summary, this application provides a training method, device, terminal, medium, and analysis method for a lightweight multimodal spinal model with vertebral segment awareness. This application achieves low-cost localized deployment by constructing a lightweight multimodal architecture. Simultaneously, through specific data input and a progressive three-stage training strategy, a lightweight model with spinal specialty capabilities is constructed and trained, thereby obtaining more accurate spinal image analysis results. Therefore, this application effectively overcomes various shortcomings of existing technologies and has high industrial applicability.

[0100] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this application should still be covered by the claims of this application.

Claims

1. A training method for a lightweight multimodal spinal model with vertebral segment awareness, characterized in that, include: Build a lightweight multimodal architecture; The lightweight multimodal architecture is trained in three stages to obtain a trained lightweight multimodal model of spine specialty with vertebral segment awareness; wherein, the three stages of training include: general medical and orthopedic basic training, spine specialty enhancement and segment awareness logic injection training, and clinical workflow and long text alignment training.

2. The training method for a lightweight multimodal spinal model with vertebral segment awareness according to claim 1, characterized in that, The lightweight multimodal architecture includes a visual encoder and a text encoder; wherein the visual encoder is connected to a multilayer perceptron; the output of the multilayer perceptron and the output of the text encoder are concatenated and used as the input of a pre-trained lightweight large language model.

3. The training method for a lightweight multimodal spinal model with vertebral segment awareness according to claim 2, characterized in that, The parameter size of the lightweight large language model is 7B.

4. The training method for a lightweight multimodal spinal model with vertebral segment awareness according to claim 1, characterized in that, The basic medical and orthopedic training includes: Obtain a general medical multimodal dataset and a non-spinal orthopedic dataset to construct a basic training set for medicine and orthopedics; Using the aforementioned medical and orthopedic basic training set and based on the DeepSpeed ​​Zero2 optimization strategy, the lightweight multimodal architecture is trained to obtain an initial training model.

5. The training method for a lightweight multimodal spinal model with vertebral segment awareness according to claim 4, characterized in that, The spinal specialty enhancement and segmental perception logic injection training includes: Obtain a spinal specialty instruction dataset; wherein, a portion of the spinal specialty instruction data in the dataset includes long thought chains; the long thought chains include: anatomical localization, image analysis, neural connections, and diagnostic conclusions; Using the aforementioned spinal specialty instruction dataset and based on the DeepSpeed ​​Zero2 optimization strategy, the initial training model is trained to obtain an intermediate training model.

6. The training method for a lightweight multimodal spinal model with vertebral segment awareness according to claim 5, characterized in that, The clinical workflow and long text alignment training includes: Acquire multi-round consultation dialogue data and full-process medical report data to construct a clinical data training set; Using the aforementioned clinical data training set and based on the DeepSpeed ​​Zero3 Offloading optimization strategy, the intermediate training model is trained to obtain a trained lightweight multimodal large model of spine specialty with vertebral segment awareness.

7. A training device for a lightweight multimodal spinal model with vertebral segment sensing, characterized in that, include: Modules for building lightweight multimodal architectures; The three-stage training module is used to perform three-stage training on the lightweight multimodal architecture to obtain a trained lightweight spinal specialty multimodal large model with vertebral segment awareness; wherein, the three-stage training includes: general medical and orthopedic basic training, spinal specialty reinforcement and segment awareness logic injection training, and clinical workflow and long text alignment training.

8. A method for analyzing spinal images, characterized in that, The method includes: Acquire and preprocess the spinal images to be analyzed; The spinal image is input into a lightweight multimodal spinal model with vertebral segment awareness to obtain corresponding analysis results; wherein the lightweight multimodal spinal model with vertebral segment awareness is trained by the training method of any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 6, or the method of claim 8.

10. An electronic terminal, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the method of any one of claims 1 to 6, or the method of claim 8.

Images