A hallux valgus intelligent judgment and prognosis analysis method and system

Abstract

The application discloses a hallux valgus intelligent judgment and prognosis analysis method and system, and belongs to the technical field of medical image processing, computer-aided diagnosis and biomechanical simulation. The hallux valgus prognosis analysis method comprises the following steps: acquiring foot multi-modal data of hallux valgus to be judged and performing pretreatment, inputting the pretreated foot multi-modal data into a pre-trained multi-modal large model, and outputting a judgment result of hallux valgus; creating a digital twin, performing virtual intervention simulation, and predicting a prognosis effect; and visualizing a process and result data of hallux valgus judgment and prognosis analysis and generating a structured report. The application can achieve the effects of fusing multi-dimensional information, revealing pathological biomechanical mechanisms, and performing individualized prognosis simulation analysis.

Description

Technical Field

[0001] This invention relates to a method and system for intelligent determination and prognostic analysis of hallux valgus, belonging to the interdisciplinary fields of medical image processing, computer-aided diagnosis, and biomechanical simulation. Specifically, it relates to a method and system for automated, precise, and personalized intelligent determination and prognostic analysis of hallux valgus by comprehensively utilizing multimodal medical data, advanced artificial intelligence models, and biomechanical simulation technology. Background Technology

[0002] Hallux valgus is a common foot deformity, especially prevalent in women with an incidence rate as high as 30%. It typically presents as lateral deviation of the big toe and medial deviation of the first metatarsophalangeal joint, leading to swelling and protrusion of the first metatarsophalangeal joint. Repeated friction between this protrusion and shoes causes bursitis near the metatarsophalangeal joint, and in severe cases, can further induce forefoot deformities, affecting daily walking activities. Currently, the diagnosis of hallux valgus mainly relies on physical examination and X-ray measurements by a professional physician to formulate a subsequent treatment plan, which presents several problems, including inconvenience in accessing medical care and a relatively cumbersome process. Existing hallux valgus assessment techniques have the following limitations:

[0003] Subjectivity and inconsistency: Traditional diagnosis relies on doctors manually marking and measuring bony landmarks, which involves human error and inter-observer differences.

[0004] Limited information dimensions: Currently, most publicly available patents for AI-based hallux valgus assisted diagnosis focus on two-dimensional analysis of foot X-rays, using convolutional neural networks for angle measurement or classification.

[0005] Lack of prognostic analysis capabilities: Most existing technologies are limited to the assessment stage, that is, determining whether hallux valgus exists and its severity, but cannot provide personalized predictions and simulations of the disease progression trend and the postoperative effects of different treatment options (such as conservative treatment and different surgical procedures), which is crucial for clinical decision-making.

[0006] Limited model generalization ability: Traditional deep learning models rely on a large amount of precisely labeled data, are sensitive to image quality and shooting position, and have poor interpretability, making it difficult for clinicians to fully trust them. Summary of the Invention

[0007] The purpose of this invention is to provide a method and system for intelligent diagnosis and prognostic analysis of hallux valgus. This invention aims to overcome the shortcomings of existing technologies by using artificial intelligence technology and biomechanical simulation to provide a more objective, multi-dimensional, prognostic analysis-capable, and model-generalization-capable intelligent diagnosis and prognostic analysis method and system for hallux valgus based on a multimodal large model and biomechanical simulation.

[0008] To achieve the above objectives, the present invention is implemented using the following technical solution:

[0009] In a first aspect, the present invention provides an intelligent method for determining hallux valgus, comprising:

[0010] Acquire and preprocess multimodal foot data for patients with hallux valgus, including weight-bearing X-ray images of the foot, three-dimensional surface scanning point cloud data of the foot, dynamic plantar pressure distribution time-series data, and clinical text descriptions of the patients.

[0011] The pre-processed multimodal foot data is input into the pre-trained multimodal large model, and the hallux valgus determination result is output.

[0012] The process and results of hallux valgus diagnosis are visualized and generated, and a structured report on hallux valgus diagnosis is generated. The process and results of hallux valgus diagnosis include the original image, AI annotation and measurement of preprocessed data, pathological component radar map of comprehensive pathological feature vector, and biomechanical simulation cloud map.

[0013] During the offline training phase, the multimodal large model has a simulation verification unit built at its output end.

[0014] The multimodal large model is used to fuse and analyze preprocessed foot multimodal data to generate a comprehensive pathological feature vector and judgment result for hallux valgus;

[0015] The simulation verification unit is used to construct a foot biomechanical model or a simplified musculoskeletal model based on the comprehensive pathological feature vector and the three-dimensional surface scanning point cloud data of the foot, perform simulation calculations to verify the hallux valgus determination result output by the multimodal large model, and generate reward or supervision signals based on the verification results and feed them back to the multimodal large model. The multimodal large model is then fine-tuned based on the signals fed back by the simulation verification module.

[0016] In the above technical solution, static and dynamic foot data are collected and preprocessed. A pre-trained multimodal large model is used for intelligent determination of hallux valgus. During the offline training phase of the multimodal large model, a comprehensive pathological feature vector and determination results are generated. The simulation verification unit at the output end constructs a biomechanical model based on the comprehensive pathological feature vector and 3D point cloud data, performs simulation calculations, and compares and verifies the determination results. The results are then fed back to the multimodal large model. By physically verifying and correcting the AI ​​determination results, the accuracy of the large model's judgment is enhanced. The above determination method can integrate multi-dimensional information, reveal the pathological biomechanical mechanism, reduce subjectivity and inconsistency, and make the results more accurate and reliable.

[0017] Intelligent determination of multimodal simulation models is achieved by using multimodal data containing 3D point clouds. Verification and retraining of large multimodal models are carried out by constructing a foot biomechanical model. The information is multidimensional and avoids the problem of limited model generalization ability to a certain extent. The trained multimodal simulation model has good interpretability.

[0018] Furthermore, the training of the multimodal large model also includes:

[0019] Preprocessing of foot multimodal sample data;

[0020] Based on known anatomical and biomechanical causal prior knowledge, a causal graph containing four nodes—bone deformity, soft tissue changes, mechanical environment, and clinical symptoms—is constructed. The edge weights that strengthen the rule that bone deformity leads to changes in the mechanical environment are set and input into the multimodal large model.

[0021] A loss function for a multimodal large model is constructed, which is based on classification / regression loss and adds a causal consistency loss term. When using the loss function to train the multimodal large model with data-label pairs, a causal graph is used as a soft constraint to penalize predictions that violate strong causal priors, so that the multimodal large model outputs a comprehensive pathological feature vector and judgment result that conforms to causal priors based on the training sample data.

[0022] A foot biomechanical model or musculoskeletal model is constructed using a simulation verification unit based on comprehensive pathological feature vectors and three-dimensional surface scanning point cloud data of the foot. The foot biomechanical model or simplified musculoskeletal model is then imported into the finite element model or musculoskeletal model. Static and dynamic simulation calculations are performed using a reduced-order simulation method to obtain the simulation results.

[0023] The simulation results are compared with foot multimodal data. If the simulation results are consistent with the clinical abnormal biomechanical environment reflected by the foot multimodal data, a reward signal is fed back to the multimodal large model and the judgment result is retained for output. Otherwise, the incorrect result is marked and fed back to the multimodal large model.

[0024] The loss function is calculated based on the feedback results of the simulation verification unit, and the multimodal large model is fine-tuned. When the loss function converges to a stable state, the target multimodal large model is obtained.

[0025] In the above technical solution, preprocessed multimodal foot data is input into a large multimodal model. The information of each modality is converted into feature vectors, and data-label pairs are obtained. Known anatomical and biomechanical causal prior knowledge is used as soft constraints and input in text form. Both are simultaneously input into the large multimodal model, and a unified output result is obtained. After training the large multimodal model, the output result has better descriptiveness. For example, the addition of the constraint that arch collapse may increase the pressure on the second and third metatarsals makes the model output not only information about arch collapse, but also the possibility of increased pressure on the second and third metatarsals caused by arch collapse.

[0026] A causal consistency loss term is added to the model's loss function to supervise model training. Model parameters are optimized, and the output is consistent with the causal graph. If the model strongly predicts large-angle deformities based solely on pain text without sufficient imaging evidence, it will be penalized. The technical solution incorporates static morphological information, foot soft tissue status, dynamic gait information, and considerations of deep biomechanical causal relationships. This makes the model learning more aligned with medical logical reasoning, compensating for the shortcomings of existing hallux valgus diagnosis techniques. A multimodal large model trained through a causal inference enhancement strategy is fused and analyzed to extract comprehensive pathological feature vectors for preliminary hallux valgus diagnosis. The data includes foot image data, 3D point cloud data, pressure data, textual information, and considerations of deep biomechanical causal relationships, resulting in more accurate output pathological feature vectors.

[0027] In the above technical solution, a foot biomechanical model or a simplified musculoskeletal model is imported into a finite element model or a musculoskeletal model. The standing phase (static) simulation and gait phase (dynamic) simulation calculations are performed using the reduced-order simulation method in the finite element model or musculoskeletal model to obtain results including the stress distribution of the first metatarsophalangeal joint, the load of the intermetarticular ligaments, and the tension of the plantar fascia. The process of hallux valgus determination realizes the physical verification of the determination results of the multimodal large model. If the pathological components obtained after the foot multimodal data is input into the multimodal large model can reproduce the abnormal mechanical environment observed in clinical practice, a reward signal is fed back to the multimodal large model. If it cannot reproduce the abnormal mechanical environment, an error signal is fed back to the multimodal large model for error correction and fine-tuning.

[0028] Therefore, the multimodal simulation model, which has been verified by causal priors and simulation physics, has higher accuracy in intelligently determining hallux valgus.

[0029] Optionally, the preprocessing of the multimodal foot sample data includes answer annotation of the sample images. The answer annotation includes: annotating the main body region of the foot in the foot X-ray image, annotating each toe sub-region of the foot, and providing detailed information description of the foot image. The detailed information description of the foot image includes the hallux valgus status of the current image, the distance from other toes, patient information, and the patient's clinical text description.

[0030] The patient information in the above technical solution includes the patient's gender, age, height, and weight. The patient's clinical text description includes the chief complaint, medical history, and physical examination record. The training file formed by the patient's detailed annotation information, combined with three-dimensional point cloud data and foot pressure data, can further enhance the accuracy of subsequent training model judgments. The hallux valgus condition of the image includes specific information on the hallux valgus angle (HVA), intermetatarsal angle (IMA), medial soft tissue tension component, and first metatarsal stability component, as well as the corresponding hallux valgus judgment results, such as normal, mild hallux valgus, moderate hallux valgus, and severe hallux valgus. The annotated data is trained in a multimodal large model. The annotation is based on manually annotated information, which improves the accuracy of the output hallux valgus judgment results.

[0031] Furthermore, the preprocessing includes: processing the images in the multimodal data, performing spatial registration and data pairing of image data, point cloud data, time series data and text data, and aligning the information of the multimodal data;

[0032] Image processing includes one or a combination of image rotation, image noise reduction, image affine transformation, and image scaling.

[0033] In the above technical solution, the acquired multimodal data is paired, that is, the data from different modalities are processed separately and then aligned, including aligning X-ray images with 3D scan point cloud data. The processed data facilitates feature length alignment in the next fusion and analysis step. When capturing CT images, there are interferences from environmental factors such as different backgrounds, lighting, and angles, which can lead to unsatisfactory analysis results of subsequent large-scale models. In order to better robustly analyze images captured under different environments, it is necessary to perform operations such as image rotation, image noise reduction, image affine transformation, and image scaling on the acquired training data.

[0034] Furthermore, the multimodal big model adopts a hierarchical multimodal big model, which includes: a visual encoding unit, a temporal data encoding unit, a text encoding unit, and a multimodal interaction understanding core;

[0035] Among them, the weight-bearing X-ray of the foot and the three-dimensional surface scanning point cloud data of the foot are input into the visual coding unit for feature extraction, and the extracted features are fused with spatial attention to output a visual fusion feature vector.

[0036] The dynamic plantar pressure distribution time series data is input into the time series coding unit to extract abnormal mechanical features and output a time series feature embedding vector.

[0037] The patient's clinical text description is input into the text encoding unit, which extracts symptom keywords and semantic information, and outputs a text feature embedding vector.

[0038] Visual fusion feature vectors, temporal feature embedding vectors, and text feature embedding vectors are used as multimodal inputs. Feature alignment and cross-modal feature fusion are performed through a cross-modal attention mechanism to output a comprehensive pathological feature vector and a judgment result.

[0039] The comprehensive pathological feature vector includes traditional angular measurements labeled during data preprocessing and interpretable pathological components.

[0040] In the above technical solution, the three branch units correspond to their respective inputs and outputs. Then, through a cross-modal attention mechanism, the features of the three branch units are fused. During fusion, the features are aligned, meaning that the feature lengths of the three branches are the same, for example, all are 256-dimensional feature vectors. After fusion, the network layer of the multimodal interactive understanding core outputs all the information needed for hallux valgus determination, namely, the angle measurement values, pathological components, hallux valgus conditions, etc., which are labeled during preprocessing. The input and output are completely corresponding. After the labeled information passes through each network structure, the pathological feature vector and the determination result are output to the simulation verification unit for physical verification.

[0041] Multimodal data is used for feature alignment and fusion. Multimodal data contains both static and dynamic information, and has multiple information dimensions, which reduces the judgment error caused by relying solely on two-dimensional data for diagnosis in existing technologies.

[0042] Optionally, the angle measurements include hallux valgus angle and intermetatarsal angle, and the pathological components that can be explained include medial soft tissue tension component and first metatarsal stability component.

[0043] In the above technical solution, based on the hallux valgus angle and intermetatarsal angle, the medial soft tissue tension component and the first metatarsal stability component are added for judgment, and multi-dimensional data are used for joint judgment to increase the accuracy of hallux valgus determination results.

[0044] Furthermore, the visual encoding unit includes a dual-channel visual encoder, one of which is an X-ray image encoder based on VisionTransformer, used to extract deep features including bone morphology, joint space and bone density from the weight-bearing X-ray of the foot; the other is a three-dimensional point cloud encoder based on PointNet++, used to extract features including toe three-dimensional morphology, soft tissue contour and degree of curvature from the three-dimensional scan point cloud data. The two features are spatially attention-fused in the visual encoding unit and output as a visual fusion feature vector.

[0045] The temporal data encoding unit uses a temporal convolutional network combined with a self-attention mechanism to encode the movement trajectory of the dynamic plantar pressure center and the pressure peak and time phase of each region in the temporal data of dynamic plantar pressure distribution. It extracts abnormal mechanical features in the gait cycle as temporal feature embedding vectors for output.

[0046] The text encoding unit includes a pre-trained biomedical language model for encoding clinical texts and extracting symptom keywords and semantic information as text feature embedding vectors for output.

[0047] In the above technical solution, the visual encoding unit extracts and integrates features of the foot's skeletal morphology, joint space, bone density, three-dimensional toe morphology, soft tissue contour, and degree of curvature; the temporal data extracts foot pressure data; the text encoding unit extracts symptom information; and the multimodal interactive understanding core fuses and analyzes the input data of the branch units through a cross-modal attention mechanism to extract comprehensive pathological feature vector features for hallux valgus determination. The above data includes considerations of the foot's soft tissue state and dynamic gait information, providing more accurate data support for the output of pathological feature vectors.

[0048] Optionally, the pre-trained biomedical language model includes BioBERT.

[0049] Among the aforementioned technical solutions, BioBERT is a biomedical language model specifically designed for biomedical text mining tasks, such as biomedical named entity recognition, relation extraction, and question answering systems. BioBERT is pre-trained based on Google's BERT model and has undergone further fine-tuning in the biomedical field. It performs exceptionally well in processing biomedical text and is suitable for extracting clinical text and annotation information related to hallux valgus.

[0050] Furthermore, the foot biomechanical model or simplified musculoskeletal model is constructed by data extraction and model parameterization using a pre-trained U-Net structured neural network model. The weight-bearing X-ray of the foot is input into the neural network model to construct the skeleton of the model. The three-dimensional surface scan point cloud data of the foot is input to construct the soft tissue contour of the model. The pathological components of the comprehensive pathological feature vector are input to be converted into the boundary conditions or material property parameters of the model.

[0051] In the above technical solution, the role of the U-Net structured neural network model is to convert the visual features of X-ray images and 3D surface scan point clouds into quantified pathological values, and then map them to physical parameters through a regression head. Using thousands of data samples generated based on high-fidelity finite element simulation, a U-Net structured neural network is pre-trained. This neural network takes simplified representations of foot skeletal morphology, such as weight-bearing X-ray images, 3D surface scan point clouds, and pathological components, as input to construct a foot biomechanical model or a simplified musculoskeletal model. It can obtain predicted maps of plantar pressure distribution, peak stress in major joints, etc., and perform comparative verification of the mechanical environment. If the verification and judgment results are consistent, a reward signal is fed back, and the judgment process and results are output.

[0052] The core mapping of the above technical solution, which transforms pathological components into model boundary conditions or material property parameters, is the nonlinear relationship between pathological feature vectors (bone, soft tissue, or clinical vectors) and model physical parameters (boundary conditions or material properties), achieved by pre-trained U-Net. Bone pathological components affect both boundary conditions and material parameters, soft tissue components are mainly mapped to material parameters, and clinical components are mainly mapped to boundary conditions. The role of U-Net is to convert the visual features of images or point clouds into quantified pathological values, and then complete the mapping to physical parameters through a regression head.

[0053] Secondly, the present invention provides a method for prognostic analysis of hallux valgus, which, based on the intelligent hallux valgus determination method described in any one of the first aspects, further includes the following steps:

[0054] A digital twin is constructed based on a foot biomechanical model or a simplified musculoskeletal model built using a simulation verification unit. Treatment recommendations are provided based on the assessment results. Virtual intervention simulations are then performed on the digital twin based on the selected treatment recommendations to predict prognostic effects.

[0055] The process and results of hallux valgus prognosis analysis are visualized and the structured report of hallux valgus prognosis analysis is generated. The process and results of hallux valgus prognosis analysis include virtual surgical animation comparison and prognostic index charts.

[0056] In the above technical solution, a digital twin is constructed based on a foot biomechanical model or a simplified musculoskeletal model to conduct virtual intervention simulation of the treatment plan. This enables personalized prediction and simulation of the disease progression trend and the postoperative effects of different treatment plans (such as conservative treatment and different surgical procedures), and realizes the predictive ability of prognosis analysis based on the determination of hallux valgus.

[0057] Furthermore, a digital twin is constructed based on a foot biomechanical model or a simplified musculoskeletal model built using simulation verification units; treatment suggestions are given based on the hallux valgus assessment results; virtual intervention simulations are conducted on the digital twin to predict prognostic effects, specifically including:

[0058] A proxy model is trained using a foot biomechanical model or a simplified musculoskeletal model, which serves as the core computing engine for building a digital twin. The digital twin is then combined with data interaction and visualization interaction modules to construct a digital twin system.

[0059] The patient's hallux valgus assessment data is input into the digital twin system. The digital twin system includes a decision layer. The decision layer retrieves the clinical database based on the hallux valgus assessment data and the baseline prediction data of the surrogate model, matches and recommends appropriate treatment plans, and outputs the treatment results from the clinical database. At the same time, the decision layer calls the intervention parameters corresponding to the treatment plan, inputs the intervention parameters into the surrogate model, and the surrogate model outputs the parameter prediction results. The intervention process is dynamically visualized on the digital twin. The digital twin system outputs the final prognostic result, which includes the treatment result and the parameter prediction result.

[0060] The surrogate model takes geometric, mechanical, and intervention parameters extracted from a foot biomechanical model or musculoskeletal model as input, and uses mechanical, kinematic, and tissue stress distribution data output from finite element or musculoskeletal simulation as labels for supervised training. When the input data includes intervention parameters, the surrogate model outputs the predicted mechanical, kinematic, and tissue stress distribution data; when the input data does not include intervention parameters, it outputs the baseline predicted mechanical, kinematic, and tissue stress distribution data.

[0061] In the above technical solution, the clinical database integrates databases of common surgical procedures (such as Chevron, Scarf, and Lapidus procedures). The decision-making logic of the digital twin decision layer can recommend conservative treatment or several feasible surgical procedures based on the type of hallux valgus deformity. The intervention parameters are input into the surrogate model, and virtual osteotomy, displacement, and fixation are automatically performed on the digital twin. The system predicts postoperative foot alignment, joint contact stress, and secondary changes in adjacent joints. It quantitatively evaluates indicators such as the deformity correction rate, stability gain, and risk of adjacent joint degeneration for each procedure, as well as the first metatarsal curvature grade. Conservative treatment simulates the impact of wearing different types of orthotics on arch support and pressure redistribution, predicts the degree of symptom relief, and provides data support for the selection of surgical procedures.

[0062] During prognostic analysis, the surrogate model provides prediction results in seconds for rapid solution previews; for the 1-2 preferred solutions determined in the final analysis, high-precision finite element simulation can be started for final confirmation, balancing speed and accuracy.

[0063] Optionally, the proxy model includes a machine learning model, a deep learning model, a neural network model, a response surface model, a multinomial model, or a hybrid model.

[0064] Optionally, the proxy model is ResNet50.

[0065] Optionally, the clinical database stores conservative treatment options and surgical treatment options, including Chevron, Scarf, and Lapidus procedures;

[0066] Intervention parameters for conservative treatment include the effect of wearing different sizes of orthotics on arch support and pressure redistribution; intervention parameters for surgical treatment include osteotomy angle and displacement distance.

[0067] Thirdly, the present invention provides a hallux valgus intelligent determination system for implementing the hallux valgus intelligent determination method described in any one of the first aspects, comprising:

[0068] The multimodal data acquisition and preprocessing module acquires and preprocesses multimodal foot data of patients with hallux valgus to be diagnosed. The multimodal foot data includes foot weight-bearing X-ray, foot three-dimensional surface scanning point cloud data, dynamic plantar pressure distribution time series data, and the patient's clinical text description.

[0069] The multimodal large model module is used to input pre-processed foot multimodal data into a pre-trained multimodal large model and output the hallux valgus determination result.

[0070] The visualization interaction and report generation module is used to visualize the determination process and result data of hallux valgus, and generate a structured report of hallux valgus determination. The determination process and result data of hallux valgus include the original image, AI annotation and measurement of preprocessed data, pathological component radar map of comprehensive pathological feature vector, and biomechanical simulation cloud map.

[0071] The above technical solution achieves intelligent determination of hallux valgus by setting up a multimodal data acquisition and preprocessing module, a multimodal simulation model analysis module, and a visualization interaction and report generation module. It can determine the hallux valgus condition based on multimodal data, avoid the limitation of model generalization ability, and has practical significance and good application prospects for accurate judgment of hallux valgus symptoms.

[0072] Fourthly, the present invention provides a hallux valgus prognostic analysis system for implementing the hallux valgus prognostic analysis method described in any of the second aspects, comprising:

[0073] The multimodal data acquisition and preprocessing module acquires and preprocesses multimodal foot data of patients with hallux valgus to be diagnosed. The multimodal foot data includes foot weight-bearing X-ray, foot three-dimensional surface scanning point cloud data, dynamic plantar pressure distribution time series data, and the patient's clinical text description.

[0074] The multimodal large model module is used to input pre-processed foot multimodal data into a pre-trained multimodal large model and output the hallux valgus determination result.

[0075] The digital twin and prognostic analysis module is used to construct a digital twin based on the foot biomechanical model or simplified musculoskeletal model of the simulation verification unit, provide treatment suggestions based on the judgment results, and perform virtual intervention simulation on the digital twin according to the selected treatment suggestions to predict the prognostic effect.

[0076] The visualization and report generation module is used to visualize the process and results of hallux valgus diagnosis, as well as the prognostic analysis process and results. It also generates a structured report on hallux valgus diagnosis and prognostic analysis. The diagnosis process and results, as well as the prognostic analysis process and results, include original images, AI annotation and measurement of preprocessed data, pathological component radar charts and biomechanical simulation cloud maps of comprehensive pathological feature vectors, virtual surgical animation comparisons, and prognostic index charts.

[0077] The above technical solution, by setting up a multimodal data acquisition and preprocessing module, a multimodal simulation model module, a digital twin and prognostic analysis module, and a visualization interaction and report generation module, realizes intelligent judgment and prognostic analysis of hallux valgus. It can determine the hallux valgus condition based on multimodal data and perform prognostic analysis of hallux valgus treatment plans, which has practical significance and good application prospects in the field of hallux valgus.

[0078] Fifthly, the present invention provides a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the hallux valgus intelligent determination method as described in any of the first aspects and / or the hallux valgus prognostic analysis method as described in any of the second aspects.

[0079] Compared with the prior art, the beneficial effects achieved by the present invention are as follows:

[0080] This invention collects and preprocesses static and dynamic foot data, and uses a pre-trained multimodal large model to intelligently determine hallux valgus. The multimodal large model generates a comprehensive pathological feature vector and a determination result. The simulation verification unit constructs a biomechanical model based on the comprehensive pathological feature vector and 3D point cloud data, performs simulation calculations, and compares and verifies the determination result. The result is then fed back to the multimodal large model. By physically verifying and correcting the AI ​​determination result, the accuracy of the large model's judgment is enhanced. This determination method can integrate multi-dimensional information, reveal the pathological biomechanical mechanism, reduce subjectivity and inconsistency, and make the results more accurate and reliable.

[0081] Intelligent judgment of multimodal simulation models is achieved by using multimodal data containing 3D point clouds. The model is validated and retrained by constructing a foot biomechanical model. The information is multidimensional and avoids the problem of limited model generalization ability to a certain extent. The trained model has good interpretability.

[0082] This invention constructs a digital twin based on a foot biomechanical model or a simplified musculoskeletal model to simulate virtual intervention for treatment plans. It can make personalized predictions and simulations of the disease progression trend and the postoperative effects of different treatment plans (such as conservative treatment and different surgical procedures), and realize the predictive ability of prognosis analysis based on hallux valgus determination. Attached Figure Description

[0083] Figure 1 This is a schematic diagram of the system modules for intelligent hallux valgus determination according to the present invention;

[0084] Figure 2 This is a schematic diagram of the system modules of the hallux valgus prognosis analysis system of the present invention. Detailed Implementation

[0085] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present invention and the specific features in the embodiments are detailed descriptions of the technical solution of the present invention, rather than limitations thereof. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments can be combined with each other.

[0086] The term "and / or" simply describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Additionally, the character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0087] The technical concept of this invention is as follows: Acquire multimodal data including weight-bearing X-ray images of the foot in cases of hallux valgus, three-dimensional surface scanning point cloud data of the foot, time-series data of dynamic plantar pressure distribution, and clinical text descriptions of the patient. Extract feature parameters reflecting the hallux valgus status of the patient. Utilize a pre-trained multimodal simulation model to determine the hallux valgus condition and output the determination process parameters and results. Construct a digital twin based on a biomechanical simulation model built within the multimodal simulation model. The digital twin queries a clinical database, performs virtual animation simulation based on the intervention parameters of the selected treatment plan, and outputs prognostic prediction results, thus performing hallux valgus determination and prognostic analysis. This method effectively overcomes the shortcomings of existing technologies in determining hallux valgus, accurately assessing the pathological condition of hallux valgus and performing prognostic analysis.

[0088] Example 1

[0089] This embodiment introduces an intelligent method for determining hallux valgus, including:

[0090] S1: Acquire and preprocess multimodal foot data of the patient to be diagnosed with hallux valgus. The multimodal foot data includes foot weight-bearing X-ray, foot three-dimensional surface scanning point cloud data, dynamic plantar pressure distribution time series data, and the patient's clinical text description.

[0091] S2: Input the pre-processed multimodal foot data into the pre-trained multimodal large model and output the hallux valgus determination result;

[0092] S3: Visualize the process and results of hallux valgus diagnosis and generate a structured report on hallux valgus diagnosis. The process and results of hallux valgus diagnosis include the original image, AI annotation and measurement of preprocessed data, pathological component radar map of comprehensive pathological feature vector, and biomechanical simulation cloud map.

[0093] During the offline training phase, the multimodal large model has a simulation verification unit built at its output end.

[0094] The multimodal large model is used to fuse and analyze preprocessed foot multimodal data to generate a comprehensive pathological feature vector and judgment result for hallux valgus;

[0095] The simulation verification unit is used to construct a foot biomechanical model or a simplified musculoskeletal model based on the comprehensive pathological feature vector and the three-dimensional surface scanning point cloud data of the foot, perform simulation calculations to verify the hallux valgus determination result output by the multimodal large model, and generate reward or supervision signals based on the verification results and feed them back to the multimodal large model. The multimodal large model is then fine-tuned based on the signals fed back by the simulation verification module.

[0096] Specifically,

[0097] In step S1, the weight-bearing X-ray of the foot includes anteroposterior and lateral views, the dynamic plantar pressure distribution time-series data is detected by a pressure plate or smart insole, and the patient's clinical text description includes chief complaint, medical history and physical examination record;

[0098] Preprocessing includes processing images by spatially registering and matching image data, time-series data, point cloud data, and text annotations, and aligning information in multimodal data; image processing includes one or a combination of image rotation, image noise reduction, image affine transformation, and image scaling.

[0099] In step S2, the multimodal big model adopts a hierarchical multimodal big model, which includes: a visual encoding unit, a temporal data encoding unit, a text encoding unit, and a multimodal interaction understanding core;

[0100] In this embodiment, the visual encoding unit includes a dual-channel visual encoder, one of which is an X-ray image encoder based on VisionTransformer, and the other is a 3D point cloud encoder based on PointNet++. The weight-bearing X-ray of the foot with hallux valgus and the 3D surface scanning point cloud data of the foot are input into the visual encoding unit. The weight-bearing X-ray of the foot is used by the X-ray image encoder based on VisionTransformer to extract deep features such as bone morphology, joint space and bone density. The 3D surface scanning point cloud data of the foot is used by the 3D point cloud encoder based on PointNet++ to extract features such as toe 3D morphology, soft tissue contour and degree of curvature. The features extracted by the two channels are fused by spatial attention to output a visual fusion feature vector.

[0101] The temporal data encoding unit uses a temporal convolutional network combined with a self-attention mechanism. The dynamic plantar pressure distribution temporal data is input into the temporal encoding unit. The temporal data encoding unit encodes the moving trajectory of the dynamic plantar pressure center and the pressure peak and time phase of each region, extracts abnormal mechanical features, and outputs a temporal feature embedding vector.

[0102] The text encoding unit includes a pre-trained biomedical language model. In this embodiment, BioBERT is used. The patient's clinical text description is input into BioBERT. BioBERT encodes the clinical text, extracts symptom keywords and semantic information, and outputs a text feature embedding vector.

[0103] Visual fusion feature vector, temporal feature embedding vector and text feature embedding vector are used as multimodal inputs. Feature alignment and cross-modal feature fusion are performed through cross-modal attention mechanism. The output is a comprehensive pathological feature vector and preliminary judgment result. The comprehensive pathological feature vector includes HVA, IMA and medial soft tissue tension component and first metatarsal stability component labeled in step S1.

[0104] The foot biomechanical model or simplified musculoskeletal model is constructed by data extraction and model parameterization using a pre-trained U-Net structured neural network model. The model's skeleton is constructed by inputting a weight-bearing X-ray of the foot, the soft tissue contour of the model is constructed by inputting three-dimensional surface scan point cloud data of the foot, and the pathological components of the comprehensive pathological feature vector are converted into the model's boundary conditions or material property parameters.

[0105] The training of the multimodal large model also includes:

[0106] Preprocessing of foot multimodal sample data;

[0107] Based on known anatomical and biomechanical causal prior knowledge, a causal graph containing four nodes—bone deformity, soft tissue changes, mechanical environment, and clinical symptoms—is constructed. The edge weights that strengthen the rule that bone deformity leads to changes in the mechanical environment are set and input into the multimodal large model.

[0108] A loss function for a multimodal large model is constructed, which is based on classification / regression loss and adds a causal consistency loss term. When using the loss function to train the multimodal large model with data-label pairs, a causal graph is used as a soft constraint to penalize predictions that violate strong causal priors, so that the multimodal large model outputs a comprehensive pathological feature vector and judgment result that conforms to causal priors based on the training sample data.

[0109] A foot biomechanical model or musculoskeletal model is constructed using a simulation verification unit based on comprehensive pathological feature vectors and three-dimensional surface scanning point cloud data of the foot. The foot biomechanical model or simplified musculoskeletal model is then imported into the finite element model or musculoskeletal model. Static and dynamic simulation calculations are performed using a reduced-order simulation method to obtain the simulation results.

[0110] The simulation results are compared with foot multimodal data. If the simulation results are consistent with the clinical abnormal biomechanical environment reflected by the foot multimodal data, a reward signal is fed back to the multimodal large model and the judgment result is retained for output. Otherwise, the incorrect result is marked and fed back to the multimodal large model.

[0111] The loss function is calculated based on the feedback results of the simulation verification unit, and the multimodal large model is fine-tuned. When the loss function converges to a stable state, the target multimodal large model is obtained.

[0112] The preprocessing of multimodal foot sample data includes answer annotation of image data. The answer annotation includes: (1) annotating the main body region of the foot in the X-ray image of the foot, that is, the bounding rectangle of the main body of the foot in the image, which needs to completely include the foot region; (2) annotating each toe sub-region of the foot, that is, each toe needs to be annotated, for example, the region where the first toe is located is annotated as 1, the region where the second toe is located is annotated as 2, and so on for other sub-regions; (3) providing detailed information description of the currently annotated foot image, including the hallux valgus status of the current image, patient information and the patient's clinical text description. The hallux valgus status includes the specific information of hallux valgus angle (HVA), intermetatarsal angle (IMA), medial soft tissue tension component, first metatarsal stability component and corresponding hallux valgus judgment results, such as normal, mild hallux valgus, moderate hallux valgus and severe hallux valgus. The hallux valgus status corresponding to HVA and IMA indicators is shown in Table 1.

[0113] The known anatomical and biomechanical causal prior knowledge is input in text form, including: bony deformities can directly lead to changes in the local biomechanical environment; bony deformities can directly cause changes in the morphology and tension of surrounding soft tissues; bony deformities can directly manifest as changes in clinical symptoms; changes in the biomechanical environment can further aggravate changes in soft tissues; changes in the biomechanical environment can further induce or aggravate changes in clinical symptoms.

[0114] The parameterization function of the cause-effect graph is: C=F(X, f M (X),f s (X, f M (X)));where C represents the clinical outcome, X represents the bony deformity, f M (X) represents mechanics, f s (X, f M (X) represents soft tissue; the causal relationships in the cause-and-effect diagram are: bony deformity → change in mechanical environment; bony deformity → change in soft tissue; bony deformity → change in clinical symptoms; change in mechanical environment → change in soft tissue; change in mechanical environment → change in clinical symptoms;

[0115] The loss function is: L total =λ1L MSE ( ,M) + λ2L MSE ( ,S) + λ3L MSE ( ,C) + λ4L causal ;

[0116] Among them, L MSE L represents the mean squared error loss; causal λ1 represents the loss of causal structural consistency; X represents skeletal deformity; M represents mechanical; S represents soft tissue; C represents clinical outcome; λ1 to λ4 represent weighting coefficients.

[0117] The mapping relationship between the pathological feature vector and the boundary conditions or material parameters is: [B,P] = F(V,θ); where V: comprehensive pathological feature vector (including components such as bony deformities, soft tissue abnormalities, and clinical symptoms); B: model boundary conditions (such as constraints, loads, contact conditions, etc.); P: material property parameters (such as elastic modulus, Poisson's ratio, stiffness, etc. of bone / soft tissue); F: nonlinear mapping function driven by pre-trained U-Net (the core is the mapping of features extracted by U-Net to physical parameters); θ: pre-trained weights of U-Net (fixed / fine-tuned) to ensure the generalization of the mapping.

[0118] The simulation results include the stress distribution of the first metatarsophalangeal joint, the load on the intermetrotarsal ligaments, and the tension of the plantar fascia. Hallux valgus assessment includes the degree of hallux valgus, the presence of a fracture, and the extent of height loss. The stress distribution of the first metatarsophalangeal joint refers to the force / area borne at various points on the articular surface formed by the first metatarsal and the proximal phalanx, reflecting the risk of hallux valgus deformity, joint degeneration, and cartilage damage, measured in MPa (N / mm²). 2 );

[0119] The stress of the first metatarsophalangeal joint is calculated using the following formula: In the formula, σ represents joint stress, F represents joint contact force, and A represents joint contact area. In this embodiment, a finite element model is used to construct a biomechanical model of the foot. The finite element model is calculated according to the following formula: {σ}=[D]{ε}, where [D] represents the material stiffness matrix and {ε} represents the strain vector.

[0120] Intermetatarsal ligament load: refers to the tensile / shear force on the ligaments connecting the metatarsals (such as the transverse intermetatarsal ligament), reflecting ligament laxity, risk of rupture, and foot arch stability. Unit: N (Newton).

[0121] The load on the intermetarticular ligaments is calculated using the following formula: In the formula, Indicates ligament tension. Indicates ligament stiffness. Indicates elongation.

[0122] Plantar fascia tension: refers to the tensile force generated when the plantar fascia is stretched during walking and standing, reflecting plantar fasciitis, arch collapse, and loss of height. Unit: N (Newton).

[0123] Plantar fascia tension is calculated using the following formula: In the formula, Indicates fascial tension. This represents the nonlinear stiffness that varies with deformation.

[0124] Table 1: Criteria for Determining the Grade of Hallux Valgus.

[0125]

[0126] Example 2

[0127] This embodiment describes a method for prognostic analysis of hallux valgus, including:

[0128] S1: Acquire and preprocess multimodal foot data of the patient to be diagnosed with hallux valgus. The multimodal foot data includes foot weight-bearing X-ray, foot three-dimensional surface scanning point cloud data, dynamic plantar pressure distribution time series data, and the patient's clinical text description.

[0129] S2: Input the pre-processed multimodal foot data into the pre-trained multimodal large model and output the hallux valgus determination result;

[0130] S3: Construct a digital twin based on a foot biomechanical model or a simplified musculoskeletal model built on a simulation verification unit, provide treatment suggestions based on the judgment results, and perform virtual intervention simulation on the digital twin based on the selected treatment suggestions to predict the prognosis.

[0131] S4: Visualize the process and results of hallux valgus diagnosis, as well as the prognostic analysis process and results, and generate a structured report on hallux valgus diagnosis and prognostic analysis. The diagnosis process and results, as well as the prognostic analysis process and results, include original images, AI annotation and measurement of preprocessed data, pathological component radar charts and biomechanical simulation cloud maps of comprehensive pathological feature vectors, virtual surgical animation comparisons, and prognostic index charts.

[0132] Specifically,

[0133] Steps S1-S2 are designed the same as in Example 1. In step S3, a digital twin is constructed based on the foot biomechanical model or simplified musculoskeletal model built by the simulation verification unit; treatment suggestions are given based on the hallux valgus assessment results; virtual intervention simulation is carried out on the digital twin to predict the prognosis, specifically including:

[0134] A proxy model is trained using a foot biomechanical model or a simplified musculoskeletal model, which serves as the core computing engine for building a digital twin. The digital twin is then combined with data interaction and visualization interaction modules to construct a digital twin system.

[0135] The patient's hallux valgus assessment data is input into the digital twin system. The digital twin system includes a decision layer. The decision layer retrieves the clinical database based on the hallux valgus assessment data and the baseline prediction data of the surrogate model, matches and recommends appropriate treatment plans, and outputs the treatment results from the clinical database. At the same time, the decision layer calls the intervention parameters corresponding to the treatment plan, inputs the intervention parameters into the surrogate model, and the surrogate model outputs the parameter prediction results. The intervention process is dynamically visualized on the digital twin. The digital twin system outputs the final prognostic result, which includes the treatment result and the parameter prediction result.

[0136] The surrogate model takes geometric, mechanical, and intervention parameters extracted from a foot biomechanical model or musculoskeletal model as input, and uses mechanical, kinematic, and tissue stress distribution data output from finite element or musculoskeletal simulation as labels for supervised training. When the input data includes intervention parameters, the surrogate model outputs the predicted mechanical, kinematic, and tissue stress distribution data; when the input data does not include intervention parameters, it outputs the baseline predicted mechanical, kinematic, and tissue stress distribution data.

[0137] The proxy model is ResNet50; the mechanical data includes joint contact stress, ground reaction force, muscle tension, and skeletal force, including magnitude and direction; the kinematic data includes joint angles, skeletal displacement, gait trajectory, and movement speed, indicating how and how much the foot moves; the tissue stress distribution includes pressure, tension, and strain of bones, ligaments, and soft tissues, indicating where the force is applied and whether injury is possible.

[0138] In this system, the decision-making layer of the digital twin performs post-processing analysis based on the prediction results of the surrogate model. Its decision-making logic includes retrieving corresponding treatment plans from the clinical database based on mechanical data, HVA, IMA, etc. For example, it can retrieve clinical databases based on HVA=32°, joint contact stress or bone displacement angle output by the surrogate model, etc. The decision-making layer can also calculate the prognostic HVA and IMA values ​​based on the treatment results and parameter prediction results. The HVA and IMA values ​​are included in the final prognostic results.

[0139] The clinical database stores conservative and surgical treatment options. Surgical options include Chevron, Scarf, and Lapidus procedures. Intervention parameters for conservative treatment include the impact of wearing different orthotics on arch support and pressure redistribution. Intervention parameters for surgical treatment include osteotomy angle and displacement distance. After the intervention parameters are input into the proxy model, virtual osteotomy, displacement, and fixation are automatically performed on the digital twin. This predicts postoperative foot alignment, joint contact stress, and secondary changes in adjacent joints. It quantitatively assesses the deformity correction rate, stability gain, and risk of adjacent joint degeneration for each option, as well as the first metatarsal curvature grade. It also simulates the impact of wearing different orthotics on arch support and pressure redistribution, predicts the degree of symptom relief, and outputs the prediction results of different options, providing data support for surgical option selection.

[0140] Example 3

[0141] This embodiment provides a hallux valgus intelligent determination system. A schematic diagram of the system modules for the hallux valgus intelligent determination system is shown below. Figure 1 As shown, the hallux valgus intelligent determination system based on this embodiment implements the hallux valgus intelligent determination method described in Embodiment 1, including:

[0142] The multimodal data acquisition and preprocessing module acquires and preprocesses multimodal foot data of patients with hallux valgus to be diagnosed. The multimodal foot data includes foot weight-bearing X-ray, foot three-dimensional surface scanning point cloud data, dynamic plantar pressure distribution time series data, and the patient's clinical text description.

[0143] The multimodal large model module is used to input pre-processed foot multimodal data into a pre-trained multimodal large model and output the hallux valgus determination result.

[0144] The visualization interaction and report generation module is used to visualize the determination process and result data of hallux valgus, and generate a structured report of hallux valgus determination. The determination process and result data of hallux valgus include the original image, AI annotation and measurement of preprocessed data, pathological component radar map of comprehensive pathological feature vector, and biomechanical simulation cloud map.

[0145] Example 4

[0146] This embodiment provides a hallux valgus prognosis analysis system. A schematic diagram of the system modules for the hallux valgus prognosis analysis system is shown below. Figure 2 As shown, the hallux valgus prognostic analysis system based on this embodiment implements the hallux valgus prognostic analysis method described in Embodiment 2, including:

[0147] The multimodal data acquisition and preprocessing module acquires and preprocesses multimodal foot data of patients with hallux valgus to be diagnosed. The multimodal foot data includes foot weight-bearing X-ray, foot three-dimensional surface scanning point cloud data, dynamic plantar pressure distribution time series data, and the patient's clinical text description.

[0148] The multimodal large model module is used to input pre-processed foot multimodal data into a pre-trained multimodal large model and output the hallux valgus determination result.

[0149] The digital twin and prognostic analysis module is used to construct a digital twin based on the foot biomechanical model or simplified musculoskeletal model of the simulation verification unit, provide treatment suggestions based on the judgment results, and perform virtual intervention simulation on the digital twin according to the selected treatment suggestions to predict the prognostic effect.

[0150] The visualization and report generation module is used to visualize the process and results of hallux valgus diagnosis, as well as the prognostic analysis process and results. It also generates a structured report on hallux valgus diagnosis and prognostic analysis. The diagnosis process and results, as well as the prognostic analysis process and results, include original images, AI annotation and measurement of preprocessed data, pathological component radar charts and biomechanical simulation cloud maps of comprehensive pathological feature vectors, virtual surgical animation comparisons, and prognostic index charts.

[0151] Example 5

[0152] This embodiment, based on the hallux valgus prognostic analysis system of Embodiment 4, introduces an application for hallux valgus prognostic analysis during patient visits, including the following:

[0153] S1: When a patient seeks medical attention, the hallux valgus intelligent judgment and prognosis analysis system acquires anteroposterior and lateral X-ray images of the foot, three-dimensional surface scanning point cloud data of the foot, dynamic plantar pressure distribution time series data based on pressure plate, and clinical text descriptions such as the patient's chief complaint, medical history and physical examination records through the multimodal data acquisition and preprocessing module. The preprocessing submodule performs spatial registration and information alignment on the acquired image data, point cloud data, pressure time series data and text descriptions.

[0154] S2: Input the preprocessed data into the multimodal simulation model module, and automatically output the judgment process and results.

[0155] In this module, X-ray images are processed by a Vision Transformer encoder to extract features such as first metatarsal varus and sesamoid bone displacement; 3D scan point clouds are processed by a PointNet++ encoder to show a low arch; a temporal coding unit processes plantar pressure data showing a significant increase in subhead pressure of the second and third metatarsal heads; a text coding unit records redness, swelling, and pain on the medial side of the big toe; the multimodal interactive understanding core of the multimodal large model uses cross-modal attention to associate this information for feature alignment and data pairing, outputting a comprehensive pathological feature vector, in which the first metatarsal stability component scores extremely low, while the forefoot transverse arch collapse component scores high, and calculates... With HVA=32° and IMA=16°, the hallux valgus result was determined. The patient was diagnosed with moderate hallux valgus. The proportional feature vector and the determination result were output to the simulation verification unit. The simulation verification unit generated a simplified foot model of the patient based on the comprehensive pathological feature vector output by the multimodal large model and the three-dimensional scanning point cloud data of the foot. The low stability component was converted into the weakening of the stiffness of the first and second metatarsal ligaments. The simulation calculation obtained the stress data of the first metatarsophalangeal joint. The determination process and results, including the stress concentration on the medial side of the first metatarsophalangeal joint and the abnormal distribution of forefoot pressure in the dynamic plantar pressure distribution time series data, were output.

[0156] S4: The digital twin and prognostic analysis module is activated. A digital twin of the patient's foot is constructed based on a simplified foot model. The geometric and mechanical parameters of the patient's foot biomechanical model are input into the surrogate model. The decision layer of the digital twin queries the clinical database based on the patient's hallux valgus assessment results and the baseline prediction results of the surrogate model, recommending several feasible surgical procedures. The doctor selects the virtual Scarf osteotomy in the visual interactive interface. The decision layer retrieves and outputs the corresponding treatment results for Scarf osteotomy from the clinical database, and inputs the intervention parameters of the osteotomy, lateralization, and fixation operations of Scarf osteotomy into the surrogate model. The proxy model calculates the input intervention parameters and outputs the predicted mechanical data, kinematic data, and tissue stress distribution data. At the same time, the digital twin performs standard osteotomy, lateral displacement, and fixation operations. The decision-making layer calculates the predicted results based on the treatment results in the clinical database and the parameter prediction results output by the proxy model. The postoperative HVA is predicted to be corrected to 10°, the IMA is predicted to be corrected to 8°, and the stress distribution of the first metatarsophalangeal joint returns to normal. However, the simulation indicates that the stress of the second metatarsal bone is slightly increased. During this process, the digital twin provides a synchronous virtual animation. The digital twin system can also provide simulation results of Chevron osteotomy for comparison.

[0157] S5: The visualization interaction and report generation module displays the original images of the patient's foot, AI annotation and measurement, pathological component radar map, biomechanical simulation cloud map, virtual surgical animation comparison and prognostic index charts from the above steps. Finally, the system generates a report that details the advantages and disadvantages of the two options, Scarf osteotomy and Chevron osteotomy, to assist doctors and patients in making joint decisions.

[0158] Example 6

[0159] This embodiment provides a computer-readable medium storing a computer program thereon. When the computer program is executed by a processor, it implements the hallux valgus intelligent determination method as described in Embodiment 1 and / or the hallux valgus prognosis analysis method as described in Embodiment 2.

[0160] In summary, this invention uses artificial intelligence algorithms to automatically identify the pathological characteristics of hallux valgus in patients, assess the potential impact on patients, and construct a simulation verification unit at the output end of a multimodal large model to physically verify the AI ​​judgment results. This allows for fine-tuning and training of the multimodal large model, resulting in a more accurate multimodal large model. Furthermore, the patient's personalized biomechanical model constructed through the simulation verification unit creates a digital foot twin, providing data support for surgical plan selection. This invention has significant value for promotion and application.

[0161] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0162] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0163] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0164] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0165] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims. All of these forms are within the protection scope of the present invention.

Claims

1. A hallux valgus intelligent determination method, characterized by, include: Acquire and preprocess multimodal foot data for patients with hallux valgus, including weight-bearing X-ray images of the foot, three-dimensional surface scanning point cloud data of the foot, dynamic plantar pressure distribution time-series data, and clinical text descriptions of the patients. The pre-processed multimodal foot data is input into the pre-trained multimodal large model, and the hallux valgus determination result is output. The process and results of hallux valgus diagnosis are visualized and generated, and a structured report on hallux valgus diagnosis is generated. The process and results of hallux valgus diagnosis include the original image, AI annotation and measurement of preprocessed data, pathological component radar map of comprehensive pathological feature vector, and biomechanical simulation cloud map. During the offline training phase, the multimodal large model has a simulation verification unit built at its output end. The multimodal large model is used to fuse and analyze preprocessed foot multimodal data to generate a comprehensive pathological feature vector and judgment result for hallux valgus; The simulation verification unit is used to construct a foot biomechanical model or a simplified musculoskeletal model based on the comprehensive pathological feature vector and the three-dimensional surface scanning point cloud data of the foot, perform simulation calculations to verify the hallux valgus determination result output by the multimodal large model, and generate reward or supervision signals based on the verification results and feed them back to the multimodal large model. The multimodal large model is then fine-tuned based on the signals fed back by the simulation verification unit.

2. The hallux valgus intelligent determination method according to claim 1, characterized in that, The training of the multimodal large model also includes: Preprocessing of foot multimodal sample data; Based on known anatomical and biomechanical causal prior knowledge, a causal graph containing four nodes—bone deformity, soft tissue changes, mechanical environment, and clinical symptoms—is constructed. The edge weights that strengthen the rule that bone deformity leads to changes in the mechanical environment are set and input into the multimodal large model. A loss function for a multimodal large model is constructed, which is based on classification / regression loss and adds a causal consistency loss term. When using the loss function to train the multimodal large model with data-label pairs, a causal graph is used as a soft constraint to penalize predictions that violate strong causal priors, so that the multimodal large model outputs a comprehensive pathological feature vector and judgment result that conforms to causal priors based on the training sample data. A foot biomechanical model or musculoskeletal model is constructed using a simulation verification unit based on comprehensive pathological feature vectors and three-dimensional surface scanning point cloud data of the foot. The foot biomechanical model or simplified musculoskeletal model is then imported into the finite element model or musculoskeletal model. Static and dynamic simulation calculations are performed using a reduced-order simulation method to obtain the simulation results. The simulation results are compared with foot multimodal data. If the simulation results are consistent with the clinical abnormal biomechanical environment reflected by the foot multimodal data, a reward signal is fed back to the multimodal large model and the judgment result is retained for output. Otherwise, the incorrect result is marked and fed back to the multimodal large model. The loss function is calculated based on the feedback results of the simulation verification unit, and the multimodal large model is fine-tuned. When the loss function converges to a stable state, the target multimodal large model is obtained.

3. The intelligent method for determining hallux valgus according to claim 1, characterized in that, The preprocessing includes: processing the images in the multimodal data, performing spatial registration and data pairing of image data, point cloud data, time series data and text data, and aligning the information of the multimodal data; Image processing includes one or a combination of image rotation, image noise reduction, image affine transformation, and image scaling.

4. The intelligent method for determining hallux valgus according to claim 1, characterized in that, The multimodal big model adopts a hierarchical multimodal big model, which includes: a visual encoding unit, a temporal data encoding unit, a text encoding unit, and a multimodal interaction understanding core; Among them, the weight-bearing X-ray of the foot and the three-dimensional surface scanning point cloud data of the foot are input into the visual coding unit for feature extraction, and the extracted features are fused with spatial attention to output a visual fusion feature vector. The dynamic plantar pressure distribution time series data is input into the time series coding unit to extract abnormal mechanical features and output a time series feature embedding vector. The patient's clinical text description is input into the text encoding unit, which extracts symptom keywords and semantic information, and outputs a text feature embedding vector. Visual fusion feature vectors, temporal feature embedding vectors, and text feature embedding vectors are used as multimodal inputs. Feature alignment and cross-modal feature fusion are performed through a cross-modal attention mechanism to output a comprehensive pathological feature vector and a judgment result. The comprehensive pathological feature vector includes traditional angular measurements labeled during data preprocessing and interpretable pathological components.

5. The intelligent method for determining hallux valgus according to claim 4, characterized in that, The visual encoding unit includes a dual-channel visual encoder. One channel is an X-ray image encoder based on Vision Transformer, used to extract deep features including bone morphology, joint space and bone density from foot weight-bearing X-ray images. The other channel is a 3D point cloud encoder based on PointNet++, used to extract features including toe 3D morphology, soft tissue contour and degree of curvature from 3D scan point cloud data. The two features are spatially attention-fused in the visual encoding unit to output a visual fusion feature vector. The temporal data encoding unit uses a temporal convolutional network combined with a self-attention mechanism to encode the movement trajectory of the dynamic plantar pressure center and the pressure peak and time phase of each region in the temporal data of dynamic plantar pressure distribution. It extracts abnormal mechanical features in the gait cycle as temporal feature embedding vectors for output. The text encoding unit includes a pre-trained biomedical language model for encoding clinical texts and extracting symptom keywords and semantic information as text feature embedding vectors for output.

6. The intelligent method for determining hallux valgus according to claim 1, characterized in that, The foot biomechanical model or simplified musculoskeletal model is constructed by data extraction and model parameterization using a pre-trained U-Net structured neural network model. The weight-bearing X-ray of the foot is input into the neural network model to construct the skeleton of the foot biomechanical model or simplified musculoskeletal model. The three-dimensional surface scan point cloud data of the foot is input to construct the soft tissue contour of the foot biomechanical model or simplified musculoskeletal model. The pathological components of the comprehensive pathological feature vector are input and converted into the boundary conditions or material property parameters of the foot biomechanical model or simplified musculoskeletal model.

7. A method for prognostic analysis of hallux valgus based on the intelligent determination method for hallux valgus according to any one of claims 1-6, characterized in that, Includes the following steps: A digital twin is constructed based on a foot biomechanical model or a simplified musculoskeletal model built using a simulation verification unit. Treatment recommendations are provided based on the assessment results. Virtual intervention simulations are then performed on the digital twin based on the selected treatment recommendations to predict prognostic effects. The process and results of hallux valgus prognosis analysis are visualized and the structured report of hallux valgus prognosis analysis is generated. The process and results of hallux valgus prognosis analysis include virtual surgical animation comparison and prognostic index charts.

8. The prognostic analysis method for hallux valgus according to claim 7, characterized in that, A digital twin is constructed based on a foot biomechanical model or a simplified musculoskeletal model built using simulation verification units; treatment suggestions are given based on hallux valgus assessment results; virtual intervention simulations are conducted on the digital twin to predict prognostic effects, specifically including: A proxy model is trained using a foot biomechanical model or a simplified musculoskeletal model, which serves as the core computing engine for building a digital twin. The digital twin is then combined with data interaction and visualization interaction modules to construct a digital twin system. The patient's hallux valgus assessment data is input into the digital twin system. The digital twin system includes a decision layer. The decision layer retrieves the clinical database based on the hallux valgus assessment data and the baseline prediction data of the surrogate model, matches and recommends appropriate treatment plans, and outputs the treatment results from the clinical database. At the same time, the decision layer calls the intervention parameters corresponding to the treatment plan, inputs the intervention parameters into the surrogate model, and the surrogate model outputs the parameter prediction results. The intervention process is dynamically visualized on the digital twin. The digital twin system outputs the final prognostic result, which includes the treatment result and the parameter prediction result. The surrogate model takes geometric, mechanical, and intervention parameters extracted from a foot biomechanical model or musculoskeletal model as input, and uses mechanical, kinematic, and tissue stress distribution data output from finite element or musculoskeletal simulation as labels for supervised training. When the input data includes intervention parameters, the surrogate model outputs the predicted mechanical, kinematic, and tissue stress distribution data; when the input data does not include intervention parameters, it outputs the baseline predicted mechanical, kinematic, and tissue stress distribution data.

9. A hallux valgus intelligent determination system, used to implement the hallux valgus intelligent determination method according to any one of claims 1-6, characterized in that, include: The multimodal data acquisition and preprocessing module acquires and preprocesses multimodal foot data of patients with hallux valgus to be diagnosed. The multimodal foot data includes foot weight-bearing X-ray, foot three-dimensional surface scanning point cloud data, dynamic plantar pressure distribution time series data, and the patient's clinical text description. The multimodal large model module is used to input pre-processed foot multimodal data into a pre-trained multimodal large model and output the hallux valgus determination result. The visualization interaction and report generation module is used to visualize the determination process and result data of hallux valgus, and generate a structured report of hallux valgus determination. The determination process and result data of hallux valgus include the original image, AI annotation and measurement of preprocessed data, pathological component radar map of comprehensive pathological feature vector, and biomechanical simulation cloud map.

10. A hallux valgus prognostic analysis system, used to implement the hallux valgus prognostic analysis method according to any one of claims 7-8, characterized in that, include: The multimodal data acquisition and preprocessing module acquires and preprocesses multimodal foot data of patients with hallux valgus to be diagnosed. The multimodal foot data includes foot weight-bearing X-ray, foot three-dimensional surface scanning point cloud data, dynamic plantar pressure distribution time series data, and the patient's clinical text description. The multimodal large model module is used to input pre-processed foot multimodal data into a pre-trained multimodal large model and output the hallux valgus determination result. The digital twin and prognostic analysis module is used to construct a digital twin based on the foot biomechanical model or simplified musculoskeletal model of the simulation verification unit, provide treatment suggestions based on the judgment results, and perform virtual intervention simulation on the digital twin according to the selected treatment suggestions to predict the prognostic effect. The visualization and report generation module is used to visualize the process and results of hallux valgus diagnosis, as well as the prognostic analysis process and results. It also generates a structured report on hallux valgus diagnosis and prognostic analysis. The diagnosis process and results, as well as the prognostic analysis process and results, include original images, AI annotation and measurement of preprocessed data, pathological component radar charts and biomechanical simulation cloud maps of comprehensive pathological feature vectors, virtual surgical animation comparisons, and prognostic index charts.

Images