Virtual patient generation method and system based on multi-modal large model
By integrating patient medical history, imaging results, and laboratory indicators through a multimodal large model, and using convolutional neural networks and recurrent neural networks to capture local patterns and temporal changes, the problem of medical information integration has been solved, enabling efficient medical decision support and personalized health management.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-09
- Publication Date
- 2026-03-13
AI Technical Summary
Existing technologies struggle to effectively integrate and deeply understand medical information from different sources and in different forms, which limits the comprehensiveness and accuracy of medical decision support, especially when constructing a holistic health profile of a patient, making it difficult to accurately capture the dynamic patterns and potential correlations in disease progression.
A multimodal large model-based approach is adopted, which uses convolutional neural networks and recurrent neural networks to process patient medical history, imaging results and laboratory indicators, capture local patterns and temporal changes, and calculate the overall similarity score by combining a weighted average method to generate an optimized decision sequence and perform iterative updates.
It significantly improves the accuracy and personalization of medical decision-making, achieves deep integration and dynamic adjustment of multi-source heterogeneous data, and ensures the real-time and accuracy of analysis results.
Smart Images
Figure CN121662387A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of large model data processing technology, and in particular relates to a method and system for generating virtual patients based on multimodal large models. Background Technology
[0002] In the healthcare field, the comprehensive analysis and application of data are crucial for improving diagnostic accuracy and designing personalized treatment plans. This field is not only a vital pillar for safeguarding human health but also a frontier for promoting the integration of science and medicine. However, although current technologies have made some progress in processing medical information, existing methods often struggle to meet the complex needs of integrating and deeply understanding data from different sources and in different formats. In particular, when faced with diverse information, the lack of effective collaborative analysis capabilities limits the comprehensiveness and accuracy of medical decision support.
[0003] Looking deeper, a core technological challenge in the healthcare field lies in effectively linking information from different senses and recording methods. In particular, there is an inherent heterogeneity among various types of medical information; for example, written patient histories and imaging results differ greatly in expression and content focus. This heterogeneity makes it difficult to uncover the inherent connections between information. Furthermore, this lack of connection leads to another critical problem: when constructing a holistic health profile of a patient, it is impossible to accurately capture the dynamic patterns and potential correlations in disease progression, thus affecting a comprehensive assessment of the patient's condition.
[0004] Specifically, in real-world medical settings, doctors often need to simultaneously refer to the patient's verbal medical history, imaging examination images, and various laboratory test results to diagnose a condition. However, due to the lack of a unified integration path between these different types of information, doctors may be unable to quickly identify the connections hidden within these different pieces of information, such as the correspondence between a certain abnormal indicator and a specific lesion on an imaging image. This information silo phenomenon significantly reduces the efficiency and depth of medical analysis.
[0005] Therefore, breaking down the barriers between different types of medical information and achieving deep correlation and dynamic comprehensive analysis of multi-source data has become a key issue in improving medical decision support capabilities. Summary of the Invention
[0006] To address the aforementioned technical problems, this invention proposes a method and system for generating virtual patients based on a multimodal large model. This method can provide patients with more accurate and timely health management solutions.
[0007] To achieve the above objectives, the present invention provides a method for generating virtual patients based on a multimodal large model, comprising:
[0008] Keyword sequences and semantic vectors are extracted from the patient's medical history text. Convolutional neural networks are used to process the keyword sequences and semantic vectors to capture local patterns and obtain a set of medical history-related features.
[0009] Based on the medical history-related feature set, the texture descriptor of the corresponding region is obtained from the image results. The temporal changes of the texture descriptor are analyzed by a recurrent neural network to determine the dynamic law vector of the image.
[0010] Based on the image dynamic pattern vector, numerical sequences and abnormal markers are extracted from the test indicators to obtain the potential feature group of the test.
[0011] Based on the integrated health profiling framework, the dynamic pattern vector of the images and the potential feature groups of the laboratory tests are integrated, and the overall similarity score is calculated using a weighted average method to obtain the patient status assessment index.
[0012] The patient status assessment indicators are used to retrieve matching treatment path templates from the decision support database. Conditional branching logic is used to compare the deviation between the indicators and the templates to determine the optimized decision sequence.
[0013] Feedback reports are generated based on the optimized decision sequence, and newly acquired dynamic patterns and potential features are incorporated through an iterative update mechanism to obtain virtual patient generation results.
[0014] Optionally, obtaining the medical history-related feature set includes:
[0015] Keyword sequences and semantic vectors are extracted from medical history texts using a pre-defined feature extraction model to complete the initial data extraction and obtain an initial text representation. The semantic vectors are calculated using a word embedding model.
[0016] ;
[0017] in, Let L be the semantic vector of the medical history text, and L be the number of keywords in the text. Let be the word embedding vector of the i-th keyword;
[0018] Based on the initial text representation, a convolutional neural network is used to process the keyword sequence and semantic vector, capturing local patterns and determining the association of key information in the medical history text. The convolution operation is as follows:
[0019] ;
[0020] in, For the j-th convolutional feature, The convolution kernel vector, Let be a local vector consisting of the j-th to j+l-1-th elements in the input sequence, where l is the length of the convolution kernel. The bias term is ReLU, and the activation function is ReLU.
[0021] Based on the key information in the medical history text, a feature set associated with the medical history is generated.
[0022] Optionally, obtaining the texture descriptor of the corresponding region from the image results based on the medical history-related feature set includes:
[0023] By extracting pixel distribution information of the corresponding region from the image data, and using a pre-established segmentation model to divide the image data into regions, preliminary regional range data is obtained.
[0024] Based on the preliminary area range data, texture description information within the corresponding area is obtained. The texture description information is then normalized using a standardization method to determine the texture descriptor. The normalization process includes: ;
[0025] in, Here is the normalized texture descriptor, and T is the original texture feature value. min T is the minimum value of this texture feature. max This represents the maximum value of the texture feature.
[0026] Optionally, analyzing the temporal changes of texture descriptors using a recurrent neural network to determine the dynamic law vector of the image includes:
[0027] A recurrent neural network (RNN) is used to model and analyze the temporal changes of texture descriptors, identifying the continuous patterns of these changes. The hidden layer state updates of the RNN include:
[0028] ;
[0029] in, Let h be the hidden layer state at time t. t-1 Let W be the hidden layer state at time t-1. h W is the hidden layer weight matrix. x Let T be the input layer weight matrix. t Let b be the texture descriptor at time t. h For hidden layer bias terms;
[0030] If the continuous pattern of time-series changes meets the preset threshold range, the continuous pattern is further segmented to obtain the segmented trend data.
[0031] By extracting key node information of dynamic patterns from the segmented trend data, the periodicity of dynamic patterns can be determined.
[0032] Based on the periodic characteristics of dynamic patterns, a vector representation model is constructed to obtain the dynamic pattern vector of the image.
[0033] Optionally, extracting numerical sequences and anomaly markers from laboratory indicators based on the image dynamic pattern vector to obtain a potential feature set for laboratory tests includes:
[0034] Based on the aforementioned image dynamic pattern vector, determine the initial vector dataset;
[0035] Based on the initial vector dataset, the corresponding numerical sequences are extracted from the test indicators, and a standardized sequence set is obtained by applying standardized data processing methods.
[0036] Based on the normalized sequence set, the abnormal labeling information is obtained. If the abnormal labeling information exceeds a preset threshold, the corresponding sequence is labeled to identify an abnormal sequence subset.
[0037] By using the correspondence between subsets of abnormal sequences and image dynamic pattern vectors, a similarity calculation method is employed to determine the matching degree distribution between the two.
[0038] Based on the matching degree distribution, potential features are grouped to obtain preliminary feature grouping results.
[0039] Based on the preliminary feature grouping results and the dynamic vector characteristics of the images, the support vector machine algorithm is used to optimize the feature grouping to obtain the potential feature group for the test.
[0040] Optionally, based on the comprehensive health profiling framework, the image dynamic pattern vector and laboratory latent feature groups are integrated, and a weighted average method is used to calculate the overall similarity score, resulting in patient status assessment indicators including:
[0041] Based on the obtained image dynamic pattern vector and the potential feature group of the test, feature fusion technology is used to integrate the correlation information between image dynamics and test features to obtain the fused feature set.
[0042] Based on the fused feature set, a weighted average method is applied to calculate the contribution of each feature to the overall similarity and obtain the overall similarity score.
[0043] If the overall similarity metric score exceeds the preset threshold range, the patient status is classified and processed, and the priority category of status assessment is determined by combining the comprehensive framework information.
[0044] Based on the priority category of the status assessment, the corresponding assessment indicator weights are obtained, and the patient status assessment indicators are determined using mapping rules.
[0045] Optionally, matching treatment pathway templates are retrieved from the decision support database using the patient status assessment indicators. Conditional branching logic is then used to compare the deviations between the indicators and the templates to determine the optimized decision sequence, which includes:
[0046] The patient status assessment indicators were initially cleaned and formatted to obtain a structured indicator dataset;
[0047] Based on the structured indicator dataset, patient status is classified to determine the classification results. The classification uses the K-means clustering algorithm, and the cluster center updates include:
[0048] ;
[0049] in, Let n be the j-th cluster center after the (t+1)-th iteration. j For the j-th cluster The number of samples in the cluster, where x is the index data sample in the cluster;
[0050] If the classification result meets the preset threshold range, the corresponding treatment path template is retrieved from the database to obtain the matching path template data;
[0051] The matching path template data is compared with the indicator dataset using conditional branching logic to determine whether there is an indicator deviation.
[0052] If there is a deviation in the indicators, a logical adjustment mechanism is used to correct the path template data, resulting in an adjusted decision sequence. The correction method is as follows:
[0053] ;
[0054] in, These are the corrected template indicator values. For correction factor, This represents the combined deviation between the indicator and the template.
[0055] The present invention also provides a virtual patient generation system based on a multimodal large model, comprising: a medical history association feature set module, a pattern vector acquisition module, a latent feature group acquisition module, a patient status assessment module, a decision sequence module, and a virtual patient generation module;
[0056] The medical history association feature set module is used to extract keyword sequences and semantic vectors from the patient's medical history text, and to use a convolutional neural network to process the keyword sequences and semantic vectors to capture local patterns and obtain a medical history association feature set.
[0057] The regularity vector acquisition module is used to obtain the texture descriptor of the corresponding region from the image results based on the medical history association feature set, and to determine the dynamic regularity vector of the image by analyzing the temporal changes of the texture descriptor through a recurrent neural network.
[0058] The latent feature group acquisition module is used to extract numerical sequences and abnormal markers from the test indicators based on the image dynamic pattern vector to obtain the test latent feature group.
[0059] The patient status assessment module is used to integrate the image dynamic pattern vector and the laboratory potential feature group according to the comprehensive health profile framework, and calculate the overall similarity score using a weighted average method to obtain the patient status assessment index.
[0060] The decision sequence module is used to retrieve a matching treatment path template from the decision support database using the patient status assessment index, and to determine the optimized decision sequence by comparing the deviation between the index and the template using conditional branching logic.
[0061] The virtual patient generation module is used to generate a feedback report based on the optimized decision sequence, and to incorporate newly acquired dynamic patterns and potential features through an iterative update mechanism to obtain the virtual patient generation result.
[0062] Compared with the prior art, the present invention has the following advantages and technical effects:
[0063] This invention addresses the challenge of integrating and analyzing multi-source heterogeneous data, including patient medical history, imaging results, and laboratory indicators. Through feature extraction, dynamic pattern capture, and similarity calculation, it constructs a logical association analysis process from medical history-related features to dynamic pattern vectors in imaging, and then to potential feature groups in laboratory tests. First, it uses convolutional neural networks and recurrent neural networks to capture local patterns and temporal changes in medical history and imaging data. Then, combining anomaly markers and preset thresholds for laboratory indicators, it fuses multi-dimensional features to form a comprehensive health profile framework. A weighted average method is then used to calculate patient status assessment indicators, ultimately generating an optimized decision sequence and feedback report. This invention continuously incorporates new data through an iterative update mechanism, ensuring the real-time nature and accuracy of the analysis results. Its core lies in the deep fusion and dynamic adjustment of multimodal data, significantly improving the accuracy and personalization of medical decisions and providing an innovative solution for the comprehensive assessment of complex diseases. Attached Figure Description
[0064] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:
[0065] Figure 1This is a flowchart of a virtual patient generation method based on a multimodal large model according to an embodiment of the present invention. Detailed Implementation
[0066] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.
[0067] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.
[0068] This embodiment proposes a method for generating virtual patients based on a multimodal large model, such as... Figure 1 As shown, the specific steps include:
[0069] Keyword sequences and semantic vectors are extracted from patient medical history texts. Convolutional neural networks are used to process the keyword sequences and semantic vectors to capture local patterns and obtain a set of medical history-related features.
[0070] Based on the medical history-related feature set, the texture descriptor of the corresponding region is obtained from the image results. The temporal changes of the texture descriptor are analyzed by recurrent neural network to determine the dynamic law vector of the image.
[0071] Based on the dynamic pattern vector of images, numerical sequences and abnormal markers are extracted from laboratory indicators to obtain potential feature groups for laboratory tests.
[0072] Based on the integrated health profiling framework, the dynamic pattern vector of images and the potential feature group of laboratory tests are integrated, and the overall similarity score is calculated using a weighted average method to obtain the patient status assessment index.
[0073] By retrieving matching treatment pathway templates from the decision support database using patient status assessment indicators, and comparing the deviations between the indicators and the templates using conditional branching logic, the optimized decision sequence is determined.
[0074] Feedback reports are generated based on the optimized decision sequence, and newly acquired dynamic patterns and potential features are incorporated through an iterative update mechanism to obtain virtual patient generation results.
[0075] Furthermore, obtaining the medical history-related feature set includes:
[0076] Keyword sequences and semantic vectors are extracted from medical history texts using a pre-defined feature extraction model to complete the initial data extraction and obtain an initial text representation. The semantic vectors are calculated using a word embedding model.
[0077] ;
[0078] in, Let L be the semantic vector of the medical history text, and L be the number of keywords in the text. Let be the word embedding vector of the i-th keyword;
[0079] Based on the initial text representation, a convolutional neural network is used to process the keyword sequence and semantic vector, capturing local patterns and determining the association of key information in the medical history text. The convolution operation is as follows:
[0080] ;
[0081] in, For the j-th convolutional feature, The convolution kernel vector, Let be a local vector consisting of the j-th to j+l-1-th elements in the input sequence, where l is the length of the convolution kernel. The bias term is ReLU, and the activation function is ReLU.
[0082] Based on the association of key information in the medical history text, a feature set associated with the medical history is generated.
[0083] Specifically, when processing medical history texts, the first step is to extract keyword sequences and semantic vectors from the text using a pre-defined feature extraction model. Assuming the medical history text is "The patient has a long-term cough, accompanied by chest pain, and has experienced significant recent weight loss," the feature extraction model will identify keywords such as "cough," "chest pain," and "weight loss," and generate corresponding semantic vector representations. These vectors, generated through a pre-trained language model, can capture the deep semantic information of the text, laying the foundation for subsequent analysis. This approach reduces manual intervention and improves efficiency in the initial data extraction. When processing based on the initial text representation, a convolutional neural network is used to capture local patterns. For the aforementioned keyword sequence, the convolutional operation identifies the association between "cough" and "chest pain," determining that it may point to a respiratory system problem. This capture of local patterns helps extract key information associations, providing a basis for subsequent feature integration. Its beneficial effect is that it can quickly focus on the core issues in the medical history. When integrating features based on local patterns, a feature set associated with the medical history is generated. Suppose that through integration, the system discovers that "weight loss" and "cough" might indicate a potential serious disease such as lung cancer. The system will structure these features into a set, including information such as symptoms and duration. This structured representation facilitates subsequent analysis and significantly improves data readability and applicability. When validating the structured information, if "duration" data is missing, the system will determine its completeness using a preset threshold. For example, if more than 30% of key fields are missing, it is marked as incomplete and a supplementation mechanism is triggered. This validation ensures data quality, avoids misjudgments due to missing information, and improves the reliability of the analysis. When performing contextual matching with patient information, assuming the patient is a 65-year-old male with a smoking history, the system will associate the feature data with the background, speculating that "cough" and "chest pain" may be related to chronic obstructive pulmonary disease (COPD). This personalized mapping improves the specificity of the features, supporting accurate diagnosis. In the classification model's grouping processing, the system may categorize the above feature set into "respiratory symptoms" and "systemic symptoms," and determine its applicability in acute or chronic medical history scenarios. The categorized categories help clarify the focus of the medical history, improving the accuracy of subsequent analysis. When generating the final medical history-related feature set, the system integrates all information to form a comprehensive feature set including symptoms, background, and classification, which is used to support diagnosis or research. This in-depth analysis lays a solid foundation for subsequent applications and significantly improves the practical value of medical history data.
[0084] Furthermore, obtaining texture descriptors for corresponding regions from image results based on a medical history-related feature set includes:
[0085] By extracting pixel distribution information of the corresponding region from the image data, and using a pre-established segmentation model to divide the image data into regions, preliminary regional range data is obtained.
[0086] Based on the preliminary area range data, texture description information within the corresponding area is obtained. The texture description information is then normalized using a standardization method to determine the texture descriptor. The normalization process includes: ;
[0087] in, Here is the normalized texture descriptor, and T is the original texture feature value. min T is the minimum value of this texture feature. max This represents the maximum value of the texture feature.
[0088] Based on the preliminary regional range data, the texture description information within the corresponding region is obtained, and the texture description information is normalized using a standardization processing method to determine the texture descriptor.
[0089] Furthermore, by analyzing the temporal changes of texture descriptors using a recurrent neural network, the dynamic law vector of the image is determined, including:
[0090] A recurrent neural network (RNN) is used to model and analyze the temporal changes of texture descriptors, identifying the continuous patterns of these changes. The hidden layer state updates of the RNN include:
[0091] ;
[0092] in, Let h be the hidden layer state at time t. t-1 Let W be the hidden layer state at time t-1. h W is the hidden layer weight matrix. x Let T be the input layer weight matrix. t Let b be the texture descriptor at time t. h For hidden layer bias terms;
[0093] If the continuous pattern of time-series changes meets the preset threshold range, the continuous pattern is further segmented to obtain the segmented trend data.
[0094] By extracting key node information of dynamic patterns from the segmented trend data, the periodicity of dynamic patterns can be determined.
[0095] Based on the periodic characteristics of dynamic patterns, a vector representation model is constructed to obtain the dynamic pattern vector of the image.
[0096] Specifically, when processing medical imaging data, assuming the target is brain MRI images, the goal is to analyze the dynamic changes in specific brain regions. First, by extracting pixel distribution information, we can focus on areas with significant grayscale value distribution in the image, such as the hippocampus. Using a pre-trained segmentation model, the image is divided into several preliminary regions, obtaining approximate outline data of the hippocampus, assuming its pixel range covers 10% of the image. Specifically, after obtaining the preliminary region data, texture description information within these regions is extracted, such as grayscale co-occurrence matrix features, reflecting the roughness or smoothness of pixels within the region. Next, a standardization method is used to normalize the texture feature values to the range of 0 to 1; for example, the original feature values 100 and 200 are normalized to 0.5 and 1.0 respectively, ensuring the comparability of features from different image data. This results in a more consistent texture feature set, which is helpful for subsequent analysis. For the processed texture feature set, a recurrent neural network is used to model the temporal changes of the feature set. Suppose image data is collected at 10 consecutive time points. The texture feature values at each time point form a sequence. A recurrent neural network can capture the trend of the feature values gradually increasing from 0.5 to 0.8, thus determining the continuous pattern of temporal change. If this pattern meets a preset threshold range, such as a change amplitude of less than 0.3, further segmentation is performed, dividing the 10 time points into 3 segments and analyzing the change trend data of each segment separately. This segmentation process can reveal local dynamic patterns in greater detail and improve the accuracy of analysis. Specifically, key node information is extracted from the segmented change trend data. For example, a sudden change in the feature value to 0.8 at the 3rd time point may be a turning point in the dynamic pattern. Combined with the stable changes at other time points, the periodic feature is determined to be a cycle of 5 time points. Based on this periodic feature, a vector representation model is constructed, describing the dynamic change as a multi-dimensional vector, such as a feature description containing a cycle length of 5 and a change amplitude of 0.3. This vector description intuitively reflects the dynamic pattern of the image data, which is helpful for subsequent diagnosis or prediction.
[0097] Furthermore, based on the dynamic pattern vector of the images, numerical sequences and anomaly markers are extracted from the laboratory indicators to obtain the potential feature groups of the laboratory tests, including:
[0098] Determine the initial vector dataset based on the dynamic law vector of the image;
[0099] Based on the initial vector dataset, the corresponding numerical sequences are extracted from the test indicators, and a normalized sequence set is obtained by using standardized data processing methods.
[0100] Based on the normalized sequence set, the abnormal labeling information is obtained. If the abnormal labeling information exceeds the preset threshold, the corresponding sequence is labeled to identify the abnormal sequence subset.
[0101] By using the correspondence between subsets of abnormal sequences and image dynamic pattern vectors, a similarity calculation method is employed to determine the matching degree distribution between the two.
[0102] Based on the matching degree distribution, potential features are grouped to obtain preliminary feature grouping results.
[0103] Based on the preliminary feature grouping results and combined with the dynamic vector characteristics of the images, the support vector machine algorithm is used to optimize the feature grouping to obtain the potential feature groups for analysis.
[0104] Specifically, in the field of image dynamics analysis, the step of extracting regular vector data from image dynamics can be achieved through feature extraction from a series of continuous image frames. Assuming the focus is on the dynamic changes of a specific region, the system first preprocesses each image frame to extract the grayscale distribution information of key areas, forming an initial vector representation. This vector data reflects the changing trend of the image over time; for example, in 100 consecutive images, the grayscale value of a certain region gradually changes from 50 to 80, exhibiting a linear growth pattern. This process lays the foundation for subsequent mapping rules. When determining the initial vector dataset using pre-established mapping rules, the regular vectors in the image dynamics can be compared with preset feature templates. Assuming the mapping rules define a specific vector label corresponding to a certain change pattern, such as periodic fluctuations corresponding to label A, the system will classify the image region into a dataset under label A based on the frequency of grayscale value changes. This dataset may contain multiple vectors; for example, vector 1 represents a fluctuation period of 5 seconds, and vector 2 represents a fluctuation amplitude of 20 grayscale units, providing diverse data support for subsequent analysis. Regarding the step of extracting and standardizing numerical sequences from laboratory indicators, we can envision extracting continuous values for a certain indicator from blood test data. For example, a patient's indicator values over 7 days might be 3.2, 3.5, 3.8, 4.0, 4.3, 4.5, and 4.7. After standardization, these values are adjusted to the range of 0 to 1, forming a standardized sequence set, facilitating subsequent anomaly detection. This process ensures that data with different dimensions can be compared uniformly. When detecting anomaly markers and labeling anomalous sequence subsets, assuming a preset threshold of 0.8, if a value in the standardized sequence reaches 0.9, the system will label it as an anomaly and further analyze whether its subset has continuity. For example, if data from three consecutive days exceeds the threshold, it is classified as an anomaly subset. This labeling helps focus on potential problem areas. When determining the matching degree distribution between the anomalous sequence subset and the regular vector using similarity calculation methods, cosine similarity can be used as a metric. Assuming the feature vector of the outlier subset is [0.5, 0.3, 0.2], and the regularity vector is [0.6, 0.4, 0.1], the system calculates the similarity score between the two and generates a distribution map showing that the matching degree is concentrated in a certain interval. This distribution provides a basis for subsequent grouping. For the grouping of potential features, features can be divided into high-relevance groups and low-relevance groups based on the matching degree distribution. Assuming the high-relevance group contains features with a matching degree greater than 0.7, these features may be closely related to periodic changes in the dynamics of the image. The preliminary grouping results lay the foundation for subsequent optimization.
[0105] Furthermore, based on the comprehensive health profiling framework, the dynamic pattern vector of images and the latent feature groups of laboratory tests are integrated, and a weighted average method is used to calculate the overall similarity score, resulting in patient status assessment indicators including:
[0106] Based on the obtained image dynamic pattern vector and the potential feature group of the test, feature fusion technology is used to integrate the correlation information between image dynamics and test features to obtain the fused feature set.
[0107] Based on the fused feature set, a weighted average method is applied to calculate the contribution of each feature to the overall similarity and obtain the overall similarity score.
[0108] If the overall similarity metric score exceeds the preset threshold range, the patient status is classified and processed, and the priority category of status assessment is determined by combining the comprehensive framework information.
[0109] Based on the priority category of the status assessment, the corresponding assessment indicator weights are obtained, and the patient status assessment indicators are determined by using mapping rules.
[0110] Specifically, data is extracted from the latent feature set, and preliminary screening is performed based on the abnormal marker values to obtain a preliminary abnormal data set. If the abnormal marker values in the preliminary abnormal data set exceed a preset threshold, a comparison tool is used to classify and label the abnormal data to determine the abnormal data category. Based on the distribution of abnormal data categories, relevant data in the medical history feature set is obtained, and preliminary matching is performed using associated feature items to determine the matching consistency result. If the matching consistency result meets the preset standard, a fusion algorithm is used to integrate the latent feature set and the medical history feature set to obtain an integrated feature dataset. For the integrated feature dataset, a secondary verification is performed using feature matching degree to determine the preliminary framework structure of the comprehensive health map. Based on the preliminary framework structure of the comprehensive health map, a portrait construction method is used to refine and adjust the health portrait frame to obtain the final health portrait framework.
[0111] More specifically, in the process of processing health profile data, acquiring dynamic image information and laboratory feature data is a crucial first step. Dynamic image information can be understood as data on the dynamic changes of a patient's body parts captured by medical imaging equipment, such as ultrasound images of the heart beating, while laboratory feature data includes various indicators from blood tests, such as blood glucose levels or white blood cell counts. Suppose that in a single processing step, dynamic image information shows abnormal cardiac contractile function, while laboratory data shows elevated blood lipid levels. Combining these two data points initially forms a pattern vector pointing to potential cardiovascular problems, with the potential feature content suggesting a risk of insufficient blood supply to the heart. For feature fusion technology, integrating the correlation information between dynamic image information and laboratory feature data is a complex process. Suppose that imaging data indicates weakened activity in a certain area of the heart, while laboratory data shows a cholesterol level of 5.5 mmol / L, significantly higher than the normal range. Fusion technology can correlate these two, leading to a preliminary conclusion that there may be a problem with blood supply to the heart. The fused feature set integrates this information into a unified feature group, facilitating subsequent analysis. When applying a weighted average method to calculate the contribution of each feature, weights can be assigned based on the reliability of the dynamic image information and laboratory feature data. Assuming an imaging data weight of 0.6 and a laboratory data weight of 0.4, the weighted similarity score is 85, exceeding the preset threshold of 75. This indicates a potentially high-risk patient condition requiring further classification. This method more objectively reflects the importance of each data point. For patient condition classification and priority assessment, assuming a high similarity score, and combining this with the comprehensive framework information, the patient condition is categorized as high-risk, with a priority assessment of Level 1. At this point, cardiovascular-related indicators will be the primary focus, and further examinations will be prioritized. This classification method helps to quickly identify key issues.
[0112] Furthermore, matching treatment pathway templates are retrieved from the decision support database using patient status assessment indicators. Conditional branching logic is then used to compare the deviations between the indicators and the templates to determine the optimized decision sequence, which includes:
[0113] The patient status assessment indicators were initially cleaned and formatted to obtain a structured indicator dataset;
[0114] Based on the structured indicator dataset, patient status is classified to determine the classification results. The classification uses the K-means clustering algorithm, and the cluster center updates include:
[0115] ;
[0116] in, Let n be the j-th cluster center after the (t+1)-th iteration. j For the j-th cluster The number of samples in the cluster, where x is the index data sample in the cluster;
[0117] Based on the structured indicator dataset, the patient status is classified and the classification result of the patient status is determined.
[0118] If the classification result meets the preset threshold range, the corresponding treatment path template is retrieved from the database to obtain the matching path template data;
[0119] The matching path template data is compared with the indicator dataset through conditional branching logic to determine whether there is any indicator deviation.
[0120] If there is a deviation in the indicators, a logical adjustment mechanism is used to correct the path template data, resulting in an adjusted decision sequence. The correction method is as follows:
[0121] ;
[0122] in, These are the corrected template indicator values. For correction factor, This represents the combined deviation between the indicator and the template.
[0123] Specifically, when retrieving patient status-related assessment indicator data from the decision support database, indicators relevant to the patient's current health profile, such as key values like blood pressure and blood sugar, can be filtered out first through the data interface. For example, if a patient's blood pressure is 135 / 85 mmHg and their blood sugar is 6.5 mmol / L, these data will be initially recorded and entered into the cleaning process. During cleaning, some data may be found to have inconsistent formats or missing values. In such cases, preset rules will be used to complete or remove data to ensure the accuracy of subsequent analysis. When grading structured indicator datasets, the classification model may categorize patient status into mild, moderate, and severe levels based on historical data and expert rules. For example, if a patient's data shows elevated blood pressure but not reaching a critical level, the model may classify them as moderate risk. This classification result will directly influence the selection of subsequent treatment pathways, providing a basis for personalized treatment plans. When retrieving treatment pathway templates, if the classification result is moderate risk, the system will extract template data suitable for moderate-risk patients from the database, such as recommended medication dosages and follow-up frequency. Assuming the template specifies weekly blood pressure checks, if the patient's data falls within this range, the template is applied directly. This method quickly matches a suitable treatment plan, reducing manual intervention. When comparing the indicator dataset with the template data using conditional branching logic, it may reveal significant fluctuations in the patient's actual blood pressure, deviating from the template's suggested stable control target. The system records this difference and triggers an adjustment mechanism. This comparison promptly identifies potential problems, ensuring the applicability of the treatment path. After detecting an indicator deviation, the logical adjustment mechanism may adjust the check-up frequency from once a week to once every three days, based on the patient's specific situation, to more closely monitor blood pressure changes. The adjusted treatment path will better suit the patient's needs, preventing further risk escalation. When generating the optimized decision sequence, the system translates the adjusted path into specific execution steps, such as medication administration time and check-up arrangements, and outputs them in a clear order. This process ensures that healthcare professionals can quickly understand and execute the decisions.
[0124] This embodiment also provides a virtual patient generation system based on a multimodal large model, including: a medical history association feature set module, a pattern vector acquisition module, a latent feature group acquisition module, a patient status assessment module, a decision sequence module, and a virtual patient generation module;
[0125] The medical history association feature set module is used to extract keyword sequences and semantic vectors from patient medical history texts. It uses a convolutional neural network to process the keyword sequences and semantic vectors to capture local patterns and obtain the medical history association feature set.
[0126] The regularity vector acquisition module is used to obtain the texture descriptor of the corresponding region from the image results based on the medical history-related feature set, and to determine the dynamic regularity vector of the image by analyzing the temporal changes of the texture descriptor through a recurrent neural network.
[0127] The latent feature group acquisition module is used to extract numerical sequences and abnormal markers from laboratory indicators based on the dynamic pattern vector of images, and to obtain the latent feature group of the laboratory test.
[0128] The patient status assessment module integrates image dynamic pattern vectors and laboratory potential feature groups based on the comprehensive health profile framework, and calculates the overall similarity score using a weighted average method to obtain the patient status assessment index.
[0129] The decision sequence module is used to retrieve matching treatment path templates from the decision support database through patient status assessment indicators, and use conditional branching logic to compare the deviation between the indicators and the templates to determine the optimized decision sequence.
[0130] The virtual patient generation module is used to generate feedback reports based on the optimized decision sequence. It adopts an iterative update mechanism to incorporate newly acquired dynamic patterns and potential features to obtain virtual patient generation results.
[0131] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
Claims
1. A method for generating virtual patients based on a multimodal large model, characterized in that, include: Keyword sequences and semantic vectors are extracted from the patient's medical history text. Convolutional neural networks are used to process the keyword sequences and semantic vectors to capture local patterns and obtain a set of medical history-related features. Based on the medical history-related feature set, the texture descriptor of the corresponding region is obtained from the image results. The temporal changes of the texture descriptor are analyzed by a recurrent neural network to determine the dynamic law vector of the image. Based on the image dynamic pattern vector, numerical sequences and abnormal markers are extracted from the test indicators to obtain the potential feature group of the test. Based on the integrated health profiling framework, the dynamic pattern vector of the images and the potential feature groups of the laboratory tests are integrated, and the overall similarity score is calculated using a weighted average method to obtain the patient status assessment index. The patient status assessment indicators are used to retrieve matching treatment path templates from the decision support database. Conditional branching logic is used to compare the deviation between the indicators and the templates to determine the optimized decision sequence. Feedback reports are generated based on the optimized decision sequence, and newly acquired dynamic patterns and potential features are incorporated through an iterative update mechanism to obtain virtual patient generation results.
2. The method for generating virtual patients based on a multimodal large model according to claim 1, characterized in that, Obtaining the medical history-related feature set includes: Keyword sequences and semantic vectors are extracted from medical history texts using a pre-defined feature extraction model to complete the initial data extraction and obtain an initial text representation. The semantic vectors are calculated using a word embedding model. ； in, Let L be the semantic vector of the medical history text, and L be the number of keywords in the text. Let be the word embedding vector of the i-th keyword; Based on the initial text representation, a convolutional neural network is used to process the keyword sequence and semantic vector, capturing local patterns and determining the association of key information in the medical history text. The convolution operation is as follows: ； in, For the j-th convolutional feature, The convolution kernel vector, Let be a local vector consisting of the j-th to j+l-1-th elements in the input sequence, where l is the length of the convolution kernel. The bias term is ReLU, and the activation function is ReLU. Based on the key information in the medical history text, a feature set associated with the medical history is generated.
3. The method for generating virtual patients based on a multimodal large model according to claim 1, characterized in that, The texture descriptor of the corresponding region obtained from the image results based on the medical history-related feature set includes: By extracting pixel distribution information of the corresponding region from the image data, and using a pre-established segmentation model to divide the image data into regions, preliminary regional range data is obtained. Based on the preliminary area range data, texture description information within the corresponding area is obtained. The texture description information is then normalized using a standardization method to determine the texture descriptor. The normalization process includes: ; in, Here is the normalized texture descriptor, and T is the original texture feature value. min T is the minimum value of this texture feature. max This represents the maximum value of the texture feature.
4. The method for generating virtual patients based on a multimodal large model according to claim 1, characterized in that, By analyzing the temporal changes of texture descriptors using recurrent neural networks, the dynamic law vectors of images are determined, including: A recurrent neural network (RNN) is used to model and analyze the temporal changes of texture descriptors, identifying the continuous patterns of these changes. The hidden layer state updates of the RNN include: ； in, Let h be the hidden layer state at time t. t-1 Let W be the hidden layer state at time t-1. h W is the hidden layer weight matrix. x Let T be the input layer weight matrix. t Let b be the texture descriptor at time t. h For hidden layer bias terms; If the continuous pattern of time-series changes meets the preset threshold range, the continuous pattern is further segmented to obtain the segmented trend data. By extracting key node information of dynamic patterns from the segmented trend data, the periodicity of dynamic patterns can be determined. Based on the periodic characteristics of dynamic patterns, a vector representation model is constructed to obtain the dynamic pattern vector of the image.
5. The method for generating virtual patients based on a multimodal large model according to claim 1, characterized in that, Based on the image dynamic pattern vector, numerical sequences and anomaly markers are extracted from laboratory indicators to obtain potential feature groups for laboratory testing, including: Based on the aforementioned image dynamic pattern vector, determine the initial vector dataset; Based on the initial vector dataset, the corresponding numerical sequences are extracted from the test indicators, and a standardized sequence set is obtained by applying standardized data processing methods. Based on the normalized sequence set, the abnormal labeling information is obtained. If the abnormal labeling information exceeds a preset threshold, the corresponding sequence is labeled to identify an abnormal sequence subset. By using the correspondence between subsets of abnormal sequences and image dynamic pattern vectors, a similarity calculation method is employed to determine the matching degree distribution between the two. Based on the matching degree distribution, potential features are grouped to obtain preliminary feature grouping results. Based on the preliminary feature grouping results and the dynamic vector characteristics of the images, the support vector machine algorithm is used to optimize the feature grouping to obtain the potential feature group for the test.
6. The method for generating virtual patients based on a multimodal large model according to claim 1, characterized in that, Based on the comprehensive health profiling framework, the dynamic pattern vector of the images and the latent feature groups of the laboratory tests are integrated, and a weighted average method is used to calculate the overall similarity score, resulting in patient status assessment indicators including: Based on the obtained image dynamic pattern vector and the potential feature group of the test, feature fusion technology is used to integrate the correlation information between image dynamics and test features to obtain the fused feature set. Based on the fused feature set, a weighted average method is applied to calculate the contribution of each feature to the overall similarity and obtain the overall similarity score. If the overall similarity metric score exceeds the preset threshold range, the patient status is classified and processed, and the priority category of status assessment is determined by combining the comprehensive framework information. Based on the priority category of the status assessment, the corresponding assessment indicator weights are obtained, and the patient status assessment indicators are determined using mapping rules.
7. The method for generating virtual patients based on a multimodal large model according to claim 1, characterized in that, The patient status assessment indicators are used to retrieve matching treatment pathway templates from the decision support database. Conditional branching logic is then used to compare the deviations between the indicators and the templates to determine the optimized decision sequence, which includes: The patient status assessment indicators were initially cleaned and formatted to obtain a structured indicator dataset; Based on the structured indicator dataset, patient status is classified to determine the classification results. The classification uses the K-means clustering algorithm, and the cluster center updates include: ； in, Let n be the j-th cluster center after the (t+1)-th iteration. j For the j-th cluster The number of samples in the cluster, where x is the index data sample in the cluster; If the classification result meets the preset threshold range, the corresponding treatment path template is retrieved from the database to obtain the matching path template data; The matching path template data is compared with the indicator dataset using conditional branching logic to determine whether there is an indicator deviation. If there is a deviation in the indicators, a logical adjustment mechanism is used to correct the path template data, resulting in an adjusted decision sequence. The correction method is as follows: ； in, These are the corrected template indicator values. For correction factor, This represents the combined deviation between the indicator and the template.
8. The system implementing the method according to any one of claims 1-7, characterized in that, include: The system includes a medical history-related feature set module, a pattern vector acquisition module, a latent feature group acquisition module, a patient status assessment module, a decision sequence module, and a virtual patient generation module. The medical history association feature set module is used to extract keyword sequences and semantic vectors from the patient's medical history text, and to use a convolutional neural network to process the keyword sequences and semantic vectors to capture local patterns and obtain a medical history association feature set. The regularity vector acquisition module is used to obtain the texture descriptor of the corresponding region from the image results based on the medical history association feature set, and to determine the dynamic regularity vector of the image by analyzing the temporal changes of the texture descriptor through a recurrent neural network. The latent feature group acquisition module is used to extract numerical sequences and abnormal markers from the test indicators based on the image dynamic pattern vector to obtain the test latent feature group. The patient status assessment module is used to integrate the image dynamic pattern vector and the laboratory potential feature group according to the comprehensive health profile framework, and calculate the overall similarity score using a weighted average method to obtain the patient status assessment index. The decision sequence module is used to retrieve a matching treatment path template from the decision support database using the patient status assessment index, and to determine the optimized decision sequence by comparing the deviation between the index and the template using conditional branching logic. The virtual patient generation module is used to generate a feedback report based on the optimized decision sequence, and to incorporate newly acquired dynamic patterns and potential features through an iterative update mechanism to obtain the virtual patient generation result.