A medical skill assessment and training method and device based on virtual patients

Abstract

The present application provides a method and device for medical skill assessment and training based on virtual patients, relating to the field of digital computing technology. The method includes: obtaining image data, pathological data, and physiological signal data corresponding to historical patients to generate a three-dimensional virtual patient model; performing multi-scale feature encoding on the operation trajectory, instrument contact pressure, and biological tissue deformation parameters corresponding to the historical operation data of the target user to generate an operation feature vector with time-series correlation; inputting the operation feature vector into the three-dimensional virtual patient model to obtain a physiological parameter change sequence corresponding to the operation feature vector; jointly analyzing the operation feature vector and the physiological parameter change sequence to obtain an evaluation index matrix; generating an adaptive training scenario according to the evaluation index matrix; the adaptive training scenario includes complication simulation information, abnormal sign trigger information, and emergency state intervention information, so as to train the target user according to the mixed reality interaction device.

Description

Technical Field

[0001] This application relates to the field of digital computing technology, and in particular, to a method and device for medical skill assessment and training based on virtual patients. Background Art

[0002] In the field of medical education and skill training, traditional teaching methods mainly rely on real patients or cadaver models for practical operations. These methods have many limitations, such as high costs, limited resources, and the inability to provide opportunities for personalized and repeated training. With the development of computer technology and artificial intelligence, the method for medical skill assessment and training based on virtual patients has emerged, aiming to provide a more efficient, economical, and safe learning and training environment.

[0003] Traditional virtual patient training systems mostly focus on simulating the physiological responses of patients, but often lack precise feedback and evaluation mechanisms related to real operations. These systems cannot generate personalized training scenarios based on the operation data of users. Summary of the Invention

[0004] This application provides a method and device for medical skill assessment and training based on virtual patients, aiming to solve the problem that with the digitalization and onlineization of financial services, high-concurrency traffic attacks (such as DDoS attacks, repeated requests caused by malicious plugins, etc.) have become an increasingly serious problem.

[0005] In a first aspect, this application provides a device for medical skill assessment and training based on virtual patients, including:

[0006] A data acquisition module, configured to acquire image data, pathological data, and physiological signal data corresponding to historical patients, and generate a virtual patient three-dimensional model according to the image data, pathological data, and physiological signal data; the virtual patient three-dimensional model has anatomical accuracy and physiological dynamic response characteristics;

[0007] An operation acquisition module, configured to acquire historical operation data of a target user, perform multi-scale feature encoding on operation trajectories, instrument contact pressures, and biological tissue deformation parameters corresponding to the historical operation data, and generate an operation feature vector with time series correlation;

[0008] A vector input module, configured to input the operation feature vector into the virtual patient three-dimensional model, and acquire a physiological parameter change sequence corresponding to the operation feature vector; the physiological parameter change sequence is used to determine the coupled responses of the circulatory system, respiratory system, and nervous system corresponding to the operation feature vector;

[0009] An evaluation acquisition module for jointly analyzing the operation feature vector and the physiological parameter change sequence to obtain an evaluation index matrix; the evaluation index matrix at least includes an operation standardization score, a clinical decision rationality score, and an emergency response ability score;

[0010] A training completion module for generating an adaptive training scenario according to the evaluation index matrix; the adaptive training scenario includes complication simulation information, abnormal sign trigger information, and emergency state intervention information to complete the training of the target user according to a preset mixed reality interaction device and the adaptive training scenario.

[0011] In some embodiments, generating the virtual patient three-dimensional model based on the imaging data, pathological data, and physiological signal data includes: performing cross-modal alignment on the imaging data, pathological data, and physiological signal data; constructing a dynamic volume rendering framework based on a neural radiance field, and generating tissue texture information corresponding to the imaging data, pathological data, and physiological signal data under the dynamic volume rendering framework; according to a physiological simulation algorithm based on finite element analysis, establishing a blood vessel wall stress-strain model and a dynamic response mechanism corresponding to the imaging data, pathological data, and physiological signal data according to the tissue texture information; generating the virtual patient three-dimensional model according to the blood vessel wall stress-strain model and the dynamic response mechanism.

[0012] Exemplarily, performing cross-modal alignment on the imaging data, pathological data, and physiological signal data includes: performing multi-scale feature extraction on the imaging data according to a preset 3D Swin Transformer encoder to obtain imaging feature information; performing temporal modeling on the pathological data according to a preset Bi-LSTM encoder to obtain pathological feature information; decoupling the time-frequency domain features of the physiological signal through a wavelet transform kernel to obtain physiological feature information; completing cross-modal alignment of the imaging data, pathological data, and physiological signal data according to the imaging feature information, pathological feature information, and physiological feature information.

[0013] Exemplarily, before generating the virtual patient three-dimensional model according to the blood vessel wall stress-strain model and the dynamic response mechanism, it further includes: verifying the authenticity of the blood vessel wall stress-strain model and the dynamic response mechanism according to a discriminator constructed based on a generative adversarial network; the discriminator includes a cyclic consistency loss function for ensuring that the phase synchronization error between the respiratory rhythm and the cardiovascular pulsation corresponding to the virtual patient three-dimensional model is less than a preset error; if the authenticity verification passes, generating the virtual patient three-dimensional model according to the blood vessel wall stress-strain model and the dynamic response mechanism.

[0014] In some embodiments, the multi-scale feature encoding of the operation trajectory, instrument contact pressure, and biological tissue deformation parameters corresponding to the historical operation data to generate an operation feature vector with time-series correlation includes: performing kinematic modeling on the operation trajectory according to a preset spatio-temporal graph convolutional network to establish a dynamic spatial relationship graph between the end effector of the instrument and the anatomical landmark points; constructing a dual-channel attention mechanism to perform multi-physical field coupling analysis on the instrument contact pressure and biological tissue deformation parameters to generate a real-time deformation thermal map of the instrument action area; generating a hierarchical Transformer encoder according to the operation feature vector corresponding to the dynamic spatial relationship graph and the real-time deformation thermal map.

[0015] In some embodiments, the inputting of the operation feature vector into the virtual patient three-dimensional model to obtain a physiological parameter change sequence corresponding to the operation feature vector includes: calculating the blood pressure waveform propagation information and simulated airway pressure gradient information corresponding to the operation feature vector in the virtual patient three-dimensional model; obtaining a causal inference chain of preset operation features and pathophysiological responses; generating the physiological parameter change sequence according to the blood pressure waveform propagation information, simulated airway pressure gradient information, operation feature vector, and the causal inference chain; the physiological parameter change sequence includes at least blood loss information and nerve reflex information.

[0016] In some embodiments, the joint analysis of the operation feature vector and the physiological parameter change sequence to obtain an evaluation index matrix includes: constructing an anomaly detection module based on a graph neural network to determine at least one mutation point in the physiological parameter change sequence; obtaining abnormal operation information corresponding to the mutation point in the operation feature vector; generating the evaluation index matrix according to the abnormal operation information.

[0017] In some embodiments, the generation of an adaptive training scenario according to the evaluation index matrix includes: determining the ability defect information of the target user according to the evaluation index matrix; quickly obtaining concurrent simulated disease information corresponding to the ability defect information according to the model-agnostic meta-learning algorithm; the complication simulation information includes a disease evolution tree; performing cross-modal enhanced rendering on the virtual patient three-dimensional model according to the concurrent simulated disease information; the cross-modal enhanced rendering includes at least converting the heart sound features corresponding to the concurrent simulated disease information into a tactile feedback matrix and mapping the abnormal blood gas analysis values corresponding to the concurrent simulated disease information into a visualized metabolic pathway disorder hot spot map; generating the adaptive training scenario according to the tactile feedback matrix, the visualized metabolic pathway disorder hot spot map, and the virtual patient three-dimensional model.

[0018] In some embodiments, completing the training of the target user according to a preset mixed reality interaction device and an adaptive training scenario includes: obtaining the eye movement trajectory information, operation force information of the target user wearing the mixed reality interaction device, and the virtual physiological parameter change information corresponding to the virtual patient three-dimensional model; dynamically adjusting the complexity of the mixed reality interaction device according to the eye movement trajectory information, operation force information, and the virtual physiological parameter change information to complete the training of the target user.

[0019] In a second aspect, the present application further provides a medical skill assessment and training method based on a virtual patient, including:

[0020] Obtaining the image data, pathological data, and physiological signal data corresponding to a historical patient, and generating a virtual patient three-dimensional model according to the image data, pathological data, and physiological signal data; the virtual patient three-dimensional model has anatomical accuracy and physiological dynamic response characteristics;

[0021] Obtaining the historical operation data of the target user, performing multi-scale feature encoding on the operation trajectory, instrument contact pressure, and biological tissue deformation parameters corresponding to the historical operation data, and generating an operation feature vector with time series correlation;

[0022] Inputting the operation feature vector into the virtual patient three-dimensional model to obtain a physiological parameter change sequence corresponding to the operation feature vector; the physiological parameter change sequence is used to determine the coupled responses of the circulatory system, respiratory system, and nervous system corresponding to the operation feature vector;

[0023] Performing joint analysis on the operation feature vector and the physiological parameter change sequence to obtain an evaluation index matrix; the evaluation index matrix at least includes an operation standardization score, a clinical decision-making rationality score, and an emergency handling ability score;

[0024] Generating an adaptive training scenario according to the evaluation index matrix; the adaptive training scenario includes complication simulation information, abnormal sign trigger information, and emergency state intervention information to train the target user according to a preset mixed reality interaction device and the adaptive training scenario.

[0025] In a third aspect, the present application further provides a computer device, which includes a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program and implement the above-mentioned medical skill assessment and training method based on a virtual patient when executing the computer program.

[0026] In a fourth aspect, the present application also provides a computer-readable storage medium storing a computer program, which when executed by a processor causes the processor to implement the above-mentioned medical skill assessment and training method based on virtual patients.

[0027] The present application discloses a medical skill assessment and training method and device based on virtual patients. The

[0028] First, relevant medical data of historical patients is collected, including imaging data (such as CT, MRI, etc.), pathological data (such as tissue section analysis), and physiological signal data (such as electrocardiogram, blood pressure, etc.) to construct a virtual patient model. Using the above data, a three-dimensional virtual patient model with anatomical accuracy and physiological dynamic response characteristics is generated. This means that the model is not only visually realistic but also can simulate real physiological reactions. Analyze the historical operation data of the target user, including operation trajectories, instrument contact pressures, and biological tissue deformation parameters, and encode these data into operation feature vectors with time-series correlation. Input the operation feature vectors into the virtual patient model to simulate the corresponding physiological parameter change sequences, which can reflect the coupled responses of the circulatory, respiratory, and nervous systems. Jointly analyze the operation feature vectors and physiological parameter change sequences to generate a matrix containing multiple evaluation indicators, such as operation standardization scores, clinical decision-making rationality scores, emergency response ability scores, etc. According to the evaluation indicator matrix, generate adaptive training scenarios, including complication simulation information, abnormal sign trigger information, and emergency state intervention information, to improve the pertinence and effectiveness of training. Use a preset mixed reality interaction device and adaptive training scenarios to complete the training of the target user and improve their medical skills.

[0029] By simulating real clinical environments and patient responses, this method can provide more realistic and personalized medical training, thereby improving the skill levels of medical students and doctors. Compared with traditional clinical internships, using virtual patient models can reduce the need for real patients, lower training costs, and potential medical risks. By simulating emergency states and complications, this method helps improve doctors' ability to handle emergencies in real clinical environments. The adaptive training scenarios generated according to the evaluation indicator matrix can provide personalized training for the specific needs and skill levels of different users. By simulating different clinical scenarios, doctors can practice and improve their clinical decision-making abilities without risk. This method can be used as a tool for medical research to help researchers test new medical technologies and treatment methods in a simulated environment.

[0030] In summary, the present application provides an innovative medical skill assessment and training method, aiming to improve the quality and efficiency of medical education, while reducing costs and risks, through virtual patient models and adaptive training scenarios.

[0031] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and do not limit this application. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions of the embodiments of this application, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings in the following description are some embodiments of this application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0033] Figure 1 is a schematic flow chart of the steps of a medical skill assessment and training method based on a virtual patient provided by an embodiment of this application;

[0034] Figure 2 is a schematic block diagram of a medical skill assessment and training device based on a virtual patient provided by an embodiment of this application;

[0035] Figure 3 is a schematic block diagram of the structure of a computer device provided by an embodiment of this application.

[0036] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and do not limit this application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0037] The following will clearly and completely describe the technical solutions in the embodiments of this application with reference to the drawings in the embodiments of this application. Obviously, the described embodiments are some, but not all, of the embodiments of this application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in this application without creative efforts fall within the scope of protection of this application.

[0038] The flow chart shown in the drawings is only an example, and does not necessarily include all contents and operations / steps, nor does it necessarily execute in the described order. For example, some operations / steps can also be decomposed, combined, or partially merged, so the actual execution order may change according to the actual situation.

[0039] It should be understood that the terms used in this specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application. As used in the specification of this application and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an", and "the" are intended to include the plural forms.

[0040] It should also be understood that the term "and / or" used in the specification and appended claims of the present application refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0041] The following will, with reference to the accompanying drawings, elaborate on some embodiments of the present application. Without conflict, the following embodiments and the features in the embodiments can be combined with each other.

[0042] In the field of medical education and skills training, traditional teaching methods mainly rely on real patients or cadaver models for practical operations. These methods have many limitations, such as high costs, limited resources, and the inability to provide opportunities for personalized and repeated training. With the development of computer technology and artificial intelligence, medical skills assessment and training methods based on virtual patients have emerged, aiming to provide a more efficient, economical, and safe learning and training environment.

[0043] Traditional virtual patient training systems mostly focus on simulating the physiological responses of patients, but often lack accurate feedback and evaluation mechanisms related to real operations. These systems cannot generate personalized training scenarios based on the operation data of users.

[0044] Therefore, there is an urgent need for a method to solve at least one of the above problems.

[0045] Please refer to Figure 1 , Figure 1 which is a schematic flowchart of the steps of a medical skills assessment and training method based on virtual patients provided by an embodiment of the present application. This method can be implemented by a computer device, and the computer device can be deployed on a single server or a server cluster. It can also be deployed on a handheld terminal, a laptop, a wearable device, or a robot, etc.

[0046] As Figure 1 shown, the medical skills assessment and training method based on virtual patients specifically includes steps S101 to S105:

[0047] Step S101, obtain the image data, pathological data, and physiological signal data corresponding to historical patients, and generate a virtual patient three-dimensional model according to the image data, pathological data, and physiological signal data; the virtual patient three-dimensional model has anatomical accuracy and physiological dynamic response characteristics.

[0048] Specifically, in step S101, the computer device not only needs to collect and process imaging, pathological, and physiological signal data, but also needs to ensure that the integration of these data can reflect the overall health status of the patient. This involves advanced data fusion techniques to integrate data from different sources and types into a unified model. For example, imaging data may need to be enhanced in detail through image processing techniques, pathological data may need to extract key information through natural language processing techniques, and physiological signal data may need to filter out noise and outliers through signal processing techniques.

[0049] For example, adopt advanced data fusion techniques such as deep learning to integrate data from different sources into a unified framework. Continuously optimize the 3D model to improve its simulation degree and response speed, making it closer to the physiological and anatomical characteristics of real patients. Develop complex algorithms to simulate the physiological responses of patients, including but not limited to the dynamic changes of the cardiovascular system, respiratory system, and nervous system.

[0050] This high-precision virtual patient model can be an important tool in medical education to help students better understand human body structures and physiological functions. The virtual patient model can also be used in medical research to simulate physiological changes under different disease states, providing new ideas for studying disease mechanisms.

[0051] Step S102: Obtain the historical operation data of the target user, perform multi-scale feature encoding on the operation trajectory, instrument contact pressure, and biological tissue deformation parameters corresponding to the historical operation data, and generate an operation feature vector with time series correlation.

[0052] Specifically, in step S102, the collection and analysis of operation data need to consider the complexity and diversity of operations. This means that we need to develop sensors and algorithms that can capture subtle operation differences. For example, in surgical operations, the way of holding the instrument, cutting depth, and speed are all key factors affecting the surgical outcome, and these all need to be captured by high-precision sensors.

[0053] Expansion of specific implementation methods:

[0054] Sensor technology: Develop or integrate high-precision sensors such as force feedback sensors, motion capture systems, etc., to capture every detail of the user's operation.

[0055] Data preprocessing: Preprocess the collected raw data, including filtering, normalization, and feature extraction, to improve the accuracy of subsequent analysis.

[0056] Feature vector construction: Construct a multi-dimensional feature vector that includes not only the physical parameters of the operation, but also the spatio-temporal parameters and physiological feedback parameters of the operation.

[0057] Expansion of beneficial effects:

[0058] Skill assessment: Through meticulous analysis of operation data, the surgical skill level of users can be more accurately evaluated, providing a basis for skill improvement.

[0059] Personalized training: Based on the operation feature vectors of users, personalized training plans can be designed to improve the pertinence and efficiency of training.

[0060] Step S103: Input the operation feature vector into the virtual patient three-dimensional model to obtain the physiological parameter change sequence corresponding to the operation feature vector; the physiological parameter change sequence is used to determine the coupling responses of the circulatory system, respiratory system, and nervous system corresponding to the operation feature vector.

[0061] Specifically, in step S103, when inputting the operation feature vector into the virtual patient model, it is necessary to ensure that the model can accurately respond to these operations. This involves complex physical engines and physiological simulation algorithms to ensure that every detail of the operation can be reflected in the model. By integrating advanced physical engines, simulate the interaction between surgical instruments and virtual patient tissues, including operations such as cutting and suturing. Develop physiological simulation algorithms to simulate the impact of surgical operations on the patient's physiological state, such as bleeding and pain responses. Build a real-time feedback system to feedback the physiological parameter changes of the model to users in real time to simulate a real surgical environment.

[0062] By simulating a real surgical environment, users can practice complex surgical operations without risk. By simulating the impact of surgical operations on the patient's physiological state, surgical risks can be evaluated, providing a reference for surgical planning.

[0063] Step S104: Conduct a joint analysis of the operation feature vector and the physiological parameter change sequence to obtain an evaluation index matrix; the evaluation index matrix includes at least an operation standardization score, a clinical decision-making rationality score, and an emergency response ability score.

[0064] Specifically, in step S104, the joint analysis needs to comprehensively consider the operation feature vector and the physiological parameter change sequence, which involves complex data analysis and machine learning techniques. For example, deep learning networks can be used to analyze these data and extract key evaluation indicators.

[0065] Adopt advanced data analysis techniques, such as cluster analysis and principal component analysis, to extract key information from a large amount of data. Develop machine learning models, such as neural networks and support vector machines, to conduct classification and regression analysis on the operation feature vector and the physiological parameter change sequence. Build a comprehensive evaluation index matrix that includes not only the evaluation of operation skills but also the evaluation of surgical risks and the patient's physiological state.

[0066] Through the comprehensive evaluation index matrix, the surgical skills and surgical risks of users can be comprehensively evaluated, providing a scientific basis for surgical decision-making.

[0067] Step S105: Generate an adaptive training scenario according to the evaluation index matrix; the adaptive training scenario includes complication simulation information, abnormal sign trigger information, and emergency state intervention information to train the target user based on the preset mixed reality interaction device and the adaptive training scenario.

[0068] Specifically, step S105 aims to test the performance of the virtual patient model through a series of quantitative and qualitative evaluation methods and optimize it according to the evaluation results. This may include the accuracy of the model, response time, user interaction experience, as well as the stability and reliability of the model. Determine specific indicators for evaluating the performance of the virtual patient model, such as simulation accuracy, computational efficiency, user satisfaction, etc. Evaluate the accuracy of the model in terms of anatomical structure, physiological response, etc. by comparing it with real patient data. Measure the response speed of the model to user operations to ensure real-time performance in actual applications. Test the stability of the model under long-term operation or high-load conditions.

[0069] Evaluate the practicality and interactivity of the model through the operation experience of actual users (such as medical students, doctors). Invite domain experts to review the medical accuracy and educational value of the model. Analyze test data and user feedback to identify the strengths and weaknesses of the model. According to the evaluation results, adjust and optimize the model, such as improving the algorithm, adding new functions, or optimizing the user interface. Take evaluation and optimization as a continuous iterative process to continuously improve the model to meet user needs and improve performance.

[0070] Through systematic evaluation and optimization, the quality of the virtual patient model can be significantly improved, making it closer to the physiological and anatomical characteristics of real patients. The optimized model can provide a smoother and more intuitive user experience, improving user satisfaction and the practicality of the model. A high-quality virtual patient model can be used as an effective educational tool to help students and doctors better learn and master medical knowledge and skills. Continuous performance evaluation and optimization promote the development of related technologies, such as innovation in the fields of image processing, machine learning, etc. By testing and training in a virtual environment, the risks and costs in actual surgeries can be reduced, and surgical safety can be improved. A virtual patient model with superior performance can be applied to a wider range of fields, such as clinical decision support, new drug testing, etc.

[0071] By implementing step S105, it can be ensured that the virtual patient model plays the greatest utility in education, training, and clinical applications, while promoting the development and innovation of related technologies.

[0072] In some embodiments, generating a virtual patient three-dimensional model based on the imaging data, pathological data, and physiological signal data includes: performing cross-modal alignment on the imaging data, pathological data, and physiological signal data; constructing a dynamic volume rendering framework based on neural radiance fields, and generating tissue texture information corresponding to the imaging data, pathological data, and physiological signal data under the dynamic volume rendering framework; establishing a vascular wall stress-strain model and a dynamic response mechanism corresponding to the imaging data, pathological data, and physiological signal data according to a physiological simulation algorithm based on finite element analysis; and generating the virtual patient three-dimensional model according to the vascular wall stress-strain model and the dynamic response mechanism.

[0073] First, it is necessary to integrate data from different sources. Imaging data (such as CT, MRI) provides anatomical structure information of the patient, pathological data provides detailed information on tissue lesions, and physiological signal data (such as electrocardiogram, blood pressure) provides physiological state information of the patient. These data need to be matched and integrated through cross-modal alignment techniques to ensure their consistency in time and space.

[0074] Denoise, enhance, and segment the imaging data to extract key anatomical structures; perform text analysis on the pathological data to extract lesion characteristics; and filter and normalize the physiological signal data to reduce the influence of noise and outliers.

[0075] Use neural radiance fields (NeRF) technology to construct a dynamic volume rendering framework. NeRF is a deep learning method for three-dimensional scene representation and rendering, capable of synthesizing a continuous volume field of new views from sparse views. Under this framework, generate corresponding tissue texture information according to the integrated imaging data, pathological data, and physiological signal data. These texture information not only includes the morphological characteristics of the tissue, but also its physiological properties, such as the distribution of blood vessels and blood flow status.

[0076] Use finite element analysis (FEA) technology to establish a stress-strain model of the vascular wall according to the tissue texture information. FEA is a numerical analysis method used to predict the response of a structure under physical actions, such as stress, strain, and displacement. According to the vascular wall stress-strain model, establish a dynamic response mechanism to simulate the changes of the vascular wall under different physiological states, such as dilation and contraction when blood pressure changes. Integrate the established vascular wall stress-strain model and the dynamic response mechanism into the virtual patient three-dimensional model to generate a virtual patient model with anatomical accuracy and physiological dynamic response characteristics. Verify the accuracy and reliability of the model by comparing it with actual patient data, and optimize it according to the feedback to improve the simulation degree and practicality of the model.

[0077] Through the cross-modal alignment technology and the dynamic volume rendering framework of neural radiance fields, the generated virtual patient model has high-precision anatomical features and tissue texture information, and can more realistically simulate the physiological state of patients.

[0078] By using the stress-strain model of the blood vessel wall and the dynamic response mechanism established by finite element analysis, the virtual patient model can simulate real physiological dynamic changes, such as the dilation and contraction of blood vessels, providing a more realistic simulation environment for medical training and research.

[0079] The virtual patient model can be customized according to different patient data and can be reused infinitely, without being restricted by actual patient resources, improving the efficiency and accessibility of medical education and research.

[0080] Through simulation rather than actual surgical operations, the virtual patient model provides a safe training environment, reducing surgical risks and costs. At the same time, due to the reusability of the model, the economic burden of medical education and skill training is reduced.

[0081] The high-precision virtual patient model not only has important value in medical education, helping students and doctors better understand and master human body structures and physiological functions, but also can simulate physiological changes under different disease states in medical research, providing new tools and methods for disease mechanism research and new therapy development.

[0082] Exemplarily, the cross-modal alignment of the image data, pathological data, and physiological signal data includes: extracting multi-scale features from the image data according to a preset 3D Swin Transformer encoder to obtain image feature information; performing temporal modeling on the pathological data according to a preset Bi-LSTM encoder to obtain pathological feature information; decoupling the time-frequency domain features of the physiological signal through a wavelet transform kernel to obtain physiological feature information; and completing the cross-modal alignment of the image data, pathological data, and physiological signal data according to the image feature information, pathological feature information, and physiological feature information.

[0083] Extract multi-scale features from image data (such as CT, MRI scans) by using a preset 3D Swin Transformer encoder. The 3D Swin Transformer is a deep learning model based on the Transformer architecture, which is particularly suitable for processing 3D image data. Through the 3D Swin Transformer encoder, rich spatial features and context information, including multi-scale features such as shape, texture, and position, can be extracted from the image data.

[0084] Temporal modeling of pathological data is performed using a preset Bi-LSTM (Bidirectional Long Short-Term Memory Network) encoder. Bi-LSTM is a deep learning model particularly suitable for processing sequential data and capable of capturing long-term dependencies in the data. Through the Bi-LSTM encoder, key temporal features can be extracted from the pathological data, including the development process and change trends of lesions, etc.

[0085] Time-frequency domain feature decoupling of physiological signal data (such as electrocardiogram, blood pressure monitoring) is carried out through a wavelet transform kernel. Wavelet transform is an effective time-frequency analysis tool that can decompose a signal into components of different scales and frequencies. Through the wavelet transform kernel, key time-frequency domain features can be extracted from the physiological signal data, including heart rate, respiratory rate, and rhythm, etc.

[0086] According to the extracted image feature information, pathological feature information, and physiological feature information, cross-modal alignment of image data, pathological data, and physiological signal data is completed using feature fusion technology. Through feature fusion technology, data of different modalities can be mapped into a unified feature space to ensure their consistency in time and space, providing accurate input data for the subsequent generation of virtual patient models.

[0087] By using a 3D Swin Transformer encoder, a Bi-LSTM encoder, and a wavelet transform kernel, key feature information can be extracted from data of different modalities, improving the accuracy and reliability of data fusion. The cross-modal alignment technology enables the virtual patient model to comprehensively consider image, pathological, and physiological signal data, enhancing the generalization ability and applicability of the model. Accurate cross-modal alignment ensures the consistency of the virtual patient model in terms of anatomical features, pathological changes, and physiological dynamics, improving the simulation degree and practicality of the model. The cross-modal alignment technology provides more accurate training data for medical education and skills training, helping to improve the training effect and the accuracy of evaluation. The cross-modal alignment technology promotes the cross-fusion of data from different disciplines such as imaging, pathology, and physiology, providing new tools and methods for multidisciplinary cross-research. Accurate cross-modal alignment technology helps doctors to more comprehensively understand the health status of patients, improving the scientificity and accuracy of clinical decision-making.

[0088] Through the implementation of examples, the method can achieve accurate cross-modal alignment of image data, pathological data, and physiological signal data, providing high-quality input data for the generation of virtual patient models, thereby improving the accuracy, simulation degree, and practicality of the models.

[0089] Exemplarily, before generating the virtual patient three-dimensional model according to the vascular wall stress-strain model and the dynamic response mechanism, it further includes: verifying the authenticity of the vascular wall stress-strain model and the dynamic response mechanism by a discriminator constructed based on a generative adversarial network; the discriminator includes a cycle consistency loss function for ensuring that the phase synchronization error between the respiratory rhythm corresponding to the virtual patient three-dimensional model and the cardiovascular pulsation is less than a preset error; if the authenticity verification passes, generating the virtual patient three-dimensional model according to the vascular wall stress-strain model and the dynamic response mechanism.

[0090] In the example, a generative adversarial network (GAN) is used to construct a discriminator whose task is to distinguish real patient data from data generated by the model. The discriminator needs to be able to identify the authenticity of the vascular wall stress-strain model and the dynamic response mechanism. A cycle consistency loss function is integrated into the discriminator, which is used to ensure the consistency of the physiological dynamics (such as respiratory rhythm and cardiovascular pulsation) of the virtual patient model with the real patient data.

[0091] The data generated based on the vascular wall stress-strain model and the dynamic response mechanism is input into the discriminator. The discriminator evaluates the authenticity of the input data, and the cycle consistency loss function ensures that the phase synchronization error between the physiological rhythm of the virtual patient model and the real patient data is controlled within a preset error range. If the discriminator determines that the generated data is not real enough, the feedback will be used to optimize the vascular wall stress-strain model and the dynamic response mechanism until the generated data passes the authenticity verification of the discriminator. Once the vascular wall stress-strain model and the dynamic response mechanism pass the authenticity verification, the virtual patient three-dimensional model can be generated according to these mechanisms. After generating the model, the anatomical features and physiological dynamic responses of the model are further refined to ensure that the model is visually and functionally close to a real patient.

[0092] By using the discriminator of the adversarial generative network for authenticity verification, the authenticity of the virtual patient model can be significantly improved. This ensures that the model has a high degree of accuracy and reliability when simulating the physiological dynamics of real patients. The application of the cycle consistency loss function ensures that the phase synchronization error between the respiratory rhythm and cardiovascular pulsation of the virtual patient model and the real patient data is controlled within a preset range, thereby improving the physiological synchronization of the model. The feedback mechanism of the discriminator allows for iterative optimization of the blood vessel wall stress-strain model and dynamic response mechanism until the generated model meets the authenticity requirements, which helps to improve the efficiency and quality of the model generation process. The authenticity-verified virtual patient model can provide a more accurate training and simulation environment for medical education and research, thereby improving the training effect and the reliability of research. By performing authenticity verification before model generation, the risks and costs caused by discovering inaccurate models in actual applications can be reduced. This provides a safer and more economical solution for medical education and clinical practice. The authenticity verification process ensures that the virtual patient model can be customized according to the individual's physiological characteristics, which is of great significance for the development of personalized medicine.

[0093] Through the implementation of examples, the method can generate a virtual patient model that is both realistic and highly personalized, providing a powerful tool for medical education, skills training, and clinical research.

[0094] It should be noted that the cycle consistency loss function (Cycle Consistency Loss) is a loss function used in the CycleGAN (Cycle Generative Adversarial Network) to ensure that the generated images remain consistent after being transformed in two directions. In CycleGAN, there are two generators (G and F) and two discriminators (Dx and Dy), where G is responsible for transforming images from domain X to domain Y, and F is responsible for transforming images from domain Y back to domain X. The purpose of the cycle consistency loss function is to ensure that the images transformed by G and F are as similar as possible to the original images. The mathematical expression is as follows:

[0095] Lcyc = Ex~pdata(x)[||F(G(x)) - x||1];

[0096] Lcyc is the cycle consistency loss. G(x) is the result of the generator G transforming the image x from domain X to domain Y. || ||1 is the L1 norm, which is used to calculate the difference between two images.

[0097] In some embodiments, performing multi-scale feature encoding on the operation trajectory, instrument contact pressure, and biological tissue deformation parameters corresponding to the historical operation data to generate an operation feature vector with time series correlation, including: performing kinematic modeling on the operation trajectory according to a preset spatio-temporal graph convolutional network to establish a dynamic spatial relationship graph between the end effector of the instrument and the anatomical landmark points; constructing a dual-channel attention mechanism to perform multi-physical field coupling analysis on the instrument contact pressure and biological tissue deformation parameters to generate a real-time deformation heat map of the instrument action area; generating a hierarchical Transformer encoder according to the operation feature vector corresponding to the dynamic spatial relationship graph and the real-time deformation heat map.

[0098] Use a preset spatio-temporal graph convolutional network (ST-GCN) to perform kinematic modeling on the operation trajectory. ST-GCN is a deep learning model capable of processing spatio-temporal data and is suitable for analyzing dynamic patterns in the operation trajectory. Through ST-GCN, establish a dynamic spatial relationship graph between the end effector of the instrument and the anatomical landmark points, capturing the movement trajectory of the end effector of the instrument and the relative position changes with the anatomical structure during the operation.

[0099] Construct a dual-channel attention mechanism to analyze the instrument contact pressure and biological tissue deformation parameters respectively. This mechanism can identify and emphasize the most important physical field features during the operation. Combining the outputs of the dual-channel attention mechanism, generate a real-time deformation heat map of the instrument action area, visually showing the deformation and pressure distribution during the interaction between the instrument and the biological tissue.

[0100] Application of the hierarchical Transformer encoder: Use the hierarchical Transformer encoder to process the operation feature vector corresponding to the dynamic spatial relationship graph and the real-time deformation heat map. The Transformer encoder is good at processing sequence data and can capture the time series correlation in the feature vector. Through the hierarchical Transformer encoder, generate operation feature vectors with time series correlation, which integrate multi-scale features of the operation trajectory, instrument contact pressure, and biological tissue deformation parameters.

[0101] Through kinematic modeling and multi - physical field coupling analysis, the dynamic interaction between instruments and biological tissues during surgical operations can be understood and predicted more precisely, improving the accuracy and safety of surgical operations. The generated operation feature vectors can provide rich feedback information for surgical training, helping trainees better understand and master surgical skills and enhancing the effectiveness of surgical training. The real - time deformation heat map and operation feature vectors can assist doctors in surgical planning and execution by monitoring key parameters during the surgical process in real - time and optimizing surgical strategies. The surgical navigation system integrating multi - scale feature encoding and operation feature vector generation methods can provide more intelligent navigation and assistance, improving the intelligence level of the system. This embodiment provides a new method for surgical data analysis and feature extraction, providing a new perspective and tool for the innovation and development of surgical techniques. The operation feature vectors and real - time deformation heat map provide detailed data support for the recording and evaluation of the surgical process, enhancing the traceability of the surgical process and the objectivity of evaluation.

[0102] Through the implementation of the embodiment, technological progress can be achieved in multiple aspects such as surgical operation analysis, surgical training, surgical planning, and surgical navigation, improving the safety and effectiveness of surgery and laying a foundation for the future development of surgical techniques at the same time.

[0103] In some embodiments, inputting the operation feature vector into the virtual patient three - dimensional model to obtain the physiological parameter change sequence corresponding to the operation feature vector includes: calculating the blood pressure waveform propagation information and simulated airway pressure gradient information corresponding to the operation feature vector in the virtual patient three - dimensional model; obtaining a causal inference chain of preset operation features and pathophysiological responses; generating the physiological parameter change sequence according to the blood pressure waveform propagation information, simulated airway pressure gradient information, operation feature vector, and the causal inference chain; the physiological parameter change sequence includes at least blood loss information and nerve reflex information.

[0104] By inputting the operation feature vector into the virtual patient three - dimensional model. These feature vectors contain key information during the surgical operation, such as instrument kinematic features, instrument contact pressure, and biological tissue deformation parameters, etc.

[0105] In the virtual patient three - dimensional model, calculate the corresponding blood pressure waveform propagation information according to the operation feature vector. This involves simulating the flow and pressure changes of blood in the vascular system to reflect the impact of the surgical operation on the circulatory system. Similarly, simulate the airway pressure gradient information to reflect the impact of the surgical operation on the respiratory system, especially in cases involving airway or lung surgeries.

[0106] Using medical knowledge and expert systems, establish a causal inference chain between preset operating characteristics and pathophysiological responses. This includes determining how specific surgical operations affect specific physiological parameters such as blood pressure, heart rate, respiratory rate, etc. Combine blood pressure waveform propagation information, simulated airway pressure gradient information, operating feature vectors, and the causal inference chain to generate a sequence of physiological parameter changes. This sequence details the impact of surgical operations on the physiological state of the virtual patient model, including but not limited to bleeding volume information and nerve reflex information.

[0107] By simulating blood pressure waveform propagation and airway pressure gradients, and considering the causal relationship between operating characteristics and pathophysiological responses, the authenticity and accuracy of surgical simulation can be significantly improved. The sequence of physiological parameter changes provides important feedback information for surgical training, helping trainees understand the impact of surgical operations on the patient's physiological state, thereby enhancing the effectiveness of surgical training. The sequence of physiological parameter changes can help doctors predict and plan the physiological changes that may be caused by surgical operations before surgery, optimize surgical strategies and decisions. By simulating the impact of surgical operations on the physiological state, potential complications can be identified and prevented in advance, improving the safety of surgery. The sequence of physiological parameter changes provides new tools and data for medical research and education, contributing to a deeper understanding of the relationship between surgical operations and physiological responses. This embodiment involves knowledge and technologies from multiple disciplines such as medicine, engineering, and computer science, promoting interdisciplinary cooperation and driving the development of related fields.

[0108] Through the implementation of the embodiment, technological progress can be achieved in multiple aspects such as surgical simulation, surgical training, surgical planning, and decision-making, improving the safety and effectiveness of surgery, while providing new perspectives and tools for medical research and education.

[0109] In some embodiments, the joint analysis of the operating feature vector and the sequence of physiological parameter changes to obtain an evaluation index matrix includes: constructing an anomaly detection module based on a graph neural network to determine at least one mutation point in the sequence of physiological parameter changes; obtaining abnormal operation information corresponding to the mutation point in the operating feature vector; generating the evaluation index matrix according to the abnormal operation information.

[0110] Construct an anomaly detection module using a graph neural network. This module can process complex physiological parameter change sequence data and identify anomaly patterns or mutation points therein. By analyzing the time series data of physiological parameter changes, the GNN can identify mutation points representing significant changes in physiological states, which may indicate potential problems or complications during surgical operations. In the operation feature vector, according to the determined mutation points, extract the abnormal operation information corresponding to these mutation points. This may include improper use of instruments, excessive operation force, or deviation of the surgical path, etc. Use machine learning techniques, such as clustering or classification algorithms, to extract features related to the mutation points from the operation feature vector to identify specific abnormal operations. Define a series of evaluation metrics for quantifying the quality and effectiveness of surgical operations, such as operation precision, safety, and efficiency, etc. According to the abnormal operation information and predefined evaluation metrics, construct an evaluation metric matrix. This matrix provides a quantitative evaluation for each surgical operation, helping to identify the strengths and weaknesses in the operation.

[0111] By identifying abnormal patterns in surgical operations, timely measures can be taken to prevent potential complications, thus improving the safety of the surgery. The evaluation metric matrix provides detailed feedback on surgical operations, helping doctors and medical students identify and improve operation skills and optimize the surgical process. In surgical training, the evaluation metric matrix can be used as a teaching tool to help trainees understand the standards and expectations of surgical operations and improve the effectiveness of training. The joint analysis method provides a new perspective to observe and understand the relationship between surgical operations and physiological responses, promoting the innovation and development of surgical techniques. The evaluation metric matrix provides a quantitative and objective method to evaluate the quality of surgical operations, reducing the influence of subjective judgment and improving the accuracy and reliability of evaluation. By analyzing the operation feature vector and physiological parameter change sequence, it can provide support for clinical decision-making and help doctors make better decisions during the surgery. The embodiments involve knowledge in multiple fields such as medicine, data science, and artificial intelligence, promoting cooperation between different disciplines and driving the development of related technologies.

[0112] Through the implementation of the embodiments, technological progress can be achieved in multiple aspects such as surgical safety, surgical operation optimization, surgical training, and surgical technique innovation, improving the overall quality and effectiveness of the surgery.

[0113] In some embodiments, generating an adaptive training scenario according to the evaluation metric matrix includes: determining the ability defect information of the target user according to the evaluation metric matrix; quickly obtaining the concurrent simulated disease information corresponding to the ability defect information according to the model-agnostic meta-learning algorithm; the complication simulation information includes a disease evolution tree; performing cross-modal enhanced rendering on the virtual patient three-dimensional model according to the concurrent simulated disease information; the cross-modal enhanced rendering at least includes converting the heart sound features corresponding to the concurrent simulated disease information into a tactile feedback matrix and mapping the abnormal blood gas analysis values corresponding to the concurrent simulated disease information into a visualized metabolic pathway disorder hot spot map; generating the adaptive training scenario according to the tactile feedback matrix, the visualized metabolic pathway disorder hot spot map, and the virtual patient three-dimensional model.

[0114] Based on the evaluation metric matrix, analyze the performance of the target user during the surgical operation to determine their ability defect information. This may include problems such as insufficient operation accuracy and improper handling of specific complications. Use the model-agnostic meta-learning (MAML) algorithm to quickly adapt and obtain the concurrent simulated disease information corresponding to the target user's ability defect information. The MAML algorithm can quickly learn from a small amount of data and adapt to new tasks, and is suitable for the rapid generation of personalized training scenarios. Construct a disease evolution tree to describe in detail the possible complications and their development paths, providing a structured representation for the simulated disease information.

[0115] Convert the heart sound features corresponding to the concurrent simulated disease information into a tactile feedback matrix. This involves processing audio signals into tactile signals to simulate the tactile feedback during the surgical operation. Map the abnormal blood gas analysis values corresponding to the concurrent simulated disease information into a visualized metabolic pathway disorder hot spot map. This involves converting biochemical data into a graphical representation to intuitively display the obstacles and abnormalities in the metabolic pathway.

[0116] Apply the tactile feedback matrix to the virtual patient three-dimensional model to simulate the tactile feedback during the surgical operation and enhance the user's actual operation experience. Overlay the visualized metabolic pathway disorder hot spot map on the virtual patient three-dimensional model to provide visual feedback on the metabolic abnormalities. Combine the tactile feedback matrix, the visualized metabolic pathway disorder hot spot map, and the virtual patient three-dimensional model to generate an adaptive training scenario. This scenario is customized for the ability defects of the target user to provide a personalized training experience.

[0117] By analyzing the evaluation index matrix and generating adaptive training scenarios, customized training content can be provided for each user, improving the personalization and pertinence of training. The adaptive training scenarios provide specialized training for users' ability deficiencies, helping users master surgical operation skills faster, improving the accuracy and efficiency of surgical operations. By simulating complications and providing haptic feedback, users' awareness of potential problems during surgery can be enhanced, preventing and dealing with complications in advance, and improving the safety of surgery. The cross-modal enhanced rendering technology provides multiple sensory feedbacks such as vision and touch, enhancing users' multi-modal interaction experience and improving the realism and immersion of training. The embodiments provide a new medical education method, improving the educational effect through simulation and multi-modal feedback, and promoting the innovation and development of medical education. The adaptive training scenarios can help users better understand the impact of surgical operations on patients' physiological states and improve surgical decision-making abilities. The embodiments involve knowledge in multiple fields such as medicine, computer science, and human-computer interaction, promoting cooperation between different disciplines and driving the development of related technologies.

[0118] Through the implementation of the embodiments, technological progress can be achieved in multiple aspects such as surgical training, surgical safety, and medical education, improving the overall quality and effect of surgery.

[0119] In some embodiments, the training of the target user according to the preset mixed reality interaction device and adaptive training scenario includes: obtaining the eye movement trajectory information, operation force information of the target user wearing the mixed reality interaction device, and virtual physiological parameter change information corresponding to the virtual patient three-dimensional model; dynamically adjusting the complexity of the mixed reality interaction device according to the eye movement trajectory information, operation force information, and the virtual physiological parameter change information, and completing the training of the target user.

[0120] Using the eye tracking technology in the mixed reality interaction device, the eye movement trajectory information of the target user during the training process is obtained in real time. These information can help analyze the user's attention focus and visual behavior patterns. Through the sensors of the interaction device, the force information exerted by the user when operating the virtual patient three-dimensional model is collected to evaluate the user's operation skills and force control ability. Synchronously record the virtual physiological parameter change information in the virtual patient model, which reflects the impact of the user's operation on the state of the virtual patient.

[0121] By developing an algorithm, the complexity of the mixed reality interaction device is dynamically adjusted according to the user's eye movement trajectory information, operation force information, and virtual physiological parameter change information. This may include adjusting the graphics rendering quality, the sensitivity of the interaction feedback, or the difficulty of the training scenario.

[0122] Adjust the training content and difficulty in real time according to the user's real-time performance and progress to provide a personalized training experience. For example, if the user performs poorly in a certain operation skill, the system can automatically increase the complexity of the relevant training module to strengthen the training of this skill.

[0123] During the training process, the system provides real-time feedback to help the user understand their performance and progress. This includes feedback on operation skills, visual attention, and responses to changes in virtual physiological parameters. After the training, the system comprehensively analyzes the user's performance throughout the training process, provides an evaluation report on the training effect, and points out the user's strengths and areas for improvement.

[0124] By dynamically adjusting the complexity of the mixed reality interaction device, a customized training plan can be provided for each user, improving the adaptability and personalization of the training.

[0125] The mixed reality technology provides a highly interactive and immersive training environment, enabling users to participate more deeply in the training, improving the training effect and attractiveness. By simulating a real surgical environment and physiological responses, users can practice surgical skills and decision-making abilities in a safe environment, reducing errors and risks in actual surgeries. Dynamically adjusting the training difficulty can ensure the effective utilization of training resources, prevent users from wasting time on overly simple or difficult training content, and improve training efficiency.

[0126] The mixed reality technology integrates various sensory information such as vision and touch, helping users integrate and coordinate different sensory and motor skills during training. Training in a virtual environment can avoid risks in actual surgeries and at the same time allow users to repeat the same operations multiple times to deepen their understanding and mastery of surgical skills. By continuously collecting and analyzing the user's training data, the development of the user's skills can be evaluated, providing a basis for continuous skill improvement.

[0127] In some embodiments, by developing a multi-user interaction system, multiple users can be allowed to participate in the same surgical simulation training scenario simultaneously, simulating a real surgical team collaboration environment. Each user plays a specific role in the system (such as the surgeon, assistant, anesthesiologist, etc.) and is assigned different tasks and responsibilities according to the role. The system provides real-time voice and text communication tools to simulate communication and coordination during surgery, and at the same time provides feedback and evaluation on team collaboration.

[0128] By simulating real surgical team collaboration, improve the user's team collaboration ability and communication skills. Train users to effectively allocate tasks and coordinate resources in a high-pressure environment to improve surgical efficiency.

[0129] In some embodiments, by adding an emergency situation module, such as massive hemorrhage, cardiac arrest, etc., to the surgical simulation training, the rapid response and handling capabilities of users in emergency situations are trained. Provide decision tree guidance to help users make correct decisions in emergency situations, and at the same time evaluate the decision-making effects of users. By simulating emergency situations, evaluate the psychological stress response of users and provide psychological adjustment training. Train users' rapid response and correct handling capabilities in emergency situations to reduce mistakes in actual surgeries. By simulating emergency situations, improve users' psychological endurance and stress-coping abilities.

[0130] In some embodiments, by developing a set of skill assessment algorithms, the surgical operation skills of users are quantified, such as cutting accuracy, suture quality, etc. According to the assessment results, provide real-time operation feedback and improvement suggestions to help users adjust operation skills in a timely manner. According to the assessment results of users, the system plans personalized skill improvement paths and recommends targeted training content. By means of quantitative assessment and real-time feedback, help users improve their surgical operation skills. Provide personalized skill improvement paths according to the actual situation of users to improve the pertinence and efficiency of training.

[0131] In some embodiments, by constructing a comprehensive knowledge base containing knowledge of multiple disciplines such as medicine, engineering, and psychology, rich background knowledge is provided for surgical simulation training. Knowledge application training: In surgical simulation training, design specific training modules that require users to apply interdisciplinary knowledge to solve problems in surgeries. Knowledge integration assessment: Evaluate the application of interdisciplinary knowledge by users in training and provide feedback and improvement suggestions. Through the application training of interdisciplinary knowledge, improve users' ability to comprehensively apply knowledge of different disciplines to solve practical problems. Encourage users to attempt interdisciplinary knowledge integration in surgical simulation training to promote the innovation of new surgical techniques and methods.

[0132] Please refer to Figure 2 , Figure 2 FIG. is a schematic structural diagram of a virtual patient-based medical skill assessment and training device 200 provided by an embodiment of the present application. The virtual patient-based medical skill assessment and training device 200 is used to execute the steps of the virtual patient-based medical skill assessment and training method shown in the above embodiments. The virtual patient-based medical skill assessment and training device 200 may be a single server or a server cluster, or the virtual patient-based medical skill assessment and training device 200 may be a terminal, and the terminal may be a handheld terminal, a notebook computer, a wearable device, or a robot, etc.

[0133] As Figure 2 shown, the virtual patient-based medical skill assessment and training device 200 includes:

[0134] The data acquisition module 201 is configured to acquire the imaging data, pathological data, and physiological signal data corresponding to historical patients, and generate a virtual patient three-dimensional model according to the imaging data, pathological data, and physiological signal data; the virtual patient three-dimensional model has anatomical accuracy and physiological dynamic response characteristics;

[0135] The operation acquisition module 202 is configured to acquire the historical operation data of the target user, perform multi-scale feature encoding on the operation trajectory, instrument contact pressure, and biological tissue deformation parameters corresponding to the historical operation data, and generate an operation feature vector with time series correlation;

[0136] The vector input module 203 is configured to input the operation feature vector into the virtual patient three-dimensional model to obtain a physiological parameter change sequence corresponding to the operation feature vector; the physiological parameter change sequence is used to determine the coupling response of the circulatory system, respiratory system, and nervous system corresponding to the operation feature vector;

[0137] The evaluation acquisition module 204 is configured to perform joint analysis on the operation feature vector and the physiological parameter change sequence to obtain an evaluation index matrix; the evaluation index matrix includes at least an operation standardization score, a clinical decision rationality score, and an emergency response ability score;

[0138] The training completion module 205 is configured to generate an adaptive training scenario according to the evaluation index matrix; the adaptive training scenario includes complication simulation information, abnormal sign trigger information, and emergency state intervention information, so as to complete the training of the target user according to a preset mixed reality interaction device and the adaptive training scenario.

[0139] In some embodiments, the generating the virtual patient three-dimensional model according to the imaging data, pathological data, and physiological signal data includes: performing cross-modal alignment on the imaging data, pathological data, and physiological signal data; constructing a dynamic volume rendering framework based on neural radiance fields, and generating tissue texture information corresponding to the imaging data, pathological data, and physiological signal data under the dynamic volume rendering framework; according to a physiological simulation algorithm based on finite element analysis, establishing a blood vessel wall stress-strain model and a dynamic response mechanism corresponding to the imaging data, pathological data, and physiological signal data according to the tissue texture information; generating the virtual patient three-dimensional model according to the blood vessel wall stress-strain model and the dynamic response mechanism.

[0140] Exemplarily, the cross-modal alignment of the imaging data, pathological data, and physiological signal data includes: extracting multi-scale features from the imaging data according to a preset 3D Swin Transformer encoder to obtain imaging feature information; performing temporal modeling on the pathological data according to a preset Bi-LSTM encoder to obtain pathological feature information; decoupling the time-frequency domain features of the physiological signal through a wavelet transform kernel to obtain physiological feature information; and completing the cross-modal alignment of the imaging data, pathological data, and physiological signal data according to the imaging feature information, pathological feature information, and physiological feature information.

[0141] Exemplarily, before generating the virtual patient three-dimensional model according to the vascular wall stress-strain model and the dynamic response mechanism, it further includes: verifying the authenticity of the vascular wall stress-strain model and the dynamic response mechanism according to a discriminator constructed based on a generative adversarial network; the discriminator includes a cycle consistency loss function for ensuring that the phase synchronization error between the respiratory rhythm and the cardiovascular pulsation corresponding to the virtual patient three-dimensional model is less than a preset error; if the authenticity verification passes, generating the virtual patient three-dimensional model according to the vascular wall stress-strain model and the dynamic response mechanism.

[0142] In some embodiments, the multi-scale feature encoding of the operation trajectory, instrument contact pressure, and biological tissue deformation parameters corresponding to the historical operation data to generate an operation feature vector with time series correlation includes: performing kinematic modeling on the operation trajectory according to a preset spatio-temporal graph convolutional network to establish a dynamic spatial relationship graph between the end effector of the instrument and the anatomical landmark points; constructing a two-channel attention mechanism to perform multi-physical field coupling analysis on the instrument contact pressure and biological tissue deformation parameters to generate a real-time deformation thermal map of the instrument action area; and generating a hierarchical Transformer encoder according to the operation feature vector corresponding to the dynamic spatial relationship graph and the real-time deformation thermal map.

[0143] In some embodiments, inputting the operation feature vector into the virtual patient three-dimensional model to obtain a sequence of physiological parameter changes corresponding to the operation feature vector includes: calculating the blood pressure waveform propagation information and the simulated airway pressure gradient information corresponding to the operation feature vector in the virtual patient three-dimensional model; obtaining a causal inference chain of preset operation features and pathophysiological responses; generating the sequence of physiological parameter changes according to the blood pressure waveform propagation information, the simulated airway pressure gradient information, the operation feature vector, and the causal inference chain; and the sequence of physiological parameter changes includes at least blood loss information and nerve reflex information.

[0144] In some embodiments, the joint analysis of the operation feature vector and the physiological parameter change sequence to obtain an evaluation index matrix includes: constructing an anomaly detection module based on a graph neural network to determine at least one mutation point in the physiological parameter change sequence; obtaining abnormal operation information corresponding to the mutation point in the operation feature vector; and generating the evaluation index matrix according to the abnormal operation information.

[0145] In some embodiments, generating an adaptive training scenario according to the evaluation index matrix includes: determining the ability defect information of the target user according to the evaluation index matrix; quickly obtaining the concurrent simulated disease information corresponding to the ability defect information according to the model-agnostic meta-learning algorithm; the complication simulation information includes a disease evolution tree; performing cross-modal enhanced rendering on the virtual patient three-dimensional model according to the concurrent simulated disease information; the cross-modal enhanced rendering at least includes converting the heart sound features corresponding to the concurrent simulated disease information into a tactile feedback matrix and mapping the abnormal blood gas analysis values corresponding to the concurrent simulated disease information into a visualized metabolic pathway disorder heat map; and generating the adaptive training scenario according to the tactile feedback matrix, the visualized metabolic pathway disorder heat map, and the virtual patient three-dimensional model.

[0146] In some embodiments, completing the training of the target user according to a preset mixed reality interaction device and an adaptive training scenario includes: obtaining the eye movement trajectory information, operation force information of the target user wearing the mixed reality interaction device, and virtual physiological parameter change information corresponding to the virtual patient three-dimensional model; and dynamically adjusting the complexity of the mixed reality interaction device according to the eye movement trajectory information, operation force information, and virtual physiological parameter change information to complete the training of the target user.

[0147] It should be noted that those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the above-described device and each module can refer to the corresponding processes in the foregoing method embodiments, and will not be elaborated herein.

[0148] The above device can be implemented in the form of a computer program, and the computer program can run on a computer device as shown in Figure 3 shown.

[0149] Please refer to Figure 3 , Figure 3 which is a schematic block diagram of the structure of a computer device in an embodiment. The computer device can be a server.

[0150] Referring to Figure 3 , the computer device includes a processor, a memory, and a network interface connected through a system bus. Among them, the memory can include a non-volatile storage medium and an internal memory.

[0151] The non-volatile storage medium can store an operating system and a computer program. The computer program includes program instructions which, when executed, can cause the processor to execute any one of the medical skill assessment and training methods based on virtual patients.

[0152] The processor is used to provide computing and control capabilities to support the operation of the entire computer device.

[0153] The internal memory provides an environment for the operation of the computer program in the non-volatile storage medium. When the computer program is executed by the processor, it can cause the processor to execute any one of the medical skill assessment and training methods based on virtual patients.

[0154] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art can understand that Figure 3 the structure shown in is only a block diagram of some structures related to the solution of this application, and does not constitute a limitation on the computer device to which the solution of this application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements.

[0155] It should be understood that the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0156] Wherein, in one embodiment, the processor is used to run the computer program stored in the memory to implement the following steps:

[0157] Obtain the image data, pathological data, and physiological signal data corresponding to the historical patient, and generate a virtual patient three-dimensional model according to the image data, pathological data, and physiological signal data; the virtual patient three-dimensional model has anatomical accuracy and physiological dynamic response characteristics;

[0158] Obtain the historical operation data of the target user, perform multi-scale feature encoding on the operation trajectory, instrument contact pressure, and biological tissue deformation parameters corresponding to the historical operation data to generate an operation feature vector with time series correlation;

[0159] Input the operation feature vector into the virtual patient three-dimensional model to obtain the physiological parameter change sequence corresponding to the operation feature vector; the physiological parameter change sequence is used to determine the coupling response of the circulatory system, respiratory system, and nervous system corresponding to the operation feature vector;

[0160] Perform joint analysis on the operation feature vector and the physiological parameter change sequence to obtain an evaluation index matrix; the evaluation index matrix at least includes an operation standardization score, a clinical decision rationality score, and an emergency response ability score;

[0161] Generate an adaptive training scenario according to the evaluation index matrix; the adaptive training scenario includes complication simulation information, abnormal sign trigger information, and emergency state intervention information to train the target user according to a preset mixed reality interaction device and the adaptive training scenario.

[0162] An embodiment of the present application also provides a computer-readable storage medium, the computer-readable storage medium stores a computer program, the computer program includes program instructions, and the processor executes the program instructions to implement any one of the medical skill evaluation and training methods based on a virtual patient provided by the embodiments of the present application.

[0163] Among them, the computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiment, such as the hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk equipped on the computer device, a smart media card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc.

[0164] The above is only the specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of various equivalent modifications or substitutions within the technical scope disclosed by the present application, and these modifications or substitutions should all be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A medical skill assessment and training device based on virtual patients, characterized in that, Including: A data acquisition module, configured to acquire imaging data, pathological data, and physiological signal data corresponding to historical patients, and generate a virtual patient three-dimensional model based on the imaging data, pathological data, and physiological signal data; The virtual patient three-dimensional model has anatomical accuracy and physiological dynamic response characteristics; An operation acquisition module, configured to acquire historical operation data of a target user, perform multi-scale feature encoding on the operation trajectory, instrument contact pressure, and biological tissue deformation parameters corresponding to the historical operation data, and generate an operation feature vector with time series correlation; A vector input module, configured to input the operation feature vector into the virtual patient three-dimensional model to obtain a physiological parameter change sequence corresponding to the operation feature vector; the physiological parameter change sequence is used to determine the coupling response of the circulatory system, respiratory system, and nervous system corresponding to the operation feature vector; An evaluation acquisition module, configured to perform joint analysis on the operation feature vector and the physiological parameter change sequence to obtain an evaluation index matrix; the evaluation index matrix at least includes an operation standardization score, a clinical decision-making rationality score, and an emergency response ability score; A training completion module, configured to generate an adaptive training scenario according to the evaluation index matrix, including: determining the ability defect information of the target user according to the evaluation index matrix; quickly obtaining the concurrent simulation disease information corresponding to the ability defect information according to the model-agnostic meta-learning algorithm; the complication simulation information includes a disease evolution tree; performing cross-modal enhanced rendering on the virtual patient three-dimensional model according to the concurrent simulation disease information; the cross-modal enhanced rendering at least includes converting the heart sound characteristics corresponding to the concurrent simulation disease information into a tactile feedback matrix and mapping the abnormal blood gas analysis values corresponding to the concurrent simulation disease information into a visualized metabolic pathway disorder heat map; generating an adaptive training scenario according to the tactile feedback matrix, the visualized metabolic pathway disorder heat map, and the virtual patient three-dimensional model; the adaptive training scenario includes complication simulation information, abnormal sign trigger information, and emergency state intervention information, so as to complete the training of the target user according to a preset mixed reality interaction device and the adaptive training scenario, including: acquiring the eye movement trajectory information, operation force information of the target user wearing the mixed reality interaction device, and virtual physiological parameter change information corresponding to the virtual patient three-dimensional model; dynamically adjusting the complexity of the mixed reality interaction device according to the eye movement trajectory information, operation force information, and virtual physiological parameter change information to complete the training of the target user.

2. The device according to claim 1, characterized in that, The generating the virtual patient three-dimensional model based on the imaging data, pathological data, and physiological signal data includes: Performing cross-modal alignment on the imaging data, pathological data, and physiological signal data; Constructing a dynamic volume rendering framework based on a neural radiance field, and generating tissue texture information corresponding to the imaging data, pathological data, and physiological signal data under the dynamic volume rendering framework; According to a physiological simulation algorithm based on finite element analysis, establishing a blood vessel wall stress-strain model and a dynamic response mechanism corresponding to the imaging data, pathological data, and physiological signal data according to the tissue texture information; Generate the three-dimensional virtual patient model according to the vascular wall stress-strain model and the dynamic response mechanism.

3. The device according to claim 2, wherein, The cross-modal alignment of the image data, pathological data, and physiological signal data includes: Perform multi-scale feature extraction on the image data according to a preset 3D Swin Transformer encoder to obtain image feature information; Perform temporal modeling on the pathological data according to a preset Bi-LSTM encoder to obtain pathological feature information; Decouple the time-frequency domain features of the physiological signal through a wavelet transform kernel to obtain physiological feature information; Complete the cross-modal alignment of the image data, pathological data, and physiological signal data according to the image feature information, pathological feature information, and physiological feature information.

4. The device according to claim 2, characterized in that, Before generating the three-dimensional virtual patient model according to the vascular wall stress-strain model and the dynamic response mechanism, it further includes: Perform authenticity verification on the vascular wall stress-strain model and the dynamic response mechanism according to a discriminator constructed based on a generative adversarial network; the discriminator includes a cyclic consistency loss function to ensure that the phase synchronization error between the respiratory rhythm and the cardiovascular pulsation corresponding to the three-dimensional virtual patient model is less than a preset error; If the authenticity verification passes, generate the three-dimensional virtual patient model according to the vascular wall stress-strain model and the dynamic response mechanism.

5. The device according to claim 1, characterized in that, The multi-scale feature encoding of the operation trajectory, instrument contact pressure, and biological tissue deformation parameters corresponding to the historical operation data to generate an operation feature vector with time series correlation includes: Perform kinematic modeling on the operation trajectory according to a preset spatio-temporal graph convolutional network to establish a dynamic spatial relationship graph between the end effector of the instrument and the anatomical landmark points; Construct a dual-channel attention mechanism to perform multi-physical field coupling analysis on the instrument contact pressure and biological tissue deformation parameters to generate a real-time deformation heat map of the instrument action area; Generate a hierarchical Transformer encoder according to the operation feature vector corresponding to the dynamic spatial relationship graph and the real-time deformation heat map.

6. The device according to claim 1, characterized in that, Inputting the operation feature vector into the three-dimensional virtual patient model to obtain the physiological parameter change sequence corresponding to the operation feature vector includes: Calculate the blood pressure waveform propagation information and the simulated airway pressure gradient information corresponding to the operation feature vector in the three-dimensional virtual patient model; Obtain a causal inference chain of preset operation features and pathophysiological responses; Generate the physiological parameter change sequence according to the blood pressure waveform propagation information, the simulated airway pressure gradient information, the operation feature vector, and the causal inference chain; the physiological parameter change sequence includes at least blood loss information and nerve reflex information.

7. The device according to claim 1, wherein The joint analysis of the operation feature vector and the physiological parameter change sequence to obtain an evaluation index matrix includes: Construct an anomaly detection module based on a graph neural network to determine at least one mutation point in the physiological parameter change sequence; Obtain the abnormal operation information corresponding to the mutation point in the operation feature vector; Generate the evaluation index matrix according to the abnormal operation information.

8. A medical skill assessment and training method based on virtual patients, characterized in that, Includes: Obtain the imaging data, pathological data, and physiological signal data corresponding to historical patients, and generate a virtual patient three-dimensional model based on the imaging data, pathological data, and physiological signal data; The virtual patient three-dimensional model has anatomical accuracy and physiological dynamic response characteristics; Obtain the historical operation data of the target user, perform multi-scale feature encoding on the operation trajectory, instrument contact pressure, and biological tissue deformation parameters corresponding to the historical operation data, and generate an operation feature vector with time series correlation; Input the operation feature vector into the virtual patient three-dimensional model to obtain the physiological parameter change sequence corresponding to the operation feature vector; the physiological parameter change sequence is used to determine the coupling response of the circulatory system, respiratory system, and nervous system corresponding to the operation feature vector; Perform joint analysis on the operation feature vector and the physiological parameter change sequence to obtain an evaluation index matrix; The evaluation index matrix at least includes an operation standardization score, a clinical decision-making rationality score, and an emergency response ability score; Generate an adaptive training scenario according to the evaluation index matrix, including: determining the ability defect information of the target user according to the evaluation index matrix; quickly obtaining the concurrent simulated disease information corresponding to the ability defect information according to the model-agnostic meta-learning algorithm; the complication simulation information includes a disease evolution tree; according to the concurrent simulated disease information, perform cross-modal enhanced rendering on the virtual patient three-dimensional model; the cross-modal enhanced rendering at least includes converting the heart sound characteristics corresponding to the concurrent simulated disease information into a tactile feedback matrix and mapping the abnormal blood gas analysis values corresponding to the concurrent simulated disease information into a visualized metabolic pathway disorder heat map; generate an adaptive training scenario according to the tactile feedback matrix, the visualized metabolic pathway disorder heat map, and the virtual patient three-dimensional model; the adaptive training scenario includes complication simulation information, abnormal sign trigger information, and emergency state intervention information, so as to train the target user according to a preset mixed reality interaction device and the adaptive training scenario, including: obtaining the eye movement trajectory information, operation force information of the target user wearing the mixed reality interaction device, and the virtual physiological parameter change information corresponding to the virtual patient three-dimensional model; dynamically adjust the complexity of the mixed reality interaction device according to the eye movement trajectory information, operation force information, and virtual physiological parameter change information to complete the training of the target user.

Images