Medical skill assessment and training method and device based on virtual patient

Abstract

The invention provides a medical skill assessment and training method and device based on a virtual patient, and relates to the technical field of digital computation.The method comprises the steps that image data, pathological data and physiological signal data corresponding to historical patients are obtained, and a virtual patient three-dimensional model is generated; performing multi-scale feature coding on the operation track, the instrument contact pressure and the biological tissue deformation parameter corresponding to the historical operation data of the target user to generate an operation feature vector with time sequence relevance; inputting the operation feature vector into a virtual patient three-dimensional model, and obtaining a physiological parameter change sequence corresponding to the operation feature vector; performing conjoint analysis on the operation feature vector and the physiological parameter change sequence to obtain an evaluation index matrix; generating an adaptive training scene according to the evaluation index matrix; the adaptive training scene comprises complication simulation information, abnormal sign triggering information and emergency state intervention information, so as to train the target user according to the mixed reality interaction equipment.

Description

Technical Field

[0001] The present application relates to the field of digital computing technology, and in particular to a medical skill assessment and training method and equipment based on virtual patients. Background Art

[0002] In the field of medical education and skills training, traditional teaching methods mainly rely on real patients or cadaver models for practical operations. These methods have many limitations, such as high cost, limited resources, and the inability to provide personalized and repeated training opportunities. With the development of computer technology and artificial intelligence, medical skills assessment and training methods based on virtual patients have emerged, aiming to provide a more efficient, economical and safe learning and training environment.

[0003] Traditional virtual patient training systems focus on simulating the physiological responses of patients, but often lack accurate feedback and evaluation mechanisms related to real operations. These systems cannot generate personalized training scenarios based on the user's operation data. Summary of the invention

[0004] The present application provides a medical skills assessment and training method and device based on virtual patients, aiming to solve the problem that high-concurrency traffic attacks (such as DDoS attacks, repeated requests caused by malicious plug-ins, etc.) have become an increasingly serious problem as financial services are digitized and online.

[0005] In a first aspect, the present application provides a medical skills assessment and training device based on a virtual patient, comprising: A data acquisition module, used to acquire image data, pathological data and physiological signal data corresponding to historical patients, and generate a virtual patient three-dimensional model according to the image data, pathological data and physiological signal data; the virtual patient three-dimensional model has anatomical accuracy and physiological dynamic response characteristics; An operation acquisition module is used to acquire the historical operation data of the target user, perform multi-scale feature encoding on the operation trajectory, instrument contact pressure and biological tissue deformation parameters corresponding to the historical operation data, and generate an operation feature vector with time series correlation; A vector input module, used to input the operation characteristic vector into the three-dimensional model of the virtual patient, and obtain a physiological parameter change sequence corresponding to the operation characteristic vector; the physiological parameter change sequence is used to determine the coupling response of the circulatory system, respiratory system and nervous system corresponding to the operation characteristic vector; An evaluation acquisition module is used to jointly analyze the operation feature vector and the physiological parameter change sequence to obtain an evaluation index matrix; the evaluation index matrix at least includes an operation standardization score, a clinical decision rationality score, and an emergency response capability score; The training completion module is used to generate an adaptive training scenario based on the evaluation indicator matrix; the adaptive training scenario includes complication simulation information, abnormal sign triggering information and emergency intervention information, so as to complete the training of the target user according to the preset mixed reality interaction device and adaptive training scenario.

[0006] In some embodiments, generating a three-dimensional model of a virtual patient based on the image data, pathological data and physiological signal data includes: performing cross-modal alignment on the image data, pathological data and physiological signal data; constructing a dynamic volume rendering framework based on a neural radiation field, and generating tissue texture information corresponding to the image data, pathological data and physiological signal data under the dynamic volume rendering framework; establishing a vascular wall stress-strain model and a dynamic response mechanism corresponding to the image data, pathological data and physiological signal data according to the tissue texture information based on a physiological simulation algorithm based on finite element analysis; and generating the three-dimensional model of the virtual patient based on the vascular wall stress-strain model and the dynamic response mechanism.

[0007] Exemplarily, the cross-modal alignment of the image data, pathological data and physiological signal data includes: performing multi-scale feature extraction on the image data according to a preset 3D Swin Transformer encoder to obtain image feature information; performing time series modeling on the pathological data according to a preset Bi-LSTM encoder to obtain pathological feature information; performing time-frequency domain feature decoupling on the physiological signal through a wavelet transform kernel to obtain physiological feature information; and completing cross-modal alignment of the image data, pathological data and physiological signal data according to the image feature information, pathological feature information and physiological feature information.

[0008] Exemplarily, before generating the virtual patient three-dimensional model according to the vascular wall stress-strain model and the dynamic response mechanism, it also includes: verifying the authenticity of the vascular wall stress-strain model and the dynamic response mechanism according to a discriminator constructed based on a generative adversarial network; the discriminator includes a cycle consistency loss function, which is used to ensure that the phase synchronization error between the respiratory rhythm and cardiovascular pulsation corresponding to the virtual patient three-dimensional model is less than a preset error; if the authenticity verification is passed, generating the virtual patient three-dimensional model according to the vascular wall stress-strain model and the dynamic response mechanism.

[0009] In some embodiments, the operation trajectory, instrument contact pressure and biological tissue deformation parameters corresponding to the historical operation data are multi-scale feature encoded to generate an operation feature vector with time series correlation, including: kinematic modeling of the operation trajectory according to a preset spatiotemporal graph convolutional network, and establishing a dynamic spatial relationship diagram between the instrument end effector and the anatomical landmarks; constructing a dual-channel attention mechanism, performing multi-physical field coupling analysis on the instrument contact pressure and biological tissue deformation parameters, and generating a real-time deformation heat map of the instrument action area; generating a hierarchical Transformer encoder according to the operation feature vector corresponding to the dynamic spatial relationship diagram and the real-time deformation heat map.

[0010] In some embodiments, the inputting of the operation characteristic vector into the virtual patient three-dimensional model to obtain the physiological parameter change sequence corresponding to the operation characteristic vector includes: calculating the blood pressure waveform propagation information and simulated airway pressure gradient information corresponding to the operation characteristic vector in the virtual patient three-dimensional model; obtaining the causal reasoning chain of preset operation characteristics and pathophysiological responses; generating the physiological parameter change sequence according to the blood pressure waveform propagation information and simulated airway pressure gradient information, the operation characteristic vector and the causal reasoning chain; the physiological parameter change sequence includes at least bleeding volume information and nerve reflex information.

[0011] In some embodiments, the joint analysis of the operation feature vector and the physiological parameter change sequence to obtain the evaluation index matrix includes: constructing an anomaly detection module based on a graph neural network to determine at least one mutation point in the physiological parameter change sequence; obtaining abnormal operation information corresponding to the mutation point in the operation feature vector; and generating the evaluation index matrix based on the abnormal operation information.

[0012] In some embodiments, generating an adaptive training scenario based on the evaluation indicator matrix includes: determining the capability deficiency information of the target user based on the evaluation indicator matrix; quickly acquiring concurrent simulated symptom information corresponding to the capability deficiency information based on a model-independent meta-learning algorithm; the complication simulation information includes a disease evolution tree; performing cross-modal enhanced rendering on the three-dimensional model of the virtual patient based on the concurrent simulated symptom information; the cross-modal enhanced rendering at least includes converting the heart sound features corresponding to the concurrent simulated symptom information into a tactile feedback matrix and mapping the blood gas analysis abnormal values ​​corresponding to the concurrent simulated symptom information into a visualized metabolic pathway disorder hotspot map; generating the adaptive training scenario based on the tactile feedback matrix, the visualized metabolic pathway disorder hotspot map and the three-dimensional model of the virtual patient.

[0013] In some embodiments, the training of the target user is completed according to a preset mixed reality interaction device and an adaptive training scenario, including: obtaining eye movement trajectory information, operation force information and virtual physiological parameter change information corresponding to the virtual patient three-dimensional model of the target user wearing the mixed reality interaction device; dynamically adjusting the complexity of the mixed reality interaction device according to the eye movement trajectory information, operation force information and the virtual physiological parameter change information to complete the training of the target user.

[0014] In a second aspect, the present application also provides a medical skills assessment and training method based on a virtual patient, comprising: Acquire image data, pathological data and physiological signal data corresponding to historical patients, and generate a virtual patient three-dimensional model according to the image data, pathological data and physiological signal data; the virtual patient three-dimensional model has anatomical accuracy and physiological dynamic response characteristics; Acquire historical operation data of the target user, perform multi-scale feature encoding on the operation trajectory, instrument contact pressure and biological tissue deformation parameters corresponding to the historical operation data, and generate an operation feature vector with time series correlation; Inputting the operation characteristic vector into the three-dimensional model of the virtual patient to obtain a physiological parameter change sequence corresponding to the operation characteristic vector; the physiological parameter change sequence is used to determine the coupled response of the circulatory system, respiratory system and nervous system corresponding to the operation characteristic vector; Performing a joint analysis on the operation characteristic vector and the physiological parameter change sequence to obtain an evaluation index matrix; the evaluation index matrix at least includes an operation standardization score, a clinical decision rationality score, and an emergency response capability score; An adaptive training scenario is generated according to the evaluation index matrix; the adaptive training scenario includes complication simulation information, abnormal sign triggering information and emergency intervention information, so as to train the target user according to a preset mixed reality interaction device and an adaptive training scenario.

[0015] In a third aspect, the present application also provides a computer device, comprising a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program and implement the medical skills assessment and training method based on a virtual patient as described above when executing the computer program.

[0016] In a fourth aspect, the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the processor implements the medical skills assessment and training method based on a virtual patient as described above.

[0017] This application discloses a medical skills assessment and training method and equipment based on virtual patients. First, a virtual patient model is constructed by collecting relevant medical data of historical patients, including imaging data (such as CT, MRI, etc.), pathological data (such as tissue section analysis) and physiological signal data (such as electrocardiogram, blood pressure, etc.). The above data are used to generate a virtual patient three-dimensional model with anatomical accuracy and physiological dynamic response characteristics. This means that the model is not only visually realistic, but also can simulate real physiological reactions. The historical operation data of the target user is analyzed, including operation trajectory, instrument contact pressure and biological tissue deformation parameters, and these data are encoded into operation feature vectors with time series correlation. The operation feature vector is input into the virtual patient model to simulate the corresponding physiological parameter change sequence, which can reflect the coupled response of the circulatory system, respiratory system and nervous system. The operation feature vector and the physiological parameter change sequence are jointly analyzed to generate a matrix containing multiple evaluation indicators, such as operation standardization score, clinical decision rationality score, emergency response capability score, etc. According to the evaluation indicator matrix, adaptive training scenarios are generated, including complication simulation information, abnormal sign trigger information and emergency intervention information, so as to improve the pertinence and effectiveness of training. Utilize preset mixed reality interactive devices and adaptive training scenarios to complete the training of target users and improve their medical skills.

[0018] By simulating real clinical environments and patient responses, this method can provide more realistic and personalized medical training, thereby improving the skills of medical students and doctors. Compared with traditional clinical internships, the use of virtual patient models can reduce the need for real patients, reduce training costs and potential medical risks. By simulating emergency conditions and complications, this method helps improve doctors' ability to respond to emergencies in real clinical environments. Adaptive training scenarios generated based on the evaluation indicator matrix can provide personalized training for the specific needs and skill levels of different users. By simulating different clinical scenarios, doctors can practice and improve their clinical decision-making skills without risk. This method can be used as a tool for medical research, helping researchers test new medical technologies and treatments in a simulated environment.

[0019] In summary, this application provides an innovative medical skills assessment and training method, which aims to improve the quality and efficiency of medical education while reducing costs and risks through virtual patient models and adaptive training scenarios.

[0020] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0022] Figure 1 is a schematic flow chart of the steps of a medical skill assessment and training method based on a virtual patient provided in one embodiment of the present application; Figure 2 is a schematic block diagram of a medical skill assessment and training device based on a virtual patient provided in one embodiment of the present application; Figure 3 It is a schematic block diagram of the structure of a computer device provided in one embodiment of the present application.

[0023] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. DETAILED DESCRIPTION

[0024] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0025] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0026] It should be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0027] It should also be understood that the term “and / or” used in the specification and appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0028] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0029] In the field of medical education and skills training, traditional teaching methods mainly rely on real patients or cadaver models for practical operations. These methods have many limitations, such as high cost, limited resources, and the inability to provide personalized and repeated training opportunities. With the development of computer technology and artificial intelligence, medical skills assessment and training methods based on virtual patients have emerged, aiming to provide a more efficient, economical and safe learning and training environment.

[0030] Traditional virtual patient training systems focus on simulating the physiological responses of patients, but often lack accurate feedback and evaluation mechanisms related to real operations. These systems cannot generate personalized training scenarios based on the user's operation data.

[0031] Therefore, a method is urgently needed to solve at least one of the above problems.

[0032] See also Figure 1 , Figure 1 This is a schematic flow chart of the steps of a medical skill assessment and training method based on a virtual patient provided in an embodiment of the present application. The method can be implemented by a computer device, which can be deployed on a single server or a server cluster. It can also be deployed on a handheld terminal, a laptop computer, a wearable device or a robot, etc.

[0033] like Figure 1 As shown, the medical skill assessment and training method based on a virtual patient specifically includes steps S101 to S105: Step S101, obtaining image data, pathological data and physiological signal data corresponding to historical patients, and generating a virtual patient three-dimensional model according to the image data, pathological data and physiological signal data; the virtual patient three-dimensional model has anatomical accuracy and physiological dynamic response characteristics.

[0034] Specifically, in step S101, the computer device not only needs to collect and process image, pathological and physiological signal data, but also needs to ensure that the integration of these data can reflect the patient's overall health status. This involves advanced data fusion technology, integrating data from different sources and types into a unified model. For example, image data may need to be enhanced through image processing technology, pathological data may need to be extracted through natural language processing technology, and physiological signal data may need to be filtered out noise and outliers through signal processing technology.

[0035] For example, advanced data fusion technologies, such as deep learning, are used to integrate data from different sources into a unified framework. Continuously optimize the 3D model to improve its simulation and response speed, making it closer to the physiological and anatomical characteristics of real patients. Develop complex algorithms to simulate the physiological responses of patients, including but not limited to the dynamic changes of the cardiovascular system, respiratory system, and nervous system.

[0036] This high-precision virtual patient model can be used as an important tool in medical education to help students better understand the structure and physiological functions of the human body. The virtual patient model can also be used in medical research to simulate physiological changes under different disease states and provide new ideas for the study of disease mechanisms.

[0037] Step S102: Acquire historical operation data of the target user, perform multi-scale feature encoding on the operation trajectory, instrument contact pressure and biological tissue deformation parameters corresponding to the historical operation data, and generate an operation feature vector with time series correlation.

[0038] Specifically, in step S102, the collection and analysis of operation data needs to take into account the complexity and diversity of the operation. This means that we need to develop sensors and algorithms that can capture subtle differences in operation. For example, during a surgical operation, the way the instrument is held, the cutting depth, and the speed are all key factors that affect the outcome of the operation, and these need to be captured by high-precision sensors.

[0039] Specific implementation method expansion: Sensor technology: Develop or integrate high-precision sensors, such as force feedback sensors, motion capture systems, etc., to capture every detail of user operations.

[0040] Data preprocessing: Preprocess the collected raw data, including filtering, normalization, and feature extraction, to improve the accuracy of subsequent analysis.

[0041] Feature vector construction: Construct a multi-dimensional feature vector, which includes not only the physical parameters of the operation, but also the spatiotemporal parameters of the operation and physiological feedback parameters.

[0042] Beneficial effects extension: Skill assessment: Through detailed analysis of operational data, the user's surgical skill level can be more accurately assessed, providing a basis for skill improvement.

[0043] Personalized training: Based on the user's operation feature vector, a personalized training plan can be designed to improve the pertinence and efficiency of training.

[0044] Step S103, input the operation feature vector into the virtual patient three-dimensional model to obtain the physiological parameter change sequence corresponding to the operation feature vector; the physiological parameter change sequence is used to determine the coupling response of the circulatory system, respiratory system and nervous system corresponding to the operation feature vector.

[0045] Specifically, in step S103, the operation feature vector is input into the virtual patient model, and it is necessary to ensure that the model can accurately respond to these operations. This involves complex physics engines and physiological simulation algorithms to ensure that every detail of the operation can be reflected in the model. By integrating advanced physics engines, the interaction between surgical instruments and virtual patient tissues is simulated, including cutting, suturing and other operations. Develop physiological simulation algorithms to simulate the impact of surgical operations on the patient's physiological state, such as bleeding, pain response, etc. Build a real-time feedback system to feed back changes in the model's physiological parameters to the user in real time to simulate a real surgical environment.

[0046] By simulating a real surgical environment, users can practice complex surgical operations without risk. By simulating the impact of surgical operations on the patient's physiological state, surgical risks can be assessed and provide a reference for surgical planning.

[0047] Step S104: jointly analyze the operation feature vector and the physiological parameter change sequence to obtain an evaluation index matrix; the evaluation index matrix at least includes an operation standardization score, a clinical decision rationality score, and an emergency response capability score.

[0048] Specifically, in step S104, the joint analysis needs to comprehensively consider the operational feature vector and the physiological parameter change sequence, which involves complex data analysis and machine learning techniques. For example, a deep learning network can be used to analyze these data and extract key evaluation indicators.

[0049] Use advanced data analysis techniques, such as cluster analysis and principal component analysis, to extract key information from large amounts of data. Develop machine learning models, such as neural networks and support vector machines, to perform classification and regression analysis on operational feature vectors and physiological parameter change sequences. Construct a comprehensive evaluation index matrix that includes not only the evaluation of operational skills, but also the evaluation of surgical risks and the evaluation of the patient's physiological status.

[0050] Through the comprehensive evaluation index matrix, the user's surgical skills and surgical risks can be comprehensively evaluated, providing a scientific basis for surgical decisions.

[0051] Step S105: Generate an adaptive training scenario based on the evaluation indicator matrix; the adaptive training scenario includes complication simulation information, abnormal sign triggering information, and emergency intervention information, so as to train the target user according to the preset mixed reality interactive device and the adaptive training scenario.

[0052] Specifically, step S105 aims to test the performance of the virtual patient model through a series of quantitative and qualitative evaluation methods, and optimize it according to the evaluation results. This may include the accuracy of the model, response time, user interaction experience, and stability and reliability of the model. Determine specific indicators for evaluating the performance of the virtual patient model, such as simulation accuracy, computational efficiency, user satisfaction, etc. Evaluate the accuracy of the model in terms of anatomical structure, physiological response, etc. by comparing with real patient data. Measure the response speed of the model to user operations to ensure real-time performance in practical applications. Test the stability of the model under long-term operation or high-load conditions.

[0053] Evaluate the practicality and interactivity of the model through the operational experience of actual users (such as medical students and doctors). Invite domain experts to review the medical accuracy and educational value of the model. Analyze test data and user feedback to identify the strengths and weaknesses of the model. Based on the evaluation results, adjust and optimize the model, such as improving the algorithm, adding new features, or optimizing the user interface. Use evaluation and optimization as a continuous iterative process to continuously improve the model to meet user needs and improve performance.

[0054] Through systematic evaluation and optimization, the quality of virtual patient models can be significantly improved, making them closer to the physiological and anatomical characteristics of real patients. The optimized model can provide a smoother and more intuitive user experience, improve user satisfaction and the practicality of the model. High-quality virtual patient models can be used as effective educational tools to help students and doctors better learn and master medical knowledge and skills. Continuous performance evaluation and optimization have promoted the development of related technologies, such as innovations in image processing, machine learning and other fields. By testing and training in a virtual environment, the risks and costs in actual surgery can be reduced and surgical safety can be improved. Virtual patient models with superior performance can be applied to a wider range of fields, such as clinical decision support, new drug testing, etc.

[0055] By implementing step S105, it is possible to ensure that the virtual patient model plays the greatest role in education, training and clinical applications, while promoting the development and innovation of related technologies.

[0056] In some embodiments, generating a three-dimensional model of a virtual patient based on the image data, pathological data and physiological signal data includes: performing cross-modal alignment on the image data, pathological data and physiological signal data; constructing a dynamic volume rendering framework based on a neural radiation field, and generating tissue texture information corresponding to the image data, pathological data and physiological signal data under the dynamic volume rendering framework; establishing a vascular wall stress-strain model and a dynamic response mechanism corresponding to the image data, pathological data and physiological signal data according to the tissue texture information based on a physiological simulation algorithm based on finite element analysis; and generating the three-dimensional model of the virtual patient based on the vascular wall stress-strain model and the dynamic response mechanism.

[0057] First, data from different sources need to be integrated. Imaging data (such as CT and MRI) provides information about the patient's anatomical structure, pathological data provides detailed information about tissue lesions, and physiological signal data (such as electrocardiogram and blood pressure) provides information about the patient's physiological state. These data need to be matched and integrated through cross-modal alignment technology to ensure their consistency in time and space.

[0058] De-noise, enhance and segment the image data to extract key anatomical structures; perform text analysis on pathological data to extract lesion features; filter and normalize physiological signal data to reduce the impact of noise and outliers.

[0059] A dynamic volume rendering framework is constructed using Neural Radiance Fields (NeRF) technology. NeRF is a deep learning method for 3D scene representation and rendering that can synthesize continuous volume fields of new views from sparse views. In this framework, the corresponding tissue texture information is generated based on the integrated image data, pathological data, and physiological signal data. This texture information includes not only the morphological characteristics of the tissue, but also its physiological characteristics, such as the distribution of blood vessels and blood flow status.

[0060] Finite Element Analysis (FEA) technology is used to establish a stress-strain model of the vascular wall based on tissue texture information. FEA is a numerical analysis method used to predict the response of a structure under physical action, such as stress, strain, and displacement. Based on the vascular wall stress-strain model, a dynamic response mechanism is established to simulate the changes of the vascular wall under different physiological conditions, such as expansion and contraction when blood pressure changes. The established vascular wall stress-strain model and dynamic response mechanism are integrated into the virtual patient 3D model to generate a virtual patient model with anatomical accuracy and physiological dynamic response characteristics. The accuracy and reliability of the model are verified by comparison with actual patient data, and optimization is performed based on feedback to improve the simulation and practicality of the model.

[0061] Through cross-modal alignment technology and the dynamic volume rendering framework of neural radiation fields, the generated virtual patient model has high-precision anatomical features and tissue texture information, which can more realistically simulate the patient's physiological state.

[0062] The vascular wall stress-strain model and dynamic response mechanism established using finite element analysis enable the virtual patient model to simulate real physiological dynamic changes, such as vascular expansion and contraction, providing a more realistic simulation environment for medical training and research.

[0063] Virtual patient models can be personalized based on different patient data and can be reused indefinitely without being limited by actual patient resources, thus improving the efficiency and accessibility of medical education and research.

[0064] By simulating rather than performing actual surgical operations, the virtual patient model provides a safe training environment, reduces surgical risks and costs, and, due to the reusability of the model, reduces the economic burden of medical education and skills training.

[0065] High-precision virtual patient models are not only of great value in medical education, helping students and doctors better understand and master the structure and physiological functions of the human body, but can also simulate physiological changes under different disease states in medical research, providing new tools and methods for disease mechanism research and new therapy development.

[0066] Exemplarily, the cross-modal alignment of the image data, pathological data and physiological signal data includes: performing multi-scale feature extraction on the image data according to a preset 3D Swin Transformer encoder to obtain image feature information; performing time series modeling on the pathological data according to a preset Bi-LSTM encoder to obtain pathological feature information; performing time-frequency domain feature decoupling on the physiological signal through a wavelet transform kernel to obtain physiological feature information; and completing cross-modal alignment of the image data, pathological data and physiological signal data according to the image feature information, pathological feature information and physiological feature information.

[0067] Multi-scale feature extraction is performed on image data (such as CT and MRI scans) by using the preset 3D Swin Transformer encoder. 3D Swin Transformer is a deep learning model based on the Transformer architecture, which is particularly suitable for processing 3D image data. Through the 3D Swin Transformer encoder, rich spatial features and contextual information can be extracted from image data, including multi-scale features such as shape, texture, and position.

[0068] The pathological data is modeled in time series by using the preset Bi-LSTM (bidirectional long short-term memory network) encoder. Bi-LSTM is a deep learning model that is particularly suitable for processing sequence data and can capture long-term dependencies in the data. Through the Bi-LSTM encoder, key time series features can be extracted from the pathological data, including the development process and change trend of the lesion.

[0069] The time-frequency domain features of physiological signal data (such as electrocardiogram and blood pressure monitoring) are decoupled through the wavelet transform kernel. Wavelet transform is an effective time-frequency analysis tool that can decompose signals into components of different scales and frequencies. Through the wavelet transform kernel, key time-frequency domain features can be extracted from physiological signal data, including heart rate, respiratory rate and rhythm.

[0070] Based on the extracted image feature information, pathological feature information and physiological feature information, feature fusion technology is used to complete the cross-modal alignment of image data, pathological data and physiological signal data. Through feature fusion technology, data from different modalities can be mapped into a unified feature space to ensure their consistency in time and space, providing accurate input data for subsequent virtual patient model generation.

[0071] By using 3D Swin Transformer encoder, Bi-LSTM encoder and wavelet transform kernel, key feature information can be extracted from data of different modalities, improving the accuracy and reliability of data fusion. Cross-modal alignment technology enables the virtual patient model to comprehensively consider imaging, pathological and physiological signal data, enhancing the generalization ability and applicability of the model. Accurate cross-modal alignment ensures the consistency of the virtual patient model in anatomical features, pathological changes and physiological dynamics, and improves the simulation and practicality of the model. Cross-modal alignment technology provides more accurate training data for medical education and skill training, which helps to improve the training effect and the accuracy of evaluation. Cross-modal alignment technology promotes the cross-fusion of data from different disciplines such as imaging, pathology and physiology, and provides new tools and methods for multidisciplinary cross-disciplinary research. Accurate cross-modal alignment technology helps doctors understand the health status of patients more comprehensively and improve the scientificity and accuracy of clinical decision-making.

[0072] Through the implementation of examples, the method can achieve accurate cross-modal alignment of imaging data, pathological data and physiological signal data, providing high-quality input data for the generation of virtual patient models, thereby improving the accuracy, simulation and practicality of the model.

[0073] Exemplarily, before generating the virtual patient three-dimensional model according to the vascular wall stress-strain model and the dynamic response mechanism, it also includes: verifying the authenticity of the vascular wall stress-strain model and the dynamic response mechanism according to a discriminator constructed based on a generative adversarial network; the discriminator includes a cycle consistency loss function, which is used to ensure that the phase synchronization error between the respiratory rhythm and cardiovascular pulsation corresponding to the virtual patient three-dimensional model is less than a preset error; if the authenticity verification is passed, generating the virtual patient three-dimensional model according to the vascular wall stress-strain model and the dynamic response mechanism.

[0074] In this example, a generative adversarial network (GAN) is used to build a discriminator whose task is to distinguish between real patient data and data generated by the model. The discriminator needs to be able to identify the authenticity of the vascular wall stress-strain model and dynamic response mechanism. The discriminator integrates a cycle consistency loss function, which is used to ensure that the physiological dynamics of the virtual patient model (such as respiratory rhythm and cardiovascular pulsation) are consistent with real patient data.

[0075] The data generated based on the vascular wall stress-strain model and dynamic response mechanism are input into the discriminator. The discriminator evaluates the authenticity of the input data, and the cycle consistency loss function ensures that the phase synchronization error between the physiological rhythm of the virtual patient model and the real patient data is controlled within the preset error range. If the discriminator determines that the generated data is not realistic enough, feedback will be used to optimize the vascular wall stress-strain model and dynamic response mechanism until the generated data passes the authenticity verification of the discriminator. Once the vascular wall stress-strain model and dynamic response mechanism pass the authenticity verification, a virtual patient 3D model can be generated based on these mechanisms. After the model is generated, the anatomical features and physiological dynamic response of the model are further refined to ensure that the model is close to the real patient in both visual and functional aspects.

[0076] The authenticity of the virtual patient model can be significantly improved by using the discriminator of the adversarial network for authenticity verification. This ensures that the model has a high degree of accuracy and reliability in simulating the physiological dynamics of real patients. The application of the cycle consistency loss function ensures that the phase synchronization error of the respiratory rhythm and cardiovascular pulsation of the virtual patient model with the real patient data is controlled within a preset range, thereby improving the physiological synchronization of the model. The feedback mechanism of the discriminator allows iterative optimization of the vascular wall stress-strain model and the dynamic response mechanism until the generated model meets the authenticity requirements, which helps to improve the efficiency and quality of the model generation process. The virtual patient model with authenticity verification can provide a more accurate training and simulation environment for medical education and research, thereby improving the training effect and the reliability of research. By performing authenticity verification before model generation, the risk and cost caused by finding that the model is inaccurate in actual application can be reduced. This provides a safer and more economical solution for medical education and clinical practice. The authenticity verification process ensures that the virtual patient model can be customized according to the individual's physiological characteristics, which is of great significance for the development of personalized medicine.

[0077] Through the implementation of examples, the method can generate a virtual patient model that is both realistic and highly personalized, providing a powerful tool for medical education, skills training and clinical research.

[0078] It should be noted that the Cycle Consistency Loss is a loss function used in the CycleGAN to ensure that the generated image remains consistent after conversion in both directions. In CycleGAN, there are two generators (G and F) and two discriminators (Dx and Dy), where G is responsible for converting the image from domain X to domain Y, and F is responsible for converting the image from domain Y back to domain X. The purpose of the Cycle Consistency Loss function is to ensure that the image converted by G and F is as similar as possible to the original image. The mathematical expression is as follows: Lcyc=Ex~pdata(x)[∣∣F(G(x))-x∣∣1]; Lcyc is the cycle consistency loss. G(x) is the result of the generator G converting image x from domain X to domain Y. ||| |||1 is the L1 norm, which is used to calculate the difference between two images.

[0079] In some embodiments, the operation trajectory, instrument contact pressure and biological tissue deformation parameters corresponding to the historical operation data are multi-scale feature encoded to generate an operation feature vector with time series correlation, including: kinematic modeling of the operation trajectory according to a preset spatiotemporal graph convolutional network, and establishing a dynamic spatial relationship diagram between the instrument end effector and the anatomical landmarks; constructing a dual-channel attention mechanism, performing multi-physical field coupling analysis on the instrument contact pressure and biological tissue deformation parameters, and generating a real-time deformation heat map of the instrument action area; generating a hierarchical Transformer encoder according to the operation feature vector corresponding to the dynamic spatial relationship diagram and the real-time deformation heat map.

[0080] The operation trajectory is kinematically modeled using a preset spatiotemporal graph convolutional network (ST-GCN). ST-GCN is a deep learning model that can process spatiotemporal data and is suitable for analyzing dynamic patterns in the operation trajectory. Through ST-GCN, a dynamic spatial relationship graph between the instrument end effector and anatomical landmarks is established to capture the movement trajectory of the instrument end effector and the relative position change with the anatomical structure during the operation.

[0081] A dual-channel attention mechanism is constructed to analyze the instrument contact pressure and biological tissue deformation parameters respectively. This mechanism can identify and emphasize the most important physical field features during the operation. Combined with the output of the dual-channel attention mechanism, a real-time deformation heat map of the instrument action area is generated to intuitively display the deformation and pressure distribution during the interaction between the instrument and the biological tissue.

[0082] Application of Hierarchical Transformer Encoder: Hierarchical Transformer Encoder is used to process the operation feature vectors corresponding to the dynamic spatial relationship graph and the real-time deformation heat map. Transformer encoder is good at processing sequence data and can capture the time series correlation in the feature vector. Through the hierarchical Transformer encoder, the operation feature vectors with time series correlation are generated, which integrate the multi-scale features of the operation trajectory, instrument contact pressure and biological tissue deformation parameters.

[0083] Through kinematic modeling and multi-physics field coupling analysis, the dynamic interaction between instruments and biological tissues during surgical operations can be understood and predicted more accurately, improving the accuracy and safety of surgical operations. The generated operation feature vector can provide rich feedback information for surgical training, help trainees better understand and master surgical skills, and enhance the effectiveness of surgical training. Real-time deformation thermal maps and operation feature vectors can assist doctors in surgical planning and execution, and optimize surgical strategies by real-time monitoring of key parameters during surgery. The surgical navigation system that integrates multi-scale feature encoding and operation feature vector generation methods can provide more intelligent navigation and assistance, and improve the intelligence level of the system. This embodiment provides a new surgical data analysis and feature extraction method, which provides new perspectives and tools for the innovation and development of surgical technology. The operation feature vector and real-time deformation thermal map provide detailed data support for the recording and evaluation of the surgical process, enhancing the traceability of the surgical process and the objectivity of the evaluation.

[0084] Through the implementation of the embodiments, technological progress can be achieved in multiple aspects such as surgical operation analysis, surgical training, surgical planning and surgical navigation, thereby improving the safety and effectiveness of surgery and laying the foundation for the future development of surgical technology.

[0085] In some embodiments, the inputting of the operation characteristic vector into the virtual patient three-dimensional model to obtain the physiological parameter change sequence corresponding to the operation characteristic vector includes: calculating the blood pressure waveform propagation information and simulated airway pressure gradient information corresponding to the operation characteristic vector in the virtual patient three-dimensional model; obtaining the causal reasoning chain of preset operation characteristics and pathophysiological responses; generating the physiological parameter change sequence according to the blood pressure waveform propagation information and simulated airway pressure gradient information, the operation characteristic vector and the causal reasoning chain; the physiological parameter change sequence includes at least bleeding volume information and nerve reflex information.

[0086] By inputting the operation feature vectors into the virtual patient 3D model, these feature vectors contain key information during the surgical operation, such as instrument kinematic characteristics, instrument contact pressure, and biological tissue deformation parameters.

[0087] In the virtual patient 3D model, the corresponding blood pressure waveform propagation information is calculated based on the operation feature vector. This involves simulating the flow and pressure changes of blood in the vascular system to reflect the impact of surgical operations on the circulatory system. Similarly, airway pressure gradient information is simulated to reflect the impact of surgical operations on the respiratory system, especially in cases involving airway or lung surgery.

[0088] Using medical knowledge and expert systems, a causal reasoning chain between preset operation characteristics and pathophysiological responses is established. This includes determining how specific surgical operations affect specific physiological parameters, such as blood pressure, heart rate, respiratory rate, etc. Combining blood pressure waveform propagation information, simulated airway pressure gradient information, operation feature vectors, and causal reasoning chains, a physiological parameter change sequence is generated. This sequence describes in detail the impact of surgical operations on the physiological state of the virtual patient model, including but not limited to bleeding volume information and neural reflex information.

[0089] By simulating the propagation of blood pressure waveforms and airway pressure gradients, and considering the causal relationship between operational characteristics and pathophysiological responses, the authenticity and accuracy of surgical simulation can be significantly improved. The sequence of physiological parameter changes provides important feedback information for surgical training, helping trainees understand the impact of surgical operations on the patient's physiological state, thereby enhancing the effectiveness of surgical training. The sequence of physiological parameter changes can help doctors predict and plan the physiological changes that may be caused by surgical operations before surgery, and optimize surgical strategies and decisions. By simulating the impact of surgical operations on physiological states, possible complications can be identified and prevented in advance, thereby improving the safety of surgery. The sequence of physiological parameter changes provides new tools and data for medical research and education, which helps to deeply understand the relationship between surgical operations and physiological responses. This embodiment involves knowledge and technology from multiple disciplines such as medicine, engineering, and computer science, promotes interdisciplinary cooperation, and promotes the development of related fields.

[0090] Through the implementation of the embodiments, technological progress can be achieved in multiple aspects such as surgical simulation, surgical training, surgical planning and decision-making, thereby improving the safety and effectiveness of surgery, while providing new perspectives and tools for medical research and education.

[0091] In some embodiments, the joint analysis of the operation feature vector and the physiological parameter change sequence to obtain the evaluation index matrix includes: constructing an anomaly detection module based on a graph neural network to determine at least one mutation point in the physiological parameter change sequence; obtaining abnormal operation information corresponding to the mutation point in the operation feature vector; and generating the evaluation index matrix based on the abnormal operation information.

[0092] A graph neural network is used to build an anomaly detection module that can process complex physiological parameter change sequence data and identify abnormal patterns or mutation points therein. By analyzing the time series data of physiological parameter change sequences, GNN can identify mutation points that represent significant changes in physiological state, which may indicate potential problems or complications in surgical operations. In the operation feature vector, according to the identified mutation points, the abnormal operation information corresponding to these mutation points is extracted. This may include improper use of instruments, excessive operation force, or deviation of surgical path. Using machine learning techniques, such as clustering or classification algorithms, features related to mutation points are extracted from the operation feature vector to identify specific abnormal operations. A series of evaluation indicators are defined to quantify the quality and effect of surgical operations, such as operation accuracy, safety, and efficiency. Based on the abnormal operation information and predefined evaluation indicators, an evaluation indicator matrix is ​​constructed. The matrix provides a quantitative evaluation for each surgical operation and helps identify strengths and weaknesses in the operation.

[0093] By identifying abnormal patterns in surgical operations, timely measures can be taken to prevent potential complications, thereby improving the safety of surgery. The evaluation indicator matrix provides detailed feedback on surgical operations, which helps doctors and medical students identify and improve operating skills and optimize surgical procedures. In surgical training, the evaluation indicator matrix can be used as a teaching tool to help trainees understand the standards and expectations of surgical operations and improve the effectiveness of training. The joint analysis method provides a new perspective to observe and understand the relationship between surgical operations and physiological responses, and promotes the innovation and development of surgical techniques. The evaluation indicator matrix provides a quantitative and objective method to evaluate the quality of surgical operations, reduces the influence of subjective judgment, and improves the accuracy and reliability of evaluation. By analyzing the operation feature vector and the sequence of physiological parameter changes, support can be provided for clinical decision-making, helping doctors make better decisions during surgery. The embodiments involve knowledge in multiple fields such as medicine, data science, and artificial intelligence, promote cooperation between different disciplines, and promote the development of related technologies.

[0094] Through the implementation of the embodiments, technological progress can be achieved in multiple aspects such as surgical safety, surgical operation optimization, surgical training, and surgical technology innovation, thereby improving the overall quality and effect of the surgery.

[0095] In some embodiments, generating an adaptive training scenario based on the evaluation indicator matrix includes: determining the capability deficiency information of the target user based on the evaluation indicator matrix; quickly acquiring concurrent simulated symptom information corresponding to the capability deficiency information based on a model-independent meta-learning algorithm; the complication simulation information includes a disease evolution tree; performing cross-modal enhanced rendering on the three-dimensional model of the virtual patient based on the concurrent simulated symptom information; the cross-modal enhanced rendering at least includes converting the heart sound features corresponding to the concurrent simulated symptom information into a tactile feedback matrix and mapping the blood gas analysis abnormal values ​​corresponding to the concurrent simulated symptom information into a visualized metabolic pathway disorder hotspot map; generating the adaptive training scenario based on the tactile feedback matrix, the visualized metabolic pathway disorder hotspot map and the three-dimensional model of the virtual patient.

[0096] Based on the evaluation indicator matrix, the target user's performance in the surgical operation is analyzed to determine their ability deficiency information. This may include problems such as insufficient operation accuracy and improper handling of specific complications. The model-independent meta-learning (MAML) algorithm is used to quickly adapt to and obtain concurrent simulated disease information corresponding to the target user's ability deficiency information. The MAML algorithm can quickly learn and adapt to new tasks from a small amount of data, and is suitable for the rapid generation of personalized training scenarios. Construct a disease evolution tree, describe possible complications and their development paths in detail, and provide a structured representation for simulated disease information.

[0097] By converting the heart sound features corresponding to the concurrent simulated disease information into a tactile feedback matrix. This involves processing the audio signal into a tactile signal to simulate the tactile feedback in the surgical operation. Mapping the abnormal values ​​of blood gas analysis corresponding to the concurrent simulated disease information into a visualized metabolic pathway obstacle heat map. This involves converting the biochemical data into a graphical representation to intuitively display the obstacles and abnormalities in the metabolic pathway.

[0098] The tactile feedback matrix is ​​applied to the virtual patient 3D model to simulate the tactile feedback during surgical operations and enhance the user's actual operation experience. The visualized metabolic pathway disorder heat map is superimposed on the virtual patient 3D model to provide visual feedback on metabolic abnormalities. The tactile feedback matrix, the visualized metabolic pathway disorder heat map and the virtual patient 3D model are combined to generate an adaptive training scenario. The scenario is customized for the target user's ability deficiencies and provides a personalized training experience.

[0099] By analyzing the evaluation index matrix and generating adaptive training scenarios, customized training content can be provided for each user to improve the personalization and pertinence of training. Adaptive training scenarios provide specialized training for user ability deficiencies, which helps users master surgical operation skills faster and improve the accuracy and efficiency of surgical operations. By simulating complications and providing tactile feedback, users can improve their awareness of possible problems during surgery, prevent and deal with complications in advance, and improve the safety of surgery. Cross-modal enhanced rendering technology provides multiple sensory feedback such as vision and touch, enhances the user's multimodal interactive experience, and improves the realism and immersion of training. The embodiment provides a new medical education method that improves the educational effect through simulation and multimodal feedback, and promotes the innovation and development of medical education. Adaptive training scenarios can help users better understand the impact of surgical operations on the patient's physiological state and improve surgical decision-making capabilities. The embodiment involves knowledge in multiple fields such as medicine, computer science, and human-computer interaction, promotes cooperation between different disciplines, and promotes the development of related technologies.

[0100] Through the implementation of the embodiments, technological progress can be achieved in multiple aspects such as surgical training, surgical safety, and medical education, thereby improving the overall quality and effect of surgery.

[0101] In some embodiments, the training of the target user is completed according to a preset mixed reality interaction device and an adaptive training scenario, including: obtaining eye movement trajectory information, operation force information and virtual physiological parameter change information corresponding to the virtual patient three-dimensional model of the target user wearing the mixed reality interaction device; dynamically adjusting the complexity of the mixed reality interaction device according to the eye movement trajectory information, operation force information and the virtual physiological parameter change information to complete the training of the target user.

[0102] Eye tracking technology in mixed reality interactive devices is used to obtain real-time eye tracking information of target users during training. This information can help analyze the user's attention focus and visual behavior patterns. The sensors of the interactive device are used to collect information on the force applied by the user when operating the virtual patient 3D model to evaluate the user's operating skills and force control ability. The simulated physiological parameter change information in the virtual patient model is recorded simultaneously, which reflects the impact of the user's operation on the virtual patient's state.

[0103] By developing an algorithm, the complexity of the mixed reality interactive device can be dynamically adjusted based on the user's eye movement information, operation force information, and virtual physiological parameter change information. This may include adjusting the quality of graphics rendering, the sensitivity of interactive feedback, or the difficulty of training scenarios.

[0104] The training content and difficulty are adjusted in real time according to the user's real-time performance and progress to provide a personalized training experience. For example, if the user performs poorly on a certain operation skill, the system can automatically increase the complexity of the relevant training module to strengthen the training of this skill.

[0105] During the training process, the system provides real-time feedback to help users understand their performance and progress. This includes feedback on operating skills, visual attention, and responses to changes in virtual physiological parameters. After the training, the system comprehensively analyzes the user's performance throughout the training process, provides an evaluation report on the training effect, and points out the user's strengths and areas for improvement.

[0106] By dynamically adjusting the complexity of mixed reality interactive devices, a tailored training plan can be provided to each user, improving the adaptability and personalization of training.

[0107] Mixed reality technology provides a highly interactive and immersive training environment, allowing users to participate more deeply in training, improving the effectiveness and attractiveness of training. By simulating real surgical environments and physiological reactions, users can practice surgical skills and decision-making in a safe environment, reducing errors and risks in actual surgery. Dynamically adjusting the difficulty of training can ensure the effective use of training resources, avoid users wasting time on training content that is too simple or too difficult, and improve training efficiency.

[0108] Mixed reality technology integrates multiple sensory information such as vision and touch, which helps users integrate and coordinate different sensory and motor skills during training. Training in a virtual environment can avoid the risks of actual surgery, while allowing users to repeat the same operation many times to deepen their understanding and mastery of surgical skills. By continuously collecting and analyzing users' training data, the development of users' skills can be evaluated, providing a basis for continuous skill improvement.

[0109] In some embodiments, a multi-user interactive system is developed to allow multiple users to participate in the same surgical simulation training scenario at the same time, simulating a real surgical team collaboration environment. Each user plays a specific role in the system (such as the surgeon, assistant, anesthesiologist, etc.), and is assigned different tasks and responsibilities based on the role. The system provides real-time voice and text communication tools to simulate communication and coordination during surgery, while providing feedback and evaluation of team collaboration.

[0110] By simulating real surgical team collaboration, users' team collaboration and communication skills can be improved. Users can be trained to effectively allocate tasks and coordinate resources under high-pressure environments to improve surgical efficiency.

[0111] In some embodiments, by adding emergency modules such as massive bleeding and cardiac arrest to surgical simulation training, users are trained to react quickly and handle emergencies. A decision tree guide is provided to help users make correct decisions in emergencies, while evaluating the effectiveness of their decisions. By simulating emergencies, the user's psychological stress response is evaluated and psychological adjustment training is provided. The user's ability to react quickly and handle emergencies correctly is trained to reduce mistakes in actual surgery. By simulating emergencies, the user's psychological tolerance and ability to cope with stress are improved.

[0112] In some embodiments, a skill evaluation algorithm is developed to quantify the user's surgical skills, such as cutting accuracy, suturing quality, etc. The method provides real-time operational feedback and improvement suggestions based on the evaluation results to help users adjust their operating skills in a timely manner. Based on the user's evaluation results, the system plans a personalized skill improvement path and recommends targeted training content. Through quantitative evaluation and real-time feedback, users are helped to improve their surgical skills. According to the user's actual situation, a personalized skill improvement path is provided to improve the pertinence and efficiency of training.

[0113] In some embodiments, a comprehensive knowledge base containing knowledge from multiple disciplines such as medicine, engineering, and psychology is constructed to provide rich background knowledge for surgical simulation training. Knowledge application training: In surgical simulation training, specific training modules are designed to require users to apply interdisciplinary knowledge to solve problems during surgery. Knowledge fusion evaluation: Evaluate the application of interdisciplinary knowledge by users in training, and provide feedback and suggestions for improvement. Through interdisciplinary knowledge application training, improve the user's ability to comprehensively apply knowledge from different disciplines to solve practical problems. Encourage users to try interdisciplinary knowledge integration in surgical simulation training to promote the innovation of new surgical techniques and methods.

[0114] See also Figure 2 , Figure 2 1 is a schematic diagram of the structure of a medical skill assessment and training device 200 based on a virtual patient provided in an embodiment of the present application. The medical skill assessment and training device 200 based on a virtual patient is used to execute the steps of the medical skill assessment and training method based on a virtual patient shown in the above embodiments. The medical skill assessment and training device 200 based on a virtual patient can be a single server or a server cluster, or the medical skill assessment and training device 200 based on a virtual patient can be a terminal, which can be a handheld terminal, a laptop computer, a wearable device or a robot, etc.

[0115] like Figure 2 As shown, the medical skill assessment and training device 200 based on a virtual patient includes: The data acquisition module 201 is used to acquire the image data, pathological data and physiological signal data corresponding to the historical patient, and generate a virtual patient three-dimensional model according to the image data, pathological data and physiological signal data; the virtual patient three-dimensional model has anatomical accuracy and physiological dynamic response characteristics; The operation acquisition module 202 is used to acquire the historical operation data of the target user, perform multi-scale feature encoding on the operation trajectory, instrument contact pressure and biological tissue deformation parameters corresponding to the historical operation data, and generate an operation feature vector with time series correlation; A vector input module 203 is used to input the operation feature vector into the three-dimensional model of the virtual patient to obtain a physiological parameter change sequence corresponding to the operation feature vector; the physiological parameter change sequence is used to determine the coupled response of the circulatory system, respiratory system and nervous system corresponding to the operation feature vector; An evaluation acquisition module 204 is used to jointly analyze the operation feature vector and the physiological parameter change sequence to obtain an evaluation index matrix; the evaluation index matrix at least includes an operation standardization score, a clinical decision rationality score, and an emergency response capability score; The training completion module 205 is used to generate an adaptive training scenario based on the evaluation index matrix; the adaptive training scenario includes complication simulation information, abnormal sign triggering information and emergency intervention information, so as to complete the training of the target user according to the preset mixed reality interaction device and adaptive training scenario.

[0116] In some embodiments, generating a three-dimensional model of a virtual patient based on the image data, pathological data and physiological signal data includes: performing cross-modal alignment on the image data, pathological data and physiological signal data; constructing a dynamic volume rendering framework based on a neural radiation field, and generating tissue texture information corresponding to the image data, pathological data and physiological signal data under the dynamic volume rendering framework; establishing a vascular wall stress-strain model and a dynamic response mechanism corresponding to the image data, pathological data and physiological signal data according to the tissue texture information based on a physiological simulation algorithm based on finite element analysis; and generating the three-dimensional model of the virtual patient based on the vascular wall stress-strain model and the dynamic response mechanism.

[0117] Exemplarily, the cross-modal alignment of the image data, pathological data and physiological signal data includes: performing multi-scale feature extraction on the image data according to a preset 3D Swin Transformer encoder to obtain image feature information; performing time series modeling on the pathological data according to a preset Bi-LSTM encoder to obtain pathological feature information; performing time-frequency domain feature decoupling on the physiological signal through a wavelet transform kernel to obtain physiological feature information; and completing cross-modal alignment of the image data, pathological data and physiological signal data according to the image feature information, pathological feature information and physiological feature information.

[0118] Exemplarily, before generating the virtual patient three-dimensional model according to the vascular wall stress-strain model and the dynamic response mechanism, it also includes: verifying the authenticity of the vascular wall stress-strain model and the dynamic response mechanism according to a discriminator constructed based on a generative adversarial network; the discriminator includes a cycle consistency loss function, which is used to ensure that the phase synchronization error between the respiratory rhythm and cardiovascular pulsation corresponding to the virtual patient three-dimensional model is less than a preset error; if the authenticity verification is passed, generating the virtual patient three-dimensional model according to the vascular wall stress-strain model and the dynamic response mechanism.

[0119] In some embodiments, the operation trajectory, instrument contact pressure and biological tissue deformation parameters corresponding to the historical operation data are multi-scale feature encoded to generate an operation feature vector with time series correlation, including: kinematic modeling of the operation trajectory according to a preset spatiotemporal graph convolutional network, and establishing a dynamic spatial relationship diagram between the instrument end effector and the anatomical landmarks; constructing a dual-channel attention mechanism, performing multi-physical field coupling analysis on the instrument contact pressure and biological tissue deformation parameters, and generating a real-time deformation heat map of the instrument action area; generating a hierarchical Transformer encoder according to the operation feature vector corresponding to the dynamic spatial relationship diagram and the real-time deformation heat map.

[0120] In some embodiments, the inputting of the operation characteristic vector into the virtual patient three-dimensional model to obtain the physiological parameter change sequence corresponding to the operation characteristic vector includes: calculating the blood pressure waveform propagation information and simulated airway pressure gradient information corresponding to the operation characteristic vector in the virtual patient three-dimensional model; obtaining the causal reasoning chain of preset operation characteristics and pathophysiological responses; generating the physiological parameter change sequence according to the blood pressure waveform propagation information and simulated airway pressure gradient information, the operation characteristic vector and the causal reasoning chain; the physiological parameter change sequence includes at least bleeding volume information and nerve reflex information.

[0121] In some embodiments, the joint analysis of the operation feature vector and the physiological parameter change sequence to obtain the evaluation index matrix includes: constructing an anomaly detection module based on a graph neural network to determine at least one mutation point in the physiological parameter change sequence; obtaining abnormal operation information corresponding to the mutation point in the operation feature vector; and generating the evaluation index matrix based on the abnormal operation information.

[0122] In some embodiments, generating an adaptive training scenario based on the evaluation indicator matrix includes: determining the capability deficiency information of the target user based on the evaluation indicator matrix; quickly acquiring concurrent simulated symptom information corresponding to the capability deficiency information based on a model-independent meta-learning algorithm; the complication simulation information includes a disease evolution tree; performing cross-modal enhanced rendering on the three-dimensional model of the virtual patient based on the concurrent simulated symptom information; the cross-modal enhanced rendering at least includes converting the heart sound features corresponding to the concurrent simulated symptom information into a tactile feedback matrix and mapping the blood gas analysis abnormal values ​​corresponding to the concurrent simulated symptom information into a visualized metabolic pathway disorder hotspot map; generating the adaptive training scenario based on the tactile feedback matrix, the visualized metabolic pathway disorder hotspot map and the three-dimensional model of the virtual patient.

[0123] In some embodiments, the training of the target user is completed according to a preset mixed reality interaction device and an adaptive training scenario, including: obtaining eye movement trajectory information, operation force information and virtual physiological parameter change information corresponding to the virtual patient three-dimensional model of the target user wearing the mixed reality interaction device; dynamically adjusting the complexity of the mixed reality interaction device according to the eye movement trajectory information, operation force information and the virtual physiological parameter change information to complete the training of the target user.

[0124] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described device and each module can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0125] The above-mentioned device can be implemented in the form of a computer program. Figure 3 Runs on the computer device shown.

[0126] See also Figure 3 , Figure 3 FIG. 1 is a schematic block diagram of a computer device in an embodiment. The computer device may be a server.

[0127] See also Figure 3 The computer device includes a processor, a memory and a network interface connected through a system bus, wherein the memory may include a non-volatile storage medium and an internal memory.

[0128] The non-volatile storage medium can store an operating system and a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor can execute any medical skill assessment and training method based on a virtual patient.

[0129] The processor is used to provide computing and control capabilities and support the operation of the entire computer equipment.

[0130] The internal memory provides an environment for the operation of the computer program in the non-volatile storage medium. When the computer program is executed by the processor, the processor can execute any medical skill assessment and training method based on a virtual patient.

[0131] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art will understand that Figure 3 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0132] It should be understood that the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0133] In one embodiment, the processor is used to run a computer program stored in the memory to implement the following steps: Acquire image data, pathological data and physiological signal data corresponding to historical patients, and generate a virtual patient three-dimensional model according to the image data, pathological data and physiological signal data; the virtual patient three-dimensional model has anatomical accuracy and physiological dynamic response characteristics; Acquire historical operation data of the target user, perform multi-scale feature encoding on the operation trajectory, instrument contact pressure and biological tissue deformation parameters corresponding to the historical operation data, and generate an operation feature vector with time series correlation; Inputting the operation characteristic vector into the three-dimensional model of the virtual patient to obtain a physiological parameter change sequence corresponding to the operation characteristic vector; the physiological parameter change sequence is used to determine the coupled response of the circulatory system, respiratory system and nervous system corresponding to the operation characteristic vector; Performing a joint analysis on the operation characteristic vector and the physiological parameter change sequence to obtain an evaluation index matrix; the evaluation index matrix at least includes an operation standardization score, a clinical decision rationality score, and an emergency response capability score; An adaptive training scenario is generated according to the evaluation index matrix; the adaptive training scenario includes complication simulation information, abnormal sign triggering information and emergency intervention information, so as to train the target user according to a preset mixed reality interaction device and an adaptive training scenario.

[0134] A computer-readable storage medium is also provided in an embodiment of the present application, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and the processor executes the program instructions to implement any one of the medical skills assessment and training methods based on virtual patients provided in the embodiments of the present application.

[0135] The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiment, such as a hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc., equipped on the computer device.

[0136] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A medical skills assessment and training device based on a virtual patient, characterized in that: include: A data acquisition module, used to acquire image data, pathological data and physiological signal data corresponding to historical patients, and generate a virtual patient three-dimensional model based on the image data, pathological data and physiological signal data; The virtual patient three-dimensional model has anatomical accuracy and physiological dynamic response characteristics; An operation acquisition module is used to acquire the historical operation data of the target user, perform multi-scale feature encoding on the operation trajectory, instrument contact pressure and biological tissue deformation parameters corresponding to the historical operation data, and generate an operation feature vector with time series correlation; A vector input module, used to input the operation characteristic vector into the three-dimensional model of the virtual patient, and obtain a physiological parameter change sequence corresponding to the operation characteristic vector; the physiological parameter change sequence is used to determine the coupling response of the circulatory system, respiratory system and nervous system corresponding to the operation characteristic vector; An evaluation acquisition module is used to jointly analyze the operation feature vector and the physiological parameter change sequence to obtain an evaluation index matrix; the evaluation index matrix at least includes an operation standardization score, a clinical decision rationality score, and an emergency response capability score; The training completion module is used to generate an adaptive training scenario based on the evaluation indicator matrix; the adaptive training scenario includes complication simulation information, abnormal sign triggering information and emergency intervention information, so as to complete the training of the target user according to the preset mixed reality interaction device and adaptive training scenario.

2. The device according to claim 1, characterized in that The generating of a virtual patient three-dimensional model according to the image data, pathological data and physiological signal data comprises: Performing cross-modality alignment on the imaging data, pathological data, and physiological signal data; Constructing a dynamic volume rendering framework based on neural radiation field, and generating tissue texture information corresponding to the image data, pathological data and physiological signal data under the dynamic volume rendering framework; According to a physiological simulation algorithm based on finite element analysis, a vascular wall stress-strain model and a dynamic response mechanism corresponding to the image data, pathological data and physiological signal data are established according to the tissue texture information; The virtual patient three-dimensional model is generated according to the vascular wall stress-strain model and the dynamic response mechanism.

3. The device according to claim 2, characterized in that The cross-modality alignment of the image data, pathological data and physiological signal data includes: Performing multi-scale feature extraction on the image data according to a preset 3D Swin Transformer encoder to obtain image feature information; Performing time series modeling on the pathological data according to a preset Bi-LSTM encoder to obtain pathological feature information; Decoupling the physiological signal in time and frequency domains through a wavelet transform kernel to obtain physiological characteristic information; The image data, pathological data and physiological signal data are aligned across modalities according to the image feature information, pathological feature information and physiological feature information.

4. The device according to claim 2, characterized in that Before generating the virtual patient three-dimensional model according to the vascular wall stress-strain model and the dynamic response mechanism, the method further includes: The authenticity of the vascular wall stress-strain model and the dynamic response mechanism is verified according to a discriminator constructed based on a generative adversarial network; the discriminator includes a cycle consistency loss function for ensuring that the phase synchronization error between the respiratory rhythm and the cardiovascular pulsation corresponding to the three-dimensional model of the virtual patient is less than a preset error; If the authenticity verification is passed, the virtual patient three-dimensional model is generated according to the vascular wall stress-strain model and the dynamic response mechanism.

5. The device according to claim 1, characterized in that The multi-scale feature encoding of the operation trajectory, instrument contact pressure and biological tissue deformation parameters corresponding to the historical operation data to generate an operation feature vector with time series correlation includes: Performing kinematic modeling on the operation trajectory according to a preset spatiotemporal graph convolutional network, and establishing a dynamic spatial relationship diagram between the instrument end effector and the anatomical landmarks; A dual-channel attention mechanism is constructed to perform multi-physical field coupling analysis on the contact pressure of the device and the deformation parameters of the biological tissue, and generate a real-time deformation thermal map of the device action area; A hierarchical Transformer encoder is generated according to the operation feature vector corresponding to the dynamic spatial relationship graph and the real-time deformation heat map.

6. The device according to claim 1, characterized in that The step of inputting the operation feature vector into the virtual patient three-dimensional model to obtain a physiological parameter change sequence corresponding to the operation feature vector includes: Calculating blood pressure waveform propagation information and simulated airway pressure gradient information corresponding to the operation characteristic vector in the virtual patient three-dimensional model; Obtain the causal reasoning chain between the pre-defined operational characteristics and pathophysiological responses; The physiological parameter change sequence is generated according to the blood pressure waveform propagation information and the simulated airway pressure gradient information, the operation characteristic vector and the causal reasoning chain; the physiological parameter change sequence at least includes bleeding volume information and nerve reflex information.

7. The device according to claim 1, characterized in that The step of jointly analyzing the operation feature vector and the physiological parameter change sequence to obtain an evaluation index matrix includes: Constructing an anomaly detection module based on a graph neural network to determine at least one mutation point in the physiological parameter change sequence; Acquire abnormal operation information corresponding to the mutation point in the operation feature vector; The evaluation index matrix is ​​generated according to the abnormal operation information.

8. The device according to claim 1, characterized in that Generating an adaptive training scenario according to the evaluation indicator matrix includes: Determining the capability deficiency information of the target user according to the evaluation index matrix; Rapidly obtaining concurrent simulated disease information corresponding to the capability deficiency information according to a model-independent meta-learning algorithm; the complication simulation information includes a disease evolution tree; According to the concurrent simulated disease information, cross-modal enhanced rendering is performed on the three-dimensional model of the virtual patient; the cross-modal enhanced rendering at least includes converting the heart sound features corresponding to the concurrent simulated disease information into a tactile feedback matrix and mapping the blood gas analysis abnormal values ​​corresponding to the concurrent simulated disease information into a visualized metabolic pathway disorder hotspot map; The adaptive training scenario is generated according to the tactile feedback matrix, the visualized metabolic pathway obstacle hot spot map and the virtual patient three-dimensional model.

9. The device according to claim 1, characterized in that The training of the target user is completed according to the preset mixed reality interaction device and the adaptive training scenario, including: Acquire eye movement trajectory information, operation force information, and virtual physiological parameter change information corresponding to the three-dimensional model of the virtual patient of the target user wearing the mixed reality interaction device; According to the eye movement trajectory information, the operation force information and the virtual physiological parameter change information, the complexity of the mixed reality interaction device is dynamically adjusted to complete the training of the target user.

10. A medical skills assessment and training method based on virtual patients, characterized in that: include: Acquire image data, pathological data and physiological signal data corresponding to historical patients, and generate a virtual patient three-dimensional model according to the image data, pathological data and physiological signal data; The virtual patient three-dimensional model has anatomical accuracy and physiological dynamic response characteristics; Acquire historical operation data of the target user, perform multi-scale feature encoding on the operation trajectory, instrument contact pressure and biological tissue deformation parameters corresponding to the historical operation data, and generate an operation feature vector with time series correlation; Inputting the operation characteristic vector into the three-dimensional model of the virtual patient to obtain a physiological parameter change sequence corresponding to the operation characteristic vector; the physiological parameter change sequence is used to determine the coupled response of the circulatory system, respiratory system and nervous system corresponding to the operation characteristic vector; Performing a joint analysis on the operation characteristic vector and the physiological parameter change sequence to obtain an evaluation index matrix; the evaluation index matrix at least includes an operation standardization score, a clinical decision rationality score, and an emergency response capability score; An adaptive training scenario is generated according to the evaluation index matrix; the adaptive training scenario includes complication simulation information, abnormal sign triggering information and emergency intervention information, so as to train the target user according to a preset mixed reality interaction device and an adaptive training scenario.

Images