Multi-modal blood transfusion special operation practical training method and system based on virtual reality

Abstract

The invention relates to the technical field of medical virtual reality training, and discloses a multi-mode blood transfusion special operation practical training method and system based on virtual reality. The method comprises the following steps: establishing a virtual reality blood transfusion special practical training basic scene; collecting multi-mode operation data; generating an operation normativity judgment mark set; predicting the risk by using a bidirectional long-short-term memory network and a time perception attention network; outputting a controlled operation process state track; and implementing comprehensive scoring in a segmented accumulation mode. The technical problem that in the prior art, a blood transfusion special operation training mode depending on static rule scoring is difficult to carry out real-time quantitative evaluation on the operation standardability of an operator especially under the condition of multi-step continuous intervention is solved. According to the method, the virtual practical training scene is constructed, and the multi-modal time sequence modeling and the risk prediction mechanism based on the AI model are combined, so that the dynamic risk perception of the whole process of the blood transfusion special practical training is realized, and the normalization of the practical training and the real-time performance of the evaluation are improved.

Description

Technical Field

[0001] This invention relates to the field of medical virtual reality training technology, and in particular to a multimodal blood transfusion operation training method and system based on virtual reality. Background Technology

[0002] Currently, blood transfusion-related procedures are an important part of clinical medical activities, involving multiple stages such as pre-transfusion compatibility testing (blood typing, irregular antibody screening, crossmatching), blood issuance, bedside transfusion, autologous blood collection, and management of abnormal situations. These procedures are characterized by numerous steps, strict standardization requirements, strong continuity, and a high concentration of safety risks. Especially in actual transfusion procedures, operators need to complete multiple delicate actions within a limited time, continuously monitoring their posture, sequence, and pace. Any non-standard behavior can easily lead to safety hazards.

[0003] Current training methods for blood transfusion procedures mainly include theoretical lectures, video demonstrations, and repetitive practice in physical models or simple simulation environments. With the development of virtual reality technology, some training systems have begun to incorporate virtual simulation techniques to visualize and interactively train blood transfusion or related medical procedures. However, these systems mostly focus on demonstrating the procedure or simulating single actions, typically only judging whether a step has been completed or whether an error condition has been triggered, making it difficult to reflect the continuous behavioral changes of the operator during actual operation. For example, most virtual training systems collect operational behavior data in a relatively singular dimension, often focusing only on isolated parameters such as position, angle, or time, lacking collaborative perception and unified modeling of multimodal operational data.

[0004] Furthermore, in actual blood transfusion procedures, operators often exhibit behaviors such as repeatedly adjusting their posture, hesitating in their actions, or having an unstable rhythm. While these behaviors may not immediately constitute a violation, they can gradually amplify risks during continuous procedures. Current technologies, lacking the ability to analyze the temporal characteristics of operational behaviors, typically cannot anticipate these risk evolution trends and can only provide alerts after obvious violations or operational failures. This is insufficient to meet the requirements of process safety and real-time controllability in specialized blood transfusion operation training.

[0005] Therefore, there is an urgent need for a multimodal transfusion operation training method based on virtual reality, which can collect multimodal data and perform time-series modeling of the entire transfusion operation process in a virtual training environment, and conduct process-level and objective evaluation of operation standardization, risk evolution trend and operation quality, so as to improve the safety and standardization of transfusion operation training. Summary of the Invention

[0006] To address the aforementioned technical shortcomings, the purpose of this invention is to propose a multimodal transfusion operation training method based on virtual reality. This method aims to solve the technical problem that existing transfusion operation training methods, which rely on static rule scoring, are particularly difficult to quantitatively assess in real time the operator's operational standardization under multi-step continuous intervention conditions.

[0007] To solve the above-mentioned technical problems, the present invention adopts the following technical solution: The present invention provides a multimodal blood transfusion operation training method based on virtual reality.

[0008] The virtual reality-based multimodal transfusion operation training method includes:

[0009] Step S10: Build a basic virtual reality blood transfusion training scenario, initialize the virtual and real space mapping of the basic virtual reality blood transfusion training scenario, and output the virtual blood transfusion training scenario model. And virtual and real space mapping parameter set ;

[0010] Step S20: Based on the virtual blood transfusion training scenario model And virtual and real space mapping parameter set Multimodal operation data at time t is acquired in real time, and a time-aligned multimodal feature fusion method is used to uniformly encode the multimodal operation data into multimodal temporal operation vectors. ;

[0011] Step S30: Based on multimodal timing operation vectors During blood transfusion training, the system performs real-time assessment of operational compliance and outputs a set of operational compliance assessment flags. ;

[0012] Step S40: Determine the standardization of operations based on the set of flags. An operational risk prediction model constructed using a bidirectional long short-term memory network (BiLSTM) and a time-aware attention network (TGAT) performs the task of predicting operational risk trends and outputs the state trajectory of the controlled operational process. ;

[0013] Step S50: Based on the controlled operation process state trajectory The operation quality score is processed using a segmented cumulative method, and the results of the blood transfusion-specific operation assessment are output.

[0014] Preferably, in step S10, a basic virtual reality blood transfusion training scenario is constructed, and the virtual-real space mapping of the basic virtual reality blood transfusion training scenario is initialized, and a virtual blood transfusion training scenario model is output. And virtual and real space mapping parameter set The steps specifically include:

[0015] Step S101: Based on the preset blood transfusion-specific skill operation specification database, build a virtual reality blood transfusion training basic scenario in the Unreal Engine UE5 system. The virtual reality blood transfusion training basic scenario includes a blood transfusion laboratory scenario, a blood transfusion dispensing room scenario, a ward scenario, a preoperative blood collection room scenario, and a virtual patient model.

[0016] Step S102: Introduce the simulated arm entity and the operating handle entity, and obtain the entity space coordinate system corresponding to the simulated arm entity and the operating handle entity. Obtain the virtual space coordinate system corresponding to the basic scenario of virtual reality blood transfusion training. Based on the physical space coordinate system and virtual space coordinate system Through a preset homogeneous transformation matrix Perform spatial unified mapping processing to output a virtual blood transfusion training scenario model. And virtual and real space mapping parameter set .

[0017] Preferably, in step S10, the entity space coordinate system and virtual space coordinate system The spatial mapping relationship is as follows: ;in, To represent the physical simulated arm or control handle in the virtual space coordinate system The horizontal coordinate of the three-dimensional spatial position; To represent the physical simulated arm or operating handle in the solid space coordinate system The vertical coordinate of the three-dimensional spatial position; To represent the physical simulated arm or operating handle in the solid space coordinate system The vertical coordinate of the three-dimensional spatial position; This represents the physical simulation arm or control handle in the solid space coordinate system. The horizontal coordinate of the three-dimensional spatial position; This represents the physical simulation arm or control handle in the solid space coordinate system. The vertical coordinate of the three-dimensional spatial position; This represents the physical simulation arm or control handle in the solid space coordinate system. The vertical coordinate of the three-dimensional spatial position;

[0018] Preset homogeneous transformation matrix The expression is: ;in, , , , , , , , and Cosine coefficients representing the directions between the physical coordinate system and the virtual coordinate system are used to describe spatial rotation relationships. and These represent the horizontal, vertical, and longitudinal translations of the origin in the physical coordinate system into the virtual coordinate system, respectively.

[0019] Preferably, in step S20, the virtual blood transfusion training scenario model is used. And virtual and real space mapping parameter set Multimodal operation data at time t is acquired in real time, and a time-aligned multimodal feature fusion method is used to uniformly encode the multimodal operation data into multimodal temporal operation vectors. The steps specifically include:

[0020] Step S201: Based on the virtual blood transfusion training scenario model And virtual and real space mapping parameter set Real-time acquisition of multimodal operation data at time t, including the real-time spatial orientation vector of the puncture needle. Real-time needle insertion force sequence during puncture Duration of tourniquet application Sequence of procedures related to blood transfusion ;in, ,in, Model of puncture needle in virtual blood transfusion training scenario The horizontal coordinate in space; Model of puncture needle in virtual blood transfusion training scenario The spatial ordinate; Model of puncture needle in virtual blood transfusion training scenario The vertical coordinate in space; The angle between the puncture needle and the virtual blood vessel axis;

[0021] Step S202: Target the real-time spatial attitude vector of the puncture needle Real-time needle insertion force sequence during puncture Duration of tourniquet application Sequence of procedures related to blood transfusion A time-aligned multimodal feature fusion method is used for unified encoding, outputting a multimodal temporal operation vector. .

[0022] Preferably, in step S30, the multimodal timing operation vector is used. During blood transfusion training, the system performs real-time assessment of operational compliance and outputs a set of operational compliance assessment flags. The steps specifically include:

[0023] Step S301: Construct a set of threshold parameters for operation specifications based on a pre-set database of blood transfusion-specific skills operation specifications. Operational specification threshold parameter set Including standard puncture angle Permissible instantaneous deviation of puncture angle Permissible tourniquet application time Standard procedure sequence for blood collection and puncture Simultaneously, temporal consistency constraint parameters are introduced, including the length of the puncture angle sliding time window. Threshold for puncture angle fluctuation and needle insertion force fluctuation threshold ;

[0024] Step S302: Perform timing manipulation on multimodal vectors Real-time spatial attitude vector of the puncture needle The angle between the puncture needle inside and the virtual blood vessel axis Length of sliding time window at puncture angle Internal calculation of the sliding mean of puncture angle Fluctuation range of puncture angle ;

[0025] Based on the sliding mean of puncture angle Fluctuation range of puncture angle The system performs a dual-determination process, which includes instantaneous deviation determination and timing stability determination; the instantaneous deviation determination outputs an instantaneous deviation flag. , Timing stability determination outputs a timing stability flag. , ;

[0026] Step S303: Based on the duration of tourniquet application and permitted tourniquet application time Perform tourniquet application timeliness determination and output tourniquet application timeliness indicator. , ;

[0027] Slide the time window length at the puncture angle Internally based on real-time needle insertion force sequence Needle force fluctuation was calculated using the standard deviation method. Based on needle insertion force fluctuation and needle insertion force fluctuation threshold Perform needle insertion force fluctuation determination and output needle insertion force fluctuation flag. , ;

[0028] Step S304: Based on the standard procedure sequence for blood collection and puncture. Sequence of procedures related to blood transfusion Perform a sequence consistency comparison and output a step consistency determination flag. ;

[0029] Step S305: Final determination based on instantaneous deviation flag Time-series stability flags Tourniquet application time limit marking Indicators of needle insertion force fluctuation Consistency judgment flag of steps Jointly construct and output a set of operational standardization judgment marks. .

[0030] Preferably, in step S40, the determination of operational standardization is based on the set of flags. An operational risk prediction model constructed using a bidirectional long short-term memory network (BiLSTM) and a time-aware attention network (TGAT) performs the task of predicting operational risk trends and outputs the state trajectory of the controlled operational process. The steps specifically include:

[0031] Step S401: Convert the multimodal timing operation vector and set of operational standardization judgment criteria The input is a pre-defined bidirectional long short-term memory (BiLSTM) network, which is used to process multimodal temporal manipulation vectors. and set of operational standardization judgment criteria The temporal evolution pattern is modeled bidirectionally, and the temporal hidden state sequence is output by the bidirectional long short-term memory network BiLSTM. ;

[0032] Step S402: Based on the temporal hidden state sequence and set of operational standardization judgment criteria Construct an operational risk time graph G, G = (V, E), where nodes V represent operational risk states; edges E represent the propagation relationships between adjacent time points; input the operational risk time graph G into a pre-defined time-aware attention network TGAT, and TGAT outputs a risk propagation node representation vector. ;

[0033] Step S403: Based on the risk propagation node representation vector The sigmoid function is used for risk probability mapping, and the output is correlated with the set of operational compliance judgment flags. The corresponding risk probability set is used to form the state trajectory of the controlled operation process. .

[0034] Preferably, in step S50, the controlled operation process state trajectory is determined. The steps for processing operational quality scores using a segmented cumulative method and outputting transfusion-specific operational assessment results specifically include:

[0035] Step S501: Obtain the controlled operation process state trajectory Where n is the total number of controlled transfusion process states in the controlled operation process state trajectory. Represents the state trajectory of a controlled operation process. Status of the i-th segment of the controlled blood transfusion process ; for the controlled operation process status trajectory Status of the i-th segment of the controlled blood transfusion process Preset corresponding state risk weight coefficients Among them, the status of controlled blood transfusion process At least three states are included: "Allow to continue", "Pause and correct", and "Interrupted", and the blood transfusion process status is controlled. Real-time announcements are broadcast via a voice system; further, the status of the i-th segment of the controlled transfusion process is calculated using a ratio method. The state trajectory of the entire controlled operation process Time allocation ;

[0036] Step S502: Based on state risk weight coefficient and time percentage A comprehensive scoring function for blood transfusion procedures is constructed using a linear weighting method, and its output corresponds to the status of the i-th segment of the controlled blood transfusion process. The corresponding score for the first blood transfusion procedure;

[0037] Step S503: Based on the controlled operation process state trajectory The transition relationship between adjacent states is calculated using the first-order Markov source entropy method to calculate the state of the i-th segment of the controlled blood transfusion process. The corresponding state transition entropy value It matches the standard entropy interval in the preset state distribution feature template library and outputs the state of the i-th segment of the controlled blood transfusion process. The corresponding score for the second blood transfusion procedure;

[0038] Step S504: Combine the scores of the first and second blood transfusion operations to output the blood transfusion operation evaluation results.

[0039] This invention also provides a virtual reality-based multimodal blood transfusion operation training system, comprising:

[0040] The scene construction and virtual-real mapping initialization module is used to build the basic scene for virtual reality blood transfusion training, initialize the virtual-real space mapping of the basic scene, and output the virtual blood transfusion training scene model. And virtual and real space mapping parameter set ;

[0041] The multimodal operation data acquisition and timing coding module is used for virtual blood transfusion training scenario models. And virtual and real space mapping parameter set Multimodal operation data at time t is acquired in real time, and a time-aligned multimodal feature fusion method is used to uniformly encode the multimodal operation data into multimodal temporal operation vectors. ;

[0042] The real-time operation compliance determination module is used for multimodal time-series operation vectors. During blood transfusion training, the system performs real-time assessment of operational compliance and outputs a set of operational compliance assessment flags. ;

[0043] The operational risk trend prediction module is used to predict operational risks based on a set of operational standardization criteria. An operational risk prediction model constructed using a bidirectional long short-term memory network (BiLSTM) and a time-aware attention network (TGAT) performs the task of predicting operational risk trends and outputs the state trajectory of the controlled operational process. ;

[0044] The operation quality assessment and scoring module is used to evaluate the status trajectory of controlled operation processes. The operation quality score is processed using a segmented cumulative method, and the results of the blood transfusion-specific operation assessment are output.

[0045] The present invention also provides a virtual reality-based multimodal blood transfusion operation training device, comprising: a memory, a processor, and a virtual reality-based multimodal blood transfusion operation training program stored in the memory and executable on the processor. When the virtual reality-based multimodal blood transfusion operation training program is executed by the processor, a virtual reality-based multimodal blood transfusion operation training method is implemented.

[0046] The present invention also provides a computer program product, including a virtual reality-based multimodal blood transfusion operation training program, which, when executed by a processor, implements the virtual reality-based multimodal blood transfusion operation training method.

[0047] The beneficial effects of this invention are as follows: By building a basic virtual reality blood transfusion training scenario and introducing a multimodal operation data time-series acquisition and fusion modeling mechanism, this invention can synchronously acquire and time-series analyze key operation parameters such as puncture angle, operation sequence, and duration in a virtual-real combined operation environment, thereby achieving real-time identification and immediate feedback on the standardization of blood transfusion operations, effectively reducing potential risks caused by non-standard operation procedures.

[0048] This invention incorporates the results of operational standardization judgments into an operational risk prediction model based on bidirectional long short-term memory networks and time-aware attention networks, and combines this with a segmented cumulative scoring mechanism to conduct process-level assessments of transfusion training operation quality. This approach can characterize the temporal evolution of operational risks during transfusion training and distinguish between short-term correctable deviations and sustained high-risk operational behaviors, thereby avoiding assessment biases caused by relying solely on results and improving the safety and reliability of transfusion-specific operational training. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 This is a flowchart illustrating the first embodiment of a virtual reality-based multimodal blood transfusion operation training method according to the present invention.

[0051] Figure 2 This is a schematic diagram showing the comparison of features before and after time alignment in the first embodiment of a virtual reality-based multimodal blood transfusion operation training method of the present invention.

[0052] Figure 3 This is a heat map diagram illustrating the real-time determination of multi-dimensional operational standardization in a multimodal blood transfusion operation training method based on virtual reality, according to the present invention.

[0053] Figure 4 This is a schematic diagram illustrating the continuous evolution trajectory of operational risks in a virtual reality-based multimodal blood transfusion operation training method according to the present invention.

[0054] Figure 5 This is a schematic diagram illustrating the real-time switching of the controlled blood transfusion process state in a virtual reality-based multimodal blood transfusion operation training method according to the present invention.

[0055] Figure 6 This is a schematic diagram of the equipment for a virtual reality-based multimodal blood transfusion operation training method according to the present invention. Detailed Implementation

[0056] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0057] Example 1: As Figure 1 The diagram shown is a flowchart of the first embodiment of the multimodal blood transfusion operation training method based on virtual reality of the present invention. The first embodiment of the multimodal blood transfusion operation training method based on virtual reality of the present invention is presented.

[0058] In the first embodiment, the virtual reality-based multimodal blood transfusion operation training method includes:

[0059] Step S10: Build a basic virtual reality blood transfusion training scenario, initialize the virtual and real space mapping of the basic virtual reality blood transfusion training scenario, and output the virtual blood transfusion training scenario model. And virtual and real space mapping parameter set ;

[0060] It should be noted that the virtual reality blood transfusion training basic scene constructed in this step is not only for visual display, but also focuses on the blood transfusion-related operation process. It performs structured modeling of the experimental table layout, instrument placement, operation space scale, and personnel interaction area involved in the blood transfusion training process, so that the virtual scene is consistent with the actual blood transfusion operation environment in terms of spatial structure and functional division, providing a unified spatial benchmark for the subsequent collection and mapping of multimodal operation data.

[0061] Understandably, virtual-real space mapping initialization refers to establishing a spatial correspondence between the basic virtual reality blood transfusion training scenario and the physical training equipment. By calibrating the pose parameters of the physical blood collection simulation arm, dedicated operating handle, and other interactive devices in physical space, it ensures that they have unique virtual coordinates and posture descriptions in the virtual blood transfusion training scenario model. This guarantees that the operator's actions in the physical space can be accurately mapped to the corresponding positions and objects in the virtual scene.

[0062] It should be understood that, unlike existing technologies that only rely on preset virtual models for simple interaction, this step initializes the virtual-real space mapping parameter set through virtual-real space mapping, enabling the virtual blood transfusion training scenario model to have dynamically updated spatial mapping capabilities. When the position of the physical equipment changes slightly or the operator's posture deviates, the operation object in the virtual scene can still be corrected in real time through the mapping parameters, thereby avoiding the problem of operation data distortion caused by spatial inconsistency and providing a reliable spatial basis for subsequent operation standardization judgment and risk trend analysis.

[0063] For example, in practical implementation, a blood collection simulation arm and operating table can be set up in a physical training environment. The spatial coordinates and orientation information of the blood collection simulation arm can be obtained through a spatial positioning device and written into the virtual transfusion training scenario model as part of the virtual-real space mapping parameter set. When the operator performs a puncture operation on the physical blood collection simulation arm, the position of their hand, the operating angle, and the contact position can be synchronously reflected in the corresponding blood vessel model position in the virtual reality transfusion training basic scenario, thereby achieving a consistent mapping between the physical operation action and the virtual operation object.

[0064] Step S20: Based on the virtual blood transfusion training scenario model And virtual and real space mapping parameter set Multimodal operation data at time t is acquired in real time, and a time-aligned multimodal feature fusion method is used to uniformly encode the multimodal operation data into multimodal temporal operation vectors. ;

[0065] It should be noted that multimodal operational data refers to operational information synchronously acquired by various types of data acquisition units during blood transfusion training. This includes, but is not limited to, information on the operator's hand position changes in space, information on the posture changes of the operating instruments, information on the duration of the operational actions, and information on interactive events corresponding to specific operational steps. The multimodal operational data originates from the collaborative perception of the virtual blood transfusion training scenario model and the physical interactive devices. Its data acquisition process is uniformly calibrated based on the virtual-real space mapping parameter set output in step S10, thereby ensuring the comparability of different modal data under the same spatial and temporal reference.

[0066] Understandably, by performing time-aligned feature fusion processing on multimodal operational data, multi-source operational data that originally differed in sampling frequency, triggering timing, and data format can be uniformly transformed into a temporal operational expression with a clear time sequence, fully reflecting the operator's continuous operational behavior during blood transfusion training. This multimodal temporal operational vector not only describes the individual operational action itself but also reflects the connection relationship and rhythmic changes between adjacent operational actions, providing a continuous and complete data foundation for subsequent analysis of operational standardization and risk evolution trends.

[0067] For example, such as Figure 2 As shown in the figure, the upper part of the figure displays the original multimodal operation data before time alignment. Continuous curves represent operator hand spatial position change data acquired by high-frequency sampling units; this type of data is characterized by high sampling frequency and strong temporal continuity. Discrete or scattered data represents operator device posture change data acquired by medium-frequency sampling units; its sampling period differs from the hand position data. Vertical pulse data represents interactive event information corresponding to specific operation steps; this type of data is usually generated by event triggering and appears discretely distributed on the time axis. It can be seen that the above multimodal operation data exhibit significant differences in sampling frequency, triggering timing, and temporal resolution. Directly analyzing them together can easily lead to problems where different operation information cannot correspond in the time dimension. The lower part of the figure shows the representation of multimodal temporal operation data after time alignment processing. By introducing a unified reference time axis, time interpolation, resampling, and event expansion processing are performed on multimodal operation data from different data acquisition units, mapping the originally dispersed multi-source operation data across different time scales to the same time reference, thus forming a multimodal temporal operation vector with a clear temporal order. The aligned multimodal temporal operation vector simultaneously contains hand spatial position features, instrument posture features, and operation event state features at the same time point, which can completely depict the instantaneous operation state of the operator during blood transfusion training.

[0068] Step S30: Based on multimodal timing operation vectors During blood transfusion training, the system performs real-time assessment of operational compliance and outputs a set of operational compliance assessment flags. ;

[0069] It should be noted that this invention does not employ a single threshold or single action triggering method to standardize transfusion operations. Instead, it introduces a multi-indicator joint judgment mechanism that combines instantaneous deviation judgment and temporal stability judgment, focusing on the most risk-sensitive key operational elements during transfusion training. Specifically, the puncture angle-related judgment not only focuses on whether the puncture angle exceeds the standard range at a certain moment, but also further introduces the mean and fluctuation amplitude within a sliding time window to describe the operator's overall stability over a continuous operational period. The tourniquet application and needle insertion force judgments supplement the operational standardization from the temporal and mechanical stability dimensions, respectively, thus constructing an operational standardization judgment framework covering multi-dimensional characteristics of "space-time-sequence".

[0070] Understandably, by jointly constructing indicators such as instantaneous deviation, temporal stability, tourniquet application timeliness, needle insertion force fluctuation, and procedural consistency, it is possible to conduct hierarchical and multi-dimensional real-time evaluation of operational behaviors during transfusion training. Even if an operation does not significantly exceed the threshold in a single indicator, it can still be identified through the joint judgment mechanism if it exhibits an abnormal trend in temporal continuity or operational sequence. This approach makes the judgment of operational standardization no longer dependent on "whether a single abnormal condition is triggered," but rather reflects the dynamic changes in the operator's overall operational quality, providing a more reliable input basis for subsequent risk trend prediction.

[0071] It should be understood that, compared with the common methods in the prior art of "judging based on a single frame attitude threshold" or "based on an overall score after the operation is completed", the temporal consistency constraint parameter introduced in step S30 of this invention enables the judgment of operation standardization to have process awareness capabilities. Traditional methods often cannot distinguish between "short-term correctable deviations" and "continuous non-standard operations", while this invention, through the synergistic constraint of the puncture angle sliding time window, angle fluctuation threshold, and needle insertion force fluctuation threshold, can effectively filter out occasional jitter or instantaneous accidental touches, while accurately identifying continuous unstable operations, thereby significantly reducing the misjudgment rate and improving the safety of training.

[0072] For example, such as Figure 3As shown, the horizontal axis represents the time evolution during the transfusion training process, and the vertical axis represents the multiple judgment dimensions involved in the operation standardization assessment. Different judgment dimensions correspond to the standardization of puncture angle, temporal stability, tourniquet application timeliness, needle insertion force fluctuation, and consistency of operation steps, respectively. Each pixel in the graph represents the degree of operational non-standardization at the corresponding time position and corresponding standardization judgment dimension. The color, from light to dark, indicates the trend of operation changing from a standard state to a non-standard state. Areas with continuously deepening colors indicate persistent non-standard behavior by the operator in the corresponding operation dimension over a continuous period, while scattered light-colored areas represent short-term fluctuations or correctable instantaneous deviations. It can be seen that in the middle time period of the graph, multiple standardization judgment dimensions simultaneously show continuous areas of high non-standardization, indicating that the operator not only deviated from the puncture angle during this period but also experienced increased fluctuations in needle insertion force, abnormal tourniquet application timeliness, and decreased consistency of operation steps.

[0073] Step S40: Determine the standardization of operations based on the set of flags. An operational risk prediction model constructed using a bidirectional long short-term memory network (BiLSTM) and a time-aware attention network (TGAT) performs the task of predicting operational risk trends and outputs the state trajectory of the controlled operational process. ;

[0074] It should be noted that in step S40, this invention does not perform static risk assessment solely based on the set of operational compliance judgment markers. Instead, it uses the multimodal temporal operation vector and the set of operational compliance judgment markers as joint inputs, introducing a bidirectional long short-term memory network (BiLSTM) and a time-aware attention network (TGAT) for phased modeling. BiLSTM focuses on characterizing the temporal correlation of operational behaviors, used to identify the time-dependent characteristics of risk evolution; TGAT focuses on characterizing the propagation strength and direction of risk states between adjacent time nodes, used to model the process of risk transformation from local non-compliant operations to overall process risk. Through this combination, the discrete judgment result of "whether the operation is compliant" can be further transformed into a continuous state expression of "how the risk evolves over time."

[0075] Understandably, during transfusion training, operational risks are often not triggered instantaneously by a single abnormal operation, but rather are procedural risks formed by the accumulation of multiple minor non-standard behaviors over time and their interaction between steps. For example, instability in the puncture angle within a short period may not directly lead to risk, but if it repeatedly occurs in adjacent time periods along with fluctuations in needle insertion force or excessive tourniquet application, it will significantly increase the probability of operational failure or concurrent risks. This invention uses BiLSTM to jointly model the forward and backward temporal information of operational behaviors, simultaneously considering both the "operational background before the abnormality" and the "risk development trend after the abnormality"; furthermore, it uses TGAT to adaptively allocate the propagation weight of risk in the time graph, enabling the risk prediction results to truly reflect the diffusion path and cumulative effect of risk in the transfusion process.

[0076] For example, such as Figure 4 As shown, this trajectory is generated by inputting a multimodal temporal operation vector and an operation compliance judgment flag set into a BiLSTM and TGAT, reflecting the dynamic trend of operational risk throughout the transfusion training process. It can be seen that the risk probability does not abruptly change at a single moment, but gradually increases or decreases with the superposition and propagation of minor non-compliant operations in multiple adjacent time periods, reflecting the temporal accumulation and stage correlation of operational risk. Figure 5 As shown, this is a controlled operation process state trajectory corresponding to the risk probability at each moment, with its state value continuously switching between "allow continuation," "pause and correct," and "interrupt." When the risk probability is in a low range, the process remains in the allow continuation state; when the risk probability rises and crosses a preset correction threshold, the process state switches to pause and correct in real time; when the risk probability further increases and exceeds the interruption threshold, the process state immediately switches to the interruption state. As the risk probability falls back, the process state can recover from the interruption or pause and correction state back to the allow continuation state. Figure 4 and Figure 5 It can be intuitively seen that this invention does not perform discrete, static, and ex-post judgment of operational risks. Instead, it uses BiLSTM to jointly model the time-dependent relationship and TGAT risk propagation path to achieve continuous quantitative expression of operational risks and real-time linkage switching of process control status. This allows it to more realistically reflect the entire process characteristics of risk formation, diffusion, and mitigation during controlled blood transfusion training.

[0077] Step S50: Based on the controlled operation process state trajectory The operation quality score is processed using a segmented cumulative method, and the results of the blood transfusion-specific operation assessment are output.

[0078] It should be noted that this invention does not perform a one-time score based on the risk probability or the final process state at a single moment. Instead, it uses the controlled operation process state trajectory output in step S40 as the scoring line, dividing the entire transfusion training process into multiple continuous scoring segments according to the process timeline, and performing segmented cumulative scoring processing within each segment based on the corresponding process state type. This method ensures that the operation quality assessment results truly reflect the operator's overall performance at different risk stages, rather than just reflecting a local or instantaneous state.

[0079] Understandably, the segmented accumulation method refers to: using the state transition nodes in the controlled operation process state trajectory as segment boundaries, and considering the time interval of continuous operation in the same process state as a scoring sub-segment; for each scoring sub-segment, calculating the corresponding stage score contribution value based on its corresponding process state type (allowed to continue, paused and corrected, or interrupted) and the duration of the sub-segment; and then accumulating the stage score contribution values ​​of each scoring sub-segment in chronological order to form a comprehensive operational quality score result covering the entire transfusion training process. This mechanism can avoid the disproportionately amplified impact of short-term high-risk fluctuations or instantaneous interruptions on the overall evaluation result.

[0080] It should be understood that the scoring accumulation strategy is designed differently for different process states: when the process state is "allowed to continue," the scoring is mainly positive, reflecting the operator's stability and continuous compliance in the low-risk, standardized operation phase; when the process state is "paused and corrected," the scoring accumulation rate is significantly reduced or turns into punitive accumulation, reflecting the operator's ability to control operational deviations in the medium-risk phase; when the process state is "interrupted," the corresponding scoring segment can directly trigger strong penalties or deduction rules, reflecting the substantial impact of high-risk operations on the overall operational quality. All of the above scoring mechanisms are executed segment by segment based on the process state trajectory, rather than solely based on the final state determination result. For example, in a controlled transfusion training process, the operator's process state remained in "allowed to continue" for several time periods in the early stages, with corresponding scoring segments continuously accumulating positive scores; subsequently, due to fluctuations in the puncture angle and needle insertion rhythm, the process state switched to "paused and corrected" multiple times, reducing the scoring accumulation rate in the corresponding segments; if the risk further spreads and triggers the "interrupted" state in certain time periods, a significant deduction occurs in the corresponding scoring segment. Ultimately, even if the operator returns to the "allow to continue" status in a later stage, the cumulative impact of the early and mid-stage risks will still be fully retained in the final transfusion-specific operation assessment results, so that the assessment results can comprehensively reflect the operator's overall operational quality level throughout the entire transfusion process.

[0081] Example 2: Furthermore, the present invention provides a virtual reality-based multimodal blood transfusion operation training system, employing a virtual reality-based multimodal blood transfusion operation training method from the above embodiments, which can solve the technical problem of virtual reality-based multimodal blood transfusion operation training. Compared with the prior art, the beneficial effects of the virtual reality-based multimodal blood transfusion operation training system provided by the present invention are the same as those of the virtual reality-based multimodal blood transfusion operation training method provided in the above embodiments, and other technical features of the virtual reality-based multimodal blood transfusion operation training system are the same as those disclosed in the above embodiments, and will not be repeated here.

[0082] Example 3: This invention provides a multimodal blood transfusion operation training device based on virtual reality. Please refer to... Figure 6A virtual reality-based multimodal blood transfusion operation training device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the virtual reality-based multimodal blood transfusion operation training method described in Embodiment 1 above. The virtual reality-based multimodal blood transfusion operation training device in this embodiment may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital radio receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. This virtual reality-based multimodal blood transfusion operation training device is merely an example and should not impose any limitations on the functionality and scope of use of this embodiment. A virtual reality-based multimodal blood transfusion operation training device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory 1002 or a program loaded from a storage device 1003 into a random access memory 1004. The random access memory 1004 also stores various programs and data required for the operation of the virtual reality-based multimodal blood transfusion operation training device. The processing unit 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An I / O interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. The communication device 1009 allows a virtual reality-based multimodal blood transfusion operation training device to wirelessly or wiredly communicate with other devices to exchange data. While the figure shows a virtual reality-based multimodal blood transfusion operation training device with various systems, it should be understood that implementing or having all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0083] Example 4: This invention also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described method for multimodal blood transfusion operation training based on virtual reality. The computer program product provided by this invention can solve the technical problem of multimodal blood transfusion operation training based on virtual reality. Compared with the prior art, the beneficial effects of the computer program product provided by this invention are the same as those of the multimodal blood transfusion operation training method based on virtual reality provided in the above embodiments, and will not be repeated here.

[0084] In particular, according to the embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this invention.

[0085] It should be understood that the various parts disclosed in this invention can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics may be combined in any suitable manner in one or more embodiments or examples.

[0086] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A multimodal transfusion operation training method based on virtual reality, characterized in that, The methods include: Step S10: Construct a basic virtual reality blood transfusion training scenario, initialize the virtual and real space mapping of the basic virtual reality blood transfusion training scenario, and output the virtual blood transfusion training scenario model. And virtual and real space mapping parameter set ; Step S20: Based on the virtual blood transfusion specialized training scenario model And virtual and real space mapping parameter set Multimodal operation data at time t is acquired in real time, and a time-aligned multimodal feature fusion method is used to uniformly encode the multimodal operation data into multimodal temporal operation vectors. ; Step S30: Based on multimodal timing operation vectors In the process of blood transfusion training, a real-time judgment task is performed to determine the standardization of operations, and a set of judgment criteria for standardization of operations is output. ; Step S40: Determine the standardization of operations based on the set of flags. An operational risk prediction model constructed using a bidirectional long short-term memory network (BiLSTM) and a time-aware attention network (TGAT) performs the task of predicting operational risk trends and outputs the state trajectory of the controlled operational process. ; Step S50: Based on the controlled operation process state trajectory The operation quality score is processed using a segmented cumulative method, and the results of the blood transfusion-specific operation assessment are output.

2. The multimodal blood transfusion operation training method based on virtual reality as described in claim 1, characterized in that, In step S10, a basic virtual reality blood transfusion training scenario is built, and the virtual-real space mapping of the basic virtual reality blood transfusion training scenario is initialized, outputting a virtual blood transfusion training scenario model. And virtual and real space mapping parameter set The steps specifically include: Step S101: Based on the preset blood transfusion-specific skill operation specification database, build a virtual reality blood transfusion training basic scenario in the Unreal Engine UE5 system. The virtual reality blood transfusion training basic scenario includes a blood transfusion laboratory scenario, a blood transfusion dispensing room scenario, a ward scenario, a preoperative blood collection room scenario, and a virtual patient model. Step S102: Introduce the simulated arm entity and the operating handle entity, and obtain the entity space coordinate system corresponding to the simulated arm entity and the operating handle entity. Obtain the virtual space coordinate system corresponding to the basic scenario of virtual reality blood transfusion training. Based on the physical space coordinate system and virtual space coordinate system Through a preset homogeneous transformation matrix Perform spatial unified mapping processing to output a virtual blood transfusion training scenario model. And virtual and real space mapping parameter set .

3. The multimodal blood transfusion operation training method based on virtual reality as described in claim 2, characterized in that, In step S10, the entity space coordinate system and virtual space coordinate system The spatial mapping relationship is as follows: ;in, To represent the physical simulated arm or control handle in the virtual space coordinate system The horizontal coordinate of the three-dimensional spatial position; To represent the physical simulated arm or operating handle in the solid space coordinate system The vertical coordinate of the three-dimensional spatial position; To represent the physical simulated arm or operating handle in the solid space coordinate system The vertical coordinate of the three-dimensional spatial position; This represents the physical simulation arm or control handle in the solid space coordinate system. The horizontal coordinate of the three-dimensional spatial position; This represents the physical simulation arm or control handle in the solid space coordinate system. The vertical coordinate of the three-dimensional spatial position; This represents the physical simulation arm or control handle in the solid space coordinate system. The vertical coordinate of the three-dimensional spatial position; Preset homogeneous transformation matrix The expression is: ;in, , , , , , , , and Cosine coefficients representing the directions between the physical coordinate system and the virtual coordinate system are used to describe spatial rotation relationships. and These represent the horizontal, vertical, and longitudinal translations of the origin in the physical coordinate system into the virtual coordinate system, respectively.

4. The multimodal blood transfusion operation training method based on virtual reality as described in claim 1, characterized in that, In step S20, based on the virtual blood transfusion training scenario model And virtual and real space mapping parameter set Multimodal operation data at time t is acquired in real time, and a time-aligned multimodal feature fusion method is used to uniformly encode the multimodal operation data into multimodal temporal operation vectors. The steps specifically include: Step S201: Based on the virtual blood transfusion training scenario model And virtual and real space mapping parameter set Real-time acquisition of multimodal operation data at time t, including the real-time spatial orientation vector of the puncture needle. Real-time needle insertion force sequence during puncture Duration of tourniquet application Sequence of procedures related to blood transfusion ;in, ,in, Model of puncture needle in virtual blood transfusion training scenario The horizontal coordinate in space; Model of puncture needle in virtual blood transfusion training scenario The spatial ordinate; Model of puncture needle in virtual blood transfusion training scenario The vertical coordinate in space; The angle between the puncture needle and the virtual blood vessel axis; Step S202: Target the real-time spatial attitude vector of the puncture needle Real-time needle insertion force sequence during puncture Duration of tourniquet application Sequence of procedures related to blood transfusion A time-aligned multimodal feature fusion method is used for unified encoding, outputting a multimodal temporal operation vector. .

5. The multimodal transfusion operation training method based on virtual reality as described in claim 4, characterized in that, In step S30, based on the multimodal timing operation vector During blood transfusion-specific training, a real-time assessment task is performed to determine operational compliance, and a set of operational compliance assessment criteria is output. The steps specifically include: Step S301: Construct a set of threshold parameters for operation procedures based on a pre-set database of blood transfusion-specific operation procedures. Operational specification threshold parameter set Including standard puncture angle Permissible instantaneous deviation of puncture angle Permissible tourniquet application time Standard procedure sequence for blood collection and puncture Simultaneously, temporal consistency constraint parameters are introduced, including the length of the puncture angle sliding time window. Threshold for puncture angle fluctuation and needle insertion force fluctuation threshold ; Step S302: Perform timing manipulation on multimodal vectors Real-time spatial attitude vector of the puncture needle The angle between the puncture needle inside and the virtual blood vessel axis Length of sliding time window at puncture angle Internal calculation of the sliding mean of puncture angle Fluctuation range of puncture angle ; Based on the sliding mean of puncture angle Fluctuation range of puncture angle The system performs a dual-determination process, which includes instantaneous deviation determination and timing stability determination; the instantaneous deviation determination outputs an instantaneous deviation flag. , Timing stability determination outputs a timing stability flag. , ; Step S303: Based on the duration of tourniquet application and permitted tourniquet application time Perform tourniquet application timeliness determination and output tourniquet application timeliness indicator. , ; Slide the time window length at the puncture angle Internally based on real-time needle insertion force sequence Needle force fluctuation was calculated using the standard deviation method. Based on needle insertion force fluctuation and needle insertion force fluctuation threshold Perform needle insertion force fluctuation determination and output needle insertion force fluctuation flag. , ; Step S304: Based on the standard procedure sequence for blood collection and puncture. Sequence of procedures related to blood transfusion Perform a sequence consistency comparison and output a step consistency determination flag. ; Step S305: Final determination based on instantaneous deviation flag Time-series stability flags Tourniquet application time limit marking Indicators of needle insertion force fluctuation Consistency judgment flag of steps Jointly construct and output a set of operational standardization judgment marks. .

6. The multimodal blood transfusion operation training method based on virtual reality as described in claim 1, characterized in that, In step S40, the judgment flag set based on the operation standardization is determined. An operational risk prediction model constructed using a bidirectional long short-term memory network (BiLSTM) and a time-aware attention network (TGAT) performs the task of predicting operational risk trends and outputs the state trajectory of the controlled operational process. The steps specifically include: Step S401: Convert the multimodal timing operation vector and set of operational standardization judgment criteria The input is a pre-defined bidirectional long short-term memory (BiLSTM) network, which is used to process multimodal temporal manipulation vectors. and set of operational standardization judgment criteria The temporal evolution pattern is modeled bidirectionally, and the temporal hidden state sequence is output by the bidirectional long short-term memory network BiLSTM. ; Step S402: Based on the temporal hidden state sequence and set of operational standardization judgment criteria Construct an operational risk time graph G, G = (V, E), where nodes V represent operational risk states; edges E represent the propagation relationships between adjacent time points; input the operational risk time graph G into a pre-defined time-aware attention network TGAT, and TGAT outputs a risk propagation node representation vector. ; Step S403: Based on the risk propagation node representation vector The sigmoid function is used for risk probability mapping, and the output is correlated with the set of operational compliance judgment flags. The corresponding risk probability set is used to form the state trajectory of the controlled operation process. .

7. The multimodal blood transfusion operation training method based on virtual reality as described in claim 1, characterized in that, In step S50, according to the controlled operation process state trajectory The steps for processing operational quality scores using a segmented cumulative method and outputting transfusion-specific operational assessment results specifically include: Step S501: Obtain the controlled operation process state trajectory Where n is the total number of controlled transfusion process states in the controlled operation process state trajectory. Represents the state trajectory of a controlled operation process. Status of the i-th segment of the controlled blood transfusion process ; for the controlled operation process status trajectory Status of the i-th segment of the controlled blood transfusion process Preset corresponding state risk weight coefficients Among them, the status of controlled blood transfusion process At least three states are included: "Allow to continue", "Pause and correct", and "Interrupted", and the blood transfusion process status is controlled. Real-time announcements are broadcast via a voice system; further, the status of the i-th segment of the controlled transfusion process is calculated using a ratio method. The state trajectory of the entire controlled operation process Time allocation ; Step S502: Based on state risk weight coefficient and time percentage A comprehensive scoring function for blood transfusion procedures is constructed using a linear weighting method, and its output corresponds to the status of the i-th segment of the controlled blood transfusion process. The corresponding score for the first blood transfusion procedure; Step S503: Based on the controlled operation process state trajectory The transition relationship between adjacent states is calculated using the first-order Markov source entropy method to calculate the state of the i-th segment of the controlled blood transfusion process. The corresponding state transition entropy value It matches the standard entropy interval in the preset state distribution feature template library and outputs the state of the i-th segment of the controlled blood transfusion process. The corresponding score for the second blood transfusion procedure; Step S504: Combine the scores of the first and second blood transfusion operations to output the blood transfusion operation evaluation results.

8. A virtual reality-based multimodal blood transfusion operation training system, applied to the virtual reality-based multimodal blood transfusion operation training method according to any one of claims 1 to 7, characterized in that, The virtual reality-based multimodal blood transfusion operation training system includes: The scene construction and virtual-real mapping initialization module is used to build the basic scene for virtual reality blood transfusion training, initialize the virtual-real space mapping of the basic scene, and output the virtual blood transfusion training scene model. And virtual and real space mapping parameter set ; The multimodal operation data acquisition and timing coding module is used for virtual blood transfusion training scenario models. And virtual and real space mapping parameter set Multimodal operation data at time t is acquired in real time, and a time-aligned multimodal feature fusion method is used to uniformly encode the multimodal operation data into multimodal temporal operation vectors. ; The real-time operation compliance determination module is used for multimodal time-series operation vectors. During blood transfusion training, the system performs real-time assessment of operational compliance and outputs a set of operational compliance assessment flags. ; The operational risk trend prediction module is used to predict operational risks based on a set of operational standardization criteria. An operational risk prediction model constructed using a bidirectional long short-term memory network (BiLSTM) and a time-aware attention network (TGAT) performs the task of predicting operational risk trends and outputs the state trajectory of the controlled operational process. ; The operation quality assessment and scoring module is used to evaluate the status trajectory of controlled operation processes. The operation quality score is processed using a segmented cumulative method, and the results of the blood transfusion-specific operation assessment are output.

9. A multimodal blood transfusion operation training device based on virtual reality, characterized in that, The virtual reality-based multimodal blood transfusion operation training device includes: a memory, a processor, and a virtual reality-based multimodal blood transfusion operation training program stored in the memory and executable on the processor. When the virtual reality-based multimodal blood transfusion operation training program is executed by the processor, it implements a virtual reality-based multimodal blood transfusion operation training method according to any one of claims 1 to 7.

10. A computer program product, characterized in that, The computer program product includes a virtual reality-based multimodal blood transfusion operation training program, which, when executed by a processor, implements a virtual reality-based multimodal blood transfusion operation training method according to any one of claims 1 to 7.

Images