Multi-mode human tracheal intubation virtual simulation training system
By combining virtual simulation systems with VR and artificial intelligence technologies, the high cost and risks of traditional endotracheal intubation training have been solved, providing real-time feedback and improving trainees' operational skills and safety.
Patent Information
- Application Number
- CN202411247482.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-06
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2044-09-06
AI Technical Summary
Traditional training methods for human endotracheal intubation rely on actual medical equipment and real patients, which are costly and risky, and make it difficult to provide immediate and effective feedback, thus affecting the learning outcomes of trainees.
A multi-mode virtual simulation training system for human tracheal intubation was designed. It utilizes VR headsets and artificial intelligence data processing to collect video of trainees' movements, identify and understand key semantics, and provide real-time feedback to correct errors, including voice, text, and animation prompts.
It enables multi-mode endotracheal intubation training in a safe environment, provides real-time feedback, improves trainees' operational skills and safety, and ensures mastery of correct operating procedures.
Smart Images

Figure CN118762579B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of intelligent simulation, and more specifically, to a multi-mode human tracheal intubation virtual simulation training system. Background Art
[0002] Human tracheal intubation is an important medical operation that requires medical staff to have superb skills and rich experience. Traditional human tracheal intubation training methods often rely on actual medical equipment and real patients, which is not only costly but also has certain risks. In addition, traditional training methods are limited in time and space, and it is difficult for trainees to obtain immediate and effective feedback during actual operations, which may reduce the learning feedback effect of trainees in actual training operations, resulting in improper or incorrect operations.
[0003] Therefore, an optimized human endotracheal intubation training system is desired. Summary of the invention
[0004] In order to solve the above technical problems, this application is proposed.
[0005] According to one aspect of the present application, a multi-mode human tracheal intubation virtual simulation training system is provided, which includes:
[0006] Virtual device access module, used to access students' VR headsets;
[0007] A mode selection module, used to select a virtual simulation training mode for human tracheal intubation, wherein the virtual simulation training mode for human tracheal intubation includes a tracheal intubation mode under normal conditions, a rapid intubation mode under emergency conditions, and an intubation mode for pediatric patients;
[0008] A tracheal intubation simulation training module, used to execute the virtual simulation training mode of human tracheal intubation, and provide the trainees with human tracheal intubation simulation training;
[0009] The training feedback module is used to generate a training feedback report based on the trainees’ tracheal intubation simulation training process;
[0010] Wherein, the endotracheal intubation simulation training module includes:
[0011] A trainee action video acquisition unit is used to obtain action videos of the monitored trainees during the human tracheal intubation simulation training;
[0012] An action video key frame sampling unit is used to perform key frame sampling on the action video of the monitored trainee during the human tracheal intubation simulation training to obtain a time sequence of the trainee's action key frames;
[0013] A student action multi-scale semantic information transfer and fusion unit is used to pass each student action key frame in the time series of the student action key frames through a student action multi-scale information transfer and fusion module based on a pyramid structure to obtain a sequence of student action multi-scale semantic fusion feature maps;
[0014] The student action full-time domain semantic enhancement aggregation unit is used to obtain a full-time key semantic representation feature map of the student action by passing the sequence of the multi-scale semantic fusion feature map of the student action through a significant semantic enhancement aggregation module based on the global semantic association between sequences;
[0015] The trainee intubation operation prompt unit is used to generate prompt results based on the full-time key semantic representation feature map of the reinforced trainee action, and the prompt results are used to represent the operation steps of voice prompts, text prompts, icon prompts and animation demonstrations to guide the monitored trainees to perform correct operations during the simulation training process.
[0016] In the above multi-mode human tracheal intubation virtual simulation training system, the trainee action multi-scale semantic information transmission and fusion unit includes:
[0017] A shallow feature extraction subunit for the student's action key frame, used for passing the student's action key frame through a convolution layer with a convolution kernel of 3×3 to obtain a shallow feature map of the student's action key frame;
[0018] The student action key frame shallow residual fusion enhancement sub-unit is used to obtain the student action key frame shallow residual fusion feature map by passing the student action key frame shallow feature map through the student action shallow residual information fusion enhancement module;
[0019] A deep feature extraction subunit for the student's action key frame is used to pass the shallow residual fusion feature map of the student's action key frame through a convolution layer with a convolution kernel of 3×3 to obtain a deep feature map of the student's action key frame;
[0020] The student action key frame deep residual fusion enhancement subunit is used to obtain the student action key frame deep residual fusion feature map by passing the student action key frame deep feature map through the student action deep residual information fusion enhancement module;
[0021] The scale modulation subunit is used to input the deep residual fusion feature map of the student action key frame into the scale modulation module based on the SPPF layer to obtain the deep modulation feature map of the student action key frame;
[0022] An upsampling subunit, used for upsampling the student action key frame deep modulation feature map to obtain an upsampled student action key frame deep modulation feature map;
[0023] A student action key frame multi-scale feature fusion sub-unit, used for fusing the up-sampled student action key frame deep modulation feature map with the student action key frame shallow residual fusion feature map to obtain a student action key frame multi-scale feature map;
[0024] The student action multi-scale residual information fusion enhancement sub-unit is used to pass the student action key frame multi-scale feature map through the student action multi-scale residual information fusion enhancement module to obtain the student action multi-scale semantic fusion feature map.
[0025] In the multi-mode human tracheal intubation virtual simulation training system, the student action key frame shallow residual fusion enhancement subunit is used to:
[0026] The shallow feature map of the student's action key frame is modulated by a point convolution layer to obtain a shallow feature map of the student's action key frame modulated by a channel dimensionality reduction;
[0027] The shallow feature map of the key frame of the student's action modulated by the channel dimension reduction is subjected to feature extraction through a convolution layer with a convolution kernel of 3×3 to obtain a shallow implicit information feature map of the key frame of the student's action;
[0028] The shallow implicit information feature map of the student's action key frame is modulated by a point convolution layer to obtain a shallow feature map of the student's action key frame after channel modulation;
[0029] The shallow feature map of the student's action key frame after channel modulation and the shallow feature map of the student's action key frame after channel dimension reduction modulation are fused to obtain the shallow residual fusion feature map of the student's action key frame.
[0030] In the above multi-mode human tracheal intubation virtual simulation training system, the scale modulation subunit is used to:
[0031] The deep residual fusion feature map of the student action key frame is passed through a scale modulation module based on a point convolution layer to obtain a scale-adjusted deep residual fusion feature map of the student action key frame;
[0032] Performing maximum pooling processing on each feature matrix along the channel dimension in the scale-adjusted student action key frame deep residual fusion feature map to obtain the student action key frame deep residual fusion feature vector;
[0033] Based on the deep residual fusion feature vector of the student action key frame, the scale-adjusted student action key frame deep residual fusion feature map is channel-weighted enhanced to obtain the student action key frame deep modulation feature map.
[0034] In the above multi-mode human tracheal intubation virtual simulation training system, the student action full-time domain semantic reinforcement aggregation unit includes:
[0035] The student action multi-scale fusion semantic representation sub-unit is used to respectively calculate the mean and maximum value of each student action multi-scale semantic fusion feature map in the sequence of the student action multi-scale semantic fusion feature map to obtain a student action multi-scale fusion semantic global representation vector composed of the mean values of multiple feature maps and a student action multi-scale fusion semantic salient representation vector composed of the maximum values of multiple feature maps;
[0036] The activation expression subunit is used to perform one-dimensional convolution encoding on the multi-scale fusion semantic global representation vector of the student's action and then pass it through The activation function is used to process the global representation vector of the multi-scale fusion semantics of the student's action;
[0037] A multi-scale fusion global semantic activation representation subunit is used to encode the activation expression student action multi-scale fusion semantic global representation vector through a point convolution layer and then perform matrix multiplication with the first trainable linear transformation weight matrix to obtain the student action multi-scale fusion global semantic activation representation feature vector;
[0038] The activation expression multi-scale fusion semantic salient subunit is used to perform one-dimensional convolution encoding on the multi-scale fusion semantic salient representation vector of the student's action and then pass it through The activation function is used to process the multi-scale fusion semantic salient representation vector of the activation expression student's action;
[0039] A multi-scale fusion salient semantic activation representation subunit is used to encode the activation expression student action multi-scale fusion semantic salient representation vector through a point convolution layer and then perform matrix multiplication with a second trainable linear transformation weight matrix to obtain a student action multi-scale fusion salient semantic activation representation feature vector;
[0040] The student action global-salient semantic association subunit is used for fusing the student action multi-scale fusion global semantic activation representation feature vector and the student action multi-scale fusion salient semantic activation representation feature vector to obtain the student action global-salient semantic association activation feature vector, and then performing nonlinear activation processing on the student action global-salient semantic association activation feature vector to obtain the student action global-salient semantic association weight feature vector;
[0041] A reinforcement subunit, used to perform positional dot multiplication on the sequence of multi-scale semantic fusion feature maps of the student's movements with the global-prominent semantic association weight feature vector of the student's movements and add the sequence of multi-scale semantic fusion feature maps of the student's movements to obtain a sequence of reinforced multi-scale semantic fusion feature maps of the student's movements;
[0042] The full-time key semantic representation subunit is used to add up the multi-scale semantic fusion feature maps of enhanced student actions in the sequence of the multi-scale semantic fusion feature maps of enhanced student actions according to their positions to obtain the full-time key semantic representation feature map of enhanced student actions.
[0043] In the multi-mode human tracheal intubation virtual simulation training system, the trainee action global-prominent semantic association subunit is used to:
[0044] Fusing the student action multi-scale fusion global semantic activation representation feature vector and the student action multi-scale fusion salient semantic activation representation feature vector to obtain the student action global-salient semantic association activation feature vector;
[0045] Using the hyperbolic tangent function and The function performs activation processing on the student action global-salient semantic association activation feature vector to obtain the student action global-salient feature hyperbolic tangent activation semantic representation vector and the student action global-salient feature S-type activation semantic representation vector;
[0046] The hyperbolic tangent activation semantic representation vector of the global-salient feature of the student's action and the S-type activation semantic representation vector of the global-salient feature of the student's action are multiplied point by point to obtain the global-salient semantic association weight feature vector of the student's action.
[0047] In the above-mentioned multi-mode human tracheal intubation virtual simulation training system, the student intubation operation prompt unit is used to: pass the full-time key semantic representation feature map of the enhanced student action through the decoder-based operation step prompt generator to obtain the prompt result.
[0048] In the above-mentioned multi-mode human tracheal intubation virtual simulation training system, it also includes a training module for training the multi-scale information transmission fusion module of the trainee action based on the pyramid structure, the significant semantic reinforcement aggregation module based on the global semantic association between sequences, and the operation step prompt generator based on the decoder;
[0049] Wherein, the training module includes:
[0050] A training data acquisition unit, used to acquire training data, wherein the training data includes a video of the training movements of the monitored trainee during the human tracheal intubation simulation training;
[0051] A key frame sampling unit is used to perform key frame sampling on the training action video of the monitored trainee during the human tracheal intubation simulation training to obtain a time sequence of key frames of the training trainee's actions;
[0052] A multi-scale semantic information transfer and fusion unit, used for respectively passing each training student action key frame in the time sequence of the training student action key frame through a student action multi-scale information transfer and fusion module based on a pyramid structure to obtain a sequence of training student action multi-scale semantic fusion feature maps;
[0053] A full-time domain semantic reinforcement aggregation unit is used to pass the sequence of the multi-scale semantic fusion feature graphs of the training student's movements through a significant semantic reinforcement aggregation module based on global semantic associations between sequences to obtain a full-time key semantic representation feature graph of the training reinforcement student's movements;
[0054] A decoding loss unit, used for prompting a generator with a decoder-based operation step to obtain a decoding loss function value by using the full-time key semantic representation feature graph of the training reinforcement student's action;
[0055] A difference loss calculation unit, used to calculate the difference loss between the actual prompt result and the predicted prompt result to obtain a loss function value for the full temporal key semantic representation of the student's action;
[0056] A loss calculation unit, used for calculating a weighted sum of the decoding loss function value and the loss function value of the full temporal key semantic representation of the reinforcement student action to obtain a loss function value;
[0057] A training unit is used to train the pyramid-structured student action multi-scale information transfer fusion module, the saliency semantic reinforcement aggregation module based on global semantic association between sequences, and the decoder-based operation step prompt generator through gradient back propagation based on the loss function value.
[0058] Compared with the prior art, the present application provides a multi-mode human tracheal intubation virtual simulation training system, which monitors and collects the trainees' action videos during the human tracheal intubation simulation training in the tracheal intubation simulation training module, and introduces data processing and image analysis algorithms based on artificial intelligence and machine vision at the back end to analyze the action videos, so as to identify and understand the key semantics of the trainees in the human tracheal intubation operation, thereby judging the student's operation progress and whether there are any wrong actions, so as to give corresponding prompts, including voice prompts, text prompts, icon prompts and animation demonstrations to guide the monitored trainees to perform correct operation steps during the simulation training process. In this way, it is possible to help trainees correct errors in time and provide correct operation demonstrations in the form of instant feedback, thereby realizing more intelligent multi-mode human tracheal intubation virtual simulation training and ensuring that trainees master the correct tracheal intubation operation process and actions. BRIEF DESCRIPTION OF THE DRAWINGS
[0059] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.
[0060] Figure 1 A block diagram of a multi-mode human tracheal intubation virtual simulation training system according to an embodiment of the present application;
[0061] Figure 2 A system architecture diagram of a multi-mode human tracheal intubation virtual simulation training system according to an embodiment of the present application;
[0062] Figure 3 A block diagram of a training module of a multi-mode human tracheal intubation virtual simulation training system according to an embodiment of the present application;
[0063] Figure 4 It is a block diagram of a tracheal intubation simulation training module in a multi-mode human tracheal intubation virtual simulation training system according to an embodiment of the present application. DETAILED DESCRIPTION
[0064] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.
[0065] As shown in this application and claims, unless the context clearly indicates an exception, the words "a", "an", "an" and / or "the" do not refer to the singular and may also include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of the steps and elements that have been clearly identified, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements.
[0066] Although the present application makes various references to certain modules in the system according to the embodiments of the present application, any number of different modules can be used and run on the user terminal and / or server. The modules are only illustrative, and different aspects of the system and method can use different modules.
[0067] Flowcharts are used in the present application to illustrate the operations performed by the system according to the embodiments of the present application. It should be understood that the preceding or following operations are not necessarily performed accurately in order. On the contrary, various steps may be processed in reverse order or simultaneously as required. Meanwhile, other operations may also be added to these processes, or a certain step or several steps of operations may be removed from these processes.
[0068] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.
[0069] It should be understood that AR and VR technologies can provide users with an immersive experience by simulating real-world views and superimposing computer-generated images in the user's field of view. In the field of medical training, these technologies can be used to simulate complex surgical procedures and clinical situations, allowing trainees to learn and practice tracheal intubation in a risk-free environment. At the same time, the system can monitor the trainees' movements in real time and provide multiple forms of instant feedback to help them correct mistakes and master the correct operating steps.
[0070] In the technical solution of the present application, a multi-mode human tracheal intubation virtual simulation training system is proposed. Figure 1 It is a block diagram of a multi-mode human tracheal intubation virtual simulation training system according to an embodiment of the present application. Figure 2 : is a system architecture diagram of a multi-mode human tracheal intubation virtual simulation training system according to an embodiment of the present application. Figure 1 and Figure 2 As shown, according to the embodiment of the present application, the multi-mode human tracheal intubation virtual simulation training system 300 includes: a virtual device access module 310, which is used to access the VR head-mounted device of the trainee; a mode selection module 320, which is used to select the virtual simulation training mode of human tracheal intubation, wherein the virtual simulation training mode of human tracheal intubation includes the tracheal intubation mode under normal circumstances, the rapid intubation mode under emergency circumstances, and the intubation mode of pediatric patients; the tracheal intubation simulation training module 330, which is used to execute the virtual simulation training mode of human tracheal intubation and provide the trainee with human tracheal intubation simulation training; the training feedback module 340, which is used to generate a training feedback report according to the tracheal intubation simulation training process of the trainee. The multi-mode human tracheal intubation virtual simulation training system 300 provides a variety of training modes to adapt to different scenarios and the learning and training needs of trainees. These modes include the tracheal intubation mode under normal circumstances, the rapid intubation mode under emergency circumstances, and the intubation mode of pediatric patients, ensuring that trainees can get adequate training in various situations.
[0071] In particular, the virtual device access module 310 and the mode selection module 320 are used to access the trainee's VR head-mounted device; and select the virtual simulation training mode of human tracheal intubation, wherein the virtual simulation training mode of human tracheal intubation includes the tracheal intubation mode under normal circumstances, the rapid intubation mode under emergency circumstances, and the intubation mode for pediatric patients. It should be understood that through these different modes of virtual simulation training, trainees can fully master the skills of tracheal intubation, including operations under normal circumstances, rapid responses in emergency situations, and special technical requirements for pediatric patients. This diversified training mode can improve the trainees' practical ability and adaptability.
[0072] In particular, the tracheal intubation simulation training module 330 is used to execute the virtual simulation training mode of human tracheal intubation to provide the trainee with human tracheal intubation simulation training. Figure 4 As shown, the tracheal intubation simulation training module 330 includes: a student action video acquisition unit 331, which is used to obtain the action video of the monitored student during the human tracheal intubation simulation training process; an action video key frame sampling unit 332, which is used to perform key frame sampling on the action video of the monitored student during the human tracheal intubation simulation training process to obtain a time sequence of the student action key frames; a student action multi-scale semantic information transfer fusion unit 333, which is used to transfer each student action key frame in the time sequence of the student action key frames through the student action multi-scale information transfer fusion module based on the pyramid structure. A sequence of multi-scale semantic fusion feature maps of trainee movements is obtained; a full-time domain semantic reinforcement aggregation unit 334 of trainee movements is used to obtain a reinforced full-time key semantic representation feature map of trainee movements through a saliency semantic reinforcement aggregation module based on global semantic associations between sequences; a trainee intubation operation prompt unit 335 is used to generate a prompt result based on the reinforced full-time key semantic representation feature map of trainee movements, wherein the prompt result is used to represent voice prompts, text prompts, icon prompts and animation demonstrations of operation steps to guide the monitored trainee to perform correct operations during the simulation training process.
[0073] Specifically, the student action video acquisition unit 331 and the action video key frame sampling unit 332 are used to obtain the action video of the monitored student during the human tracheal intubation simulation training process; and the action video of the monitored student during the human tracheal intubation simulation training process is sampled with key frames to obtain the time sequence of the student's action key frames. Considering that the action video contains a large amount of action semantics about the monitored student during the human tracheal intubation simulation training process, some of these semantics are crucial for identifying and understanding the intubation process and correctness tasks, while some are sufficient redundant information. Based on this, in order to quickly identify and record the key actions in the student's operation, so as to reduce the burden of data processing and improve the efficiency of system operation, in the technical solution of the present application, the action video of the monitored student during the human tracheal intubation simulation training process is sampled with key frames to obtain the time sequence of the student's action key frames. By sampling key frames and establishing a time sequence, the system can more accurately analyze the student's operation process, identify the student's key actions in the operation, and provide a basis for subsequent data processing and feedback.
[0074] Specifically, the student action multi-scale semantic information transfer fusion unit 333 is used to pass each student action key frame in the time series of the student action key frame through the student action multi-scale information transfer fusion module based on the pyramid structure to obtain a sequence of student action multi-scale semantic fusion feature maps. Because in the process of identifying and understanding the semantics of student actions, both the shallow features and deep features of the student action key frames are very important. In the pyramid network, the bottom network is mainly responsible for extracting low-level features and spatial detail information of the image, while the high-level network is responsible for extracting more abstract high-level features. However, in the traditional network structure, the position information extracted by the bottom network is difficult to directly affect the high-level network, resulting in the distortion of the position information in the feature extraction of different levels. In addition, in the pyramid network, the transmission of features usually requires a large amount of calculation, especially when processing large-scale video image frame data. With the increase of the number of network layers, the amount of calculation increases exponentially, which may cause the training and reasoning process to become very time-consuming. Based on this, in order to solve the problem that the shallow position information in the pyramid network has limited influence on the student's action features and the amount of calculation is too large when transmitting feature information, in the technical solution of the present application, each student action key frame in the time series of the student action key frame is further passed through the student action multi-scale information transmission and fusion module based on the pyramid structure to obtain a sequence of student action multi-scale semantic fusion feature maps. In particular, in the technical solution of the present application, the student action multi-scale information transmission and fusion module based on the pyramid structure includes a feature extraction network and a feature fusion network. In the feature fusion network, information transmission is achieved by residual fusion and jump connection to effectively fuse the shallow features and deep features in the student action key frames, which can effectively transmit the position information extracted from the bottom layer to the high layer, so that the deep features related to the student's action can better fuse the bottom layer position information, thereby improving the expression ability and accuracy of the action semantic features. In addition, the pyramid structure is used to extract residual features and transfer information for the action features of students at different levels, which reduces the amount of calculation in the feature transfer process. By extracting features and aggregating information at different scales, efficient transmission and utilization of feature information is achieved, while reducing unnecessary computational overhead and improving the computational efficiency and speed of the network.
[0075] In an embodiment of the present application, each student action key frame in the time series of the student action key frames is respectively passed through a student action multi-scale information transfer fusion module based on a pyramid structure to obtain a sequence of student action multi-scale semantic fusion feature maps, including: passing the student action key frames through a convolution layer with a convolution kernel of 3×3 to obtain a shallow feature map of the student action key frames; passing the shallow feature map of the student action key frames through a student action shallow residual information fusion enhancement module to obtain a shallow residual fusion feature map of the student action key frames; passing the shallow residual fusion feature map of the student action key frames through a convolution layer with a convolution kernel of 3×3 to obtain a deep feature map of the student action key frames; passing the deep feature map of the student action key frames through a student action layer. The deep residual information fusion enhancement module of the student's action is used to obtain the deep residual fusion feature map of the student's action key frame; the deep residual fusion feature map of the student's action key frame is input into the scale modulation module based on the SPPF layer to obtain the deep modulation feature map of the student's action key frame; the deep modulation feature map of the student's action key frame is upsampled to obtain the upsampled deep modulation feature map of the student's action key frame; the upsampled deep modulation feature map of the student's action key frame is fused with the shallow residual fusion feature map of the student's action key frame to obtain the multi-scale feature map of the student's action key frame; the multi-scale feature map of the student's action key frame is passed through the student's action multi-scale residual information fusion enhancement module to obtain the multi-scale semantic fusion feature map of the student's action.
[0076] Among them, the process of passing the shallow feature map of the student action key frame through the student action shallow residual information fusion enhancement module to obtain the shallow residual fusion feature map of the student action key frame includes: passing the shallow feature map of the student action key frame through a point convolution layer to perform channel dimensionality reduction modulation to obtain the channel dimensionality reduction modulated shallow feature map of the student action key frame; passing the channel dimensionality reduction modulated shallow feature map of the student action key frame through a convolution layer with a convolution kernel of 3×3 to perform feature extraction to obtain the shallow implicit information feature map of the student action key frame; passing the shallow implicit information feature map of the student action key frame through a point convolution layer to perform channel dimensionality increase modulation to obtain the shallow feature map of the student action key frame after channel modulation; fusing the shallow feature map of the student action key frame after channel modulation and the shallow feature map of the student action key frame modulated by the channel dimensionality reduction to obtain the shallow residual fusion feature map of the student action key frame. More specifically, the process of inputting the deep residual fusion feature map of the student action key frame into the scale modulation module based on the SPPF layer to obtain the deep modulation feature map of the student action key frame includes: passing the deep residual fusion feature map of the student action key frame through the scale modulation module based on the point convolution layer to obtain the scale-adjusted deep residual fusion feature map of the student action key frame; performing maximum pooling processing on each feature matrix along the channel dimension in the scale-adjusted deep residual fusion feature map of the student action key frame to obtain the deep residual fusion feature vector of the student action key frame; performing channel-weighted enhancement on the scale-adjusted deep residual fusion feature map of the student action key frame based on the deep residual fusion feature vector of the student action key frame to obtain the deep modulation feature map of the student action key frame.
[0077] Specifically, the full-time domain semantic reinforcement aggregation unit 334 of the trainee's action is used to obtain a full-time key semantic representation feature map of the reinforced trainee's action through a saliency semantic reinforcement aggregation module based on global semantic associations between sequences through the sequence of the multi-scale semantic fusion feature maps of the trainee's action. It should be understood that since each multi-scale semantic fusion feature map of the trainee's action in the sequence of the multi-scale semantic fusion feature map of the trainee's action respectively contains fusion semantics between different levels of features related to the trainee's action, these semantics have a correlation relationship based on the full time domain and different importance in the time dimension, the full-time domain semantic association features and important semantics of these trainee's actions are crucial for the subsequent identification and understanding of the correctness of the trainee's actions and the operation process during the human tracheal intubation operation. Based on this, in order to identify and strengthen the features that are crucial to the semantic understanding of the trainees' operation actions, so as to help the system more accurately evaluate the performance of the trainees in the tracheal intubation simulation training steps, and identify the process of the trainees' intubation operation, so as to give immediate feedback, in the technical solution of the present application, the sequence of the multi-scale semantic fusion feature map of the trainees' actions is further subjected to the saliency semantic reinforcement aggregation module based on the global semantic association between sequences to obtain the feature map of the key semantic representation of the full temporal sequence of the trainees' actions. It should be understood that the analysis of the global semantic association between sequences can reveal the logical and temporal relationship between the actions, and this analysis helps the system understand the upper and lower processes of the trainees' operations, not just isolated actions. And through the processing of the saliency semantic reinforcement aggregation module based on the global semantic association between sequences, it is possible to learn and capture the correlation relationship and interactive semantics based on the full time domain between the multi-level fusion semantics of the trainees' actions, and identify which action semantic features under the action key frames are more important for the subsequent intubation action correctness judgment and intubation operation process judgment, so as to give a higher weight to the action semantics of this importance for prominent expression, and at the same time give a smaller weight to the image frame semantics with less contribution, so as to improve the feature utilization ability of key features. Then, the information loss is compensated by adding the corresponding positions with the original multi-scale semantic fusion feature map of the student's action. Finally, the multi-scale semantic features of the student's action after the key feature saliency enhancement are aggregated and transferred by adding the feature map according to the position to obtain the key semantic aggregation representation feature information of the student's action in the full time sequence, which helps to further enhance the full-time domain expression and understanding of the student's action semantics.
[0078] In an embodiment of the present application, the sequence of the multi-scale semantic fusion feature maps of the student's actions is passed through a saliency semantic reinforcement aggregation module based on global semantic associations between sequences to obtain a reinforcement of the full-time key semantic representation feature map of the student's actions, including: respectively calculating the mean and maximum values of the multi-scale semantic fusion feature maps of each student's action in the sequence of the multi-scale semantic fusion feature maps of the student's actions to obtain a multi-scale fusion semantic global representation vector of the student's actions composed of a plurality of feature map means and a multi-scale fusion semantic salient representation vector of the student's actions composed of a plurality of feature map maximum values; performing one-dimensional convolution encoding on the multi-scale fusion semantic global representation vector of the student's actions and then performing the aggregation module. The activation function is processed to obtain a global representation vector of the multi-scale fusion semantics of the activation expression of the student's action; the activation expression of the multi-scale fusion semantics of the global representation vector of the student's action is encoded through a point convolution layer and then matrix multiplied with the first trainable linear transformation weight matrix to obtain a multi-scale fusion global semantic activation representation feature vector of the student's action; the multi-scale fusion semantic salient representation vector of the student's action is one-dimensionally convolutionally encoded and then passed The activation function is processed to obtain an activation expression student action multi-scale fusion semantic salient representation vector; the activation expression student action multi-scale fusion semantic salient representation vector is encoded through a point convolution layer and then matrix multiplied with a second trainable linear transformation weight matrix to obtain a student action multi-scale fusion salient semantic activation representation feature vector; after fusing the student action multi-scale fusion global semantic activation representation feature vector and the student action multi-scale fusion salient semantic activation representation feature vector to obtain the student action global-salient semantic association activation feature vector, the student action global-salient semantic association activation feature vector is subjected to nonlinear activation processing to obtain the student action global-salient semantic association weight feature vector; with the student action global-salient semantic association weight feature vector, the sequence of the student action multi-scale semantic fusion feature map is point-multiplied by position and added to the sequence of the student action multi-scale semantic fusion feature map to obtain a sequence of enhanced student action multi-scale semantic fusion feature maps; each enhanced student action multi-scale semantic fusion feature map in the sequence of the enhanced student action multi-scale semantic fusion feature maps is added by position to obtain the enhanced student action full-time key semantic representation feature map.
[0079] Among them, after fusing the multi-scale fusion global semantic activation representation feature vector of the student's action and the multi-scale fusion salient semantic activation representation feature vector of the student's action to obtain the global-salient semantic association activation feature vector of the student's action, a process of performing nonlinear activation processing on the global-salient semantic association activation feature vector of the student's action to obtain the global-salient semantic association weight feature vector of the student's action includes: fusing the multi-scale fusion global semantic activation representation feature vector of the student's action and the multi-scale fusion salient semantic activation representation feature vector of the student's action to obtain the global-salient semantic association activation feature vector of the student's action; using the hyperbolic tangent function and The function activates the global-salient semantic association activation feature vector of the student's action to obtain the global-salient feature hyperbolic tangent activation semantic representation vector of the student's action and the global-salient feature S-type activation semantic representation vector of the student's action; multiplies the global-salient feature hyperbolic tangent activation semantic representation vector of the student's action and the global-salient feature S-type activation semantic representation vector of the student's action by position point to obtain the global-salient semantic association weight feature vector of the student's action.
[0080] In summary, in the above embodiment, the sequence of the multi-scale semantic fusion feature graphs of the student's actions is processed through a significant semantic enhancement aggregation module based on global semantic associations between sequences to obtain a full-time key semantic representation feature graph of the enhanced student's actions, including: the sequence of the multi-scale semantic fusion feature graphs of the student's actions is processed through the significant semantic enhancement aggregation module based on global semantic associations between sequences with the following significant semantic enhancement aggregation formula to obtain the full-time key semantic representation feature graph of the enhanced student's actions; wherein the significant semantic enhancement aggregation formula is:
[0081] ;
[0082] ;
[0083] ;
[0084] ;
[0085] ;
[0086] ;
[0087] in, is a sequence of multi-scale semantic fusion feature maps of the student's actions, represents the global mean operation of all eigenvalues in the feature map. Represents the operation of calculating the maximum value of all eigenvalues in the feature map, It is the multi-scale fusion semantic mean representation vector of the student's action. The maximum value of the multi-scale fusion semantic representation vector for the student's action, represents one-dimensional convolutional coding, express Activation function, and They represent the trainable weight matrices for linear transformation, and They are the global representation feature vector of the multi-scale fusion semantics of the student's action and the prominent representation feature vector of the multi-scale fusion semantics of the student's action, for function, for function, It means point multiplication by position. In order to strengthen the sequence of multi-scale semantic fusion feature maps of students' actions, is the positional addition operation of the feature map, Respectively represent each enhanced student action multi-scale semantic fusion feature map in the sequence of enhanced student action multi-scale semantic fusion feature maps, A feature graph representing the full temporal key semantics of the reinforcement student's actions.
[0088] Specifically, the trainee intubation operation prompt unit 335 is used to generate a prompt result based on the full-time key semantic representation feature map of the enhanced trainee action, and the prompt result is used to represent the voice prompt, text prompt, icon prompt and animation demonstration of the operation steps to guide the monitored trainee to perform the correct operation during the simulation training process. In particular, in a specific example of the present application, the full-time key semantic representation feature map of the enhanced trainee action is passed through the decoder-based operation step prompt generator to obtain the prompt result. That is, the full-time domain key semantic aggregation representation feature of the trainee action after enhancement is used to perform decoding regression, so as to identify and understand the full-time domain key semantics of the trainee in the human tracheal intubation operation, so as to judge the operation process of the trainee and whether there is an erroneous action, so as to give corresponding prompts and generate prompt results, including voice prompts, text prompts, icon prompts and animation demonstrations of the operation steps to guide the monitored trainee to perform the correct operation during the simulation training process. In this way, it is possible to help trainees correct errors in time and provide correct operation demonstrations in the form of instant feedback, thereby realizing a more intelligent multi-mode human tracheal intubation virtual simulation training, ensuring that trainees master the correct tracheal intubation operation process and actions.
[0089] In a preferred example, the sequence of the multi-scale semantic fusion feature maps of the student's movements expresses the cross-scale and cross-depth fusion image semantic features of the key frames of the student's movements. Therefore, after the sequence of the multi-scale semantic fusion feature maps of the student's movements passes through a significant semantic enhancement aggregation module based on global semantic associations between sequences, the enhanced student's movement full-time key semantic representation feature map obtained will also have a lack of interpretability of decoding regression due to the multi-scale and multi-depth cross-time domain feature aggregation of complex image semantics based on the significant differences in image semantic associations between time domain sequences in the time direction, thereby affecting the accuracy of the decoding results.
[0090] Based on this, the present application optimizes the key semantic representation feature graph of the full temporal sequence of the strengthening student's action, including the steps of:
[0091] Expanding the full-time key semantic representation feature graph of the reinforcement student's action into a full-time key semantic representation feature vector of the reinforcement student's action;
[0092] Calculate the self-association matrix of the key semantic representation feature vector of the full temporal sequence of the reinforcement student's action, that is:
[0093] ;
[0094] in, Represents the full-time key semantic feature vector of the reinforcement student action, represents the transpose of a vector, represents matrix multiplication, represents the autocorrelation matrix;
[0095] The feature vector representing the key semantics of the full temporal sequence of the reinforcement student's action is multiplied by the self-association matrix in a query manner to obtain an intermediate feature vector, that is:
[0096] ;
[0097] in, Represents the full-time key semantic feature vector of the reinforcement student action, represents the autocorrelation matrix, represents matrix multiplication, represents the intermediate eigenvector;
[0098] Calculate the first Row vector and Inner product sum of row vectors distance, to obtain the first weight matrix and the second weight matrix , expressed as:
[0099] ;
[0100] ;
[0101] in, and The autocorrelation matrix is represented by Row vector and row vector, represents the transpose of a vector, represents matrix multiplication, represents the autocorrelation matrix, represents the first weight matrix, represents the first weight matrix The eigenvalues of the positions, represents the second weight matrix, represents the second weight matrix The eigenvalues of the positions, Represents the calculation vector distance;
[0102] The first weight matrix and the second weight matrix After matrix multiplication, further matrix multiplication is performed with the intermediate feature vector to obtain the optimized feature vector representing the key semantics of the full temporal sequence of the reinforcement student's action, namely:
[0103] ;
[0104] in, represents the first weight matrix, represents the second weight matrix, represents the intermediate eigenvector, represents matrix multiplication, A feature vector representing the optimized key semantic representation of the full temporal sequence of the reinforcement student's action;
[0105] The optimized reinforcement student action full-time key semantic representation feature vector is restored to the optimized reinforcement student action full-time key semantic representation feature graph. Finally, the optimized reinforcement student action full-time key semantic representation feature graph is passed through the decoder-based operation step prompt generator to obtain the prompt result.
[0106] That is, for the bit-by-bit scattering association response space of the feature value of the reinforcement student action key semantic representation feature graph of the full temporal sequence, the deep embedding structure in the high-order association space is obtained through the joint embedding space of the vector inner product embedding space and the vector independent embedding space, and the decomposable association response sub-dimension of the bit-by-bit scattering association response space of the feature value of the reinforcement student action key semantic representation feature graph of the full temporal sequence is represented as a set of spatial sub-dimensions to further perform the dependency space mapping of the decomposable association response sub-dimension of the deep embedding structure of the reinforcement student action key semantic representation feature graph of the full temporal sequence, so as to improve the interpretability of the solution space of the decoding regression solution of the reinforcement student action key semantic representation feature graph based on the association distribution through the complex dimensional representation of the association response, thereby improving the accuracy of the decoding result obtained from the reinforcement student action key semantic representation feature graph of the full temporal sequence.
[0107] In particular, the training feedback module 340 is used to generate a training feedback report based on the trainee's tracheal intubation simulation training process. It should be understood that in a virtual simulation system, real-time interaction is crucial. By gradually guiding the trainee's intubation simulation training process and operation steps, the trainee can be effectively given correct operation demonstrations and key points to ensure that the trainee can get correct guidance in each step. In this process, the key is how to perform semantic recognition on the trainee's operating actions to judge the progress of the trainee's human tracheal intubation and whether there are any action errors, thereby generating correct operation demonstration prompts and error warning prompts. This form of instant feedback helps trainees learn and adapt quickly during the training process. In particular, through feedback in various forms such as voice, text, and icons, the system can ensure that trainees can get correct guidance in each step.
[0108] It should be understood that before using the above-mentioned neural network model for inference, it is necessary to train the multi-scale information transmission fusion module of the student action based on the pyramid structure, the significant semantic reinforcement aggregation module based on the global semantic association between sequences, and the decoder-based operation step prompt generator. That is to say, according to the multi-mode human tracheal intubation virtual simulation training system 300 of the present application, it also includes a training phase for training the multi-scale information transmission fusion module of the student action based on the pyramid structure, the significant semantic reinforcement aggregation module based on the global semantic association between sequences, and the decoder-based operation step prompt generator.
[0109] Figure 3 FIG. 4 is a block diagram of a training module of a multi-mode human tracheal intubation virtual simulation training system according to an embodiment of the present application. Figure 3As shown, according to the multi-mode human tracheal intubation virtual simulation training system 300 of the embodiment of the present application, it includes: a training module 400, including: a training data acquisition unit 410, used to acquire training data, wherein the training data includes a training action video of a monitored trainee during a human tracheal intubation simulation training process; a key frame sampling unit 420, used to perform key frame sampling on the training action video of the monitored trainee during the human tracheal intubation simulation training process to obtain a time series of training trainee action key frames; a multi-scale semantic information transfer fusion unit 430, used to pass each training trainee action key frame in the time series of the training trainee action key frames through a trainee action multi-scale information transfer fusion module based on a pyramid structure to obtain a sequence of training trainee action multi-scale semantic fusion feature maps; a full-time domain semantic reinforcement aggregation unit 440, used to pass the sequence of the training trainee action multi-scale semantic fusion feature maps through a global semantic relationship between sequences A saliency semantic reinforcement aggregation module is connected to obtain a training reinforcement student action full-time key semantic representation feature map; a decoding loss unit 450 is used to pass the training reinforcement student action full-time key semantic representation feature map through a decoder-based operation step prompt generator to obtain a decoding loss function value; a difference loss calculation unit 460 is used to calculate the difference loss between the real prompt result and the predicted prompt result to obtain the reinforcement student action full-time key semantic representation loss function value; a loss calculation unit 470 is used to calculate the weighted sum of the decoding loss function value and the reinforcement student action full-time key semantic representation loss function value to obtain a loss function value; a training unit 480 is used to train the pyramid-structured student action multi-scale information transfer fusion module, the saliency semantic reinforcement aggregation module based on global semantic association between sequences, and the decoder-based operation step prompt generator based on the loss function value through gradient back propagation.
[0110] In a preferred example, the sequence of multi-scale semantic fusion feature maps of the training student's movements expresses the cross-scale and cross-depth fusion image semantic features of the key frames of the student's movements. Therefore, after the sequence of multi-scale semantic fusion feature maps of the training student's movements passes through a significant semantic enhancement aggregation module based on global semantic associations between sequences, the obtained training enhanced student's movement full-time key semantic representation feature map will also have decoding regression recognition difficulties due to the complex image semantic multi-scale and multi-depth cross-time domain feature aggregation based on the significant differences in image semantic associations between time domain sequences in the time direction, thereby affecting the decoding training efficiency.
[0111] Therefore, the applicant of the present application further introduces a decoding loss function in the model training process, such as a predetermined loss function other than the difference loss function between the real prompt result and the training predicted prompt result, that is, training the model through gradient back propagation based on the loss function includes the following steps:
[0112] Expanding the training reinforcement student action full-time key semantic representation feature graph into the training reinforcement student action full-time key semantic representation feature vector;
[0113] Calculate the first training reinforcement student action full time sequence key semantic representation weight matrix and the second training reinforcement student action full time sequence key semantic representation weight matrix based on the training reinforcement student action full time sequence key semantic representation feature vector, wherein the first and second training reinforcement student action full time sequence key semantic representation weight matrices The feature values of the positions are the first and second feature vectors of the key semantic representation of the full temporal sequence of the training reinforcement student's action. Eigenvalues and The mean and half of the absolute value of the difference of the eigenvalues;
[0114] Performing a query-based matrix multiplication of the training reinforcement student action full-time key semantic representation feature vector with the first training reinforcement student action full-time key semantic representation weight matrix and the second training reinforcement student action full-time key semantic representation weight matrix to obtain the first training reinforcement student action full-time key semantic representation intermediate vector and the second training reinforcement student action full-time key semantic representation intermediate vector;
[0115] Calculate the vector inner product of the intermediate vector of the full temporal key semantic representation of the first training reinforcement student action and the intermediate vector of the full temporal key semantic representation of the second training reinforcement student action to obtain the loss term of the full temporal key semantic representation of the first reinforcement student action;
[0116] The first training reinforcement student action full time sequence key semantic representation weight matrix and the second training reinforcement student action full time sequence key semantic representation weight matrix are matrix multiplied, and the result matrix is calculated. norm to obtain the second reinforcement student action full temporal key semantic representation loss term;
[0117] The first reinforcement student action full temporal key semantic representation loss term is subtracted from the product of a predetermined weight hyperparameter and the second reinforcement student action full temporal key semantic representation loss term to obtain a reinforcement student action full temporal key semantic representation loss function value;
[0118] Based on the weighted sum of the loss function value of the full temporal key semantic representation of the student's actions and the decoding loss function value, the model parameters are optimized through gradient back propagation.
[0119] Among them, the loss function value of the key semantic representation of the full temporal sequence of the enhanced student action is specifically expressed as:
[0120] ;
[0121] in, , .
[0122] in, The training strengthens the trainee's full-time key semantic representation feature vector of the action, and They are respectively the weight matrix of the key semantic representation of the full temporal sequence of the first training reinforcement student action and the weight matrix of the key semantic representation of the full temporal sequence of the second training reinforcement student action. and The first training strengthens the student's action full temporal key semantic representation weight matrix and the second training strengthens the student's action full temporal key semantic representation weight matrix. The eigenvalues of the positions, and are the first and second key semantic representation feature vectors of the full temporal sequence of the training reinforcement student's action. Eigenvalues and Eigenvalues, is matrix multiplication, To calculate the matrix norm, is the predetermined weight hyperparameter, To strengthen the key semantic representation of the loss function value of the entire temporal sequence of students' actions.
[0123] That is, in the above preferred example, the loss function value of the full temporal key semantic representation of the enhanced student action is used to perform the query composition of the detail inner product space within the training enhanced student action full temporal key semantic representation feature graph through the structured feature representation of the short-range and long-range cross-scale detail links of the training enhanced student action full temporal key semantic representation feature graph, so as to approximate the low-rank independent observable composition of the link detail composition provided by the structured detail interaction of the training enhanced student action full temporal key semantic representation feature graph. In this way, by training with the loss function value of the full temporal key semantic representation of the enhanced student action, the distributed detail group of the training enhanced student action full temporal key semantic representation feature graph can be used to decompose the detail group on the basis of the detail complexity, so as to promote the decoding regression decomposition and recognition of the complex feature structure of the training enhanced student action full temporal key semantic representation feature graph, and improve the decoding training efficiency. This helps to better identify and understand the aggregated semantics of the trainees' key actions in the entire time domain during human tracheal intubation operations, so as to determine the trainees' operation progress and whether there are any wrong actions and give corresponding prompts. In this way, more intelligent multi-mode human tracheal intubation virtual simulation training can be achieved to ensure that trainees master the correct tracheal intubation operation procedures and actions.
[0124] As described above, the multi-mode human tracheal intubation virtual simulation training system 300 according to the embodiment of the present application can be implemented in various wireless terminals, such as a server with a multi-mode human tracheal intubation virtual simulation training algorithm. In one possible implementation, the multi-mode human tracheal intubation virtual simulation training system 300 according to the embodiment of the present application can be integrated into the wireless terminal as a software module and / or a hardware module. For example, the multi-mode human tracheal intubation virtual simulation training system 300 can be a software module in the operating system of the wireless terminal, or it can be an application developed for the wireless terminal; of course, the multi-mode human tracheal intubation virtual simulation training system 300 can also be one of the many hardware modules of the wireless terminal.
[0125] Alternatively, in another example, the multi-mode human tracheal intubation virtual simulation training system 300 and the wireless terminal may also be separate devices, and the multi-mode human tracheal intubation virtual simulation training system 300 may be connected to the wireless terminal via a wired and / or wireless network and transmit interactive information in accordance with an agreed data format.
[0126] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.
Claims
1. A multi-mode human tracheal intubation virtual simulation training system, characterized in that: include: Virtual device access module, used to access students' VR headsets; A mode selection module, used to select a virtual simulation training mode for human tracheal intubation, wherein the virtual simulation training mode for human tracheal intubation includes a tracheal intubation mode under normal conditions, a rapid intubation mode under emergency conditions, and an intubation mode for pediatric patients; A tracheal intubation simulation training module, used to execute the virtual simulation training mode of human tracheal intubation, and provide the trainees with human tracheal intubation simulation training; The training feedback module is used to generate a training feedback report based on the trainees’ tracheal intubation simulation training process; Wherein, the endotracheal intubation simulation training module includes: A trainee action video acquisition unit is used to obtain action videos of the monitored trainees during the human tracheal intubation simulation training; An action video key frame sampling unit is used to perform key frame sampling on the action video of the monitored trainee during the human tracheal intubation simulation training to obtain a time sequence of the trainee's action key frames; A student action multi-scale semantic information transfer and fusion unit is used to pass each student action key frame in the time series of the student action key frames through a student action multi-scale information transfer and fusion module based on a pyramid structure to obtain a sequence of student action multi-scale semantic fusion feature maps; The student action full-time domain semantic enhancement aggregation unit is used to obtain a full-time key semantic representation feature map of the student action by passing the sequence of the multi-scale semantic fusion feature map of the student action through a significant semantic enhancement aggregation module based on the global semantic association between sequences; A trainee intubation operation prompt unit, used to generate a prompt result based on the full-time key semantic representation feature graph of the reinforced trainee action, wherein the prompt result is used to indicate a voice prompt, a text prompt, an icon prompt and an animation demonstration of the operation steps to guide the monitored trainee to perform the correct operation during the simulation training process; The student action full-time domain semantic reinforcement aggregation unit includes: The student action multi-scale fusion semantic representation sub-unit is used to respectively calculate the mean and maximum value of each student action multi-scale semantic fusion feature map in the sequence of the student action multi-scale semantic fusion feature map to obtain a student action multi-scale fusion semantic global representation vector composed of the mean values of multiple feature maps and a student action multi-scale fusion semantic salient representation vector composed of the maximum values of multiple feature maps; The activation expression subunit is used to perform one-dimensional convolution encoding on the multi-scale fusion semantic global representation vector of the student's action and then pass it through The activation function is used to process the global representation vector of the multi-scale fusion semantics of the student's action; A multi-scale fusion global semantic activation representation subunit is used to encode the activation expression student action multi-scale fusion semantic global representation vector through a point convolution layer and then perform matrix multiplication with the first trainable linear transformation weight matrix to obtain the student action multi-scale fusion global semantic activation representation feature vector; The activation expression multi-scale fusion semantic salient subunit is used to perform one-dimensional convolution encoding on the multi-scale fusion semantic salient representation vector of the student's action and then pass it through The activation function is used to process the multi-scale fusion semantic salient representation vector of the activation expression student's action; A multi-scale fusion salient semantic activation representation subunit is used to encode the activation expression student action multi-scale fusion semantic salient representation vector through a point convolution layer and then perform matrix multiplication with a second trainable linear transformation weight matrix to obtain a student action multi-scale fusion salient semantic activation representation feature vector; The student action global-salient semantic association subunit is used for fusing the student action multi-scale fusion global semantic activation representation feature vector and the student action multi-scale fusion salient semantic activation representation feature vector to obtain the student action global-salient semantic association activation feature vector, and then performing nonlinear activation processing on the student action global-salient semantic association activation feature vector to obtain the student action global-salient semantic association weight feature vector; A reinforcement subunit, used to perform positional dot multiplication on the sequence of multi-scale semantic fusion feature maps of the student's movements with the global-prominent semantic association weight feature vector of the student's movements and add the sequence of multi-scale semantic fusion feature maps of the student's movements to obtain a sequence of reinforced multi-scale semantic fusion feature maps of the student's movements; The full-time key semantic representation subunit is used to add the multi-scale semantic fusion feature maps of the enhanced student actions in the sequence of the multi-scale semantic fusion feature maps of the enhanced student actions according to the position to obtain the full-time key semantic representation feature map of the enhanced student actions; The student action multi-scale semantic information transmission and fusion unit comprises: A shallow feature extraction subunit for the student's action key frame, used for passing the student's action key frame through a convolution layer with a convolution kernel of 3×3 to obtain a shallow feature map of the student's action key frame; The student action key frame shallow residual fusion enhancement sub-unit is used to obtain the student action key frame shallow residual fusion feature map by passing the student action key frame shallow feature map through the student action shallow residual information fusion enhancement module; A deep feature extraction subunit for the student's action key frame is used to pass the shallow residual fusion feature map of the student's action key frame through a convolution layer with a convolution kernel of 3×3 to obtain a deep feature map of the student's action key frame; The student action key frame deep residual fusion enhancement subunit is used to obtain the student action key frame deep residual fusion feature map by passing the student action key frame deep feature map through the student action deep residual information fusion enhancement module; The scale modulation subunit is used to input the deep residual fusion feature map of the student action key frame into the scale modulation module based on the SPPF layer to obtain the deep modulation feature map of the student action key frame; An upsampling subunit, used for upsampling the student action key frame deep modulation feature map to obtain an upsampled student action key frame deep modulation feature map; A student action key frame multi-scale feature fusion sub-unit, used for fusing the up-sampled student action key frame deep modulation feature map with the student action key frame shallow residual fusion feature map to obtain a student action key frame multi-scale feature map; The student action multi-scale residual information fusion enhancement sub-unit is used to pass the student action key frame multi-scale feature map through the student action multi-scale residual information fusion enhancement module to obtain the student action multi-scale semantic fusion feature map.
2. The multi-mode human tracheal intubation virtual simulation training system according to claim 1 is characterized in that: The student action key frame shallow residual fusion enhancement sub-unit is used to: The shallow feature map of the student's action key frame is modulated by a point convolution layer to obtain a shallow feature map of the student's action key frame modulated by a channel dimensionality reduction; The shallow feature map of the key frame of the student's action modulated by the channel dimension reduction is subjected to feature extraction through a convolution layer with a convolution kernel of 3×3 to obtain a shallow implicit information feature map of the key frame of the student's action; The shallow implicit information feature map of the student's action key frame is modulated by a point convolution layer to obtain a shallow feature map of the student's action key frame after channel modulation; The shallow feature map of the student's action key frame after channel modulation and the shallow feature map of the student's action key frame after channel dimension reduction modulation are fused to obtain the shallow residual fusion feature map of the student's action key frame.
3. The multi-mode human tracheal intubation virtual simulation training system according to claim 2 is characterized in that: The scale modulation subunit is used for: The deep residual fusion feature map of the student action key frame is passed through a scale modulation module based on a point convolution layer to obtain a scale-adjusted deep residual fusion feature map of the student action key frame; Performing maximum pooling processing on each feature matrix along the channel dimension in the scale-adjusted student action key frame deep residual fusion feature map to obtain the student action key frame deep residual fusion feature vector; Based on the deep residual fusion feature vector of the student action key frame, the scale-adjusted student action key frame deep residual fusion feature map is channel-weighted enhanced to obtain the student action key frame deep modulation feature map.
4. The multi-mode human tracheal intubation virtual simulation training system according to claim 3 is characterized in that: The student action global-prominent semantic association subunit is used to: Fusing the student action multi-scale fusion global semantic activation representation feature vector and the student action multi-scale fusion salient semantic activation representation feature vector to obtain the student action global-salient semantic association activation feature vector; Using the hyperbolic tangent function and The function performs activation processing on the student action global-salient semantic association activation feature vector to obtain the student action global-salient feature hyperbolic tangent activation semantic representation vector and the student action global-salient feature S-type activation semantic representation vector; The hyperbolic tangent activation semantic representation vector of the global-salient feature of the student's action and the S-type activation semantic representation vector of the global-salient feature of the student's action are multiplied point by point to obtain the global-salient semantic association weight feature vector of the student's action.
5. The multi-mode human tracheal intubation virtual simulation training system according to claim 4 is characterized in that: The trainee intubation operation prompt unit is used to: pass the full-time key semantic representation feature map of the enhanced trainee action through a decoder-based operation step prompt generator to obtain the prompt result.
6. The multi-mode human tracheal intubation virtual simulation training system according to claim 5 is characterized in that: It also includes a training module for training the pyramid-structure-based student action multi-scale information transfer fusion module, the saliency semantic reinforcement aggregation module based on global semantic association between sequences, and the decoder-based operation step prompt generator; Wherein, the training module includes: A training data acquisition unit, used to acquire training data, wherein the training data includes a video of the training movements of the monitored trainee during the human tracheal intubation simulation training; A key frame sampling unit is used to perform key frame sampling on the training action video of the monitored trainee during the human tracheal intubation simulation training to obtain a time sequence of key frames of the training trainee's actions; A multi-scale semantic information transfer and fusion unit, used for respectively passing each training student action key frame in the time sequence of the training student action key frame through a student action multi-scale information transfer and fusion module based on a pyramid structure to obtain a sequence of training student action multi-scale semantic fusion feature maps; A full-time domain semantic reinforcement aggregation unit is used to pass the sequence of the multi-scale semantic fusion feature graphs of the training student's movements through a significant semantic reinforcement aggregation module based on global semantic associations between sequences to obtain a full-time key semantic representation feature graph of the training reinforcement student's movements; A decoding loss unit, used for prompting a generator with a decoder-based operation step to obtain a decoding loss function value by using the full-time key semantic representation feature graph of the training reinforcement student's action; A difference loss calculation unit, used to calculate the difference loss between the actual prompt result and the predicted prompt result to obtain a loss function value for the key semantic representation of the full temporal sequence of the reinforcement student's action; A loss calculation unit, used for calculating a weighted sum of the decoding loss function value and the loss function value of the full temporal key semantic representation of the reinforcement student action to obtain a loss function value; A training unit is used to train the pyramid-structured student action multi-scale information transfer fusion module, the saliency semantic reinforcement aggregation module based on global semantic association between sequences, and the decoder-based operation step prompt generator through gradient back propagation based on the loss function value.
Citation Information
Patent Citations
Infectious disease nursing guidance system and method based on virtual reality technology
CN111627266A
Fishing boat operation behavior identification method based on deep learning
CN117975341A
