Method and system for evaluating infusion operation of intern medical staff

Abstract

The invention discloses a method and system for evaluating infusion operation of intern medical staff, and belongs to the field of infusion, and the method comprises the steps that the operation of the intern medical staff is collected in multiple modes, and the multiple modes comprise a visual mode, a touch mode and a physiological signal mode; inputting the collected operation of the intern medical staff into the multi-modal infusion evaluation model, performing feature extraction by the multi-modal infusion evaluation model in a modal manner, and performing cross-modal attention fusion on feature vectors of different modals; and finally, comparing the calculated total feature tensor Hfusion with a standard total feature tensor Hfusion ', and outputting a final score of the practice medical personnel according to a preset scoring standard. According to the scheme, multi-dimensional data such as visual, tactile and physiological signals are synchronously collected through a multi-modal fusion method, operation details are comprehensively covered, comprehensive evaluation is carried out, and evaluation objectivity is improved.

Description

Technical Field

[0001] The present invention belongs to the field of infusion, and relates to a method and system for evaluating the infusion operation of medical trainees. Background Art

[0002] Intravenous infusion is a fundamental and critical procedure in clinical nursing, and its accuracy is directly related to patient safety and treatment outcomes. However, current assessments of infusion procedures for medical trainees still rely primarily on manual observation or simplified simulation equipment, which has the following significant shortcomings:

[0003] Highly subjective: Traditional assessment relies on the instructor's experience, is easily influenced by individual subjective judgment, and lacks a unified quantitative standard;

[0004] Single data dimension: Existing methods focus on visual operation records, ignoring multimodal data such as tactile pressure and physiological signals, making it difficult to fully reflect the operation quality;

[0005] Feedback lag: Manual evaluation is inefficient and cannot provide real-time feedback on errors, making it difficult for interns to make timely improvements;

[0006] Insufficient standardization: Clinical operation standards are not systematically integrated into the evaluation model, which may lead to a disconnect between the scores and the actual clinical scenarios.

[0007] Although recent studies have attempted to incorporate sensors or machine learning technologies, most are limited to single-modality analysis and lack the ability to adapt to dynamic scenarios. Therefore, there is an urgent need for an intelligent assessment method that integrates multimodal data, incorporates clinical knowledge graphs, and can dynamically adapt to different scenarios to improve training efficiency and operational standardization. Summary of the Invention

[0008] The purpose of the present invention is to provide a method and system for evaluating the infusion operation of trainee medical staff. Through a multimodal fusion method, multi-dimensional data such as vision, touch, and physiological signals are synchronously collected to comprehensively cover the operation details and improve the objectivity of the evaluation.

[0009] The technical solution adopted in the present invention is as follows:

[0010] A method for evaluating infusion operation of trainee medical personnel, the method comprising:

[0011] Multimodal acquisition of the intern medical staff's operations, wherein the multimodal acquisition includes a visual modality, a tactile modality, and a physiological signal modality;

[0012] The collected operations of medical interns are input into a multimodal infusion evaluation model, which extracts features based on the modalities and performs cross-modal attention fusion on the feature vectors of different modalities:

[0013] Hfusion =Φ CA (H vis ,H tac ,H bio ,H kg )

[0014] Among them, H fusion is the total feature tensor after fusion, Φ cA is the cross-modal attention fusion function, H represents the feature tensor, H vis is the visual feature tensor, H tac is the tactile feature tensor, H bio is the physiological signal feature tensor, H kg is the knowledge graph feature tensor;

[0015] The calculated total feature tensor H fusion Compared with the standard total feature tensor H fusion ', output the final score of the intern medical staff according to the preset scoring criteria.

[0016] in

[0017] H vis =Ψ vis (V) = ViT(3DCNN(V))

[0018] H tac =Ψ tac (T) = BiLSTM(T)

[0019] H bio =Ψ bio (B)=DilatedCNN(B)

[0020] H kg =Ψ kg (K) = GAT(k)

[0021] In the formula, Ψ represents the feature extraction function, Ψ vis is the visual feature extraction function, Ψ tac is the tactile feature extraction function, Ψ bio is the physiological signal processing function, Ψ kg Embedding function for knowledge graph.

[0022] In order to better implement this solution, further, in the cross-modal attention fusion function, the weights of vision, touch, physiological signals and knowledge graph are dynamically allocated: the multimodal infusion evaluation model is based on the visual feature tensor H of the infusion operation of the intern medical staff in the simulated scene. vis , tactile feature tensor H tac , simulate the patient's physiological signal feature tensor H bioAccording to the different infusion scenarios set and the different operations captured by the interns, the feature tensors H of the knowledge graph modules are corresponding to different kg Dynamically allocate visual feature tensor H vis , tactile feature tensor H tac , physiological signal feature tensor H bio The weight of .

[0023] In order to better implement this solution, further, in the cross-modal attention fusion, the visual feature tensor H is first established through cross-modal attention vis and the tactile feature tensor H vis The physiological signal feature tensor H bio And the fused visual-tactile features are gated and fused, where

[0024] Cross-modal attention builds visual feature tensor H vis and the tactile feature tensor H tac The association process is:

[0025] Query-key-value mapping:

[0026] Q=H vis W Q

[0027] K=H tac W K

[0028] V=H tac W V

[0029] Calculate the attention weight A:

[0030]

[0031] Among them, Q, K, and V are query, key, and value matrices respectively, and d k is the scaling factor, T is the length of the time dimension;

[0032] Initial feature fusion obtains the visual-tactile fusion vector H vis-tac :

[0033] H vis-tac =A·V

[0034] The physiological signal feature tensor H bio , knowledge graph feature tensor H kg and the visual-tactile fusion vector H vis-tac Fusion is performed to obtain the total feature tensor H fusion :

[0035] LSTM models the delayed characteristics of physiological responses to obtain the intermediate tensor H bio-timed:

[0036] H bio-timed =LSTM(H bio ,H vis-tac )

[0037] Finally, gated multimodal fusion is performed:

[0038] H fusion =g⊙H vis-tac +(1-g)⊙(αH bio-timed +βH kg )

[0039] g=σ(W g [H vis-tac ||H bio-timed ||H kg ]+b g )

[0040] Among them, g is the dynamic gating value generated based on the visual-tactile confidence, with a range of (0-1); α and β are modal balance coefficients, generally set to 0.5, σ is the activation function, which compresses the output to the (0, 1) interval, indicating the modal weight (such as 0.7 indicates visual-tactile dominance), and W g is the weight matrix, which assigns weights to each tensor; ‖ is the concatenation operation, which connects different modal features in the feature dimension; b g is a bias scalar used to adjust the activation threshold.

[0041] In order to better implement this solution, further, the visual feature tensor H vis The method of obtaining is to use a 3D convolution model to process the collected operation video, process local spatiotemporal details, and detect local anomalies. In order to better detect certain parameters, such as the angle between the infusion needle and the patient's skin, it is also possible to add recognition functions to the visual feature processing function, or add an infrared recognition device to identify the needle tube angle. However, these methods are relatively conventional and are not the core focus of this solution. They are not described in detail in this article and can be expanded according to needs when used. The convolution kernel parameters are (t×h×w) and the calculation formula is:

[0042]

[0043] Where C represents the number of input channels, T represents the length of the time dimension, i.e., the number of consecutive video frames, H represents the input height, i.e., the video resolution height, such as 256 pixels, corresponding to the vertical range of the operation area, W represents the input width, i.e., the video resolution width, such as 256 pixels, corresponding to the horizontal range of the operation area, and K represents the input width, i.e., the video resolution width, such as 256 pixels, corresponding to the horizontal range of the operation area. c',c,t,h,wrepresents the convolution kernel weight, where c' is the output channel, t is the time dimension kernel size, that is, the number of frames of continuous action capture, h is the spatial kernel height, w is the spatial kernel width, I is the input feature map, i is the time axis coordinate, j is the height axis position, k is the width axis position, b c' Represents the bias term, that is, the preset feature activation threshold;

[0044] Residual connections are then used to alleviate the vanishing gradient. The formula is:

[0045]

[0046] In addition, a collaborative cascade architecture is formed using ViT and 3D convolutional layers to capture global semantic associations. The 3D feature map needs to be expanded into a spatiotemporal tag sequence, including:

[0047] Chunked embedding:

[0048]

[0049] Among them, z p is the embedded representation of the pth block of the input video, carrying local semantic and position information, E is the embedding matrix, learning to map the original pixels to the feature space, the vec function flattens the block into a vector, V p represents the p-th spatiotemporal block of the input video, Representation position encoding, marking the spatial-temporal position of each block;

[0050] Multi-head self-attention:

[0051] MSA(Z)=Concat(head1,…,head h )W O

[0052]

[0053] Among them, MSA(Z) is the final output of multi-head self-attention, Z is the input feature matrix, that is, the output z of the previous block embedding p The Concat function represents the concatenation of the outputs of h heads in the feature dimension. i is the output of the i-th attention head, such as the output feature of the “device motion tracking head”, W O is the output projection matrix, Q i , K i 、V i are query, key, and value matrices respectively, d k is a scaling factor to prevent the dot product result from being too large and causing gradient instability;

[0054] Here Q i , Ki 、V i The three matrices are:

[0055]

[0056] Q i , K i 、V i The input feature matrix Z is used to project the i-th head onto d k dimensional space to obtain the query, key, and value matrix,

[0057] Perform residual connection and layer normalization:

[0058] Z'=LayerNorm(Z+MSA(Z))

[0059] The final output visual feature tensor H vis :

[0060] H vis =LayerNorm(Z'+FFN(Z'))

[0061] Among them, FFN is a feed-forward network.

[0062] In order to better implement this solution, further, the tactile feature tensor H tac The method of obtaining the pressure is to use the long short-term memory network LSTM to analyze the simulated patient pressure change curve and identify the three-stage characteristics of "touch-puncture-stabilization". The collected pressure time series data T = {t1,…,t n The gating mechanism formula of} is:

[0063] i t =σ(W ii x t +W hi h t-1 +b i )

[0064] f t =σ(W if x t +W hf h t-1 +b f )

[0065] g t =tanh(W ig x t +W hg h t-1 +b g )

[0066] o t =σ(W io xt +W ho h t-1 +b o )

[0067] c t =f t ⊙c t-1 +i t ⊙g t

[0068] h t =o t ⊙tanh(c t )

[0069] Among them, x t represents the step input at the current moment, which is the real-time pressure sequence collected by the tactile data acquisition system with pressure sensor, h t-1 Represents the hidden state before the current moment, W ii 、W if 、W ig 、W io is the input weight matrix, W hi 、W hf 、W hg 、W ho is the hidden state weight matrix, b i 、b f 、b g 、b o is the bias term, i.e. the preset minimum effective pressure threshold. Among the six gate states, i t is the input gate, f t is the forget gate, g t is the candidate memory, o t is the output gate, c t is the cell state of LSTM, i.e. long-term memory unit, h t is in hidden state;

[0070] Bidirectional LSTM is used to capture the previous and next dependencies for comprehensive judgment. The formula is:

[0071]

[0072] in, represents the final hidden state of the forward LSTM, encoding the forward temporal features of the complete force application process. Represents the final hidden state of the backward LSTM, encoding the reverse temporal dependency.

[0073] Among them, the forward LSTM captures the force application process (such as the pressure increasing trend), and the backward LSTM identifies abnormal withdrawal (such as the sudden drop in pressure when the blood vessel is penetrated).

[0074] In order to better implement this solution, further, the physiological signal feature tensor H bio The method for obtaining comprises the following steps performed in sequence:

[0075] Step Q1: The simulated patient experiences a corresponding simulated situation in the simulated scenario;

[0076] Step Q2: collecting physical data of the simulated patient;

[0077] Step Q3: Use 1D convolution model to extract local anomalies. The formula is:

[0078]

[0079] Among them, O is the single layer output, c is the input channel index, c' is the output channel index, K is the convolution kernel width, that is, the time window length, k is the convolution kernel position index, i is the output position index, W c',c,k is the convolution kernel weight, I c,(i+k) is the input signal value, that is, the input value of the cth channel at time point i+k, b c' is the bias term, i.e. the preset feature activation threshold,

[0080] The dilated convolution is used to capture long-term trends, and its formula is:

[0081]

[0082] Among them, d is the expansion rate, that is, the sampling interval, x i is the value of the input signal x at time point i, k is the convolution kernel position index, x (i+d·k) is the position of the input signal after expansion, W k is the convolution kernel weight, K is the convolution kernel width, that is, the time window length;

[0083] The final output physiological signal feature tensor H bio for:

[0084] H bio =StackedLayers(O)

[0085] By stacking multiple 1D convolutional layers and dilated convolutional layers, high-order features are gradually extracted.

[0086] The WaveNet structure is then used to stack causal convolutional layers to ensure temporal causality and predict the probability of subsequent symptoms in simulated patients.

[0087] In order to better implement this solution, the method for obtaining the knowledge graph feature tensor is as follows:

[0088] The first step is knowledge embedding. TransR is used to convert clinical normative knowledge into vectors:

[0089] h r +r≈t r where h r =hM r , t r =tM r

[0090] Where h is the head entity vector, t is the tail entity vector, M r is the relation-specific projection matrix, h r is the head entity after relation-specific projection, t r is the embedding vector of the tail entity; after mapping the entity to the relation space, keep the head entity + relation ≈ tail entity

[0091] Then comes the graph attention propagation, whose node update formula is:

[0092]

[0093] The final output knowledge graph feature tensor H kg for:

[0094] H kg =[h′1,h′2,…,h′ N ]

[0095] That is, the knowledge graph feature tensor H kg is the embedding matrix of all nodes in the graph.

[0096] A system for evaluating infusion practices of medical trainees, based on the aforementioned method for evaluating infusion practices of medical trainees, includes a multimodal data acquisition tool and a multimodal infusion evaluation model. The multimodal data acquisition tool includes:

[0097] A camera that records medical staff's operation video in real time, a tactile data acquisition system that records medical staff's hand pressure data in real time, and a data acquisition and display device that records simulated patient vital sign monitoring data in real time;

[0098] The multimodal infusion evaluation model includes a cross-modal attention fusion layer, a visual feature extractor, a tactile feature extractor, a physiological signal processing module, and a knowledge graph embedding module. The cross-modal attention fusion layer fuses the visual feature tensor H extracted by the visual feature extractor. vis , the tactile feature tensor H extracted by the tactile feature extractor vis , the physiological signal feature tensor H extracted by the physiological signal processing module bio According to the different infusion scenarios set and the different operations captured by the interns, the feature tensors H of the knowledge graph modules are corresponding to differentkg Dynamically allocate visual feature tensor H vis , tactile feature tensor H vis , physiological signal feature tensor H bio The weight of the fused feature tensor H is then output. fusion , and then output the final score of the intern medical staff according to the preset scoring criteria.

[0099] In this solution, some repeated letters are used because the same letters are used to express different algorithms. However, since there are many algorithm formulas, using different letters will lead to greater complexity. Therefore, the letters in each algorithm are explained one by one separately, and the same letters in different formulas should not be understood as the same expression.

[0100] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

[0101] 1. The method and system for evaluating infusion procedures of medical trainees described in this invention utilizes a 360-degree camera, a tactile data acquisition system, and physiological monitoring equipment to simultaneously collect multi-dimensional data, including visual, tactile, and physiological signals, comprehensively covering operational details and enhancing the objectivity of the evaluation.

[0102] 2. The method and system described in this invention for evaluating infusion procedures performed by medical trainees utilizes a cross-modal attention mechanism to dynamically adjust the weights of each modality based on the operational scenario and real-time data, flexibly adapting to complex clinical scenarios such as abnormal vascular punctures or sudden changes in patient vital signs.

[0103] 3. The method and system for evaluating infusion procedures performed by medical trainees, as described in this invention, integrates the experience of senior medical staff with clinical standards, constructs a structured knowledge graph, ensures that the scoring criteria are highly consistent with real-world medical scenarios, and enhances the professionalism and authority of the evaluation.

[0104] 4. The method and system described in this invention for evaluating infusion procedures of medical trainees uses an artificial intelligence model to quickly compare interns' procedures with standard features, generating scores and improvement suggestions in real time, significantly shortening training cycles and reducing labor costs.

[0105] 5. The method and system described in the present invention for evaluating the infusion operation of trainee medical staff combine technologies such as spatiotemporal convolutional network (3DCNN), bidirectional LSTM and void convolution to capture local operation anomalies, temporal pressure changes and long-term physiological trends, respectively, to improve the sensitivity of error recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0106] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and should not be considered as limiting the scope. A person of ordinary skill in the art can also derive other relevant drawings based on these drawings without inventive effort, among which:

[0107] Figure 1 It is a flow chart of the method for evaluating the infusion operation of medical trainees according to the present invention. DETAILED DESCRIPTION

[0108] In order to make the objectives, technical solutions, and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present invention and are not intended to limit the present invention. That is, the embodiments described herein are only some embodiments of the present invention, not all embodiments. Generally, the components of the embodiments of the present invention described and illustrated in the drawings herein may be arranged and designed in various different configurations.

[0109] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention as claimed, but is merely intended to represent selected embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative work are within the scope of protection of the present invention.

[0110] It should be noted that relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus comprising the element.

[0111] The features and performance of the present invention are further described in detail below with reference to the embodiments.

[0112] Example 1

[0113] A preferred embodiment of the present invention provides a method for evaluating the infusion operation of medical trainees, such as Figure 1 ,include:

[0114] Multimodal acquisition of the intern medical staff's operations, wherein the multimodal acquisition includes a visual modality, a tactile modality, and a physiological signal modality;

[0115] The collected operations of medical interns are input into a multimodal infusion evaluation model, which extracts features based on the modalities and performs cross-modal attention fusion on the feature vectors of different modalities:

[0116] H fusion =Φ CA (H vis ,H tac ,H bio ,H kg )

[0117] Among them, H fusion is the total feature tensor after fusion, Φ CA is the cross-modal attention fusion function, H represents the feature tensor, H vis is the visual feature tensor, H tac is the tactile feature tensor, H bio is the physiological signal feature tensor, H kg is the knowledge graph feature tensor;

[0118] The calculated total feature tensor H fusion Compared with the standard total feature tensor H fusion ', output the final score of the intern medical staff according to the preset scoring criteria.

[0119] The multimodal infusion evaluation model includes modules for analyzing the visual modality, tactile modality, and physiological signal modality of medical trainees, as well as a knowledge graph module, which breaks down intravenous infusion in different situations into multiple steps.

[0120] The core principle of training a multimodal infusion assessment model is to break down intravenous infusion into multiple steps under different circumstances. This approach requires systematic design that combines clinical standards with teaching requirements. Below, we illustrate this with a simple example:

[0121] 1. Operation process breakdown

[0122] (1) Intravenous infusion is broken down into seven main steps (e.g., assessment, preparation, puncture) and 28 sub-steps (e.g., venting technique, tourniquet tightness), with each step having a clearly defined start and end boundary and time window (e.g., disinfection operation must be completed within 30 seconds).

[0123] (2) Set quality indicators for key nodes, such as:

[0124] Puncture angle error range: ±3°; if the detailed puncture angle error obtained from the video is large, you can set up an infrared sensor to collect data;

[0125] Disinfection area standard: ≥5cm diameter circular area, disinfected in two times.

[0126] 2. Quantification of sterility principle

[0127] Establish 15 aseptic operation checkpoints, including:

[0128] Hand disinfection time (seven-step hand washing method ≥ 40 seconds)

[0129] Article placement specifications (distance between contaminated area and clean area ≥ 30cm)

[0130] Statistics of the number of times crossing the sterile area (5 points will be deducted for each crossing)

[0131] 3. Composite evaluation system

[0132] Combining the three dimensions of operational quality (weight 60%), efficiency (weight 20%), and humanistic care (weight 20%):

[0133] Operation quality: puncture success rate, drip rate adjustment accuracy;

[0134] Efficiency: the deviation between the total time and the standard time (10 minutes);

[0135] Humanistic care: standardization of communication language and patient comfort assessment.

[0136] If patient comfort is difficult to assess because the intern nurse is using a simulated patient, it can be eliminated and the focus of humanistic care can be placed on the standardization of communication language. The audio of the intern nurse can be collected from the video to identify whether it contains keywords that require the patient to be reminded.

[0137] In addition, during the initial model training, key nodes will be labeled to establish infusion process standards. This requires the establishment of a labeling team, which can specifically include:

[0138] 1. Establish an expert committee

[0139] The annotation guidance group is composed of clinical head nurses (60%), infection control experts (20%), and medical education experts (20%), and is responsible for:

[0140] Arbitration of controversial cases (e.g., vascular positioning standards for patients with special body types);

[0141] Dynamic update of marking rules (revised annually or regularly)

[0142] 2. Gradual training of labelers

[0143] A three-level certification system can be implemented:

[0144] Level 1: Master basic operation annotation (recognize the completeness of steps);

[0145] Level 2: Able to mark complex errors (e.g., aseptic operation violation accompanied by communication failure);

[0146] Level 3: Has the right to make suggestions for optimization of the annotation system (special position for senior nurses).

[0147] At the same time, a quality control mechanism for annotations needs to be set up, which can include:

[0148] 1. Layered review process

[0149] Primary annotation → cross-validation (30% sample re-inspection) → expert final review (10% high-risk operation spot check)

[0150] Establish annotation consistency indicators (such as using Cohen's Kappa coefficient to measure)

[0151] 2. Dynamic Optimization Strategy

[0152] Establish an infusion error sample library and update it in real time, such as Figure 1 The dotted part in the figure uses the error sample library to reversely correct the annotation standard:

[0153] Collect typical error types that exceed a certain number of errors (e.g., 100) (e.g., special violation scenarios such as "dipping a cotton swab twice in disinfectant")

[0154] Every time a certain number of abnormal cases are added (e.g. 50 abnormal cases), a revision of the standard is triggered.

[0155] Establishing an evaluation system requires a deep intersection of clinical medicine, education, and data science. Through continuous iteration, a comprehensive assessment capability encompassing "operational standards, clinical thinking, and humanistic care" can be developed. Practical application requires dynamic adjustments based on the latest version of the "Intravenous Infusion Procedure and Scoring Standard" (for example, the current latest version is the revised ISO / TS23908, released in 2025).

[0156] At the same time, application extensions can be made for labeled data, including the following:

[0157] 1. Construction of teaching case library

[0158] The labeled data can be converted into a 3D training scene to establish:

[0159] (1) Standard operation demonstration video (with annotated heat map)

[0160] (2) Error case simulation system (e.g. insufficient disinfection range triggering a virtual patient infection alarm)

[0161] 2. Intelligent feedback generation

[0162] A personalized assessment report can be automatically generated based on the annotation system:

[0163] (1) Identification of weak links (e.g., "steps where aseptic operation violations are concentrated")

[0164] (2) Targeted training modules are recommended (such as AR training for puncture angle correction).

[0165] This solution is only a basic method for establishing a multimodal evaluation model. These feasible extension directions can be set up and developed according to specific needs.

[0166] Example 2

[0167] In this embodiment, based on the first embodiment, the weights of vision, touch, physiological signals and knowledge graph in the cross-modal attention fusion function are dynamically allocated in the following way: the multimodal infusion evaluation model is based on the visual feature tensor H of the infusion operation of the intern medical staff in the simulated scene. vis , tactile feature tensor H tac , simulate the patient's physiological signal feature tensor H bio According to the different infusion scenarios set and the different operations captured by the interns, the feature tensors H of the knowledge graph modules are corresponding to different kg Dynamically allocate visual feature tensor H vis , tactile feature tensor H tac , physiological signal feature tensor H bio The weight of .

[0168] In cross-modal attention fusion, the visual feature tensor H is first established through cross-modal attention. vis and the tactile feature tensor H vis The physiological signal feature tensor H bio And the fused visual-tactile features are gated and fused, where

[0169] Cross-modal attention builds visual feature tensor H vis and the tactile feature tensor H tac The association process is:

[0170] Query-key-value mapping:

[0171] Q=H vis W Q

[0172] K=H tac W K

[0173] V=H tac W V

[0174] Calculate the attention weight A:

[0175]

[0176] Among them, Q, K, and V are query, key, and value matrices respectively, and d k is the scaling factor, T is the length of the time dimension;

[0177] Initial feature fusion obtains the visual-tactile fusion vector H vis-tac :

[0178] H vis-tac =A·V

[0179] The physiological signal feature tensor H bio , knowledge graph feature tensor H kg and the visual-tactile fusion vector H vis-tac Fusion is performed to obtain the total feature tensor H fusion :

[0180] LSTM models the delayed characteristics of physiological responses to obtain the intermediate tensor H bio-timed :

[0181] H bio-timed =LSTM(H bio ,H vis-tac )

[0182] Finally, gated multimodal fusion is performed:

[0183] H fusion =g⊙H vis-tac +(1-g)⊙(αH bio-timed +βH kg )

[0184] g=σ(W g [H vis-tav ||H bio-timed ||H kg ]+b g )

[0185] Where g is the dynamic gating value generated based on visual-tactile confidence, α and β are modal balance coefficients, σ ​​is the activation function, and W g is the weight matrix, which assigns weights to each tensor; || is the splicing operation b g is the bias scalar.

[0186] When the visual-tactile confidence is high (e.g., sufficient lighting and normal sensor function), g≈1, and the situation primarily depends on the characteristics of the operation. This is categorized into vision-dominated and touch-dominated scenarios. In vision-dominated scenarios, such as when the camera clearly captures the needle tip, the visual modality weight is >70%. In touch-dominated scenarios, such as during blind needle insertion, the pressure signal weight is increased to 65%.

[0187] When a physiological abnormality is detected (such as a sudden drop in blood pressure), g is reduced and the weight of the physiological signal is enhanced, thereby achieving dynamic weight distribution of visual, tactile, and physiological signals.

[0188] In this process, the feature tensor H of the knowledge graph module kg Different knowledge graph matching will be performed according to specific scenarios, generally including at least step integrity testing and sterility standard testing, among which

[0189] Step completeness detection uses CTC loss to align operation sequences:

[0190]

[0191] Build a violation pattern classifier for sterility compliance testing:

[0192] p(v|θ)=sigmid(W c H fusion +b c )

[0193] Wherein the visual feature tensor H vis The method of obtaining is to use a 3D convolution model to process the collected operation video, process local spatiotemporal details, and detect local anomalies. In order to better detect certain parameters, such as the angle between the infusion needle and the patient's skin, it is also possible to add recognition functions to the visual feature processing function, or add an infrared recognition device to identify the needle tube angle. However, these methods are relatively conventional and are not the core focus of this solution. They are not described in detail in this article and can be expanded according to needs when used. The convolution kernel parameters are (t×h×w) and the calculation formula is:

[0194]

[0195] Where C represents the number of input channels, T represents the length of the time dimension, i.e., the number of consecutive video frames, H represents the input height, i.e., the video resolution height, such as 256 pixels, corresponding to the vertical range of the operation area, W represents the input width, i.e., the video resolution width, such as 256 pixels, corresponding to the horizontal range of the operation area, and K represents the input width, i.e., the video resolution width, such as 256 pixels, corresponding to the horizontal range of the operation area. c',c,t,h,wrepresents the convolution kernel weight, where c' is the output channel, t is the time dimension kernel size, that is, the number of frames of continuous action capture, h is the spatial kernel height, w is the spatial kernel width, I is the input feature map, i is the time axis coordinate, j is the height axis position, k is the width axis position, b c' Represents the bias term, that is, the preset feature activation threshold;

[0196] Residual connections are then used to alleviate the vanishing gradient. The formula is:

[0197]

[0198] Here, the shallow detail features (such as hand micro-movements) are retained through jump connections to prevent the deep network from losing key operation information.

[0199] In addition, a collaborative cascade architecture is formed using ViT and 3D convolutional layers to capture global semantic associations. The 3D feature map needs to be expanded into a spatiotemporal tag sequence, including:

[0200] Chunked embedding:

[0201]

[0202] Among them, z p is the embedded representation of the pth block of the input video, carrying local semantic and position information, E is the embedding matrix, learning to map the original pixels to the feature space, the vec function flattens the block into a vector, V p represents the p-th spatiotemporal block of the input video, Representation position encoding, marking the spatial-temporal position of each block;

[0203] Multi-head self-attention:

[0204] MSA(Z)=Concat(head1,…,head h )W O

[0205]

[0206] Among them, MSA(Z) is the final output of multi-head self-attention, Z is the input feature matrix, that is, the output z of the previous block embedding p The Concat function represents the concatenation of the outputs of h heads in the feature dimension. i is the output of the i-th attention head, such as the output feature of the “device motion tracking head”, W O is the output projection matrix, Q i , K i 、V i are query, key, and value matrices respectively, d kis a scaling factor to prevent the dot product result from being too large and causing gradient instability;

[0207] Here Q i , K i 、V i The three matrices are:

[0208]

[0209] Q i , K i 、V i The input feature matrix Z is used to project the i-th head onto d k dimensional space to obtain the query, key, and value matrix,

[0210] Perform residual connection and layer normalization:

[0211] Z'=LayerNorm(Z+MSA(Z))

[0212] The final output is:

[0213] H vis =LayerNorm(Z'+FFN(Z'))

[0214] Among them, H vis is the final visual feature output, and FFN is a feed-forward network.

[0215] The following uses the disinfection process as an example to illustrate the collection and calculation process of visual features:

[0216] The key points in the nurse's standard operation are:

[0217] 1. Use a cotton swab to disinfect a circular area with a diameter of ≥5cm centered on the puncture point;

[0218] 2. Keep the cotton swab at a 30° angle to the skin and apply about 30g of force;

[0219] 3. Alternate the disinfection directions (clockwise → counterclockwise).

[0220] The focus of our data collection is to use the top camera to record the disinfection trajectory (60 frames per second) and the macro lens to capture the deformation of the cotton swab fiber.

[0221]

[0222] The medical correspondence in the example is: the RGB color channel of the c input channel corresponds to the color change of the disinfectant; the t time dimension detects whether the disinfection duration is ≥ 15 seconds; the h and w spatial dimensions mainly calculate whether the disinfection area meets the standard; and the important feature learned by k is whether the cotton swab motion trajectory pattern satisfies the alternation of two disinfection directions.

[0223] Residual connection formula

[0224]

[0225] The medical correspondence in

[15] is to preserve shallow details and prevent the deep network from losing key actions (such as changes in disinfection direction). For example, the third-layer features may lose the cotton swab rotation direction information, and the residual connection ensures that this information is passed to the deep layer.

[0226] The tactile feature tensor H tac The method of obtaining the pressure is to use the long short-term memory network LSTM to analyze the simulated patient pressure change curve and identify the three-stage characteristics of "touch-puncture-stabilization". The collected pressure time series data T = {t1,…,t n The gating mechanism formula of} is:

[0227] i t =σ(W ii x t +W hi h t-1 +b i )

[0228] f t =σ(W if x t +W hf h t-1 +b f )

[0229] g t =tanh(W ig x t +W hg h t-1 +b g )

[0230] o t =σ(W io x t +W ho h t-1 +b o )

[0231] c t =f t ⊙c t-1 +i t ⊙g t

[0232] h t =o t ⊙tanh(c t )

[0233] Among them, x trepresents the current step input, which is the real-time pressure sequence collected by the tactile glove with pressure sensor, h t-1 Represents the hidden state before the current moment, W ii 、W if 、W ig 、W io is the input weight matrix, W hi 、W hf 、W hg 、W ho is the hidden state weight matrix, b i 、b f 、b g 、b o is the bias term, i.e. the preset minimum effective pressure threshold. Among the six gate states, i t is the input gate, f t is the forget gate, g t is the candidate memory, o t is the output gate, c t is the cell state of LSTM, i.e. long-term memory unit, h t is in hidden state;

[0234] Forget Gate f t Adjustment: During the puncture process, the continuous pressure signal (such as the stable stage of needle holding) will be enhanced, while the transient jitter noise (such as hand tremor) will be suppressed;

[0235] Bidirectional LSTM is used to capture the previous and next dependencies for comprehensive judgment. The formula is:

[0236]

[0237] in, represents the final hidden state of the forward LSTM, encoding the forward temporal features of the complete force application process. Represents the final hidden state of the backward LSTM, encoding the reverse temporal dependency.

[0238] Among them, the forward LSTM captures the force application process (such as the pressure increasing trend), and the backward LSTM identifies abnormal withdrawal (such as the sudden drop in pressure when the blood vessel is penetrated).

[0239] The following example illustrates that if the nurse's standard operation is

[0240] 1. Hold the needle at an angle of 15-30° and insert the needle at a constant speed;

[0241] 2. The resistance of the blood vessel wall is about 80g, and the pressure drops to 40g after penetration;

[0242] 3. Hand shaking causes pressure fluctuations of less than ±20g.

[0243] We used a tactile glove to collect pressure data 1000 times per second, while the syringe piston rod monitored the advancement resistance.

[0244] Then the input gate i t Determine the current pressure value x t How much information needs to be stored in memory, such as when a sudden pressure drop is detected (possibly penetrating the posterior wall of the blood vessel), the input gate i t The value of increases, emphasizing the importance of the current abnormal signal; the forget gate f t Control the degree of retention of historical memory. In the stable needle holding stage, the forget gate f t The value is close to 1, and the correct pressure pattern (40-60g) is continuously memorized; the candidate memory g t Generate a new feature representation for the current time step, such as converting the original pressure data into a "stable / fluctuating" feature, such as g t Positive values ​​represent stable pressure; cell state c t Integrate long-term memory with current information and continuously update cell state c t To reflect the trend of operation quality, for example: when f t =0.9, retaining 90% of historical memory; when i t =0.8, absorb 80% of the current pressure characteristics; output gate o t Control the output ratio of cell status information, such as when abnormal jitter is detected (o t When the signal level is reduced, non-critical information transmission is suppressed and abnormal signals are highlighted.

[0245] Forward propagation Identify the timing characteristics of normal operation: pressure increase phase (needle insertion): 10g→80g→40g, typical time consumption: 2.3±0.5 seconds;

[0246] Backpropagation Detect reverse abnormal patterns: such as false withdrawal: the pressure drops suddenly from 80g to 20g (indicating excessive relaxation after vessel penetration), compensatory jitter: pressure fluctuations accompanied by high-frequency acceleration changes (>3G).

[0247] The physiological signal feature tensor H bio The method for obtaining comprises the following steps performed in sequence:

[0248] Step Q1: The simulated patient experiences a corresponding simulated situation in the simulated scenario;

[0249] Step Q2: collecting physical data of the simulated patient;

[0250] Step Q3: Use 1D convolution model to extract local anomalies. The formula is:

[0251]

[0252] Among them, O is the single layer output, c is the input channel index, c' is the output channel index, K is the convolution kernel width, that is, the time window length, k is the convolution kernel position index, i is the output position index, W c',c,k is the convolution kernel weight, I c,(i+k) is the input signal value, that is, the input value of the cth channel at time point i+k, b c' is the bias term, i.e. the preset feature activation threshold,

[0253] The dilated convolution is used to capture long-term trends, and its formula is:

[0254]

[0255] Among them, d is the expansion rate, that is, the sampling interval, x i is the value of the input signal x at time point i, k is the convolution kernel position index, x (i+d·k) is the position of the input signal after expansion, W k is the convolution kernel weight, K is the convolution kernel width, that is, the time window length; the expansion factor is used to expand the receptive field to capture the long-term trend of blood pressure changes (such as continuous low blood pressure warning shock risk)

[0256] The final output physiological signal feature tensor H bio for:

[0257] H bio =StackedLayers(O)

[0258] By stacking multiple 1D convolutional layers and dilated convolutional layers, high-order features are gradually extracted.

[0259] The WaveNet structure is then used to stack causal convolutional layers to ensure temporal causality and predict the probability of subsequent symptoms in simulated patients.

[0260] Let's use an example to illustrate the clinical scenario:

[0261] The nurse accidentally injured the artery during puncture, and the simulated patient showed:

[0262] Sudden increase in blood pressure (systolic blood pressure from 120 to 160 mmHg)

[0263] Decreased blood oxygen saturation (SpO2 from 98% to 89%)

[0264] The standard requirement is: identify and activate the emergency plan within 10 seconds.

[0265] In this scenario, the data collection we need to focus on are: continuous output of ECG waveform by the electrocardiogram monitor (lead II) and beat-to-beat blood pressure monitoring (FinometerPro, sampling rate 200Hz).

[0266] Then in the 1D convolutional model, k = 5: analyze physiological signals within a 5-second time window and detect local anomalies:

[0267] Widened QRS complex (may indicate ventricular arrhythmia)

[0268] Blood pressure fluctuation standard deviation >15% (a sign of vascular damage)

[0269] In dilated convolution, d = 2: interval sampling, the receptive field is expanded to 15 seconds, and gradual changes are recognized:

[0270] Slowly decreasing blood pressure (5 mmHg drop every 10 seconds → shock warning)

[0271] Blood oxygen saturation continues to decrease (2% decrease per minute → risk of pneumothorax).

[0272] The WaveNet architecture predicts the probability of complications in the next 3 seconds (e.g., predicting the risk of ventricular fibrillation based on the current ECG).

[0273] The method for obtaining the knowledge graph feature tensor is:

[0274] The first step is knowledge embedding. TransR is used to convert clinical normative knowledge into vectors:

[0275] h r +r≈t r where h r =hM r , t r =tM r

[0276] Where h is the head entity vector, t is the tail entity vector, M r is the relation-specific projection matrix, h r is the head entity after relation-specific projection, t r is the embedding vector of the tail entity; after mapping the entity to the relation space, keep the head entity + relation ≈ tail entity

[0277] Then comes the graph attention propagation, whose node update formula is:

[0278]

[0279] Among them, α ij is the attention weight.

[0280] The final output knowledge graph feature tensor H kg for:

[0281] H kg =[h′1,h′2,…,h′ N ]

[0282] That is, the knowledge graph feature tensor H kg is the embedding matrix of all nodes in the graph.

[0283] If the clinical scenario is:

[0284] The nurse is preparing to administer an infusion of 10% potassium chloride solution. The standard requirements are:

[0285] 1. A central venous catheter must be used;

[0286] 2. Infusion rate ≤10mmol / h.

[0287] In the TransR knowledge embedding formula in this scenario, h = "potassium chloride", r = "route of administration", and t = "central vein". When the model detects peripheral intravenous infusion, the calculation of || h r +rt r || The distance increases, triggering an alarm. In the attention propagation formula, node i = "disinfection operation", neighbor node j = "infection risk", if the disinfection time is insufficient (h i Abnormal), it will lead to: attention weight α ij It increased to 0.85, and the correlation triggered a "high risk of infection" warning.

[0288] Example 3

[0289] A system for evaluating infusion operations of medical trainees, based on the methods for evaluating infusion operations of medical trainees described in Examples 1 and 2, includes a multimodal data acquisition tool and a multimodal infusion evaluation model. The multimodal data acquisition tool includes:

[0290] The camera records the medical staff's operation video in real time, with a panoramic view. The equipment configuration is:

[0291] 1. 360° panoramic camera: Install an Insta360 Pro 2 camera (8K resolution, 60fps) 30cm above the operating table;

[0292] 2. Macro lens: Install a Sony Alpha 7IV + 90mm macro lens just above the puncture area (to capture the details of the needle tip);

[0293] 3. Head-mounted camera: Nurse wearing GoPro HERO10 Black (first-person perspective recording of hand movements)

[0294] At the same time, we also need to control the lighting, we use:

[0295] 1. Ring fill light: Install Dracast Halo 300 LED lights (color temperature 5600K, illumination 2000lux) around the operating table

[0296] 2. Polarizing filter: eliminates interference from instrument reflections.

[0297] For spatial calibration use:

[0298] 1. Checkerboard calibration plate: Lay ArUco markers on the surface of the operating table (accuracy ±0.1mm)

[0299] 2. Multi-camera synchronization: Use Genlock synchronization signal generator (time alignment error <1ms)

[0300] The tactile data acquisition system records the hand pressure data of medical staff in real time, including:

[0301] 1. Pressure sensing device

[0302] High-precision tactile gloves are used here:

[0303] Model: TactGlove Mk3 (16 pressure sensors on the palm and 9 tactile points on the fingertips)

[0304] Measuring range: 0-200g, resolution ±3g

[0305] Sampling rate: 1kHz (transmitted via Bluetooth 5.2)

[0306] 2. Equipment modification

[0307] (1) Smart tourniquet: integrated strain gauge sensor (measures tightness, range 0-50N) and built-in IMU (detects tying angle)

[0308] (2) Syringe force feedback: A micro weighing sensor (HX711 module, accuracy ±0.5g) is installed on the piston rod

[0309] 3. Surface electromyography acquisition

[0310] sEMG sensor: Delsys TrignoAvanti sensor (sampling rate 1926Hz) placed on the nurse's forearm flexor muscles

[0311] Monitor muscle tension and operational stability.

[0312] A data acquisition and display device that records the patient's vital signs monitoring data in real time, using a vital signs simulator:

[0313] Model: Laerdal SimMan 3G

[0314] Connection parameters:

[0315] ECG lead: II lead simulates ECG waveform (abnormal waveform injection function)

[0316] Blood pressure cuff: Dynamic simulation of systolic blood pressure fluctuations of 80-180 mmHg

[0317] Blood oxygen sensor: SpO2 programmable adjustment (range 70-100%)

[0318] At the same time, a blood vessel simulation device is also required, namely a bionic vein model: it uses a silicone material with a Shore hardness of 20A to simulate veins, has a built-in pressure sensor array (Tekscan4011, with a spacing of 2mm), and uses infrared markers for 3D motion capture system tracking.

[0319] The multimodal infusion evaluation model includes a cross-modal attention fusion layer, a visual feature extractor, a tactile feature extractor, a physiological signal processing module, and a knowledge graph embedding module. The cross-modal attention fusion layer fuses the visual feature tensor H extracted by the visual feature extractor. vis , the tactile feature tensor H extracted by the tactile feature extractor vis , the physiological signal feature tensor H extracted by the physiological signal processing module bio According to the different infusion scenarios set and the different operations captured by the interns, the feature tensors H of the knowledge graph modules are corresponding to different kg Dynamically allocate visual feature tensor H vis , tactile feature tensor H vis , physiological signal feature tensor H bio The weight of the fused feature tensor H is then output. fusion , and then output the final score of the intern medical staff according to the preset scoring criteria.

[0320] The above description is only a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions and improvements made by any technician familiar with the field within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for evaluating the infusion operation of trainee medical staff, characterized in that: The method comprises: Multimodal acquisition of intern medical staff's operations, including visual modality, tactile modality, and physiological signal modality: The collected operations of medical interns are input into a multimodal infusion evaluation model, which extracts features based on the modalities and performs cross-modal attention fusion on the feature vectors of different modalities: H fusion =Φ CA (H vis ,H tac ,H bio ,H kg ) Among them, H fusion is the total feature tensor after fusion, Φ CA is the cross-modal attention fusion function, H represents the feature tensor, H vis is the visual feature tensor, H tac is the tactile feature tensor, H bio is the physiological signal feature tensor, H kg is the knowledge graph feature tensor; The calculated total feature tensor H fusion Compared with the standard total feature tensor H fusion ′, output the final score of the intern medical staff according to the preset scoring criteria.

2. A method for evaluating the infusion operation of medical trainees according to claim 1, characterized in that: In the cross-modal attention fusion function, the weights of vision, touch, physiological signals and knowledge graph are dynamically allocated: the multimodal infusion evaluation model is based on the visual feature tensor H of the infusion operation of the intern medical staff in the simulated scene. vis , tactile feature tensor H tac , simulate the patient's physiological signal feature tensor H bio According to the different infusion scenarios set and the different operations captured by the interns, the feature tensors H of the knowledge graph modules are corresponding to different kg Dynamically allocate visual feature tensor H vis , tactile feature tensor H tac , physiological signal feature tensor H bio The weight of .

3. A method for evaluating the infusion operation of medical trainees according to claim 1, characterized in that: In cross-modal attention fusion, the visual feature tensor H is first established through cross-modal attention. vis and the tactile feature tensor H vis The physiological signal feature tensor H bio And the fused visual-tactile features are gated and fused, where Cross-modal attention builds visual feature tensor H vis and the tactile feature tensor H tac The association process is: Query-key-value mapping: Q=H vis W Q K=H tac W K V=H tac W V Calculate the attention weight A: Among them, Q, K, and V are query, key, and value matrices respectively, and d k is the scaling factor, T is the length of the time dimension; Initial feature fusion obtains the visual-tactile fusion vector H vis-tac : H vis-tac =A·V The physiological signal feature tensor H bio , knowledge graph feature tensor H kg and the visual-tactile fusion vector H vis-tac Fusion is performed to obtain the total feature tensor H fusion : LSTM models the delayed characteristics of physiological responses to obtain the intermediate tensor H bio-timed : H bio-timed =LSTM(H bio ,H vis-tac ) Finally, gated multimodal fusion is performed: H fusion =g⊙H vis-tac +(1-g)⊙(αH bio-timed +βH kg ) g=σ(W g [H vis-tac ||H bio-timed ||H kg ]+b g ) Where g is the dynamic gating value generated based on visual-tactile confidence, α and β are modal balance coefficients, σ ​​is the activation function, and W g is the weight matrix, which assigns weights to each tensor; || is the splicing operation b g is the bias scalar.

4. A method for evaluating the infusion operation of medical trainees according to claim 1, characterized in that: Wherein the visual feature tensor H vis The method of obtaining is to use the 3D convolution model to process the collected operation video. The convolution kernel parameters are (t×h×w) and the calculation formula is: Among them, C represents the number of input channels, T represents the length of the time dimension, H represents the input height, w represents the input width, K c′,c,t,h，w Represents the convolution kernel weight, where c′ is the output channel, t is the time dimension kernel size, h is the spatial kernel height, w is the spatial kernel width, I is the input feature map, i is the time axis coordinate, j is the height axis position, k is the width axis position, and b is the time axis coordinate. c′ represents the bias term; Residual connections are then used to alleviate the vanishing gradient. The formula is: In addition, a collaborative cascade architecture is formed using ViT and 3D convolutional layers to expand the 3D feature map into a spatiotemporal tag sequence, including: Chunked embedding: Among them, z p is the embedding representation of the pth block of the input video, E is the embedding matrix, the vec function flattens the block into a vector, V p represents the p-th spatiotemporal block of the input video, Representation position encoding, marking the spatial-temporal position of each block; Multi-head self-attention: MSA(Z)=Concat(head1,…,head h )W O Among them, MSA(Z) is the final output of multi-head self-attention, Z is the input feature matrix, that is, the output z of the previous block embedding p The Concat function represents the concatenation of the outputs of h heads in the feature dimension. i is the output of the i-th attention head, W O is the output projection matrix, Q i , K i 、V i are query, key, and value matrices respectively, d k is the scaling factor; Perform residual connection and layer normalization: Z′=LayerNorm(Z+MSA(Z)) The final output visual feature tensor H vis : H vis =LayerNorm(Z′+FFN(Z′)) Among them, FFN is a feed-forward network.

5. The method for evaluating the infusion operation of medical trainees according to claim 1, characterized in that: The tactile feature tensor H tac The method of obtaining the pressure is to use the long short-term memory network LSTM to analyze the simulated patient pressure change curve and identify the three-stage characteristics of "touch-puncture-stabilization". The collected pressure time series data T = {t1, ..., t n The gating mechanism formula of} is: i t =σ(W ii x t +W hi h t-1 +b i ) f t =σ(W if x t +W hf h t-1 +b f ) g t =tanh(W ig x t +W hg h t-1 +b g ) o t =σ(W io x t +W ho h t-1 +b o ) c t =f t ⊙c t-1 +i t ⊙g t h t =o t ⊙tanh(c t ) Among them, x t Indicates the current moment step input, that is, the real-time pressure sequence collected, h t-1 Represents the hidden state before the current moment, W ii 、W if 、W ig 、W io is the input weight matrix, W hi 、W hf 、W hg 、W ho is the hidden state weight matrix, b i 、b f 、b g 、b o is the bias term, i.e. the preset minimum effective pressure threshold. Among the six gate states, i t is the input gate, f t is the forget gate, g t is the candidate memory, ot is the output gate, c t is the cell state of LSTM, i.e. long-term memory unit, h t is in hidden state; Bidirectional LSTM is used to capture the previous and next dependencies for comprehensive judgment to obtain the tactile feature tensor H tac : in, represents the final hidden state of the forward LSTM, Represents the final hidden state of the backward LSTM.

6. A method for evaluating the infusion operation of medical trainees according to claim 1, characterized in that: The physiological signal feature tensor H bio The method for obtaining comprises the following steps performed in sequence: Step Q1: The simulated patient experiences a corresponding simulated situation in the simulated scenario; Step Q2: collecting physical data of the simulated patient; Step Q3: Use 1D convolution model to extract local anomalies. The formula is: Among them, O is the single layer output, c is the input channel index, c′ is the output channel index, K is the convolution kernel width, k is the convolution kernel position index, i is the output position index, W c′,c，k is the convolution kernel weight, I c，(i+k) is the input signal value, b c′ is the bias term; The dilated convolution is used to capture long-term trends, and its formula is: Where d is the expansion rate, x i is the value of the input signal x at time point i, k is the convolution kernel position index, x (i+d·k) is the position of the input signal after expansion, W k is the convolution kernel weight, K is the convolution kernel width, The final output physiological signal feature tensor H bio for: H bio =StackedLayers(O) By stacking multiple 1D convolutional layers and dilated convolutional layers, high-order features are gradually extracted.

7. A method for evaluating the infusion operation of medical trainees according to claim 1, characterized in that: The knowledge graph feature tensor H kg The method to obtain is: The first step is knowledge embedding. TransR is used to convert clinical normative knowledge into vectors: h r +r≈t r where h r =hM r ,t r =tM r Where h is the head entity vector, t is the tail entity vector, M r is the relation-specific projection matrix, h r is the head entity after relation-specific projection, t r is the embedding vector of the tail entity; Then comes the graph attention propagation, whose node update formula is: The final output knowledge graph feature tensor H kg for: H kg =[h′1,h′2,…,h′ N ] That is, the knowledge graph feature tensor H kg is the embedding matrix of all nodes in the graph.

8. A system for evaluating the infusion operation of medical trainees, based on the method for evaluating the infusion operation of medical trainees according to any one of claims 1 to 7, characterized in that: The multimodal data acquisition tool and the multimodal infusion assessment model are included. The multimodal data acquisition tool includes: A camera that records medical staff's operation video in real time, a tactile data acquisition system that records medical staff's hand pressure data in real time, and a data acquisition and display device that records simulated patient vital sign monitoring data in real time; The multimodal infusion evaluation model includes a cross-modal attention fusion layer, a visual feature extractor, a tactile feature extractor, a physiological signal processing module, and a knowledge graph embedding module. The cross-modal attention fusion layer fuses the visual feature tensor H extracted by the visual feature extractor. vis , the tactile feature tensor H extracted by the tactile feature extractor tac , the physiological signal feature tensor H extracted by the physiological signal processing module bio According to the different infusion scenarios set and the different operations captured by the interns, the feature tensors H of the knowledge graph modules are corresponding to different kg Dynamically allocate visual feature tensor H vis , tactile feature tensor H tac , physiological signal feature tensor H bio The weight of the output fusion feature tensor H fusion , and then output the final score of the intern medical staff according to the preset scoring criteria.

Images