Venipuncture training error correction system and method based on multi-modal perception

Abstract

The embodiment of the invention discloses a vein puncture training error correction system and method based on multi-modal perception, and belongs to the technical field of medical teaching systems. The problem that the real-time performance and the accuracy of error correction in venipuncture training are low is solved. The data acquisition unit is used for acquiring corresponding venipuncture training operation data and blood vessel response data of the trainee in the current training mode; the coding unit is used for coding the blood vessel response data in real time and constructing a blood vessel state coding sequence; the association unit is used for associating the venipuncture training operation data with the blood vessel state coding sequence to form a coupled data stream; the prediction unit is used for inputting the coupled data flow into a preset venipuncture deviation prediction model, performing operation deviation prediction and generating compensation parameters based on an operation deviation prediction result; and the error correction unit is used for sending an early warning to the trainee based on the operation deviation prediction result and the compensation parameter, and giving operation guide data to realize vein puncture training error correction.

Description

Technical Field

[0001] This application relates to the field of medical teaching system technology, and in particular to a multimodal perception-based system and method for error correction in intravenous puncture training. Background Technology

[0002] Venous puncture is a crucial clinical nursing skill, and the proficiency and accuracy of its application directly impact patient treatment outcomes. Current mainstream skills training commonly utilizes simulated arms made of materials such as silicone as basic training aids.

[0003] Currently, during training, the assessment and correction of trainees' operations rely on the on-site observation and manual guidance of instructors. Instructors must visually monitor the trainee's needle holding posture, needle insertion angle, puncture depth, and vessel fixation technique throughout the entire process, judging whether the operation is standardized based on their personal experience. When problems such as angle deviation, excessive needle insertion, or vessel slippage are found, immediate corrections are made through verbal prompts or demonstrations.

[0004] However, manual assessment lacks objective quantitative standards. Key parameters such as angle, depth, and force can only be qualitatively described as too large or too deep, making it difficult to provide precise numerical feedback. Secondly, guidance from instructors is often delayed, with errors only being pointed out after completion. Trainees struggle to adjust and develop correct muscle memory in a timely manner, resulting in low real-time accuracy and precision in error correction during intravenous puncture training. Summary of the Invention

[0005] This application provides a multimodal perception-based error correction system and method for intravenous puncture training, which addresses the following technical problems: manual evaluation lacks objective quantitative standards, and errors are often only pointed out after completion, resulting in low real-time performance and accuracy of error correction during intravenous puncture training.

[0006] The embodiments of this application adopt the following technical solutions: This application provides a multimodal perception-based intravenous puncture training error correction system. It includes a data acquisition unit, an encoding unit, an association unit, a prediction unit, and an error correction unit. The data acquisition unit acquires intravenous puncture training operation data and vascular response data transmitted by a multimodal sensor installed within a simulated arm, corresponding to the trainee's intravenous puncture training in the current training mode. The simulated arm corresponds to different training modes, each simulating the arm shape and vascular pattern of patients of different ages and conditions. The encoding unit encodes the vascular response data in real time and sorts it chronologically to construct a vascular state encoding sequence. The association unit temporally associates the intravenous puncture training operation data with the vascular state encoding sequence, forming a coupled data stream between the training operation and changes in vascular state. The prediction unit inputs the coupled data stream into a pre-set intravenous puncture deviation prediction model to predict operational deviations in the intravenous puncture training operation under the current training mode and generates compensation parameters based on the prediction results. The error correction unit issues warnings to the trainee and provides operational guidance data based on the operational deviation prediction results and compensation parameters, thereby achieving intravenous puncture training error correction.

[0007] In one implementation of this application, the encoding unit is specifically used for: determining the trainee's current operation stage based on intravenous puncture training operation data, and outputting the corresponding operation stage identifier; wherein the operation stage includes at least one of the following: vascular fixation stage, needle tip approach stage, puncture into the vascular wall stage, and catheter insertion stage; converting vascular response data into a feature vector composed of multi-dimensional feature values; wherein the multi-dimensional feature values ​​include at least one of the following: mean vascular pressure, gradient of vascular pressure change, variance of vascular displacement, and rate of vascular deformation; matching the value in the feature vector corresponding to the current moment with the transition condition corresponding to the current operation stage identifier, and determining the vascular state corresponding to the current moment based on the matching result; wherein the transition condition is a preset vascular state judgment rule based on the operation stage identifier and the feature vector; combining and encoding the vascular state and the feature vector, and arranging them in chronological order to generate a vascular state encoding sequence.

[0008] In one implementation of this application, the association unit is specifically used for: establishing a first storage area for caching intravenous puncture training operation data, and establishing a second storage area for caching vascular state codes; when writing the current vascular state code into the second storage area, determining a backtracking time window based on the operation stage and timestamp associated with the current vascular state code; traversing the first storage area and filtering operation data whose timestamps satisfy the backtracking time window, as a set of historical operation data related to the current vascular state code; determining the association strength weight between each operation data in the historical operation data set and the current vascular state code based on a preset weight rule library; wherein the preset weight rule library stores the corresponding weight values ​​using operation stage, operation feature type, and vascular state code as a joint index; encapsulating the current vascular state code, the historical operation data set, and the corresponding association strength weights into a time-stamped coupled data unit, and outputting them in chronological order to form a coupled data stream.

[0009] In one implementation of this application, the system further includes an input sequence construction unit, specifically used for: generating a mode condition vector based on the current training mode, and fusing the mode condition vector with each coupled data unit in the coupled data stream to generate a mode-conditional coupled data stream; wherein, the mode condition vector is a category identifier reflecting the current training mode; and constructing an input sequence based on multiple coupled data units in the mode-conditional coupled data stream, and inputting it into a preset venipuncture deviation prediction model.

[0010] In one implementation of this application, the prediction unit is specifically used for: determining the correlation between coupled data units at different times in the input sequence through the attention mechanism of a pre-set venipuncture deviation prediction model; wherein the weight parameters used to calculate the correlation in the attention mechanism are dynamically modulated by the pattern condition vector; determining the influence weight of each historical coupled data unit in the input sequence on the vascular state of the current coupled data unit based on the modulated correlation; weighting and fusing the operation feature vectors in the historical coupled data units based on the influence weights to generate the operation attribution vector corresponding to the current coupled data unit; wherein the operation attribution vector is used to represent the contribution distribution of each historical training operation that leads to the current vascular state; and inputting the operation attribution vector, the current vascular state encoding, and the pattern condition vector into the prediction head of the pre-set venipuncture deviation prediction model to determine the operation deviation prediction result through the prediction head.

[0011] In one implementation of this application, the system further includes a parameter calculation unit, specifically used for: extracting the predicted deviation type, deviation value, and predicted occurrence time based on the operation deviation result; determining a reference compensation benchmark value for the deviation type based on the vascular physiological feature library corresponding to the current training mode; calculating a dynamic compensation coefficient based on a preset compensation coefficient mapping function and the deviation value; wherein the compensation coefficient mapping function is configured such that the larger the deviation value, the larger the dynamic compensation coefficient; calculating a time urgency coefficient for compensation intervention based on the time difference between the predicted occurrence time and the current time; and generating compensation parameters through weighted calculation based on the reference compensation benchmark value, the dynamic compensation coefficient, and the time urgency coefficient; wherein the compensation parameters include at least one of a mechanical parameter for tactile feedback guidance and a display parameter for visual feedback guidance.

[0012] In one implementation of this application, the error correction unit is specifically used to: generate a warning signal for local directional tactile feedback within the simulated arm based on the deviation prediction result and compensation parameters; and generate visual operation guidance containing the target path and real-time deviation indication on the real-world image of the simulated arm based on the compensation parameters; and dynamically adjust the operation guidance data according to the trainee's adjustment of training operations relative to the compensation parameters to achieve error correction in intravenous puncture training.

[0013] In one implementation of this application, the system further includes a tactile feedback unit, specifically used for: determining the type of operation deviation based on the operation deviation prediction result; when the deviation type is needle insertion angle deviation, driving the targeted tactile array within the simulated arm based on the compensation direction, so that the tactile units in the targeted tactile array located on the opposite side of the compensation direction generate a directional vibration sequence simulating the lateral compression sensation of the blood vessel wall; when the deviation type is needle insertion depth deviation, driving the tactile unit located directly below the simulated blood vessel to generate a preset pulse vibration sequence to simulate the bottoming sensation of the needle tip touching the blood vessel bed; wherein, the targeted tactile array consists of multiple independently controlled micro-vibration tactile units distributed in a grid pattern in the surrounding tissue layer of the simulated blood vessel.

[0014] In one implementation of this application, the system further includes an operation guidance unit, specifically used for: calling a preset three-dimensional vascular mechanical model based on the target vascular pattern corresponding to the compensation parameters, and obtaining the deformation safety domain and the corresponding target needle insertion path of the vascular based on the collected vascular wall pressure data; determining the dynamic spatial tolerance range of the needle insertion operation based on the deformation safety domain and the corresponding target needle insertion path; acquiring the spatial position of the simulated puncture needle tip in real time, and determining the current angular deviation and radial distance deviation of the spatial position relative to the target needle insertion path; superimposing a visual guide line corresponding to the target needle insertion path and a tolerance color band corresponding to the dynamic spatial tolerance range on the real-scene image corresponding to the simulated arm; and visually marking the deviation state of the spatial position of the needle tip in the real-scene image based on the current angular deviation and radial distance deviation.

[0015] This application provides a method for error correction in intravenous puncture training based on multimodal perception. It includes: acquiring intravenous puncture training operation data and vascular response data of the trainee in the current training mode using a multimodal sensor installed in a simulated arm; wherein the simulated arm corresponds to different training modes, and each training mode is composed of an arm morphology and vascular pattern simulating patients of different ages and conditions; encoding the vascular response data in real time and sorting it according to time order to construct a vascular state encoding sequence; temporally associating the intravenous puncture training operation data and the vascular state encoding sequence to form a coupled data stream between the training operation and changes in vascular state; inputting the coupled data stream into a preset intravenous puncture deviation prediction model to predict the operation deviation of the intravenous puncture training operation in the current training mode, and generating compensation parameters based on the operation deviation prediction results; issuing a warning to the trainee and providing operation guidance data based on the operation deviation prediction results and compensation parameters, thereby achieving error correction in intravenous puncture training.

[0016] The at least one technical solution adopted in this application embodiment can achieve the following beneficial effects: This application embodiment simulates the vascular characteristics of patients of different ages and conditions using a simulated arm, and combines multimodal sensors to collect operational training data and vascular response data in real time, allowing trainees to practice under near-real clinical conditions, thus improving the relevance and effectiveness of training. Secondly, by analyzing the temporal correlation between operational data and vascular status, the model predicts deviation trends in real time, allowing for correction in the early stages of errors, thereby improving training efficiency. Furthermore, this application embodiment transforms subjective operational feel into objective, quantifiable data indicators, and provides intuitive and precise guidance through a combination of tactile and visual methods. This achieves an immediate and specific feedback mechanism, helping trainees accelerate skill development. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings: Figure 1 A schematic diagram of a multimodal perception-based training error correction system for venipuncture provided in an embodiment of this application; Figure 2 This is a flowchart of a method for training and correcting errors in venipuncture based on multimodal perception, provided in an embodiment of this application. Detailed Implementation

[0018] This application provides a multimodal perception-based training error correction system and method for venipuncture.

[0019] To enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this application.

[0020] Figure 1 A schematic diagram of a multimodal perception-based training error correction system for venipuncture is provided in an embodiment of this application, as shown below. Figure 1As shown, the multimodal perception-based intravenous puncture training error correction system includes a data acquisition unit, an encoding unit, an association unit, a prediction unit, and an error correction unit. The data acquisition unit acquires intravenous puncture training operation data and vascular response data transmitted by multimodal sensors installed within the simulated arm, corresponding to the trainee's current training mode. The simulated arm corresponds to different training modes, each simulating the arm shape and vascular patterns of patients of different ages and conditions. The encoding unit encodes the vascular response data in real time and sorts it chronologically to construct a vascular state encoding sequence. The association unit temporally associates the intravenous puncture training operation data with the vascular state encoding sequence, forming a coupled data stream between the training operation and changes in vascular state. The prediction unit inputs the coupled data stream into a pre-set intravenous puncture deviation prediction model to predict operational deviations in the intravenous puncture training operation under the current training mode and generates compensation parameters based on the prediction results. The error correction unit issues warnings to the trainee based on the operational deviation prediction results and compensation parameters, and provides operational guidance data to achieve intravenous puncture training error correction.

[0021] In one implementation of this application, an embodiment features a simulated arm capable of simulating various physiological and pathological characteristics. This simulated arm integrates replaceable or adjustable vascular modules. Each module is parametrically designed based on the physiological characteristics corresponding to the target population or specific diseases, simulating the corresponding vascular elasticity, wall thickness, inner diameter, and subcutaneous tissue properties. This allows the simulated arm to exhibit differentiated tactile resistance and vascular biomechanical feedback under different training modes.

[0022] Secondly, in key areas of the simulated arm, such as around the preset puncture point, a distributed array of pressure sensor matrices, optical positioning markers, and ultrasonic simulation probes are embedded. During the puncture procedure, the pressure sensor matrix collects real-time data on the multidimensional pressure distribution and gradient changes during finger pressure and needle insertion; the optical positioning system tracks the displacement of the skin surface markers to obtain the relative positional changes of the blood vessels and the spatial trajectory of the needle; and the ultrasonic simulation probe generates simulated echo signals reflecting the deformation of the blood vessel wall and the dynamic changes in the lumen, based on the currently activated training mode parameters. All sensor data are aligned through a unified time-domain synchronization module.

[0023] Furthermore, the data acquisition unit extracts training operation data and vascular response data based on the activated training mode. The training operation data includes at least a three-dimensional vector of the needle insertion angle, a time series of the needle insertion depth, a puncture velocity profile, and pressure distribution characteristics. The vascular response data includes at least the vascular displacement trajectory, vessel wall deformation rate, dynamic changes in lumen diameter, and vessel wall stress distribution.

[0024] In one implementation of this application, the encoding unit is specifically used for: determining the trainee's current operation stage based on intravenous puncture training operation data, and outputting the corresponding operation stage identifier; wherein the operation stage includes at least one of the following: vascular fixation stage, needle tip approach stage, puncture of the vascular wall stage, and catheter insertion stage. The vascular response data is converted into a feature vector composed of multi-dimensional feature values; wherein the multi-dimensional feature values ​​include at least one of the following: mean vascular pressure, gradient of vascular pressure change, variance of vascular displacement, and rate of vascular deformation. The value in the feature vector corresponding to the current moment is matched with the transition condition corresponding to the current operation stage identifier, and the vascular state corresponding to the current moment is determined based on the matching result; wherein the transition condition is a preset vascular state judgment rule based on the operation stage identifier and the feature vector. The vascular state and the feature vector are combined and encoded, and arranged in chronological order to generate a vascular state encoding sequence.

[0025] Specifically, the acquired intravenous puncture training data is analyzed by the encoding unit to determine the current operational stage. For example, when the pressure sensor acquires a continuous and stable pressure signal and the displacement sensor shows that the blood vessel position is relatively fixed, the blood vessel fixation stage is determined and an identifier is output. When the optical positioning system recognizes that the needle tip has entered the preset approach range and the pressure signal shows pre-contact characteristics, the needle tip approach stage is determined. When the pressure signal shows a sharp increase and is accompanied by ultrasound showing initial deformation of the blood vessel wall, the blood vessel wall insertion stage is determined. When the pressure signal tends to stabilize and the ultrasound signal shows that the lumen remains open, the catheter insertion stage is determined.

[0026] Secondly, feature processing is performed on the vascular response data to extract multidimensional feature vectors. Specifically, a sliding window analysis is performed on the acquired pressure signal to calculate the average pressure value within the window as the mean vascular pressure feature, and the first derivative of the pressure change is calculated as the pressure change gradient feature. The displacement variance of the vascular marker points in the two-dimensional plane is calculated from the optical positioning data as a quantitative index of vascular displacement stability. The rate of change of vascular wall deformation over time is analyzed from the ultrasound simulation signal to extract the vascular deformation rate feature. These feature values ​​are standardized and combined to form a multidimensional feature vector representing the vascular response at the current moment.

[0027] Furthermore, this application embodiment pre-establishes a conditional state transition rule base based on the operation stage. A set of vascular state judgment rules is predefined for each operation stage, and these rules are stored in the form of condition-conclusion associations. For example, in the vascular fixation stage, the vascular state judgment rule can be: if the vascular displacement variance is less than the threshold T1 and the average pressure is greater than the threshold P1, then the state is stable and fixed; if the displacement variance is greater than T1, then the state is at risk of slippage. In the vascular wall puncture stage, the vascular state judgment rule can be: if the deformation rate exceeds the threshold D1 and the pressure gradient is positive, then the state is that the vascular wall is under pressure; if the deformation rate suddenly drops accompanied by a sharp drop in pressure, then the state is that the puncture was successful.

[0028] Furthermore, at each sampling moment, the system's encoding unit acquires the current operation stage identifier, loads the corresponding state judgment rule set, matches the values ​​of each dimension in the real-time feature vector with the rule conditions, and selects the blood vessel state corresponding to the rule with the highest matching degree as the current judgment result. The system combines the judged blood vessel state identifier with the corresponding feature vector for encoding. In this embodiment, the format of "state code: feature vector" is adopted, and these encoding units are stored in a circular buffer in chronological order to form a continuous blood vessel state encoding sequence.

[0029] In one implementation of this application, the association unit in this embodiment is specifically used for: establishing a first storage area for caching intravenous puncture training operation data, and establishing a second storage area for caching vascular state codes. When writing the current vascular state code into the second storage area, a backtracking time window is determined based on the operation stage and timestamp associated with the current vascular state code. The first storage area is traversed, and operation data whose timestamps satisfy the backtracking time window are selected as a set of historical operation data related to the current vascular state code. Based on a preset weight rule base, the association strength weight between each operation data in the historical operation data set and the current vascular state code is determined; wherein, the preset weight rule base stores the corresponding weight values ​​using the operation stage, operation feature type, and vascular state code as a joint index. The current vascular state code, the historical operation data set, and the corresponding association strength weights are encapsulated into a coupled data unit with a timestamp and output in chronological order to form a coupled data stream.

[0030] Specifically, the association unit sets up two independent circular buffers in memory, serving as the first storage area and the second storage area, respectively. The first storage area is used to cache venipuncture training operation data in chronological order, with each operation data record containing a timestamp, operation type identifier, and multi-dimensional operation feature vector. The second storage area is used to cache vessel status codes in chronological order, with each status code record containing a timestamp, vessel status identifier, and feature vector. Both buffers employ the same timing management mechanism to ensure the consistency of data writing and reading timing. When a new vessel status code is generated and ready to be written to the second storage area, the association unit first parses the operation stage identifier associated with the status code. Based on a preset configuration mapping table, differentiated backtracking time window lengths are set for different operation stages. For example, the vessel fixation stage corresponds to a longer backtracking window to analyze the cumulative effect of fixation operations, while the vessel wall insertion stage corresponds to a shorter backtracking window to focus on the instantaneous operation impact. The association unit uses the timestamp of the current status code as a reference to backtrack the window duration corresponding to that stage to determine the time filtering range.

[0031] Furthermore, the association unit traverses all operation data records in the first storage area, filtering out records whose timestamps fall within the backtracking time window to form an initial historical operation dataset. To further improve association accuracy, data cleaning is performed based on the characteristics of the operation stage; for example, remote operation records from the vessel fixation stage are ignored during the tube insertion stage. The filtered historical operation data is arranged in ascending order of timestamps, forming a set of historical operation data with a temporal causal relationship to the current vessel status code. Secondly, the error correction system in this embodiment is equipped with a multi-dimensional weighted rule base. This rule base uses the operation stage, operation feature type, and vessel status code as a joint primary key, storing weight values ​​pre-annotated by expert knowledge or trained by machine learning. For each operation data record in the historical operation data set, the system extracts its operation stage and feature type, combines them with the current vessel status code, and performs a joint query in the weighted rule base to obtain the corresponding association strength weight. For operation data with composite features, the association unit calculates the comprehensive association strength using a feature-weighted summation method.

[0032] Finally, the association unit encapsulates the current vessel state code, the filtered historical operation data set, and their corresponding association strength weights to generate coupled data units with unified timestamps. Each coupled data unit adopts a standardized data structure, including header information, a state code block, a historical operation data block, and an association weight block. The system outputs these coupled data units sequentially in chronological order, forming a time-aligned, causally linked coupled data stream, providing a structured input sequence for subsequent bias prediction models.

[0033] In one implementation of this application, the multimodal perception-based venipuncture training error correction system further includes an input sequence construction unit. This unit is specifically used to: generate a mode condition vector based on the current training mode, and fuse the mode condition vector with each coupled data unit in the coupled data stream to generate a mode-conditional coupled data stream; wherein the mode condition vector is a category identifier reflecting the current training mode. Multiple coupled data units in the mode-conditional coupled data stream constitute an input sequence, which is then input into a pre-set venipuncture deviation prediction model.

[0034] Specifically, this embodiment of the application pre-defines a pattern embedding table, which stores different training patterns, such as edema in the elderly, high elasticity in children, and hypotension in shock, each corresponding to a fixed-dimensional vector representation. The input sequence construction unit uses the current pattern identifier as an index to retrieve the corresponding pattern condition vector from the embedding table. This vector is obtained through training on large-scale multi-pattern venous puncture data and can encode the differences in vascular characteristics under different physiological and pathological states. Next, the input sequence construction unit performs a feature-level fusion operation between the pattern condition vector and the coupled data stream. For each data unit in the coupled data stream, its internal operational feature vector and vascular state feature vector are concatenated with the pattern condition vector, respectively. Specifically, if the operational feature vector has a dimension of M, the vascular state feature vector has a dimension of N, and the pattern condition vector has a dimension of K, then the enhanced feature vector formed after concatenation has a dimension of M+N+K, thereby ensuring that pattern information is injected into the data representation at each time step.

[0035] Furthermore, the input sequence construction unit normalizes the fused enhanced feature vector. Since different features have different dimensions and numerical ranges, a standardization method based on the statistical features of the current training mode is adopted. The mean and standard deviation of the historical data of the mode are used to normalize each feature dimension, eliminating the differences in feature distribution between modes and improving the stability of model training.

[0036] Secondly, each augmented data unit, after fusion and normalization, is repackaged, retaining its original timestamp and associated weight information to form a new pattern-conditionalized data unit. These units are arranged chronologically, forming a complete pattern-conditionalized coupled data stream, where each data point carries explicit training pattern semantic information. Finally, a fixed-length continuous data unit is extracted from the pattern-conditionalized coupled data stream to form the model input sequence. A sliding window mechanism is used to extract L consecutive data units backward from the current time point, forming an input sequence matrix of dimension [L, M+N+K]. This sequence simultaneously contains temporal information, operator-vessel interaction information, and training pattern information, providing comprehensive input features for subsequent bias prediction.

[0037] In one implementation of this application, the prediction unit in the multimodal perception-based venipuncture training error correction system is specifically used to: determine the correlation between coupled data units at different times in the input sequence through the attention mechanism of a pre-set venipuncture deviation prediction model; wherein, the weight parameters used to calculate the correlation in the attention mechanism are dynamically modulated by the pattern condition vector. Based on the modulated correlation, the influence weight of each historical coupled data unit in the input sequence on the vascular state of the current coupled data unit is determined. Based on the influence weight, the operation feature vectors in the historical coupled data units are weighted and fused to generate the operation attribution vector corresponding to the current coupled data unit; wherein, the operation attribution vector is used to represent the contribution distribution of each historical training operation that leads to the current vascular state. The operation attribution vector, the current vascular state code, and the pattern condition vector are jointly input into the prediction head of the pre-set venipuncture deviation prediction model to determine the operation deviation prediction result through the prediction head.

[0038] Specifically, for each data unit in the sequence, a query vector, key vector, and value vector are generated through three independent fully connected layers. The weight parameters and bias terms of the fully connected layers are dynamically adjusted using a pattern conditional vector. Specifically, a conditional parameterization technique is used to map the pattern vector to weight modulation coefficients, enabling the same network layer to produce differentiated feature transformation effects under different training modes. Next, the system performs pattern-conditional attention weight calculation. For the data unit at the current time step, its query vector is multiplied by the key vectors of all data units at all time steps in the sequence to obtain an initial relevance score. During this process, a pattern conditional vector is introduced to scale the dot product result. This is achieved by adding a scaling factor based on the pattern vector during the calculation, ensuring that the attention focusing mechanism can adaptively adjust under different physiological modes.

[0039] Further, the prediction unit inputs the correlation score calculated in the previous step into the Softmax function for normalization, making the sum of the weights of all time steps equal to 1, forming a standardized attention weight distribution. Each weight value in this distribution represents the importance of the corresponding historical data unit to the prediction of the current vessel state; a higher weight indicates a greater impact of the historical operation on the current state. Then, the value vectors of all time step data units in the sequence are weighted and summed according to their corresponding attention weights to obtain the context vector of the current time step. Specifically, the system extracts the operation feature portion from the value vector and performs separate weighted fusion to form an operation attribution vector specifically representing the distribution of historical operation contributions. The value of each dimension in this vector reflects the contribution strength of the corresponding historical operation feature to the current vessel state. Finally, the operation attribution vector, the current vessel state code, and the pattern condition vector are concatenated and input into the multilayer perceptron of the prediction head. The network parameters of the prediction head are also modulated by the pattern condition vector, causing it to adopt different decision boundaries in different training modes. After forward propagation calculation, the prediction head outputs the probability distribution of various operational deviations occurring within the future time window. The system selects the deviation types with probabilities exceeding the threshold, along with their confidence levels, prediction occurrence times, and other information, and combines them to form a complete operational deviation prediction result.

[0040] In one implementation of this application, the multimodal perception-based venous puncture training error correction system further includes a parameter calculation unit, specifically used for: extracting the predicted deviation type, deviation value, and predicted occurrence time based on the operation deviation result; determining a reference compensation benchmark value for the deviation type based on the vascular physiological feature library corresponding to the current training mode; calculating a dynamic compensation coefficient based on a preset compensation coefficient mapping function and the deviation value; wherein the compensation coefficient mapping function is configured such that the larger the deviation value, the larger the dynamic compensation coefficient; calculating the time urgency coefficient of the compensation intervention based on the time difference between the predicted occurrence time and the current time; and generating compensation parameters through weighted calculation based on the reference compensation benchmark value, the dynamic compensation coefficient, and the time urgency coefficient; wherein the compensation parameters include at least one of the mechanical parameters for tactile feedback guidance and the display parameters for visual feedback guidance.

[0041] Specifically, the parameter calculation unit determines the deviation type code, such as angle deviation, depth deviation, and force deviation, by parsing the data structure output by the prediction module; it also determines the quantified deviation value and parses the specific time when the deviation prediction occurred. These parameters are stored in a temporary buffer as the basic input for compensation calculation. Next, based on the identifier of the current training mode and the extracted deviation type, a pre-set vascular physiological feature database is accessed. This database stores standard compensation reference values ​​for various deviations under different physiological states. For example, in the elderly arteriosclerosis mode, the benchmark compensation force for angle deviation is 0.8N, while in the children's vascular high elasticity mode, the corresponding benchmark value is 0.5N. Matching records are retrieved to obtain the reference compensation benchmark value.

[0042] Secondly, this application embodiment provides a predefined mapping function, with each deviation type corresponding to an independent function. Taking angular deviation as an example, a piecewise linear function is used: when the deviation value is within the range of 0-5 degrees, the compensation coefficient is 1.0; when it is between 5-10 degrees, the coefficient increases linearly to 1.5; and when it is above 10 degrees, the coefficient increases exponentially to 2.0. The system inputs the extracted deviation value into the corresponding function to calculate the dynamic compensation coefficient. Then, the time urgency coefficient is calculated, using the current system timestamp as a reference, to calculate the time difference Δt between the predicted occurrence time and the actual occurrence time. The parameter calculation unit uses an S-shaped function to calculate the urgency coefficient. ; in This is the sensitivity parameter (default value 0.5). This is the time offset (default value 2 seconds). When When the coefficient is relatively small, a coefficient close to 1.0 indicates a high degree of urgency; when... As the coefficient increases, it gradually decreases to 0.5, reflecting a decrease in urgency.

[0043] Finally, the parameter calculation unit performs a weighted calculation to generate the final compensation parameters, using a multiplicative weighted model: final parameter = baseline value × dynamic coefficient × urgency coefficient. For haptic feedback guidance, the system generates a set of mechanical parameters including force magnitude, direction, and duration; for visual feedback guidance, it generates a set of display parameters including guide line transparency and color intensity.

[0044] In one implementation of this application, the error correction unit in the multimodal perception-based intravenous puncture training error correction system is used to: generate a warning signal for localized tactile feedback within the simulated arm based on the deviation prediction result and compensation parameters; and generate visual operation guidance including the target path and real-time deviation indication on the real-world image of the simulated arm based on the compensation parameters. The operation guidance data is dynamically adjusted according to the trainee's adjustments to the training operation relative to the compensation parameters to achieve intravenous puncture training error correction.

[0045] Specifically, a localized tactile warning signal is generated based on the mechanical parameters in the compensation parameters. The system analyzes the force magnitude, direction, and duration data in the compensation parameters and converts them into drive commands to control the tactile actuators within the simulated arm. It also reads the values ​​in the compensation parameters regarding the transparency, color coding, and indicator size of the guide lines. Combined with the real-time acquired spatial position data of the simulated arm, a visual guide layer is overlaid on the real-world image using an augmented reality rendering engine.

[0046] Furthermore, after tactile and visual guidance is initiated, the error correction unit continuously collects real-time operational data from the trainee, including changes in needle angle, depth, force, and speed. By comparing the real-time operational data with the target values ​​in the compensation parameters, it calculates the timeliness index and accuracy index of operational adjustments. These two indices reflect the trainee's response speed and adjustment precision to the guidance signals, respectively. Then, the guidance parameters are dynamically adjusted based on the operational adjustments. When the timeliness index is below a threshold, the system gradually increases the intensity of tactile feedback and increases the flashing frequency of visual guidance; when the accuracy index continues to improve, the system reduces the guidance intensity according to a preset gradient, for example, reducing the tactile intensity by 10% every 10 seconds and increasing the transparency of visual guidance by 15%. If a reverse deviation occurs in the operation, the system immediately restores the initial guidance intensity and triggers an audible alert.

[0047] In one implementation of this application, the tactile feedback unit in the multimodal perception-based venous puncture training error correction system is specifically used to determine the type of operational deviation based on the operational deviation prediction result. When the deviation type is needle insertion angle deviation, the targeted tactile array within the simulated arm is driven based on the compensation direction, so that the tactile units located on the opposite side of the compensation direction in the targeted tactile array generate a directional vibration sequence simulating the lateral compression sensation of the blood vessel wall. When the deviation type is needle insertion depth deviation, the tactile unit located directly below the simulated blood vessel is driven to generate a preset pulse vibration sequence to simulate the bottoming sensation of the needle tip touching the vascular bed. The targeted tactile array consists of multiple independently controlled micro-vibration tactile units distributed in a grid pattern within the surrounding tissue layer of the simulated blood vessel.

[0048] Specifically, by analyzing the predicted operational deviation results, the system determines whether the current deviation is due to needle angle deviation or depth deviation. For angle deviation, the predicted left or right deviation direction and deviation angle value are further extracted; for depth deviation, the predicted depth difference (too deep or too shallow) is extracted. This identification process is based on a preset deviation type coding rule to ensure accurate differentiation of different types of operational errors. Secondly, for angle deviation, the system calculates the activation strategy for the targeted tactile array, determining the tactile unit group to be activated based on the compensation direction. If it is a right deviation requiring leftward compensation, the tactile unit located on the left side of the target blood vessel is activated. The vibration intensity is calculated based on the deviation angle value, using a linear mapping relationship: a deviation of 5 degrees corresponds to intensity level 3, a deviation of 10 degrees corresponds to intensity level 6, and the maximum is no more than level 10. The vibration sequence is designed as a pulse train with a frequency of 80Hz and a duration of 200ms, simulating the continuous lateral pressure of the blood vessel wall on the needle tip.

[0049] Furthermore, to address depth deviation issues, the system locates and activates deep tactile feedback units. Based on the 3D vascular model, the system determines the position directly below the target vessel and activates the corresponding deep tactile unit. The vibration mode is designed in three stages: the first stage is a weak cue (intensity 2, frequency 40Hz, duration 100ms); the second stage is an enhanced warning (intensity 5, frequency 60Hz, duration 150ms); and the third stage is a strong warning (intensity 8, frequency 80Hz, duration 200ms), simulating the process of the needle tip gradually approaching and eventually contacting the vascular bed. The system then generates tactile drive commands and sends them to the execution unit. The calculated vibration parameters are encoded into standardized control command frames and transmitted to the tactile array controller. The commands include time synchronization markers to ensure that multiple tactile units can accurately start and stop vibration according to a preset timing sequence. Finally, the system establishes a tactile feedback effect monitoring and adjustment mechanism. During tactile feedback execution, the system monitors the operator's response in real time. If the needle angle or depth is not adjusted as expected within a preset time, the system automatically increases the vibration intensity level and extends the vibration duration. Simultaneously, the parameter settings and operator response data for each haptic feedback are recorded to optimize the personalized adaptation of subsequent feedback strategies.

[0050] In one implementation of this application, the operation guidance unit in the multimodal perception-based venous puncture training and error correction system is specifically used to: call a preset three-dimensional vascular mechanical model based on the target vascular pattern corresponding to the compensation parameters; and obtain the deformation safety domain and the corresponding target needle insertion path of the vascular based on the collected vascular wall pressure data. Based on the deformation safety domain and the corresponding target needle insertion path, the dynamic spatial tolerance range of the needle insertion operation is determined, the spatial position of the simulated puncture needle tip is acquired in real time, and the current angular deviation and radial distance deviation of the spatial position relative to the target needle insertion path are determined. A visual guide line corresponding to the target needle insertion path and a tolerance color band corresponding to the dynamic spatial tolerance range are superimposed on the real-world image corresponding to the simulated arm. Based on the current angular deviation and radial distance deviation, the deviation status of the needle tip's spatial position is visually marked in the real-world image.

[0051] Specifically, based on the current training mode identifier and compensation parameters, the system retrieves the corresponding 3D vascular mechanical model from the model library. This model includes the geometric parameters, material properties, and boundary conditions of the blood vessel under specific physiological conditions. The system inputs real-time multi-point pressure data of the blood vessel wall as a dynamic load into the model, and calculates the stress distribution and deformation field of the blood vessel under the current operation in real time through finite element analysis. Regions with equivalent stress below the material stress threshold are marked as deformation safety regions, and the geometric centerline of this safety region is extracted as the theoretically optimal target needle insertion path. The dynamic tolerance range is determined based on the spatial geometric characteristics of the deformation safety region. With the target needle insertion path as the central axis, the minimum radial distance of the safety region at each sampling point on this axis is calculated, and these distance values ​​are used as the tolerance radius at that point. The system corrects the tolerance radius with a safety factor according to the needle diameter and operational stability requirements, generating a dynamic spatial tolerance channel that changes with the path position. This channel represents the safe space range within which needle tip deviation is allowed.

[0052] Furthermore, the spatial coordinates of the needle tip are tracked in real time using an optical positioning device. An infrared optical positioning system installed within the simulated arm acquires the three-dimensional coordinates of the needle tip marker, which are then mapped onto the blood vessel model coordinate system after coordinate transformation. The system calculates the shortest distance from the current position of the needle tip to the target insertion path as the radial distance deviation, and simultaneously calculates the angle between the current movement direction vector of the needle tip and the tangent direction vector of the target path as the angular deviation value. Then, in the real-time video stream, a semi-transparent green ribbon-like surface is used to render a visual guide line for the target insertion path, with the line width dynamically adjusted according to the blood vessel diameter. Around the guide line, the system renders a gradually transparent tolerance color band: the 0-30% tolerance radius area from the guide line is displayed in green, the 30-70% area gradually turns yellow, and the 70-100% area gradually turns red, forming an intuitive spatial safety indication. Finally, the system implements real-time deviation marking and status feedback for the needle tip position. A three-dimensional deviation indicator is overlaid at the current position of the needle tip: when the radial distance deviation is within the safe range, the indicator displays a green circular mark; when the deviation exceeds the safe range, the circular mark gradually turns red and displays the specific deviation value. Simultaneously, the angle deviation gauge is displayed on the side of the screen, showing the current angle deviation value in both pointer and numerical form. All visualization elements are updated at a frequency of 60Hz to ensure real-time synchronization with operation.

[0053] Figure 2 This is a flowchart illustrating a multimodal perception-based method for training and correcting errors in venipuncture, as provided in an embodiment of this application. Figure 2 As shown, the multimodal perception-based method for training and error correction in venipuncture includes the following steps: Step 101: By using a multimodal sensor installed in the simulated arm, acquire the venous puncture training operation data and vascular response data of the trainee in the current training mode; wherein, the simulated arm corresponds to different training modes, and the different training modes are composed of the simulated arm morphology and vascular patterns of patients of different ages and with different diseases. Step 102: Encode the vascular response data in real time and sort it according to time order to construct a vascular state coding sequence; Step 103: Temporally correlate the venipuncture training operation data with the vascular state coding sequence to form a coupled data stream between the training operation and the vascular state changes; Step 104: Input the coupled data stream into the preset venipuncture deviation prediction model to predict the operation deviation of the venipuncture training operation in the current training mode, and generate compensation parameters based on the operation deviation prediction results. Step 105: Based on the operation deviation prediction results and compensation parameters, issue an early warning to the trainee and provide operation guidance data to achieve error correction in intravenous puncture training.

[0054] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of apparatus, devices, and non-volatile computer storage media are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0055] The above descriptions are merely embodiments of this application and are not intended to limit the scope of this application. For those skilled in the art, various modifications and variations can be made to the embodiments of this application. These modifications or substitutions do not cause the essence of the corresponding technical solutions to depart from the spirit and scope of the technical solutions in the embodiments of this application.

Claims

1. A multimodal perception-based training and error correction system for venipuncture, characterized in that, The system includes a data acquisition unit, an encoding unit, an association unit, a prediction unit, and an error correction unit, wherein: The data acquisition unit is used to acquire the intravenous puncture training operation data and vascular response data of the trainee in the current training mode transmitted by the multimodal sensor set in the simulated arm; wherein, the simulated arm corresponds to different training modes, and the different training modes are composed of the arm shape and vascular pattern of patients of different ages and different diseases simulated by the simulated arm. The encoding unit is used to encode the vascular response data in real time and sort it according to time order to construct a vascular state encoding sequence. The association unit is used to temporally associate the vein puncture training operation data with the vascular state encoding sequence to form a coupled data stream between the training operation and the vascular state change. The prediction unit is used to input the coupled data stream into a preset venipuncture deviation prediction model to predict the operation deviation of the venipuncture training operation in the current training mode, and generate compensation parameters based on the operation deviation prediction results. The error correction unit is used to issue an early warning to the trainee and provide operation guidance data based on the operation deviation prediction result and the compensation parameter, so as to realize error correction in intravenous puncture training.

2. The venipuncture training error correction system based on multimodal perception according to claim 1, characterized in that, The encoding unit is specifically used for: Based on the intravenous puncture training operation data, the trainee's current operation stage is determined, and the corresponding operation stage identifier is output; wherein, the operation stage includes at least one of the following: vascular fixation stage, needle tip approach stage, puncture of the vascular wall stage, and catheter insertion stage. The vascular response data is converted into a feature vector composed of multi-dimensional feature values; wherein, the multi-dimensional feature values ​​include at least one of the following: mean vascular pressure, gradient of vascular pressure change, variance of vascular displacement, and rate of vascular deformation. The value in the feature vector corresponding to the current moment is matched with the transition condition corresponding to the current operation stage identifier. Based on the matching result, the blood vessel state corresponding to the current moment is determined. The transition condition is a preset blood vessel state judgment rule based on the operation stage identifier and the feature vector. The blood vessel state is combined with the feature vector and encoded, and then arranged in chronological order to generate the blood vessel state encoding sequence.

3. The venipuncture training error correction system based on multimodal perception according to claim 1, characterized in that, The associated unit is specifically used for: A first storage area is established for caching the training operation data of the venipuncture, and a second storage area is established for caching the vascular status code; When writing the current blood vessel status code into the second storage area, a backtracking time window is determined based on the operation stage and timestamp associated with the current blood vessel status code. Traverse the first storage area and filter out the operation data whose timestamps satisfy the backtracking time window, as a set of historical operation data related to the current blood vessel status code; Based on a pre-set weighted rule base, the association strength weight between each operation data in the historical operation data set and the current blood vessel state code is determined; wherein, the pre-set weighted rule base stores the corresponding weight values ​​using operation stage, operation feature type and blood vessel state code as a joint index. The current blood vessel state code, the historical operation data set, and the corresponding association strength weights are encapsulated into a timestamped coupled data unit and output in chronological order to form the coupled data stream.

4. The venipuncture training error correction system based on multimodal perception according to claim 1, characterized in that, The system also includes an input sequence construction unit, specifically used for: Based on the current training mode, a mode condition vector is generated, and the mode condition vector is fused with each coupled data unit in the coupled data stream to generate a mode-conditional coupled data stream; wherein, the mode condition vector is a category identifier that reflects the current training mode; Multiple coupled data units in the coupled data stream based on the pattern conditionalization constitute an input sequence, which is then input into the preset venipuncture deviation prediction model.

5. A multimodal perception-based venipuncture training error correction system according to claim 4, characterized in that, The prediction unit is specifically used for: The correlation between coupled data units at different times in the input sequence is determined by the attention mechanism of the preset venipuncture deviation prediction model; wherein, the weight parameters used to calculate the correlation in the attention mechanism are dynamically modulated by the pattern condition vector. Based on the modulated correlation, the influence weight of each historical coupled data unit in the input sequence on the vascular state of the current coupled data unit is determined; Based on the influence weights, the operation feature vectors in the historical coupled data units are weighted and fused to generate the operation attribution vector corresponding to the current coupled data unit; wherein, the operation attribution vector is used to represent the contribution distribution of each historical training operation that leads to the current vascular state; The operation attribution vector, the current vascular state code, and the pattern condition vector are input together into the prediction head of the preset venous puncture deviation prediction model to determine the operation deviation prediction result through the prediction head.

6. The venipuncture training error correction system based on multimodal perception according to claim 1, characterized in that, The system also includes a parameter calculation unit, specifically used for: Based on the operational deviation results, the predicted deviation type, deviation value, and predicted occurrence time are extracted; Based on the vascular physiological feature library corresponding to the current training mode, a reference compensation benchmark value is determined for the deviation type. Based on the preset compensation coefficient mapping function and the deviation value, a dynamic compensation coefficient is calculated; wherein, the compensation coefficient mapping function is configured such that the larger the deviation value, the larger the dynamic compensation coefficient. Calculate the time urgency coefficient for compensation intervention based on the time difference between the predicted occurrence time and the current time; Based on the reference compensation benchmark value, the dynamic compensation coefficient, and the time urgency coefficient, the compensation parameters are generated through weighted calculation; wherein, the compensation parameters include at least one of the mechanical parameters for haptic feedback guidance and the display parameters for visual feedback guidance.

7. The venipuncture training error correction system based on multimodal perception according to claim 1, characterized in that, The error correction unit is specifically used for: Based on the deviation prediction results and the compensation parameters, a warning signal with localized tactile feedback is generated within the simulated arm. Furthermore, based on the compensation parameters, a visual operation guide containing the target path and real-time deviation indication is generated on the real-world image of the simulated arm. Based on the trainee's adjustments to the training operations relative to the compensation parameters, the operation guidance data is dynamically adjusted to achieve error correction in intravenous puncture training.

8. The venipuncture training error correction system based on multimodal perception according to claim 7, characterized in that, The system also includes a haptic feedback unit, specifically used for: Based on the predicted operational deviation results, the type of operational deviation is determined; When the deviation type is needle insertion angle deviation, the targeted tactile array in the simulated arm is driven based on the compensation direction, so that the tactile unit in the targeted tactile array located on the opposite side of the compensation direction generates a directional vibration sequence that simulates the lateral compression sensation of the blood vessel wall. When the deviation type is needle insertion depth deviation, the tactile unit located directly below the simulated blood vessel is driven to generate a preset pulse vibration sequence to simulate the sensation of the needle tip touching the bottom of the vascular bed. The targeted tactile array consists of multiple independently controlled micro-vibration tactile units distributed in a grid pattern in the surrounding tissue layer of the simulated blood vessel.

9. A multimodal perception-based training and error correction system for venipuncture according to claim 7, characterized in that, The system also includes an operation guidance unit, specifically used for: Based on the target blood vessel pattern corresponding to the compensation parameters, a preset three-dimensional mechanical model of the blood vessel is invoked, and based on the collected blood vessel wall pressure data, the deformation safety domain and the corresponding target needle insertion path of the blood vessel are obtained. Based on the deformation safety domain and the corresponding target needle insertion path, determine the dynamic spatial tolerance range of the needle insertion operation; The spatial position of the simulated puncture needle tip is acquired in real time, and the current angular deviation and radial distance deviation of the spatial position relative to the target needle insertion path are determined. On the real-world image corresponding to the simulated arm, a visual guide line corresponding to the target needle insertion path is superimposed, as well as a tolerance color band corresponding to the dynamic space tolerance range is superimposed; Based on the current angular deviation and radial distance deviation, the spatial position of the needle tip is visually marked in the real-world image to indicate the deviation status.

10. A method for error correction during venipuncture training based on multimodal perception, characterized in that, The method includes: By using multimodal sensors installed in the simulated arm, the trainee's intravenous puncture training operation data and vascular response data in the current training mode are acquired; wherein, the simulated arm corresponds to different training modes, and the different training modes are composed of the simulated arm morphology and vascular patterns of patients of different ages and with different diseases. The vascular response data is encoded in real time and sorted according to time order to construct a vascular state encoding sequence; The venipuncture training operation data is temporally correlated with the vascular state encoding sequence to form a coupled data stream between the training operation and changes in vascular state. The coupled data stream is input into a preset venipuncture deviation prediction model to predict the operation deviation of venipuncture training operations under the current training mode, and to generate compensation parameters based on the operation deviation prediction results. Based on the predicted operational deviation and the compensation parameters, an early warning is issued to the trainee, and operational guidance data is provided to achieve error correction in intravenous puncture training.

Images