Teaching system for intravenous drug allocation based on deep learning

Abstract

The invention relates to the technical field of intravenous drug configuration teaching, in particular to a teaching system for intravenous drug configuration based on deep learning, which constructs a three-dimensional scene and collects operation data of a user in a virtual environment by acquiring live-action video data of intravenous drug configuration. Through semantic segmentation, an operation track is tracked in combination with a long and short-term memory network, a variable path matrix is generated, an operation score is generated by using a convolutional neural network, wrong operation is identified in time, and real-time feedback is provided. According to the method, training can be carried out only through a computer or VR equipment without depending on a large number of sites and equipment, students are helped to carry out timely correction in each step of operation through model evaluation and real-time feedback, and the operation accuracy and the learning effect are improved. Difficulty gradient adjustment is carried out in combination with scores, it is ensured that the training difficulty is matched with the skill level of students, personalized teaching is achieved, and the teaching effect is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of intravenous drug preparation teaching, and in particular to a teaching system for intravenous drug preparation based on deep learning. Background Art

[0002] Intravenous drug preparation is an extremely important operation in the medical industry, especially in terms of drug mixing, dosage adjustment and label management. Any operational errors will directly affect the patient's treatment effect and may even lead to serious medical accidents. Traditional intravenous drug preparation operation training mainly relies on human training and manual operation, and there are the following problems: drug preparation requires precise operation steps and strict specifications, which places high demands on learners' operation skills. Manual training is difficult to achieve fine operation feedback, and the teaching process relies on the guidance of senior medical personnel, and requires a lot of space and equipment, which is difficult to apply on a large scale. At the same time, each learner has different operation skill levels. Traditional training often adopts a unified standard and has a weak response to personalized learning needs. High-level learners cannot obtain more challenging training content, while learners with weaker skills find it difficult to adapt to more difficult operation tasks. Summary of the invention

[0003] To solve the above problems, the present invention provides a teaching system for intravenous drug configuration based on deep learning, which realizes high-precision intravenous drug configuration operation training through virtual simulation and deep learning algorithm, and provides real-time feedback and personalized training support.

[0004] To achieve the above object, the technical solution adopted by the present invention is:

[0005] A teaching system for intravenous drug dispensing based on deep learning, comprising a data acquisition module, a three-dimensional space simulation module, a variable capture module, an operation scoring module and a personalized practice module connected in sequence, wherein the personalized practice module is connected to the three-dimensional space simulation module;

[0006] The data acquisition module is used to obtain real-life video data of intravenous drug dispensing and perform data annotation;

[0007] The three-dimensional space simulation module is used to construct a three-dimensional scene of intravenous drug configuration based on the real-scene video data of intravenous drug configuration, and initialize the type and quantity of intravenous drugs in the three-dimensional scene; the three-dimensional space virtual module is also used to receive user-side operation instructions to perform intravenous drug configuration operations in the three-dimensional scene;

[0008] The variable capture module is used to perform the following steps:

[0009] S1, obtaining video data of a three-dimensional scene for intravenous drug dispensing operation;

[0010] S2, performing frame decomposition on the operation video data of the three-dimensional scene of intravenous drug configuration to obtain a frame image set, performing semantic segmentation training on the variable objects in the frame image set, and obtaining a variable capture model;

[0011] S3, a frame image set of operation video data of a three-dimensional scene of intravenous drug configuration, performing variable tracking through a variable capture model and generating a variable path matrix;

[0012] The operation scoring module is used to take the variable path matrix as input and the corresponding scoring data as output, to construct a scoring model based on a convolutional neural network, and is also used to generate a scoring result for the real-time input real-scene video data through the scoring model and extract the deduction items to provide wrong operation prompts;

[0013] The personalized training module is used to perform gradient adjustments on the types and quantities of intravenous drugs in the three-dimensional scene according to the scoring results.

[0014] Furthermore, the three-dimensional scene of intravenous drug configuration includes prescription review, label printing, label placement, drug placement verification, mixing and preparation, finished product verification, finished product packaging and zoning placement; the variable objects include intravenous drugs, drug labels, tools and operating tables.

[0015] Furthermore, the construction of a three-dimensional scene of intravenous drug configuration based on real-life video data of intravenous drug configuration includes the following steps:

[0016] Perform frame segmentation on real-scene video data, extract key feature points in each frame to generate a feature point matrix, including feature points of intravenous drugs, drug labels, tools, and operating tables;

[0017] The feature points of different perspectives are associated through a multi-perspective image matching algorithm to generate a corresponding depth information matrix;

[0018] Based on the depth information matrix, the variable objects in the scene are reconstructed in three dimensions to generate a three-dimensional scene of intravenous drug configuration.

[0019] Further, the S2 comprises the following steps:

[0020] Output the frame image set to the backbone network of DeepLab for feature extraction to obtain a feature map;

[0021] The feature map is integrated with multi-scale context information through the atrous spatial pyramid pooling module.

[0022] The feature map processed by the dilated spatial pyramid pooling module is restored to its original size through deconvolution operation to generate a variable segmentation mask;

[0023] The generated variable segmentation mask is cross-entropy-loss calculated with the labeled data, and the network weights are adjusted through the Adam optimizer to obtain the variable capture model.

[0024] Furthermore, S3 comprises the following steps:

[0025] Based on the frame image set of the operation video data of the three-dimensional scene of intravenous drug configuration, the corresponding variable segmentation mask is generated through the variable capture model, and a unique identifier is generated for each variable object therein, and its spatial position and posture are initialized;

[0026] Based on the long short-term memory network, the variable objects in different frame images are continuously tracked, and their motion trajectories between different frames are output to generate a preliminary path matrix.

[0027] The preliminary path matrix is ​​de-noised using the Kalman filter algorithm to generate a variable path matrix.

[0028] Furthermore, the formula of the long short-term memory network is as follows:

[0029]

[0030] Among them, h t It is the hidden state output at the current moment, which is used to reflect the status of the current operation step; o t is the output of the output gate, which is used to control the information passed to the next moment; f t is the output of the forget gate, which is used to determine the cell state C at the previous moment t-1 Information that needs to be retained; C t-1 It is the information of the previous frame image; is the information of the candidate current frame image; α is the weight parameter, which is used to dynamically adjust the weights of different variable objects; Δ t It is the time difference change of the variable object during the operation.

[0031] Furthermore, the scoring data includes an accuracy score, a standardization score and an efficiency score.

[0032] Furthermore, the extraction of deduction items for incorrect operation prompts includes the following steps:

[0033] The convolutional neural network is used to analyze the spatiotemporal characteristics of the variable path matrix in real time, monitor the trajectory deviation of the variable object during the operation process, and identify the non-compliant operation steps by calculating the Euclidean distance and dynamic time warping value between the operation path and the standard path;

[0034] Detected operational deviations are marked as penalty items and visualized in real time via visual or text.

[0035] Furthermore, the step of gradient-adjusting the type and quantity of intravenous drugs in the three-dimensional scene according to the scoring result includes:

[0036] According to the user's scoring result, it is compared with the preset scoring threshold. If the scoring result is greater than or equal to the threshold, the type and quantity of intravenous drug configuration in the three-dimensional scene are increased through the preset gradient; if the scoring result is lower than the scoring threshold, the type and quantity of intravenous drug configuration in the three-dimensional scene are reduced through the preset gradient.

[0037] The beneficial effects of the present invention are as follows: the present invention constructs a three-dimensional scene of intravenous drug configuration by acquiring real-life video data of intravenous drug configuration, collects user operation data in the three-dimensional virtual scene for semantic segmentation, tracks the operation trajectory of the variable object through a long short-term memory network (LSTM), obtains a variable path matrix, and generates an operation score through a convolutional neural network, timely identifies wrong operations and provides real-time feedback. It does not require a large amount of venues and equipment to reduce the cost of venues and equipment, and only needs to access the three-dimensional scene through a computer or VR device to practice intravenous drug configuration. Compared with the traditional manual training method that relies on post-evaluation, this solution can provide instant feedback on errors through semantic segmentation and model evaluation during the operation process, helping trainees to correct them in time in each operation step, greatly improving the accuracy of the operation and the learning effect. Then, by combining the operation scoring module and the personalized practice module, intelligent adjustment based on the user's operation performance is realized. Specifically, when the system detects that the user's operation score is high, it automatically increases the type and quantity of drugs to increase the difficulty of training, thereby providing a more challenging task; conversely, if the score is low, the system dynamically reduces the complexity of the task, reduces the types of drugs and operation steps, to ensure that the user can gradually master the basic operation skills. This gradient adjustment mechanism ensures that the training difficulty of each student matches his or her skill level, truly realizes personalized teaching path planning, and significantly improves the pertinence and effectiveness of training. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a structural schematic diagram of a teaching system for intravenous drug configuration based on deep learning in the present invention.

[0039] Figure 2 It is a flow chart of the execution steps of the variable capture module in the present invention. DETAILED DESCRIPTION

[0040] See also Figure 1-2 As shown, the present invention is about a teaching system for intravenous drug configuration based on deep learning, comprising a data acquisition module, a three-dimensional space simulation module, a variable capture module, an operation scoring module and a personalized practice module connected in sequence, wherein the personalized practice module is connected to the three-dimensional space simulation module;

[0041] The data acquisition module is used to obtain real-life video data of intravenous drug dispensing and perform data annotation;

[0042] The three-dimensional space simulation module is used to construct a three-dimensional scene of intravenous drug configuration based on the real-scene video data of intravenous drug configuration, and initialize the type and quantity of intravenous drugs in the three-dimensional scene; the three-dimensional space virtual module is also used to receive user-side operation instructions to perform intravenous drug configuration operations in the three-dimensional scene;

[0043] The variable capture module is used to perform the following steps:

[0044] S1, obtaining video data of a three-dimensional scene for intravenous drug dispensing operation;

[0045] S2, performing frame decomposition on the operation video data of the three-dimensional scene of intravenous drug configuration to obtain a frame image set, performing semantic segmentation training on the variable objects in the frame image set, and obtaining a variable capture model;

[0046] S3, a frame image set of operation video data of a three-dimensional scene of intravenous drug configuration, performing variable tracking through a variable capture model and generating a variable path matrix;

[0047] The operation scoring module is used to take the variable path matrix as input and the corresponding scoring data as output, to construct a scoring model based on a convolutional neural network, and is also used to generate a scoring result for the real-time input real-scene video data through the scoring model and extract the deduction items to provide wrong operation prompts;

[0048] The personalized training module is used to perform gradient adjustments on the types and quantities of intravenous drugs in the three-dimensional scene according to the scoring results.

[0049] In some embodiments, the data acquisition module is used to obtain real-life video data of intravenous drug configuration and perform data annotation. Data acquisition mainly includes all key links in the user's operation, such as prescription review, label printing, label placement, drug placement verification, mixed preparation, finished product verification, finished product packaging and partition placement. The collected data is converted into a frame image set, and the system uses an automated annotation tool to annotate the key variable objects (drugs, labels, tools, etc.) in the frame image to ensure that each frame image has accurate variable position, type and posture information. These annotation information provides a high-quality data source for subsequent deep learning model training. Next, the three-dimensional space simulation module is responsible for constructing a virtual three-dimensional scene of intravenous drug configuration based on the collected real-life data. In order to improve the simulation effect of training, the module reconstructs the depth information of the collected frame images through a multi-view image matching algorithm. Specifically, the system generates a depth information matrix for each object by matching feature points of images from different perspectives. Based on this, the system can present key contents such as drug types, tool placement, label positions, etc. in the process of intravenous drug configuration in a three-dimensional virtual scene to ensure a high degree of consistency between the simulation scene and the real scene. The variable capture module is the core link of this system, which is used to accurately and dynamically track the user's operation behavior in the three-dimensional scene. First, the system extracts the video data of the user's operation from the three-dimensional scene and splits it into a set of frame images. On the basis of semantic segmentation, the DeepLab network is used to calibrate the variable objects in the frame image pixel by pixel. The convolution layer of the network can extract low-level features in the image, while the atrous spatial pyramid pooling (ASPP) module accurately segments the variable objects (such as drugs and tools) in complex scenes through multi-scale context information extraction and generates feature maps. Through the deconvolution operation, the feature map is restored to the same size as the original frame image to obtain the variable segmentation mask. On this basis, the system calculates the segmentation accuracy through the cross entropy loss function and adjusts the parameters through the Adam optimizer to optimize the weights of the semantic segmentation model. Once the variable capture model is trained, the system enters the real-time tracking link of the user's operation. The position and state information of each variable object during the user's operation will be recorded and analyzed in time series through the long short-term memory network (LSTM). The LSTM model is particularly suitable for processing time series data. Its memory unit can save the historical state of the operation process and dynamically update the motion trajectory of the variable object in combination with the characteristics of the current frame image. The system generates a variable path matrix to record the spatial position changes of the variable object at each moment and the temporal characteristics of the operation behavior. In order to ensure the accuracy of the path matrix, the system uses the Kalman filter algorithm to filter out the noise in the variable path and generate a smooth and accurate motion trajectory. Next, the operation scoring module performs operation evaluation based on the generated variable path matrix.The scoring model uses a convolutional neural network (CNN) to extract spatial and temporal features from the path matrix. The layer-by-layer convolution operation can capture key details in the operation, such as the mixing order of drugs and the use of tools. The scoring module measures the accuracy and standardization of the operation by calculating the Euclidean distance and dynamic time warping (DTW) value between the operation path and the standard path. The system can identify different types of erroneous operations (such as incorrect label pasting position, inaccurate drug dosage, etc.) and quantify each erroneous operation. At the same time, the system extracts abnormal behaviors in the operation through the feature mapping layer, generates deduction items and feeds them back to the user. This scoring process not only ensures the objectivity of the operation evaluation, but also generates specific improvement suggestions for different operation links. Finally, the personalized practice module automatically adjusts the difficulty of the operation task in the three-dimensional scene by analyzing the scoring results. According to the user's performance, the system will dynamically adjust the type and quantity of drugs to optimize the complexity of the training task. Specifically, when the score reaches the preset threshold, the system increases the type and quantity of drugs through a gradient algorithm to increase the difficulty of training; when the score is low, the system reduces the types of drugs and the complexity of the task to help users gradually master basic operating skills. This gradient adjustment mechanism combines the strategy optimization algorithm of reinforcement learning, which can adjust the training content in real time according to the user's operating performance, ensuring that the training tasks match the user's skill level, thereby achieving highly personalized teaching results.

[0050] Furthermore, the three-dimensional scene of intravenous drug configuration includes prescription review, label printing, label placement, drug placement verification, mixing and preparation, finished product verification, finished product packaging and zoning placement; the variable objects include intravenous drugs, drug labels, tools and operating tables.

[0051] It should be noted that in order to simulate the process of drug review, users need to check the type and dosage of prescribed drugs in a virtual environment. The system records the user's verification operation in the review process through the variable capture module, and ensures that the type and quantity of drugs match the actual prescription. Printing labels is an important part of the drug configuration process. Users generate corresponding drug labels by operating a virtual printer in a three-dimensional scene. The system tracks variable objects in the printing process in real time, including drug labels and printing tools. After printing, users need to correctly paste the labels on the drug containers. Operation details such as position deviation and inaccurate pasting angle during labeling will be detected by the system. In the drug placement process, the system monitors variable objects such as drug displacement and placement order to ensure that users place drugs in the correct order and position. This process involves precise tracking of drug quantity, position and operation sequence. The variable capture module analyzes the user's operation trajectory through the LSTM network and models the drug placement behavior in time series. When the user's drug placement order does not conform to the preset process or there is an error in the drug quantity, the system will identify the error and reflect it in the score. The mixed preparation process is more complex and involves the mixing and dosage control of multiple drugs. The three-dimensional scene realizes the process through the simulation of virtual operation tables, mixing tools and drug containers. The variable capture module dynamically tracks the drug containers and mixing tools in each mixing step to ensure that the user operates according to the specified mixing ratio. The system calculates the time series data and dynamic changes of the drug amount during the drug mixing process to determine whether the operation meets the standards and generate a score. After completing the mixing and preparation, the user needs to check and package the finished drug.

[0052] Furthermore, the construction of a three-dimensional scene of intravenous drug disposition based on real-scene video data of intravenous drug disposition includes the following steps:

[0053] Perform frame segmentation on real-scene video data, extract key feature points in each frame to generate a feature point matrix, including feature points of intravenous drugs, drug labels, tools, and operating tables;

[0054] The feature points of different perspectives are associated through a multi-perspective image matching algorithm to generate a corresponding depth information matrix;

[0055] Based on the depth information matrix, the variable objects in the scene are reconstructed in three dimensions to generate a three-dimensional scene of intravenous drug configuration.

[0056] In some embodiments, first, the system performs frame segmentation on the real-life video data of intravenous drug configuration. This process decomposes the video data into a continuous sequence of frame images, each frame representing a time point in the operation process. The purpose of this frame segmentation is to ensure that the dynamic operation in the video data can be analyzed frame by frame, providing basic data for subsequent feature extraction and three-dimensional reconstruction. The information of objects such as intravenous drugs, drug labels, tools and operating tables contained in each frame will be parsed and processed in the next step. In each frame, the system extracts key feature points through a convolutional neural network (CNN). These feature points cover the position information and boundaries of variable objects such as intravenous drugs, drug labels, tools and operating tables. The feature extraction process of the convolutional neural network includes several layers of convolution operations, which gradually identify important objects in the image by extracting low-level, intermediate and high-level features layer by layer. These feature points generate a feature point matrix by marking the key parts in each frame image. The feature point matrix not only records the two-dimensional spatial position of each variable object in the image, but also captures geometric information such as the size, shape and posture change of the object. Next, the system associates frame images from different angles through a multi-view image matching algorithm. For image data acquired from multiple angles, matching between feature points is the core step of 3D reconstruction. Multi-view image matching algorithms usually match based on the similarity of feature points, and achieve association between feature points by finding corresponding feature points of the same object under different viewpoints. Specifically, the system extracts feature descriptors in each frame of the image and calculates the similarity between feature points through algorithms such as SIFT (Scale Invariant Feature Transform) or ORB (Oriented Fast Binary Feature). The matching algorithm identifies feature point matching pairs under different viewpoints through the nearest neighbor search method, and generates a depth information matrix based on these matching pairs. The generated depth information matrix contains the depth information of all variable objects in each frame, which is calculated by triangulation or stereo vision algorithm. The stereo vision algorithm infers the depth of each object relative to the camera through the parallax between multiple frames (i.e., the position offset of the object under different viewpoints). This depth information not only provides the Z-axis position information of the object for 3D reconstruction, but also reflects the relative height and distance of the object in the scene. Based on the depth information matrix, the system performs 3D reconstruction of the variable objects in the scene. The 3D reconstruction process expands the 2D feature points of each frame into 3D space and combines the depth information to build a 3D model of the object. This process is usually based on Delaunay triangulation or Poisson surface reconstruction algorithm to generate a 3D mesh. Delaunay triangulation is used to connect feature points into a 3D polygonal mesh to ensure that the generated mesh conforms to the physical structure and maintains geometric consistency. Poisson surface reconstruction converts known point cloud data into a continuous 3D surface to describe the boundary and volume information of the object.Through these algorithms, the system can accurately reproduce the structure and shape of drugs, tools and operating tables in virtual three-dimensional space.

[0057] Further, the S2 comprises the following steps:

[0058] Output the frame image set to the backbone network of DeepLab for feature extraction to obtain a feature map;

[0059] The feature map is integrated with multi-scale context information through the atrous spatial pyramid pooling module.

[0060] The feature map processed by the dilated spatial pyramid pooling module is restored to its original size through deconvolution operation to generate a variable segmentation mask;

[0061] The generated variable segmentation mask is cross-entropy-loss calculated with the labeled data, and the network weights are adjusted through the Adam optimizer to obtain the variable capture model.

[0062] In some embodiments, first, the frame image set is sent to the backbone network of DeepLab for feature extraction after preprocessing. As a deep convolutional neural network (DCNN), the DeepLab network has high accuracy and efficiency in image semantic segmentation. In the feature extraction stage, the convolution layer of the backbone network is used to process the input frame image set layer by layer, and a feature map is generated by extracting low-level to high-level image features such as edges, textures, and shapes. The backbone network usually uses pre-trained models, such as ResNet or Xception, which can be trained on large-scale data sets to enable it to have strong feature representation capabilities. The convolution operation captures important information in the image through the local receptive field and gradually constructs a high-dimensional feature representation of the image. Then, the feature map extracted by the backbone network will be multi-scale context information fused through the dilated spatial pyramid pooling (ASPP) module. The core function of the ASPP module is to use dilated convolutions of different scales to perform multi-level processing on the feature map to obtain context information under different receptive fields. Atrous convolution expands the receptive field by inserting holes between convolution kernels without increasing computational complexity, enabling the network to capture a wider range of contextual dependencies without losing detailed information. For example, in the intravenous drug configuration scenario, atrous convolution can simultaneously capture the subtle edges of the drug label and the spatial layout of the entire operating table. Through the multi-scale processing of the ASPP module, the network can not only capture local details, but also integrate global information, thereby improving segmentation accuracy. After feature fusion is completed, the system restores the feature map to the size of the original image through a deconvolution operation. The main purpose of the deconvolution operation is to upsample the downsampled feature map to restore the spatial resolution of the original image. This operation makes the size of the feature map consistent with the input image through interpolation or reverse convolution, ensuring that the generated variable segmentation mask has the same size as the original frame image. The output of the deconvolution layer is a high-resolution feature map containing semantic information, which can accurately locate the boundaries and positions of variable objects during the segmentation process. The generated variable segmentation mask represents the category to which each pixel in the image belongs, including objects such as intravenous drugs, drug labels, tools, and operating tables. The predicted category of each pixel is evaluated by calculating the cross entropy loss function. The cross-entropy loss function measures the segmentation performance of the model by comparing the difference between the predicted segmentation mask and the annotated data. Specifically, the cross-entropy loss measures the degree of deviation between the probability distribution of the predicted category and the true category distribution. In the intravenous drug configuration scenario, accurate variable segmentation is crucial for subsequent operation evaluation, so the calculation result of the loss function is used to guide further optimization of the model. In order to reduce the training error of the model, the system uses the Adam optimizer to adjust the weight parameters of the network.The Adam optimizer is a commonly used adaptive gradient descent algorithm. Its advantage is that it can dynamically adjust the learning rate based on the estimation of the first-order and second-order moments, so that the network can converge at a faster speed at different training stages. Specifically, the Adam optimizer calculates the learning rate of each parameter through historical gradient information and adjusts the network weights according to the current loss gradient, thereby effectively reducing the cross entropy loss. During the optimization process, the Adam optimizer can avoid the oscillation problem in traditional gradient descent and ensure that the model quickly converges to the global optimum.

[0063] Furthermore, S3 comprises the following steps:

[0064] Based on the frame image set of the operation video data of the three-dimensional scene of intravenous drug configuration, the corresponding variable segmentation mask is generated through the variable capture model, and a unique identifier is generated for each variable object therein, and its spatial position and posture are initialized;

[0065] Based on the long short-term memory network, the variable objects in different frame images are continuously tracked, and their motion trajectories between different frames are output to generate a preliminary path matrix.

[0066] The preliminary path matrix is ​​de-noised using the Kalman filter algorithm to generate a variable path matrix.

[0067] In some embodiments, after generating the segmentation mask, the system assigns a unique identifier to each variable object. The generation of the identifier is based on the feature vector of each object, which is composed of multi-dimensional features extracted by CNN, including the geometric shape, texture, color and position information of the object. In this way, the system can ensure that each variable object remains unique during subsequent processing, and will not cause confusion of objects due to the similarity between frames. The assignment of unique identifiers is an important basis for subsequent time series tracking, ensuring that each object can be correctly identified and distinguished between multiple frame images. Next, the system initializes the spatial position and posture of each variable object. Spatial position initialization is achieved by extracting the coordinate information of the object in the three-dimensional scene, including the coordinate values ​​on the X, Y and Z axes. Posture initialization is based on the angle and direction of the object in the scene. For example, the placement angle of the tool, the inclination of the drug container, etc. These initial parameters provide a reference basis for the subsequent motion trajectory calculation, ensuring that the position information of each object at different time points is consistent and coherent. Based on these initialization information, the system continuously tracks the variable objects in different frame images through a long short-term memory network (LSTM). LSTM networks are good at processing time-dependent sequence data. In the operation scenario of intravenous drug configuration, the motion trajectory of each object has strong temporal characteristics. For example, during the operation, the movement of drug containers, the pasting of labels, and the use of tools all involve continuous time actions. The memory unit of LSTM can save the historical state of variable objects and predict the position of objects in the next frame by combining the input features of the current frame. Through the time series model, the system can not only capture the movement of variable objects in space, but also capture the dynamic changes of objects during operation, such as the rotation angle of tools and the pouring process of drugs. The preliminary path matrix output by LSTM records the spatial position and posture information of each variable object at different times. However, since video data may have noise or interference, such as light changes, object occlusion, etc., these factors may cause discontinuity or inaccuracy of the object's motion trajectory. Therefore, the system introduces the Kalman filter algorithm to optimize the preliminary path matrix. Kalman filtering is a recursive optimal estimation algorithm that can dynamically adjust the position data in the path matrix and eliminate the influence of noise by combining prior information and current observations. Specifically, the Kalman filter predicts the next moment position of the object, combines the observed actual position of the object, and uses the weighted calculation of the Gaussian distribution to smooth the final output path. Through the optimization of the Kalman filter, the generated variable path matrix can accurately reflect the motion trajectory of the variable object. This path matrix not only describes the position change of each object in the intravenous drug configuration scenario, but also reflects the temporal dynamic characteristics of the operation process, such as the displacement of the drug container, the trajectory of the tool, etc.The final path matrix provides a key data basis for subsequent operation evaluation and scoring, ensuring that the system can accurately determine whether the user's operation meets the predetermined standards.

[0068] Furthermore, the formula of the long short-term memory network is as follows:

[0069]

[0070] Among them, h t It is the hidden state output at the current moment, which is used to reflect the status of the current operation step; o t is the output of the output gate, which is used to control the information passed to the next moment; f t is the output of the forget gate, which is used to determine the cell state C at the previous moment t-1 Information that needs to be retained; C t-1 It is the information of the previous frame image; is the information of the candidate current frame image; α is the weight parameter, which is used to dynamically adjust the weights of different variable objects; Δ t It is the time difference change of the variable object during the operation.

[0071] Specifically, the forget gate f t ·C t-1 Determines how much state information C is retained from the previous moment (i.e. the previous frame image) t-1 This mechanism ensures that the LSTM network can reasonably remember the operation state of the previous moment and continue to track the changes of the object in the current frame. For example, during the intravenous drug dispensing process, the position and posture of the drug in the previous frame are partially retained to ensure the consistency of tracking. Input gate Used to update the state at the current moment, combined with the new information of the variable object in the current frame image (such as position, posture, etc.). By processing the current frame data, LSTM can flexibly adjust Realize timely tracking and adjustment of variable objects. t It is used to characterize the motion information of variable objects between different frames, which is related to the object's motion speed or position change between frames. It represents the magnitude of the object's change between time t and t-1. This term can be used to capture the object's motion trend and trajectory. For example, if the drug container is moved between two frames, the time difference term can quantify this movement information. α is used to adjust the impact of the time difference term on the current state update. By flexibly adjusting α, the system can adapt to the motion characteristics of variable objects in different scenarios. For faster movements, the system may need to increase the value of α to enhance sensitivity to time differences, while for slow operations, the system may reduce this value to reduce the interference of short-term changes on state updates. Hidden state h tIt is calculated by the nonlinear activation function tanh of LSTM, which combines the current input information and historical state information. t It not only contains the operation information in the current frame, but also reflects the dynamic changes of the operation object at different times, especially through the time difference change Δ t The motion trajectory reflected. Through this formula, the LSTM network can effectively capture the motion trajectory of variable objects at different times in the intravenous drug dispensing scenario, ensuring the accuracy of object tracking and state updates between multiple frames. This is crucial for continuous operation tracking in complex operation scenarios, helping the system better understand the user's operation behavior and provide data support for subsequent scoring and feedback.

[0072] Furthermore, the scoring data includes an accuracy score, a standardization score and an efficiency score.

[0073] Furthermore, the extraction of deduction items for incorrect operation prompts includes the following steps:

[0074] The convolutional neural network is used to analyze the spatiotemporal characteristics of the variable path matrix in real time, monitor the trajectory deviation of the variable object during the operation process, and identify the non-compliant operation steps by calculating the Euclidean distance and dynamic time warping value between the operation path and the standard path;

[0075] Detected operational deviations are marked as penalty items and visualized in real time via visual or text.

[0076] In some embodiments, first, the system obtains the variable path matrix in the user operation. The variable path matrix is ​​generated by the system through a long short-term memory network (LSTM) or a Kalman filter algorithm, and records the motion trajectory and time series change information of the variable objects during the intravenous drug configuration process. The spatial position and time series data of each variable object constitute the matrix. The matrix reflects the user's operation behavior when configuring the drug, such as the displacement of the drug container, the pasting position of the label, and the operation trajectory of the tool. The system first analyzes the spatiotemporal characteristics of the variable path matrix in real time through a convolutional neural network. The convolution layer of the convolutional neural network extracts the spatial and temporal feature information in the path matrix layer by layer to ensure that the detailed changes in the operation behavior are captured. The convolutional neural network is particularly suitable for processing such multidimensional feature data, and can effectively identify spatial deviations in the trajectory and time series anomalies in the operation. For example, during the process of drug label pasting, the network can capture the slight offset of the user's hand movement in space, or detect errors in the order of placing the drug containers. Next, the system evaluates the difference in spatial position by calculating the Euclidean distance between the user's operation path and the standard path. Euclidean distance is a common spatial distance measurement method used to calculate the straight-line distance between two points. For the motion trajectory of each variable object, the system compares each spatial point in the user's operation path with the corresponding point in the standard path and calculates the Euclidean distance between them. The larger the distance, the more obvious the deviation between the user's operation and the standard path. For example, if the user does not move the drug container according to the preset standard path during the drug preparation process, the system will identify the operation error through the calculated larger Euclidean distance. In order to handle the temporal characteristics in the operation, the system further adopts the dynamic time warping (DTW) algorithm. DTW is used to measure the similarity of two time series and can effectively capture the deviation between the user's operation and the standard path in the time dimension. Specifically, DTW calculates the time deviation between the user's operation trajectory and the standard trajectory by aligning the data points in the time series. For example, if the user's action is too fast or too slow in the drug mixing operation, the system identifies the difference in the time dimension through DTW calculation, ensuring that not only the errors in the spatial position are captured, but also the abnormalities in the operation time can be identified. After identifying the deviations in the operation, the system marks these deviations as penalty items. Each penalty item is associated with a specific operation step and variable object, such as incorrect drug label position, excessive drug dosage, incorrect tool placement, etc. The generation of penalty items is based on the results of convolutional neural networks and spatiotemporal feature analysis, which can perform detailed error analysis for each specific operation. The system quantifies these deviations and calculates the penalty value for each operation step according to the preset scoring criteria. Finally, the system visualizes the penalty items in real time in visual or text form. The specific visualization method can be graphical error highlighting, such as marking the error node on the user's operation path, or intuitively displaying the error location in the operation in a three-dimensional scene.At the same time, the system can display detailed descriptions of operational errors to users through text prompts to help them understand where the error is. For example, if the user does not correctly paste the drug label when dispensing the drug, the system will highlight the incorrect position of the label in the 3D virtual scene and explain the degree of deviation of the pasting position through text prompts.

[0077] Furthermore, the step of gradient-adjusting the type and quantity of intravenous drugs in the three-dimensional scene according to the scoring result includes:

[0078] According to the user's scoring result, it is compared with the preset scoring threshold. If the scoring result is greater than or equal to the threshold, the type and quantity of intravenous drug configuration in the three-dimensional scene are increased through the preset gradient; if the scoring result is lower than the scoring threshold, the type and quantity of intravenous drug configuration in the three-dimensional scene are reduced through the preset gradient.

[0079] Specifically, when the user's operation score is greater than or equal to the system's preset score threshold, the system increases the types and quantity of intravenous drug configurations in the three-dimensional scene through preset gradient parameters. At this time, the system will add more drug types and configuration steps based on the original operation scene. The added drug types may include more complex drug configuration operations, such as mixing multiple drugs, preparing specific doses of drugs, etc. The gradient parameters are used to control the increased operation difficulty, and are usually adjusted step by step according to the user's current operation ability to ensure that the challenge is gradually increased without causing excessive difficulty jumps. For example, after the user completes a specific drug configuration step many times, the system will add additional drug types or complex operation steps to further test the user's operation level. When the user's score result is lower than the system's score threshold, the system will reduce the types and quantity of intravenous drug configurations in the three-dimensional scene through preset gradients. At this time, the system will simplify the operation tasks, reduce the types of drugs in the operation scene, and reduce the complexity of the tasks. By reducing the types and steps of drugs, the system helps users concentrate on practicing basic operations and gradually improve their operation proficiency. For example, when a user makes a mistake in drug dosage configuration, the system may reduce the types of drugs that need to be processed and provide the user with more basic drug operation tasks to help him master the most basic drug preparation skills. This adjustment reduces the variables that the user needs to deal with through a preset gradient, ensuring that the user can effectively learn in a simpler task environment.

[0080] The above implementation modes are merely descriptions of the preferred implementation modes of the present invention, and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary engineering and technical personnel in the field shall fall within the protection scope determined by the claims of the present invention.

Claims

1. A teaching system for intravenous drug preparation based on deep learning, characterized in that: It includes a data acquisition module, a three-dimensional space simulation module, a variable capture module, an operation scoring module and a personalized practice module which are connected in sequence, and the personalized practice module is connected to the three-dimensional space simulation module; The data acquisition module is used to obtain real-life video data of intravenous drug dispensing and perform data annotation; The three-dimensional space simulation module is used to construct a three-dimensional scene of intravenous drug configuration based on the real-scene video data of intravenous drug configuration, and initialize the type and quantity of intravenous drugs in the three-dimensional scene; the three-dimensional space virtual module is also used to receive user-side operation instructions to perform intravenous drug configuration operations in the three-dimensional scene; The variable capture module is used to perform the following steps: S1, obtaining video data of a three-dimensional scene for intravenous drug dispensing operation; S2, performing frame decomposition on the operation video data of the three-dimensional scene of intravenous drug configuration to obtain a frame image set, performing semantic segmentation training on the variable objects in the frame image set, and obtaining a variable capture model; S3, a frame image set of operation video data of a three-dimensional scene based on intravenous drug configuration, performing variable tracking through a variable capture model and generating a variable path matrix; The operation scoring module is used to take the variable path matrix as input and the corresponding scoring data as output, to construct a scoring model based on a convolutional neural network, and is also used to generate a scoring result for the real-time input real-scene video data through the scoring model and extract the deduction items to provide wrong operation prompts; The personalized training module is used to perform gradient adjustments on the types and quantities of intravenous drugs in the three-dimensional scene according to the scoring results.

2. According to claim 1, a deep learning-based intravenous drug preparation teaching system is characterized in that: The three-dimensional scene of intravenous drug configuration includes prescription review, label printing, label placement, drug placement verification, mixed preparation, finished product verification, finished product packaging and zoning placement; the variable objects include intravenous drugs, drug labels, tools and operating tables.

3. According to claim 1, a teaching system for intravenous drug preparation based on deep learning is characterized in that: The method of constructing a three-dimensional scene of intravenous drug disposition based on real-life video data of intravenous drug disposition comprises the following steps: Perform frame segmentation on real-scene video data, extract key feature points in each frame to generate a feature point matrix, including feature points of intravenous drugs, drug labels, tools, and operating tables; The feature points of different perspectives are associated through a multi-perspective image matching algorithm to generate a corresponding depth information matrix; Based on the depth information matrix, the variable objects in the scene are reconstructed in three dimensions to generate a three-dimensional scene of intravenous drug configuration.

4. According to claim 1, a teaching system for intravenous drug preparation based on deep learning is characterized in that: The S2 comprises the following steps: Output the frame image set to the backbone network of DeepLab for feature extraction to obtain a feature map; The feature map is integrated with multi-scale context information through the atrous spatial pyramid pooling module. The feature map processed by the dilated spatial pyramid pooling module is restored to its original size through deconvolution operation to generate a variable segmentation mask; The generated variable segmentation mask is cross-entropy-loss calculated with the labeled data, and the network weights are adjusted through the Adam optimizer to obtain the variable capture model.

5. The teaching system for intravenous drug preparation based on deep learning according to claim 1 is characterized in that: The S3 comprises the following steps: Based on the frame image set of the operation video data of the three-dimensional scene of intravenous drug configuration, the corresponding variable segmentation mask is generated through the variable capture model, and a unique identifier is generated for each variable object therein, and its spatial position and posture are initialized; Based on the long short-term memory network, the variable objects in different frame images are continuously tracked, and their motion trajectories between different frames are output to generate a preliminary path matrix. The preliminary path matrix is ​​de-noised using the Kalman filter algorithm to generate a variable path matrix.

6. The teaching system for intravenous drug preparation based on deep learning according to claim 5, characterized in that: The formula of the long short-term memory network is as follows: Among them, h t It is the hidden state output at the current moment, which is used to reflect the status of the current operation step; o t is the output of the output gate, which is used to control the information passed to the next moment; f t is the output of the forget gate, which is used to determine the cell state C at the previous moment t-1 Information that needs to be retained; C t-1 It is the information of the previous frame image; is the information of the candidate current frame image; α is the weight parameter, which is used to dynamically adjust the weights of different variable objects; Δ t It is the time difference change of the variable object during the operation.

7. The teaching system for intravenous drug preparation based on deep learning according to claim 1 is characterized in that: The scoring data includes an accuracy score, a standardization score, and an efficiency score.

8. The teaching system for intravenous drug preparation based on deep learning according to claim 1 is characterized in that: The extracting of deduction items for error operation prompts comprises the following steps: The convolutional neural network is used to analyze the spatiotemporal characteristics of the variable path matrix in real time, monitor the trajectory deviation of the variable object during the operation process, and identify the non-compliant operation steps by calculating the Euclidean distance and dynamic time warping value between the operation path and the standard path; Detected operational deviations are marked as penalty items and visualized in real time via visual or text.

9. The teaching system for intravenous drug preparation based on deep learning according to claim 1, characterized in that: The step of gradient-adjusting the type and quantity of intravenous drugs in the three-dimensional scene according to the scoring result includes: According to the user's scoring result, it is compared with the preset scoring threshold. If the scoring result is greater than or equal to the threshold, the type and quantity of intravenous drug configuration in the three-dimensional scene are increased through the preset gradient; if the scoring result is lower than the scoring threshold, the type and quantity of intravenous drug configuration in the three-dimensional scene are reduced through the preset gradient.

Images