A method for detecting intravenous infusion level based on semantic segmentation

Abstract

This invention provides a method for detecting intravenous infusion level based on semantic segmentation, comprising the following steps: labeling the infusion bottle and infusion bag images to obtain a dataset; extracting image features from the dataset using a feature extraction module to generate low-level and high-level feature maps; inputting the high-level feature map into a deep feature extraction module to obtain a first feature map; inputting the low-level feature map and the first feature map into a feature fusion module to obtain a fused feature map; inputting the fused feature map into a fully connected module to obtain semantic segmentation results for images in the dataset with and without liquid levels; iteratively training a semantic segmentation network based on the liquid level labeling, semantic segmentation results, and a target loss function to obtain a semantic segmentation network model; inputting the image of the infusion bottle or infusion bag to be tested into the semantic segmentation network model to output the intravenous infusion level detection result. This invention can accurately and efficiently identify the liquid level status during intravenous injection.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to a method for detecting intravenous infusion level based on semantic segmentation. Background Technology

[0002] Intravenous infusion is a common clinical treatment method used in hospitals. During routine infusions, medical staff need to frequently monitor the medication level in the IV bag to change the fluid or remove the needle as needed. For patients in the intensive care unit and those without family members and suffering from infectious diseases, close monitoring is even more crucial. If the IV bag empties during the infusion process, backflow of blood may occur, causing swelling and pain at the injection site in mild cases, and potentially leading to shock in severe cases. When there are many patients receiving infusions, monitoring and managing these processes becomes a significant burden.

[0003] Intravenous fluid level detection is a crucial part of infusion monitoring and treatment. Automation of infusion monitoring is only possible by replacing manual labor with automated equipment for fluid level detection. Existing methods primarily utilize capacitance and ultrasound. Capacitive level detection requires a constant dielectric constant for the medium and the liquid surface to achieve accurate measurement; ultrasonic level detection devices are significantly affected by the gas composition of the transmission medium and the geometry of the infusion bag and its container. All of these methods malfunction when affected by external environmental factors. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the above-mentioned background technology and provide a method for detecting intravenous infusion level based on semantic segmentation, which can accurately and efficiently identify the intravenous infusion level and is not affected by external environmental factors.

[0005] The technical solution adopted in this invention is: a method for detecting intravenous infusion level based on semantic segmentation, comprising the following steps:

[0006] The images of infusion bottles and bags are labeled with fluid levels to obtain a dataset;

[0007] Input the dataset into the semantic segmentation network;

[0008] The semantic segmentation network includes a feature extraction module, a deep feature extraction module, a feature fusion module, and a fully connected module;

[0009] The feature extraction module extracts image features from the dataset and generates low-level feature maps and high-level feature maps;

[0010] The high-level feature map is input into the deep feature extraction module for feature extraction to obtain the first feature map;

[0011] The low-level feature map and the first feature map are input into the feature fusion module for fusion to obtain a fused feature map;

[0012] The fused feature map is input into the fully connected module, which classifies the pixels on the fused feature map to obtain semantic segmentation results for images in the dataset with and without liquid surfaces.

[0013] The semantic segmentation network model is obtained by iteratively training the semantic segmentation network based on the liquid level annotations and semantic segmentation results of the images in the dataset and the target loss function.

[0014] The image of the infusion bottle or bag to be tested is input into the semantic segmentation network model; the semantic segmentation network model outputs the intravenous infusion level detection result.

[0015] In the above technical solution, the feature extraction module includes a convolution module and multiple inverse residual modules; the feature extraction module includes a first-stage module and a second-stage module. The first-stage module is composed of a convolution module, a convolutional attention module, and three inverse residual modules connected in sequence. The input of the first-stage module is an image of an infusion bottle or infusion bag. After feature extraction by the first-stage module, a low-level feature map is obtained from the image of the infusion bottle or infusion bag. The second-stage module is composed of fourteen inverse residual modules connected in sequence. The input of the second-stage module is the low-level feature map. After feature extraction by the inverse residual modules in the second-stage module, a high-level feature map is obtained from the low-level feature map.

[0016] In the above technical solution, the convolutional module of the feature extraction module sequentially includes a convolutional layer, a batch normalization layer, and an activation layer. The convolutional layer uses the input of the convolutional module as its input to perform feature extraction. The convolutional layer is used to extract the same distribution feature matrix and input data features. Performing a convolution operation on an image is essentially using a convolutional kernel to slide across the image, multiplying the value on the image with the value on the corresponding convolutional kernel, and then summing all the multiplied values ​​as the output value at that position. Finally, the kernel slides through the entire image to obtain the feature map after convolution calculation. The batch normalization layer performs batch normalization calculation on the output of the convolutional layer and then uses the output of the batch normalization layer as the input of the activation layer. The activation layer uses the output of the batch normalization layer as its input to perform linear activation calculation and uses the output of the activation layer as the output of the convolutional module.

[0017] In the above technical solution, the convolutional attention module of the feature extraction module includes: a channel attention structure and a spatial attention structure connected in sequence. The output of the convolutional module is used as the input of the convolutional attention module, that is, as the input of the channel attention structure. The output result obtained by multiplying the input and output of the channel attention structure element-wise is used as the input of the spatial attention structure. The output of the convolutional attention module is obtained by multiplying the input and output of the spatial attention structure element-wise.

[0018] In the above technical solution, the inverse residual module of the feature extraction module includes a convolution module, a depthwise separable convolution module, and a convolutional layer connected in sequence. When the feature map is input to the inverse residual module, the convolution module extracts features from the feature map and uses the result of feature extraction as the input of the depthwise separable convolution module. The depthwise separable convolution module extracts features from the output of the convolution module and uses the result as the input of the convolutional layer. The convolutional layer extracts features from the output of the depthwise separable convolution module and adds the result of feature extraction to the input of the inverse residual module pixel by pixel before outputting the result.

[0019] In the above technical solution, the deep feature extraction module includes a convolutional module, three depthwise separable convolutional modules, a global average pooling layer, a dimensional concatenation operation, a convolutional layer, and an upsampling layer. The convolutional module, the three depthwise separable convolutional modules, and the global average pooling layer in the deep feature extraction module extract features from the high-level feature map to obtain five feature maps. After performing a dimensional feature concatenation operation on the five obtained feature maps, they are sequentially input into the convolutional layer and the upsampling layer to obtain the first feature map.

[0020] In the above technical solution, the feature fusion module includes a convolution module, a dimension concatenation operation, a convolutional layer, and an upsampling layer. The convolution module in the feature fusion module reduces the dimensionality of the low-level feature map, and after performing a feature concatenation operation on the first feature map in the dimension, the low-level feature map and the first feature map are sequentially input into the convolutional layer and the upsampling layer to obtain the fused feature map.

[0021] In the above technical solution, the fully connected module includes a fully connected layer and a softmax activation function; the fully connected layer classifies each pixel in the fused feature map to obtain the category to which each pixel belongs, and the category includes liquid part and liquid-free part; the output of the fully connected layer is input into the softmax function to perform probability mapping to obtain the probability value of the category to which each pixel belongs.

[0022] In the above technical solution, the process of iteratively training the semantic segmentation network based on the liquid level annotation and semantic segmentation results of the images in the dataset and the target loss function to obtain the semantic segmentation network model includes: using the loss function to calculate the probability of each pixel in the fused feature map belonging to the category and the loss value of the true liquid level annotation result; using the backpropagation algorithm to calculate the loss value to update the parameters in the semantic segmentation network; iterating the above process for a predetermined number of rounds to obtain the semantic segmentation network model.

[0023] In the above technical solution, the process of using the backpropagation algorithm to calculate the loss value and update the parameters in the semantic segmentation network includes: using the loss backpropagation algorithm in the semantic segmentation network to calculate the gradient of the loss value with respect to each parameter; using an optimization strategy to optimize the semantic segmentation network; and using the gradient descent algorithm to update the parameters of the semantic segmentation network based on the gradient and the optimization strategy.

[0024] The present invention also provides a semantic segmentation-based intravenous infusion level detection system, which is used to implement the semantic segmentation-based intravenous infusion level detection method as described in the above technical solution.

[0025] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the semantic segmentation-based intravenous infusion level detection method as described in the above technical solution.

[0026] The beneficial effects of this invention are: it achieves automated identification of the fluid level on infusion bags or bottles by constructing a semantic segmentation network model. After detecting the intravenous infusion level using this method, the detection results can be displayed in real time on a display device, allowing medical staff to be promptly aware of changes in the fluid level of the infusion bottle or bag, or to trigger alarms by setting thresholds, providing convenience for both medical staff and patients. It has broad application prospects in the medical field.

[0027] Furthermore, the feature extraction module of this invention uses a depthwise separable convolution module, which reduces the network parameters and computational load during the feature extraction process and improves the inference speed of the network.

[0028] Furthermore, the feature extraction module constructed in this invention features a relatively stable distribution of input data at each layer, accelerating model learning speed. It is insensitive to the range of input data in the network, simplifying the parameter tuning process and making network learning more stable. It allows the network to use saturated activation functions, alleviating the gradient vanishing problem and, to some extent, regularizing the model. Each convolutional module includes feature extraction capabilities and enhances the network's linear mapping ability. The feature extraction module of this invention uses a convolutional attention module to allow the network to focus more on extracting features of the liquid-containing and liquid-free areas of the infusion bottles and bags in the image during subsequent feature extraction. This addresses the problem that focusing only on extracting global image information results in insufficiently clear features of the liquid-containing and liquid-free areas of the infusion bottles and bags, leading to low accuracy in pixel-by-pixel classification.

[0029] Furthermore, the deep feature extraction module of this invention extracts five feature maps from a high-level feature map. These five feature maps are then concatenated dimensionally, allowing the network to capture more details and effectively addressing the image resolution and information loss issues caused by downsampling in semantic segmentation, thereby improving segmentation accuracy. The depthwise separable convolution module within the deep feature extraction module further reduces network parameters and computational load, improving the network's inference speed.

[0030] Furthermore, the feature fusion module of the present invention performs feature concatenation operation between the low-level feature map and the first feature map in terms of dimension, and then inputs them sequentially into the convolutional layer and the upsampling layer to obtain the fused feature map. This makes the fused feature map have more detailed information of the original infusion image, which is beneficial to further improve the accuracy and precision of segmentation.

[0031] Furthermore, this invention uses a dataset of infusion images taken under different lighting conditions and from different shooting angles to train the semantic segmentation network, which can improve the generalization ability of the trained model.

[0032] Furthermore, the convolutional attention module constructed in this invention enables the network to focus more on extracting features of liquid-containing and liquid-free areas of infusion bottles and bags in images, thereby improving the accuracy of classifying liquid-containing and liquid-free parts of infusion bottles or bags in subsequent pixel-by-pixel classification tasks.

[0033] Furthermore, the inverse residual module constructed in this invention uses depthwise separable convolution, which can optimize the network structure, reduce the amount of computation, and thus improve the inference speed. The inverse residual module also uses linear activation, that is, only convolutional layers are used in the depthwise separable convolution without adding activation functions, which makes it less likely to cause information loss when extracting features in low dimensions.

[0034] Furthermore, the semantic segmentation network training process of this invention saves the deep learning classification network model with the minimum loss value as the final output of the training, effectively improving the accuracy of the final model used to determine the infusion fluid level. Attached Figure Description

[0035] Figure 1 This is a schematic diagram of the process of the present invention;

[0036] Figure 2 This is a diagram of the semantic segmentation network structure of the present invention;

[0037] Figure 3 The following are the original and labeled drawings of the infusion bottle and infusion bag in this specific embodiment;

[0038] Figure 4 This is a flowchart of the semantic segmentation network operation of the present invention;

[0039] Figure 5 This is a flowchart of the feature extraction module operation of the present invention;

[0040] Figure 6 This is a flowchart of the convolution module operation of the present invention;

[0041] Figure 7 This is a flowchart of the operation of the convolutional attention module of the present invention.

[0042] Figure 8 This is a flowchart of the operation of the reverse residual module of the present invention;

[0043] Figure 9 This is a flowchart of the deep feature extraction module of the present invention;

[0044] Figure 10 This is a flowchart illustrating the training process of the semantic segmentation network model of this invention.

[0045] Figure 11 This is a curve showing the change in loss value during the training process of the semantic segmentation network of this invention.

[0046] Figure 12 This is a diagram showing the separation of the liquid-filled and liquid-free portions of the infusion bottle of the present invention, which is also a diagram showing the liquid level detection effect.

[0047] Figure 13 This is a diagram illustrating the separation of the liquid-containing and liquid-free portions of the infusion bag of the present invention.

[0048] Figure 14 This is a flowchart of the operation of a semantic segmentation-based intravenous infusion level detection device.

[0049] Figure 15 This is a schematic diagram of a device for detecting intravenous infusion level. Detailed Implementation

[0050] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments to facilitate a clear understanding of the present invention, but these descriptions do not constitute a limitation on the present invention.

[0051] The technical solution adopted in this invention is: a method for detecting intravenous infusion level based on semantic segmentation, comprising the following steps:

[0052] The images of infusion bottles and bags are labeled with fluid levels to obtain a dataset;

[0053] Input the dataset into the semantic segmentation network;

[0054] The semantic segmentation network includes a feature extraction module, a deep feature extraction module, a feature fusion module, and a fully connected module;

[0055] The feature extraction module extracts image features from the dataset and generates low-level feature maps and high-level feature maps;

[0056] The high-level feature map is input into the deep feature extraction module for feature extraction to obtain the first feature map;

[0057] The low-level feature map and the first feature map are input into the feature fusion module for fusion to obtain a fused feature map;

[0058] The fused feature map is input into the fully connected module, which classifies the pixels on the fused feature map to obtain semantic segmentation results for images in the dataset with and without liquid surfaces.

[0059] The semantic segmentation network model is obtained by iteratively training the semantic segmentation network based on the liquid level annotations and semantic segmentation results of the images in the dataset and the target loss function.

[0060] The image of the infusion bottle or bag to be tested is input into the semantic segmentation network model; the semantic segmentation network model outputs the intravenous infusion level detection result.

[0061] like Figure 1 As shown in the figure, this specific embodiment provides a method for detecting intravenous infusion level based on semantic segmentation, including the following steps:

[0062] S101, acquire images of infusion bottles and bags under different conditions.

[0063] In this embodiment, images of infusion bags or bottles during infusion are captured using cameras installed in the ward or infusion area. A total of 8030 infusion images were collected by taking pictures of the infusion bags or bottles under different lighting conditions and from different shooting angles. Using this dataset of infusion images taken under different lighting conditions and from different shooting angles to train the semantic segmentation network improves the generalization ability of the trained model.

[0064] S102, label the images of infusion bottles and bags and divide the dataset.

[0065] After obtaining 8030 images of infusions, the LabelMe software was used to label the liquid-filled and liquid-free portions of the infusion bottles and bags in the images. The dataset contains images of infusion bottles and bags with corresponding labels, as shown below. Figure 3 As shown in the diagram. During the dataset annotation process, class 0 represents no fluid (gray), and class 1 represents fluid (blue). After annotating 8030 infusion images, the annotation files were converted to VOC dataset format to obtain the dataset. During training, a portion of the images was randomly selected for the training set, and the remaining portion was used for the test set. The random ratio of the training and test sets was 90% training set and 10% test set.

[0066] S103, Construction of Semantic Segmentation Network

[0067] In this embodiment, a semantic segmentation model is used to detect the intravenous infusion level. By classifying the pixels of the infusion bottle or bag as having a liquid surface, not having a liquid surface, and background pixels, the model can segment the infusion bottle or bag as having a liquid surface or not, thus realizing the detection of the intravenous infusion level.

[0068] The semantic segmentation network comprises a feature extraction module, a deep feature extraction module, a feature fusion module, and a fully connected module, all constructed using default parameters. When using an image of an infusion bottle or bag as network input, the feature extraction module extracts features from the image to obtain low-level and high-level feature maps. The high-level feature map is then input into the deep feature extraction module for further feature extraction to obtain a first feature map. The low-level and first feature maps are then fused in the feature fusion module to obtain a fused feature map. The fused map is then input into the fully connected module to obtain semantic segmentation results for the infusion bottle or bag with and without a liquid level, i.e., the intravenous infusion level detection result.

[0069] like Figure 4 The diagram shown is a flowchart of the semantic segmentation network operation provided by this invention, which specifically includes the following:

[0070] S201, the feature extraction module extracts features from the image of the infusion bottle or infusion bag to obtain low-level feature maps and high-level feature maps.

[0071] like Figure 5 The diagram shown is a flowchart of the feature extraction module in this embodiment. The feature extraction module consists of a convolution module, a convolutional attention module, and multiple inverse residual modules.

[0072] like Figure 6The diagram shown is a flowchart of the convolution module operation in this embodiment. The convolution module sequentially includes a convolutional layer, a batch normalization layer, and an activation layer. Specifically, the first layer of the convolution module is the convolutional layer, which uses the input of the convolution module as its input for feature extraction. The main function of the convolutional layer is to extract the feature matrix with the same distribution and input data features. Performing a convolution operation on an image involves sliding the convolution kernel across the image, multiplying the value on the image with the value on the corresponding convolution kernel, and then summing all the multiplied values ​​as the output value at that position. Finally, the kernel slides through the entire image to obtain the feature map after convolution calculation. Specifically, the batch normalization layer is the second layer of the convolution module, responsible for batch normalizing the output of the convolutional layer; the output of the batch normalization layer is then used as the input of the activation layer.

[0073] Batch normalization layers can serve the following functions:

[0074] (1) This makes the distribution of input data in each layer of the network relatively stable, thus accelerating the learning speed of the model.

[0075] (2) This makes the model less sensitive to the range of input data in the network, simplifies the parameter tuning process, and makes the network learning more stable.

[0076] (3) Allow the network to use saturated activation functions (such as sigmoid, tanh, etc.) to alleviate the gradient vanishing problem.

[0077] (4) It has a regularization effect on the model to a certain extent.

[0078] Specifically, to enhance the expressive power of the entire network, an activation function is introduced at the back end of the convolutional layers as the activation layer of the convolutional module. This activation layer is the third and final layer of the convolutional module. It takes the output of the batch normalization layer as its input, performs linear activation calculations, and uses its output as the output of the convolutional module. This ensures that each convolutional module incorporates feature extraction capabilities and enhances the network's linear mapping ability. In this embodiment, the activation function used is ReLU6.

[0079] like Figure 7 The diagram shows the operation flowchart of the convolutional attention module of this invention. The output of the convolutional module serves as the input of the convolutional attention module, that is, as the input of the channel attention structure. The input of the channel attention structure is used as the input of two parallel max pooling layers and average pooling layers. The results processed by the max pooling and average pooling layers are respectively input into the fully connected layers. The outputs of the two fully connected layers are added element-wise, and then the result obtained by the sigmoid activation function is multiplied by the input of the channel attention structure to obtain the output of the channel attention structure.

[0080] The output of the channel attention structure is used as the input to the spatial attention structure. This input is then fed into two parallel max-pooling and average-pooling layers. The outputs of the two pooling layers are concatenated dimensionally, and the concatenated result is input into a depthwise separable convolutional module for feature extraction. The output of this module is then used as the input to a sigmoid activation function. Finally, the result obtained through the sigmoid activation function is multiplied element-wise with the input to the spatial attention structure to obtain the output of the convolutional attention module.

[0081] like Figure 8 The diagram shown is a flowchart of the inverse residual module operation in this embodiment. The inverse residual module consists of a convolution module, a depthwise separable convolution module, a convolutional layer, and an add operation. Specifically, the convolution module is the first layer of the inverse residual module, and its input is also the input of the convolution module. When the feature map is input to the inverse residual module, the convolution module extracts features from the feature map and uses the result as the input of the depthwise separable convolution module. The depthwise separable convolution module is the second layer of the inverse residual module. After extracting features from the output of the convolution module, it uses the result as the input of the convolutional layer. The convolutional layer is the third layer of the inverse residual module. It extracts features from the output of the depthwise separable convolution module and adds the result of the feature extraction to the input of the inverse residual module pixel by pixel. The inverse residual module in this embodiment uses depthwise separable convolution, which can optimize the network structure, reduce the amount of computation, and thus improve the inference speed. The inverse residual module also uses linear activation, that is, only convolutional layers are used in the depthwise separable convolution without adding activation functions, which makes it less likely to cause information loss when extracting features in low dimensions.

[0082] In this embodiment, the feature extraction module consists of a convolutional module, a convolutional attention module, and multiple inverse residual modules. The feature extraction module is divided into two stages. Stage one consists of the convolutional module, the convolutional attention module, and three inverse residual modules connected sequentially. The input to stage one is the image of the infusion bottle or bag, which is the input to the feature extraction module. After feature extraction in stage one, the image of the infusion bottle or bag yields a feature map with dimensions of 256×256×24, i.e., a low-level feature map.

[0083] Phase 2 consists of fourteen inverse residual modules connected in sequence. The input to Phase 2 is a feature map with a size and dimension of 256×256×24. After the inverse residual modules in Phase 2 extract features from the feature map with a size and dimension of 256×256×24, a feature map with a size and dimension of 32×32×320 is obtained, which is the high-level feature map.

[0084] S202, input the advanced feature map into the deep feature extraction module to extract features and obtain the first feature map.

[0085] like Figure 9 The diagram shown is a flowchart of the deep feature extraction module in this embodiment. The deep feature extraction module includes a convolutional module, three depthwise separable convolutional modules, a global average pooling layer, a regular convolutional layer, and an upsampling layer.

[0086] The convolutional module in the deep feature extraction module is the same as the convolutional module in the feature extraction module, both consisting of convolutional layers, batch normalization layers, and activation layers connected in sequence.

[0087] The depthwise separable convolution module in the depth feature extraction module consists of a channel-wise convolution layer, a pointwise convolution layer, a batch normalization layer, and an activation layer. The depthwise separable convolution module is primarily used to address the image resolution and information loss issues caused by downsampling in semantic segmentation problems, thereby improving segmentation accuracy.

[0088] Specifically, the first layer of the depthwise separable convolutional module is a channel-wise convolutional layer, and the input to the depthwise separable convolutional module is also the input to the channel-wise convolutional layer. After the channel-wise convolutional layer extracts features from the input high-level feature map, the result of the feature extraction is used as the input to the second pointwise convolutional layer of the depthwise separable convolutional module.

[0089] Specifically, the pointwise convolutional layer takes the output of the channelwise convolutional layer as input, performs feature extraction on the output of the channelwise convolutional layer, and then uses the extracted feature results as input to the batch normalization layer.

[0090] Specifically, the batch normalization layer is the third layer of the depthwise separable convolutional module, responsible for batch normalizing the output of the pointwise convolutional layer; then the output of the batch normalization layer is used as the input of the activation layer.

[0091] Specifically, the activation layer is the fourth and final layer of the depthwise separable convolutional module. The activation layer takes the output of the batch normalization layer as its input for linear activation calculation, and uses its output as the output of the depthwise separable convolutional module. This ensures that each depthwise separable convolutional module includes feature extraction capabilities and enhances the network's linear mapping ability. In this embodiment, the activation function used in the activation layer is ReLU6.

[0092] The main function of the global average pooling layer in the deep feature extraction module is to retain the main features while reducing parameters and computational cost, thereby accelerating the training and inference speed of the network. The global average pooling layer can aggregate and statistically analyze features at different locations in the feature map, gradually realizing the representation of image features from high to low levels. Furthermore, the global average pooling layer helps enhance the network's adaptability to image variations. Specifically, the global average pooling layer reduces the amount of data by selecting the average value of regions. Its typical operation involves dividing the input image into several rectangular regions and averaging each sub-region. In this embodiment, the average pooling method divides the original H*W feature map into H / 2*W / 2 2*2 matrices, averages each 2*2 matrix, and merges the averages of all 2*2 matrices into a new H / 2*W / 2 feature map, achieving a 4-fold downsampling of the feature map. Using the average pooling method retains the relevant feature information while reducing the overall computational cost of the network.

[0093] After obtaining the high-level feature map, input the high-level feature map into the deep feature extraction module to extract features and obtain the first feature map.

[0094] Specifically, the high-level feature map is input into the convolution module for feature extraction to obtain features. Figure 1 The high-level feature map is input into the first depthwise separable convolutional module for feature extraction to obtain the features. Figure 2 The high-level feature map is input into a second depthwise separable convolutional module for feature extraction to obtain the features. Figure 3 The high-level feature map is input into a third depthwise separable convolutional module for feature extraction to obtain the features. Figure 4 The high-level feature map is input into the global average pooling module for feature extraction to obtain the features. Figure 5 ;

[0095] Specifically, the features Figure 1 ,feature Figure 2 ,feature Figure 3 ,feature Figure 4 ,feature Figure 5 Features are obtained after performing a concatenation operation along the channel dimension. Figure 6 Use a 1×1 ordinary convolution to apply features Figure 6 After channel compression, the input is fed into the upsampling layer for upsampling, resulting in a first feature map with dimensions of 128×128×512.

[0096] S203, input the low-level feature map and the first feature map into the feature fusion module to obtain the fused feature map.

[0097] After obtaining the low-level feature map and the first feature map, the low-level feature map and the first feature map are used as input to the feature fusion module. First, the low-level feature map is compressed by channels using a 1×1 ordinary convolution to obtain a feature map with a size and dimension of 128×128×48. The feature map with a size and dimension of 128×128×48 is concatenated with the first feature map in the channel dimension to obtain a feature map with a size and dimension of 128×128×560. The feature map with a size and dimension of 128×128×560 is compressed by channels using a 1×1 ordinary convolution and then upsampled to obtain a fused feature map with a size and dimension of 512×513×3.

[0098] S204. Input the fused feature map into the fully connected module to obtain the segmentation results of the infusion bag with and without liquid surface, which is also the result of intravenous infusion level detection.

[0099] The fully connected module in this embodiment includes a fully connected layer and a softmax function.

[0100] After obtaining the fused feature map, it is input into a fully connected layer. The fully connected layer classifies each pixel in the fused feature map to obtain the category to which each pixel belongs, thus obtaining the segmentation results of the liquid-containing and liquid-free parts. The output of the fully connected layer is then input into a softmax function to perform probability mapping, obtaining the probability value of the category to which each pixel belongs.

[0101] S104, Training of the semantic segmentation network model

[0102] The following will be based on the appendix Figure 10 Introducing the training process of semantic segmentation network models:

[0103] S301, Based on the classification results and the true results of each pixel in the fused feature map, the loss value is calculated using a loss function.

[0104] The input image of an infusion bottle or bag is processed by network inference to obtain the classification result of each pixel. The classification result includes the probability of each pixel in the image belonging to the category. However, the classification result is not necessarily correct. Since the correct category of the image has been marked in advance when the image is acquired, i.e. the true result, the difference between the classification result and the true result can be calculated using a loss function, which is the loss value.

[0105] In this embodiment, the cross-entropy loss function is used to calculate the loss value between the classification result and the true result of the image pixels. The cross-entropy loss function is a commonly used loss function in classification problems, and its specific formula is shown in Formula 1:

[0106]

[0107] Among them, y i p(y) is the true class probability value of a certain image pixel i. i ) represents the predicted class probability value of a certain image pixel i, n is the number of classes, and BCELoss is the classification loss value.

[0108] BCELoss represents the difference between the true sample label and the predicted probability. The smaller the loss value, the higher the prediction accuracy.

[0109] S302, Calculate the gradient of the loss value with respect to each learnable parameter using the backpropagation algorithm based on the loss value.

[0110] Semantic segmentation network model training involves adjusting the values ​​of learnable parameters in each layer of a convolutional neural network to make the predicted value of the input image of an IV bottle or bag closer to the true value. The specific evaluation metric for the difference between the predicted and true values ​​is the loss value; the smaller the loss value, the closer the predicted value is to the true value, and the higher the accuracy of the network's prediction.

[0111] In this embodiment, the semantic segmentation network is a convolutional neural network. The values ​​of each parameter in the convolutional neural network are calculated using chain rule differentiation during backpropagation to obtain the gradient of the loss value with respect to each parameter. Then, gradient descent is used to update the parameters, adjusting them in the opposite direction of the gradient. This results in the fastest and highest rate of change in the loss value. The set of these gradients represents the optimal parameter tuning direction that minimizes the number of training epochs and reduces the loss value.

[0112] S303, Select an optimization strategy based on the semantic segmentation network.

[0113] The learning rate (lr) is a crucial hyperparameter in deep learning, representing the step size by which the model adjusts its weights in the reverse direction of the gradient during parameter optimization. In other words, it controls the magnitude of weight adjustments made by the model during optimization. A high learning rate means larger weight updates, allowing for faster finding of a new set of weights and minimizing the network's training loss. However, a high learning rate can also lead to excessively large weight updates, preventing the network from reaching its optimal point. A low learning rate ensures that no local minima are missed, but it also means that convergence will take longer. Therefore, choosing an appropriate learning rate during model training is extremely important.

[0114] The optimizer is an optimization strategy based on the backpropagation algorithm. Its role is to help the model adjust its parameters more quickly and effectively during training.

[0115] In this embodiment, an initial learning rate of 0.001 is selected, the learning rate descent strategy is cosine decay, and the optimizer is the Adam gradient descent algorithm. The cosine decay strategy uses cosine correlation to decay the learning rate, and the decay graph is similar to that of a cosine function.

[0116] In this embodiment, the Adam optimizer is chosen as the gradient descent strategy. Adam is an algorithm that combines Momentum and RMSProp, introducing the first-order momentum of Momentum to accumulate the gradient and the second-order momentum of RMSProp. This allows for fast convergence while minimizing fluctuations. Furthermore, two correction terms are added to enable parameter self-renewal. The Adam optimizer has the following advantages: simple implementation, computational efficiency, and low memory requirements; parameter updates are unaffected by gradient scaling; hyperparameters are highly interpretable, and the update step size can be limited to a general range, typically requiring no adjustment or only minimal fine-tuning; and it naturally implements the step-size annealing process (automatically adjusting the learning rate).

[0117] S304 updates the parameters using the gradient descent algorithm based on the gradient and the optimization strategy.

[0118] When using the optimizer Adam to update parameters, the update process can be represented by the following formula:

[0119]

[0120] m t =β1m t-1 +(1-β1)g t (3)

[0121]

[0122]

[0123]

[0124]

[0125] in:

[0126] t: Represents the number of update steps

[0127] g t : Represents the objective function L(θ) t The gradient obtained by differentiating with respect to θ

[0128] β1: First-order moment attenuation coefficient

[0129] β2: Second-order moment attenuation coefficient

[0130] β1 raised to the power of t

[0131] β2 raised to the power of t

[0132] m t gradient g t The first moment, i.e., the gradient g t Expectations

[0133] v t gradient g t The second moment, i.e., the gradient g t Expectations

[0134] m t bias correction

[0135] v t bias correction

[0136] θ t+1 Updated parameters

[0137] The parameter update process is as follows: obtain the gradient g. t Then, the gradient g is calculated. t First moment m t and gradient g t The second moment v t Then calculate m t and v t bias correction and Then according to m t and v t bias correction m t and v t bias correction The new parameters can be obtained from the parameters θt, learning rate η, and parameters ∈ from the previous time step.

[0138] S305, the semantic segmentation network is trained through a predetermined discussion to obtain a semantic segmentation network model.

[0139] The image of the infusion bottle or bag is first processed sequentially through steps S201-S204 to obtain the image pixel classification result. The image pixel classification result and the true image pixel result (the true result being the annotation result executed in S102) are then processed through steps S301-S304 to update the learnable parameters in the deep learning network model. After updating all learnable parameters, one round of training is complete. Generally, network training involves hundreds of iterations to allow the model to converge and obtain a deep learning classification network model with the minimum loss value.

[0140] In this embodiment, when the network has been trained for 70 epochs, the loss value tends to stabilize. At this point, the deep learning classification network model with the smallest loss value is saved as the final output of this training. The loss value change curve during the training process is shown below. Figure 11 As shown.

[0141] After obtaining the final semantic segmentation network model, S306 uses MIoU and MPA evaluation metrics to quantify and evaluate the model.

[0142] This specific embodiment uses the classic evaluation metrics MIoU and MPA in semantic segmentation to evaluate the performance of the semantic segmentation model on the infusion bottle and infusion bag dataset. Semantic segmentation is a classification task, and its prediction results typically fall into four categories: TP (true positive): the actual result is positive, and the prediction is also positive; FP (false positive): the actual result is negative, and the prediction is also positive; TN (true negative): the actual result is positive, and the prediction is also negative; FN (false negative): the actual result is negative, and the prediction is also negative. MIoU is the square of the intersection-union ratio for each class in the dataset. This ratio can be transformed into TP (intersection) divided by the sum of TP, FP, and FN (union). That is: MIoU = TP / (FP + FN + TP). Its calculation formula is shown in Equation 9:

[0143]

[0144] Where: let i represent the true class, j represent the predicted class, and p ij p represents the number of true class i predicted as class j. ii p represents the number of correct predictions. ji represents the number of true categories i in predicted category j, and k represents the total number of categories.

[0145] First, the PA for each class is calculated, and then the PA of all classes is averaged, i.e., MPA. The calculation formula is shown in Formula 10:

[0146]

[0147] Images of infusion bottles and bags from the test set were input into the semantic segmentation network model for testing, and the model was quantitatively evaluated using MIoU and MPA evaluation metrics. The MIoU and MPA evaluation metrics for this embodiment are shown in Table 1.

[0148] Table 1

[0149]

[0150] S105, input the image of the infusion bottle or bag to be segmented into the semantic segmentation network model for inference, and obtain the segmentation results of the infusion bottle or bag image with and without liquid surface, that is, obtain the intravenous infusion level detection result.

[0151] After obtaining the semantic segmentation network model, the image of the infusion bottle or bag to be detected is input into the semantic segmentation network model for inference. This yields segmentation results for the infusion bottle or bag image, indicating whether or not there is a liquid surface, thus obtaining the intravenous infusion level detection result. Figure 12 , Figure 13 .

[0152] The above is a detailed description of a semantic segmentation-based intravenous infusion level detection method provided in the embodiments of this application. The following will describe a semantic segmentation-based intravenous infusion level detection device provided in the embodiments of this application. Figure 14 As shown, when detecting intravenous infusion level in a certain scenario, the acquisition module S401 is used to acquire images of the infusion bottle and bag during patient infusion via a camera. Next, the annotation module S402 annotates the images of the infusion bottle and bag to obtain a dataset. Then, the training module S403 inputs the annotated images of the infusion bottle and bag from the dataset into a semantic segmentation network for training, resulting in a semantic segmentation network model. Finally, the inference module S404 inputs the real-time images of the infusion bag or bottle captured by the camera into the semantic segmentation network model to obtain semantic segmentation results indicating whether the infusion bag has a liquid level or not, thus obtaining the intravenous infusion level detection result.

[0153] The aforementioned semantic segmentation-based intravenous infusion level detection device is used to execute the semantic segmentation-based intravenous infusion level detection method, which will not be described in detail here.

[0154] The intravenous infusion level detection device provided in this application embodiment is as follows: Figure 15 As shown, the device includes: a central processing unit (CPU), a graphics card (GPU), a storage unit, a monitoring camera, and a communication interface. After the monitoring camera acquires the infusion image, it transmits the data to the embedded device via the communication interface. The embedded device reads the image of the infusion bottle or bag, loads the input image onto the GPU model, and performs forward inference calculations to obtain semantic segmentation results for whether the infusion bag has a liquid level or not, i.e., the intravenous infusion level detection result. The embedded device then displays the intravenous infusion level detection result on a display device in real time.

[0155] The contents not described in detail in this specification are existing technologies known to those skilled in the art.

Claims

1. A method for detecting the level of intravenous infusion based on semantic segmentation, characterized in that: The method comprises the following steps: The infusion bottle and infusion bag images are marked with liquid levels to obtain a data set; The data set is input into a semantic segmentation network; The semantic segmentation network comprises a feature extraction module, a deep feature extraction module, a feature fusion module, and a full connection module; The feature extraction module extracts image features in the data set to generate low-level feature maps and high-level feature maps; The high-level feature maps are input into the deep feature extraction module for feature extraction to obtain first feature maps; The low-level feature maps and the first feature maps are input into the feature fusion module for fusion to obtain fused feature maps; The fused feature maps are input into the full connection module, which classifies the pixel points on the fused feature maps to obtain semantic segmentation results of images in the data set with or without liquid surfaces; The semantic segmentation network is iteratively trained according to the liquid level marking of the images in the data set, the semantic segmentation results, and a target loss function to obtain a semantic segmentation network model; The images of the infusion bottle or infusion bag to be tested are input into the semantic segmentation network model, and the semantic segmentation network model outputs the intravenous infusion liquid level detection result. The feature extraction module comprises a convolution module, a convolution attention module, and a plurality of reverse residual modules; the feature extraction module comprises a one-stage module and a two-stage module; the one-stage module is composed of a convolution module, one convolution attention module, and three reverse residual modules connected in sequence, and the input of the one-stage module is the image of the infusion bottle or infusion bag; the image of the infusion bottle or infusion bag is subjected to feature extraction by the one-stage module to obtain a low-level feature map; the two-stage module is composed of fourteen reverse residual modules connected in sequence, and the input of the two-stage module is the low-level feature map; the low-level feature map is subjected to feature extraction by the reverse residual modules in the two-stage module in sequence to obtain a high-level feature map; The deep feature extraction module comprises a convolution module, three depth separable convolution modules, a global average pooling layer, a dimension splicing operation, a convolution layer, and an upsampling layer; the convolution module, the three depth separable convolution modules, and the global average pooling layer in the deep feature extraction module perform feature extraction on the high-level feature map to obtain five feature maps; after the five feature maps are subjected to a feature splicing operation in the dimension, the five feature maps are input into the convolution layer and the upsampling layer in sequence to obtain the first feature map.

2. The method of claim 1, wherein: The convolution module of the feature extraction module comprises a convolution layer, a batch normalization layer, and an activation layer in sequence; the input of the convolution module is taken as the input of the convolution layer for feature extraction; the convolution layer is used for extracting the same distribution feature matrix and inputting data features; the convolution operation on the image is actually sliding the convolution kernel on the image, multiplying the values on the image with the values on the corresponding convolution kernel, adding all the multiplied values as the output value of the position, and finally sliding through the entire image to obtain the result feature map after convolution calculation; the batch normalization layer performs batch normalization calculation on the output of the convolution layer, and the output of the batch normalization layer is taken as the input of the activation layer; The output of the batch normalization layer is taken as the input of the activation layer for linear activation calculation, and the output of the activation layer is taken as the output of the convolution module.

3. The method of claim 2, wherein: The convolution attention module of the feature extraction module comprises a channel attention structure and a spatial attention structure connected in sequence; the output of the convolution module is taken as the input of the convolution attention module, that is, as the input of the channel attention structure; the output result obtained by element-wise multiplication of the input and output of the channel attention structure is taken as the input of the spatial attention structure; and the output of the spatial attention structure is element-wise multiplied with the input of the spatial attention structure to obtain the output of the convolution attention module.

4. The method of claim 3, wherein: The inverted residual module of the feature extraction module comprises a convolution module, a depth separable convolution module and a convolution layer connected in sequence; when a feature map is input into the inverted residual module, the convolution module extracts features of the feature map and takes the feature extraction result as the input of the depth separable convolution module; the depth separable convolution module extracts features of the output of the convolution module and takes the result as the input of the convolution layer; and the convolution layer extracts features of the output of the depth separable convolution module and outputs the result of pixel-wise addition with the input of the inverted residual module.

5. The method of claim 1, wherein: The feature fusion module comprises a convolution module, a dimension splicing operation, a convolution layer and an upsampling layer; the convolution module in the feature fusion module reduces the dimension of a low-level feature map, and the low-level feature map after dimension reduction and the first feature map are subjected to a feature splicing operation in the dimension to be sequentially input into the convolution layer and the upsampling layer to obtain a fused feature map.

6. The method of claim 1, wherein: The full connection module comprises a full connection layer and a softmax activation function; the full connection layer classifies each pixel point in the fused feature map to obtain the class to which each pixel point belongs, and the class comprises a liquid portion and a non-liquid portion; and the output result of the full connection layer is input into the softmax function for probability mapping to obtain the probability value of the class to which each pixel point belongs.

7. The method of claim 6, wherein: The process of iteratively training the semantic segmentation network according to the liquid level label and the semantic segmentation result of the images in the data set and the target loss function comprises: using the loss function to calculate the loss value of the probability of the class to which each pixel point in the fused feature map belongs and the liquid level label true result; using the back propagation algorithm to calculate the loss value to update the parameters in the semantic segmentation network, and iteratively performing the above process for a predetermined number of times to obtain the semantic segmentation network model.

8. The method of claim 7, wherein: The process of using the back propagation algorithm to calculate the loss value to update the parameters in the semantic segmentation network comprises: using the loss back propagation algorithm to calculate the gradient of each parameter with respect to the loss value in the semantic segmentation network; and using an optimization strategy to optimize the semantic segmentation network according to the gradient and the optimization strategy, and using a gradient descent algorithm to update the parameters of the semantic segmentation network according to the gradient and the optimization strategy.

Images