Automatic liquid change reminding method and device based on neural network and medium

Abstract

The invention discloses an automatic liquid change reminding method and device based on a neural network and a medium, and the method comprises the steps: employing a thermal imaging technology to shoot infusion bottle pictures in different scenes in real time, and carrying out the centralized preprocessing of all shot pictures; constructing a hybrid neural network, performing feature extraction on the preprocessed image set, and performing feature enhancement processing on the extracted image; the extracted infusion bottle images are classified, and whether the liquid in the current infusion bottle needs to be replaced or not is predicted according to the classification result; and according to a prediction result, collecting a data set, and performing classification to form a training set and a test set to verify the output accuracy of the model. Pictures are provided in real time through the thermal imaging system, the intelligent computing system can accurately predict whether the infusion bottle needs to be replaced or not in real time through the deep learning algorithm LeViT and remind related medical staff, and the automation level and safety of medical care are improved.

Description

Technical Field

[0001] The present invention relates to the field of image detection technology, and in particular to a method, device and medium for automatically reminding liquid replacement based on a neural network. Background Art

[0002] Existing medical resources are already quite tight, especially when encountering epidemic diseases, a large number of patients flock to the hospital and urgently need infusion treatment. At this time, one medical staff often looks after ten or more patients at the same time. Each patient should receive the same care, but even a seemingly simple fluid change can make medical staff exhausted: for example, the time for medical staff to frequently check the remaining amount of infusion bags prevents them from focusing on other nursing work. For example, medical staff who manually monitor the infusion process are easily affected by factors such as fatigue and distraction, which may lead to missing the best time for fluid change. For example, timely replacement of infusion bags is crucial for the treatment and safety of patients, and timely fluid change avoids blood reflux or other complications caused by empty infusion tubes. Summary of the invention

[0003] In view of this, the present invention provides a method, device and medium for automatically reminding liquid replacement based on a neural network to solve the above problems.

[0004] In order to solve the above technical problems, the present invention provides an automatic reminder method for changing liquid based on a neural network, comprising: Take pictures of infusion bottles in different scenes, and perform centralized preprocessing on all the pictures after classification; Construct a hybrid neural network to extract features from the preprocessed image set and perform feature enhancement on the extracted images; Classify the extracted infusion bottle images, and based on the classification results, predict whether the liquid in the current infusion bottle needs to be replaced; According to the prediction results, the data set is collected and classified into training set and test set to verify the output accuracy of the model.

[0005] As an optional method, centralized preprocessing of all captured images includes: screening and repairing images of liquids in infusion bottles; The screening includes: screening all liquid surface images of infusion bottles of different sizes and positions in accordance with the national medicine package standards in different scenes, different times and different capacities, to build image richness, and remove images of infusion bottles in which the liquid surface is completely blocked, and images in which the scales on the infusion bottles are extremely unclear due to severe wear and tear during use or obstruction by impurities; Fix for resizing and color normalizing images.

[0006] As an optional method, filter the liquid level pictures in infusion bottles of different sizes and positions, including: The images of infusion bottles in different scenes are classified based on the color pixel values ​​of the liquid in the bottles to form a first liquid data set and a second liquid data set. The image translation is performed on the part of the first liquid data set where the pixel value exceeds a preset threshold, and the part is converted into an image that meets the characteristics of the second liquid data set, which includes: The unpaired first liquid dataset and the second liquid dataset are used as input parameters, the first liquid dataset is converted into images of the second liquid dataset through a generator based on contrastive unpaired translation, and the conversion process is stimulated by a standard adversarial loss; wherein, When the image of the first liquid data set is converted into the paired image of the second liquid data set by the generator, the discriminator is used to constrain the converted reconstructed image to maintain the same color block level features as its original image; Afterwards, the part of the first liquid data set whose anchor pixel value exceeds the preset threshold is segmented as the contour association feature of the first liquid data set and the second liquid data set; based on the contour association feature, a contour association feature data set is constructed, and the above translation is trained through binary cross entropy loss; the anchor pixel value is the pixel value within the area of ​​the infusion bottle in the image.

[0007] As an optional method, repair is used to unify the size and color of the image, as follows: S1. Crop the image according to a preset size, align the view center with the geometric center of the infusion bottle, remove other irrelevant areas including the background, and only keep the infusion bottle area; S2, adjusting the image contrast, using adaptive histogram equalization to perform histogram equalization on the contrast, increasing the image contrast of the infusion bottle after removing irrelevant areas, so that the contrast between the liquid surface and the infusion bottle is more obvious; S3, adjusting the image brightness to make the liquid surface in the infusion bottle clearer, and discarding the image in which the liquid surface of the infusion bottle is still unclear after the brightness is adjusted; S4, sharpening the image, enhancing the high-frequency components of the infusion bottle liquid surface image through a high-pass filter, thereby enhancing the image details and making the liquid surface contour in the infusion bottle clearer; S5, image denoising, using median filtering to remove image noise and improve image quality; S6. Perform image restoration on images where the liquid surface of the infusion bottle is blocked by pipelines and / or trademarks.

[0008] As an optional method, in the above S6, the image repairing for the image where the pipeline and / or the trademark obstructs the liquid surface of the infusion bottle includes the following method: The Sobel operator is used to process the image in the horizontal and vertical directions, and the image brightness gradient is approximately calculated to obtain a clearer liquid level image edge of the liquid surface image; By scanning the accumulated number of edge points of each liquid surface position information in the horizontal direction, it is determined whether each liquid surface position information is a temporary qualified line. The qualified standard of the qualified line is: the accumulated number of edge points reaches more than one tenth of the entire line; If only one qualified line is greater than the preset value, the liquid level position information represented by the qualified line is determined to be the position of the current liquid level; if multiple qualified lines are greater than the preset value, the qualified lines are sorted according to their lengths, and the longest qualified line is determined first; The liquid level position represented by the longest qualified line is compared with the liquid level of the previous frame. If there is no position change, the qualified line is determined not to be the liquid level position, and the position not being the liquid level and the liquid level length are recorded; if there is a position change, it is determined to be the liquid level position and the liquid level length is recorded; wherein, useless horizontal edges are excluded by setting a horizontal projection line segment length threshold.

[0009] As an optional method, image restoration also includes: The integral projection method is used to detect the liquid level of the infusion bottle. The image is projected in both horizontal and vertical directions, and the liquid level of the image is located by horizontal projection. If the liquid surface in the image is not flat due to interference, the liquid level height is corrected by taking the pixel length of each column of the liquid level component as the liquid level height of the column, and then selecting the average value of the maximum liquid level height value as the pixel height of the liquid level; The spectral clustering algorithm is used to perform grayscale segmentation on the bottle body in the image and construct a similarity matrix. The matrix is ​​decomposed to extract its eigenvectors. The number of clusters is selected according to the eigenvectors corresponding to the preset number of largest eigenvalues, which are used as the representation of data points in low-dimensional space. Then, the DBSCAN algorithm is used to cluster these data points. In the obtained clustering results, the category with the lowest grayscale value is identified, and the maximum grayscale value in the category is selected as the threshold of the binary processing; the image is converted into a binary image according to the determined threshold, and the pixels in the binary image above the threshold are regarded as liquid parts, and the pixels below the threshold are regarded as background or other non-liquid parts and are eliminated; Finally, the average value of the maximum liquid level height value is selected as the pixel coordinate of the liquid level and determined as the liquid pixel height.

[0010] As an optional method, in the process of collecting images of the liquid level in the infusion bottle, it is necessary to manually mark the actual liquid volume corresponding to the liquid level in the infusion bottle and whether the liquid needs to be changed, and select a large number of mixed infusion bottles of different sizes that meet the national drug package standards, with liquid levels in different states, infusion bottles with different liquid colors, and infusion bottles with slight differences in color themselves, and the images of the liquid volumes that have been manually marked are input into the model.

[0011] As an optional method, a hybrid neural network is constructed to extract features from the preprocessed image set and perform feature enhancement on the extracted images, including: Convolutional layers, pooling layers, and fully connected layers are set based on the LeViT deep learning algorithm. The convolutional layer is used to extract local features in the images of all infusion bottles that meet the national medicine package standards. The pooling layer is used to reduce the dimension of the output of the convolutional layer and output the main features of the liquid in the infusion bottle. The fully connected layer is used to classify the output of the pooling layer and predict whether the liquid level status of the infusion bottle needs to be classified.

[0012] As an optional method, the extracted infusion bottle image is classified, and based on the classification result, it is predicted whether the liquid in the current infusion bottle needs to be replaced, including: A classifier is used to classify the extracted infusion bottle image features and predict whether the liquid in the current infusion bottle needs to be replaced. Softmax is used as the classifier, and neurons are randomly discarded in the process to avoid fitting the infusion bottle image set with the generated model parameters.

[0013] On the other hand, the present invention provides an electronic device, as an optional method, comprising: a memory for storing a computer program; a processor for implementing the steps of the above-mentioned neural network-based automatic fluid replacement reminder method when executing the computer program.

[0014] On the other hand, the present invention provides a computer-readable storage medium. As an optional method, the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned neural network-based automatic fluid replacement reminder method are implemented.

[0015] The beneficial effects of the present invention are: The invention provides pictures in real time through the thermal imaging system, and the intelligent computing system can accurately and in real time predict whether the infusion bottle needs to be replaced and remind relevant medical staff through the deep learning algorithm, thereby improving the automation level and safety of medical care. It can realize real-time and accurate monitoring of the liquid volume in the infusion bottle and automatically remind the replacement of the liquid, improve the quality of care, and reduce medical accidents caused by insufficient liquid. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A schematic flow chart of the automatic fluid replacement reminder method provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0017] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is further described in detail below in conjunction with specific implementation methods.

[0018] Existing medical resources are already quite tight, especially when encountering epidemic diseases, a large number of patients flock to the hospital and urgently need infusion treatment. At this time, one medical staff often looks after ten or more patients at the same time. Each patient should receive the same care, but even a seemingly simple fluid change can make medical staff exhausted: for example, the time for medical staff to frequently check the remaining amount of infusion bags prevents them from focusing on other nursing work. For example, medical staff who manually monitor the infusion process are easily affected by factors such as fatigue and distraction, which may lead to missing the best time for fluid change. For example, timely replacement of infusion bags is crucial for the treatment and safety of patients, and timely fluid change avoids blood reflux or other complications caused by empty infusion tubes.

[0019] In order to improve nursing efficiency and reduce the risk of human errors causing patients to suffer "additional pain" and "injury to the injured", timely fluid replacement ensures patient safety, optimizes medical staff resources, allows them to better plan and allocate their time, and ensures timely processing when the infusion bag is exhausted, thereby optimizing the use of medical resources, reducing the inconvenience and anxiety of patients waiting for fluid replacement, improving patients' medical experience and satisfaction, and automatically recording fluid replacement time, so that medical staff can track the patient's treatment progress, and also provide data support for medical quality control and research. This embodiment adopts mature deep learning algorithms: LeViT (LightweightVision Transformer) is used to extract image features and deep learning, the cross entropy algorithm is used for back propagation to solve model parameters, and the classification algorithm softmax predicts whether the liquid in the infusion bottle needs to be replaced, and timely pushes patients who need fluid replacement to medical staff.

[0020] See also Figure 1 This embodiment provides a method for automatically reminding liquid replacement based on a neural network, comprising: Thermal imaging technology is used to capture images of infusion bottles in different scenes in real time, and all the captured images are centrally preprocessed; a hybrid neural network is constructed to extract features from the preprocessed image set, and feature enhancement is performed on the extracted images; the extracted infusion bottle images are classified, and based on the classification results, it is predicted whether the liquid in the current infusion bottle needs to be replaced; based on the prediction results, the data set is collected and classified into training sets and test sets to verify the output accuracy of the model.

[0021] After the above solution completes the recognition of the infusion bottle image, in an optional solution, this embodiment provides a system for automatically reminding to change the liquid, which can be a thermal imaging system and an intelligent computing system. The thermal imaging system is used to monitor the status of the liquid in the infusion bottle: The infrared thermal imager monitors the real-time status of the liquid in the infusion bottle in the hospital ward and generates thermal imaging images that meet the requirements in real time. The intelligent computing system can automatically analyze whether the current infusion bottle needs to be replaced through the liquid status in the infusion bottle provided by the thermal imaging system and the trained image classification model. If the liquid needs to be replaced, the bed information of the patient who needs to be replaced will be automatically pushed to the medical staff to remind the medical staff to replace the liquid in time.

[0022] In this embodiment, when constructing a classification model for whether fluid replacement is needed, three steps are included, namely data extraction, data governance, and construction of a hybrid neural network LeViT deep learning algorithm for classification prediction.

[0023] Before extraction, thermal imaging technology is used to take pictures of infusion bottles in different scenes in real time, and all the pictures taken are centrally preprocessed. The screening includes: screening the liquid surface pictures in infusion bottles of different scenes, different times and different shapes to construct the picture richness.

[0024] The screening includes: screening different scenes, different times and different capacities, all of which meet the national medicine package standards: "YBB00032005-2015 Soda-lime Glass Infusion Bottle", "YBB00012004-2015 Low-borosilicate Glass Infusion Bottle", "YBB00022005-2-2015 Medium-borosilicate Glass Infusion Bottle" in different specifications and sizes in different positions, for constructing image richness; and eliminating images in which the liquid surface is completely blocked in the collected infusion bottle images, and images in which the scales on the infusion bottle are extremely unclear due to severe wear during use or obstruction by impurities; repair is used to unify the size and color of the images.

[0025] Since different patients need to be injected with different liquids, this embodiment takes into account that there are injection liquids other than conventional transparent body fluid colors, such as levofloxacin hydrochloride injection in light yellow, Qingkailing in brownish red, nutrient solution in milky white, vitamin B12 injection in rose red clear solution, etc., which may make the boundary between the liquid surface and the container wall blurred or difficult to distinguish when identifying the liquid surface due to its absorption and reflection characteristics of light. Compared with transparent liquids that allow light to pass directly through, such colored liquids are more difficult to clearly show the boundary between the liquid surface and the container wall. Therefore, this embodiment is based on the idea of ​​contrast non-paired translation, converting colored liquid images into colorless transparent liquid images to overcome the color problem and uniformly identify the infusion bottle image. This embodiment is implemented as follows: The images of infusion bottles in different scenes are classified based on the color pixel values ​​of the liquid in the bottles to form a first liquid data set with color and a second liquid data set without color. Liquid color pixels are searched in the first liquid data set, and the image translation is performed on the part of the first liquid data set where the pixel value exceeds a preset threshold, and the part is converted into an image that meets the characteristics of the second liquid data set, which includes: The unpaired first liquid data set and the second liquid data set are used as input parameters, and by comparing unpaired translations, the first liquid data set is converted into an image of the second liquid data set by a generator in a manner that converts an image in the source domain into an image in the target domain, and the conversion process is stimulated by a standard adversarial loss. Because this embodiment performs conversion through a generator in the conversion of the preceding step, and adversarial loss is a core concept used in a generative adversarial network, which involves a generator and a discriminator. The goal of the generator is to generate data that is as close to the real data as possible, while the goal of the discriminator is to distinguish between the generated data and the real data. The adversarial loss promotes the generator to produce high-quality data through this competition mechanism. Additional training data can also be generated when the amount of data is small. This makes the liquid color conversion learning of this embodiment more efficient.

[0026] Therefore, when the image of the first liquid data set is converted into the paired image of the second liquid data set through the generator, the discriminator is used to constrain the converted reconstructed image to maintain the same color block level features as its original image.

[0027] Afterwards, the part of the first liquid data set where the anchor pixel value exceeds the preset threshold is segmented as the contour association feature of the first liquid data set and the second liquid data set. The anchor pixel value is the pixel value within the area of ​​the infusion bottle in the image to avoid confusion of liquid surface edge pixels caused by dispersion, scattering, etc. Based on the contour association feature, a contour association feature data set is constructed, and the above translation is trained through binary cross entropy loss to complete the conversion from colored liquid to colorless liquid for the next step of inspection.

[0028] Considering that 1. the image size is inconsistent, for example, the infusion bottles in some images are distorted for some reason, which will increase the difficulty of convergence in the training process or make the entire training process non-convergent; 2. Since the images of the infusion bottles come from thermal imagers of different models or manufacturers, the colors of the infusion bottles have color differences, that is, color inconsistencies. Too much color difference will also cause the training process to be too long or non-convergent. Therefore, it is necessary to repair the images, for example, uniformly adjust the images of the infusion bottles to a uniform size of 224*224, and perform color normalization to normalize the colors to between [0,1]. Specifically, when screening, the liquid surface images of different scenes include indoor scenes, outdoor scenes, scenes of handheld infusion bottles, and scenes of infusion bottles suspended on brackets; infusion bottles of different shapes include bottled infusion bottles and bagged infusion bottles; wherein, the screening also includes: eliminating the images of the collected infusion bottle images where the liquid surface is completely blocked.

[0029] To this end, this embodiment is achieved as follows: S1. Crop the image according to a preset size, and remove other irrelevant areas including the background by aligning the view center with the geometric center of the infusion bottle, and only retain the infusion bottle area; for example, the background, the infusion line, etc.

[0030] S2, adjusting the image contrast, using adaptive histogram equalization to perform histogram equalization on the contrast, increasing the image contrast of the infusion bottle after removing irrelevant areas, so that the contrast between the liquid surface and the infusion bottle is more obvious; S3, adjusting the image brightness to make the liquid surface in the infusion bottle clearer, and discarding the image in which the liquid surface of the infusion bottle is still unclear after the brightness is adjusted; S4, sharpening the image, enhancing the high-frequency components of the infusion bottle liquid surface image through a high-pass filter, thereby enhancing the image details and making the liquid surface contour in the infusion bottle clearer; S5, image denoising, using median filtering to remove image noise and improve image quality; S6. Perform image restoration on images where the liquid surface of the infusion bottle is blocked by pipelines and / or trademarks.

[0031] In the above S6, the image repairing for the image where the liquid surface of the infusion bottle is blocked by the pipeline and / or the trademark includes the following method: The Sobel operator is used to process the image in the horizontal and vertical directions, and the image brightness gradient is approximately calculated to obtain a clearer liquid level image edge of the liquid surface image; by scanning the cumulative number of edge points of each liquid surface position information in the horizontal direction, it is determined whether each liquid surface position information is a temporary qualified line, and the qualified standard of the qualified line is: the cumulative number of edge points reaches more than one tenth of the entire row; if only one qualified line is greater than the preset value, the liquid surface position information represented by the qualified line is determined to be the position of the current liquid level; if multiple qualified lines are greater than the preset value, the qualified lines are sorted according to the length, and the longest qualified line is determined first; the liquid surface position represented by the longest qualified line is compared with the liquid level of the previous frame image. If there is no position change, it is determined that the qualified line is not the position of the liquid level, and the position and liquid surface length that are not the liquid level are recorded; if there is a position change, it is determined to be the position of the liquid level and the liquid surface length is recorded; wherein, since the horizontal edge of the infusion bottle trademark may interfere with the liquid level detection, this embodiment excludes useless horizontal edges by setting a horizontal projection line segment length threshold.

[0032] Afterwards, the liquid level of the infusion bottle is detected by the integral projection method, and the image is projected in both horizontal and vertical directions, and the liquid level of the image is located by the horizontal projection method; if the liquid surface in the image is not flat due to interference, the liquid level height is corrected by taking the pixel length of each column of the liquid level component as the liquid level height of the column, and then selecting the average value of the maximum value of the liquid level height as the pixel height of the liquid level; the grayscale segmentation of the bottle body part in the image is performed by the spectral clustering algorithm to construct a similarity matrix; the matrix is ​​feature decomposed to extract its eigenvectors, and the number of clusters is selected according to the eigenvectors corresponding to the largest eigenvalues ​​of the preset number, which are used as the representation of data points in low-dimensional space, and then the DBSCAN algorithm is used to cluster these data points; in the obtained clustering results, the category with the lowest grayscale value is identified, and the largest grayscale value is selected in the category as the threshold for binarization processing; the image is converted into a binary image according to the determined threshold, and the pixels in the binary image above the threshold are regarded as liquid parts, and the pixels below the threshold are regarded as background or other non-liquid parts, which are eliminated. Finally, the average value of the maximum liquid level height value is selected as the pixel coordinate of the liquid level and determined as the liquid pixel height, which can reduce the influence of light.

[0033] When inputting parameters for the screening model, the liquid surface images of the different scenes include indoor scenes, outdoor scenes, scenes of hand-held infusion bottles and scenes of infusion bottles suspended on brackets. Infusion bottles in different positions are tilted at 10 degrees and 45 degrees, for example; the infusion bottles of different shapes include bottled infusion bottles, all of which comply with the national drug package standards: "YBB00032005-2015 Soda-lime Glass Infusion Bottle", "YBB00012004-2015 Low-borosilicate Glass Infusion Bottle", "YBB00022005-2-2015 Medium-borosilicate Glass Infusion Bottle" and infusion bottles of different sizes. In addition, considering that the liquid level in the infusion bottle under natural conditions can present many states, in the process of collecting the liquid level image in the infusion bottle, it is necessary to manually mark the actual liquid volume corresponding to the liquid level in the infusion bottle and whether the liquid needs to be changed. Different medicines will cause differences in machine learning results due to slight differences in color. Taking these factors into consideration, a large number of mixed infusion bottles of different sizes that meet the national medicine package standards, with liquid levels in different states, infusion bottles with different liquid colors, and infusion bottles with slight differences in color themselves, and the liquid volume images that have been manually annotated are selected and input into the model.

[0034] When performing feature extraction, in this embodiment, a hybrid neural network LeViT is selected for feature extraction. LeViT consists of multiple convolutional layers, pooling layers, and fully connected layers. During training, the images of infusion bottles in multiple scenes (the infusion bottle may be outdoors, indoors, or handheld, etc.) are taken into consideration. In this embodiment, 13 convolutional layers and pooling layers are used to extract the image features of the infusion bottle in more detail, and 3 fully connected layers are used to output the main features of the infusion bottle image. Among them, the convolutional layer is responsible for extracting local features in the infusion bottle image: image features of the liquid surface of the liquid in the infusion bottle being tilted, image features when the liquid surface of the liquid in the infusion bottle is partially blocked, image features when the text on the infusion bottle is on the liquid surface, and image features of different transparency of the infusion bottle due to different materials such as soda-lime glass infusion bottles, low borosilicate glass infusion bottles, medium borosilicate glass infusion bottles, etc. The pooling layer is responsible for reducing the dimension of the output of the convolutional layer, that is, outputting the main features of the liquid in the infusion bottle and ignoring some non-main features such as when the liquid in the infusion bottle is partially blocked. The fully connected layer is responsible for classifying the output of the pooling layer, that is, predicting whether the liquid level in the infusion bottle needs to be replaced.

[0035] In addition, this embodiment uses the neural network LeViT to extract features from the preprocessed image, and adopts the Relu activation function to increase the nonlinear ability of the neural network. That is, in the deep learning process, there will be images of infusion bottles in a certain epoch. For example, many images of infusion bottles at the same angle in the same scene will cause the subsequently generated model to have a linear relationship with the data set, resulting in the model being not universal. The Relu activation function is used to randomly activate neurons to solve these problems.

[0036] When classifying, this embodiment uses a classifier to classify the extracted infusion bottle image features, and predicts whether the liquid in the current infusion bottle needs to be replaced, that is, the extraction of the above-mentioned infusion bottle features is calculated and divided into two categories: 1. Need to replace the liquid, that is, 1; 2. No need to replace the liquid, that is, 0. This classifier uses softmax, and LeViT constructs a deep hybrid neural network by using a series of small-size convolution kernels and pooling layers of size 3x3, and achieves good results. In the classification process, it may be found that the infusion bottle image sets of several epochs are too similar, resulting in the infusion bottle image set fitting the generated model parameters. The generated model parameters will be unusable in other scenarios. In order to solve this problem, in the training process, this embodiment uses dropout, that is, randomly discarding neurons. In the training process, the cross-entropy loss function (cross-entropy) is used in the LeViT model parameter process to evaluate the loss of the deep learning process and to back-propagate the model parameters.

[0037] And, in order to verify the effectiveness of the present invention, this embodiment carried out the following experiments: Dataset: This embodiment collects 60,000 pictures of liquid in infusion bottles taken in real time by a thermal imaging camera, of which 50,000 pictures are training sets and 10,000 pictures are test sets.

[0038] Model: This embodiment uses a neural network model LeViT, which consists of 13 convolutional layers and pooling layers and 3 fully connected layers.

[0039] Training: In this example, the LeViT model is trained on 50,000 images in the training set for 10,000 epochs.

[0040] Test: This example tests the LeViT model on 10,000 images in the test set, and the accuracy of the model reaches 98.8%.

[0041] The experimental results show that the LeViT-based infusion bottle liquid volume prediction method of the present invention can effectively predict whether the liquid in the current infusion bottle needs to be replaced.

[0042] In this way, after completing the image recognition, the model parameters of the deep learning algorithm LeViT network are generated by using TPU training in the training set, and the intelligent computing system loads the model parameters. In the production environment, only a general-configuration CPU is needed to complete the prediction of the real-time image of the liquid in the infusion bottle transmitted by the thermal imaging system. If the liquid needs to be replaced, the patient's bed can be pushed to the relevant medical staff. This can reduce the physical and mental fatigue of medical staff when a large number of patients need infusion, thereby maximizing the care received by patients.

[0043] Through the above scheme, this embodiment improves nursing efficiency, reduces the "additional pain" and "injury" caused by human errors, ensures patient safety through timely fluid replacement, optimizes medical staff resources, allows them to better plan and allocate their time, and ensures timely processing when the infusion bag is exhausted, thereby optimizing the use of medical resources, reducing the inconvenience and anxiety caused by waiting for fluid replacement, and improving patients' medical experience and satisfaction. It can also automatically record the fluid replacement time, which is convenient for medical staff to track the patient's treatment progress, and also provides data support for medical quality control and research.

[0044] On the other hand, this embodiment also provides an electronic device, which can be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.) or other types of mobile devices, or a stationary computing device such as a desktop computer or a PC. The electronic device can also be a mobile or stationary server.

[0045] As an optional manner, the method comprises: a memory for storing a computer program; and a processor for executing the computer program to realize: Use thermal imaging technology to take real-time pictures of infusion bottles in different scenes, and centrally pre-process all the pictures taken; Construct a hybrid neural network to extract features from the preprocessed image set and perform feature enhancement on the extracted images; Classify the extracted infusion bottle images, and based on the classification results, predict whether the liquid in the current infusion bottle needs to be replaced; According to the prediction results, the data set is collected and classified into training set and test set to verify the output accuracy of the model.

[0046] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the automatic reminder method for changing liquids based on a neural network are of the same concept. For details not described in detail in the technical scheme of the computing device, please refer to the description of the technical scheme of the automatic reminder method for changing liquids based on a neural network.

[0047] On the other hand, this embodiment further provides a computer-readable storage medium. As an optional manner, the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following is implemented: Use thermal imaging technology to take real-time pictures of infusion bottles in different scenes, and centrally pre-process all the pictures taken; Construct a hybrid neural network to extract features from the preprocessed image set and perform feature enhancement on the extracted images; Classify the extracted infusion bottle images, and based on the classification results, predict whether the liquid in the current infusion bottle needs to be replaced; According to the prediction results, the data set is collected and classified into training set and test set to verify the output accuracy of the model.

[0048] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the automatic reminder method for liquid replacement based on a neural network belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be found in the description of the technical scheme of the automatic reminder method for liquid replacement based on a neural network.

[0049] An embodiment of the present application further provides a chip storing a computer program, which, when executed by the chip, implements the steps of the neural network-based automatic fluid replacement reminder method.

[0050] The above describes specific embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0051] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0052] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described action sequence, because according to the present application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0053] The above are only preferred embodiments of the present invention. It should be noted that the above preferred embodiments should not be regarded as limiting the present invention, and the protection scope of the present invention should be based on the scope defined by the claims. For ordinary technicians in this technical field, several improvements and modifications can be made without departing from the spirit and scope of the present invention, and these improvements and modifications should also be regarded as the protection scope of the present invention.

Claims

1. A method for automatically reminding liquid replacement based on a neural network, characterized in that: include: Use thermal imaging technology to take pictures of infusion bottles in different scenes, and then pre-process all the pictures after classification; Construct a hybrid neural network to extract features from the preprocessed image set and perform feature enhancement on the extracted images; Classify the extracted infusion bottle images, and based on the classification results, predict whether the liquid in the current infusion bottle needs to be replaced; According to the prediction results, the data set is collected and classified into training sets and test sets to verify the output accuracy of the model.

2. The method for automatically reminding liquid replacement based on a neural network according to claim 1, characterized in that: The centralized preprocessing of all the photographed images includes: screening and repairing the images of the liquid in the infusion bottle; The screening includes: screening all liquid surface images of infusion bottles of different sizes and positions in accordance with the national medicine package standards in different scenes, different times and different capacities, for constructing image richness, and eliminating images of infusion bottles in which the liquid surface is completely blocked, and images in which the scales on the infusion bottles are extremely unclear due to severe wear during use or blocked by impurities; The repair is used to unify the size and normalize the color of the image.

3. The method for automatically reminding liquid replacement based on a neural network according to claim 2, characterized in that: Filter the liquid level pictures in infusion bottles of different sizes and positions, including: The images of infusion bottles in different scenes are classified based on the color pixel values ​​of the liquid in the bottles to form a first liquid data set and a second liquid data set, and the image translation is performed on the part of the first liquid data set where the pixel value exceeds a preset threshold to convert it into an image that meets the characteristics of the second liquid data set, which includes: The first liquid dataset and the second liquid dataset without pairing are used as input parameters, the first liquid dataset is converted into images of the second liquid dataset through a generator based on contrast unpaired translation, and the conversion process is stimulated by a standard adversarial loss; wherein, When the image of the first liquid data set is converted into a paired image of the second liquid data set by the generator, a constraint is performed by the discriminator so that the converted reconstructed image maintains the same color block level features as its original image; Afterwards, the part of the first liquid data set whose anchor pixel value exceeds the preset threshold is segmented as the contour association feature of the first liquid data set and the second liquid data set; based on the contour association feature, a contour association feature data set is constructed, the above translation process is trained through binary cross entropy loss, and all the first liquid data sets are converted into the second liquid data set; the anchor pixel value is the pixel value within the area of ​​the infusion bottle in the image.

4. The method for automatically reminding liquid replacement based on a neural network according to claim 2, characterized in that: The repair is used to unify the size and normalize the color of the image, and is performed as follows: S1. Crop the image according to a preset size, align the view center with the geometric center of the infusion bottle, remove other irrelevant areas including the background, and only keep the infusion bottle area; S2, adjusting the image contrast, using adaptive histogram equalization to perform histogram equalization on the contrast, increasing the image contrast of the infusion bottle after removing irrelevant areas, so that the contrast between the liquid surface and the infusion bottle is more obvious; S3, adjusting the image brightness to make the liquid surface in the infusion bottle clearer, and discarding the image in which the liquid surface of the infusion bottle is still unclear after the brightness is adjusted; S4, sharpening the image, enhancing the high-frequency components of the infusion bottle liquid surface image through a high-pass filter, thereby enhancing the image details and making the liquid surface contour in the infusion bottle clearer; S5, image denoising, using median filtering to remove image noise and improve image quality; S6. Perform image restoration on images where the liquid surface of the infusion bottle is blocked by pipelines and / or trademarks.

5. The method for automatically reminding liquid replacement based on a neural network according to claim 4, characterized in that: In the above S6, the image repairing for the image where the liquid surface of the infusion bottle is blocked by the pipeline and / or the trademark includes the following method: The Sobel operator is used to process the image in the horizontal and vertical directions, and the image brightness gradient is approximately calculated to obtain a clearer liquid level image edge of the liquid surface image; By scanning the accumulated number of edge points of each liquid level position information in the horizontal direction, it is determined whether each liquid level position information is a temporary qualified line, and the qualified standard of the qualified line is: the accumulated number of edge points reaches more than one tenth of the entire line; If only one qualified line is greater than the preset value, the liquid level position information represented by the qualified line is determined to be the position of the current liquid level; if multiple qualified lines are greater than the preset value, the qualified lines are sorted according to their lengths, and the longest qualified line is determined first; The liquid level position represented by the longest qualified line is compared with the liquid level of the previous frame. If there is no position change, the qualified line is determined not to be the liquid level position, and the position not being the liquid level and the liquid level length are recorded; if there is a position change, it is determined to be the liquid level position and the liquid level length is recorded; wherein, useless horizontal edges are excluded by setting a horizontal projection line segment length threshold.

6. The method for automatically reminding liquid replacement based on a neural network according to claim 5, characterized in that: The image restoration further comprises: The integral projection method is used to detect the liquid level of the infusion bottle. The image is projected from both horizontal and vertical directions, and the liquid level of the image is located by horizontal projection. If the liquid surface in the image is not flat due to interference, the liquid level height is corrected by taking the pixel length of each column of the liquid level component as the liquid level height of the column, and then selecting the average value of the maximum liquid level height value as the pixel height of the liquid level; The spectral clustering algorithm is used to perform grayscale segmentation on the bottle body in the image to construct a similarity matrix. The similarity matrix is ​​subjected to eigendecomposition to extract its eigenvectors. The number of clusters is selected according to the eigenvectors corresponding to the preset number of largest eigenvalues, which are used as the representation of data points in low-dimensional space. The DBSCAN algorithm is then used to cluster these data points. In the obtained clustering results, the category with the lowest grayscale value is identified, and the maximum grayscale value in the category is selected as the threshold of the binarization process; the image is converted into a binary image according to the determined threshold, and the pixels in the binary image above the threshold are regarded as liquid parts, and the pixels below the threshold are regarded as background or other non-liquid parts, and are eliminated; Finally, the average value of the maximum liquid level height value is selected as the pixel coordinate of the liquid level and determined as the liquid pixel height.

7. The method for automatically reminding liquid replacement based on a neural network according to claim 1, characterized in that: In the process of collecting images of the liquid surface in the infusion bottle, it is necessary to manually mark the actual liquid volume corresponding to the liquid surface of the infusion bottle and whether the liquid needs to be changed, and select a large number of mixed infusion bottles of different sizes that meet the national drug package standards, with liquid surfaces in different states, infusion bottles with different liquid colors, and infusion bottles with slight differences in color themselves, and the images of the liquid volumes that have been manually marked are input into the model.

8. The method for automatically reminding liquid replacement based on a neural network according to claim 1, characterized in that: The hybrid neural network is constructed to extract features from the preprocessed image set and perform feature enhancement processing on the extracted images, including: A convolution layer, a pooling layer and a fully connected layer are set based on the LeViT deep learning algorithm; the convolution layer is used to extract local features in the images of all infusion bottles that meet the national medicine package standards, and the pooling layer is used to reduce the dimension of the output of the convolution layer and output the main features of the liquid in the infusion bottle; the fully connected layer is used to classify the output of the pooling layer and predict whether the liquid level state of the infusion bottle needs to be classified.

9. The method for automatically reminding liquid replacement based on a neural network according to claim 1, characterized in that: The step of classifying the extracted infusion bottle image and predicting whether the liquid in the current infusion bottle needs to be replaced according to the classification result includes: A classifier is used to classify the extracted infusion bottle image features and predict whether the liquid in the current infusion bottle needs to be replaced. Softmax is used as the classifier, and neurons are randomly discarded in the process to avoid fitting the infusion bottle image set with the generated model parameters.

10. An electronic device, characterized in that: include: Memory for storing computer programs; A processor is used to implement the steps of the automatic fluid replacement reminder method based on a neural network as described in any one of claims 1 to 9 when executing the computer program.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the neural network-based automatic fluid replacement reminder method as described in any one of claims 1 to 9 are implemented.

Images