An intelligent infusion state monitoring method and robot based on machine vision and multi-source data fusion

Abstract

The application discloses a kind of based on machine vision and multi-source data fusion infusion state intelligent monitoring method and robot, belong to medical infusion monitoring and mobile robot cross technical field.It includes: robot autonomous navigation to sickbed and adaptively adjusts camera posture and collects the RGB-D image sequence of infusion bottle;Using semantic segmentation and principal component analysis extracts medicine liquid area and bottle body area, accurately calculates liquid level remaining proportion by the projection height ratio along bottle body principal axis direction, and realizes inclination correction;In drip tube region of interest, dense optical flow algorithm and morphological filtering are used to identify effective droplet and calculate real-time drip speed;Liquid level remaining proportion and real-time drip speed are input into Mamdani fuzzy logic judge to make multi-source data fusion decision;According to alarm level, trigger hierarchical response including nurse station terminal, portable terminal and robot local sound and light alarm, and support multiple confirmation methods to form nursing closed loop.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary field of medical infusion monitoring and mobile robot technology, specifically involving an intelligent monitoring method for infusion status based on machine vision and multi-source data fusion, and a mobile robot that executes the method. Background Technology

[0002] Intravenous infusion is one of the most common methods of drug administration in clinical treatment. During the infusion process, the patient or caregiver needs to continuously monitor the remaining amount of medication and the drip rate. If the medication is about to run out or the drip rate becomes abnormal, medical staff must be notified immediately for handling; otherwise, it may lead to medical risks such as backflow of blood or air embolism.

[0003] Currently, infusion monitoring in wards mainly relies on the following methods: (1) Manual rounds: Nurses make regular rounds in the wards to visually observe the progress of the infusion. This method consumes a lot of nursing manpower and is prone to gaps in rounds at night or during busy periods, making it impossible to respond in real time.

[0004] (2) Fixed electronic monitoring instrument: An infrared beam sensor or a weighing sensor is installed on each infusion stand to determine the infusion status by detecting changes in the frequency or weight of the droplets. This type of solution requires a "one machine per bed" deployment, which has high hardware costs, complex wiring, and the sensors are easily affected by factors such as ambient light and infusion tube shaking, resulting in a high false alarm rate.

[0005] (3) Fixed visual monitoring scheme: For example, Chinese patent document CN120783273A discloses a computer vision-based infusion detection method and system, which uses an industrial camera installed next to the bed to photograph the infusion bottle and uses image processing technology to identify the liquid level and drip rate. However, this scheme is still a fixed deployment, which has the limitations of inconvenient installation and maintenance and the fact that a single device can only monitor a single bed. In addition, its liquid level detection algorithm does not take into account the working condition of the infusion bottle tilting during actual use, resulting in a large error in the liquid level calculation. Its alarm mechanism uses a single threshold trigger and lacks a comprehensive judgment of liquid level and drip rate, which can easily lead to problems such as "there is still a lot of medicine but it is not detected in time due to abnormal drip rate" or "frequent alarms cause nurse fatigue".

[0006] (4) Mobile robot inspection scheme: Existing technologies have proposed using mobile robots equipped with sensors for ward inspection, but existing schemes mostly use a combination of multiple sensors (such as vision + weight + infrared), which results in high system complexity and limited accuracy of the vision detection module, making it difficult to independently complete highly reliable infusion status monitoring.

[0007] In summary, the existing technologies have the following pressing technical problems: fixed monitoring equipment has high deployment costs and low utilization rates; visual solutions are poorly adaptable to infusion bottle tilting conditions and have insufficient accuracy in level detection; alarm decisions rely on a single threshold and lack comprehensive intelligent judgment of level and drip rate; and the monitoring results are not sufficiently linked with the hospital's existing information system, failing to form a complete nursing loop. Summary of the Invention

[0008] 1. Technical problem to be solved: This invention aims to provide an intelligent monitoring method and robot for infusion status based on machine vision and multi-source data fusion, solving the technical problems of infusion monitoring relying on manual labor, low accuracy of non-contact monitoring, and lack of system linkage alarm and predictive early warning capabilities in the prior art.

[0009] 2. Technical Solution: To solve the above problems, the present invention adopts the following technical solution.

[0010] In a first aspect, the present invention provides an intelligent monitoring method for infusion status based on machine vision and multi-source data fusion, applicable to autonomously navigable mobile robots. The method includes the following steps: Step S1: The mobile robot autonomously navigates to the monitoring point according to the preset ward patrol route; upon arrival, it adjusts the posture of the depth camera it carries through the mechanical gimbal so that the infusion bottle is in the center of the image and acquires an RGB-D image sequence containing the infusion bottle. Step S2: Use a semantic segmentation model to segment the first mask corresponding to the liquid area and the second mask corresponding to the bottle body area from each frame of the RGB-D image sequence; extract the principal axis direction of the bottle body area as the projection axis through principal component analysis; calculate the projection height of the liquid area and the total projection height of the bottle body area along the projection axis direction, and use the ratio of the two as the liquid level remaining ratio of the current frame; wherein, when the deviation angle between the principal axis direction and the vertical direction exceeds a preset threshold, tilt correction is performed, the principal axis direction is used as the projection axis for calculation, and the corrected liquid level remaining ratio is used as the output; Step S3: Within the dropper region of interest in the RGB-D image sequence, the motion vector of each pixel is calculated using a dense optical flow algorithm; pixel clusters with downward motion vectors whose vertical components are greater than a preset velocity threshold are selected as candidate droplets, and the candidate droplets are clustered and morphologically filtered to identify effective droplets; the number of effective droplets per unit time is counted, and the real-time drop rate is calculated. Step S4: Input the remaining liquid level ratio and the real-time drip rate as input variables to the pre-constructed Mamdani-type fuzzy logic judge; the fuzzy logic judge performs fuzzy reasoning based on the preset membership function and fuzzy rule base, and outputs the alarm level representing the infusion status after defuzzification processing; Step S5: Trigger a graded response action according to the alarm level: When the alarm level is Level 1, generate a warning message containing the estimated remaining time and push it to the nurse station terminal; when the alarm level is Level 2, generate an alarm message and push it to the nurse station terminal and at least one portable terminal simultaneously, and control the mobile robot to execute a local audible and visual alarm until a confirmation instruction from the nurse is received.

[0011] Furthermore, step S2, "extracting the principal axis direction of the bottle body region as the projection axis through principal component analysis," specifically includes: Obtain the minimum bounding rectangle of the second mask of the bottle body region, calculate the covariance matrix of the coordinates of all pixels within the rectangle, solve for the eigenvalues ​​and eigenvectors of the covariance matrix, and determine the direction of the eigenvector corresponding to the largest eigenvalue as the principal axis direction of the bottle body region.

[0012] Furthermore, step S3, "selecting pixel clusters with downward motion vectors as candidate droplets, and performing clustering and morphological filtering on the candidate droplets to identify valid droplets," specifically includes: The Farneback dense optical flow algorithm is used to calculate the motion vector of each pixel within the region of interest of the dropper; Filter out pixels whose vertical component is greater than a preset speed threshold; The selected pixels are clustered using a density-based clustering algorithm to form several pixel clusters; Pixel clusters with an area smaller than the first area threshold or an aspect ratio exceeding a preset range are removed, and the remaining pixel clusters are determined as valid droplets.

[0013] Further, in step S4, the fuzzy subset of the input variable "remaining liquid level ratio" of the fuzzy logic judge includes at least {high, medium, low, extremely low}, and the fuzzy subset of "real-time drip rate" includes at least {normal, slow, too slow, stopped}; the fuzzy rule base includes at least the following rules: If the remaining liquid level is "low" and the real-time drip rate is "normal", then output the first alarm level; If the remaining liquid level is "extremely low" or the real-time drip rate is "stopped", then output the second alarm level; If the remaining liquid level is "high" or "medium" and the real-time drip rate is "normal", the output indicates a normal level.

[0014] Furthermore, the calculation method for "estimated remaining time" in step S5 is as follows: When the real-time drip rate V When it is not zero, T left =( R × V total ) / ( V × k ),in, T left To estimate the remaining time, R This refers to the remaining percentage of the liquid level. V total The preset rated capacity of the infusion bottle, V The real-time drip rate, k The conversion coefficient between the dropping rate and the liquid level drop rate, obtained in advance through calibration experiments; When the real-time drip rate V When it is zero, the estimated remaining time is... T left Set to infinity or a preset alarm status flag.

[0015] Furthermore, the method of "receiving the nurse's confirmation processing instruction" in step S5 includes at least one of the following: Receive confirmation input on the interactive interface of the nurse station terminal; Receive confirmation input on the touch screen of the mobile robot; The mobile robot receives a preset confirmation password through its voice acquisition module, and after the voiceprint verification of the confirmation password is successful, it is confirmed that the confirmation processing instruction has been received.

[0016] Furthermore, the method further includes step S6: Each alarm event and its confirmation and handling event automatically generates a structured electronic nursing record. The electronic nursing record includes at least the patient identifier, bed number, alarm level, alarm time, remaining fluid level percentage, real-time drip rate, response time, and the identifier of the handling nurse. The electronic nursing records are synchronized to the hospital information system using a standardized medical data exchange protocol.

[0017] Furthermore, the method also includes a processing step for low-light nighttime modes: When the ambient light intensity is lower than the preset illuminance threshold, the method for segmenting the liquid region based on the depth map is switched to the method for segmenting the liquid region based on the depth map. The method for segmenting the liquid region based on the depth map in the RGB-D image sequence uses the depth value of the image center region as the seed point and a region growing algorithm is used to segment the bottle body region. The liquid region is extracted by comparing the depth difference between the foreground and the background of the bottle body region.

[0018] Secondly, the present invention provides a mobile robot for performing the intelligent monitoring method for infusion status described in the first aspect or any preferred embodiment thereof. The mobile robot includes: The mobile chassis and navigation module are used for planning and autonomous positioning of ward patrol routes; A depth camera and a mechanical gimbal, wherein the depth camera is used to acquire RGB-D image sequences, and the mechanical gimbal is used to adaptively adjust the shooting posture of the depth camera based on visual feedback; The embedded processing unit is equipped with a semantic segmentation model, an optical flow calculation module, and a fuzzy logic detector, which is used to process the acquired image sequence and output the alarm level. The wireless communication module is used to push warning or alarm information to the nurse station terminal and / or portable terminal in a hierarchical manner; The human-computer interaction module includes a voice broadcast unit, a touch screen, and a visual indication unit, which are used to output local alarm prompts and receive confirmation and processing instructions from nurses.

[0019] Furthermore, the mechanical gimbal's "adaptive adjustment of the depth camera's shooting posture based on visual feedback" specifically includes: Acquire a frame of RGB image and use an object detection network to detect the bounding box of the infusion bottle; Calculate the pixel deviation between the center point coordinates of the bounding box and the preset image center point coordinates; Based on the pixel deviation, a proportional control algorithm is used to generate control commands for the gimbal motor to adjust the yaw and pitch angles of the gimbal until the pixel deviation is less than a preset threshold.

[0020] 3. Beneficial effects: Compared with the prior art, the technical solution provided by this invention has the following advantages: (1) Reduce deployment costs and improve resource utilization: The mobile robot patrol monitoring architecture of "one robot for multiple beds" is adopted to replace the traditional fixed "one robot for one bed" deployment mode. A single robot can cover multiple ward beds through autonomous patrol, which significantly reduces hardware procurement and maintenance costs.

[0021] (2) Improve the accuracy of liquid level detection: By extracting the main axis direction of the bottle body through principal component analysis and performing projection calculation along the main axis direction, the tilting error caused by suspension or contact of the infusion bottle in actual use is effectively corrected, making the liquid level detection results more accurate and reliable.

[0022] (3) Improve the robustness of droplet velocity detection: The Farneback dense optical flow algorithm combined with DBSCAN clustering and morphological filtering can accurately identify droplets under complex lighting and background interference, and effectively suppress false detections caused by instantaneous noise through the continuous frame discrimination mechanism.

[0023] (4) Intelligent hierarchical alarm: The Mamdani fuzzy logic judge is introduced to integrate information from two dimensions, namely the remaining liquid level ratio and the real-time drip rate, to make a decision and output multiple alarm levels. This avoids the "false alarm" or "missed alarm" problems of traditional single threshold alarms and is more in line with actual clinical needs.

[0024] (5) Forming a complete nursing closed loop: alarm information is pushed to the nurse station terminal and portable terminal in a hierarchical manner, providing multiple confirmation and processing methods and recording the response time. At the same time, alarm and processing records are synchronized to the hospital HIS system through standardized protocols, realizing the whole process closed loop management from perception, decision-making, alarm, confirmation to recording.

[0025] (6) Adapt to complex lighting environment: Set a low light mode at night and automatically switch to the image segmentation method based on depth map when the ambient light is insufficient, so as to ensure that the monitoring system can work stably at all times of day and night.

[0026] It should be noted that the structures not described in this invention are not related to the design points and improvement directions of this invention, and are the same as or can be implemented using existing technologies, so they will not be elaborated here. Attached Figure Description

[0027] Figure 1 This is a schematic diagram of the overall process of the intelligent monitoring method for infusion status provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the hardware structure of the mobile robot in an embodiment of the present invention; Figure 3 This is a schematic diagram illustrating the principle of liquid level tilt correction in an embodiment of the present invention; Figure 4 This is a schematic diagram of the drop rate detection process based on dense optical flow and clustering in an embodiment of the present invention; Figure 5 This is a schematic diagram of the membership function curve of the fuzzy logic determiner in an embodiment of the present invention; Figure 6 This is a sequence diagram of information interaction between hierarchical alarm and confirmation processing in an embodiment of the present invention. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the scope of protection of the invention.

[0029] Example 1: Overall System Structure See Figure 2 The mobile robot provided in this embodiment includes: a mobile chassis, a depth camera, a mechanical gimbal, an embedded processing unit, a wireless communication module, a voice broadcasting module, a touch screen, and an LED breathing light strip.

[0030] The mobile chassis is a wheeled differential drive chassis, with built-in LiDAR, inertial measurement unit, and wheeled odometer. The robot achieves autonomous localization and path planning using a pre-built two-dimensional grid map of the ward and an adaptive Monte Carlo localization algorithm.

[0031] The depth camera uses an RGB-D depth camera (such as the Intel RealSense D435i), which can simultaneously acquire color images with a resolution of 1280×720 and corresponding depth maps at a frame rate of 30 frames per second.

[0032] The mechanical gimbal is a two-degree-of-freedom gimbal with a yaw angle range of ±45° and a pitch angle range of -30° to +30°. It is used to adaptively adjust the shooting posture of the depth camera based on visual feedback to ensure that the infusion bottle is centered in the field of view.

[0033] The embedded processing unit is an NVIDIA Jetson Orin NX edge computing platform, which is equipped with a lightweight semantic segmentation model, a Farneback optical flow computing module, and a Mamdani fuzzy logic detector. All visual algorithms run locally in real time without needing to be uploaded to the cloud.

[0034] The wireless communication module supports Wi-Fi and 4G / 5G communication, interacts with the nurse station terminal via the MQTT protocol, and pushes high-priority alarm notifications to the nurse's handheld terminal via Firebase Cloud Messaging.

[0035] Example 2: Overall Flowchart of Intelligent Monitoring Method for Infusion Status See Figure 1 The intelligent monitoring method for infusion status in this embodiment includes the following steps: Step S1: Robot Autonomous Navigation and Adaptive Image Acquisition The mobile robot stores a two-dimensional semantic grid map of each ward, with the coordinates of the monitoring point for each bed pre-marked on the map. The monitoring point is usually set at a distance of 0.5m from the edge of the bed and at a height of 1.4m to ensure that the depth camera can completely capture the area of ​​the infusion bottle and drip tube.

[0036] The robot proceeds sequentially to each monitoring point according to a pre-set patrol route. Upon reaching the target point, it initiates the adaptive alignment procedure of the mechanical gimbal. First, a depth camera captures a frame of RGB image, which is then input into a lightweight object detection network (SSD-MobileNet) to detect the bounding box of the infusion bottle and obtain the center coordinates of the bounding box. u box , v box ). Calculate the bounding box center and the image center ( u c , v c Pixel deviation: Δ u = u box - u c ; Δ v = v box - v c ; A proportional controller is used to convert pixel deviation into gimbal angular velocity control commands: ω yaw = Kp yaw × Δ u ; ω pitch = Kp pitch × Δ v ; in, Kp yaw and Kp pitch This is the proportional gain coefficient. In this embodiment, Kp yaw Take 0.05 rad / pixel. Kp pitch The value is set to 0.03 rad / pixel. This value has been experimentally calibrated to ensure rapid convergence of the gimbal without overshoot oscillation.

[0037] Iteratively execute the above detection-deviation calculation-attitude adjustment process until |Δ u| < 10 pixels and |Δ v |< 10 pixels, indicating that the infusion bottle is centered in the image, the gimbal is locked, and the depth camera begins to continuously acquire a sequence of 150 frames (approximately 5 seconds) of RGB-D images.

[0038] Step S2: Liquid level detection and tilt correction In this embodiment, the semantic segmentation model adopts the U-Net architecture, and the encoder backbone network is MobileNetV3-Large (pre-trained on ImageNet). The model is fine-tuned and trained on a dataset of 8500 infusion bottle images labeled with the drug liquid region and the bottle body region. The training loss function is a weighted sum of cross-entropy loss and Dice loss (weight ratio of 1:1), the optimizer is Adam, and the initial learning rate is 1×10⁻⁶. -3 , train 50 rounds.

[0039] For each frame of RGB image acquired, the semantic segmentation model outputs a mask for the liquid medicine region. M liquid Bottle area mask M bottle .

[0040] See Figure 3 To eliminate the impact of infusion bottle tilt on liquid level calculation, this embodiment employs a tilt correction method based on principal component analysis: (1) Obtain the mask of the bottle body area M bottle Find the smallest bounding rectangle, and calculate the coordinates of all pixels within that rectangle. x i , y i The covariance matrix of )

[0041] Solve for the eigenvalues ​​and eigenvectors of the covariance matrix, and determine the direction of the eigenvector corresponding to the largest eigenvalue as the principal axis direction of the bottle. Let the principal axis direction angle be denoted as... θ .

[0042] (2) Mask the liquid area M liquid Projecting along the principal axis θ yields a one-dimensional projection histogram. P ( t The projection formula is: t = x·cosθ + y·sinθ in( x , y () represents the pixel coordinates in the mask. t These are the projected coordinate values.

[0043] (3) Define the liquid level projection height in the projection histogram P(t). H liquid The length of a continuous non-zero interval along the principal axis from the projected position of the bottle bottom to the projected position of the highest point of the liquid; the total projected height of the bottle body is defined. H total It represents the total length of the interval from the projected position of the bottle bottom to the projected position of the bottle mouth.

[0044] (4) Calculate the remaining liquid level ratio R : R = ( H liquid / H total ) × 100%; When the principal axis direction angle θ If the deviation from the vertical direction exceeds 5°, the above tilt correction process is automatically activated; otherwise, vertical projection can be used directly to simplify the calculation. To suppress single-frame noise, the arithmetic mean of the calculation results of 5 consecutive frames is taken as the final output liquid level remaining ratio.

[0045] Step S3: Drop rate detection See Figure 4 The drip rate detection is performed within the region of interest (ROI) of the dropper. The ROI is obtained by empirically extracting a rectangular region containing the Murphy dropper by extending a preset pixel range downward from the bottle mouth region in the semantic segmentation model output.

[0046] Within the region of interest in the eyedropper algorithm, the Farneback dense optical flow algorithm is used to calculate the motion vector of each pixel between adjacent frames. The algorithm parameters are set as follows: number of pyramid layers 3, window size 15, and number of iterations 3.

[0047] Motion vector for each pixel ( u , v ), filter for pixels that meet the following conditions: v > 2.0 pixels / frame (vertical downward movement) Perform DBSCAN clustering on the selected pixels, with the parameter set to: neighborhood radius. ε = 5 pixels, minimum number of neighborhood points of the core point MinPts = 10.

[0048] For each pixel cluster obtained from clustering, perform morphological filtering: Remove clusters (noise) with an area smaller than 50 pixels; Clusters (non-droplet shapes) with a bounding box aspect ratio greater than 3:1 are removed.

[0049] The clusters retained after the above filtering are determined to be valid droplets. The number of valid droplets in the current frame is then counted. N frame .

[0050] The drip rate calculation uses the sliding time window method: setting the time window length. T = 10 seconds, cumulative total number of effective droplets in each frame within the window N total Real-time drip rate V The calculation formula is: V = ( N total / T ) × 60 (drops / minute) The sliding window updates every 1 second.

[0051] Furthermore, when no satisfying condition is detected for 30 consecutive frames (i.e., 1 second) v > 2.0 pixels per frame moving downwards, and the remaining percentage of the current liquid level R When the drip rate is >20%, it is determined that the drip rate is abnormally slow, triggering the slow drip rate warning mechanism. This information will be used as one of the inputs to the fuzzy logic judge.

[0052] Step S4: Multi-source data fusion judgment based on fuzzy logic The remaining liquid level ratio obtained in step S2 R and the real-time drip rate obtained in step S3 V As input variables, the input is a pre-built Mamdani-type fuzzy logic determiner.

[0053] (1) Fuzzification of input variables See Figure 5 Define input variables R The fuzzy subsets and membership functions are as follows: High: Trapezoidal membership function with parameters [60, 75, 100, 100]. Medium: The membership function for the triangle, with parameters [20, 50, 80]. Low: The membership function for the triangle, with parameters [5, 15, 30]. Very Low: Trapezoidal membership function with parameters [0, 0, 5, 12]. The fuzzy subsets and membership functions of the input variable V are defined as follows: Normal: Trapezoidal membership function with parameters [30, 40, 80, 80]. Slow: The membership function for the triangle, with parameters [15, 25, 35]. Very Slow: The membership function for triangles, with parameters [5, 10, 20]. Stop: Trapezoidal membership function with parameters [0, 0, 3, 8]. (2) Fuzzy rule base The fuzzy rule base constructed in this embodiment is shown in the table below:

[0054] Among them, alarm levels L The definition is as follows: 0 indicates a normal state, no alarm is needed; 1 indicates a level one warning, requiring nurses' attention; 2 indicates a level two alarm, requiring immediate action from nurses.

[0055] (3) Fuzzy reasoning and defuzzification The Mamdani inference method is used, taking the minimum membership degree of each rule's antecedent as the rule activation degree. The centroid method is then used to defuzzify the output fuzzy set to obtain continuous values. L out ∈ [0, 2]. The discrete values ​​that ultimately determine the alarm level. L Round to the nearest integer in the following range: L out < 0.5 → L = 0; 0.5 ≤ L out < 1.5 → L = 1; L out ≥ 1.5 → L = 2; Step S5: Hierarchical Alarm and Closed-Loop Confirmation According to alarm level L Trigger the corresponding action: when L When the value is 0, the robot does not trigger an alarm and continues to perform the next bed monitoring or patrol task.

[0056] when L When the fluid level is 1, a Level 1 warning message is generated. The message body is in JSON format and includes the bed number and the remaining fluid level percentage. R Real-time drip rate V and estimated remaining time T left The formula for calculating the estimated remaining time is: T left= ( R × V total ) / ( V × k ); in, V total The rated capacity of the infusion bottle (e.g., 250ml or 500ml, which can be obtained by scanning the barcode of the infusion bottle with a robot or by manual input) is specified. k This is the conversion coefficient between the drip rate and the rate of liquid level descent. The conversion coefficient k is calibrated by recording the complete infusion time from a full bottle to an empty bottle under standard experimental conditions, and then deducing the value of k based on the total volume, total time, and average drip rate. In this example, for a standard disposable infusion set (20 drops ≈ 1 ml). k The value is 20.

[0057] The Level 1 warning message is pushed to the nurse station terminal screen via the MQTT protocol. At the same time, the mobile robot touch screen displays a yellow warning bar, providing information but not triggering an audible alarm.

[0058] when L When the value is 2, a level 2 alarm message is generated. In addition to being pushed to the nurse station terminal, a high-priority notification is also sent to the responsible nurse's handheld terminal via Firebase Cloud Messaging through the 4G / 5G module. The mobile robot immediately activates the local alarm: it broadcasts the voice message "The infusion in bed X is about to end, please handle it promptly" in a loop at 80 decibels, and the LED breathing light strip lights up in a red gradient flashing mode.

[0059] After the alarm is triggered, the system enters a confirmation waiting state. Nurses can confirm the confirmation using any of the following methods: Click the "Confirm Processing" button for the corresponding patient bed on the interactive interface of the nurse station terminal; Tap the "Processed" button on the mobile robot's touchscreen display; When a pre-set confirmation command (such as "IV infusion has been processed") is spoken to the robot, the system extracts the Mel-frequency cepstral coefficient features of the speech and compares them with a pre-registered nurse voiceprint template. If the verification passes, confirmation is granted. Specifically, the voiceprint verification process includes: pre-collecting speech samples from authorized nurses to establish a voiceprint model library; upon receiving the confirmation command, extracting the voiceprint feature vector of the command and performing similarity matching with models in the voiceprint model library; when the similarity exceeds a pre-set verification threshold, verification is deemed successful, and the confirmation processing command takes effect.

[0060] After confirmation, the robot immediately stops broadcasting voice messages and flashing its breathing light, records the response time of the alarm (the time interval from the alarm being issued to the confirmation), and automatically resumes its patrol mission to move on to the next monitoring point.

[0061] Step S6: Electronic nursing record generation and synchronization with HIS Each alarm event (regardless) L =1 or L =2) and its confirmation processing event automatically generate a structured electronic nursing record. The record fields include at least: patient unique identifier, bed number, alarm level, alarm timestamp, remaining fluid level percentage, real-time drip rate, nurse confirmation timestamp, response time, and processing nurse identifier.

[0062] The electronic nursing record is packaged in the HL7 / FHIR standard format and synchronized to the nursing record module of the hospital's HIS system through the hospital's intranet gateway. It is archived as an official nursing document and does not require nurses to manually enter it afterward.

[0063] Example 3: Low-light mode at night Considering that the main lights in the ward may be turned off at night, leaving only the floor lights, resulting in low ambient light levels that could affect the accuracy of RGB image segmentation, this embodiment also includes a low-light night mode.

[0064] The mobile robot is equipped with a light sensor to detect ambient light levels in real time. When the light level falls below a preset threshold (50 lux in this embodiment), the system automatically switches to a depth map-based method for segmenting the drug liquid region. (1) Extract depth map from RGB-D image sequence. Since the infusion bottle is suspended on the infusion stand, its depth value is usually in a continuous range, which is significantly different from the background wall or curtain.

[0065] (2) Depth value at the center point of the image (depth cameras are usually aimed at the center of the infusion bottle). d seed As a seed point, a depth tolerance threshold is set. δ = 15mm, and a region continuous with the depth of the seed point is grown in the depth map using the region growing algorithm. This region is the bottle body region mask.

[0066] (3) Further extract the liquid area within the bottle body area. Due to the depth difference caused by air gaps between the liquid and the back wall of the bottle or by the refraction of the liquid itself, the liquid surface position can be determined by detecting the abrupt boundary of the depth value within the bottle body area. Specifically, scan the depth value profile within the bottle body area in the vertical direction. When the depth difference between adjacent pixels exceeds 10 mm, the position is determined to be the liquid surface boundary line. The area above the boundary line is the air area (or empty bottle area), and the area below is the liquid area.

[0067] (4) Based on the extracted liquid area and bottle area, the remaining liquid level ratio is calculated using the projection height method in step S2. R .

[0068] The drop rate detection section still relies on optical flow calculation based on RGB images in night mode. Since the dropper area has independent infrared supplementary light (provided by the depth camera), the reliability of optical flow calculation can be guaranteed.

[0069] The above description is merely a preferred embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural modifications made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A method for intelligent monitoring of infusion status based on machine vision and multi-source data fusion, characterized in that, The method, applied to autonomously navigable mobile robots, includes: Step S1: The mobile robot autonomously navigates to the monitoring point according to the preset ward patrol route; upon arrival, it adjusts the posture of the depth camera it carries through the mechanical gimbal so that the infusion bottle is in the center of the image and acquires an RGB-D image sequence containing the infusion bottle. Step S2: Use a semantic segmentation model to segment the first mask corresponding to the liquid area and the second mask corresponding to the bottle body area from each frame of the RGB-D image sequence; extract the principal axis direction of the bottle body area as the projection axis through principal component analysis; calculate the projection height of the liquid area and the total projection height of the bottle body area along the projection axis direction, and use the ratio of the two as the liquid level remaining ratio of the current frame; wherein, when the deviation angle between the principal axis direction and the vertical direction exceeds a preset threshold, tilt correction is performed, the principal axis direction is used as the projection axis for calculation, and the corrected liquid level remaining ratio is used as the output; Step S3: Within the dropper region of interest in the RGB-D image sequence, the motion vector of each pixel is calculated using a dense optical flow algorithm; pixel clusters with downward motion vectors are selected as candidate droplets, and clustering and morphological filtering are performed on the candidate droplets to identify effective droplets; the number of effective droplets per unit time is counted, and the real-time drop rate is calculated. Step S4: Input the remaining liquid level ratio and the real-time drip rate as input variables to the pre-constructed Mamdani-type fuzzy logic judge; the fuzzy logic judge performs fuzzy reasoning based on the preset membership function and fuzzy rule base, and outputs the alarm level representing the infusion status after defuzzification processing; Step S5: Trigger a graded response action according to the alarm level: When the alarm level is Level 1, generate a warning message containing the estimated remaining time and push it to the nurse station terminal; when the alarm level is Level 2, generate an alarm message and push it to the nurse station terminal and at least one portable terminal simultaneously, and control the mobile robot to execute a local audible and visual alarm until a confirmation instruction from the nurse is received.

2. The method according to claim 1, characterized in that, The step S2, "extracting the principal axis direction of the bottle body region as the projection axis through principal component analysis," specifically includes: Obtain the minimum bounding rectangle of the second mask of the bottle body region, calculate the covariance matrix of the coordinates of all pixels within the rectangle, solve for the eigenvalues ​​and eigenvectors of the covariance matrix, and determine the direction of the eigenvector corresponding to the largest eigenvalue as the principal axis direction of the bottle body region.

3. The method according to claim 1, characterized in that, Step S3, "selecting pixel clusters with downward motion vectors as candidate droplets, and performing clustering and morphological filtering on the candidate droplets to identify valid droplets," specifically includes: The Farneback dense optical flow algorithm is used to calculate the motion vector of each pixel within the region of interest of the dropper; Filter out pixels whose vertical component is greater than a preset speed threshold; The selected pixels are clustered using a density-based clustering algorithm to form several pixel clusters; Pixel clusters with an area smaller than the first area threshold or an aspect ratio exceeding a preset range are removed, and the remaining pixel clusters are determined as valid droplets.

4. The method according to claim 1, characterized in that, In step S4, the fuzzy subset of the input variable "remaining liquid level ratio" of the fuzzy logic judge includes at least {high, medium, low, extremely low}, and the fuzzy subset of "real-time drip rate" includes at least {normal, slow, too slow, stopped}; the fuzzy rule base includes at least the following rules: If the remaining liquid level is "low" and the real-time drip rate is "normal", then output the first alarm level; If the remaining liquid level is "extremely low" or the real-time drip rate is "stopped", then output the second alarm level; If the remaining liquid level is "high" or "medium" and the real-time drip rate is "normal", the output indicates the normal level.

5. The method according to claim 1, characterized in that, The calculation method for "estimated remaining time" in step S5 is as follows: When the real-time drip rate V When it is not zero, T left =( R × V total ) / ( V × k ),in, T left To estimate the remaining time, R This refers to the remaining percentage of the liquid level. V total The preset rated capacity of the infusion bottle, V The real-time drip rate, k The conversion coefficient between the dropping rate and the liquid level drop rate, obtained in advance through calibration experiments; When the real-time drip rate V When it is zero, the estimated remaining time is... T left Set to infinity or a preset alarm status flag.

6. The method according to claim 1, characterized in that, The method of "receiving the nurse's confirmation processing instruction" in step S5 includes at least one of the following: Receive confirmation input on the interactive interface of the nurse station terminal; Receive confirmation input on the touch screen of the mobile robot; The mobile robot receives a preset confirmation password through its voice acquisition module, and after the voiceprint verification of the confirmation password is successful, it is confirmed that the confirmation processing instruction has been received.

7. The method according to claim 1, characterized in that, It also includes step S6: Each alarm event and its confirmation and handling event automatically generates a structured electronic nursing record. The electronic nursing record includes at least the patient identifier, bed number, alarm level, alarm time, remaining fluid level percentage, real-time drip rate, response time, and the identifier of the handling nurse. The electronic nursing records are synchronized to the hospital information system using a standardized medical data exchange protocol.

8. The method according to any one of claims 1 to 7, characterized in that, It also includes processing steps for low-light nighttime modes: When the ambient light intensity is lower than the preset illuminance threshold, the method of drug liquid region segmentation based on depth map is switched. The depth map-based liquid region segmentation method includes: using the depth map in the RGB-D image sequence, taking the depth value of the image center region as the seed point, and using a region growing algorithm to segment the bottle body region; and extracting the liquid region by comparing the depth difference between the foreground and background of the bottle body region.

9. A mobile robot, characterized in that, The mobile robot, configured to perform the intelligent monitoring method for infusion status as described in any one of claims 1 to 8, comprises: The mobile chassis and navigation module are used for planning and autonomous positioning of ward patrol routes; A depth camera and a mechanical gimbal, wherein the depth camera is used to acquire RGB-D image sequences, and the mechanical gimbal is used to adaptively adjust the shooting posture of the depth camera based on visual feedback; An embedded processing unit, on which a semantic segmentation model, an optical flow calculation module and a fuzzy logic detector are deployed, is used to process the acquired image sequence and output the alarm level; The wireless communication module is used to push warning or alarm information to the nurse station terminal and / or portable terminal in a hierarchical manner; The human-computer interaction module includes a voice broadcast unit, a touch screen, and a visual indication unit, which are used to output local alarm prompts and receive confirmation and processing instructions from nurses.

10. The mobile robot according to claim 9, characterized in that, The mechanical gimbal is used to adaptively adjust the shooting posture of the depth camera based on visual feedback, and its specific configuration includes performing the following steps: Acquire a frame of RGB image and use an object detection network to detect the bounding box of the infusion bottle; Calculate the pixel deviation between the center point coordinates of the bounding box and the preset image center point coordinates; Based on the pixel deviation, a proportional control algorithm is used to generate control commands for the gimbal motor to adjust the yaw and pitch angles of the gimbal until the pixel deviation is less than a preset threshold.

Images