Drug counting method based on gesture recognition
By installing a camera on the rescue vehicle to monitor gestures and combining the Mask R-CNN model to detect drugs, the problem of unstable detection in the rescue vehicle's item management system is solved, and efficient and accurate identification of drug counts is achieved.
Patent Information
- Application Number
- CN202510524403.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-24
- Publication Date
- 2025-08-08
AI Technical Summary
The existing rescue vehicle item management system is unstable under light changes and hand occlusion, resulting in misjudgment of the quantity of items, making it difficult to meet the requirements of accuracy and real-time.
The drug counting method based on gesture recognition is adopted, and the hand characteristics in the drawer are monitored through the camera, the gesture type is identified and the number of drugs is detected in combination with the Mask R-CNN model to achieve closed-loop verification.
It improves the accuracy and efficiency of drug counting, reduces the impact of misjudgment and occlusion interference, and meets the accuracy and real-time requirements of rescue vehicle item management.
Smart Images
Figure CN120452060A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of medical equipment, and in particular to a medicine counting method based on gesture recognition. Background Art
[0002] In medical emergency scenarios, emergency vehicles are crucial for storing emergency medicines and equipment. They typically feature a multi-layered frame structure, with retractable drawers on each layer for storing medical supplies. Real-time monitoring and management of the contents within these drawers is crucial for improving emergency response efficiency. Currently, some emergency vehicles utilize camera-based visual recognition technology, which uses image acquisition and analysis to identify the type and quantity of items within drawers, enabling dynamic monitoring of item usage. However, existing camera-based object recognition technology has the following problems in practical applications: 1. Inaccurate item detection: Due to the compact placement of items in the drawer or changes in lighting conditions, the camera may experience unstable detection frames and target position jumps when identifying items, causing the system to misjudge the number or location of items, affecting management accuracy. 2. Hand occlusion: When medical staff reach for items in a drawer, their hands block the camera's field of view, preventing the system from fully capturing the items. In this case, the algorithm may misidentify the location or number of other items due to missing local features or background interference, causing abnormal jitter in all detection targets and further reducing system reliability. These problems make it difficult for existing technologies to meet the accuracy and real-time requirements of rescue vehicle item management. There is an urgent need for an identification solution that is resistant to occlusion and has higher stability to optimize system performance. Summary of the Invention
[0003] The technical problem to be solved by the present invention is that the existing technology is difficult to meet the accuracy and real-time requirements of emergency vehicle item management. In order to solve the above problem, a medicine counting method based on gesture recognition is provided.
[0004] The object of the present invention is achieved in the following manner: A medicine counting method based on gesture recognition, wherein a camera is installed above the drawer of the emergency vehicle, and the camera is used to capture the image inside the opened drawer; the medicine counting method includes: S1: Real-time monitoring of whether hand features appear in the drawer area after it is opened; S2: If hand features appear, capture a video frame image and obtain an image with complete hand features from the video frame image; S3: Gesture recognition for images with complete hand features; S4: Determine the gesture type by querying the gesture mapping table: If it is a digital gesture, the corresponding value will be recorded and the accumulation mode will be entered; If the gesture is activated in the accumulation mode, multiple digital gestures can be input continuously for accumulation; If it is a confirmation gesture, step S5 is triggered to start counting medicines; If it is a clear or cancel gesture, the current counting result will be reset or rolled back; S5: If medicine counting is required, obtain the image after the hand feature disappears, identify the number of medicines in the image, compare the gesture input value with the actual number of medicines, and complete the closed-loop verification.
[0005] The gesture recognition includes a training phase, specifically: Establish a gesture dataset, annotate gesture labels, enhance the data including rotation, translation, and scaling, select a CNN deep network model, and input the data into the CNN deep network for training; input the monitored images with complete hand features into the trained CNN deep network to output the gesture recognition results.
[0006] The gesture recognition includes the inference stage: specifically, establishing a gesture dictionary: collecting samples of gestures commonly used by medical staff and labeling gesture categories; Construct a hierarchical dictionary structure: the first level is classified into "number", "function" and "special", and the second level is the specific gesture label; Static gesture mapping table: stores the correspondence between gestures and operations in the form of key-value pairs; Cumulative counting mode: After activation, the system accumulates the continuously input digital gesture values until a confirmation or reset command is received; during the accumulation process, the temporary counting result is displayed in real time and can be canceled midway; Special gesture handling: Clear gesture: clear the current counting pool and displayed results, and reset the system to the initial state; Pause gesture: freeze the current counting process and retain the counting pool data until the gesture is input again; Exception handling: If an undefined gesture is detected, a voice prompt "Invalid gesture, please try again" is triggered.
[0007] The gesture mapping table includes static gesture and number mapping, functional gestures and special gestures; wherein, the static gesture and number mapping includes: five fingers spread out to represent the number 5, four fingers spread out and the thumb bent to represent the number 4, three fingers spread out to represent the number 3, two fingers spread out to represent the number 2, and one finger extended to represent the number 1; the functional gestures include: spreading both hands flat to confirm the count, putting five fingers together and swinging them quickly to clear the count, and making a fist and releasing it to undo the last operation; the special gestures include: extending the little finger and thumb to enter the cumulative counting mode, and crossing the fingers to pause the counting.
[0008] The S5 specifically includes the following steps: Hand feature disappearance detection: After the gesture is confirmed, the system continues to monitor the drawer area until the hand completely moves out of the camera's field of view; Preprocessing of pharmaceutical images: De-noising and light equalization processing are performed on the images after the hands disappear to eliminate shadow interference; Drug object detection and segmentation: Use the Mask R-CNN model to locate the drug bounding box and extract the individual drug outlines through instance segmentation; Quantity statistics and verification: Count the number of independent targets after segmentation and filter out false detections by combining with the drug size database; If the difference between the detection quantity and the gesture input value exceeds the threshold, an alarm is triggered and manual review is required.
[0009] Beneficial effects of the present invention: The present invention can improve the efficiency and accuracy of drug counting and identification. BRIEF DESCRIPTION OF THE DRAWINGS
[0010] Figure 1 It is a workflow diagram of the present invention. DETAILED DESCRIPTION
[0011] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.
[0012] It should be noted that the following detailed description is illustrative and is intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same technical meanings as those commonly understood by those skilled in the art to which the present application belongs.
[0013] A medicine counting method based on gesture recognition, wherein a camera is installed above the drawer of the emergency vehicle, and the camera is used to capture the image inside the opened drawer; the medicine counting method includes: S1: Real-time monitoring of whether hand features appear in the drawer area after it is opened; S2: If hand features appear, capture a video frame image and obtain an image with complete hand features from the video frame image; S3: Gesture recognition for images with complete hand features; S4: Determine the gesture type by querying the gesture mapping table: If it is a numeric gesture (such as 1-5), the corresponding value is recorded and the accumulation mode is entered (if not activated, single counting is used); If the gesture is activated in the accumulation mode, multiple digital gestures can be input continuously for accumulation; If it is a confirmation gesture, step S5 is triggered to start counting medicines; If it is a clear or cancel gesture, the current counting result will be reset or rolled back; S5: If medicine counting is required, obtain the image after the hand feature disappears, identify the number of medicines in the image, compare the gesture input value with the actual number of medicines, and complete the closed-loop verification.
[0014] The S5 specifically includes the following steps: Hand feature disappearance detection: After the gesture is confirmed (step S4), the system continues to monitor the drawer area until the hand completely moves out of the camera's field of view (using the optical flow method to detect the disappearance of movement); Preprocessing of pharmaceutical images: De-noising and light equalization processing are performed on the images after the hands disappear to eliminate shadow interference; Drug object detection and segmentation: Use the Mask R-CNN model to locate the drug bounding box and extract the individual drug outlines through instance segmentation; Quantity counting and verification: Count the number of independent targets after segmentation, and filter out false detections (such as packaging fragments) by combining a drug size database (such as standard drug box sizes); If the difference between the detected number and the gesture input value exceeds a threshold (such as ±1), an alarm is triggered and manual review is required.
[0015] The gesture recognition includes a training phase, specifically: Establish a gesture dataset, annotate gesture labels, enhance the data including rotation, translation, and scaling, select a CNN deep network model, and input the data into the CNN deep network for training; input the monitored images with complete hand features into the trained CNN deep network to output the gesture recognition results.
[0016] The gesture recognition includes the inference stage: specifically, establishing a gesture dictionary: collecting samples of gestures commonly used by medical staff and labeling gesture categories; Construct a hierarchical dictionary structure: the first level is classified into "number", "function" and "special", and the second level is the specific gesture label; Static gesture mapping table: stores the correspondence between gestures and operations in the form of key-value pairs; Cumulative counting mode: After activation, the system accumulates the continuously input digital gesture values until a confirmation or reset command is received; during the accumulation process, the temporary counting result is displayed in real time and can be canceled midway; Special gesture handling: Clear gesture: clear the current counting pool and displayed results, and reset the system to the initial state; Pause gesture: freeze the current counting process and retain the counting pool data until the gesture is input again; Exception handling: If an undefined gesture is detected, a voice prompt "Invalid gesture, please try again" is triggered.
[0017] The gesture mapping table includes static gesture and number mapping, functional gestures and special gestures; wherein, the static gesture and number mapping includes: five fingers spread out to represent the number 5, four fingers spread out and the thumb bent to represent the number 4, three fingers spread out to represent the number 3, two fingers spread out to represent the number 2, and one finger extended to represent the number 1; the functional gestures include: spreading both hands flat to confirm the count, putting five fingers together and swinging them quickly to clear the count, and making a fist and releasing it to undo the last operation; the special gestures include: extending the little finger and thumb to enter the cumulative counting mode, and crossing the fingers to pause the counting.
[0018] The present invention can improve the efficiency and accuracy of medicine counting and identification.
[0019] The above is only a preferred embodiment of the present invention. It should be pointed out that for those skilled in the art, several changes and improvements can be made without departing from the overall concept of the present invention, and these should also be regarded as the scope of protection of the present invention.
Claims
1. A medicine counting method based on gesture recognition, characterized by: A camera is installed above the drawer of the rescue vehicle, and the camera is used to capture the image inside the opened drawer; the medicine counting method includes: S1: Real-time monitoring of whether hand features appear in the drawer area after it is opened; S2: If hand features appear, capture a video frame image and obtain an image with complete hand features from the video frame image; S3: Gesture recognition for images with complete hand features; S4: Determine the gesture type by querying the gesture mapping table: If it is a digital gesture, the corresponding value will be recorded and the accumulation mode will be entered; If the gesture is activated in the accumulation mode, multiple digital gestures can be input continuously for accumulation; If it is a confirmation gesture, step S5 is triggered to start counting medicines; If it is a clear or cancel gesture, the current counting result will be reset or rolled back; S5: If medicine counting is required, obtain the image after the hand feature disappears, identify the number of medicines in the image, compare the gesture input value with the actual number of medicines, and complete the closed-loop verification.
2. The drug counting method based on gesture recognition according to claim 1, characterized in that: The gesture recognition includes a training phase, specifically: Establish a gesture dataset, annotate gesture labels, enhance the data including rotation, translation, and scaling, select a CNN deep network model, and input the data into the CNN deep network for training; input the monitored images with complete hand features into the trained CNN deep network to output the gesture recognition results.
3. The drug counting method based on gesture recognition according to claim 1, characterized in that: The gesture recognition includes the inference stage: specifically, establishing a gesture dictionary: collecting samples of gestures commonly used by medical staff and labeling gesture categories; Construct a hierarchical dictionary structure: the first level is classified into "number", "function", and "special", and the second level is the specific gesture label; Static gesture mapping table: stores the correspondence between gestures and operations in the form of key-value pairs; Cumulative counting mode: After activation, the system accumulates the continuously input digital gesture values until a confirmation or reset command is received; during the accumulation process, the temporary counting result is displayed in real time and can be canceled midway; Special gesture handling: Clear gesture: clear the current counting pool and displayed results, and reset the system to the initial state; Pause gesture: freeze the current counting process and retain the counting pool data until the gesture is input again; Exception handling: If an undefined gesture is detected, a voice prompt "Invalid gesture, please try again" is triggered.
4. The drug counting method based on gesture recognition according to claim 1, characterized in that: The gesture mapping table includes static gesture and number mapping, functional gestures and special gestures; wherein, the static gesture and number mapping includes: five fingers spread out to represent the number 5, four fingers spread out and the thumb bent to represent the number 4, three fingers spread out to represent the number 3, two fingers spread out to represent the number 2, and one finger extended to represent the number 1; the functional gestures include: spreading both hands flat to confirm the count, putting five fingers together and swinging them quickly to clear the count, and making a fist and releasing it to undo the last operation; the special gestures include: extending the little finger and thumb to enter the cumulative counting mode, and crossing the fingers to pause the counting.
5. The drug counting method based on gesture recognition according to claim 1, characterized in that: The S5 specifically includes the following steps: Hand feature disappearance detection: After the gesture is confirmed, the system continues to monitor the drawer area until the hand completely moves out of the camera's field of view; Preprocessing of pharmaceutical images: De-noising and light equalization processing are performed on the images after the hands disappear to eliminate shadow interference; Drug object detection and segmentation: Use the Mask R-CNN model to locate the drug bounding box and extract the individual drug outlines through instance segmentation; Quantity statistics and verification: Count the number of independent targets after segmentation and filter out false detections by combining with the drug size database; If the difference between the detection quantity and the gesture input value exceeds the threshold, an alarm is triggered and manual review is required.