Rescue carriage management system and method based on visual model

Abstract

The invention discloses a rescue carriage management system and method based on a visual model, and the system comprises a main control unit, and an edge calculation unit connected with the main control unit. The main control unit is further connected with a man-machine interaction unit, a voice recognition module and an electromagnetic control unit used for controlling the electromagnetic lock to be powered on, wherein the man-machine interaction unit, the voice recognition module and the electromagnetic control unit are arranged on the working table top. The edge calculation unit is connected with a first camera installed above the multiple layers of drawers, a second camera installed on the working table top and a third camera installed on the working table top. And the edge computing unit is also connected with the network unit and is connected with the background service terminal through the network unit. According to the invention, the expiration date label of the medicine or the article is directly scanned through the camera and is automatically identified and recorded into the system, so that the manual input step is omitted and the accuracy and efficiency are improved. Medical staff only need to put medicines or articles into the drawer or directly take and use the medicines or articles according to a conventional process, and the visual identification system can automatically complete material registration or cancel after verification.

Description

Technical Field

[0001] The present invention relates to the technical field of medical equipment, and in particular to a rescue vehicle management system and a management method based on a visual model. Background Art

[0002] As a crucial piece of equipment in hospital emergency situations, the management of medicines and supplies within a rescue vehicle is directly related to the patient's life safety. For example, the rescue vehicle disclosed in Chinese Patent Publication No. CN115487024A comprises a mobile cabinet with multiple drawers on the front for storing items or medications. Traditional rescue vehicle medication and supply management relies primarily on manual verification and data entry, which presents the following issues: Manual data entry is inefficient and prone to errors: Existing systems usually require medical staff to manually select drug or item information for expiration date management. This is not only cumbersome to operate, but also prone to data entry errors due to human negligence, affecting rescue efficiency.

[0003] Reliance on additional identification such as radio frequency tags (RFID): Some hospitals use RFID tags for medication management. However, this method requires pre-attaching the tags to medications or items and scanning them each time they are restocked or used. This increases the number of steps and can change medical staff's habits, leading to resistance. The additional tag scanning required for emergency access or replenishment can affect rescue efficiency. Furthermore, RFID tags can be subject to wear, detachment, or signal interference, further reducing management reliability. Inadequate expiration date management: Manual inspection of drug expiration dates is prone to omissions, and expired drugs may not be discovered in time, posing a safety hazard. Summary of the Invention

[0004] The technical problem to be solved by the present invention is that the management of medicines and items in traditional emergency vehicles mainly relies on manual verification and manual entry. In order to solve the above problem, a emergency vehicle management system and management method based on a visual model are provided.

[0005] The object of the present invention is achieved in the following manner: A rescue vehicle management system based on a visual model, wherein the rescue vehicle includes a mobile cabinet, the upper surface of the mobile cabinet is a work surface, and the front of the mobile cabinet is also provided with multiple layers of drawers; each layer of the multiple layers of drawers is provided with an electric lock; the management system includes a main control unit and an edge computing unit connected to the main control unit, the main control unit is also respectively connected to a human-computer interaction unit, a voice recognition module, and an electromagnetic control unit for controlling the power supply of the electromagnetic lock arranged on the work surface; the edge computing unit is respectively connected to a first camera installed above the multiple layers of drawers, a second camera installed on the work surface, and a third camera installed on the work surface; the first camera is used to obtain the picture inside the opened drawer; the second camera adopts a monocular camera for drug entry and identification, and the third camera adopts a multi-camera camera for transparent medicine entry and identification; the edge computing unit is also connected to a network unit, and is connected to a background service terminal through the network unit.

[0006] Each layer of the multi-layer drawer of the rescue vehicle is provided with a three-color indicator light indicating the status of the medicine; the management system also includes a three-color light management unit for controlling the three-color indicator lights, and the three-color light management unit is connected to the main control unit.

[0007] The background service terminal is provided with a comprehensive database set, and the hospital drug data set collects the emergency vehicle pharmacopoeia list and stores it in the comprehensive database set.

[0008] A management method for an emergency vehicle management system based on a visual model, comprising a medicine replenishment method, a medicine collection method, a medicine verification method, and a medicine exchange method.

[0009] The tonic method comprises: If medicines or items need to be replenished, the replenishment page can be entered by pressing the replenishment button on the human-computer interaction unit or by inputting "replenish medicines" via the voice recognition module; The second camera is used to identify the placement of various items and send them to the edge computing module; Enable the third camera to identify the placement of the transparent medicine and send it to the edge computing module; The edge computing module extracts text from the target object, where the target text includes a name and an expiration date; The target text is sent to the main control unit, which compares the target text with the pharmacopoeia list of the rescue vehicle and sends the comparison result to the human-computer interaction unit for confirmation; If correct, the main control unit opens the corresponding drawer through the electromagnetic control unit; the drawer position is detected by the first camera; After placing the items in the corresponding drawers and the medicines in the corresponding medicine boxes, the edge computing module detects the number and location of the placed items or reagents based on the gesture recognition model and displays the results on the human-computer interaction unit; the main control module changes the three-color light indicating the status of the medicine through the three-color light management unit; and updates the medicine refill record.

[0010] The medicine taking method comprises: If the medicine or item needs to be used, click the use button on the human-computer interaction unit or input the voice "use medicine" through the voice recognition module to enter the use page; The main control unit opens the corresponding drawer through the electromagnetic control unit and detects the position of the drawer through the first camera; After taking out the corresponding drawer items or the corresponding medicine box medicine, the edge computing unit detects the number and location of the placed items or reagents based on the gesture recognition model and displays the results on the human-computer interaction unit; the main control module changes the three-color light indicating the status of the medicine through the three-color light management unit; and updates the medication record.

[0011] The verification method includes: If medicines or items need to be inventoried, click the verification button on the human-computer interaction unit or input "verify medicines" through the voice recognition module to enter the verification interface; The electromagnetic control unit opens all drawers; The main control unit detects the position of the drawer through the first camera; Pull out the 1st to 5th layer drawers or open the corresponding medicine boxes in sequence. The edge computing unit detects the number and location of the placed items or reagents, displays the results on the human-computer interaction unit, and updates the verification record.

[0012] The replacement method includes: If there are medicines or items that need to be exchanged, click the button on the human-computer interaction unit or input "exchange medicines" through the voice recognition module to enter the exchange page; The electromagnetic control unit opens the corresponding drawer; the main control unit detects the position of the drawer through the first camera; Take out expired or discarded items from the corresponding drawer or take out expired or discarded medicine from the corresponding medicine box; The edge computing unit detects the number and location of placed objects or reagents based on the gesture recognition model and displays the results on the human-computer interaction unit. The control module changes the three-color light indicating the status of the drug through the three-color light management unit. After the exchange is completed, the medicines taken will be replenished and a new exchange record will be issued.

[0013] The present invention's beneficial effects include: The system uses a camera to directly scan expiration date labels on medications or items, automatically identifying and entering them into the system, eliminating manual input steps and improving accuracy and efficiency. Medical staff simply place medications or items in drawers or access them directly according to standard procedures, and the visual recognition system automatically completes material registration or verification, eliminating the need for additional labeling or scanning, perfectly adapting to existing work habits. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is a structural diagram of a rescue vehicle of the present invention.

[0015] Figure 2 It is a hardware architecture diagram of the present invention.

[0016] Figure 3 It is a schematic diagram of the model deployment of the present invention.

[0017] Figure 4 It is a medicine replenishment flow chart of the present invention.

[0018] Figure 5 It is a medicine taking flow chart of the present invention.

[0019] Figure 6 It is a verification flow chart of the present invention.

[0020] Figure 7 It is a replacement flow chart of the present invention.

[0021] Among them, 1 is the first camera; 2 is the second camera; 3 is the third camera, and 4 is the three-color indicator light. DETAILED DESCRIPTION

[0022] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0023] It should be noted that the following detailed description is illustrative and is intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same technical meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0024] like Figure 1-2The present invention provides a rescue vehicle management system based on a visual model. The rescue vehicle includes a mobile cabinet, the upper surface of the mobile cabinet is a work surface, and the front of the mobile cabinet is also provided with multiple drawers; each layer of the multiple drawers is provided with an electric lock (the electromagnetic lock adopted by the present invention is a conventional form in this field); each layer of the multiple drawers is provided with a three-color indicator light indicating the status of the medicine (when the medicine is approaching the expiration date, the indicator light turns yellow, the indicator light for lack or expiration turns red, and the normal state is green); the management system includes a main control unit and an edge computing unit connected to the main control unit. The main control unit is also respectively connected to a human-computer interaction unit and a voice control unit arranged on the work surface. The edge computing unit is connected to an identification module, an electromagnetic control unit for controlling the power supply of the electromagnetic lock, a three-color light management unit for controlling the three-color indicator light, an electric control container, and a single precision control unit; the edge computing unit is respectively connected to a first camera installed above a multi-layer drawer, a second camera installed on the work surface, and a third camera installed on the work surface; the first camera is used to obtain the image inside the opened drawer; the second camera adopts a monocular camera for drug entry and identification, and the third camera adopts a multi-camera camera for transparent drug entry and identification; the edge computing unit is also connected to a network unit, and is connected to a background service terminal through the network unit.

[0025] like Figure 3 As shown, a comprehensive database set is provided in the background service terminal, and the hospital drug data set collects the emergency vehicle pharmacopoeia list and stores it in the comprehensive database set.

[0026] A management method for an emergency vehicle management system based on a visual model, comprising a medicine replenishment method, a medicine collection method, a medicine verification method, and a medicine exchange method.

[0027] like Figure 4 As shown, the tonic method includes: If medicines or items need to be replenished, the replenishment page can be entered by pressing the replenishment button on the human-computer interaction unit or by inputting "replenish medicines" via the voice recognition module; The second camera is used to identify the placement of various items and send them to the edge computing module; Enable the third camera to identify the placement of the transparent medicine and send it to the edge computing module; The edge computing module extracts text from the target object, where the target text includes a name and an expiration date; The target text is sent to the main control unit, which compares the target text with the pharmacopoeia list of the rescue vehicle and sends the comparison result to the human-computer interaction unit for confirmation; If correct, the main control unit opens the corresponding drawer through the electromagnetic control unit; the drawer position is detected by the first camera; After placing the items in the corresponding drawers and the medicines in the corresponding medicine boxes, the edge computing module detects the number and location of the placed items or reagents based on the gesture recognition model and displays the results on the human-computer interaction unit; the main control module changes the three-color light indicating the status of the medicine through the three-color light management unit; and updates the medicine refill record.

[0028] like Figure 5 As shown, the medicine taking method includes: If the medicine or item needs to be used, click the use button on the human-computer interaction unit or input the voice "use medicine" through the voice recognition module to enter the use page; The main control unit opens the corresponding drawer through the electromagnetic control unit and detects the position of the drawer through the first camera; After taking out the corresponding drawer items or the corresponding medicine box medicine, the edge computing unit detects the number and location of the placed items or reagents based on the gesture recognition model and displays the results on the human-computer interaction unit; the main control module changes the three-color light indicating the status of the medicine through the three-color light management unit; and updates the medication record.

[0029] like Figure 6 As shown, the verification method includes: If medicines or items need to be inventoried, click the verification button on the human-computer interaction unit or input "verify medicines" through the voice recognition module to enter the verification interface; The electromagnetic control unit opens all drawers; The main control unit detects the position of the drawer through the first camera; Pull out the 1st to 5th layer drawers in sequence or open the corresponding medicine boxes in sequence. The edge computing unit detects the number and location of the placed items or reagents, displays the results on the human-computer interaction unit, and updates the verification record.

[0030] like Figure 7 As shown, the replacement method includes: If there are medicines or items that need to be exchanged, click the button on the human-computer interaction unit or input "exchange medicines" through the voice recognition module to enter the exchange page; The electromagnetic control unit opens the corresponding drawer; the main control unit detects the position of the drawer through the first camera; Take out expired or discarded items from the corresponding drawer or take out expired or discarded medicine from the corresponding medicine box; The edge computing unit detects the number and location of placed objects or reagents based on the gesture recognition model and displays the results on the human-computer interaction unit. The control module changes the three-color light indicating the status of the drug through the three-color light management unit. After the exchange is completed, the medicines taken will be replenished and a new exchange record will be issued.

[0031] The edge computing module extracts text from the target object, including: processing the image captured by the second camera and processing the image captured by the third camera; The processing of the image captured by the second camera includes the following steps: S11: Acquire an overall image of the medicine box through a second camera on the rescue vehicle; S12: Full-image text detection: Use DBNet to detect the full-image text area and output the detection frame coordinates; enhance the detection of the following areas: the top 1 / 3 area of the medicine box (high probability area for the name); the bottom barcode area (high probability area for the batch number / expiration date); S13: Text Recognition (PaddleOCR-CRNN+CTC): Identify each detection box area one by one and output the text content with coordinates; S14: Key field extraction: including drug name extraction, batch number extraction, and expiration date extraction; S15: Structured output: Delete results with a confidence level less than 85%; Format standardization: Expiration dates are standardized to the YYYY-MM-DD format, with batch numbers excluding extra spaces / symbols; the expiration date must be greater than or equal to the current date; S16: After the search is successful, the final output is performed.

[0032] The processing of the image captured by the third camera includes the following steps: S21: Multi-angle image acquisition: multiple cameras are used to acquire 360-degree multi-angle images around the medicine bottle; S22: Image preprocessing: The preprocessing includes performing image enhancement operations on multi-angle images using a Retinex enhancement algorithm to improve uneven lighting and enhance the contrast of text on the surface of the medicine bottle; Non-local mean denoising (NL-Means) is performed on the enhanced multi-angle images to reduce the noise interference caused by translucent materials. After morphological segmentation of the denoised multi-angle image, the main area of the medicine bottle is extracted and the background interference is separated; S23: Image Registration: Based on the scale-invariant feature transform (SIFT) algorithm and the random sampling consistency (RANSAC) algorithm, feature point matching and geometric transformation alignment are performed on pre-processed images at different angles to eliminate perspective differences and ensure spatial consistency of the fused image. S24: Multi-resolution image fusion: Combines the Laplacian pyramid fusion algorithm to fuse the details and background of multi-view images in layers, retaining the optimal information from each angle. This is used to resolve local blur or reflection issues caused by translucent materials. S25: Validity period information extraction: PaddleOCR+DBNet algorithm is used to detect the text area in the fused image, and PaddleOCR+CRNN recognition algorithm is used to perform end-to-end recognition on the detected text area; S26: Expiration date verification and database entry: A fuzzy string matching algorithm is used to match the identified expiration date with the drug batch information in the database, correcting any possible identification errors, and finally entering the information into the system. The algorithm in S26 may be a Levenshtein distance algorithm.

[0033] The edge computing module detects the gesture recognition based on the recognition model, including the following steps: S31: Using the first camera to capture the image inside the opened drawer; monitoring in real time whether hand features appear in the opened drawer area; S32: If hand features appear, capture a video frame image, and obtain an image with complete hand features from the video frame image; S33: Gesture recognition for images with complete hand features; S34: Determine the gesture type by querying the gesture mapping table: If it is a digital gesture, the corresponding value will be recorded and the accumulation mode will be entered; If the gesture is activated in the accumulation mode, multiple digital gestures can be input continuously for accumulation; If it is a confirmation gesture, step S5 is triggered to start counting medicines; If it is a clear or cancel gesture, the current counting result will be reset or rolled back; S35: If medicine counting is required, an image is obtained after the hand feature disappears, the medicine quantity is recognized on the image, and the gesture input value is compared with the actual medicine quantity to complete the closed-loop verification.

[0034] The gesture recognition includes a training phase, specifically: Establish a gesture dataset, annotate gesture labels, enhance the data including rotation, translation, and scaling, select a CNN deep network model, and input the data into the CNN deep network for training; input the monitored images with complete hand features into the trained CNN deep network to output the gesture recognition results.

[0035] The gesture recognition includes the inference stage: specifically, establishing a gesture dictionary: collecting samples of gestures commonly used by medical staff and labeling gesture categories; Construct a hierarchical dictionary structure: the first level is classified into "number", "function" and "special", and the second level is the specific gesture label; Static gesture mapping table: stores the correspondence between gestures and operations in the form of key-value pairs; Cumulative counting mode: After activation, the system accumulates the continuously input digital gesture values until a confirmation or reset command is received; during the accumulation process, the temporary counting result is displayed in real time and can be canceled midway; Special gesture handling: Clear gesture: clear the current counting pool and displayed results, and reset the system to the initial state; Pause gesture: freeze the current counting process and retain the counting pool data until the gesture is input again; Exception handling: If an undefined gesture is detected, a voice prompt "Invalid gesture, please try again" is triggered.

[0036] The gesture mapping table includes static gesture and number mapping, functional gestures and special gestures; wherein, the static gesture and number mapping includes: five fingers spread out to represent the number 5, four fingers spread out and the thumb bent to represent the number 4, three fingers spread out to represent the number 3, two fingers spread out to represent the number 2, and one finger extended to represent the number 1; the functional gestures include: spreading both hands flat to confirm the count, putting five fingers together and swinging them quickly to clear the count, and making a fist and releasing it to undo the last operation; the special gestures include: extending the little finger and thumb to enter the cumulative counting mode, and crossing the fingers to pause the counting.

[0037] The S35 specifically includes the following steps: Hand feature disappearance detection: After the gesture is confirmed, the system continues to monitor the drawer area until the hand completely moves out of the camera's field of view; Preprocessing of pharmaceutical images: De-noising and light equalization processing are performed on the images after the hands disappear to eliminate shadow interference; Drug object detection and segmentation: Use the Mask R-CNN model to locate the drug bounding box and extract the individual drug outlines through instance segmentation; Quantity statistics and verification: Count the number of independent targets after segmentation and filter out false detections by combining with the drug size database; If the difference between the detection quantity and the gesture input value exceeds the threshold, an alarm is triggered and manual review is required.

[0038] The present invention has the following advantages: Automatic scanning and entry of expiration date: Through visual recognition technology, the system can directly scan the text information on the drug packaging (such as name, batch number, expiration date, etc.) and automatically enter it into the system without manual input, greatly improving management efficiency and reducing errors.

[0039] Touchless operation, in line with actual usage habits: Visual recognition does not rely on additional tags such as RFID. Medical staff only need to put medicines or items into the drawer of the emergency vehicle or use them directly. The system can automatically identify and record them, avoiding additional operation steps and truly realizing "touchless management".

[0040] Real-time expiration monitoring and early warning: Combined with the backend management system, visual recognition can automatically compare drug expiration dates and issue early warnings when drugs are approaching expiration or when inventory is low, ensuring that the rescue vehicle is always in optimal standby condition.

[0041] In summary, the application of visual recognition technology can significantly improve the intelligence level of emergency vehicle management, reduce manual intervention, improve rescue efficiency, while reducing management costs and safety risks, and provide more reliable protection for medical emergency scenarios.

[0042] The above is only a preferred embodiment of the present invention. It should be pointed out that for those skilled in the art, several changes and improvements can be made without departing from the overall concept of the present invention, and these should also be regarded as the scope of protection of the present invention.

Claims

1. A rescue vehicle management system based on a visual model, wherein the rescue vehicle comprises a mobile cabinet, the upper surface of the mobile cabinet serving as a work surface, and the front of the mobile cabinet further comprising multiple drawers; each layer of the multiple drawers being equipped with an electric lock; and characterized in that: The management system includes a main control unit and an edge computing unit connected to the main control unit. The main control unit is also respectively connected to a human-computer interaction unit, a voice recognition module, and an electromagnetic control unit for controlling the power supply of an electromagnetic lock provided on the work surface; the edge computing unit is respectively connected to a first camera installed above a multi-layer drawer, a second camera installed on the work surface, and a third camera installed on the work surface; The first camera is used to capture the image inside the opened drawer; The second camera uses a monocular camera for drug entry and identification, and the third camera uses a multi-camera camera for transparent medicine entry and identification; the edge computing unit is also connected to the network unit and connected to the background service terminal through the network unit.

2. The visual model-based emergency vehicle management system according to claim 1, characterized in that: Each layer of the multi-layer drawer of the rescue vehicle is provided with a three-color indicator light indicating the status of the medicine; the management system also includes a three-color light management unit for controlling the three-color indicator lights, and the three-color light management unit is connected to the main control unit.

3. The visual model-based emergency vehicle management system according to claim 1, characterized in that: The background service terminal is provided with a comprehensive database set, and the hospital drug data set collects the emergency vehicle pharmacopoeia list and stores it in the comprehensive database set.

4. A management method for an emergency vehicle management system based on a visual model according to any one of claims 1 to 3, characterized in that: The management method includes a medicine replenishing method, a medicine taking method, a medicine checking method and a medicine exchanging method.

5. The management method according to claim 4, characterized in that: The tonic method comprises: If medicines or items need to be replenished, the user can click the replenish button on the human-computer interaction unit or input "replenish medicines" through the voice recognition module to enter the replenishment page; The second camera is used to identify the placement of various items and send them to the edge computing module; Enable the third camera to identify the placement of the transparent medicine and send it to the edge computing module; The edge computing module extracts text from the target object, where the target text includes a name and an expiration date; The target text is sent to the main control unit, which compares the target text with the pharmacopoeia list of the rescue vehicle and sends the comparison result to the human-computer interaction unit for confirmation; If correct, the main control unit opens the corresponding drawer through the electromagnetic control unit; the drawer position is detected by the first camera; After placing the items in the corresponding drawers and the medicines in the corresponding medicine boxes, the edge computing module detects the number and location of the placed items or reagents based on the gesture recognition model and displays the results on the human-computer interaction unit; the main control module changes the three-color light indicating the status of the medicine through the three-color light management unit; and updates the medicine refill record.

6. The management method according to claim 4, characterized in that: The medicine taking method comprises: If the medicine or item needs to be used, click the use button on the human-computer interaction unit or input the voice "use medicine" through the voice recognition module to enter the use page; The main control unit opens the corresponding drawer through the electromagnetic control unit and detects the position of the drawer through the first camera; After taking out the corresponding drawer items or the corresponding medicine box medicine, the edge computing unit detects the number and location of the placed items or reagents based on the gesture recognition model and displays the results on the human-computer interaction unit; the main control module changes the three-color light indicating the status of the medicine through the three-color light management unit; and updates the medication record.

7. The management method according to claim 4, characterized in that: The verification method includes: If medicines or items need to be checked, click the check button on the human-computer interaction unit or input "check medicines" via the voice recognition module to enter the check interface; The electromagnetic control unit opens all drawers; The main control unit detects the position of the drawer through the first camera; Pull out the 1st to 5th layer drawers or open the corresponding medicine boxes in sequence. The edge computing unit detects the number and location of the placed items or reagents, displays the results on the human-computer interaction unit, and updates the verification record.

8. The management method according to claim 4, characterized in that: The replacement method includes: If there are medicines or items that need to be exchanged, click the button on the human-computer interaction unit or input "exchange medicines" through the voice recognition module to enter the exchange page; The electromagnetic control unit opens the corresponding drawer; the main control unit detects the position of the drawer through the first camera; Take out expired or discarded items from the corresponding drawer or take out expired or discarded medicine from the corresponding medicine box; The edge computing unit detects the number and location of placed objects or reagents based on the gesture recognition model and displays the results on the human-computer interaction unit. The control module changes the three-color light indicating the status of the drug through the three-color light management unit. After the exchange is completed, the medicines taken will be replenished and a new exchange record will be issued.

Images