Drug management method, device and system
By identifying T-shaped corners in the drug handling area to create a recognition zone, hands and drugs are tracked in real time, and inventory data is automatically updated. This solves the problem of lagging drug inventory data, enabling real-time updates of drug inventory and saving labor costs.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SHENZHEN NUOBO MEDICAL INSTR CO LTD
- Filing Date
- 2025-12-29
- Publication Date
- 2026-05-01
AI Technical Summary
In existing technologies, drug inventory data is updated slowly and lacks timeliness, requiring manual entry, which leads to high labor costs and is prone to errors.
By acquiring video stream images of the drug handling area, identifying T-shaped corners, constructing a recognition area, and tracking hands and drugs entering the recognition area in real time, the inventory data is automatically updated.
It enables real-time updates of drug inventory data, improves data timeliness, saves labor costs, and reduces human error.
Smart Images
Figure CN121964080A_ABST
Abstract
Description
Drug management methods, devices and systems Technical Field
[0001] This application relates to the fields of next-generation information technology and biomedical industry technology, and in particular to a drug management method, device and system. Background Technology
[0002] In existing technologies, drug verification usually requires manual operation to place the drugs into the correct compartments. However, manual operation cannot automatically update the inventory system at the same time as the drugs are put in or taken out, and it is necessary to rely on manual entry or post-event verification, which leads to a lag in inventory data updates and a lack of timeliness. Summary of the Invention
[0003] This application aims to address at least one of the technical problems existing in the prior art. To this end, this application proposes a drug management method, apparatus, and system that can simultaneously and promptly update inventory data while automatically identifying when drugs are placed or retrieved, thereby ensuring the timeliness of inventory data.
[0004] The drug management method according to a first aspect of this application includes: acquiring continuous video stream images of a drug retrieval and placement area to obtain a first image sequence, wherein the drug retrieval and placement area is the area through which drugs are retrieved or placed from a medicine box, and the medicine box includes a medicine compartment component; acquiring multiple T-shaped corner points on the first image sequence; constructing a recognition area based on at least two T-shaped corner points; performing target tracking detection on a hand and a target drug in the recognition area, and determining whether the hand and the target drug have entered the recognition area to obtain a judgment result; and updating inventory data based on the judgment result.
[0005] The drug management method according to the embodiments of this application has at least the following beneficial effects: by acquiring continuous video stream images of the area through which drugs are taken out and placed from the drug case, a first image sequence is obtained to obtain the basic layout of the drug compartments in the drug case. Multiple T-shaped corner points on the first image sequence are identified, and at least two of the T-shaped corner points are used to construct the identification area for the drug verification operation. By performing target tracking detection on the hand and the target drug in the identification area, it can be determined whether the hand and the target drug have entered the identification area. When the hand and the target drug enter the identification area, they can be monitored in real time. By monitoring the movement trajectory of the hand and the drug in the identification area in real time, it is possible to determine whether the drug has been placed or taken out. At the same time, the inventory data is automatically updated synchronously based on the judgment result to ensure the timeliness of the inventory data and save labor costs.
[0006] According to some embodiments of this application, the drug compartment assembly includes a plurality of first connectors and a plurality of second connectors; the first connector includes a first long bottom surface and a first short bottom surface connected together, the first long bottom surface and the first short bottom surface being perpendicularly arranged; the second connector includes a second long bottom surface and a second short bottom surface connected together, the second long bottom surface and the second short bottom surface being perpendicularly arranged; the first short bottom surface and the second short bottom surface abut; before acquiring multiple T-shaped corner points on the first image sequence, the assembly further includes: fitting the abutting first short bottom surface and the second short bottom surface into the horizontal side of the T-shaped corner point; and generating the vertical side of the T-shaped corner point by pointing the midpoint of the horizontal side toward the extension direction of the first long bottom surface and the second long bottom surface.
[0007] According to some embodiments of this application, before obtaining multiple T-shaped corner points on the first image sequence, the method further includes: extending the first short bottom surface or the second short bottom surface of the first connector or the second connector that is not fitted into a T-shaped corner point to the opposite side by a predetermined distance to fit the horizontal side of the T-shaped corner point.
[0008] According to some embodiments of this application, the identification region includes at least one counting region; the construction of the identification region based on at least two T-corner points includes: selecting adjacent T-shaped corner points with vertical sides on the same side and horizontal sides collinear, and using the midpoint of the horizontal side as the first construction endpoint and the second construction endpoint of the counting region; extending the first construction endpoint and the second construction endpoint along the vertical side by a preset height to obtain a third construction endpoint and a fourth construction endpoint; connecting the first construction endpoint, the second construction endpoint, the third construction endpoint, and the fourth construction endpoint to obtain the counting region.
[0009] According to some embodiments of this application, obtaining a third and fourth construction endpoint based on a preset height extended along the vertical side of the first and second construction endpoints includes: when the vertical sides of adjacent T-shaped corner points are parallel, obtaining the third and fourth construction endpoints based on a first preset height extended along the vertical side of the first and second construction endpoints; when the vertical sides of adjacent T-shaped corner points are not parallel, obtaining the third and fourth construction endpoints based on a second preset height extended along the vertical side of the first and second construction endpoints; wherein the second preset height is greater than the first preset height.
[0010] According to some embodiments of this application, after constructing the identification region based on at least two T-corner points, the method further includes: obtaining the physical parameters of the counting region, wherein the physical parameters correspond one-to-one with the counting region; comparing the physical parameters with preset standard parameters one by one, and removing the counting regions whose corresponding physical parameters are not within the range of the standard parameters.
[0011] According to some embodiments of this application, the step of performing target tracking and detection on the hand and the target drug in the recognition area, and determining whether the hand and the target drug have entered the recognition area to obtain a determination result, includes: acquiring continuous video stream images within the recognition area to obtain a second image sequence; based on the first image sequence and the second image sequence, obtaining the coordinate range of the recognition area under the first image sequence to obtain a first coordinate range; based on the first image sequence, monitoring the first bounding box coordinates of the hand and the second bounding box coordinates of the target drug respectively; if the first bounding box coordinates are within the first coordinate range, then determining that the hand has entered the recognition area; if the second bounding box coordinates are within the first coordinate range, then determining that the hand has entered the recognition area. The target drug enters the recognition area, and the real-time state of the target drug is determined based on the center point of the first bounding box coordinates and the center point of the second bounding box coordinates. The real-time state includes a first state, a second state, and a third state. When the migration sequence of the real-time state satisfies the order of the first state, the second state, and the third state, and the number of frames in which the target drug is in the third state exceeds a preset first threshold frame number, it is determined that the target drug has been placed in the recognition area. When the migration sequence of the real-time state satisfies the order of the third state, the second state, and the first state, and the number of frames in which the target drug is in the first state exceeds a preset second threshold frame number, it is determined that the target drug has been removed from the recognition area.
[0012] According to some embodiments of this application, determining the real-time state of the target drug based on the center point of the first bounding box coordinates and the center point of the second bounding box coordinates includes: calculating the distance between the center point of the first bounding box coordinates and the center point of the second bounding box coordinates to obtain a first distance; when the first distance is less than a preset threshold distance, determining that the target drug is held by the hand; when the first distance is greater than or equal to the threshold distance, determining that the target drug is not held by the hand; when the target drug is held by the hand and the center point of the second bounding box coordinates is not within the recognition area, the target drug is determined to be in the first state; when the target drug is held by the hand and the center point of the second bounding box coordinates is within the recognition area, the target drug is determined to be in the second state; when the target drug is not held by the hand and the center point of the second bounding box coordinates is within the recognition area, the target drug is determined to be in the third state.
[0013] According to a second aspect embodiment of this application, a drug management device includes: an image acquisition module configured to acquire continuous video stream images within a drug handling area to obtain a first image sequence, wherein the drug handling area is the area through which drugs are handled from a medicine box, and the medicine box includes a medicine compartment component; a region construction module configured to acquire multiple T-shaped corner points on the first image sequence; construct a recognition region based on at least two T-shaped corner points; and a judgment module configured to perform target tracking detection on a hand and a target drug within the recognition region, and determine whether the hand and the target drug have entered the recognition region to obtain a judgment result; and update inventory data based on the judgment result.
[0014] A drug management system according to a third aspect of this application includes: a controller for executing the drug management method described in the first aspect of this application; a drug cartridge including a housing and a compartment assembly, the compartment assembly being housed in the housing and including a plurality of first connectors and second connectors, the plurality of first connectors and the plurality of second connectors being alternately arranged and connected to form a ring structure; and a camera disposed above the drug retrieval area within the drug cartridge.
[0015] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description
[0016] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which: Figure 1 is a schematic flowchart of the steps of a drug administration method according to an embodiment of this application; Figure 2 is a schematic flowchart of steps S201 to S202 according to an embodiment of this application; Figure 3 is a schematic flowchart of a specific step S103; Figure 4 is a schematic flowchart of a specific step S302; Figure 5 is a schematic flowchart of steps S501 to S502 according to an embodiment of this application; Figure 6 is a schematic flowchart of a specific step S104; Figure 7 is a schematic flowchart of a specific step S606. Figure 8 is a schematic diagram of the process; Figure 9 is a structural schematic diagram of the medicine box in an embodiment of this application; Figure 10 is a structural schematic diagram of the medicine compartment assembly in the medicine box structure shown in Figure 9; Figure 11 is a structural schematic diagram of the first connector in the medicine box structure shown in Figure 9; Figure 12 is a structural schematic diagram of the second connector in the medicine box structure shown in Figure 9; Figure 13 is a schematic diagram of the T-shaped corner point and counting area in an embodiment of this application; Figure 14 is a schematic diagram of the T-shaped corner point in the multi-row multi-column medicine compartment assembly in an embodiment of this application; Figure 15 is a structural schematic diagram of the multi-row multi-column medicine compartment assembly in an embodiment of this application.
[0017] Reference numerals: Drug compartment assembly 100; First connector 101; First long bottom surface 1011; First short bottom surface 1012; Second connector 102; Second long bottom surface 1021; Second short bottom surface 1022. Detailed Implementation
[0018] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application.
[0019] In the description of this application, it should be understood that the orientation descriptions, such as up, down, front, back, left, right, etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application.
[0020] In the description of this application, "several" means one or more, "more than" means two or more, "greater than," "less than," and "exceeding" are understood to exclude the stated number, while "above," "below," and "within" are understood to include the stated number. The use of "first" and "second" in the description is merely for distinguishing technical features and should not be construed as indicating or implying relative importance, or implicitly indicating the number of indicated technical features, or implicitly indicating the order of the indicated technical features.
[0021] In the description of this application, unless otherwise expressly defined, terms such as "setup," "installation," and "connection" should be interpreted broadly, and those skilled in the art can reasonably determine the specific meaning of the above terms in this application in conjunction with the specific content of the technical solution.
[0022] Currently, drug verification and data updates require manual operation. However, when manually dispensing drugs, it is common to put drugs into the wrong compartment. Furthermore, after dispensing or removing drugs, the back-end drug database cannot be automatically updated, requiring manual entry, which is labor-intensive and results in a lag in inventory data, hindering the work.
[0023] Based on this, this application proposes a drug management method, device and system that can update drug inventory data in real time while drugs are being delivered and taken away, improving the timeliness of the database, and eliminating the need for manual data entry and verification, thus saving labor costs.
[0024] Corner points: These are typically the intersections of object edges or points with unusually prominent image texture features. They are a key local feature of an image, possessing advantages such as rotation invariance.
[0025] The drug management method of this application is mainly applied to a medicine box 200 including a medicine compartment assembly 100. In practical applications, each component of the medicine box can be flexibly set into other shapes such as circles or polygons, multi-row multi-column rectangles, or single-row single-column rectangles, according to the design requirements of the equipment and the usage scenario. Referring to Figure 9, the medicine compartment assembly forms a matrix medicine compartment structure with a single-row ring distribution; referring to Figures 14 and 15, the medicine compartment assembly 100 forms a multi-row multi-column matrix medicine compartment structure.
[0026] As an example, referring to Figures 9, 10, 11, and 12, the drug compartment assembly 100 includes a plurality of first connectors 101 and a plurality of second connectors 102; the first connectors 101 include a first long bottom surface 1011 and a first short bottom surface 1012 connected together, the first long bottom surface 1011 and the first short bottom surface 1012 being arranged perpendicularly; the second connectors 102 include a second long bottom surface 1021 and a second short bottom surface 1022 connected together, the second long bottom surface 1021 and the second short bottom surface 1022 being arranged perpendicularly; the first short bottom surface 1012 and the second short bottom surface 1022 abut against each other. It should be noted that adjacent connecting parts can be fixed by means of snap-fit, fixing posts, etc. In one embodiment, referring to FIG9, the drug compartment assembly 100 forms a single-row, ring-shaped matrix drug compartment structure. Between two adjacent first connecting parts 101 and the second connecting parts 102 between the two first connecting parts 101, the second short bottom surface 1022 of the second connecting part 102 abuts against and faces the first short bottom surface 1012 of one first connecting part 101, and the second short bottom surface of the second connecting part 102 abuts against and faces away from the first short bottom surface 1012 of another first connecting part 101. In another embodiment, referring to FIG14, the drug compartment assembly 100 forms a multi-row, multi-column matrix drug compartment structure. In the non-edge area of the drug compartment assembly 100, the first long bottom surface 1011 and the second long bottom surface 1021 are fitted together to form a higher partition between the drug compartments. In the edge region of the drug compartment assembly 100, a connector alone constitutes an inner corner or part of the sidewall of the drug compartment. The first long bottom surface 1011 is not attached to the second long bottom surface 1021, and the corresponding first short bottom surface 1012 only abuts against a second short bottom surface 1022; or the second long bottom surface 1021 is not attached to the first long bottom surface 1011, and the corresponding second short bottom surface 1022 abuts against only a first short bottom surface 1012.
[0027] The first aspect of this application is based on the drug administration method illustrated in FIG1. Referring to FIG1, FIG1 is a flowchart of the drug administration method according to an embodiment of this application. The flowchart illustrated in FIG1 includes, but is not limited to, steps S101 to S105.
[0028] Step S101: Acquire continuous video stream images within the drug retrieval and placement area to obtain a first image sequence, wherein the drug retrieval and placement area is the area through which drugs are retrieved or placed from the medicine box 200, and the medicine box 200 includes a medicine compartment assembly 100.
[0029] Step S102: Obtain multiple T-shaped corner points on the first image sequence.
[0030] Step S103: Construct the recognition region based on at least two T-shaped corner points.
[0031] Step S104: Perform target tracking detection on the hand and target drug in the recognition area, and determine whether the hand and target drug have entered the recognition area to obtain the judgment result.
[0032] Step S105: Update the inventory data based on the judgment result.
[0033] The drug management method according to the embodiments of this application has at least the following beneficial effects: by acquiring continuous video stream images of the area through which drugs are taken out and placed from the medicine box 200, a first image sequence is obtained to obtain the basic layout of the medicine compartments in the medicine box 200. Multiple T-shaped corner points on the first image sequence are identified, and at least two of the T-shaped corner points are used to construct the identification area for the drug verification operation. By performing target tracking detection on the hand and the target drug in the identification area, it can be determined whether the hand and the target drug have entered the identification area. When the hand and the target drug enter the identification area, they can be monitored in real time. By monitoring the movement trajectory of the hand and the drug in the identification area in real time, it is possible to determine whether the drug is placed or taken out. At the same time, the inventory data is automatically updated synchronously based on the judgment result to ensure the timeliness of the inventory data and save labor costs.
[0034] In step S101 of some embodiments, the area through which medicines are taken out and placed in the medicine box 200 is continuously monitored by a fixed image acquisition device to obtain a continuous video stream image, thereby obtaining a first image sequence. The medicine box 200 includes a medicine compartment component 100 to facilitate the subsequent acquisition of T-shaped corner points.
[0035] It should be noted that a continuous video stream refers to a series of static images arranged sequentially in time, i.e., a series of consecutive frames.
[0036] In some embodiments, referring to FIG2, steps S201 to S203 may be included before step S102.
[0037] Step S201: Fit the first short bottom surface 1012 and the second short bottom surface 1022 that abut together to form the horizontal side of the T-shaped corner point.
[0038] Step S202: The midpoint of the horizontal edge is directed toward the extension direction of the first long base 1011 and the second long base 1021 to generate the vertical edge of the T-shaped corner point.
[0039] Step S203: Extend the first short bottom surface 1012 or the second short bottom surface 1022 of the first connector 101 or the second connector 102 that is not fitted into a T-shaped corner point to the opposite side by a predetermined distance to fit the horizontal side of the T-shaped corner point.
[0040] In step S201 of some embodiments, the first short bottom surface 1012 and the second short bottom surface 1022 of the first connector 101 and the second connector 102 abutting each other are used as the horizontal side of the T-shaped corner point. The second short bottom surface 1022 of the second connector 102 is arranged opposite to the first short bottom surface 1012 of the first connector 101. Furthermore, the first long bottom surface 1011 of the first connector 101 and the second long bottom surface 1021 of the second connector 102 are attached to each other.
[0041] In step S202 of some embodiments, the midpoint of the horizontal side of the T-shaped corner point is taken as the starting point and extended in the direction of the extension of the first long bottom surface 1011 and the second long bottom surface 1021 to form the vertical side of the T-shaped corner point. The length of the vertical side can be preset according to actual needs. In some regular drug compartment components 100, the length of the vertical side can be the same as the length of the first long bottom surface 1011 and the second long bottom surface 1021 to obtain the desired T-shaped corner point.
[0042] It should be noted that the midpoint of the horizontal side is equivalent to the intersection of the "T-shaped structure" at the T-corner point.
[0043] In step S203 of some embodiments, referring to FIG15, when the drug compartment assembly 100 has a multi-row, multi-column structure, at the edge of the drug compartment assembly 100, there are first connectors 101 without a second long bottom edge that is attached to the first long bottom edge, or second connectors 102 without a first long bottom edge that is attached to the second long bottom edge. That is, the second short bottom surface 1022 of the second connector 102 is disposed facing the first short bottom surface 1012 of only one first connector 101, or the first short bottom surface 1012 of the first connector 101 is only disposed facing the second short bottom surface 1012 of the second connector 102. Since the 022 components are set facing each other, a T-shaped corner cannot be formed through the previous steps. The first short bottom surface 1012 or the second short bottom surface 1022 of the first connector 101 or the second connector 102 that does not fit into a T-shaped corner needs to extend a predetermined distance towards the opposite side as the horizontal side of the T-shaped corner. For example, one end of the first short bottom surface 1012 is connected to the first long bottom surface 1011, which is assumed to be the connecting end, and the other end is the free end. The connecting end extends a predetermined distance along a straight line parallel to the first short bottom surface 1012 in a direction away from the free end to serve as the horizontal side of the T-shaped corner. A specific example is shown in Figure 14, where P4 is the T-shaped corner of the medicine compartment edge.
[0044] Steps S201 and S203 of this application involve fitting the T-shaped corner points to facilitate the construction of the subsequent identification region.
[0045] It should be noted that the fitted T-shaped corner points need to be further analyzed through multi-frame image time series analysis to screen out T-shaped corner points that are stable in position and recur in consecutive frames of the first image sequence.
[0046] In step S102 of some embodiments, the acquired first image sequence needs to be preprocessed by grayscale conversion, noise reduction, contrast enhancement, etc., in order to simplify the color dimension of the first image sequence, eliminate noise interference, improve detail recognition, etc., and then obtain multiple T-shaped corner points on the first image sequence.
[0047] In some embodiments, referring to FIG3, step S103 may include, but is not limited to, steps S301 to S303.
[0048] Step S301: Select adjacent T-shaped corner points with vertical sides on the same side and horizontal sides collinear, and use the midpoint of the horizontal side as the first and second construction endpoints of the counting area.
[0049] Step S302: Based on the first and second construction endpoints, extend along the vertical side to a preset height to obtain the third and fourth construction endpoints.
[0050] Step S303: Connect the first construction endpoint, the second construction endpoint, the third construction endpoint, and the fourth construction endpoint to obtain the counting region.
[0051] In step S301 of some embodiments, two adjacent T-shaped corner points with vertical sides on the same side and horizontal sides collinear are selected, and the midpoint of the horizontal side of these two T-shaped corner points is used as the first construction endpoint and the second construction endpoint of the counting region.
[0052] In some embodiments, referring to FIG4, step S302 may also include, but is not limited to, steps S401 to S402.
[0053] Step S401: When the vertical sides of adjacent T-shaped corner points are parallel, a third construction endpoint and a fourth construction endpoint are obtained by extending the first construction endpoint and the second construction endpoint along the vertical side to a first preset height.
[0054] Step S402: When the vertical sides of adjacent T-shaped corner points are not parallel, a third construction endpoint and a fourth construction endpoint are obtained by extending the first construction endpoint and the second construction endpoint along the vertical side to a second preset height; wherein, the second preset height is greater than the first preset height.
[0055] In step S401 of some embodiments, the shape of the drug compartment assembly 100 is relatively regular, and the long bottom surface dimensions of the first connector and the second connector constituting the drug compartment assembly are fixed. The vertical sides of two adjacent T-shaped corner points are parallel to each other. At this time, after extending the first construction endpoint and the second construction endpoint along the vertical side extension direction of the corresponding T-shaped corner point by a first preset height, the third construction endpoint and the fourth construction endpoint can be obtained respectively.
[0056] In step S402 of some embodiments, the shape of the drug grid assembly 100 may not be very regular, and the vertical sides of two adjacent T-shaped corner points may not be parallel to each other. If the same first preset height is extended along this vertical side direction, the effective coverage height of the constructed area in the direction perpendicular to the horizontal side will be reduced due to geometric relationships. To ensure that the identification area has the same coverage capability as the regular area in the vertical direction of the drug grid, the extension height needs to be compensated. At this time, after the first construction endpoint and the second construction endpoint are extended by the second preset height along the vertical side extension direction of the corresponding T-shaped corner point, the third construction endpoint and the fourth construction endpoint can be obtained respectively.
[0057] Steps S401 to S402 of this embodiment of the application, by constructing the third construction endpoint and the fourth construction endpoint under different conditions, enable the construction of the counting region to use a more diverse array of pharmacy components 100.
[0058] The effects achieved by steps S301 to 303 in this embodiment are shown in Figure 13. P1, P2 and P3 are all T-shaped corner points. Among them, the two T-shaped corner points P1 and P2 are adjacent and have the same vertical side and collinear horizontal side. Finally, P1 and P2 construct the counting area A1.
[0059] In some embodiments, referring to FIG5, steps S501 to S502 may be included after step S103.
[0060] Step S501: Obtain the physical parameters of the counting region. The physical parameters correspond one-to-one with the counting region.
[0061] Step S502: Compare the physical parameters with the preset standard parameters one by one, and remove the counting areas in the counting area whose corresponding physical parameters are not within the range of the standard parameters.
[0062] In step S501 of some embodiments, physical parameters corresponding to the counting region are obtained, such as pixel area, perimeter, aspect ratio, sector angle, etc.
[0063] In step S502 of some embodiments, the physical parameters are compared with preset standard parameters. If the physical parameters are not within the range of standard parameters, it is determined that the counting area does not meet the standard, that is, it may be an open area or have excessive deformation or abnormal size. In this case, the counting area is removed, and the counting area that is finally filtered out is the counting area included in the identification area.
[0064] This application uses steps S501 to S502 to select qualified counting regions as identification regions, thereby avoiding the influence of deformed counting regions on subsequent judgments.
[0065] In some embodiments, referring to FIG6, step S104 may include, but is not limited to, steps S601 to S608.
[0066] Step S601: Obtain continuous video stream images within the recognition area to obtain the second image sequence.
[0067] Step S602: Based on the first image sequence and the second image sequence, obtain the coordinate range of the recognition region under the first image sequence to obtain the first coordinate range.
[0068] Step S603: Based on the first image sequence, monitor the coordinates of the first bounding box of the hand and the second bounding box of the target drug, respectively.
[0069] Step S604: If the coordinates of the first bounding box are within the range of the first coordinates, then it is determined that the hand has entered the recognition area.
[0070] Step S605: If the coordinates of the second bounding box are within the range of the first coordinates, then it is determined that the target drug has entered the recognition area.
[0071] Step S606: Determine the real-time state of the target drug based on the center point of the first bounding box coordinates and the center point of the second bounding box coordinates, wherein the real-time state includes a first state, a second state, and a third state.
[0072] Step S607: When the real-time state migration sequence satisfies the order of first state, second state, and third state, and the number of frames in which the target drug is in the third state exceeds the preset first threshold number of frames, it is determined that the target drug has been placed in the identification area.
[0073] Step S608: When the real-time state migration sequence satisfies the order of third state, second state, and first state, and the number of frames in which the target drug is in the first state exceeds the preset second threshold number of frames, it is determined that the target drug has been taken away from the identification area.
[0074] In step S601 of some embodiments, the recognition area is continuously monitored by the image acquisition module to obtain continuous video stream images and obtain a second image sequence.
[0075] In step S602 of some embodiments, the coordinate range of the recognition area in the preset area is determined, that is, the coordinate range of the second sequence image in the first sequence image, to obtain the first coordinate range.
[0076] In step S603 of some embodiments, the hand detection module and the drug detection module are run synchronously. The hand detection module uses a lightweight deep learning model to detect and locate the hand position in the first image sequence in real time and outputs the coordinates of the first bounding box. The drug detection module uses another target detection model to detect and locate the target drug that may be held in real time and outputs the coordinates of the second bounding box.
[0077] It should be noted that the specific operation of real-time detection is as follows: a unique identifier is assigned to each detected hand and target drug. A multi-target tracking algorithm based on motion prediction and appearance feature matching is adopted. The image frame in the first image sequence in which the hand or target drug is detected is used as the starting frame. In subsequent frames, the position, velocity and motion trajectory of each identifier target are continuously tracked to achieve real-time detection of the coordinates of the first bounding box and the second bounding box.
[0078] In step S604 of some embodiments, there are multiple recognition areas. The center coordinates of the first bounding box are calculated and geometric algorithms such as the ray method are used. If the center coordinates of the first bounding box are within the first coordinate range of any recognition area, it is determined that the hand has entered the interior of the recognition area.
[0079] In step S605 of some embodiments, the center coordinates of the second bounding box are calculated using geometric algorithms such as the ray method. If the center coordinates of the second bounding box are within the range of the first coordinates of any recognition area, it is determined that the target drug has entered the interior of the recognition area.
[0080] In some embodiments, referring to FIG7, step S606 may include, but is not limited to, steps S701 to S706.
[0081] Step S701: Calculate the distance between the center point of the first bounding box coordinates and the center point of the second bounding box coordinates to obtain the first distance.
[0082] Step S702: When the first distance is less than the preset threshold distance, it is determined that the target drug is held by the hand.
[0083] Step S703: When the first distance is greater than or equal to the threshold distance, it is determined that the target drug is not held by the hand.
[0084] Step S704: When the target drug is held by the hand and the center point of the second bounding box coordinates is not within the recognition area, the target drug is determined to be in the first state.
[0085] Step S705: When the target drug is held by the hand and the center point of the second bounding box coordinates is located within the recognition area, the target drug is determined to be in the second state.
[0086] In step S706, when the target drug is not held by the hand and the center point of the second bounding box coordinates is located in the recognition area, the target drug is judged to be in the third state.
[0087] In step S701 of some embodiments, based on the principle of spatial proximity, the minimum Euclidean distance between the center point of the first bounding box and the center point of the second bounding box in each frame is calculated to obtain the first distance.
[0088] In step S702 of some embodiments, if the first distance is less than a preset threshold distance, it is determined that the target drug is held by the hand, and a "hand-object" association is established.
[0089] In step S703 of some embodiments, if the first distance is greater than or equal to the threshold distance, it is determined that the medicine is not being held by the hand at this time.
[0090] It should be noted that the threshold distance can be adaptively adjusted based on the size of the first bounding box of the hand.
[0091] In step S704 of some embodiments, a state machine is maintained for the target drug. The state machine reflects the real-time state of the target drug. When the target drug is held by the hand and the center point of the second bounding box coordinates of the target drug is not in the recognition area, then the corresponding state machine is STATE_A (i.e., the first state).
[0092] In step S705 of some embodiments, when the target drug is held by the hand and the center point of the second bounding box coordinates of the target drug is within the recognition area, then the corresponding state machine is STATE_B (i.e., the second state).
[0093] In step S706 of some embodiments, when the target drug is not held by the hand and the center point of the second bounding box coordinates of the target drug is within the recognition area, the corresponding state machine is STATE_C (i.e., the third state).
[0094] In other embodiments, the real-time state also includes a fourth state, and the state machine is also introduced into the STATE_B_OCCLUDED state, i.e., the fourth state, when the target drug is held but obscured, that is, the target drug is obscured by the hand in the recognition area, and the state machine is transitioned to this state when the hand has not left.
[0095] In the STATE_B_OCCLUDED state (i.e., the fourth state), prediction algorithms such as Kalman filtering are used to predict the motion state of the target drug before it was occluded, maintaining its virtual motion trajectory. The current position is predicted based on the trajectory before the drug disappeared. Simultaneously, the depth and appearance features of the target drug before occlusion are pre-stored through an appearance re-identification model. When the target drug reappears, its features are re-identified by calculating the similarity between the new target drug's features and the pre-stored features, achieving accurate recovery of the tracking identifier. If, after the hand leaves, a target drug with a high feature matching degree reappears through trajectory continuity or re-identification, and the position of this target drug is still within the recognition area, the state machine transitions to the STATE_C state (i.e., the third state), and it is determined that the target drug was successfully placed at this time.
[0096] In the embodiments of this application, steps S701 to S706 involve calculating the Euclidean distance between the center point of the first bounding box and the center point of the second bounding box to determine whether the target drug is held by a hand, and setting a state machine to classify the target drug under different states.
[0097] In step S607 of some embodiments, when the state transition sequence of a target drug state machine satisfies the order STATE_A, STATE_B, STATE_C (i.e., the order of first state, second state, and third state), and the number of frames in STATE_C (i.e., the third state) of the target drug state machine exceeds the first threshold number of frames, the system determines that a successful drug placement event has occurred.
[0098] In step S608 of some embodiments, when the state transition sequence of a target drug state machine satisfies the order STATE_C, STATE_B, STATE_A (i.e., the order of third state, second state, first state), and the number of frames in STATE_A (i.e., first state) of the target drug state machine exceeds the second threshold number of frames, the system determines that a successful drug retrieval event has occurred.
[0099] It should be noted that the first and second threshold frame counts are adaptive values dynamically calculated based on context information, rather than fixed thresholds. The core calculation formula is: N = max(N_min, min(N_max, N_base + k * v)); where N is the first or second threshold frame count, v represents the instantaneous movement speed of the target drug when entering STATE_B state (i.e., the second state). This speed can be calculated from the final position change of the target drug over multiple frames (e.g., the Euclidean distance between the center points of the second bounding box coordinates in two consecutive frames), N_base represents the basic confirmation frame count constant (i.e., the minimum number of frames required in a near-static ideal placement state), k represents the speed gain coefficient, which is preset based on experience to adjust the influence of movement speed on the confirmation frame count. N_min is the preset minimum frame count boundary, and N_max is the preset maximum frame count boundary, used to ensure the real-time performance of the system response (e.g., N_min = 3, N_max = 20).
[0100] It should be noted that v and N are directly proportional. Therefore, the larger v is, the faster the action is, but the more unstable it may be. Thus, the larger N is, the more image frames are needed to confirm the stability of the placement or retrieval behavior. On the other hand, the smaller v is, the slower and more stable the action is. Thus, the smaller N is, the fewer image frames are needed for confirmation.
[0101] Steps S601 to S608 of this embodiment of the application facilitate real-time judgment of whether the hand and the drug have entered the recognition area by real-time monitoring of the bounding box coordinates and center point of the hand and the target drug, thereby ensuring the timeliness of the data; and by obtaining the state machine transition sequence, it is possible to judge in real time whether the target drug has been placed or taken away, further ensuring the timeliness.
[0102] In other embodiments, the method further includes: if the hand and the target drug cannot be detected simultaneously, or if the target drug cannot be detected, monitoring the action of the hand entering and exiting the recognition area; after the hand exits, detecting whether a new drug or a drug has been removed from the recognition area to determine that a drug removal and placement operation has occurred; and updating the drug removal and placement status based on the new drug or the drug that has been removed.
[0103] In this embodiment, during target tracking and detection, if the hand and the target drug are not detected simultaneously, or if only the hand is detected but the drug is not identified, a supplementary detection mechanism will be activated. Specifically, by monitoring the action sequence of the hand entering and exiting the recognition area, and analyzing the trajectory and dwell time of the hand, possible pick-up and put-down operations can be inferred. After the hand exits, it is detected whether a new drug appears in the recognition area or whether an existing drug is removed, and changes are judged by image comparison or target reproduction, thereby determining whether a drug pick-up and put-down operation has occurred, and updating the drug pick-up and put-down status accordingly.
[0104] This embodiment solves the problem of pick-up and put-down judgment in scenarios of occlusion or detection failure by using action sequence and region change detection, thereby improving the system's fault tolerance and robustness.
[0105] In step S105 of some embodiments, when the event of placing or removing the target drug is confirmed, the system automatically generates an inventory update instruction. This instruction carries the identification information of the target drug (such as the target drug identifier, the identifier of the identification area, the timestamp, etc.) and the operation type, and sends it to the inventory management database to achieve real-time, synchronous incremental updates of inventory data. This allows the system to remind managers to replenish the drug or to indicate that replenishment has been completed based on the updated results.
[0106] It should be noted that the identification information of the target drug can be obtained from the associated OCR (Optical Character Recognition) or RFID (Radio Frequency Identification) module.
[0107] An embodiment of the second aspect of this application also provides a drug management device. Referring to FIG8, the drug management device includes: an image acquisition module 801, configured to acquire continuous video stream images within a drug handling area to obtain a first image sequence, wherein the drug handling area is the area through which drugs are handled from a medicine box 200, and the medicine box 200 includes a medicine compartment component 100; a region construction module 802, configured to acquire multiple T-shaped corner points on the first image sequence; construct a recognition area based on at least two T-shaped corner points; and a judgment module 803, configured to perform target tracking detection on a hand and a target drug within the recognition area, and determine whether the hand and the target drug have entered the recognition area to obtain a judgment result; and update inventory data based on the judgment result.
[0108] A third aspect of this application also provides a drug management system, comprising: a controller for executing the drug management method shown in the first aspect of this application; a medicine box 200, the medicine box 200 including a shell and a medicine compartment assembly 100, the medicine compartment assembly 100 being housed in the shell, the medicine compartment assembly 100 including a plurality of first connectors 101 and second connectors 102, the plurality of first connectors 101 and the plurality of second connectors 102 being alternately arranged and connected to form a ring structure; and a camera disposed above the drug retrieval area within the medicine box 200.
[0109] In some embodiments, the drug management method of the first aspect of this application is executed by a controller, a recognition area is constructed within the drug compartment assembly 100 of the drug cartridge 200, and a camera is used to acquire a first image sequence and a second image sequence, and also to detect the hand and the target drug.
[0110] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.
[0111] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.
[0112] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.
[0113] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.
[0114] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0115] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.
[0116] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.
[0117] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0118] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0119] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0120] Furthermore, it should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent will be obtained first. Moreover, the collection, use, and processing of this data will comply with relevant laws, regulations, and standards. Additionally, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user will be obtained through pop-ups or redirects to confirmation pages. Only after obtaining the user's separate permission or consent will the necessary user-related data for the proper functioning of these embodiments be acquired.
[0121] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.
Claims
1. A method for drug administration, characterized in that, include: A first image sequence is obtained by acquiring continuous video stream images within the drug retrieval and placement area, wherein the drug retrieval and placement area is the area through which drugs are retrieved or placed from the medicine box, and the medicine box includes a medicine compartment component; multiple T-shaped corner points are acquired on the first image sequence; and a recognition region is constructed based on at least two T-shaped corner points. The hand and the target drug in the identification area are tracked and detected, and it is determined whether the hand and the target drug have entered the identification area to obtain a judgment result; the inventory data is updated based on the judgment result.
2. The drug administration method according to claim 1, characterized in that, The drug compartment assembly includes a plurality of first connectors and a plurality of second connectors; the first connector includes a first long bottom surface and a first short bottom surface connected together, the first long bottom surface and the first short bottom surface being perpendicularly arranged; the second connector includes a second long bottom surface and a second short bottom surface connected together, the second long bottom surface and the second short bottom surface being perpendicularly arranged; the first short bottom surface and the second short bottom surface abut together; before acquiring multiple T-shaped corner points on the first image sequence, the assembly further includes: fitting the abutting first short bottom surface and the second short bottom surface into the horizontal side of the T-shaped corner point; and generating the vertical side of the T-shaped corner point by pointing the midpoint of the horizontal side toward the extension direction of the first long bottom surface and the second long bottom surface.
3. The drug administration method according to claim 2, characterized in that, Before acquiring multiple T-shaped corner points on the first image sequence, the method further includes: extending the first short bottom surface or the second short bottom surface of the first connector or the second connector that is not fitted into a T-shaped corner point to the opposite side by a predetermined distance to fit the horizontal side of the T-shaped corner point.
4. The drug administration method according to claim 3, characterized in that, The identification area includes at least one counting area; the construction of the identification area based on at least two T-corner points includes: selecting adjacent T-shaped corner points with the vertical sides on the same side and the horizontal sides collinear, and taking the midpoint of the horizontal side as the first construction endpoint and the second construction endpoint of the counting area; extending the first construction endpoint and the second construction endpoint along the vertical side by a preset height to obtain a third construction endpoint and a fourth construction endpoint; connecting the first construction endpoint, the second construction endpoint, the third construction endpoint and the fourth construction endpoint to obtain the counting area.
5. The drug administration method according to claim 4, characterized in that, The step of obtaining a third and fourth construction endpoint by extending a preset height along the vertical side based on the first and second construction endpoints includes: when the vertical sides of adjacent T-shaped corner points are parallel, obtaining the third and fourth construction endpoints by extending a first preset height along the vertical side based on the first and second construction endpoints; when the vertical sides of adjacent T-shaped corner points are not parallel, obtaining the third and fourth construction endpoints by extending a second preset height along the vertical side based on the first and second construction endpoints; wherein the second preset height is greater than the first preset height.
6. A drug administration method according to claim 4, characterized in that, After constructing the recognition region based on at least two T-corner points, the method further includes: obtaining the physical parameters of the counting region, wherein the physical parameters correspond one-to-one with the counting region; comparing the physical parameters with preset standard parameters one by one, and removing the counting regions whose corresponding physical parameters are not within the range of the standard parameters.
7. A drug administration method according to claim 1, characterized in that, The step of tracking and detecting the hand and the target drug within the recognition area, and determining whether the hand and the target drug have entered the recognition area to obtain a determination result, includes: acquiring continuous video stream images within the recognition area to obtain a second image sequence; based on the first image sequence and the second image sequence, obtaining the coordinate range of the recognition area under the first image sequence to obtain a first coordinate range; based on the first image sequence, monitoring the first bounding box coordinates of the hand and the second bounding box coordinates of the target drug respectively; if the first bounding box coordinates are within the first coordinate range, determining that the hand has entered the recognition area; if the second bounding box coordinates are within the first coordinate range, determining that the target drug... The target drug enters the identification area; the real-time state of the target drug is determined based on the center point of the first bounding box coordinates and the center point of the second bounding box coordinates, wherein the real-time state includes a first state, a second state, and a third state; when the migration sequence of the real-time state satisfies the order of the first state, the second state, and the third state, and the number of frames in which the target drug is in the third state exceeds a preset first threshold frame number, it is determined that the target drug is placed in the identification area; when the migration sequence of the real-time state satisfies the order of the third state, the second state, and the first state, and the number of frames in which the target drug is in the first state exceeds a preset second threshold frame number, it is determined that the target drug is removed from the identification area.
8. A drug administration method according to claim 7, characterized in that: The step of determining the real-time state of the target drug based on the center points of the first bounding box coordinates and the second bounding box coordinates includes: calculating the distance between the center points of the first bounding box coordinates and the second bounding box coordinates to obtain a first distance; when the first distance is less than a preset threshold distance, determining that the target drug is held by the hand; when the first distance is greater than or equal to the threshold distance, determining that the target drug is not held by the hand; when the target drug is held by the hand and the center point of the second bounding box coordinates is not within the recognition area, the target drug is determined to be in the first state; when the target drug is held by the hand and the center point of the second bounding box coordinates is within the recognition area, the target drug is determined to be in the second state; when the target drug is not held by the hand and the center point of the second bounding box coordinates is within the recognition area, the target drug is determined to be in the third state.
9. A drug management device, characterized in that, The device includes: an image acquisition module configured to acquire continuous video stream images within a drug handling area to obtain a first image sequence, wherein the drug handling area is the area through which drugs are handled from a medicine box, and the medicine box includes a medicine compartment component; a region construction module configured to acquire multiple T-shaped corner points on the first image sequence; construct a recognition region based on at least two T-shaped corner points; and a judgment module configured to perform target tracking detection on a hand and a target drug within the recognition region, and determine whether the hand and the target drug have entered the recognition region to obtain a judgment result; and update inventory data based on the judgment result.
10. A drug management system, characterized in that, include: A controller for performing the method as described in any one of claims 1 to 8; a medicine cartridge comprising a housing and a medicine compartment assembly housed within the housing, the medicine compartment assembly comprising a plurality of first connectors and second connectors, the plurality of first connectors and the plurality of second connectors being alternately arranged and connected to form a ring structure; and a camera disposed above the medicine dispensing area within the medicine cartridge.