A medicine taking recognition method and device, electronic equipment and storage medium

Abstract

The present application relates to the technical field of data processing, and provides a medicine taking recognition method and device, electronic equipment and storage medium. The medicine taking recognition method comprises: recognizing whether there is a medicine taking behavior based on a first image or a first video of a first target area, wherein the first target area comprises a medicine storage area; if the medicine taking behavior exists, triggering the acquisition of a second image or a second video of a second target area, wherein the second target area comprises a cash register area; if it is recognized based on the second image or the second video that there is a cash register behavior, judging whether the medicine taking behavior and the cash register behavior are associated; if the medicine taking behavior and the cash register behavior are associated, it is recognized as an effective medicine taking. The present application solves the technical problem that the association between the medicine taking recognition and the cash register condition of the pharmacy in the prior art is poor, and improves the accuracy of effective medicine taking recognition.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method, apparatus, electronic device, and storage medium for identifying medication. Background Technology

[0002] Existing pharmacy monitoring platforms typically determine whether a medication pickup event has occurred based on the target objects in the medication storage area. However, relying solely on the identification of the medication storage area to determine valid pickup, or relying solely on the identification of the medication storage area combined with system registration, without considering the closed-loop identification of pickup and payment, may result in false alarms due to incomplete transactions. Furthermore, the significant differences between different pharmacy POS systems make it difficult to effectively link payment and pickup through a unified platform, leading to inaccuracies in identifying valid pickups. Summary of the Invention

[0003] To address the technical problem of poor correlation between medication identification and payment status in existing technologies, this invention provides a medication identification method, device, electronic device, and storage medium, which improves the accuracy of effective medication identification.

[0004] A first aspect of the present invention provides a method for identifying medication, comprising:

[0005] Based on a first image or video of a first target area, it is determined whether a drug-taking behavior exists, wherein the first target area includes a drug storage area;

[0006] If the medication dispensing behavior occurs, a second image or second video of the second target area is triggered, and the second target area includes the cashier area.

[0007] If a bank collection is identified based on the second image or the second video, determine whether the medication collection behavior is related to the bank collection behavior.

[0008] If the medication collection behavior is associated with the bank, it is identified as a valid medication collection.

[0009] In some embodiments, if the time interval between the medication retrieval and the bank collection falls within a first time range, it is determined that the medication retrieval and the bank collection are related.

[0010] In some embodiments, if at least two target objects are present in the checkout area for a continuous period exceeding a first time threshold, it is determined that the medication retrieval behavior is associated with the checkout behavior.

[0011] In some embodiments, the second target area further includes a medicine storage area. The target object in the second image or the second video is tracked from the medicine storage area. If the target object has a trajectory from the medicine storage area to the checkout area, it is determined that the medicine retrieval behavior is associated with the checkout behavior.

[0012] In some embodiments, if at least one of the target objects has a trajectory of moving from the drug storage area to the checkout area, and at least two of the target objects continuously appear in the checkout area for more than a second time threshold, then it is determined that the drug retrieval behavior is associated with the checkout behavior.

[0013] In some embodiments, the combined medication collection behavior and the bank receipt are features of the single medication collection.

[0014] A second aspect of the present invention provides a drug dispensing identification device, comprising:

[0015] A first acquisition module is configured to acquire a first target area and obtain a first image or a first video, wherein the first target area includes a drug storage area.

[0016] The second acquisition module is configured to acquire a second target area and obtain a second image or a second video, wherein the second target area includes a cashier area.

[0017] An image processing module is configured to identify whether a drug-taking behavior exists based on the first image or the first video, and is configured to identify whether a bank-collecting behavior exists based on the second image or the second video;

[0018] A trigger module is configured to trigger the second acquisition module to collect the second target area and obtain the second image or the second video when the drug-taking behavior occurs.

[0019] The judgment module is configured to determine whether the medication retrieval behavior is associated with the bank collection point; if the medication retrieval behavior is associated with the bank collection point, it is identified as a valid medication retrieval.

[0020] In some embodiments, the second target area acquired by the second acquisition module further includes the drug storage area;

[0021] The medication identification device further includes a tracking module, which is configured to track the target object in the second image or the second video from the medication storage area and determine whether the target object has a trajectory from the medication storage area to the cashier area.

[0022] A third aspect of the present invention provides an electronic device comprising:

[0023] A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the steps of any of the drug identification methods described in this specification.

[0024] A fourth aspect of the present invention provides a storage medium,

[0025] The storage medium stores a computer program, which, when executed by a processor, implements the steps of any of the drug identification methods described in this specification.

[0026] This embodiment establishes a closed-loop association between medication collection behavior and payment behavior. By using image recognition and preset association conditions to replace the complex data provided by the POS system, the relationship between medication collection behavior and payment behavior can be established. This can effectively address the impact of any POS system on valid medication collection and improve the accuracy of valid medication collection identification.

[0027] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and in order to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0028] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0029] Figure 1 This is a schematic diagram illustrating an application scenario of an exemplary embodiment of the present invention;

[0030] Figure 2 A flowchart of a drug dispensing identification process provided in Exemplary Example 1 of the present invention;

[0031] Figure 3 A flowchart for drug identification provided in exemplary embodiment 2 of the present invention;

[0032] Figure 4 A flowchart for drug identification provided in exemplary embodiment 3 of the present invention;

[0033] Figure 5 A flowchart for drug identification provided in exemplary embodiment 4 of the present invention;

[0034] Figure 6A flowchart for drug identification provided in Exemplary Embodiment 5 of the present invention;

[0035] Figure 7 This is a block diagram of a medicine dispensing and identification device provided in Exemplary Embodiment 6 of the present invention. Detailed Implementation

[0036] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0037] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0038] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0039] The electronic devices involved in the embodiments of this application may include video and image acquisition devices, video and image storage devices, and video and image analysis devices, such as network cameras (IP cameras, IPCs), digital video recorders (DVRs), network video recorders (NVRs), smart boxes (AiBoxes), etc.

[0040] Pharmacy monitoring platforms can manage medications based on the actions of staff or customers retrieving medications from the cabinet, such as whether they are holding medications when leaving the cabinet. However, the mere act of retrieving medication does not necessarily constitute a valid medication retrieval event. For example, if medication is retrieved but the staff does not process the transaction or the customer does not pay, the retrieval itself is not considered a valid event. Furthermore, due to differences in POS systems across different pharmacies, a consistent platform cannot be used to link POS events to medication retrieval activities.

[0041] In view of this, the present invention provides a method, device, electronic device and storage medium for identifying medication dispensing behavior in a manner compatible with different POS systems, and accurately identifies valid medication dispensing events.

[0042] Please refer to Figure 1 This is a schematic diagram illustrating a scenario for the medication identification method provided in this embodiment. For example... Figure 1 As shown, the pharmacy includes a medicine storage area 100, a cashier area 200, a first visual acquisition device 11 for capturing images of the medicine storage area, and a second visual acquisition device 12 for capturing images of at least the cashier area. The first visual acquisition device 11 and the second visual acquisition device 12 can be image acquisition devices or video acquisition devices. In some embodiments, the second visual acquisition device 12 is a device for capturing global images or videos of the pharmacy, including areas such as the cashier area and the medicine storage area.

[0043] Example 1

[0044] Please refer to Figure 2 This embodiment provides a method for identifying medication, which may include the following steps:

[0045] S11 identifies whether medication retrieval behavior exists based on the first image or video of the drug storage area.

[0046] S12 If the medication retrieval behavior occurs, trigger the acquisition of a second image or second video of the cashier area.

[0047] S13 If a bank collection is identified based on the second image or the second video, determine whether the drug collection behavior is related to the bank collection.

[0048] S14 If the medication collection behavior is related to the bank, it is identified as a valid medication collection.

[0049] This embodiment establishes a closed-loop association between medication collection behavior and payment behavior. By using image recognition and preset association conditions to replace the complex data provided by the POS system, the relationship between medication collection behavior and payment behavior can be established. This can effectively address the impact of any POS system on valid medication collection and improve the accuracy of valid medication collection identification.

[0050] Example 2

[0051] Please refer to Figure 3 This embodiment provides a medication dispensing identification method. This embodiment uses motion detection events in the checkout area as approximations of checkout actions, and associates checkout actions falling within a preset time with medication dispensing behavior to achieve the identification of valid medication dispensing. It may include the following steps:

[0052] S21 continuously collects images or videos of the drug storage area through the first vision acquisition device 11. When a drug retrieval behavior is detected, the drug retrieval behavior is uploaded to the pharmacy monitoring platform in real time.

[0053] In one embodiment, the determination of medication retrieval behavior is based on the order, location, and frequency of medication retrieval and placement actions, and can also be aided by images or videos of the person at the boundary of the medication storage area. Specifically, when the first visual acquisition device 11 detects a person entering the medication storage area, it acquires the person's head and shoulder data, and establishes a personnel identification code for the person entering the medication storage area based on the head and shoulder data. The personnel identification code can be the head-to-shoulder ratio obtained based on the head and shoulder data. The system monitors medication retrieval and placement actions within the time interval from when the person enters the medication storage area to when they leave, generating medication retrieval and placement action data, and associating each instance of these data with the personnel identification code. The medication retrieval and placement action data includes the time and location data of the medication retrieval and placement actions. Based on the association of each personnel identification code with the time and location data of the medication retrieval and placement actions, it is determined whether the person has engaged in medication retrieval.

[0054] For example, if the latest action occurring at all locations under a certain personnel identification code is taking medicine, then it is determined that the person has taken medicine; as another example, if the latest action occurring at one location under a certain personnel identification code is taking medicine, but the latest action occurring at another location is putting medicine away, it is still determined that the person has taken medicine; as yet another example, if the latest action occurring at all locations under a certain personnel identification code is putting medicine away, further determination can be made based on images or videos of the time the person leaves the medicine storage area.

[0055] It is understandable that the logic for judging medication-taking behavior is not limited to the examples mentioned above.

[0056] Upon receiving a notification of a medication pickup, the S22 pharmacy monitoring platform triggers the activation of the second visual acquisition device 12, which continuously acquires images or videos of at least the area including the checkout area.

[0057] When the pharmacy monitoring platform does not determine the medication collection behavior based on the collection results of the first visual acquisition device 11 and does not trigger the second visual acquisition device 12, the second visual acquisition device 12 remains in standby mode and does not perform unnecessary image or video collection. This avoids excessive interference or unnecessary redundant identification and judgment, reduces the complexity of the platform's matching of medication collection and cashiering, and improves the accuracy of cashiering behavior identification and the effectiveness of the association between cashiering behavior and medication collection behavior in subsequent processes.

[0058] S23 Based on the trigger start-up, the second vision acquisition device 12 continuously acquires images or videos of at least the area including the checkout area to identify whether there is movement of the target object in the checkout area.

[0059] If the target object moves within the checkout area, it is treated as a detection event and uploaded to the pharmacy monitoring platform.

[0060] S24 determines whether any medication collection occurred within a preset time frame prior to the detection event. If medication collection occurred within the preset time frame prior to the detection event, it is considered that there is a possibility of a correlation between the detection event and the medication collection, and the medication collection is identified as a valid medication collection. If no medication collection occurred within the preset time frame prior to the detection event, the detection event is not used as a basis for valid medication collection, and the detection event is ignored.

[0061] Possible scenarios include multiple medication collection activities occurring within a preset timeframe prior to the detected event. When multiple medication collection activities occur within this preset timeframe, the medication collection identification process can include:

[0062] Optionally, multiple medication collection behaviors that occur within a preset time range before the detected event are all considered as suspected medication collection behaviors, awaiting manual review to determine which medication collection behavior is associated with the detected event, and the associated behavior is considered as a valid medication collection behavior, while other medication collection behaviors are considered as invalid medication collection behaviors.

[0063] Optionally, the most recent medication collection behavior within a preset time range will be associated with the detected event and identified as a valid medication collection.

[0064] Optionally, a preferred time range shorter than the preset time range is preset. If a drug retrieval behavior occurs within the preferred time range, the drug retrieval behavior occurring within the preferred time range is associated with the detected event, and the association is taken as a valid drug retrieval.

[0065] For example, based on the distance from the medicine storage area to the checkout area, the system first determines whether any medicine retrieval occurred within the time interval of 30 seconds to 5 minutes before the detection time. If a single retrieval occurred, it is associated with the detection event and considered a valid retrieval. If multiple retrievals occurred within the time interval of 30 seconds to 5 minutes before the detection time, the system further determines whether any retrieval occurred within the time interval of 30 seconds to 3 minutes, 1 minute to 5 minutes, or other feasible preferred time intervals based on the distance from the specific medicine cabinet to the checkout area before the detection event. If a single retrieval occurred, it is associated with the detection event and considered a valid retrieval.

[0066] In some embodiments, the method further includes S25, which merges the associated drug retrieval behavior and detection event determined based on S24 into a single valid drug retrieval, binds and stores the merged drug retrieval behavior data, detection event data, and other data associated with this valid drug retrieval in the database, and sets a unified query ID.

[0067] This embodiment utilizes motion detection events in the checkout area to approximate checkout actions. Checkout actions falling within a preset time are associated with medication collection behaviors to identify valid medication collection. This replaces the complex data provided by the checkout system to establish the relationship between medication collection behaviors and checkout actions, effectively addressing the impact of any checkout system on valid medication collection and improving the accuracy of valid medication collection identification.

[0068] Example 3

[0069] Please refer to Figure 4 This embodiment provides a method for identifying medication collection. This embodiment uses personnel events in the checkout area as approximations of checkout actions, and associates checkout actions falling within a preset time period with medication collection behaviors to identify valid medication collection. The method may include the following steps:

[0070] S31 continuously collects images or videos of the drug storage area through the first vision acquisition device 11. When a drug retrieval behavior is detected, the drug retrieval behavior is uploaded to the pharmacy monitoring platform in real time.

[0071] In one embodiment, the determination of medication retrieval behavior is based on the order, location, and frequency of medication retrieval and placement actions, and can also be aided by images or videos of the person at the boundary of the medication storage area. Specifically, when the first visual acquisition device 11 detects a person entering the medication storage area, it acquires the person's head and shoulder data, and establishes a personnel identification code for the person entering the medication storage area based on the head and shoulder data. The personnel identification code can be the head-to-shoulder ratio obtained based on the head and shoulder data. The system monitors medication retrieval and placement actions within the time interval from when the person enters the medication storage area to when they leave, generating medication retrieval and placement action data, and associating each instance of these data with the personnel identification code. The medication retrieval and placement action data includes the time and location data of the medication retrieval and placement actions. Based on the association of each personnel identification code with the time and location data of the medication retrieval and placement actions, it is determined whether the person has engaged in medication retrieval.

[0072] For example, if the latest action occurring at all locations under a certain personnel identification code is taking medicine, then it is determined that the person has taken medicine; as another example, if the latest action occurring at one location under a certain personnel identification code is taking medicine, but the latest action occurring at another location is putting medicine away, it is still determined that the person has taken medicine; as yet another example, if the latest action occurring at all locations under a certain personnel identification code is putting medicine away, further determination can be made based on images or videos of the time the person leaves the medicine storage area.

[0073] It is understandable that the logic for judging medication-taking behavior is not limited to the examples mentioned above.

[0074] Upon receiving a notification of a medication pickup, the S32 pharmacy monitoring platform triggers the activation of the second visual acquisition device 12, which continuously acquires images or videos of at least the area including the checkout area.

[0075] When the pharmacy monitoring platform does not determine the medication collection behavior based on the collection results of the first visual acquisition device 11 and does not trigger the second visual acquisition device 12, the second visual acquisition device 12 remains in standby mode and does not perform unnecessary image or video collection. This avoids excessive interference or unnecessary redundant identification and judgment, reduces the complexity of the platform's matching of medication collection and cashiering, and improves the accuracy of cashiering behavior identification and the effectiveness of the association between cashiering behavior and medication collection behavior in subsequent processes.

[0076] S33, after being triggered and started, the second vision acquisition device 12 continuously acquires images or videos of at least the area including the checkout area, and identifies whether there are at least two people in the checkout area, and the time that the at least two people continuously appear in the checkout area exceeds the minimum duration threshold.

[0077] If there are at least two people in the checkout area, and the duration of their presence in the checkout area exceeds the minimum duration threshold, it will be considered a personnel event and uploaded to the pharmacy monitoring platform.

[0078] S34 determines whether any medication collection occurred within a preset time frame prior to the incident. If medication collection occurred within that time frame, the incident is considered potentially related to the medication collection, and the collection is deemed valid. If no medication collection occurred within the preset time frame, the incident is not considered a valid basis for medication collection and is ignored.

[0079] Similar to Example 2, Example 3 may also involve multiple medication collection activities occurring within a preset time frame prior to a personnel incident. When multiple medication collection activities occur within this preset time frame, the identification and judgment of medication collection may include:

[0080] Optionally, multiple medication collection behaviors that occurred within a preset time frame before the incident are considered as suspected medication collection behaviors, awaiting manual review to determine which medication collection behavior is associated with the incident. The associated behavior is then considered as a valid medication collection behavior, while other medication collection behaviors are considered as invalid medication collection behaviors.

[0081] Optionally, the most recent medication collection behavior within a preset time range will be associated with personnel events and identified as a valid medication collection.

[0082] Optionally, a preferred time range shorter than the preset time range is preset. If a medication retrieval behavior occurs within the preferred time range, the medication retrieval behavior occurring within the preferred time range is associated with a personnel event, and the association is considered as a valid medication retrieval.

[0083] For example, based on the distance from the medicine storage area to the checkout area, the system first determines whether any medicine retrieval occurred within the time interval of 30 seconds to 5 minutes before the detection time. If a single retrieval occurred, it is associated with the detection event and considered a valid retrieval. If multiple retrievals occurred within the time interval of 30 seconds to 5 minutes before the detection time, the system further determines whether any retrieval occurred within the time interval of 30 seconds to 3 minutes, 1 minute to 5 minutes, or other feasible preferred time intervals based on the distance from the specific medicine cabinet to the checkout area before the detection event. If a single retrieval occurred, it is associated with the detection event and considered a valid retrieval.

[0084] In some embodiments, the method further includes S35, which merges the associated medication collection behavior and personnel event determined based on S34 into a single valid medication collection, binds and stores the merged medication collection behavior data, personnel event data, and other data associated with this valid medication collection in the database, and sets a unified query ID.

[0085] This embodiment uses personnel events in the checkout area as approximations of checkout behavior. Checkout behavior falling within a preset time is associated with medication collection behavior to identify valid medication collection. This replaces the data provided by complex checkout systems to establish the relationship between medication collection behavior and checkout behavior, and can effectively cope with the impact of any checkout system on valid medication collection.

[0086] Compared with Example 2, this example is particularly applicable to medication identification in pharmacies that do not include self-checkout. It associates the identification of at least two people, including the store clerk and the customer, as a personnel event with the medication identification behavior, thus avoiding false alarms caused by unnecessary movement of the target object in the checkout area, which may be caused by Example 2.

[0087] Example 4

[0088] Please refer to Figure 5 This embodiment provides a method for identifying medication pickup. This embodiment links the personnel trajectory identified in the medication pickup area and the cashier area to establish a correlation between medication pickup and cashier behavior. The method may include the following steps:

[0089] S41 continuously collects images or videos of the drug storage area through the first vision acquisition device 11. When a drug-taking behavior is detected, the drug-taking behavior is uploaded to the pharmacy monitoring platform in real time. At the same time, the necessary characteristics of the person taking the drug are also uploaded to the pharmacy monitoring platform.

[0090] In one embodiment, the determination of medication retrieval behavior is based on the order, location, and frequency of medication retrieval and placement actions, and can also be aided by images or videos of the person at the boundary of the medication storage area. Specifically, when the first visual acquisition device 11 detects a person entering the medication storage area, it acquires the person's head and shoulder data, and establishes a personnel identification code for the person entering the medication storage area based on the head and shoulder data. The personnel identification code can be the head-to-shoulder ratio obtained based on the head and shoulder data. The system monitors medication retrieval and placement actions within the time interval from when the person enters the medication storage area to when they leave, generating medication retrieval and placement action data, and associating each instance of these data with the personnel identification code. The medication retrieval and placement action data includes the time and location data of the medication retrieval and placement actions. Based on the association of each personnel identification code with the time and location data of the medication retrieval and placement actions, it is determined whether the person has engaged in medication retrieval.

[0091] For example, if the latest action occurring at all locations under a certain personnel identification code is taking medicine, then it is determined that the person has taken medicine; as another example, if the latest action occurring at one location under a certain personnel identification code is taking medicine, but the latest action occurring at another location is putting medicine away, it is still determined that the person has taken medicine; as yet another example, if the latest action occurring at all locations under a certain personnel identification code is putting medicine away, further determination can be made based on images or videos of the time the person leaves the medicine storage area.

[0092] It is understandable that the logic for judging medication-taking behavior is not limited to the examples mentioned above.

[0093] Upon receiving a notification of a medication pickup, the S42 pharmacy monitoring platform triggers the activation of the second vision acquisition device 12, which continuously collects images or videos of areas including at least the medication storage area and the cashier area.

[0094] When the pharmacy monitoring platform does not determine the medication collection behavior based on the collection results of the first visual acquisition device 11 and does not trigger the second visual acquisition device 12, the second visual acquisition device 12 remains in standby mode and does not perform unnecessary image or video collection. This avoids excessive interference or unnecessary redundant identification and judgment, reduces the complexity of the platform's matching of medication collection and cashiering, and improves the accuracy of cashiering behavior identification and the effectiveness of the association between cashiering behavior and medication collection behavior in subsequent processes.

[0095] S43, after being triggered and started, continuously collects images or videos of areas including at least the medicine storage area and the cashier area, as well as the necessary characteristics of the people who take medicine, and tracks the people identified by the first vision acquisition device 11.

[0096] S44 identifies whether there is a trajectory of the person moving from the medicine storage area to the checkout area. If there is a trajectory of the person moving from the medicine storage area to the checkout area, and the second vision acquisition device 12 captures the checkout action in the checkout area, then the medication retrieval behavior is considered to be associated with the checkout action, and the medication retrieval behavior is identified as a valid medication retrieval. If there is a trajectory of the person moving from the medicine storage area to the checkout area, but the second vision acquisition device 12 does not capture the checkout action in the checkout area, or if there is no trajectory of the person moving from the medicine storage area to the checkout area, but the second vision acquisition device 12 captures the checkout action in the checkout area, then the prior medication retrieval behavior will not be identified as a valid medication retrieval, nor will the checkout action or the trajectory of moving from the medicine storage area to the checkout area be used as a basis for valid medication retrieval; the checkout action or the trajectory of moving from the medicine storage area to the checkout area will be ignored.

[0097] In some embodiments, the method further includes S45, which merges the associated medication collection behavior, bank collection behavior, and trajectory determined based on S44 into a single valid medication collection, binds and stores the merged medication collection behavior data, bank collection behavior data, trajectory data, and other data associated with this valid medication collection in the database, and sets a unified query ID.

[0098] This embodiment uses the linked drug dispensing area and the cashier area to identify the personnel trajectory, thus effectively linking drug dispensing behavior with cashier behavior. Compared with embodiments 2 and 3, this embodiment uses a dedicated medicine cabinet camera to shoot the drug storage area, and links it with a global camera that includes at least the drug storage area and the cashier area to track the personnel trajectory between the drug storage area and the cashier area, improving the accuracy of the correlation between drug dispensing and cashiering. It is also applicable to the pharmacy monitoring platform of self-service checkout pharmacies.

[0099] Example 5

[0100] Please refer to Figure 6 This embodiment provides a medication dispensing identification method. This embodiment links the medication dispensing area with the cashier area to identify personnel behavior, and uses this as a correlation between medication dispensing behavior and cashier behavior. It may include the following steps:

[0101] S51 continuously collects images or videos of the drug storage area through the first vision acquisition device 11. When a drug retrieval behavior is detected, the drug retrieval behavior is uploaded to the pharmacy monitoring platform in real time. At the same time, the necessary characteristics of the person who retrieved the drug are also uploaded to the pharmacy monitoring platform.

[0102] In one embodiment, the determination of medication retrieval behavior is based on the order, location, and frequency of medication retrieval and placement actions, and can also be aided by images or videos of the person at the boundary of the medication storage area. Specifically, when the first visual acquisition device 11 detects a person entering the medication storage area, it acquires the person's head and shoulder data, and establishes a personnel identification code for the person entering the medication storage area based on the head and shoulder data. The personnel identification code can be the head-to-shoulder ratio obtained based on the head and shoulder data. The system monitors medication retrieval and placement actions within the time interval from when the person enters the medication storage area to when they leave, generating medication retrieval and placement action data, and associating each instance of these data with the personnel identification code. The medication retrieval and placement action data includes the time and location data of the medication retrieval and placement actions. Based on the association of each personnel identification code with the time and location data of the medication retrieval and placement actions, it is determined whether the person has engaged in medication retrieval.

[0103] For example, if the latest action occurring at all locations under a certain personnel identification code is taking medicine, then it is determined that the person has taken medicine; as another example, if the latest action occurring at one location under a certain personnel identification code is taking medicine, but the latest action occurring at another location is putting medicine away, it is still determined that the person has taken medicine; as yet another example, if the latest action occurring at all locations under a certain personnel identification code is putting medicine away, further determination can be made based on images or videos of the time the person leaves the medicine storage area.

[0104] It is understandable that the logic for judging medication-taking behavior is not limited to the examples mentioned above.

[0105] Upon receiving a notification of a medication pickup, the S52 pharmacy monitoring platform triggers the activation of the second vision acquisition device 12, which continuously collects images or videos of areas including at least the medication storage area and the cashier area.

[0106] When the pharmacy monitoring platform does not determine the medication collection behavior based on the collection results of the first visual acquisition device 11 and does not trigger the second visual acquisition device 12, the second visual acquisition device 12 remains in standby mode and does not perform unnecessary image or video collection. This avoids excessive interference or unnecessary redundant identification and judgment, reduces the complexity of the platform's matching of medication collection and cashiering, and improves the accuracy of cashiering behavior identification and the effectiveness of the association between cashiering behavior and medication collection behavior in subsequent processes.

[0107] S53, after being triggered and started, continuously collects images or videos of areas including at least the drug storage area and the cashier area, as well as the necessary characteristics of the person who takes the medicine, and tracks the person identified by the first vision acquisition device 11.

[0108] S54 identifies whether there is a trajectory of the person moving from the medicine storage area to the cashier area. If there is a trajectory of the person moving from the medicine storage area to the cashier area, and the second vision acquisition device 12 acquires the cashier behavior in the cashier area, and there is a person event at the same time, then it is considered that the medicine retrieval behavior is related to the cashier behavior, and the medicine retrieval behavior is identified as a valid medicine retrieval.

[0109] The personnel event recognition in this embodiment is similar to that in embodiment 3: it identifies whether there are at least two people in the checkout area and whether the duration of the at least two people in the checkout area exceeds the minimum duration threshold; if there are at least two people in the checkout area and the duration of the at least two people in the checkout area exceeds the minimum duration threshold.

[0110] It is understood that identifying whether a person has moved from the drug storage area to the checkout area can be based on the trajectory of one person or multiple people. If at least one of the at least two people involved in a personnel incident has a trajectory from the drug storage area to the checkout area, the medication retrieval activity can be considered a valid retrieval, rather than requiring all individuals involved in the incident to have a trajectory from the drug storage area to the checkout area.

[0111] If there is a trajectory of the person moving from the medicine storage area to the checkout area, but the second visual acquisition device 12 does not capture the checkout behavior in the checkout area, or if there is no trajectory of the person moving from the medicine storage area to the checkout area, but the second visual acquisition device 12 captures the checkout behavior in the checkout area, or if there is a trajectory of the person moving from the medicine storage area to the checkout area and the checkout behavior, but no personnel event, the prior medication retrieval behavior will not be identified as valid medication retrieval, nor will the checkout behavior or the trajectory of moving from the medicine storage area to the checkout area or the personnel event be used as the basis for valid medication retrieval. The checkout behavior or the trajectory of moving from the medicine storage area to the checkout area will be ignored.

[0112] In some embodiments, the method further includes S55 merging the associated medication collection behavior and personnel event determined based on S54 into a single valid medication collection, binding and storing the merged medication collection behavior data, collection behavior data, trajectory data, personnel event data, and other data associated with this valid medication collection in the database, and setting a unified query ID.

[0113] This embodiment links the identification of personnel trajectories in the medication dispensing area and the cashier area, and combines continuously occurring personnel events to effectively correlate medication dispensing behavior with cashiering behavior. Compared with embodiment 4, this embodiment uses a dedicated medicine cabinet camera to photograph the medication storage area, and links it with a global camera that includes at least the medication storage area and the cashier area to track personnel trajectories between the medication storage area and the cashier area. This improves the accuracy of the correlation between medication dispensing and cashiering, and is more suitable for monitoring pharmacies that do not include self-checkout.

[0114] Example 6

[0115] Please refer to Figure 7 This embodiment provides a medicine identification device, including:

[0116] A first acquisition module is configured to acquire a first target area and obtain a first image or a first video, wherein the first target area includes a drug storage area.

[0117] The second acquisition module is configured to acquire a second target area and obtain a second image or a second video, wherein the second target area includes a cashier area.

[0118] An image processing module is configured to identify whether a drug-taking behavior exists based on the first image or the first video, and is configured to identify whether a bank-collecting behavior exists based on the second image or the second video;

[0119] A trigger module is configured to trigger the second acquisition module to collect the second target area and obtain the second image or the second video when the drug-taking behavior occurs.

[0120] The judgment module is configured to determine whether the medication retrieval behavior is associated with the bank collection point; if the medication retrieval behavior is associated with the bank collection point, it is identified as a medication retrieval.

[0121] In some embodiments, the second target area acquired by the second acquisition module further includes the drug storage area;

[0122] The medication identification device further includes a tracking module, which is configured to track the target object in the second image or the second video from the medication storage area and determine whether the target object has a trajectory from the medication storage area to the cashier area.

[0123] The drug identification device provided in this embodiment is used to perform the drug identification method as described in Embodiments 1-5.

[0124] Example 7

[0125] This embodiment provides an electronic device that can be applied to video and image acquisition devices, video and image storage devices, video and image analysis devices, etc., capable of performing drug identification methods. The electronic device provided in this embodiment can implement all the processes of the drug identification method described in the above embodiments and achieve the same beneficial effects. To avoid repetition, further details are omitted here.

[0126] Example 8

[0127] This embodiment provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the various processes of the drug identification method or the application-side drug identification method provided in this embodiment of the invention, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0128] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0129] Those skilled in the art will understand that the above embodiments are specific examples of implementing the present invention, and in practical applications, various changes in form and detail may be made without departing from the spirit and scope of the present invention.

Claims

1. A method for identifying medication, characterized in that, include: Based on a first image or video of a first target area, it is determined whether a drug-taking behavior exists, wherein the first target area includes a drug storage area; If the medication dispensing behavior occurs, a second image or second video of the second target area is triggered, and the second target area includes the cashier area. If a bank collection is identified based on the second image or the second video, determine whether the medication collection behavior is related to the bank collection behavior. If the medication collection is associated with the bank, it is identified as a valid medication collection. The process involves acquiring head and shoulder data of individuals, and establishing personnel identification codes for those entering the medication storage area based on this data. The personnel identification code is the head-to-shoulder ratio obtained from the head and shoulder data. Each personnel identification code is associated with the time data of medication retrieval and medication placement to determine if a person has retrieved medication. The process also identifies whether at least two individuals are present in the checkout area, and if these two individuals remain in the checkout area for a duration exceeding a minimum threshold, classifying this as a personnel event. The determination of medication retrieval is based on the order, location, and frequency of medication retrieval and placement. In a personnel event involving at least two individuals, at least one of them must have retrieved medication from the medication storage area. The trajectory from the drug storage area to the checkout area is considered a valid drug retrieval. The system monitors drug retrieval and dispensing behaviors from the time a person enters the drug storage area until they leave, generating drug retrieval and dispensing data and associating these data with the person's identification code. The drug retrieval and dispensing data includes the time and location of each retrieval and dispensing behavior. The identified associated drug retrieval behaviors and personnel events are merged into a single valid drug retrieval. This merged data, along with checkout behavior data, trajectory data, personnel event data, and other data associated with this valid drug retrieval, is then bound and stored in the database.

2. The drug identification method according to claim 1, characterized in that, If the time interval between the medication collection and the bank collection falls within the first time range, then it is determined that the medication collection and the bank collection are related.

3. The drug identification method according to claim 2, characterized in that, If at least two target objects are present in the checkout area for a continuous period exceeding a first time threshold, then it is determined that the medication collection behavior is associated with the checkout behavior.

4. The drug identification method according to claim 1, characterized in that, The second target area also includes a medicine storage area. The target object in the second image or the second video is tracked starting from the medicine storage area. If the target object has a trajectory from the medicine storage area to the checkout area, it is determined that the medicine retrieval behavior is related to the checkout behavior.

5. According to the drug retrieval identification method of claim 4, if at least one of the target objects has a trajectory of moving from the drug storage area to the checkout area, and at least two of the target objects continuously appear in the checkout area for more than a second time threshold, then it is determined that the drug retrieval behavior is associated with the checkout behavior.

6. The drug identification method according to claim 1, characterized in that, The combined drug retrieval behavior and the bank receipt are considered as characteristics of a single drug retrieval.

7. A medicine dispensing and identification device, characterized in that, include: A first acquisition module is configured to acquire a first target area and obtain a first image or a first video, wherein the first target area includes a drug storage area. The second acquisition module is configured to acquire a second target area and obtain a second image or a second video, wherein the second target area includes a cashier area. An image processing module is configured to identify whether a drug-taking behavior exists based on the first image or the first video, and is configured to identify whether a bank-collecting behavior exists based on the second image or the second video; A trigger module is configured to trigger the second acquisition module to collect the second target area and obtain the second image or the second video when the drug-taking behavior occurs; The judgment module is configured to determine whether the medication retrieval behavior is associated with the checkout area; if the medication retrieval behavior is associated with the checkout area, it is identified as a medication retrieval; wherein, head and shoulder data of the personnel are acquired, and a personnel identification code for the personnel entering the medication storage area is established based on the head and shoulder data; the personnel identification code is the head-to-shoulder ratio obtained based on the head and shoulder data; based on the time data of the medication retrieval behavior and the time data of the medication placement behavior associated with each personnel identification code, it is determined whether the personnel have performed a medication retrieval behavior; it is also determined whether there are at least two personnel in the checkout area, and if the duration of the at least two personnel continuously appearing in the checkout area exceeds a minimum duration threshold, it is considered as a personnel event; the determination of medication retrieval behavior is based on the order of the personnel's medication retrieval behavior and medication placement behavior. The location and frequency are determined; if at least one of the at least two individuals in a personnel incident has a trajectory from the drug storage area to the checkout area, the drug retrieval behavior is considered valid; the drug retrieval and dispensing behaviors are monitored from the time a person enters the drug storage area to the time they leave the drug storage area, generating drug retrieval and dispensing behavior data, and associating each drug retrieval and dispensing behavior data with the personnel identification code; the drug retrieval and dispensing behavior data includes the time and location data of the drug retrieval behavior, and the time and location data of the drug dispensing behavior; the determined associated drug retrieval behaviors and personnel incidents are merged into a single valid drug retrieval, and the merged drug retrieval behavior data, checkout behavior data, trajectory data, personnel incident data, and other data associated with this valid drug retrieval are bound and stored in the database.

8. The drug identification device according to claim 7, characterized in that, The second target area acquired by the second acquisition module also includes the drug storage area; The medication identification device further includes a tracking module, which is configured to track the target object in the second image or the second video from the medication storage area and determine whether the target object has a trajectory from the medication storage area to the cashier area.

9. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the steps of the drug identification method as described in any one of claims 1 to 6.

10. A storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the steps of the drug identification method as described in any one of claims 1 to 6.

Images