Vending device and vending method

Abstract

The present disclosure provides a vending device and a vending method, which relate to the computer field. Based on the detection information of whether the shopping area is invaded, and the detection information of whether the boundary area between the area for sale and other areas is crossed, the occurrence of shopping-related events such as the user entering or leaving the shopping area, the boundary area being crossed, etc., is determined, and the associated shopping-related operations are triggered in combination with the detected shopping information, so as to realize the vending. Thus, at least one problem existing in the vending business, such as abnormal settlement and large amount of calculation for item identification, is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computers, and in particular to a vending device and a vending method. Background Art

[0002] With the development of computer technology, automatic vending has become possible. People can take items from vending equipment, identify the taken items based on the gravity of the items or the image of the items, and settle the bill. Summary of the invention

[0003] Some embodiments of the present disclosure provide a vending device, comprising:

[0004] processor,

[0005] as well as

[0006] at least one of a shopping area intrusion detection sensor and a demarcation area crossing detection sensor,

[0007] Wherein, the processor is electrically connected to the shopping area intrusion detection sensor, and the processor is electrically connected to the boundary area penetration detection sensor;

[0008] The shopping area intrusion detection sensor is configured to detect information on whether the shopping area has been invaded;

[0009] The boundary area crossing detection sensor is configured to detect whether the boundary area between the for-sale item placement area and other areas is crossed;

[0010] The processor is configured to determine the occurrence of a predetermined event based on detection information of at least one of the shopping area intrusion detection sensor and the demarcation area crossing detection sensor, and trigger an operation associated with the predetermined event.

[0011] In some embodiments, the processor is configured to determine the occurrence of an event of a user entering or leaving a shopping area based on detection information of the shopping area intrusion detection sensor.

[0012] In some embodiments, the processor is configured to determine the occurrence of an event of a demarcation area being crossed or an event of a direction in which the demarcation area is crossed based on detection information of the demarcation area crossing detection sensor.

[0013] In some embodiments, the association relationship between the predetermined event and the operation is configured as at least one of the following relationships:

[0014] An operation of associating an event of a user entering a shopping area with item recognition or an operation of associating a cache of sensing information used for item recognition;

[0015] The event of the user leaving the shopping area is associated with the item settlement operation;

[0016] The event of the demarcation area being crossed is associated with an item identification operation or an associated purchase item addition or deletion operation;

[0017] The demarcation area is crossed by the event in the direction associated with the item recognition operation or the addition or deletion operation associated with the purchase item.

[0018] In some embodiments, an event of a user entering a shopping area is associated with an operation of item recognition based on at least one of item gravity and item image.

[0019] In some embodiments, the event of the demarcation area being crossed is associated with an operation of item recognition based on at least one of item gravity and item image, or an operation of adding or deleting purchase items based on at least one of item gravity and item image.

[0020] In some embodiments, the event of the demarcation area being crossed in a direction is associated with an operation of item recognition based on at least one of item gravity and item image, or is associated with an operation of adding or deleting purchase items based on at least one of item gravity and item image.

[0021] In some embodiments, the processor is configured to retrieve images of objects within a preset range based on the time when the boundary area is crossed or the time when the boundary area is crossed in the direction of the crossing, for object identification or the addition or deletion of purchased objects.

[0022] In some embodiments, the processor is configured to determine the occurrence of an abnormal event based on detection information of at least one of the shopping area intrusion detection sensor, the boundary area crossing detection sensor, and the shopping information detection sensor, and trigger an operation associated with the abnormal event.

[0023] The shopping information detection sensor is configured to detect information about items purchased by a user from a vending device that displays items for sale.

[0024] In some embodiments, the boundary area is set according to a plane whose angle with the front of the vending equipment where the items for sale are placed is smaller than a preset angle, and the distance between the boundary area and the front of the vending equipment is smaller than a preset distance.

[0025] In some embodiments, the shopping area is set according to a preset range area where the vending equipment is located.

[0026] In some embodiments, the shopping area intrusion detection sensor is a first image sensor, a photoelectric sensor working in a reflection mode, an ultrasonic ranging sensor, or a human body sensing switch, wherein the detection area of ​​the first image sensor is aligned with the shopping area.

[0027] In some embodiments, the boundary area passing detection sensor is a second image sensor, a photoelectric sensor in a through-beam working mode, wherein the detection area of ​​the second image sensor is aligned with the boundary area.

[0028] Some embodiments of the present disclosure provide a vending method, which is applied to a vending device, comprising:

[0029] Obtain detection information on whether the shopping area has been invaded;

[0030] Obtain detection information on whether the boundary area between the area where items for sale are placed and other areas has been crossed;

[0031] determining an occurrence of a predetermined event based on at least one detection information; and

[0032] An operation associated with the predetermined event is triggered to implement the vending.

[0033] In some embodiments, the occurrence of an event of a user entering or leaving a shopping area is determined based on detection information of whether the shopping area is invaded.

[0034] In some embodiments, in the event of a user entering a shopping area, an operation of caching sensing information used for item identification is triggered;

[0035] When the user leaves the shopping area, the item checkout operation is triggered.

[0036] In some embodiments, based on the detection information of whether the boundary area between the for-sale item placement area and other areas has been crossed, the occurrence of the event of the boundary area being crossed or the event of the direction of the boundary area being crossed is determined.

[0037] In some embodiments, when an event that a demarcation area is crossed occurs, an operation of object identification is triggered;

[0038] When an event occurs in which the demarcation area is crossed, the item recognition operation is triggered, as well as the addition or deletion operation of the purchased items.

[0039] In some embodiments, an operation of taking away or putting back an item is determined by combining at least one of the item gravity and the item image, and an operation of adding or deleting the purchased item is triggered based on the determined operation of taking away or putting back the item.

[0040] In some embodiments, based on the time when the boundary area is crossed or the time when the boundary area is crossed in the direction of the crossing, the image of the object within the preset range is retrieved, and the object is identified based on the retrieved image of the object or in combination with the gravity of the object.

[0041] Some embodiments of the present disclosure provide a computer-readable storage medium having a computer program stored thereon, which implements the vending method of any one of the embodiments when executed by a processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The following is a brief introduction to the drawings required for use in the embodiments or related technical descriptions. The present disclosure can be more clearly understood according to the following detailed description with reference to the drawings.

[0043] Obviously, the drawings described below are only some embodiments of the present disclosure, and a person skilled in the art can obtain other drawings based on these drawings without creative work.

[0044] Figure 1 Schematic diagrams showing some embodiments of the vending device 10 of the present disclosure.

[0045] Figure 2 Schematic diagram showing some examples of shopping areas 21 and demarcation areas 22 relative to vending equipment 23 of the present disclosure.

[0046] Figure 3 A schematic diagram showing the detection principle of the photoelectric sensor 30 working in a reflective mode.

[0047] Figure 4 FIG. 4 is a schematic diagram showing the detection principle of the ultrasonic distance measuring sensor 40 .

[0048] Figure 5 A schematic diagram showing the detection principle of the human body sensing switch 50 is shown.

[0049] Figure 6 A schematic diagram showing the detection principle of the photoelectric sensor 60 in through-beam working mode.

[0050] Figure 7 A flowchart diagram showing some embodiments of the vending method of the present disclosure.

[0051] Figure 8 A flowchart diagram showing some embodiments of the vending method of the present disclosure. DETAILED DESCRIPTION

[0052] The technical solutions in the embodiments of the present disclosure will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present disclosure.

[0053] Figure 1Schematic diagrams showing some embodiments of the vending device 10 of the present disclosure.

[0054] like Figure 1 As shown, the vending device 10 of this embodiment includes: a processor 11, and various sensors electrically connected to the processor 11, such as a shopping area intrusion detection sensor 12, a boundary area crossing detection sensor 13, a shopping information detection sensor 14, etc., but is not limited to the examples given.

[0055] The items for sale can be placed on the vending equipment. The vending equipment can be, for example, an open shelf or a closed cabinet. The above sensors can be installed on the vending equipment. The sensors can be installed at appropriate locations of the vending equipment according to their functions. The processor 11 can be integrated in the vending equipment.

[0056] The shopping area refers to an area near the vending machine where there is a high probability of shopping behavior, and can be set in advance. For example, the shopping area is set according to a preset range area of ​​the ground where the vending machine is located.

[0057] The boundary area between the area where items for sale are placed and other areas is easy to understand. With the boundary area as the boundary, items for sale are placed on one side of the boundary area, while items for sale are not placed on the other side. The boundary area can be set according to a plane whose angle with the front of the vending equipment is less than a preset angle, and the distance between the boundary area and the front of the vending equipment is less than a preset distance. If the outer contour of the vending equipment is or approximates to a rectangular parallelepiped, the outer surface of the vending equipment facing the user side (i.e., the side where the user can take the goods for sale) is regarded as the front of the vending equipment. For example, the boundary area can be set to a plane perpendicular to the ground and parallel to the front of the vending equipment, or the boundary area can be set to the front of the vending equipment.

[0058] Figure 2 Schematic diagrams showing some examples of the shopping area 21 and the demarcation area 22 of the present disclosure relative to the vending equipment 23. In some embodiments, the demarcation area can be set as an area closer to the vending equipment than the outer edge of the shopping area.

[0059] The shopping area intrusion detection sensor 12 is configured to detect whether the shopping area is invaded. The shopping area intrusion detection sensor 12 can be, for example, an image sensor (set as a first image sensor), a photoelectric sensor in a reflection working mode, an ultrasonic distance sensor, or a human body sensing switch, etc. The corresponding detection information can be image information, distance information, spectrum change information, etc. These sensors are described below.

[0060] The first image sensor may be, for example, a camera or other imaging device. The position and angle of the first image sensor may be adjusted so that the detection area (i.e., the shooting area) of the first image sensor is aligned with the shopping area to shoot an image of the shopping area. Based on the real-time image of the shopping area shot by the first image sensor, it may be determined whether the shopping area has been invaded. For example,

[0061] The real-time image of the shopping area captured by the first image sensor is compared with a reference image of the shopping area that has not been invaded. If a new object (such as a person) appears in the image of the shopping area, the shopping area is considered to be invaded by the object. In addition, other area intrusion detection technologies can also be used to implement intrusion detection of shopping areas.

[0062] The detection principle of the photoelectric sensor 30 in the reflective working mode is as follows: Figure 3 As shown. The photoelectric sensor 30 in the reflective working mode includes a light emitter 31 and a light receiver 32. The light emitter 31 emits a light signal, and the light signal irradiated to a person is reflected and received by the light receiver 32. The distance between the person and the vending machine is determined based on the time interval between the light signal received by the light receiver 32 and the light signal emitted by the light emitter 31, and whether the shopping area is invaded can be determined based on the distance.

[0063] The detection principle of the ultrasonic distance measuring sensor 40 is as follows: Figure 4 As shown. The ultrasonic distance measuring sensor 40 includes a transmitter 41 and a receiver 42. The transmitter 41 transmits ultrasonic waves, and the ultrasonic waves that hit a person are reflected and received by the receiver 42. The distance between the person and the vending machine is determined based on the time interval between the ultrasonic waves received by the receiver 42 and the ultrasonic waves emitted by the transmitter 41, and whether the shopping area is invaded can be determined based on the distance.

[0064] The detection principle of the human body induction switch 50 is as follows: Figure 5 As shown. The human body sensing switch 50 can be implemented based on infrared technology, for example. When a person enters the sensing range, the human body sensing switch 50 can detect changes in the infrared spectrum. Therefore, when the human body sensing switch 50 detects changes in the infrared spectrum, it is considered that the shopping area has been invaded.

[0065] The boundary area crossing detection sensor 13 is configured to detect whether the boundary area between the for-sale item placement area and other areas is crossed. The boundary area crossing detection sensor 13 can be, for example, an image sensor (set as a second image sensor) or a photoelectric sensor in a beaming working mode, and the corresponding detection information can be image information, signal reception information, etc. These sensors are described below.

[0066] The second image sensor can be, for example, a camera or other imaging device. The position and angle of the second image sensor installation are adjusted so that the detection area (i.e., the shooting area) of the second image sensor is aligned with the demarcation area so as to capture an image of the demarcation area. Based on the image of the demarcation area captured by the second image sensor, it can be determined whether the demarcation area is crossed. For example, at least two frames of images of the demarcation area are continuously acquired. If an object (such as a person or a hand) appears on one side of the demarcation area in some frames and on the other side of the demarcation area in other frames, it is considered that the demarcation area is crossed by the object. At the same time, the direction in which the demarcation area is crossed can also be determined. For example, in the time period T1, the object appears on one side a of the demarcation area, and in the next later time period T2, the object appears on the other side b of the demarcation area, then the direction in which the demarcation area is crossed is the direction from a to b.

[0067] The detection principle of the photoelectric sensor 60 in the beam working mode is as follows: Figure 6 As shown. The photoelectric sensor 60 in the opposite beam working mode includes a plurality of light emitters 61 and light receivers 62 arranged in pairs. These light emitters 61 and light receivers 62 arranged in pairs form a light curtain covering the boundary area. When the light curtain is not blocked, the light signal emitted by the light emitter 61 can be received by the corresponding light receiver 62. When the light curtain is blocked, the light signal emitted by the light emitter 61 cannot be received by the corresponding light receiver 62. Therefore, according to the reception of the light signal by the light receiver 62, it can be determined whether the boundary area has been crossed.

[0068] The shopping information detection sensor 14 is configured to detect information about items purchased by a user from a vending device that displays items for sale. The shopping information detection sensor 14 may be, for example, a gravity sensor, an image sensor (set as a third image sensor), or a combination thereof.

[0069] For example, a gravity sensor can be set under each layer of the tray of the vending equipment to detect the gravity information of the items in the tray. The items can be identified according to the changes in the gravity of the items. For example, if the gravity of the items for sale detected by a gravity sensor decreases, and the amount of gravity decrease is consistent with the gravity of an item a among the items on the tray where it is located (or, with the items of the vending equipment), it is considered that item a is taken away. If the user wants to buy item a, he can add item a to the user's virtual shopping cart for settlement. If the gravity of the items for sale detected by a gravity sensor increases, and the amount of gravity increase is consistent with the gravity of an item b among the items of the vending equipment, it is considered that item b is put back. If the user does not want to buy item b, he can delete item b from the user's virtual shopping cart.

[0070] The third image sensor can be set, for example, at a position in the vending machine where the item placement and taking can be photographed. The image taken by the third image sensor is identified to determine the item in the image. For example, the image features of each item for sale are obtained, and the image features of the item in the image to be identified are compared with the image features of the item for sale. The higher the similarity, the greater the possibility that the item in the image to be identified is the corresponding item for sale. For example, the item in the image to be identified is determined as the item for sale with the highest similarity. In some embodiments, for example, a dual convolutional neural network can be used for item recognition. The image to be detected is input into the first dual convolutional neural network to detect whether there is an item in the image. If an item is detected, the item area is cropped and input into the second convolutional neural network to detect what kind of item it is. In addition, the trajectory of the item can be further tracked through multiple frames of images of the item, and it can be determined based on the trajectory whether the item is an item that the user wants to buy and is taken away, or an item that the user does not want to buy and is put back.

[0071] In addition, the above-mentioned object gravity recognition and object image recognition can also be combined. For example, compare whether the object recognized by gravity is consistent with the object recognized by image. If they are consistent, the confidence of the object recognition result is relatively high. If the image recognition result includes multiple objects with a small confidence difference, but the gravity of these objects may be different, then the gravity change can be combined, and the object in the image recognition result that matches the gravity change can be used as the final recognition result. If a certain gravity change corresponds to multiple objects, but the images of these objects may be different, then the image recognition result can be combined, and the object in the gravity recognition result that matches the image recognition result can be used as the final recognition result.

[0072] Based on the detection information of at least one of the above sensors 12 , 13 , 14 , the processor 11 may determine the occurrence of a predetermined event and may further trigger an operation associated with the predetermined event.

[0073] In some embodiments, the processor 11 is configured to determine the occurrence of an event of a user entering or leaving a shopping area based on detection information of the shopping area intrusion detection sensor 12 .

[0074] For example, based on the detection information of various shopping area intrusion detection sensors 12, it can be determined whether the shopping area is invaded. If the shopping area is invaded by a user at a certain time t1, it is considered that an event of the user entering the shopping area occurred at time t1. If the behavior of the shopping area being invaded by the user disappears at the next time t2, it is considered that an event of the user leaving the shopping area occurred at time t2.

[0075] In some embodiments, the processor 11 is configured to determine the occurrence of an event of a demarcation area being crossed or an event of a demarcation area being crossed in a direction based on the detection information of the demarcation area crossing detection sensor 13 .

[0076] As described above, based on the detection information of the sensor 13 of the second image sensor type, the occurrence of the event that the boundary area is crossed and the event in the direction of the boundary area being crossed can be determined. Based on the detection information of the sensor 13 of the photoelectric sensor 60 type of the opposite beam working mode, the occurrence of the event that the boundary area is crossed can be determined.

[0077] In some embodiments, according to business requirements, the association relationship between the predetermined event and the operation can be set as follows:

[0078] 1) The event of the user entering the shopping area is associated with the operation of item recognition, that is, the operation of item recognition is performed after the event of the user entering the shopping area occurs. Thus, the calculation amount of item recognition is reduced and the speed of item recognition is improved.

[0079] As mentioned above, the object recognition is realized based on at least one of the object gravity and the object image, that is, the object recognition is realized based on the object gravity, the object recognition is realized based on the object image, and the object recognition is realized based on the combination of the object gravity and the object image. The specific object recognition method will not be described here or later.

[0080] 2) The operation of caching the sensing information used for item identification by associating the event of the user entering the shopping area, that is, after the event of the user entering the shopping area occurs, the sensing information used for item identification, such as item image information or item gravity information, is cached. Thus, the need for item identification can be met while reducing the amount of cached information.

[0081] 3) The operation of item settlement is associated with the event of the user leaving the shopping area, that is, the operation of item settlement is performed after the event of the user leaving the shopping area occurs.

[0082] Thus, the settlement operation in the vending business scenario can be carried out smoothly. Some related technologies use the closing signal of the door of the vending device to trigger the settlement operation, which may result in abnormal situations where settlement cannot be made. For example, the user does not close the door of the vending device after purchasing the goods, or the door lock cannot be closed normally due to some mechanical or electrical failures.

[0083] 4) The event of the boundary area being crossed is associated with an operation of item recognition or an operation of adding or deleting purchased items, that is, after the event of the boundary area being crossed occurs, the operation of item recognition or the operation of adding or deleting purchased items is performed.

[0084] As mentioned above, the demarcation area is an area very close to the items for sale, and images of items being taken away or put back appear densely near the events where the demarcation area is crossed. Therefore, the amount of calculation for item recognition is reduced and the speed of item recognition is increased.

[0085] 5) The event of the direction in which the boundary area is crossed is associated with the operation of item recognition or the operation of adding or deleting purchased items. That is, after the event of the direction in which the boundary area is crossed occurs, the operation of item recognition or the operation of adding or deleting purchased items is performed. Thus, the amount of calculation for item recognition is reduced and the speed of item recognition is increased.

[0086] In addition, based on the time when the boundary area is crossed or the time when the boundary area is crossed, the images of items within a preset range are retrieved for item identification or the addition or deletion of purchased items. For example, the images of items within a preset time range before and after the relevant event is retrieved, or the images of items within a preset frame range before and after the relevant event is retrieved. Thus, the amount of calculation for item identification is further reduced.

[0087] In addition, the processor 11 is configured to determine the occurrence of an abnormal event based on detection information of at least one of the shopping area intrusion detection sensor 12, the boundary area crossing detection sensor 13 and the shopping information detection sensor 14, and trigger an operation associated with the abnormal event.

[0088] Abnormal events and their associated operations can be set according to business needs. There are many situations for abnormal events and their associated operations. Here are some examples. When the purchased items are identified based on the shopping information detection sensor 14, if the item identification result based on the gravity of the item is inconsistent with the item identification result based on the item image, it means that an abnormal item identification event has occurred. You can alarm and ask the staff to handle it, or ask the purchasing user to manually select the purchased items. Alternatively, based on the detection information of the boundary area crossing detection sensor 13, it is determined that the boundary area has not been crossed, and item identification can be omitted. For example, item identification based on the item image is not performed, thereby effectively shielding the misidentification caused by abnormal events such as image misdetection or image jitter. Based on the detection information of the shopping area intrusion detection sensor 12, it is identified that dangerous items have entered the shopping area, and an alarm can be issued. Based on the detection information of the boundary area crossing detection sensor 13, it is identified that dangerous items have been put back into the vending machine, and an alarm can be issued.

[0089] Figure 7 A flowchart diagram showing some embodiments of the vending method of the present disclosure applied to vending equipment.

[0090] like Figure 7 As shown, the method 70 of this embodiment includes steps 71 to 72. The method may be executed by the processor 11, for example.

[0091] Step 71, obtaining detection information on whether the shopping area has been invaded.

[0092] For example, the processor 11 receives and analyzes the detection information of whether the shopping area is invaded, which is transmitted by the shopping area intrusion detection sensor 12, thereby knowing whether the shopping area is invaded.

[0093] Step 72, obtaining detection information of whether the boundary area between the area for sale and other areas is crossed.

[0094] For example, the processor 11 receives and analyzes the detection information of whether the boundary area is crossed transmitted by the boundary area crossing detection sensor 13, thereby knowing whether the boundary area is crossed and the crossing direction and other information.

[0095] Step 73: determining the occurrence of a predetermined event based on at least one detection information.

[0096] As described above, the processor 11 determines the occurrence of an event in which a user enters or leaves the shopping area based on the detection information of whether the shopping area has been invaded, and determines the occurrence of an event in which a boundary area has been crossed or an event in which a boundary area has been crossed, based on the detection information of whether the boundary area between the area where items for sale are placed and other areas has been crossed.

[0097] Step 74: triggering an operation associated with the predetermined event to implement automatic vending.

[0098] As mentioned above, the processor 11 can, for example, perform operations associated with events according to the preset association relationship between predetermined events and operations, such as item identification, adding or deleting purchased items, settlement and other sales-related operations.

[0099] For example, when a user enters a shopping area, an operation of caching images or sensing information such as gravity used for item recognition is triggered. When a user leaves a shopping area, an operation of item settlement is triggered. When a boundary area is crossed, an operation of item recognition is triggered. When an event occurs in the direction of a boundary area being crossed, an operation of item recognition and an operation of adding or deleting purchased items are triggered. If the boundary area is crossed in a direction away from the vending device, it means that the user may have taken the item from the vending device, and an operation of adding the purchased item can be performed. If the boundary area is crossed in a direction toward the vending device, it means that the user may have put the item back into the vending device, and an operation of deleting the purchased item can be performed.

[0100] In addition, when the type of sensor can only determine that the boundary area has been crossed but cannot determine the direction in which the boundary area has been crossed, the operation of taking the item away or putting the item back can be determined in combination with at least one of the gravity of the item and the image of the item, and based on the determined operation of taking the item away or putting the item back, the operation of adding or deleting the purchased item is triggered.

[0101] For example, when the demarcation area is crossed, the gravity of the items for sale detected by the gravity sensor decreases, indicating that the user may take the items from the vending device, triggering the add operation of the purchased items, and adding the taken items to the user's virtual shopping cart. When the demarcation area is crossed, the gravity of the items for sale detected by the gravity sensor increases, indicating that the user may put the items back to the vending device, triggering the delete operation of the purchased items, and deleting the returned items from the user's virtual shopping cart.

[0102] For example, if the boundary area is crossed and the direction of the object trajectory based on image recognition is away from the vending device, it means that the user may have taken the item from the vending device, and the operation of adding the purchased item can be performed. If the direction of the object trajectory based on image recognition is toward the vending device, it means that the user may have put the item back into the vending device, and the operation of deleting the purchased item can be performed.

[0103] In addition, based on the time when the boundary area is crossed or the time when the boundary area is crossed in the direction, the image of the object within the preset range can be retrieved, and the object recognition can be performed based on the retrieved object image, or based on the retrieved object image combined with the object gravity. Thus, the amount of calculation for object recognition can be further reduced.

[0104] Figure 8 A flowchart diagram showing some embodiments of the vending method of the present disclosure applied to vending equipment.

[0105] like Figure 8 As shown, the method 80 of this embodiment includes:

[0106] Step 81, using the detection information of the shopping area intrusion detection sensor 12 to determine whether the user has entered the preset shopping area, if no user has entered the preset shopping area, continue to execute this judgment step, if the user has entered the preset shopping area, you can execute step 82.

[0107] Step 82 , start caching the detection information of the shopping information detection sensor 14 .

[0108] When the shopping information detection sensor 14 is a gravity sensor, the item gravity detection information starts to be cached. When the shopping information detection sensor 14 is a third image sensor, the item image detection information starts to be cached.

[0109] Since the computing resources consumed by item recognition based on the gravity of the item are relatively few, and the computing resources consumed by item recognition based on the item image are relatively many, after step 82, step 84a can be directly executed. Of course, step 83 can also be continued after step 82. In the case where the demarcation area is passed through by a human hand, step 84a is then executed. The latter can save more computing resources compared to the former.

[0110] Step 83: Use the detection information of the demarcation area passing through the detection sensor 13 to determine whether the demarcation area is passed through by a human hand and information such as the passing direction. If the demarcation area is passed through by a human hand, execute step 84b. If the demarcation area is not passed through by a human hand, execute step 85.

[0111] Step 84a: Perform item recognition based on the cached item gravity detection information. The method for item recognition based on item gravity is as described above and will not be elaborated here.

[0112] Step 84b: If the demarcation area is passed through by a human hand, it indicates that the user is about to operate on the item for sale, such as taking the item for sale, putting the item for sale back, etc. At this time, the item images within a preset time range cached can be retrieved, and item recognition is performed based on the retrieved item images.

[0113] For example, if the direction of the human hand passing through is the direction approaching the vending device, it indicates that an item may be placed. Retrieve the images of the B1 seconds before the human hand passes through the demarcation area and the A1 seconds after passing through the demarcation area (B1 > A1) for item recognition. If the direction of the human hand passing through is the direction away from the vending device, it indicates that an item may be taken away. Retrieve the images of the B2 seconds before the human hand passes through the demarcation area and the A2 seconds after passing through the demarcation area (B2 < A2) for item recognition.

[0114] Since only the items within a certain period of time need to be processed, the item recognition speed can be increased and computing resources can be saved.

[0115] Step 85: Fuse the various recognition results of steps 83, 84a, and 84b.

[0116] An exemplary fusion method is that if the demarcation area is not passed through, no item recognition is performed. For example, no item recognition based on the item image is performed, thereby effectively shielding the misrecognition caused by abnormal events such as image misdetection or image jitter.

[0117] Another exemplary fusion method is to comprehensively determine the item recognition result by combining the item recognition result based on item gravity and the item recognition result based on item image. The comprehensive determination method is as described above and will not be elaborated here.

[0118] If the various recognition results are consistent, for example, the boundary area is crossed and the object recognition results of the gravity recognition method and the image recognition method are consistent, the taken item is added to the virtual shopping cart, or the returned item is deleted from the virtual shopping cart.

[0119] If the various recognition results are inconsistent, for example, the object recognition results of gravity recognition and image recognition are inconsistent, it will be marked as an abnormal event, and the relevant image or gravity detection information and recognition information will be saved as risk data for subsequent risk control.

[0120] The object recognition method based on multiple sensor fusion has stronger robustness and accuracy.

[0121] Step 86, using the detection information of the shopping area intrusion detection sensor 12 to determine whether the user has left the preset shopping area. If the user has not left the preset shopping area, continue with the above-mentioned steps related to item identification. If the user has left the preset shopping area, execute step 87.

[0122] Step 87, determining whether there is an abnormal event, such as an abnormal recognition event in which various recognition results are inconsistent, or other abnormal events.

[0123] In step 88, if there is an abnormal event, corresponding processing can be performed according to the operation corresponding to the abnormal event. The exemplary abnormal event processing method is referred to above and will not be repeated here.

[0124] Step 89: If there is no abnormal event, the items in the virtual shopping cart can be settled to complete the sales process.

[0125] Based on the detection information of whether the shopping area has been invaded and the detection information of whether the boundary area between the area for sale and other areas has been crossed, the occurrence of shopping-related events such as the user entering or leaving the shopping area and the boundary area being crossed is determined, and combined with the detected shopping information, corresponding shopping-related operations are triggered to realize automatic vending. Thus, at least one problem existing in the automatic vending business, such as abnormal settlement and large amount of calculation for item identification, is improved.

[0126] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present disclosure may take the form of a computer program product implemented on one or more computer-usable non-transient storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0127] The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present disclosure should be included in the protection scope of the present disclosure.

Claims

1. A vending device, comprising: processor, as well as Shopping area intrusion detection sensors and boundary area penetration detection sensors, Wherein, the processor is electrically connected to the shopping area intrusion detection sensor, and the processor is electrically connected to the boundary area penetration detection sensor; The shopping area intrusion detection sensor is configured to detect information on whether the shopping area has been invaded; The boundary area crossing detection sensor is configured to detect whether the boundary area between the for-sale item placement area and other areas is crossed; The processor is configured to determine the occurrence of a predetermined event based on detection information of at least one of the shopping area intrusion detection sensor and the boundary area crossing detection sensor, and trigger an operation associated with the predetermined event, wherein the occurrence of an event in which a user enters or leaves a shopping area is determined based on the detection information of the shopping area intrusion detection sensor, the event of the user entering the shopping area is associated with caching sensing information for object identification, the detection information of the shopping area intrusion detection sensor is used to determine whether the user has entered a preset shopping area, if the user enters the preset shopping area, the detection information of the shopping information detection sensor is cached to identify the object, and the event of the user leaving the shopping area is associated with an object settlement operation.

2. The vending device of claim 1, wherein: The processor is configured to determine the occurrence of an event of a boundary area being crossed or an event in a direction of a boundary area being crossed based on the detection information of the boundary area crossing detection sensor.

3. The vending device of claim 1, wherein: The association relationship between the predetermined event and the operation is configured as at least one of the following relationships: The event of a user entering a shopping area is also associated with the operation of item recognition; The event of the demarcation area being crossed is associated with an item identification operation or an associated purchase item addition or deletion operation; An operation of identifying items associated with the event of the direction in which the demarcation area is passed through, or an operation of adding or deleting associated purchase items, wherein: if the direction in which the demarcation area is passed through is a direction away from the vending device, an operation of adding purchase items associated with the event of the demarcation area being passed through in the direction away from the vending device is performed; if the direction in which the demarcation area is passed through is a direction toward the vending device, an operation of deleting purchase items associated with the event of the demarcation area being passed through in the direction toward the vending device is performed.

4. The vending device of claim 3, wherein: An event of a user entering a shopping area is associated with an operation of realizing item recognition based on at least one of item gravity and item image; Alternatively, the event of the boundary area being crossed is associated with an operation of item recognition based on at least one of item gravity and item image, or an operation of adding or deleting a purchase item based on at least one of item gravity and item image; Alternatively, the event of the direction in which the demarcation area is crossed is associated with an operation of item recognition based on at least one of item gravity and item image, or is associated with an operation of adding or deleting purchase items based on at least one of item gravity and item image.

5. The vending device of claim 3, wherein: The processor is configured to retrieve images of items within a preset range based on the time when the boundary area is crossed or the time when the boundary area is crossed in the direction of the crossing, for item identification or adding or deleting purchased items.

6. The vending device of claim 1, wherein: The processor is configured to determine the occurrence of an abnormal event based on detection information of at least one of the shopping area intrusion detection sensor, the boundary area crossing detection sensor, and the shopping information detection sensor, and trigger an operation associated with the abnormal event, The shopping information detection sensor is configured to detect information about items purchased by a user from a vending device that displays items for sale.

7. The vending device of claim 1, wherein: The demarcation area is set according to a plane whose angle with the front of the vending equipment on which the items for sale are placed is less than a preset angle, and the distance between the demarcation area and the front of the vending equipment is less than a preset distance; and / or, The shopping area is set according to a preset range area where the vending equipment is located.

8. The vending device of claim 1, wherein: The shopping area intrusion detection sensor is a first image sensor, a photoelectric sensor in a reflection working mode, an ultrasonic distance sensor, or a human body sensing switch, wherein the detection area of ​​the first image sensor is aligned with the shopping area; and / or, The boundary area passing detection sensor is a second image sensor and a photoelectric sensor in a through-beam working mode, wherein the detection area of ​​the second image sensor is aligned with the boundary area.

9. A method of selling goods, applied to a vending device, characterized in that: include: Obtain detection information on whether the shopping area has been invaded; Obtain detection information on whether the boundary area between the area where items for sale are placed and other areas has been crossed; Determining the occurrence of a predetermined event based on at least one detection information, wherein the occurrence of an event that a user enters or leaves a shopping area is determined based on the detection information of the shopping area intrusion detection sensor; and Triggering operations associated with the predetermined event to achieve sales, including: associating the event of a user entering a shopping area with caching sensing information for item identification, using detection information from an intrusion detection sensor in the shopping area to determine whether the user has entered the preset shopping area, if the user enters the preset shopping area, starting to cache detection information from a shopping information detection sensor for item identification, and associating the event of a user leaving the shopping area with item settlement.

10. The method of claim 9, wherein: In the event of a user entering a shopping area, triggering the operation of caching sensing information used for item identification; When the user leaves the shopping area, the item settlement operation is triggered.

11. The method according to any one of claims 9 to 10, wherein: Based on the detection information of whether the boundary area between the for-sale item placement area and other areas has been crossed, the occurrence of an event of the boundary area being crossed or an event of the direction in which the boundary area is crossed is determined.

12. The method of claim 11, wherein: When the boundary area is crossed, an object recognition operation is triggered; When an event of a direction in which a demarcation area is crossed occurs, an operation of item recognition and an operation of adding or deleting purchased items are triggered. If the direction in which the demarcation area is crossed is a direction away from the vending device, an operation of adding purchased items associated with the event of the demarcation area being crossed in a direction away from the vending device is performed. If the direction in which the demarcation area is crossed is a direction toward the vending device, an operation of deleting purchased items associated with the event of the demarcation area being crossed in a direction toward the vending device is performed.

13. The method of claim 12, wherein: An operation of taking away or putting back the item is determined by combining at least one of the item gravity and the item image, and an operation of adding or deleting the purchased item is triggered based on the determined operation of taking away or putting back the item.

14. The method of claim 12, wherein: Based on the time when the boundary area is crossed or the time when the boundary area is crossed in the direction of the crossing, the image of the object within the preset range is retrieved, and the object is identified based on the retrieved object image or in combination with the gravity of the object.

15. A computer-readable storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the vending method according to any one of claims 9 to 14 is implemented.

Images