Payment verification method, device and equipment based on unmanned supermarket information system

Abstract

The invention discloses a payment verification method and device based on an unmanned supermarket information system. According to the scheme, when it is detected that the to-be-purchased object enters the supermarket entrance area, the identity information of the to-be-purchased object is confirmed according to the face image so as to confirm the shopping trajectory tracking object; if it is detected that the shopping trajectory tracking object is located in the exit area of the supermarket, generating a supermarket shopping trajectory of the shopping trajectory tracking object moving in the supermarket according to a video set in the monitoring video; according to the payment information, determining a goods placement area corresponding to the paid commodity information, and obtaining a track proportion of the goods placement area in a supermarket shopping track and an area number proportion of the goods placement area and a target goods placement area in number; determining whether payment checking operation needs to be carried out according to the track proportion and the area proportion; if so, triggering payment verification equipment to execute payment checking operation; according to the scheme, abnormal conditions such as wrong payment or missing payment of commodities in the shopping process of the unmanned supermarket can be avoided, and property loss is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of the Internet and the Internet of Things, and in particular to a payment verification method, device and equipment based on an unmanned supermarket information system. Background Art

[0002] An unmanned supermarket refers to a supermarket that operates without store clerks or salespeople. It uses technologies such as the Internet of Things, automation, and intelligence to provide consumers with shopping convenience through self-checkout, automatic replenishment, and monitoring.

[0003] At present, unmanned supermarkets will set up shelves to display goods in the supermarket. Consumers can freely choose goods on the shelves according to their preferences, and then settle the goods in the self-service checkout area. Since unmanned supermarkets do not have real-time supervision by store staff, wrong payments, missed payments, or lost goods often occur, resulting in property losses in the supermarket. Summary of the invention

[0004] The main purpose of the present invention is to provide a payment verification method, device and equipment based on an unmanned supermarket information system, which can perform payment verification on commodity transactions in unmanned supermarkets, avoid commodity loss, wrong payment or missed payment, and reduce property losses in unmanned supermarkets.

[0005] To achieve the above-mentioned purpose, an embodiment of the present invention provides a control method for an unmanned supermarket information system, wherein the unmanned supermarket information system includes a supermarket monitoring device, an unmanned supermarket monitoring center platform and a payment verification device, wherein the supermarket monitoring device is arranged in an unmanned supermarket, and the unmanned supermarket is provided with a supermarket entrance area, a supermarket exit area and a supermarket cargo shelf, a monitoring area is arranged in front of the supermarket cargo shelf, and a facial image acquisition device is arranged in the supermarket entrance area; the payment verification method is executed by the unmanned supermarket monitoring center platform, and comprises: If it is detected that the shopping object enters the supermarket entrance area, the facial image of the shopping object is collected by the facial image collection device at the supermarket entrance; the identity information of the shopping object is confirmed according to the facial image; and the shopping object is determined as a shopping trajectory tracking object according to the identity information; If it is detected that the shopping trajectory tracking object is located in the supermarket exit area, a video set corresponding to the shopping trajectory tracking object is retrieved from the surveillance video, wherein the video set includes at least one video frame having the shopping trajectory tracking object; Performing target tracking detection processing on each video frame in the video set to obtain position information of a shopping trajectory tracking object in each video frame, and generating a supermarket shopping trajectory of the shopping trajectory tracking object in the supermarket according to the position information of the shopping trajectory tracking object in each video frame; When the payment information corresponding to the shopping trajectory tracking object is obtained, the goods placement area corresponding to the paid commodity information is confirmed according to the payment information, and the trajectory proportion of the goods placement area in the supermarket shopping trajectory is obtained; Identify the cross-overlapping activity points of the shopping trajectory tracking object in the supermarket shopping trajectory, determine the target goods placement area corresponding to the cross-overlapping activity points, and obtain the area quantity ratio between the goods placement area and the target goods placement area; Compare the trajectory proportion with the first set threshold, and the area quantity proportion with the second set threshold, to obtain a comparison result; Determining whether a payment verification operation needs to be performed on the shopping trajectory tracking object according to the comparison result; If necessary, a payment verification instruction is sent to the payment verification device so that the payment verification device performs a payment verification operation.

[0006] Accordingly, the embodiment of the present application further provides a payment verification device based on an unmanned supermarket information system, wherein the unmanned supermarket information system includes a supermarket monitoring device, an unmanned supermarket monitoring center platform, and a payment verification device, wherein the supermarket monitoring device is arranged in an unmanned supermarket, wherein the unmanned supermarket is provided with a supermarket entrance area, a supermarket exit area, and a supermarket cargo shelf, wherein a monitoring area is arranged in front of the supermarket cargo shelf, and a facial image acquisition device is arranged in the supermarket entrance area; the payment verification device includes: The object determination module is used to collect a facial image of the object to be purchased through a facial image acquisition device at the supermarket entrance when it is detected that the object to be purchased enters the supermarket entrance area; confirm the identity information of the object to be purchased based on the facial image; and determine that the object to be purchased is a shopping trajectory tracking object based on the identity information; The video tracking module is used to retrieve a video set corresponding to the shopping trajectory tracking object from the surveillance video if it is detected that the shopping trajectory tracking object is located in the supermarket exit area, wherein the video set includes at least one video frame having the shopping trajectory tracking object; perform target tracking detection processing on each video frame in the video set to obtain position information of the shopping trajectory tracking object in each video frame, and generate a supermarket shopping trajectory of the shopping trajectory tracking object in the supermarket according to the position information of the shopping trajectory tracking object in each video frame; A payment verification module is used to, when the payment information corresponding to the shopping trajectory tracking object is obtained, confirm the goods placement area corresponding to the paid commodity information according to the payment information, and obtain the trajectory proportion of the goods placement area in the supermarket shopping trajectory; identify the cross-overlapping activity points of the shopping trajectory tracking object in the supermarket shopping trajectory, determine the target goods placement area corresponding to the cross-overlapping activity points, and obtain the area quantity proportion of the goods placement area and the target goods placement area in terms of quantity; compare the trajectory proportion with a first set threshold, and the area quantity proportion with a second set threshold to obtain a comparison result; determine whether it is necessary to perform a payment verification operation on the shopping trajectory tracking object according to the comparison result; if necessary, send a payment verification instruction to the payment verification device to enable the payment verification device to perform the payment verification operation.

[0007] The payment verification method based on the unmanned supermarket information system provided in the embodiment of the present application can detect that when a shopping object enters the supermarket entrance area, the facial image of the shopping object is collected by the facial image collection device at the supermarket entrance; the identity information of the shopping object is confirmed according to the facial image; the shopping object is determined to be a shopping trajectory tracking object according to the identity information; if it is detected that the shopping trajectory tracking object is located in the supermarket exit area, a video set corresponding to the shopping trajectory tracking object is retrieved from the monitoring video; a supermarket shopping trajectory of the shopping trajectory tracking object in the supermarket is generated according to the video set; when the payment information corresponding to the shopping trajectory tracking object is obtained, the goods placement area corresponding to the paid commodity information is confirmed according to the payment information, the trajectory proportion of the goods placement area in the supermarket shopping trajectory is obtained, and the area quantity proportion of the goods placement area and the target goods placement area in terms of quantity is obtained; whether it is necessary to perform a payment verification operation on the shopping trajectory tracking object according to the trajectory proportion and the area proportion; if necessary, a payment verification instruction is sent to the payment verification device so that the payment verification device performs the payment verification operation; this solution can avoid the situation of commodity loss, wrong payment or missed payment during shopping in an unmanned supermarket, thereby reducing property losses. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0009] Figure 1 This is a schematic diagram of a scenario of an unmanned supermarket information system in an embodiment of the present application; Figure 2A flowchart of a payment verification method based on an unmanned supermarket information system provided in an embodiment of the present application; Figure 3 A schematic diagram of the structure of a payment verification device based on an unmanned supermarket information system provided in an embodiment of the present application; Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0010] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0011] The embodiments of the present application provide a payment verification method and a control device for an unmanned supermarket information system, which will be described in detail below.

[0012] like Figure 1 As shown, the unmanned supermarket information system may include multiple monitoring devices and an unmanned supermarket monitoring center platform, such as local or remote monitoring devices. In this application, an unmanned supermarket refers to a supermarket where no staff is required to be on duty and consumers can choose goods or settle goods by themselves. The unmanned supermarket is generally provided with a supermarket entrance, a supermarket exit and a supermarket cargo rack, and a monitoring area is provided in front of the supermarket cargo rack.

[0013] The monitoring equipment is used to shoot the monitoring area and all areas inside the unmanned supermarket to obtain monitoring videos; and the monitoring center platform can be deployed in the cloud and connected to the monitoring equipment of multiple unmanned supermarkets for data exchange.

[0014] For example, in one embodiment, see Figure 1 , which is an example of an unmanned supermarket. The unmanned supermarket information system may include multiple cameras, such as camera 1, camera 2 to camera n, etc. These cameras may be installed at various locations and areas of the unmanned supermarket according to actual application requirements, so as to shoot all areas inside the unmanned supermarket, especially the monitoring area, to obtain monitoring videos. The monitoring videos may be transmitted to the unmanned supermarket monitoring center platform via the network.

[0015] The above-mentioned unmanned supermarket information system includes supermarket monitoring equipment and an unmanned supermarket monitoring center platform. The supermarket monitoring equipment is arranged in the unmanned supermarket. The unmanned supermarket is provided with a supermarket entrance area, a supermarket exit area and a supermarket cargo shelf. A monitoring area is arranged in front of the supermarket cargo shelf. A facial image acquisition device is arranged in the supermarket entrance area. The supermarket monitoring equipment is used to shoot the monitoring area and all areas inside the unmanned supermarket to obtain monitoring videos; the unmanned supermarket monitoring center platform is remotely connected to the supermarket monitoring equipment network of at least two unmanned supermarkets.

[0016] The payment verification method is applicable to an unmanned supermarket monitoring center platform, comprising: if a shopping object is detected to enter the supermarket entrance area, a facial image of the shopping object is collected by a facial image collection device at the supermarket entrance; facial feature extraction processing is performed on the facial image to confirm the identity information of the shopping object; a shopping trajectory tracking object of the shopping object is determined according to the identity information; if the shopping trajectory tracking object is detected to be located in the supermarket exit area, a video set corresponding to the shopping trajectory tracking object is retrieved from the monitoring video, the video set including at least one video frame having the shopping trajectory tracking object; target tracking detection processing is performed on each video frame in the video set to obtain the location information of the shopping trajectory tracking object in each video frame; a supermarket shopping trajectory of the shopping trajectory tracking object in the supermarket is generated according to the location information of the shopping trajectory tracking object in each video frame; when When the payment information corresponding to the shopping trajectory tracking object is obtained, the goods placement area corresponding to the paid commodity information is confirmed according to the payment information, and the trajectory proportion of the goods placement area in the supermarket shopping trajectory is obtained; the cross-overlapping activity points of the shopping trajectory tracking object in the supermarket shopping trajectory are identified, the target goods placement areas corresponding to the cross-overlapping activity points are determined, and the area quantity proportion of the goods placement areas and the target goods placement areas is obtained; the trajectory proportion is compared with a first set threshold, and the area quantity proportion is compared with a second set threshold to obtain a comparison result; according to the comparison result, it is determined whether a payment verification operation is required for the shopping trajectory tracking object; if necessary, a payment verification instruction is sent to the payment verification device so that the payment verification device performs the payment verification operation; if not, the shopping trajectory tracking object is given permission to pass through the supermarket exit area. Based on the above-mentioned unmanned supermarket information system, an embodiment of the present application provides a payment verification method for an unmanned supermarket information system, including: when a shopping object enters the supermarket entrance, collecting feature information of the shopping object, and generating a shopping trajectory tracking object based on the feature information; when the shopping trajectory tracking object enters the supermarket exit, retrieving a video frame corresponding to the shopping trajectory tracking object from a surveillance video; generating a supermarket shopping trajectory of the shopping trajectory tracking object in the supermarket and an activity picture in a surveillance area based on the video frame; obtaining payment information corresponding to the shopping trajectory tracking object, and determining whether a payment verification operation is required based on the payment information, the supermarket shopping trajectory and the activity picture; if necessary, notifying the monitoring personnel to perform a payment verification operation based on the payment information, the supermarket shopping trajectory and the activity picture through a remote monitoring device; if not necessary, allowing the shopping trajectory tracking object to pass through the supermarket exit.

[0017] The following will describe in detail the control method of the unmanned supermarket information system from the perspective of payment verification (referred to as control device) based on the unmanned supermarket information system. The control device can be set in the unmanned supermarket as an independent entity, or it can be integrated in a local monitoring device or a remote monitoring device.

[0018] like Figure 2 As shown, the specific process of the control method of the unmanned supermarket information system can be as follows: 101. If it is detected that a shopping object enters the supermarket entrance area, a facial image of the shopping object is collected by a facial image collection device at the supermarket entrance; facial feature extraction is performed on the facial image to confirm the identity information of the shopping object; and a shopping trajectory tracking object of the shopping object is determined based on the identity information.

[0019] In one embodiment, a facial image acquisition device, a face recognition machine, etc. may be set at the supermarket entrance. When a shopping object enters the supermarket entrance, the facial image acquisition device may acquire the facial image of the shopping object for identity recognition.

[0020] In one embodiment, the unmanned supermarket monitoring center platform performs facial feature extraction processing based on the facial image to confirm the identity information of the shopping object; and determines the shopping trajectory tracking object of the shopping object based on the identity information. For example, in one embodiment, the real-name authentication of the shopping object can be performed based on the facial image, such as calling the facial image security database for matching. If the real-name authentication passes, that is, the identity information is legal, then the shopping object is confirmed as the shopping trajectory tracking object.

[0021] The object to be purchased refers to a person or other object that needs to be monitored. For the convenience of description, in the application embodiments, people, namely consumers, are used as examples for explanation.

[0022] Among them, facial feature information can include geometric features and representational features. Geometric features refer to the geometric relationship between facial features such as eyes, nose and mouth, such as distance, area and angle, etc., and representational features refer to global or local features extracted by using the grayscale information of the face image through some algorithms, such as the Local Binary Patterns (LBP) algorithm. Human features can include human height, width (fatness), color (such as clothing color and skin color), texture, depth, behavioral characteristics, and / or movement direction.

[0023] 102. If it is detected that the shopping trajectory tracking object is located in the supermarket exit area, a video set corresponding to the shopping trajectory tracking object is retrieved from the surveillance video, where the video set includes at least one video frame having the shopping trajectory tracking object.

[0024] For example, when the shopping trajectory tracking object enters the supermarket exit area, the unmanned supermarket monitoring center platform can specifically retrieve the video frame corresponding to the shopping trajectory tracking object from the monitoring video according to the feature information of the shopping trajectory tracking object.

[0025] The video set is all video data collected by the monitoring device after the shopping trajectory tracking object enters the unmanned supermarket. The video set includes video frames with the shopping trajectory tracking object, such as video frames containing consumers.

[0026] 103. Perform target tracking detection processing on each video frame in the video set to obtain position information of the shopping trajectory tracking object in each video frame; and generate a supermarket shopping trajectory of the shopping trajectory tracking object in the supermarket according to the position information of the shopping trajectory tracking object in each video frame.

[0027] After acquiring the video set of the object, the unmanned supermarket monitoring center platform uses the target tracking algorithm to process it, obtains the location information of the object in the video frame, and then generates the supermarket shopping trajectory of the object in the supermarket.

[0028] In one embodiment, the process of video target tracking processing and generating trajectories involves using a target detection algorithm to identify targets in the video and determine their positions. Subsequently, feature extraction techniques, such as deep learning models, are used to extract key features of the target. The target tracking algorithm is used based on key features to identify and match the same target in consecutive video frames to form a coherent trajectory. During the target tracking process, the system records the position changes of the target in each frame, thereby generating a movement trajectory of the target. Finally, these trajectories are drawn back onto the video frames to intuitively show the path of the target moving over time.

[0029] In one embodiment, the unmanned supermarket monitoring center platform reads video frames frame by frame from the video set; uses a target tracking detection model to perform target tracking detection on each video frame to identify the shopping trajectory tracking object and location information in the video frame, wherein the target tracking detection model is based on a neural network.

[0030] In one embodiment, in order to improve the accuracy of trajectory tracking, the target detection model or the target tracking detection model can also determine the location information and time information of the shopping trajectory tracking object in the supermarket based on the video frame, select the location information within the monitoring area, and obtain the filtered location information; generate the trajectory points of the shopping trajectory tracking object in a preset supermarket map based on the filtered location information; connect the trajectory points in the order of time information to obtain the supermarket shopping trajectory of the shopping trajectory tracking object in the supermarket.

[0031] In one embodiment, in order to improve the accuracy of the trajectory, data preprocessing is performed on the position data of the shopping trajectory tracking object in each video frame to obtain predicted position information, and the data preprocessing includes: at least one of abnormal data removal and data smoothing processing; generating trajectory points of the shopping trajectory tracking object in the supermarket indoor map according to the position information and time sequence; and connecting the trajectory points to construct the supermarket shopping trajectory of the shopping trajectory tracking object in the supermarket.

[0032] 104. When the payment information corresponding to the shopping trajectory tracking object is obtained, the goods placement area corresponding to the paid commodity information is confirmed according to the payment information, and the trajectory proportion of the goods placement area in the supermarket shopping trajectory is obtained.

[0033] The payment information may include information about the paid merchandise and / or the identification of the merchandise placement area where the merchandise is located. The goods placement area refers to the area where a certain item or a certain type of goods in an unmanned supermarket belongs on the supermarket goods shelf. The goods placement area is located in the monitoring area, and each monitoring area may include at least one goods placement area. The goods placement area corresponding to the paid goods information refers to the area where the goods corresponding to the paid goods information belong on the supermarket goods shelf.

[0034] In an embodiment of the present application, the goods placement area corresponding to the paid commodity information can be determined based on the payment information, and then the proportion of the goods placement area in the supermarket shopping trajectory (i.e., the trajectory proportion) can be obtained. Specifically, the number of goods placement areas corresponding to the paid commodity information is counted, and the total number of goods placement areas involved in the shopping trajectory tracking object is determined based on the supermarket shopping trajectory. Then, the ratio of the number of goods placement areas corresponding to the paid commodity information to the total number is calculated to obtain the proportion of the goods placement area in the supermarket shopping trajectory. For example, assuming that the consumer's supermarket shopping trajectory involves all the goods placement areas in the five monitoring areas, and each monitoring area includes N goods placement areas, and one monitoring area that is not involved includes M goods placement areas, at this time, the trajectory proportion = 5N / (5N+M), and N and M are both positive integers.

[0035] 105. Identify the cross-overlapping activity points of the shopping trajectory tracking object in the supermarket shopping trajectory, determine the target goods placement area corresponding to the cross-overlapping activity points, and obtain the area quantity ratio between the goods placement area and the target goods placement area.

[0036] The cross-overlapping activity point is a point where the shopping trajectory of the shopping trajectory tracking object crosses, indicating that the consumer may take goods in the goods placement area corresponding to the activity point. In the embodiment of the present application, considering the factor of the goods placement area where shopping is possible, the payment verification of goods is performed based on this factor, which can improve the accuracy of the payment verification of the unmanned supermarket.

[0037] Among them, the area quantity ratio is the ratio of the number of goods placement areas involved in the payment information in the supermarket shopping trajectory to the goods placement areas involved in the cross-overlapping activity points. For example, suppose that the consumer's supermarket shopping trajectory is determined to involve all the goods placement areas in the five monitoring areas based on the payment information, and each monitoring area includes N goods placement areas, and the one monitoring area not involved includes M goods placement areas. Then it is identified that the cross-overlapping activity points of the consumer in the supermarket shopping trajectory involve P goods placement areas. At this time, the area quantity ratio = 4N / P, where N, M, and P are all positive integers.

[0038] 106. Compare the trajectory proportion with the first set threshold, and the area quantity proportion with the second set threshold to obtain a comparison result; and determine whether a payment verification operation is required for the shopping trajectory tracking object according to the comparison result.

[0039] The first set threshold and the second set threshold can be set according to actual needs, for example, can be set between 1-10%, etc. The smaller the threshold, the stricter the payment verification is and the more accurate the payment verification is.

[0040] In one implementation, if the track proportion is less than a first set threshold, and the area quantity proportion is less than a second set threshold, it is determined that a payment verification operation needs to be performed on the shopping track tracking object; If the track proportion is greater than or equal to the first set threshold, and the area quantity proportion is greater than or equal to the second set threshold, it is determined that there is no need to perform a payment verification operation on the shopping track tracking object.

[0041] Among them, a trajectory proportion less than the first set threshold indicates that the shopper has a lot of contact with the goods in the supermarket shelves, but the quantity purchased is small, and vice versa.

[0042] The number of areas indicates the relationship between the goods purchased by the shopper and the supermarket shelves that are repeatedly touched. If the proportion of the number of areas is less than the second set threshold, it means that the quantity purchased is lower than the number of supermarket shelves that are repeatedly touched. In this case, there may be behavior of hiding goods, and abnormal shopping events may occur. At this time, further payment verification is required. If the proportion of the number of areas is greater than or equal to the second set threshold, it means that the quantity purchased is higher than the number of supermarket shelves that are repeatedly touched. At this time, the shopping is considered reasonable. Combined with the proportion of tracks, it can be further judged that if the proportion of tracks is greater than or equal to the first set threshold, then there is no need to perform payment verification operations.

[0043] In one embodiment, in order to improve the accuracy of payment verification, the determining whether it is necessary to perform a payment verification operation on the shopping trajectory tracking object according to the comparison result further includes: If the track proportion is less than the first set threshold, and the area quantity proportion is greater than the second set threshold, it is determined that a payment verification operation needs to be performed for the shopping track tracking object (that is, a payment verification operation needs to be performed as long as the purchase quantity is large and the shelves touched are small); If the trajectory proportion is greater than or equal to the first set threshold, and the area quantity proportion is less than the second set threshold, the trajectory proportion and the area quantity proportion are fused and calculated to obtain a fused value; based on the fused value, it is further determined whether it is necessary to perform a payment verification operation on the shopping trajectory tracking object. That is, in the case where the number of purchased goods is large and the number of shelves contacted is small, if it is found that the relationship between the purchased quantity and the number of supermarket shelves repeatedly contacted is unreasonable, and the payment and shopping abnormalities cannot be confirmed at this time, further calculation and confirmation are required. For example, the trajectory proportion and the area quantity proportion are fused and calculated to obtain a fused value; based on the fused value, it is further determined whether it is necessary to perform a payment verification operation on the shopping trajectory tracking object.

[0044] In one embodiment, in order to improve the accuracy of payment verification, the track proportion and the area quantity proportion are fused and calculated to obtain a fused value, including: Obtaining a first weight value of the track proportion and a second weight value of the area quantity proportion, wherein the first weight value represents the importance of the track proportion in determining whether payment is verified, and the second weight value represents the importance of the area quantity proportion in determining whether payment is verified; According to the first weight value and the second weight value, a weighted sum is performed on the trajectory proportion and the area quantity proportion to obtain a fused value.

[0045] The above steps are all executed by the unmanned supermarket monitoring center platform.

[0046] 107. If necessary, a payment verification instruction is sent to the payment verification device so that the payment verification device performs a payment verification operation; if not necessary, the shopping trajectory tracking object is given permission to pass through the supermarket exit area.

[0047] In one embodiment, if it is determined that a payment verification operation is required, a monitoring person is notified to perform the payment verification operation through a payment verification device.

[0048] For example, payment verification prompt information can be specifically generated and sent to the monitoring personnel; in response to the payment verification request triggered by the monitoring personnel based on the payment verification prompt information, the payment information, supermarket shopping trajectory and monitoring screen are sent to the monitoring personnel, and the supermarket shopping trajectory and monitoring screen are displayed in split screens on the payment verification device, so that the monitoring personnel can perform the payment verification operation.

[0049] If the monitoring personnel determine that there is no problem after checking the payment, the unmanned supermarket monitoring center platform can send a response message of verification passed to allow the shopping trajectory tracking object to pass through the supermarket exit. Otherwise, if it is determined that there is a problem with the payment, corresponding measures can be taken, such as sending a response message of verification failure to the unmanned supermarket monitoring center platform. After receiving the response message of verification failure, the unmanned supermarket monitoring center platform can perform operations such as alarms. The specific measures can be set according to the actual application needs and will not be elaborated here.

[0050] In one embodiment, if it is determined that the payment verification operation is not required, the shopping track tracking object is allowed to pass through the supermarket exit. For example, the door control of the supermarket exit can be controlled to open to allow the shopping track tracking object to pass through the supermarket exit.

[0051] As can be seen from the above, the payment verification method of the unmanned supermarket information system provided in the embodiment of the present application can, when detecting that the object to be shopped enters the supermarket entrance area, collect the facial image of the object to be shopped through the facial image acquisition device at the supermarket entrance; confirm the identity information of the object to be shopped based on the facial image; determine that the object to be shopped is a shopping trajectory tracking object based on the identity information; if it is detected that the shopping trajectory tracking object is located in the supermarket exit area, retrieve the video set corresponding to the shopping trajectory tracking object from the surveillance video; generate the supermarket shopping trajectory of the shopping trajectory tracking object's activities in the supermarket based on the video set; and when the shopping trajectory tracking object is obtained, When the payment information corresponding to the shopping trajectory tracking object is obtained, the goods placement area corresponding to the paid goods information is confirmed according to the payment information, the track proportion of the goods placement area in the supermarket shopping trajectory is obtained, and the area quantity proportion of the goods placement area and the target goods placement area is obtained; according to the track proportion and the area proportion, it is determined whether it is necessary to perform a payment verification operation on the shopping trajectory tracking object; if necessary, a payment verification instruction is sent to the payment verification device, so that the payment verification device performs the payment verification operation; this solution can avoid the loss, wrong payment or missed payment of goods during shopping in an unmanned supermarket, thereby reducing property losses.

[0052] Correspondingly, in order to better implement the above method, an embodiment of the present application also provides a payment verification device based on an unmanned supermarket information system (hereinafter referred to as the payment verification device), wherein the unmanned supermarket information system includes a supermarket monitoring device, an unmanned supermarket monitoring center platform and a payment verification device, the supermarket monitoring device is arranged in the unmanned supermarket, the unmanned supermarket is provided with a supermarket entrance area, a supermarket exit area and a supermarket cargo shelf, a monitoring area is arranged in front of the supermarket cargo shelf, and a facial image acquisition device is arranged in the supermarket entrance area.

[0053] like Figure 3 As shown, the payment verification device includes an object determination module 301, a video tracking module 302, and a payment verification module 303, which are specifically as follows: The object determination module 301 is used to collect a facial image of the object to be purchased through a facial image acquisition device at the supermarket entrance when it is detected that the object to be purchased enters the supermarket entrance area; confirm the identity information of the object to be purchased based on the facial image; and determine that the object to be purchased is a shopping trajectory tracking object based on the identity information; The video tracking module 302 is used to retrieve a video set corresponding to the shopping trajectory tracking object from the surveillance video if it is detected that the shopping trajectory tracking object is located in the supermarket exit area, wherein the video set includes at least one video frame having the shopping trajectory tracking object; perform target tracking detection processing on each video frame in the video set to obtain position information of the shopping trajectory tracking object in each video frame, and generate a supermarket shopping trajectory of the shopping trajectory tracking object in the supermarket according to the position information of the shopping trajectory tracking object in each video frame; The payment verification module 303 is used to, when the payment information corresponding to the shopping trajectory tracking object is obtained, confirm the goods placement area corresponding to the paid commodity information according to the payment information, and obtain the trajectory proportion of the goods placement area in the supermarket shopping trajectory; identify the cross-overlapping activity points of the shopping trajectory tracking object in the supermarket shopping trajectory, determine the target goods placement area corresponding to the cross-overlapping activity points, and obtain the area quantity proportion of the goods placement area and the target goods placement area in terms of quantity; compare the trajectory proportion with the first set threshold, and the area quantity proportion with the second set threshold to obtain a comparison result; determine whether it is necessary to perform a payment verification operation on the shopping trajectory tracking object according to the comparison result; if necessary, send a payment verification instruction to the payment verification device so that the payment verification device performs the payment verification operation.

[0054] In one embodiment, the payment verification module 303 is specifically used to: If the track proportion is less than the first set threshold, and the area quantity proportion is less than the second set threshold, it is determined that a payment verification operation needs to be performed on the shopping track tracking object; If the track proportion is greater than or equal to the first set threshold, and the area quantity proportion is greater than or equal to the second set threshold, it is determined that there is no need to perform a payment verification operation on the shopping track tracking object.

[0055] In one embodiment, the payment verification module 303 is specifically used to: If the track proportion is less than the first set threshold, and the area quantity proportion is greater than the second set threshold, it is determined that a payment verification operation needs to be performed on the shopping track tracking object; If the trajectory proportion is greater than or equal to the first set threshold and the area quantity proportion is less than the second set threshold, the trajectory proportion and the area quantity proportion are fused and calculated to obtain a fused value; based on the fused value, it is further determined whether a payment verification operation is required for the shopping trajectory tracking object.

[0056] In one embodiment, the payment verification module 303 is specifically used to: Obtaining a first weight value of the track proportion and a second weight value of the area quantity proportion, wherein the first weight value represents the importance of the track proportion in determining whether payment is verified, and the second weight value represents the importance of the area quantity proportion in determining whether payment is verified; According to the first weight value and the second weight value, a weighted sum is performed on the trajectory proportion and the area quantity proportion to obtain a fused value.

[0057] In one embodiment, the payment verification module 303 is specifically used to obtain the number of the goods placement areas and the number of goods placement areas covered by the supermarket shopping trajectory, so as to calculate the trajectory proportion of the goods placement areas in the supermarket shopping trajectory.

[0058] In one embodiment, the video tracking module 302 is specifically used to: read video frames frame by frame from the video set; use a target tracking detection model to perform target tracking detection on each video frame, and identify the shopping trajectory tracking object and location information in the video frame, wherein the target tracking detection model is based on a neural network.

[0059] In one embodiment, the video tracking module 302 is specifically used to: Performing data preprocessing on the location data of the shopping trajectory tracking object in each video frame to obtain predicted location information, wherein the data preprocessing includes at least one of: abnormal data removal and data smoothing processing; Generating the track points of the shopping track tracking object in the supermarket indoor map according to the location information and time sequence; The track points are connected to construct a supermarket shopping track of the shopping track tracking object in the supermarket.

[0060] In one embodiment, the payment verification module 303 is specifically used to: Setting a boundary box of the supermarket shopping trajectory point based on the location information of the supermarket shopping trajectory point in the supermarket shopping trajectory; Get the bounding box overlap area between supermarket shopping trajectory points; The supermarket shopping trajectory points whose bounding box overlap area is greater than the overlap threshold are determined as cross-overlapping activity points.

[0061] For the implementation of the above modules, please refer to the previous method embodiments for details, which will not be described in detail here.

[0062] It should be noted that, in specific implementation, the above modules can be arbitrarily combined, integrated into one or more modules, or implemented as independent entities. In addition, the above modules can be implemented in the form of hardware or in the form of software functional modules. If the integrated modules are implemented in the form of software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. The above-mentioned storage medium can be a read-only memory, a disk or an optical disk, etc.

[0063] From the above, it can be seen that the payment verification device based on the unmanned supermarket information system provided in the embodiment of the present application can ensure a better shopping experience while performing seamless monitoring of commodity transactions, avoiding the loss, wrong payment or missed payment of commodities, and reducing the loss of supermarket property.

[0064] like Figure 4 As shown, an embodiment of the present application further provides an electronic device 40, characterized in that it includes a processor 401 and a memory 402, wherein the memory 402 stores a computer program, and when the computer program is executed by the processor 401, the processor 401 executes the steps of any of the methods described above.

[0065] The payment verification method and device based on the unmanned supermarket information system provided by the embodiment of the present application are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A payment verification method based on an unmanned supermarket information system, characterized in that: The unmanned supermarket information system includes a supermarket monitoring device, an unmanned supermarket monitoring center platform and a payment verification device. The supermarket monitoring device is arranged in the unmanned supermarket. The unmanned supermarket is provided with a supermarket entrance area, a supermarket exit area and a supermarket cargo rack. A monitoring area is arranged in front of the supermarket cargo rack. A facial image acquisition device is arranged in the supermarket entrance area. The payment verification method is executed by the unmanned supermarket monitoring center platform, and includes: If it is detected that the shopping object enters the supermarket entrance area, the facial image of the shopping object is collected by the facial image collection device at the supermarket entrance; the identity information of the shopping object is confirmed according to the facial image; and the shopping object is determined as a shopping trajectory tracking object according to the identity information; If it is detected that the shopping trajectory tracking object is located in the supermarket exit area, a video set corresponding to the shopping trajectory tracking object is retrieved from the surveillance video, wherein the video set includes at least one video frame having the shopping trajectory tracking object; Performing target tracking detection processing on each video frame in the video set to obtain position information of a shopping trajectory tracking object in each video frame, and generating a supermarket shopping trajectory of the shopping trajectory tracking object in the supermarket according to the position information of the shopping trajectory tracking object in each video frame; When the payment information corresponding to the shopping trajectory tracking object is obtained, the goods placement area corresponding to the paid commodity information is confirmed according to the payment information, and the trajectory proportion of the goods placement area in the supermarket shopping trajectory is obtained; Identify the cross-overlapping activity points of the shopping trajectory tracking object in the supermarket shopping trajectory, determine the target goods placement area corresponding to the cross-overlapping activity points, and obtain the area quantity ratio between the goods placement area and the target goods placement area; Compare the trajectory proportion with a first set threshold, and the area quantity proportion with a second set threshold, to obtain a comparison result; Determining whether a payment verification operation needs to be performed on the shopping trajectory tracking object according to the comparison result; If necessary, a payment verification instruction is sent to the payment verification device so that the payment verification device performs a payment verification operation.

2. The method according to claim 1, characterized in that: The determining, according to the comparison result, whether it is necessary to perform a payment verification operation on the shopping track tracking object includes: If the track proportion is less than the first set threshold, and the area quantity proportion is less than the second set threshold, it is determined that a payment verification operation needs to be performed on the shopping track tracking object; If the track proportion is greater than or equal to the first set threshold, and the area quantity proportion is greater than or equal to the second set threshold, it is determined that there is no need to perform a payment verification operation on the shopping track tracking object.

3. The method according to claim 2, characterized in that The determining whether it is necessary to perform a payment verification operation on the shopping track tracking object according to the comparison result also includes: If the track proportion is less than the first set threshold, and the area quantity proportion is greater than the second set threshold, it is determined that a payment verification operation needs to be performed on the shopping track tracking object; If the trajectory proportion is greater than or equal to the first set threshold and the area quantity proportion is less than the second set threshold, the trajectory proportion and the area quantity proportion are fused and calculated to obtain a fused value; based on the fused value, it is further determined whether a payment verification operation is required for the shopping trajectory tracking object.

4. The method according to claim 3, characterized in that The trajectory proportion and the area quantity proportion are fused and calculated to obtain fused values, including: Obtaining a first weight value of the track proportion and a second weight value of the area quantity proportion, wherein the first weight value represents the importance of the track proportion in determining whether payment is verified, and the second weight value represents the importance of the area quantity proportion in determining whether payment is verified; According to the first weight value and the second weight value, a weighted sum is performed on the trajectory proportion and the area quantity proportion to obtain a fused value.

5. The method according to claim 1, characterized in that Confirming the goods placement area corresponding to the paid goods information according to the payment information, and obtaining the track proportion of the goods placement area in the supermarket shopping track, including: The number of the goods placement areas and the number of the goods placement areas covered by the supermarket shopping trajectory are obtained, so as to calculate the trajectory proportion of the goods placement areas in the supermarket shopping trajectory.

6. The method according to claim 1, characterized in that Performing target tracking detection processing on each video frame in the video set to obtain location information of the shopping trajectory tracking object in each video frame, including: Reading video frames frame by frame from the video set; A target tracking detection model is used to perform target tracking detection on each video frame to identify the shopping trajectory tracking object and location information in the video frame, wherein the target tracking detection model is based on a neural network.

7. The method according to claim 6, characterized in that Generating a supermarket shopping trajectory of the shopping trajectory tracking object in the supermarket according to the position information of the shopping trajectory tracking object in each video frame includes: Performing data preprocessing on the location data of the shopping trajectory tracking object in each video frame to obtain predicted location information, wherein the data preprocessing includes at least one of: abnormal data removal and data smoothing processing; Generating the track points of the shopping track tracking object in the supermarket indoor map according to the location information and time sequence; The track points are connected to construct a supermarket shopping track of the shopping track tracking object in the supermarket.

8. The method according to claim 1, characterized in that Identifying the cross-overlapping activity points of the shopping trajectory tracking objects in the supermarket shopping trajectory includes: Setting a boundary box of the supermarket shopping trajectory point based on the location information of the supermarket shopping trajectory point in the supermarket shopping trajectory; Get the bounding box overlap area between supermarket shopping trajectory points; The supermarket shopping trajectory points whose bounding box overlap area is greater than the overlap threshold are determined as cross-overlapping activity points.

9. A payment verification device based on an unmanned supermarket information system, characterized in that: The unmanned supermarket information system includes a supermarket monitoring device, an unmanned supermarket monitoring center platform and a payment verification device. The supermarket monitoring device is arranged in the unmanned supermarket. The unmanned supermarket is provided with a supermarket entrance area, a supermarket exit area and a supermarket cargo rack. A monitoring area is arranged in front of the supermarket cargo rack. A facial image acquisition device is arranged in the supermarket entrance area. The payment verification device includes: The object determination module is used to collect a facial image of the object to be purchased through a facial image acquisition device at the supermarket entrance when it is detected that the object to be purchased enters the supermarket entrance area; confirm the identity information of the object to be purchased based on the facial image; and determine that the object to be purchased is a shopping trajectory tracking object based on the identity information; The video tracking module is used to retrieve a video set corresponding to the shopping trajectory tracking object from the surveillance video if it is detected that the shopping trajectory tracking object is located in the supermarket exit area, wherein the video set includes at least one video frame having the shopping trajectory tracking object; perform target tracking detection processing on each video frame in the video set to obtain position information of the shopping trajectory tracking object in each video frame, and generate a supermarket shopping trajectory of the shopping trajectory tracking object in the supermarket according to the position information of the shopping trajectory tracking object in each video frame; A payment verification module is used to, when the payment information corresponding to the shopping trajectory tracking object is obtained, confirm the goods placement area corresponding to the paid commodity information according to the payment information, and obtain the trajectory proportion of the goods placement area in the supermarket shopping trajectory; identify the cross-overlapping activity points of the shopping trajectory tracking object in the supermarket shopping trajectory, determine the target goods placement area corresponding to the cross-overlapping activity points, and obtain the area quantity proportion of the goods placement area and the target goods placement area in terms of quantity; compare the trajectory proportion with a first set threshold, and the area quantity proportion with a second set threshold to obtain a comparison result; determine whether it is necessary to perform a payment verification operation on the shopping trajectory tracking object according to the comparison result; if necessary, send a payment verification instruction to the payment verification device to enable the payment verification device to perform the payment verification operation.

10. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of any one of the methods of claims 1-8.

Images