Abnormal shopping behavior detection method and apparatus for smart shopping cart, and shopping cart

Abstract

The present application discloses an abnormal shopping behavior detection method and apparatus for a smart shopping cart, and a shopping cart. The method comprises: acquiring barcode scanning data of products and basket-area video frame image data during a user's shopping behavior process; segmenting the products in each image frame of the basket-area video frame image data to obtain image data of each target product in a basket area of each image frame; performing trajectory tracking on each target product on the basis of the image data of each target product in the basket areas of a plurality of image frames; determining a movement direction and distance of a tracking trajectory of each target product on the basis of trajectory tracking data; on the basis of the direction and the distance, determining a state that each target product has been put into or taken out of a shopping cart; and detecting an abnormal shopping behavior of the user on the basis of the barcode scanning data, the trajectory tracking data and the state that each product has been put into or taken out of the shopping cart.

Description

Abnormal shopping behavior detection method and device for smart shopping cart and shopping cart

[0001] Related applications

[0002] This application claims priority to the Chinese invention patent application with application number 202410338478.9 filed on March 22, 2024, and cites the entire contents disclosed in the above patent application as part of this application. Technical Field

[0003] The present application relates to the field of self-service shopping technology, and in particular to a method and device for detecting abnormal shopping behavior of a smart shopping cart, and a shopping cart. Background Art

[0004] This section is intended to provide a background or context to the embodiments of the present application that are recited in the claims. No admission is made that the description herein is prior art by virtue of its inclusion in this section.

[0005] With the continuous development of technologies such as the Internet of Things, artificial intelligence, big data analytics, mobile payments, and smart hardware, the supermarket industry is undergoing a continuous transformation towards intelligence and digitalization. This allows supermarkets to better meet customer needs, enhance the shopping experience, optimize merchant operations, increase sales, and further expand their market influence. Checkout is a crucial part of supermarket shopping. It is against this backdrop that smart shopping carts have emerged in the supermarket environment, bringing a completely new shopping experience and operational model to both customers and merchants.

[0006] Generally speaking, smart shopping carts offer a range of advanced features, including automatic scanning and checkout, navigation and product location, item identification and weighing, personalized recommendations and advertising, data analysis, and inventory management. For example, with self-service barcode scanning and checkout, many smart shopping carts are equipped with built-in scanners, weighing devices, and checkout channels. These allow customers to instantly scan product barcodes while shopping, view the total price in real time, and complete payment through the cart's self-service checkout system before leaving the store. This self-service shopping process simplifies the traditional steps required for shopping and payment, eliminating the need for customers to interact with cashiers at the checkout counter, reducing the risk of errors and enhancing the shopping experience. At the same time, for merchants, it reduces cashier staffing and, therefore, reduces operating costs.

[0007] At the same time, when smart shopping carts are deployed in supermarkets, there's a need to detect unusual shopping behavior. Because smart shopping carts are self-service, consumers inevitably try to evade normal purchase procedures. These behaviors need to be detected and either prompted or prevented. Existing smart shopping cart solutions for detecting unusual user behavior lack accuracy. Summary of the Invention

[0008] The present invention provides a method for detecting abnormal shopping behavior in a smart shopping cart, which is used to improve the accuracy of detecting abnormal shopping behavior in the smart shopping cart. The method includes:

[0009] Obtain product scan data and basket area video frame image data during the user's shopping process;

[0010] Segment the products in each frame of the basket area video frame image data to obtain image data of each target product in the basket area of ​​each frame;

[0011] Tracking each target product according to the image data of each target product in the basket area in the multiple frames of images to obtain trajectory tracking data of each target product;

[0012] Determine the movement direction and distance of each target product's tracking trajectory based on the trajectory tracking data of each target product;

[0013] Determine whether each target product is in a shopping cart or out of a shopping cart based on the direction and distance of the tracking trajectory of each target product and a preset product movement distance threshold;

[0014] The user's abnormal shopping behavior is detected based on the code scanning data, the trajectory tracking data of each target product in the basket area, and the status of each target product being placed in or taken out of the shopping cart.

[0015] The present application also provides a device for detecting abnormal shopping behavior in a smart shopping cart, which is used to improve the accuracy of detecting abnormal shopping behavior in the smart shopping cart. The device includes:

[0016] An acquisition unit, configured to acquire the scanned code data of the product and the video frame image data of the basket area during the user's shopping behavior;

[0017] a segmentation processing unit, configured to segment the commodities in each frame of the basket area video frame image data to obtain image data of each target commodity in the basket area in each frame of the image;

[0018] a trajectory tracking unit, configured to track the trajectory of each target commodity according to the image data of each target commodity in the basket area in the multiple frames of images, and obtain trajectory tracking data of each target commodity;

[0019] A trajectory processing unit, configured to determine the movement direction and distance of each target commodity tracking trajectory based on the trajectory tracking data of each target commodity;

[0020] a state determination unit, configured to determine whether each target product is in a state of being placed in a shopping cart or taken out of a shopping cart based on the movement direction and distance of each target product's tracking trajectory and a preset product movement distance threshold;

[0021] The detection unit is used to detect the user's abnormal shopping behavior based on the code scanning data, the trajectory tracking data of each target product in the basket area, and the status of each target product being placed in or taken out of the shopping cart.

[0022] The present application also provides a smart shopping cart for improving the accuracy of detecting abnormal shopping behavior. The smart shopping cart includes:

[0023] The barcode scanner is used to obtain the product scanning data during the user's shopping behavior;

[0024] A basket area acquisition device, used to acquire video frame image data of the basket area;

[0025] The abnormal shopping behavior detection device of the smart shopping cart as described above.

[0026] An embodiment of the present application also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the above-mentioned method for detecting abnormal shopping behavior of the smart shopping cart is implemented.

[0027] An embodiment of the present application further provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the computer program implements the above-mentioned method for detecting abnormal shopping behavior of the smart shopping cart.

[0028] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the above-mentioned method for detecting abnormal shopping behavior of the smart shopping cart.

[0029] In an embodiment of the present application, a scheme for detecting abnormal shopping behavior in a smart shopping cart operates as follows: obtaining scanned code data of products and video frame image data of a basket area during a user's shopping behavior; segmenting the products in each frame of the video frame image data of the basket area to obtain image data of each target product in the basket area in each frame; tracking the trajectory of each target product based on the image data of each target product in the basket area in multiple frames of images to obtain trajectory tracking data of each target product; determining the movement direction and distance of the tracking trajectory of each target product based on the trajectory tracking data of each target product; determining whether each target product is in a state of being placed in a shopping cart or taken out of a shopping cart based on the movement direction and distance of the tracking trajectory of each target product and a preset product movement distance threshold; and detecting abnormal shopping behavior of the user based on the scanned code data, the trajectory tracking data of each target product in the basket area, and the state of each target product being placed in a shopping cart or taken out of a shopping cart.

[0030] The embodiments of the present application enable the smart shopping cart to have functions such as self-service checkout and smart shopping, while also having the function of detecting abnormal shopping behavior. The scanning data and the collected video frame image data of the basket area can be combined to detect the user's abnormal shopping behavior. The beneficial technical effects of the abnormal shopping behavior detection solution for the smart shopping cart provided by the embodiments of the present application are:

[0031] First, the embodiment of the present application segments the products in each frame of the basket area video frame image data and tracks the trajectory of the target products, thereby avoiding the problem of products stacking and blocking each other that affects the detection results, and improving the accuracy of detecting abnormal shopping behavior of the smart shopping cart.

[0032] Secondly, the embodiment of the present application determines the direction and distance of movement of each target product based on the tracking trajectory data of each target product; based on the direction and distance, it can accurately determine the status of each target product as being placed in or taken out of the car, and then based on the status, the user's abnormal shopping behavior can be accurately detected.

[0033] In summary, the embodiments of the present application can accurately detect abnormal shopping behaviors of smart shopping carts. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0035] FIG1 is a flow chart of a method for detecting abnormal shopping behavior in a smart shopping cart according to an embodiment of the present application;

[0036] FIG2 is a schematic diagram of the structure of an abnormal shopping behavior detection device for a smart shopping cart in an embodiment of the present application;

[0037] FIG3 is a schematic diagram of the connection structure of the smart shopping cart in an embodiment of the present application;

[0038] FIG4 is a schematic diagram of the connection structure of a smart shopping cart in another embodiment of the present application;

[0039] FIG5 is a schematic diagram of the structure of the smart shopping cart in an embodiment of the present application. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solutions and advantages of the embodiments of the present application more clear, the embodiments of the present application are further described in detail below with reference to the accompanying drawings. Here, the illustrative embodiments of the present application and their descriptions are used to explain the present application, but are not intended to limit the present application.

[0041] The acquisition, storage, use, and processing of data in the technical solution of this application comply with relevant laws and regulations.

[0042] This embodiment of the present application proposes a smart shopping cart solution for abnormal shopping behavior. This solution uses intelligent hardware to detect abnormal shopping behavior. The overall hardware device can be flexibly installed and removed from the shopping cart. The entire device includes a code scanner, computing and interaction equipment, a visual camera, a battery module, etc. Based on the shopping process with self-service code scanning, page interaction, and self-checkout, it focuses on detecting and preventing some potential abnormal shopping behaviors, such as placing items without scanning the code, scanning item A and placing item B, and interfering with sensors to place items. The following is a detailed introduction to the abnormal shopping behavior solution for the smart shopping cart.

[0043] FIG1 is a flow chart of a method for detecting abnormal shopping behavior in a smart shopping cart according to an embodiment of the present application. As shown in FIG1 , the method includes the following steps:

[0044] Step 301: Obtain product scan data and basket area video frame image data during the user's shopping process;

[0045] Step 302: Segment the products in each frame of the basket area video frame image data to obtain image data of each target product in the basket area in each frame of the image;

[0046] Step 303: Tracking the trajectory of each target product based on the image data of each target product in the basket area in the multiple frames of images to obtain trajectory tracking data of each target product;

[0047] Step 304: Determine the movement direction and distance of each target product's tracking trajectory based on the trajectory tracking data of each target product;

[0048] Step 305: Determine whether each target product is in a shopping cart or out of a shopping cart based on the movement direction and distance of each target product's tracking trajectory and a preset product movement distance threshold;

[0049] Step 306: Detect abnormal shopping behavior of the user based on the code scanning data, the trajectory tracking data of each target product in the basket area, and the status of each target product being placed in or taken out of the shopping cart.

[0050] In an embodiment of the present application, a method for detecting abnormal shopping behavior in a smart shopping cart operates as follows: obtaining scanned code data of products and video frame image data of a basket area during a user's shopping behavior; segmenting the products in each frame of the video frame image data of the basket area to obtain image data of each target product in the basket area in each frame; tracking the trajectory of each target product based on the image data of each target product in the basket area in multiple frames of images to obtain trajectory tracking data of each target product; determining a movement direction and distance of the tracking trajectory of each target product based on the trajectory tracking data of each target product; determining whether each target product is in a state of being placed in a shopping cart or taken out of a shopping cart based on the movement direction and distance of the tracking trajectory of each target product and a preset product movement distance threshold; and detecting abnormal shopping behavior of the user based on the scanned code data, the trajectory tracking data of each target product in the basket area, and the state of each target product being placed in a shopping cart or taken out of a shopping cart.

[0051] The embodiments of the present application enable a smart shopping cart to have functions such as self-service checkout and smart shopping, while also having the function of detecting abnormal shopping behavior. The method can combine code scanning data and collected video frame image data of the basket area to detect abnormal shopping behavior of users. The beneficial technical effects of the abnormal shopping behavior detection method of the smart shopping cart provided by the embodiments of the present application are:

[0052] First, the embodiment of the present application segments the products in each frame of the basket area video frame image data and tracks the trajectory of the target products, thereby avoiding the problem of products stacking and blocking each other that affects the detection results, and improving the accuracy of detecting abnormal shopping behavior of the smart shopping cart.

[0053] Secondly, the embodiment of the present application determines the direction and distance of movement of each target product based on the tracking trajectory data of each target product; based on the direction and distance, it can accurately determine the status of each target product as being placed in or taken out of the car, and then based on the status, the user's abnormal shopping behavior can be accurately detected.

[0054] In summary, the embodiments of the present application can accurately detect abnormal shopping behaviors of smart shopping carts.

[0055] The following is a detailed description of the abnormal shopping behavior detection method for the smart shopping cart in an embodiment of the present application.

[0056] In order to facilitate understanding of how to implement this application, the overall architecture of the smart shopping cart mentioned in the embodiment of this application is first introduced.

[0057] As shown in Figures 4 and 5, an embodiment of the present application provides an overall hardware device that can be flexibly installed and disassembled, including the necessary hardware and running software of a smart shopping cart. The necessary hardware includes a barcode scanner 10 as shown in Figure 5, a computing and interactive device (including a computing module and an interactive module as shown in Figure 4) 30, and a battery module as shown in Figure 4 (not shown in Figure 5), and a number of visual cameras 20 (acquisition devices) are added on the basis to capture video frame images of the shopper's shopping behavior process. The running software (the abnormal shopping behavior detection method for a smart shopping cart provided in an embodiment of the present application, which method can be implemented by the computing module in Figure 4, which is the abnormal shopping behavior detection device for a smart shopping cart) combines the data of the barcode scanner and the collected image data to analyze whether the shopping process is normal and whether there are some potential abnormal shopping behaviors, and generates relevant prompt information to be presented on the interactive device (such as the interactive module in Figure 4).

[0058] Specifically:

[0059] 1. Barcode scanner: This can be a common barcode scanner gun or scanner that can scan and identify the barcode on the product. The barcode scanner connects to the interactive device and sends the recognition results (product identification, such as product name and / or barcode information, as well as product price information and scanning time, etc.) to the computing and interactive device.

[0060] 2. As shown in Figure 5, the visual camera includes an RGB camera; it is installed above the shopping cart basket and above the barcode scanning area. For example, there can be more than one camera; it is connected to the computing and interaction device and transmits data. The visual camera covers two areas: the camera installed above the shopping cart basket (basket area acquisition device) has a visual range of 40 meters within the basket area; the camera installed above the barcode scanner (scanning area acquisition device) has a visual range of 50 meters within the barcode scanning area.

[0061] 3. Computing and interactive device: an electronic device with a touch screen. The computing and interactive device is an computing processing unit with a certain computing power (such as the computing unit in Figure 4), which can be installed in the handle of the shopping cart and connected to the barcode scanner and visual camera. The computing and interactive device can run the operating system and software, and present the data on the interactive screen (as implemented by the "interactive module" shown in Figure 4), thereby realizing basic functions such as self-service scanning, page interaction, and self-service checkout. The computing and interactive device can process the data transmitted by the barcode scanner and visual camera, and present the data or results on the interactive screen, thereby realizing the discovery and prevention of some potential abnormal shopping behaviors.

[0062] 4. Battery module: Provides power to the entire device. When the entire device is removed from the shopping cart, the battery module can be charged.

[0063] The abnormal shopping behavior detection method of the smart shopping cart involved in the embodiment of the present application is described in detail below with reference to Figures 4 and 5.

[0064] 1. Obtaining video frame images of the shopping process

[0065] After starting shopping, start the visual camera to obtain video frame images of the shopping process.

[0066] 2. Perform inference operations on image data

[0067] The computing module shown in FIG4 processes each frame of received image data using computer vision algorithms, machine learning algorithms, and deep learning algorithms based on the received image data.

[0068] 1. Detect and record the products at the time of scanning in the scanning area

[0069] When the scanner scans a barcode, it detects the product in the scanning area and saves the product image data corresponding to the product, that is, the video frame image data of the scanning area (this image data can be used in the following step of determining incorrect scanning behavior). As shown in the above step 301, this step obtains the product scanning data during the scanner user's shopping behavior.

[0070] 2. For the basket area, determine whether the target product is placed in the vehicle or taken out of the vehicle, that is, steps 302 to 305 above.

[0071] If an RGB camera is configured, the target tracking algorithm will be used to determine whether the product enters or leaves the shopping cart based on the trajectory of the target product. The detailed steps are as follows:

[0072] 1) Perform target detection or segmentation on products.

[0073] For video frame image data, use a target detection algorithm model or image segmentation algorithm model to detect or segment the products within the image, outputting a target bounding box B1 (first target bounding box) or target mask A1 (first target mask) containing the products. The target detection algorithm model and image segmentation algorithm model are models trained using deep learning algorithms using training data, including but not limited to convolutional neural network models and transformer network models. Target detection algorithm models include but are not limited to one-stage and two-stage algorithms, such as SSD and the YOLO series of algorithms. Segmentation algorithm models include but are not limited to YOLACT, SOLO, YOLO, and the Segmentation Anything model.

[0074] In specific implementations, the target bounding box can be a rectangular bounding box that completely encloses the target product, while the product mask is a pixel block that surrounds the target product. When to use the target bounding box and when to use the mask depends on the algorithm model used to detect the product. If an object detection model is used, the target bounding box is used; if a segmentation model is used, the mask is used.

[0075] 2) Filter product targets.

[0076] Perform multiple layers of different filtering on product targets, including the following two:

[0077] A. Filtering out falsely detected interferences

[0078] In order to prevent the algorithm model from incorrectly detecting or segmenting non-product objects in step 1) and affecting subsequent judgment, it is necessary to check and filter the detected or segmented product objects. The specific method is as follows:

[0079] First, detect or segment the interfering object. Using a target detection or image segmentation algorithm, detect or segment the interfering object (such as a hand, arm, or phone) within the video frame image data. Output is a target bounding box B2 (a second target bounding box) or a target mask A2 (a second target mask, which contains the interfering object's pixels). Both the interfering object mask and the target bounding box can completely encompass a target interfering object.

[0080] Then, the product is verified. Using the product's target detection bounding box B1 or target mask A1 from step 1), an intersection-and-union calculation is performed with the interfering object's target detection bounding box B2 or target mask A2. This calculation calculates the ratio a of the overlapping areas of the two targets. If the overlapping area ratio a is greater than a set threshold, the target detection bounding box or target mask A1 is considered an interfering object and is removed. Otherwise, the target detection bounding box or target mask A1 is determined to be a genuine product and recorded in the product set S (recording the bounding box information or mask information of each product and the information of each product image).

[0081] During specific implementation, in the above step 302, the commodities in each frame of the basket area video frame image data are segmented, and the position information and image data of each target commodity in the basket area in each frame of the image can be obtained; in the above step 303, the trajectory of each target commodity is tracked based on the position information and image data of each target commodity in the basket area in multiple frames of images, and the trajectory tracking data of each target commodity is obtained.

[0082] As can be seen from the above, in one embodiment, if the device for collecting the basket area video frame image data is an RGB camera, segmenting the products in each frame of the basket area video frame image data to obtain the location information and image data of each target product in the basket area in each frame may include:

[0083] Segmenting the products in each frame of the basket area video frame image data, outputting a first target mask containing each target product, and forming a target product set based on each target product, its corresponding first target mask, and the image information;

[0084] Segmenting the distractors in each frame of the video frame image data of the basket area, and outputting a second target mask containing the target distractors;

[0085] Using the first target mask of the target product and the second target mask of the target interferer to calculate the intersection-and-union ratio, the ratio of the overlapping area between the first target mask and the second target mask is obtained;

[0086] When the ratio of the overlapping areas of the first target mask and the second target mask is greater than a preset overlapping ratio threshold, the target product included in the first target mask is removed from the target product set as an interference object.

[0087] In specific implementation, the above-mentioned target mask intersection-union ratio calculation of the product and the interference object is used to filter the segmented and detected products, thereby avoiding the impact of incorrectly segmented interference objects on the subsequent judgment of abnormal shopping behavior, and improving the detection accuracy of subsequent abnormal shopping behavior.

[0088] As can be seen from the above, in one embodiment, if the device for collecting the basket area video frame image data is an RGB camera, target detection is performed on the products in each frame of the basket area video frame image data to obtain the position information and image data of each target product in the basket area in each frame, and the following steps may also be included:

[0089] Performing target detection on the products within each frame of the basket area video frame image data, outputting a first target bounding box containing each target product, and forming a target product set based on each target product, its corresponding first target bounding box, and image information;

[0090] Performing target detection on the interference object in each frame of the video frame image data of the basketball area, and outputting a second target bounding box containing the target interference object;

[0091] Calculate the intersection-and-union ratio of the first target bounding box of the target product and the second target bounding box of the target interferer to obtain the ratio of the overlapping area between the first target bounding box and the second target bounding box;

[0092] When the ratio of the overlapping areas of the first target bounding box and the second target bounding box is greater than a preset overlapping ratio threshold, the target product contained in the first target bounding box is removed from the target product set as an interference object.

[0093] In specific implementation, the above-mentioned target bounding box intersection-union ratio calculation of the product and the interference object is used to filter the target detected products, thereby avoiding the influence of the interference object detected by the wrong target on the subsequent judgment of abnormal shopping behavior, and improving the detection accuracy of subsequent abnormal shopping behavior.

[0094] As can be seen from the above, in one embodiment, the abnormal shopping behavior detection method of the smart shopping cart further includes: when the ratio of the overlapping area is less than a preset overlapping ratio threshold, determining that the first target mask contains content that is real and retaining the product in the target product set.

[0095] B. Filter out products that are in motion

[0096] In order to more accurately track and calculate the trajectory of the goods placed in or taken out, the embodiment of the present application proposes a method for screening out goods in motion.

[0097] As can be seen from the above, in one embodiment, the abnormal shopping behavior detection method of the smart shopping cart further includes:

[0098] Target commodities in motion are filtered out from the target commodity set and added to the final target commodity set; image data of each target commodity in the basket area in the multi-frame image is commodity data taken from the final target commodity set.

[0099] The specific method for filtering out products in motion is as follows:

[0100] Use algorithms to calibrate, screen, and filter products in motion. The specific methods are as follows:

[0101] a) Calibrate the pixels in motion in the image

[0102] Using a foreground extraction algorithm or background subtraction algorithm on the video frame data, the moving target area is calibrated and the binary image mask A3 corresponding to the calibrated area is extracted as the foreground mask A3 for each frame. The foreground extraction algorithm includes but is not limited to the Gaussian mixture model background subtraction method, frame difference method, optical flow method, KNN algorithm, etc.

[0103] b) Select the sports target

[0104] Using target mask A1, we crop mask A1' (cropped mask) from mask A3 (binary image mask) to create the same position as A1. We then calculate the intersection-over-union (IoU) of A1' and A1, i.e., the ratio a of the overlapping areas of the two masks. If the overlapping area ratio a is greater than a set threshold, the product corresponding to mask A1 is considered to be moving and is incorporated into the product set S (recording each product's bounding box information or mask information, as well as each product image information).

[0105] As can be seen from the above, in one embodiment, filtering out target products in motion from the target product set and adding them to the final target product set may include:

[0106] Calibrate the area of ​​moving pixels in each frame of the target product set, and extract the binary image mask corresponding to the calibration area as the foreground mask of each frame;

[0107] A cropping mask is cut out from the foreground mask at the same position as the first target mask, and an intersection-over-union ratio of the cropping mask and the first target mask is calculated to obtain a ratio of overlapping areas between the cropping mask and the first target mask.

[0108] When the overlapping area ratio between the cropping mask and the first target mask is greater than a preset overlapping ratio threshold, it is determined that the target product included in the first target mask is in motion, and the target product in motion is added to the final target product set.

[0109] This concludes the introduction to steps 1) detecting or segmenting products and 2) filtering products in step 302, along with their preferred implementation. Using machine learning or deep learning methods to detect, segment, and track products is more practical in real-world applications. This allows for recording and analyzing the process of product placement and removal, as well as counting the number of items placed or removed. This avoids the possibility of missing the exact number of items placed or removed due to overlapping or obstructing items during the time delay.

[0110] 3) Tracking and recording the target product, i.e., the above-mentioned step 303.

[0111] For the products in the product set S, use the information of each product to track the trajectory of each product. The methods used include but are not limited to the following methods:

[0112] A. Tracking using multi-target tracking algorithms

[0113] For each product target in the product set s, the target bounding box information or key point description data is used as the input data of multi-target tracking, and the target tracking trajectory information is finally output. The target tracking used is a common multi-target tracking algorithm, including:

[0114] a) Perform Kalman filter prediction on each target trajectory to calculate and predict the target's motion pattern and state (speed, direction, acceleration, trajectory);

[0115] b) Describe the characteristics of the target, including the appearance characteristics of the target;

[0116] c) Use the Hungarian algorithm to perform data association matching on the predicted data and feature description data.

[0117] B. Use the reid algorithm for tracking calculation

[0118] The commodity target is extracted as feature description data, and then the similarity of the feature description data of the target commodity in each frame is compared, and the feature description data within the threshold range is matched as the trajectory ID (identification) of the same target.

[0119] C. Algorithms using visual tracking

[0120] A model trained based on training data using a deep learning method can process target bounding box information or key point description data and timing information, and directly output target tracking trajectory information.

[0121] Among them, for each product in the product set S, a different trajectory id is assigned, and the number of products put in or taken out can be determined according to the number of ids.

[0122] 4) Calculate the trajectory of the commodity target, i.e., the above-mentioned step 304.

[0123] For each tracking trajectory, calculate its direction and distance of movement. The specific method is:

[0124] For each tracking trajectory, the trajectory point set information containing at least the starting point and the end point is sampled, and the trajectory point set information is input into a machine learning model to output information such as the direction of movement, the longest moving distance, etc.

[0125] Among them, the machine learning model includes but is not limited to the XGboost regression model. After training, it receives input trajectory point set information and can output information such as movement direction and movement difference.

[0126] As can be seen from the above, in one embodiment, determining the movement direction and distance of each target product tracking trajectory based on the trajectory tracking data of each target product includes:

[0127] For each tracking trajectory, sample the trajectory point set information that at least includes the starting point and the end point;

[0128] The trajectory point set information is input into the pre-trained trajectory machine learning model to output the movement direction and distance of each target product tracking trajectory.

[0129] In specific implementation, the above-mentioned method of determining the movement direction and distance of each target product tracking trajectory can improve the efficiency and accuracy of trajectory calculation, and further improve the accuracy and efficiency of detecting abnormal shopping behavior.

[0130] 5) Determine the state of the target product being placed in or taken out, i.e., the above-mentioned step 305.

[0131] According to the direction and distance of the target motion trajectory, if the motion trajectory of a product is from outside the shopping cart to inside the shopping cart, and the motion distance reaches the threshold, the product is judged to be in the state of being placed in the cart; if the motion trajectory of a product is from inside the shopping cart to outside the cart, and the motion distance reaches the threshold (preset product motion distance threshold), the product is judged to be taken out.

[0132] As can be seen from the above, in one embodiment, determining whether each target product is in a shopping cart or out of a shopping cart based on the movement direction and distance of each target product's tracking trajectory and a preset product movement distance threshold includes:

[0133] When the target product tracking trajectory moves from outside the shopping cart into the shopping cart, and the movement distance reaches a preset product movement distance threshold, the target product is determined to be in the shopping cart state; when the target product tracking trajectory moves from inside the shopping cart to outside the shopping cart, and the movement distance reaches a preset product movement distance threshold, the target product is determined to be in the shopping cart state. It should be noted that the preset product movement distance threshold can be determined based on actual conditions.

[0134] 3. The loss prevention logic module performs data processing, i.e., the above-mentioned step 306.

[0135] 1. Receive and aggregate all data

[0136] The logic processing module receives the detection and tracking data output by the calculation module (e.g., the trajectory tracking data of each target product in the basket area calculated above, and the status of each target product being placed in or removed from the shopping cart), the barcode scanning data of the scanner, and the operation instructions of the interactive interface (e.g., instructions for deleting a product item or logging in to a member account).

[0137] 2. Determine abnormal behavior based on fused data

[0138] 1) Align the scan data time and scan quantity to determine missed scans

[0139] For detection and tracking data of products placed in the basket area, scan data is collected within a threshold time t before the placement time. Within the threshold time range, if the basket area detection and identification data shows that n new products (as shown above, the number of products is determined by the number of product tracking tracks) enter the vehicle, but the newly received scan data is less than n, then it is determined that "missed scan" has occurred.

[0140] In one embodiment, the trajectory tracking data of each target product includes: the number of products determined based on the number of trajectories; the trajectory tracking data of each target product in the basket area based on the scanned code data; and the status of each target product being placed in or removed from the shopping cart. Detecting abnormal shopping behavior of a user includes:

[0141] For the trajectory tracking data of each target product in the basket area that is in the shopping cart state, the scan code data within a preset time threshold range before the placement time is obtained. Within the preset time threshold range, if the number of products in the scan code data is less than the number of products placed in the shopping cart according to the trajectory tracking data of the basket area, it is detected that the user has missed scanning.

[0142] 2) Determine the wrong scanning behavior

[0143] A. Identify the products placed in

[0144] For n commodities placed in the detection and tracking data of the basket area, commodity identification is performed on each commodity to determine the specific commodity identifier corresponding to each commodity, such as the commodity name or barcode information.

[0145] Methods for identifying products include but are not limited to: using deep learning algorithm models for classification and identification, using retrieval calculation methods for retrieval and identification, and outputting the result with the highest confidence as the identification result.

[0146] B. Compare the scan results with the product identification results

[0147] In specific implementation, the judgment of wrong scanning behavior may include the following two methods:

[0148] Implementation method one: For n scan code data, use the n product identification results entered for matching. If the comparison results are consistent, the scan code data and the identification data are offset one-to-one (the product information corresponding to the scan code data and the identification data can be matched one-to-one). If the n scan code data and the n identification data can be offset one-to-one, it is determined to be "normal behavior"; if it is finally found that there is product identification data that cannot be offset, it is determined to be "wrong scanning behavior".

[0149] As can be seen from the above, in one embodiment, based on the code scanning data, the trajectory tracking data of each target product in the basket area, and the status of each target product being placed in or removed from the shopping cart, detecting abnormal shopping behavior of the user includes:

[0150] Perform product recognition on each target product image data in the shopping cart state in the trajectory tracking data of the basket area, and determine product recognition data corresponding to each target product;

[0151] Each commodity identification information in the scanned code data is matched one by one with each commodity identification information in the commodity identification data. When it is detected that the identification information does not match one by one, it is detected that the user has made an incorrect scanning behavior.

[0152] In specific implementation, based on the process and quantity of goods being put in or taken out, combined with the scanning data of the scanner and the result data of the goods identification, the wrong scanning behavior can be judged more accurately. For example, when the order of scanning and putting in goods is out of order, using a one-to-one offset logic method can show better judgment results and present a more friendly experience.

[0153] Implementation Method 2: In addition to using product identification to determine mis-scanning, product images detected in the scanned area can also be used. Specifically, within a time range t, obtain product image data from the scanned area's detection and identification data, and simultaneously obtain an image of the product placed in the cart from the basket area. Calculate the similarity between the two product images, including but not limited to comparisons of similarity in product appearance, color, texture, and size. If significant differences are found between the two images, a mis-scan is determined.

[0154] As can be seen from the above, in one embodiment, the above-mentioned method for detecting abnormal shopping behavior of the smart shopping cart further includes: obtaining video frame image data of the code scanning area during the user's shopping behavior;

[0155] Based on the scan data, the trajectory tracking data of each target product in the basket area, and the status of each target product being placed in or removed from the shopping cart, the user's abnormal shopping behavior is detected, including:

[0156] Obtain image data of each target product in the scanning area and image data of each target product in the basket area in each frame of video frame image data of the scanning area within a preset time threshold range;

[0157] Calculate the similarity between the image data of each target product in the scan area and the image data of each target product in the basket area;

[0158] When the similarity is less than the preset product similarity threshold, it is detected that the user has made a wrong scan.

[0159] 3) Determining Multiple-Taking Behavior

[0160] Within the threshold time range, if the shopping cart deletes n (the number of items deleted on the interactive screen) items from the interactive screen, but the detection and identification data of the basket area contains data of less than n items taken out of the cart, it is determined to be "multiple-taking behavior".

[0161] As can be seen from the above, in one embodiment, the trajectory tracking data of each target product includes: the number of products determined based on the number of trajectories; the trajectory tracking data of each target product in the basket area based on the scanned code data; and the status of each target product being placed in or removed from the shopping cart. Detecting abnormal shopping behavior of a user includes:

[0162] For each target product image data in the trajectory tracking data of the basket area that is in the state of being taken out of the shopping cart, when it is detected within a preset time threshold range that the number of target products taken out of the basket area and out of the shopping cart is less than the number of products deleted on the interactive screen, it is detected that the user has performed multiple taking behavior.

[0163] 4. The following describes further preferred embodiments of the present application.

[0164] 1. First, we introduce how to identify shopper actions and judge user occlusion behavior and abnormal actions on products based on the action type recognition results.

[0165] 1) Detect and identify the occurrence of action within the basket area.

[0166] When a user action is detected in the basket area, the start and end times t1 and t2 of the action are recorded;

[0167] 2) Sampling the video frame image

[0168] Based on the determined action occurrence time, the video frame image data is truncated;

[0169] 3) Recognize the action

[0170] The captured video frame sequences are fed into a machine learning or deep learning algorithm model, which then outputs the results of the shopper's action recognition. These actions include placing items in, taking items out, arranging items, and feinting. The most important thing is to confirm that the shopper is placing or taking items out.

[0171] For the basket area, action behaviors are identified, with special designations including but not limited to the following methods: For the video frame image sequence of the entire process, a machine learning algorithm or a deep learning algorithm model is used to identify the shopper's actions and detect abnormal shopper actions (action type data), such as stacking and covering products, changing product packaging, switching product barcodes, and blocking visual sensors.

[0172] During specific implementation, in addition to the above-mentioned abnormal shopping behavior judgments such as missed scanning, wrong scanning, and multiple picking behaviors, abnormal shopping behavior detection can also be performed based on the above-mentioned action recognition results.

[0173] In specific implementations, if the visual image of the basket area shows anomalies, the computing module infers that the camera is intentionally blocked or obscured by stacked items. This is considered "obstruction behavior," preventing shoppers from placing items in the basket while the camera is blocked. If the motion recognition data for the basket area shows unusual movements, such as stacking or obstructing items or changing product packaging, this is considered "shopping behavior involving unusual actions with the products."

[0174] As can be seen from the above, in one embodiment, the present invention further includes:

[0175] When a user action is detected in the basket area, the time of the user action is recorded;

[0176] Based on the time when the user action occurs, the video frame image sequence to be detected is intercepted from the video frame image data of the basketball area;

[0177] Input the video frame image sequence to be detected into the pre-trained action type recognition model and output the user's action type data;

[0178] Based on the action type data, detect the user's obstruction behavior and shopping behavior that involves abnormal actions on products (such as stacking and obstructing products, changing product packaging, switching product barcodes, blocking visual sensors, etc.).

[0179] 2. Secondly, we introduce the solution for reporting and prompting detected abnormal shopping behaviors.

[0180] 1) If there is a "missed scan", the shopper will be prompted to take out the unscanned items, scan them again and put them back in.

[0181] 2) If there is a "wrong scan behavior", the shopper will be prompted to remove the incorrectly placed product, delete the original product entry from the interactive screen, scan the code again and put it back in.

[0182] 3) If there is "multiple-taking behavior", the shopper will be prompted or prohibited from deleting more product items.

[0183] 4) If there is "blocking behavior", the shopper will be prompted to stop blocking and allowed to continue shopping after recovery.

[0184] 5) If there is "shopping behavior with abnormal actions on the goods", it will prompt that there is abnormal shopping behavior with abnormal actions on the goods.

[0185] As can be seen from the above, in one embodiment, the abnormal shopping behavior detection method of the smart shopping cart further includes:

[0186] When detecting that the user's abnormal shopping behavior is missing a scan, prompt the user to remove the unscanned items from the shopping cart, scan the code again, and then put them back in;

[0187] When it is detected that the user's abnormal shopping behavior is a wrong scanning behavior, the user is prompted to remove the wrongly placed product from the shopping cart, delete the scanned product item corresponding to the wrongly placed product from the interactive screen, and scan the code again to add the product again;

[0188] When it is detected that a user's abnormal shopping behavior is multiple purchases, the user is prohibited from deleting the corresponding product items;

[0189] When detecting that the user's abnormal shopping behavior is blocking the screen, prompt the user to stop blocking the screen and allow the user to continue shopping after recovery.

[0190] When it is detected that the user's abnormal shopping behavior is a shopping behavior in which an abnormal action is taken on a product, the user is prompted that an abnormal action is taken on the product.

[0191] In practice, the aforementioned reporting and prompting scheme for detected abnormal shopping behavior automatically and promptly alerts shoppers of irregular behavior, allowing them to correct it, reducing manual intervention by the supermarket and improving the user experience. Specifically, the reporting scheme may include sending the detected abnormal shopping behavior to the supermarket's user terminal, allowing for manual intervention if the shopper fails to correct it.

[0192] 3. Again, introduce the solution of entering the payment process and checking the shopping list.

[0193] If the shopper clicks the interaction button and attempts to proceed to the checkout process, check whether the shopping list still has any abnormal behavior that has not been eliminated:

[0194] 1) If so, prompt the shopper to correct the behavior.

[0195] 2) If not, proceed with the payment process.

[0196] When implemented specifically, the above-mentioned solution of entering the payment process and checking the shopping list ensures smooth and safe shopping in the shopping cart.

[0197] The advantages of the abnormal shopping behavior method of the smart shopping cart provided in the embodiment of the present application are:

[0198] 1) Provide an integrated hardware device that can be flexibly installed and removed from the shopping cart. While providing functions such as self-service checkout and smart shopping, it also has the function of detecting abnormal shopping behavior of users. In particular, based on the hardware of common smart shopping carts, several visual cameras (especially cameras in the code scanning area) are added to cover the shopping cart basket area or the code scanning area to capture the shopper's shopping behavior process. There is software that can run on the entire device. This software can combine the data from the code scanner and the collected image data to analyze whether the shopping process is normal and whether there are some potential theft behaviors, and prompt information on the interactive device.

[0199] 2) Using machine learning or deep learning methods to detect, segment, and track products, or identify actions, is more practical in actual application scenarios. It can record and analyze the process of placing or removing products, and can count the number of products placed in or removed, avoiding the problem of being unable to detect the specific number of products placed in or removed due to the stacking or blocking of products during the judgment time difference.

[0200] 3) Based on the process and quantity of items being put in or taken out, combined with the barcode scanner's scanning data and the resulting product identification data, behaviors such as incorrect scanning can be more accurately judged. For example, when the order of scanning and putting in items is out of sequence, using a one-to-one offsetting logic method can produce better judgment results and present a more user-friendly experience.

[0201] In summary, the abnormal shopping behavior method of the smart shopping cart provided in the embodiments of the present application achieves:

[0202] 1) Give ordinary shopping carts the ability to perform self-service checkout and smart shopping.

[0203] 2) Effectively detect irregular behaviors of shoppers and reduce potential abnormal shopping behaviors during self-service shopping.

[0204] 3) Automatically and promptly alert shoppers of irregular behavior, allowing them to make corrections and reducing manual intervention by supermarket staff.

[0205] 4) Reduce manual checking of checkout lists for self-service shopping carts at supermarket entrances and exits, thereby reducing operating costs.

[0206] The present application also provides a device for detecting abnormal shopping behavior in a smart shopping cart, as described in the following embodiments. Because the principles underlying the device are similar to those of the method for detecting abnormal shopping behavior in a smart shopping cart, the implementation of the device can be referenced to the implementation of the method for detecting abnormal shopping behavior in a smart shopping cart, and any repetitions will not be repeated.

[0207] FIG2 is a flow chart of an abnormal shopping behavior detection device for a smart shopping cart according to an embodiment of the present application. As shown in FIG2 , the device includes:

[0208] An acquisition unit 31 is used to acquire the scanned code data of the product and the video frame image data of the basket area during the user's shopping behavior;

[0209] A segmentation processing unit 32 is configured to segment the commodities in each frame of the basket area video frame image data to obtain image data of each target commodity in the basket area in each frame of the image;

[0210] A trajectory tracking unit 33 is configured to track the trajectory of each target commodity based on the image data of each target commodity in the basket area in the multiple frames of images to obtain trajectory tracking data of each target commodity;

[0211] The trajectory processing unit 34 is used to determine the movement direction and distance of each target product tracking trajectory based on the trajectory tracking data of each target product;

[0212] A state determination unit 35 is configured to determine whether each target product is in a state of being placed in a shopping cart or taken out of a shopping cart based on the movement direction and distance of each target product's tracking trajectory and a preset product movement distance threshold;

[0213] The detection unit 36 ​​is used to detect the user's abnormal shopping behavior based on the code scanning data, the trajectory tracking data of each target product in the basket area, and the status of each target product being placed in or taken out of the shopping cart.

[0214] In one embodiment, if the device for collecting the video frame image data of the basket area is an RGB camera, the segmentation processing unit is specifically configured to:

[0215] Segment the products within each frame of the basket area video frame image data, output a first target mask containing each target product, and form a target product set based on each target product, its corresponding first target mask, and the image information; where the mask is a pixel block surrounding the product;

[0216] Segmenting the distractors in each frame of the video frame image data of the basket area, and outputting a second target mask containing the target distractors;

[0217] Using the first target mask of the target product and the second target mask of the target interferer to calculate the intersection-and-union ratio, the ratio of the overlapping area between the first target mask and the second target mask is obtained;

[0218] When the ratio of the overlapping areas of the first target mask and the second target mask is greater than a preset overlapping ratio threshold, the target product included in the first target mask is removed from the target product set as an interference object.

[0219] In one embodiment, the segmentation processing unit is further configured to: when the ratio of the overlapping areas is less than a preset overlapping ratio threshold, determine that the first target mask contains authentic products and retain them in the target product set.

[0220] In one embodiment, the abnormal shopping behavior detection device further includes:

[0221] The screening processing unit is used to filter out target commodities in motion from the target commodity set and add them to the final target commodity set; the image data of each target commodity in the basket area in the multi-frame image is the commodity data taken from the final target commodity set.

[0222] In one embodiment, the screening processing unit is specifically configured to:

[0223] Calibrate the area of ​​moving pixels in each frame of the target product set, and extract the binary image mask corresponding to the calibration area as the foreground mask of each frame;

[0224] A cropping mask is cut out from the foreground mask at the same position as the first target mask, and an intersection-over-union ratio of the cropping mask and the first target mask is calculated to obtain a ratio of overlapping areas between the cropping mask and the first target mask.

[0225] When the overlapping area ratio between the cropping mask and the first target mask is greater than a preset overlapping ratio threshold, it is determined that the target product included in the first target mask is in motion, and the target product in motion is added to the final target product set.

[0226] In one embodiment, the trajectory processing unit is specifically configured to:

[0227] For each tracking trajectory, sample the trajectory point set information that at least includes the starting point and the end point;

[0228] The trajectory point set information is input into the pre-trained trajectory machine learning model to output the movement direction and distance of each target product tracking trajectory.

[0229] In one embodiment, the state determination unit is specifically configured to:

[0230] When the movement direction of the target product tracking trajectory is from outside the shopping cart to inside the shopping cart, and the movement distance reaches the preset product movement distance threshold, the target product is determined to be in the state of being placed in the shopping cart; when the movement direction of the target product tracking trajectory is from inside the shopping cart to outside the shopping cart, and the movement distance reaches the preset product movement distance threshold, the target product is determined to be in the state of being taken out of the shopping cart.

[0231] In one embodiment, the trajectory tracking data of each target product includes: the number of products determined according to the number of trajectories; the detection unit is specifically configured to:

[0232] For the trajectory tracking data of each target product in the basket area that is in the shopping cart state, the scan code data within a preset time threshold range before the placement time is obtained. Within the preset time threshold range, if the number of products in the scan code data is less than the number of products placed in the shopping cart according to the trajectory tracking data of the basket area, it is detected that the user has missed scanning.

[0233] In one embodiment, the detection unit is specifically configured to:

[0234] Perform product recognition on each target product image data in the shopping cart state in the trajectory tracking data of the basket area, and determine product recognition data corresponding to each target product;

[0235] Each commodity identification information in the scanned code data is matched one by one with each commodity identification information in the commodity identification data. When it is detected that the identification information does not match one by one, it is detected that the user has made an incorrect scanning behavior.

[0236] In one embodiment, the abnormal shopping behavior detection device of the smart shopping cart may further include: a code scanning area image acquisition unit for acquiring video frame image data of the code scanning area during the user's shopping behavior;

[0237] The detection unit is specifically used for:

[0238] Obtain image data of each target product in the scanning area and image data of each target product in the basket area in each frame of video frame image data of the scanning area within a preset time threshold range;

[0239] Calculate the similarity between the image data of each target product in the scan area and the image data of each target product in the basket area;

[0240] When the similarity is less than the preset product similarity threshold, it is detected that the user has made a wrong scan.

[0241] In one embodiment, the trajectory tracking data of each target product includes: the number of products determined according to the number of trajectories; the detection unit is specifically configured to:

[0242] For each target product image data in the trajectory tracking data of the basket area that is in the state of being taken out of the shopping cart, when it is detected within a preset time threshold range that the number of target products taken out of the basket area and out of the shopping cart is less than the number of products deleted on the interactive screen, it is detected that the user has performed multiple taking behavior.

[0243] In one embodiment, the abnormal shopping behavior detection device of the smart shopping cart further includes:

[0244] An action detection unit, configured to record the time when a user action is detected in the basket area;

[0245] An interception unit, configured to intercept a sequence of video frame images to be detected from the video frame image data of the basket area based on the time when the user action occurs;

[0246] The recognition unit is used to input the video frame image sequence to be detected into the pre-trained action type recognition model and output the user's action type data;

[0247] The detection unit is also used to detect the user's blocking behavior and shopping behavior with abnormal actions on the goods based on the action type data.

[0248] In one embodiment, the abnormal shopping behavior detection device further includes a prompt processing unit configured to:

[0249] When detecting that the user's abnormal shopping behavior is missing a scan, prompt the user to remove the unscanned items from the shopping cart, scan the code again, and then put them back in;

[0250] When it is detected that the user's abnormal shopping behavior is a wrong scanning behavior, the user is prompted to remove the wrongly placed product from the shopping cart, delete the scanned product item corresponding to the wrongly placed product from the interactive screen, and scan the code again to add the product again;

[0251] When it is detected that a user's abnormal shopping behavior is multiple purchases, the user is prohibited from deleting the corresponding product items;

[0252] When detecting that the user's abnormal shopping behavior is blocking the screen, prompt the user to stop blocking the screen and allow the user to continue shopping after recovery.

[0253] When it is detected that the user's abnormal shopping behavior is a shopping behavior in which an abnormal action is taken on a product, the user is prompted that an abnormal action is taken on the product.

[0254] The present application also provides a smart shopping cart, as described in the following embodiments. Since the principles underlying the problem solved by the smart shopping cart are similar to those of the method for detecting abnormal shopping behavior in a smart shopping cart, the implementation of the smart shopping cart can be referenced to the implementation of the method for detecting abnormal shopping behavior in a smart shopping cart, and any repetitions will not be repeated.

[0255] FIG3 is a schematic diagram of the connection structure of the smart shopping cart in an embodiment of the present application. As shown in FIG3 , the smart shopping cart includes:

[0256] The barcode scanner 10 is used to obtain the barcode scanning data of the goods during the user's shopping behavior;

[0257] A basket area acquisition device 20 is used to acquire video frame image data of the basket area;

[0258] The abnormal shopping behavior detection device 60 is used to detect the abnormal shopping behavior of the user.

[0259] Specifically, the abnormal shopping behavior detection device is, for example, the abnormal shopping behavior detection device of the smart shopping cart shown in FIG2 .

[0260] In specific implementation, as shown in Figures 4 and 5, the smart shopping cart provided in the embodiment of the present application may also include other components such as a battery module and a visual camera. For details, please refer to the introduction of the overall architecture of the smart shopping cart above.

[0261] An embodiment of the present application also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the above-mentioned method for detecting abnormal shopping behavior of the smart shopping cart is implemented.

[0262] An embodiment of the present application further provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the above-mentioned method for detecting abnormal shopping behavior of the smart shopping cart is implemented.

[0263] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the above-mentioned method for detecting abnormal shopping behavior of the smart shopping cart.

[0264] The embodiments of the present application enable the smart shopping cart to have functions such as self-service checkout and smart shopping, while also having the function of detecting abnormal shopping behavior. The function can combine the scanned code data and the collected video frame image data of the basket area to detect the user's abnormal shopping behavior. The beneficial technical effects of the abnormal shopping behavior detection solution of the smart shopping cart provided by the embodiments of the present application are:

[0265] First, the embodiment of the present application segments the products in each frame of the basket area video frame image data and tracks the trajectory of the target products, thereby avoiding the problem of products stacking and blocking each other that affects the detection results, and improving the accuracy of detecting abnormal shopping behavior of the smart shopping cart.

[0266] Secondly, the embodiment of the present application determines the direction and distance of movement of each target product based on the tracking trajectory data of each target product; based on the direction and distance, it can accurately determine the status of each target product as being placed in or taken out of the car, and then based on the status, the user's abnormal shopping behavior can be accurately detected.

[0267] In summary, the embodiments of the present application can accurately detect abnormal shopping behaviors of smart shopping carts.

[0268] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0269] The present application is described with reference to the flow chart and / or block diagram of the method, device (system), and computer program product according to the embodiment of the present application. It should be understood that each flow process and / or box in the flow chart and / or block diagram and the combination of the flow process and / or box in the flow chart and / or block diagram can be realized by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processing machine or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for realizing the function specified in one flow chart flow or multiple flows and / or one box or multiple boxes of the block diagram.

[0270] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0271] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0272] The specific embodiments described above further illustrate the purpose, technical solutions and beneficial effects of the present application. It should be understood that the above description is only a specific embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application.

Claims

1. A method for detecting abnormal shopping behavior in a smart shopping cart, characterized in that: include: Obtain product scan data and basket area video frame image data during the user's shopping process; Segment the products in each frame of the basket area video frame image data to obtain image data of each target product in the basket area of ​​each frame; Tracking each target product according to the image data of each target product in the basket area in the multiple frames of images to obtain trajectory tracking data of each target product; Determine the movement direction and distance of each target product's tracking trajectory based on the trajectory tracking data of each target product; Determine whether each target product is in a shopping cart or out of a shopping cart based on the direction and distance of the tracking trajectory of each target product and a preset product movement distance threshold; as well as The user's abnormal shopping behavior is detected based on the code scanning data, the trajectory tracking data of each target product in the basket area, and the status of each target product being placed in or taken out of the shopping cart.

2. The method according to claim 1, wherein If the device for collecting the video frame image data of the basket area is an RGB camera, segmenting the products in each frame of the video frame image data of the basket area to obtain image data of each target product in the basket area in each frame includes: Segmenting the products in each frame of the basket area video frame image data, outputting a first target mask containing each target product, and forming a target product set based on each target product, its corresponding first target mask, and the image information; Segmenting the distractors in each frame of the video frame image data of the basket area, and outputting a second target mask containing the target distractors; Performing an intersection-and-union calculation using the first target mask of the target product and the second target mask of the target interferer to obtain a ratio of overlapping areas between the first target mask and the second target mask; and When the ratio of the overlapping areas of the first target mask and the second target mask is greater than a preset overlapping ratio threshold, the target product included in the first target mask is removed from the target product set as an interference object.

3. The method according to claim 2, wherein Also includes: When the ratio of the overlapping areas is less than a preset overlapping ratio threshold, it is determined that the first target mask contains authentic products, and the products are retained in the target product set.

4. The method according to claim 2, wherein Also includes: Filter out target products in motion from the target product set and add them to the final target product set; The image data of each target commodity in the basket area in the multiple frames of images is commodity data taken from the final target commodity set.

5. The method according to claim 4, wherein Filter out the target products in motion from the target product set and add them to the final target product set, including: Calibrate the area of ​​moving pixels in each frame of the target product set, and extract the binary image mask corresponding to the calibration area as the foreground mask of each frame; Cutting out a cropping mask at the same position as the first target mask on the foreground mask, calculating an intersection-over-union ratio of the cropping mask and the first target mask to obtain a ratio of overlapping areas between the cropping mask and the first target mask; and When the overlapping area ratio between the cropping mask and the first target mask is greater than a preset overlapping ratio threshold, it is determined that the target product included in the first target mask is in motion, and the target product in motion is added to the final target product set.

6. The method according to claim 1, wherein Determine the movement direction and distance of each target product's tracking trajectory based on the trajectory tracking data of each target product, including: For each tracking trajectory, sample the trajectory point set information including at least the starting point and the end point; and The trajectory point set information is input into the pre-trained trajectory machine learning model to output the movement direction and distance of each target product tracking trajectory.

7. The method according to claim 1, wherein Determining whether each target product is in a shopping cart state or a shopping cart state based on the movement direction and distance of each target product's tracking trajectory and a preset product movement distance threshold includes: When the movement direction of the target product tracking trajectory is from outside the shopping cart to inside the shopping cart, and the movement distance reaches the preset product movement distance threshold, the target product is determined to be in the state of being placed in the shopping cart; when the movement direction of the target product tracking trajectory is from inside the shopping cart to outside the shopping cart, and the movement distance reaches the preset product movement distance threshold, the target product is determined to be in the state of being taken out of the shopping cart.

8. The method according to claim 1, wherein The trajectory tracking data of each target product includes: the number of products determined based on the number of trajectories; based on the scanned code data, the trajectory tracking data of each target product in the basket area, and the status of each target product being placed in or removed from the shopping cart, detecting abnormal shopping behavior of the user, including: For the trajectory tracking data of each target product in the basket area that is in the shopping cart state, the scan code data within a preset time threshold range before the placement time is obtained. Within the preset time threshold range, if the number of products in the scan code data is less than the number of products placed in the shopping cart according to the trajectory tracking data of the basket area, it is detected that the user has missed scanning.

9. The method according to claim 1, wherein Detecting abnormal shopping behavior of the user based on the scanned code data, the trajectory tracking data of each target product in the basket area, and the status of each target product being placed in or taken out of the shopping cart, including: Performing product recognition on each target product image data in a shopping cart state in the trajectory tracking data of the basket area to determine product recognition data corresponding to each target product; and Each commodity identification information in the scanned code data is matched one by one with each commodity identification information in the commodity identification data. When it is detected that the identification information does not match one by one, it is detected that the user has made an incorrect scanning behavior.

10. The method according to claim 1, wherein Also includes: Obtain video frame image data of the scanned area during the user's shopping behavior; Detecting abnormal shopping behavior of the user based on the scanned code data, the trajectory tracking data of each target product in the basket area, and the status of each target product being placed in or taken out of the shopping cart, including: Obtain image data of each target product in the scanning area and image data of each target product in the basket area in each frame of video frame image data of the scanning area within a preset time threshold range; Calculate the similarity between each target product image data in the scanned area and each target product image data in the basket area; and When the similarity is less than a preset commodity similarity threshold, it is detected that the user has a wrong scanning behavior.

11. The method according to claim 1, wherein The trajectory tracking data of each target product includes: the number of products determined based on the number of trajectories; based on the scanned code data, the trajectory tracking data of each target product in the basket area, and the status of each target product being placed in or removed from the shopping cart, detecting abnormal shopping behavior of the user, including: For each target product image data in the trajectory tracking data of the basket area that is in the state of being taken out of the shopping cart, when it is detected within a preset time threshold range that the number of target products taken out of the basket area and out of the shopping cart is less than the number of products deleted on the interactive screen, it is detected that the user has performed multiple taking behavior.

12. The method according to claim 1, wherein Also includes: When a user action is detected in the basket area, the time of the user action is recorded; Based on the time when the user action occurs, a video frame image sequence to be detected is intercepted from the video frame image data of the basketball area; Inputting the video frame image sequence to be detected into the pre-trained action type recognition model, and outputting the user's action type data; and According to the action type data, the user's blocking behavior and shopping behavior with abnormal actions on the product are detected.

13. The method according to claim 1, wherein Also includes: When detecting that the user's abnormal shopping behavior is missing a scan, prompt the user to remove the unscanned items from the shopping cart, scan the code again, and then put them back in; When it is detected that the user's abnormal shopping behavior is a wrong scanning behavior, the user is prompted to remove the wrongly placed product from the shopping cart, delete the scanned product item corresponding to the wrongly placed product from the interactive screen, and scan the code again to add the product again; When it is detected that a user's abnormal shopping behavior is multiple purchases, the user is prohibited from deleting the corresponding product items; When detecting that the user's abnormal shopping behavior is blocking behavior, prompt the user to stop blocking and allow the user to continue shopping after recovery; and / or When it is detected that the user's abnormal shopping behavior is a shopping behavior in which an abnormal action is taken on a product, the user is prompted that an abnormal action is taken on the product.

14. A device for detecting abnormal shopping behavior of a smart shopping cart, characterized in that: include: An acquisition unit, configured to acquire the scanned code data of the product and the video frame image data of the basket area during the user's shopping behavior; a segmentation processing unit, configured to segment the commodities in each frame of the basket area video frame image data to obtain image data of each target commodity in the basket area in each frame of the image; a trajectory tracking unit, configured to track the trajectory of each target commodity according to the image data of each target commodity in the basket area in the multiple frames of images, and obtain trajectory tracking data of each target commodity; A trajectory processing unit, configured to determine the movement direction and distance of each target commodity tracking trajectory based on the trajectory tracking data of each target commodity; a state determination unit, configured to determine whether each target product is in a state of being placed in a shopping cart or taken out of a shopping cart based on the movement direction and distance of each target product's tracking trajectory and a preset product movement distance threshold; as well as The detection unit is used to detect the user's abnormal shopping behavior based on the code scanning data, the trajectory tracking data of each target product in the basket area, and the status of each target product being placed in or taken out of the shopping cart.

15. A smart shopping cart, characterized in that: include: The barcode scanner is used to obtain the product scanning data during the user's shopping behavior; A basket area acquisition device, used to acquire video frame image data of the basket area; as well as The abnormal shopping behavior detection device for a smart shopping cart as claimed in claim 14.

16. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 13 is implemented.

17. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 13 is implemented.

18. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 13 is implemented.

Images