Abnormal shopping behavior detection method, apparatus and system for shopping cart, and shopping cart

Abstract

The present application discloses an abnormal shopping behavior detection method, apparatus and system for a shopping cart, and a shopping cart. The method comprises: acquiring barcode scanning data of a barcode scanner and action data in a basket area detected by an action monitoring apparatus in the basket area; when it is determined, on the basis of the barcode scanning data, that a shopping action occurs, issuing a first image capture triggering signal; when it is determined, on the basis of the action data, that a shopping action occurs in the basket area, issuing a second image capture triggering signal; when it is determined, on the basis of the first image capture triggering signal and / or the second image capture triggering signal, that a shopping action occurs, triggering a visual image data acquisition apparatus to acquire video frame image data of a shopper during a shopping process; on the basis of the video frame image data, determining the shopping action type of the shopper and the number of commodities in a moving state; and on the basis of barcode scanner data, the shopping action type and the number of commodities, detecting whether the shopper has an abnormal shopping behavior.

Description

Methods, devices, systems, and shopping carts for detecting abnormal shopping behavior

[0001] Related applications

[0002] This application claims priority to Chinese Patent Application No. 202410841638.1, filed on June 26, 2024, and incorporates the entire contents of the aforementioned patent application as part of this application. Technical Field

[0003] This application relates to the field of self-service shopping technology, and in particular to a method, device, system, and shopping cart for detecting abnormal shopping behavior in a shopping cart. Background Technology

[0004] This section is intended to provide background or context for the embodiments of this application set forth in the claims. The description herein is not an admission that it is prior art simply because it is included in this section.

[0005] In recent years, with the continuous development of technologies such as the Internet of Things, artificial intelligence, big data analytics, mobile payment, and smart hardware, the supermarket industry is constantly transforming towards intelligence and digitalization in order to better meet customer needs, improve the customer shopping experience, and optimize merchant operations. Shopping checkout is a crucial part of this transformation. Smart shopping carts have emerged in the supermarket environment against this backdrop, primarily aiming to provide self-service shopping options, simplifying the traditional steps of shopping and payment, eliminating the need for customers to interact with cashiers at the checkout counter, and improving the shopping experience. Simultaneously, for merchants, it reduces the need for human cashiers, thereby reducing operating costs.

[0006] Generally, smart shopping carts have a range of advanced features, including automatic scanning and checkout, navigation and product location, item recognition and weighing, personalized recommendations and advertising, data analysis and inventory management. Self-service scanning and checkout are its core functions. These typically involve hardware such as barcode scanners, tablet computers, and vision cameras to intelligently detect and recognize products or barcodes, instantly update the shopping list on the tablet, and finally, the shopper checks the list and proceeds to checkout.

[0007] When smart shopping carts are actually deployed in supermarkets and shopping malls, there is a need to prevent abnormal shopping behaviors. Since consumers use them on their own, there will inevitably be some attempts to shop outside the normal process. Smart shopping carts need to detect, alert, or prevent these abnormal shopping behaviors.

[0008] In existing technologies, supermarkets typically use closed-circuit television (CCTV) cameras to monitor and prevent product loss. They also employ various sensors and alarm systems to enhance loss prevention capabilities. Some shopping carts are equipped with cameras that only monitor the area inside the cart. These methods often fail to comprehensively monitor customer shopping behavior, lack a holistic perspective, have blind spots, and are easily bypassed. Furthermore, these methods largely rely on manual judgment, which is inefficient and prone to errors. Due to a lack of intelligent analysis, existing systems cannot efficiently and accurately identify abnormal shopping behavior that leads to product loss. Summary of the Invention

[0009] This application provides a method for detecting abnormal shopping behavior in a shopping cart. The method is applied to an abnormal shopping behavior detection device to efficiently and accurately detect abnormal shopping behavior. The method includes:

[0010] Collect product scanning data from the barcode scanner, and collect motion data within the basket area from the motion monitoring device within the basket area;

[0011] When a shopping action is determined to have occurred based on the product scanning data, the first trigger shooting signal is issued;

[0012] When it is determined, based on motion data within the shopping basket area, that a shopping action has occurred within the shopping cart basket area, a second trigger shooting signal is issued;

[0013] When a shopping action is determined to have occurred based on a first trigger shooting signal and / or a second trigger shooting signal, a trigger shooting command is sent to the visual image data acquisition device; the visual image data acquisition device is used to acquire video frame image data of the shopper during the shopping process when it receives the trigger shooting command, and send the video frame image data to the abnormal shopping behavior detection device.

[0014] Based on video frame image data during the shopper's shopping process, determine the type of shopper's shopping action and the number of items in motion;

[0015] Based on data from product barcode scanners, the types of shopping actions performed by shoppers, and the number of items in motion, the system detects whether shoppers are engaging in abnormal shopping behavior.

[0016] This application provides a method for detecting abnormal shopping behavior in a shopping cart. This method is applied to a system to efficiently and accurately detect abnormal shopping behavior. The method includes:

[0017] The barcode scanner acquires product barcode data and sends the product barcode data to the abnormal shopping behavior detection device;

[0018] The motion monitoring device within the bicycle basket area monitors motion data within the bicycle basket area and sends the motion data within the bicycle basket area to the abnormal shopping behavior detection device;

[0019] When the abnormal shopping behavior detection device determines that a shopping action has occurred based on the product scanning data, it issues a first trigger shooting signal; when it determines that a movement has occurred in the shopping cart basket area based on the movement data in the basket area, it issues a second trigger shooting signal; when it determines that a shopping action has occurred based on the first trigger shooting signal and / or the second trigger shooting signal, it sends a trigger shooting command to the visual image data acquisition device.

[0020] When the visual image data acquisition device receives the trigger shooting command, it acquires video frame image data of the shopper during the shopping process and sends the video frame image data to the abnormal shopping behavior detection device.

[0021] The abnormal shopping behavior detection device determines the type of shopping action and the number of items in motion by using video frame image data during the shopping process; based on the data from the product barcode scanner, the type of shopping action, and the number of items in motion, it detects whether the shopper is engaging in abnormal shopping behavior.

[0022] This application provides a device for detecting abnormal shopping behavior in a shopping cart, used to efficiently and accurately detect abnormal shopping behavior. The device includes:

[0023] The data acquisition unit is used to collect product scanning data obtained by the barcode scanner, as well as motion data within the basket area monitored by the motion monitoring device within the basket area.

[0024] The first trigger shooting unit is used to issue a first trigger shooting signal when it is determined that a shopping action has occurred based on the product scanning data;

[0025] The second trigger shooting unit is used to issue a second trigger shooting signal when it is determined from the motion data within the shopping basket area that a shopping action has occurred within the shopping basket area of ​​the shopping cart.

[0026] The trigger shooting control unit is used to send a trigger shooting command to the visual image data acquisition device when a shopping action is determined to have occurred based on the first trigger shooting signal and / or the second trigger shooting signal; the visual image data acquisition device is used to collect video frame image data of the shopper during the shopping process when it receives the trigger shooting command, and send the video frame image data to the abnormal shopping behavior detection device.

[0027] The analysis unit is used to determine the type of shopping action of the shopper and the number of items in motion based on video frame image data during the shopping process;

[0028] The detection unit is used to detect whether a shopper is engaging in abnormal shopping behavior based on data from the product scanner, the type of shopping action of the shopper, and the number of products in motion.

[0029] This application provides a system for detecting abnormal shopping behavior in a shopping cart, used to efficiently and accurately detect abnormal shopping behavior. The system includes:

[0030] A barcode scanner is used to acquire product barcode data and send the product barcode data to an abnormal shopping behavior detection device.

[0031] The motion monitoring device within the bicycle basket area is used to monitor motion data within the bicycle basket area and send the motion data within the bicycle basket area to the abnormal shopping behavior detection device.

[0032] The visual image data acquisition device is used to acquire video frame image data of shoppers during the shopping process when a shooting command is received, and send the video frame image data to the abnormal shopping behavior detection device.

[0033] An abnormal shopping behavior detection device is used to issue a first trigger shooting signal when a shopping action is determined to have occurred based on product barcode scanning data; issue a second trigger shooting signal when a movement is determined to have occurred within the shopping cart basket area based on movement data within the basket area; send a trigger shooting command to a visual image data acquisition device when a shopping action is determined to have occurred based on the first and / or second trigger shooting signals; determine the type of shopping action and the number of moving products based on video frame image data during the shopper's shopping process; and detect whether the shopper is engaging in abnormal shopping behavior based on product barcode scanner data, the type of shopping action, and the number of moving products.

[0034] This application provides a shopping cart for detecting abnormal shopping behavior, used to efficiently and accurately detect abnormal shopping behavior. The shopping cart includes:

[0035] The abnormal shopping behavior detection device for shopping carts as described above.

[0036] This application also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the above-mentioned method for detecting abnormal shopping behavior in a shopping cart.

[0037] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for detecting abnormal shopping behavior in a shopping cart.

[0038] This application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the above-mentioned method for detecting abnormal shopping behavior in a shopping cart.

[0039] In this embodiment, the abnormal shopping behavior detection scheme for shopping carts involves: collecting product barcode scanning data from a barcode scanner and motion data within the basket area monitored by a motion monitoring device; issuing a first trigger shooting signal when a shopping action is determined based on the product barcode scanning data; issuing a second trigger shooting signal when a shopping action is determined based on the motion data within the basket area; and sending a trigger shooting command to a visual image data acquisition device when a shopping action is determined based on the first and / or second trigger shooting signals. The visual image data acquisition device, upon receiving the trigger shooting command, collects video frame image data of the shopper's shopping process and sends the video frame image data to the abnormal shopping behavior detection device. Based on the video frame image data of the shopper's shopping process, the type of the shopper's shopping action and the number of items in motion are determined. Based on the product barcode scanner data, the type of the shopper's shopping action, and the number of items in motion, the scheme detects whether the shopper is engaging in abnormal shopping behavior, thus efficiently and accurately detecting abnormal shopping behavior. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings:

[0041] Figure 1 is a flowchart illustrating the abnormal shopping behavior detection method for a shopping cart applied to a device in an embodiment of this application;

[0042] Figure 2 is a flowchart illustrating the abnormal shopping behavior detection method for the shopping cart applied to the system in this embodiment of the application;

[0043] Figure 3 is a schematic diagram of the abnormal shopping behavior detection device for shopping carts in an embodiment of this application;

[0044] Figure 4 is a schematic diagram of the abnormal shopping behavior detection system for shopping carts in an embodiment of this application;

[0045] Figure 5 is a schematic diagram of the circuit structure of the shopping cart in an embodiment of this application;

[0046] Figure 6 is a schematic diagram of the structure of a computer device according to an embodiment of this application. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the embodiments of this application will be further described in detail below with reference to the accompanying drawings. Here, the illustrative embodiments and descriptions of this application are used to explain this application, but are not intended to limit this application.

[0048] The acquisition, storage, use, and processing of data in this application comply with relevant laws and regulations.

[0049] This application provides a scheme for detecting abnormal shopping behavior in a shopping cart. This scheme is based on a wide-angle camera and video action recognition to prevent abnormal shopping behavior. This application proposes configuring a wide-angle vision camera and a computing and interactive device (the abnormal shopping behavior detection device in this application) on the shopping cart. The wide-angle camera collects video frame data of the shopper's shopping process, which is then processed by computer vision algorithms, including detecting target products and recognizing and judging the shopper's actions. Finally, it analyzes whether the shopper is engaging in abnormal shopping behavior. The abnormal shopping behavior detection scheme for this shopping cart is described in detail below.

[0050] Figure 1 is a flowchart illustrating the abnormal shopping behavior detection method for a shopping cart applied to a device in this embodiment of the present application. As shown in Figure 1, the method includes the following steps:

[0051] Step 101: Collect the product scanning data obtained by the barcode scanner, and the motion data within the basket area monitored by the motion monitoring device within the basket area;

[0052] Step 102: When a shopping action is determined to have occurred based on the product scanning data, the first trigger shooting signal is issued;

[0053] Step 103: When it is determined from the motion data within the shopping cart basket area that a shopping action has occurred within the shopping cart basket area, a second trigger shooting signal is issued;

[0054] Step 104: When it is determined that a shopping action has occurred based on the first trigger shooting signal and / or the second trigger shooting signal, a trigger shooting command is sent to the visual image data acquisition device; the visual image data acquisition device is used to acquire video frame image data of the shopper during the shopping process when it receives the trigger shooting command, and send the video frame image data to the abnormal shopping behavior detection device.

[0055] Step 105: Based on the video frame image data during the shopper's shopping process, determine the type of shopper's shopping action and the number of items in motion;

[0056] Step 106: Based on the data from the product barcode scanner, the shopper's shopping action type, and the number of products in motion, detect whether the shopper is engaging in abnormal shopping behavior.

[0057] The abnormal shopping behavior detection method for shopping carts provided in this application embodiment operates as follows: It collects product barcode scanning data from a barcode scanner and motion data within the shopping cart basket area monitored by a motion monitoring device within the basket area; when a shopping action is determined to have occurred based on the product barcode scanning data, a first trigger shooting signal is issued; when a shopping action is determined to have occurred within the shopping cart basket area based on the motion data within the basket area, a second trigger shooting signal is issued; when a shopping action is determined to have occurred based on the first and / or second trigger shooting signals, a trigger shooting command is sent to a visual image data acquisition device; the visual image data acquisition device, upon receiving the trigger shooting command, collects video frame image data of the shopper's shopping process and sends the video frame image data to the abnormal shopping behavior detection device; based on the video frame image data of the shopper's shopping process, it determines the type of the shopper's shopping action and the number of items in motion; based on the product barcode scanner data, the type of the shopper's shopping action, and the number of items in motion, it detects whether the shopper is engaging in abnormal shopping behavior for accurate settlement, thereby efficiently and accurately detecting abnormal shopping behavior. The method is described in detail below.

[0058] Based on the shopping cart, this application provides a method for detecting abnormal shopping behavior in a shopping cart, which integrates deep learning, machine learning, and computer vision algorithms to address abnormal shopping behaviors including but not limited to the following:

[0059] 1. Missed scan and addition: Shoppers add items to their shopping cart without scanning the barcode. This includes adding n items to the cart but having less than n barcodes. This includes various situations such as missing a single item, missing multiple items, scanning a single item and adding multiple items.

[0060] 2. Misplaced item: The shopper scans item A2, but puts a different item B2 into the cart.

[0061] The hardware devices involved in the embodiments of this application may include:

[0062] 1. Visual sensor (visual image data acquisition device as shown in Figure 4): at least one camera with a wide field of view. A wide field of view camera may include, but is not limited to, cameras with a large field of view, fisheye cameras, panoramic cameras, stereo cameras, and cameras mounted at a height. The camera's coverage area may include the area within the shopping cart basket and a certain distance outside the shopping cart basket. That is, in one embodiment, the visual image data acquisition device in this application embodiment can be a wide field of view camera. A wide field of view camera means: a camera with a field of view greater than a preset angle, such as a fisheye camera, panoramic camera, stereo camera, or a camera mounted at a height greater than a preset angle.

[0063] 2. A motion monitoring device within the shopping cart basket area. In this embodiment, the motion monitoring device within the shopping cart basket area may include: a ranging sensor (distance data acquisition device) and a depth data acquisition device (depth camera), that is, it can be configured as at least one depth camera or at least one ultrasonic sensor or millimeter-wave sensor. The sensing range of the motion monitoring device within the shopping cart basket area covers the entire shopping cart basket area. Two implementation methods of this motion monitoring device within the shopping cart basket area are described in the following embodiments.

[0064] 3. Computing and Interactive Device (abnormal shopping behavior detection device 04 as shown in Figure 4): This device connects to a visual sensor and receives image data from the visual sensor (visual image data acquisition device 03 as shown in Figure 4); it connects to a ranging sensor and receives distance data or connects to a depth data acquisition device and receives depth image data (action monitoring device 02 within the basket area as shown in Figure 4); it connects to a barcode scanner (barcode scanner 01 as shown in Figure 4) and receives the barcode scanning information from the scanner. Specifically, the computing and interactive device may include a processing unit for processing visual image data, distance data, or depth image data, performing algorithm calculations, etc. (the functions of the analysis unit 045 and detection unit 046 shown in Figure 3); it may also connect to an interactive display screen to display a shopping interaction interface.

[0065] 4. Barcode scanner: Used to scan the barcode on the product packaging and send the data to a computing device (abnormal shopping behavior detection device).

[0066] The methods involved in the embodiments of this application may include:

[0067] Using a wide-angle camera, video frame data of shoppers' shopping process is captured. This data is then processed by computer vision algorithms, including detecting target products and recognizing and judging shoppers' actions. Finally, it is determined whether the shopper has engaged in theft or damage during the shopping process; if so, a notification is issued. The general workflow is as follows:

[0068] 1. A ranging sensor continuously acquires ranging data within the basket area; or a depth camera (depth data acquisition device) acquires depth data (depth image data).

[0069] 2. The corresponding algorithm processing module (data acquisition device in Figure 4) in the device (abnormal shopping behavior detection device in Figure 4) continuously receives and processes ranging data or depth data, as well as barcode scanner data.

[0070] 3. Based on the data from the barcode scanner, determine if any new shopping activity has occurred; based on fluctuations in distance data or changes in depth image data, determine if any activity has occurred inside the basket.

[0071] 4. When an action occurs, the visual sensor (visual image data acquisition device) is triggered to start capturing video frame images.

[0072] 5. Stop the visual sensor from taking pictures when the distance data stops fluctuating or the depth data stops changing.

[0073] 6. The captured video segment data is sent to the corresponding algorithm module (such as the analysis unit and detection unit in Figure 3) for calculation and processing to confirm the action type and the quantity of goods, etc.

[0074] 7. By combining data from the barcode scanner, the type of shopping action, and the quantity of goods, determine whether there is any abnormal shopping behavior.

[0075] The detailed process of the method involved in this application embodiment is as follows:

[0076] 1. Receive data from the barcode scanner. If there is barcode scanning information, it is determined that a shopping action has occurred, and signal A (first trigger shooting signal) is sent.

[0077] 2. The ranging sensor is activated to acquire ranging data within the basket area, or the depth data acquisition device is activated to acquire depth image data.

[0078] When the device enters the shopping process, the distance sensor is activated to continuously measure the distance inside the basket. Here, we take the depth camera as an example. When the device enters the shopping process, the depth camera is activated to continuously acquire depth image data inside the basket.

[0079] As can be seen from the above, in one embodiment, the motion monitoring device in the shopping basket area is a depth data acquisition device, which is used to acquire depth image data in the shopping cart basket area and send the depth image data to the abnormal shopping behavior detection device.

[0080] When it is determined from motion data within the shopping cart basket area that a shopping action has occurred within the shopping cart basket area, a second trigger shooting signal is issued, including: when it is determined from depth image data that a shopping action has occurred within the shopping cart basket area, a second trigger shooting signal is issued.

[0081] In practice, depth image data of the shopping cart basket area acquired by the depth data acquisition device (depth camera) is used to determine whether a shopping action has occurred in the shopping cart basket area. If so, a second trigger shooting signal (hereinafter A1 signal) is issued.

[0082] That is, receiving depth images continuously acquired by a depth camera, assuming the depth image received at time t (the current time) is labeled as D. t Calculate D t With D t-1 The average depth difference between the two, where t-1 is the time before the current time.

[0083] If the average depth difference exceeds a preset average depth difference threshold, a screen disturbance is detected at time t, and the number of disturbed frames r is incremented by 1. When the value of r exceeds the preset number of disturbed frames threshold, it is determined that an action has occurred, and signal A1 (second trigger shooting signal) is issued. At this time, the number of still frames s is initialized to 0.

[0084] If the average depth difference remains at 0 or has a very small difference, then there is no image disturbance at time t, and the number of still frames s is incremented by 1. When the value of s equals the preset number of still frames threshold, the action is considered to have ended, and signal C (closing the shooting trigger signal) is issued.

[0085] As can be seen from the above, in one embodiment, when it is determined from the depth image data that a shopping action has occurred within the shopping cart basket area, a second trigger shooting signal is issued, including:

[0086] Obtain the depth image data at the current moment;

[0087] Determine the average depth difference between the current depth image data and the previous depth image data;

[0088] If the average depth difference exceeds the preset average depth difference threshold, it is determined that a screen disturbance has occurred at the current moment, and the number of disturbed frames is incremented by 1;

[0089] When the value of the disturbance frame count exceeds the preset disturbance frame count threshold, it is determined that a shopping action has occurred in the shopping cart basket area, and a second trigger shooting signal is issued.

[0090] In practice, the above-described method of determining whether shopping actions have occurred within the shopping cart basket area based on depth image data can improve the accuracy of detecting shopping actions and further improve the accuracy of subsequent abnormal behavior detection.

[0091] As can be seen from the above, in one embodiment, the abnormal shopping behavior detection method for shopping carts further includes:

[0092] Initialize the number of still frames to 0;

[0093] If the average depth difference is 0 or less than the preset depth value within the preset time period, it is determined that there is no screen disturbance at the current moment, and the number of still frames is incremented by 1.

[0094] When the value of the number of still frames equals the preset number of still frames threshold, the shopping action is determined to be over, and a signal to stop shooting is issued; the visual image data acquisition device is also used to stop shooting video frame image data when it receives the signal to stop shooting.

[0095] In practice, the above-mentioned method of stopping the recording of video frame image data can improve the accuracy of determining the end of the shopping action and further improve the accuracy of subsequent abnormal behavior detection.

[0096] Next, we will introduce a scheme for monitoring the movement within the basket area using a distance data acquisition device.

[0097] In one embodiment, the motion monitoring device within the shopping cart basket area is a distance data acquisition device, used to acquire the fluctuation of distance data within the shopping cart basket area and send the distance fluctuation data to the abnormal shopping behavior detection device.

[0098] When it is determined from the motion data within the shopping cart basket area that a shopping action has occurred within the shopping cart basket area, a second trigger shooting signal is issued, including: when it is determined from the fluctuation of distance data that a shopping action has occurred within the shopping cart basket area, a second trigger shooting signal is issued.

[0099] In practice, using a distance data acquisition device to monitor actions within the basket area can further improve the accuracy of monitoring shopping actions and further improve the accuracy of subsequent abnormal behavior detection.

[0100] In one embodiment, the above-mentioned method for detecting abnormal shopping behavior in a shopping cart further includes:

[0101] When the distance data stops fluctuating, a signal to stop shooting is issued; the visual image data acquisition device is also used to stop capturing video frame image data when it receives the signal to stop shooting.

[0102] In practice, the above-mentioned implementation method of issuing a shutdown trigger signal when the distance data no longer fluctuates can further improve the accuracy of monitoring shopping actions and further improve the accuracy of subsequent abnormal behavior detection.

[0103] 3. Wake up the vision sensor (visual image data acquisition device) to acquire video frame images.

[0104] When signal A (first trigger shooting signal) or signal A1 (second trigger shooting signal) is issued, the vision sensor is activated to start taking pictures and acquiring video frame image data. As explained above, signal A indicates that the barcode scanner has scanned the code, indicating that the shopper is engaged in shopping. Signal A1 indicates that the depth camera sensor has detected movement within the boundaries of the shopping cart. The difference between the two trigger signals is that some shoppers may want to put items into the shopping cart without scanning the code, in which case only signal A1 is received. Therefore, in order to accurately and flexibly activate the vision sensor to start taking pictures and acquire video frame image data, the occurrence of shopping action is determined based on the first trigger shooting signal and / or the second trigger shooting signal. If the vision sensor is in the process of taking pictures and receives signal A or signal A1 again, the vision sensor will not be activated again. That is, in one embodiment, the above-mentioned abnormal shopping behavior detection method for shopping carts further includes: when the vision image data acquisition device is detected to be taking video frame image data, if the first trigger shooting signal and / or the second trigger shooting signal are received again, the trigger shooting command will not be sent to the vision image data acquisition device again. This can save scheduling resources and improve the accuracy of abnormal behavior detection.

[0105] When taking photos stops, the acquired video segment data is sent to the algorithm processing module (abnormal shopping behavior detection device).

[0106] 4. The algorithm processing module performs calculations and processing on the video segment image data, including:

[0107] 1) Inspect the target product

[0108] The video frame image data obtained in step 3 is used to extract each frame image and feed it into the object detection algorithm model to obtain the object detection bounding box data B for all products in each frame image. The object detection bounding box data of all frames are then arranged in the temporal order of the video frames to form the product bounding box temporal data B. T .

[0109] Where N is the total number of goods, i is the current number of goods, and x is the total number of goods. i ,y i ,w i ,h i These are the coordinates of the four sides of the bounding box, T is the total duration of the video frame, and t is the time of the video frame.

[0110] 2) Action recognition

[0111] The video segment image data from step 3 is fused with the product bounding box temporal data from the previous step and then fed into the action recognition algorithm model. The corresponding action type and the number of products in motion are output. Action types include: placing a product into the vehicle after scanning the barcode, placing a different product into the vehicle after scanning the barcode, placing a product into the vehicle without scanning the barcode, and taking a product out of the vehicle.

[0112] Where action_cls is the action type, num is the number of items in motion, t is the time of the video frame, T is the total duration of the video frames, and I... T For video segment image data, B T Product bounding box time series data B T x i ,y i ,w i ,h i These are the coordinate data of the four sides of the bounding box. The action recognition algorithm model used in this application embodiment is based on video action recognition technology. This application embodiment innovatively designs a new method, namely, fusing image data with product bounding box temporal data and feeding it into a temporal transformer model (this model can determine the shopper's shopping action type and the number of moving products based on video frame image data during the shopper's shopping process). As can be seen from the above, in one embodiment, determining the shopper's shopping action type and the number of moving products based on video frame image data during the shopper's shopping process can include:

[0113] Extract each frame image from the video frame image data;

[0114] Each frame of image is input into a pre-trained object detection algorithm model to obtain the bounding box data of all items in each frame of image; the object detection algorithm model is pre-trained using a sample dataset of the relationship between the frame images of the items and the corresponding bounding boxes of the objects.

[0115] The bounding box data of the target detection of the goods corresponding to all frame images are arranged in the temporal order of the video frames to form the product bounding box temporal data;

[0116] The video frame image data is fused with the product bounding box time series data to obtain the current fused data of the video frame image data and the product bounding box time series data;

[0117] The current fused data is input into the action recognition algorithm model to obtain the shopper's action type and the number of items in motion. The action recognition algorithm model is pre-trained and generated based on the relationship between historical fused data and the corresponding action type and the number of items in motion sample datasets.

[0118] In practice, the details of feeding the time-series transformer model can be as follows:

[0119] a) Perform fusion representation of input data: The input video segment is extracted and divided into a preset fixed number of frames M, and then each frame image is divided into a fixed number of non-overlapping image blocks. Each image block is fed into a linear network layer and converted into a feature vector f of dimension L. Then, the corresponding N product target bounding box data in each frame image are concatenated with the above feature vector (wherein, the intersection-union ratio (IU) of each bounding box with the image block is determined, and the product bounding box information is concatenated with its feature vector if the IU with the image block is larger. If no bounding box falls into a certain image block, a fixed dimension of 0 value is added for concatenation. That is, in one embodiment, the target detection bounding box data of all products in each frame image are concatenated with the feature vectors of all image blocks in each frame image to obtain the feature vector with target detection bounding box data in each frame image, including: determining the IU of each bounding box with each image block; concatenating the bounding box with the larger IU with the image block with the feature vector of that image block to obtain the feature vector with target detection bounding box data in each frame image). Then, the temporal location information Tpos (4-dimensional) and the spatial location information of the image patch (4-dimensional) are concatenated with the feature vector. Finally, the entire video segment is represented as a matrix of shape T×M×(L+4×N+4+4).

[0120] b) Constructing a Transformer encoder: Feature vectors containing target bounding box information, temporal location encoding, and spatial location encoding of image patches are passed to a multi-layer Transformer encoder. These Transformer encoders include multi-head self-attention layers and feedforward neural network layers. This encoder is used to capture the spatiotemporal dependencies in the video sequence. Its BoundingBox-Space-Time attention mechanism enables the model to automatically learn and assign weights to different temporal frames, image patches, and product bounding boxes, thereby focusing on contextual information related to product position changes and action recognition.

[0121] This includes the innovatively designed joint BoundingBox-Space-Time attention module in the embodiments of this application, whose design features are:

[0122] First, an attention score is calculated for the input: each element of the input feature sequence (containing target bounding box information, temporal location encoding, and spatial location encoding of image patches) is projected into three spaces: Query(Q), Key(K), and Value(V) (an algorithmic concept in Transformer). The dot product between the Query vector and all Key vectors is then calculated to obtain the attention score. Next, the attention score is normalized by dividing it by a scaling factor (an algorithmic concept in Transformer) and applying the Softmax function to convert it into a probability distribution between 0 and 1. Then, a weighted summation is performed on the input feature sequence, i.e., the Value vector is weighted and summed using the normalized attention score. For this process, a fixed number of attention modules are used for multiple parallel computations to form multiple different projection spaces. Finally, the outputs of each attention module are concatenated.

[0123] As described above, in one embodiment, the current feature vector containing target detection bounding box data, temporal data location information, and spatial location information of image patches in each frame is input into a pre-built Transformer encoder. The current video frame image data is represented as a matrix as the current fusion data, including:

[0124] The target detection bounding box data, temporal data location information, and spatial location information of image patches in each frame are mapped to the projection space of the Query vector, Key vector, and Value vector in the transformer algorithm.

[0125] Determine the dot product between the query vector and all key vectors to obtain the attention score;

[0126] Divide the attention score by a preset scaling factor, apply the Softmax function, and convert the quotient of the attention score divided by the preset scaling factor into a probability distribution between 0 and 1 to obtain the normalized attention score.

[0127] The normalized attention scores are used to perform a weighted summation of the Value vectors to obtain the current fused data.

[0128] The key to the combined BoundingBox-Space-Time attention module lies in integrating the spatial and temporal information of image patches and the information of product bounding boxes into a single sequence. During training, the attention mechanism can simultaneously capture the relationships between spatially adjacent image patches, temporally adjacent frames, and the bounding boxes of the included product targets.

[0129] c) Building the classifier (action recognition algorithm model): Obtain the output of the Transformer encoder and set it as the global representation of the video segment. Then, build a classifier containing a multilayer perceptron or a linear layer. This classifier can calculate the global representation of the video segment and output the probability distribution of the action category (action type) to which the representation belongs.

[0130] d) Based on the above method for fusing input data and building the model, the model can be trained using a pre-collected and labeled training set, so that the entire model can output the action type action_cls of the video segment and the number of items in motion num according to the definition of Formula 2.

[0131] Specifically, the shopping process will be verified.

[0132] Based on the data obtained from the action recognition algorithm model in the previous step, we can obtain the different action types (action_cls) and the number of items in motion (num). The action types include: placing an item into the cart after scanning the barcode, placing an item into the cart after scanning the barcode and then replacing it with another item, placing an item into the cart without scanning the barcode, and retrieving an item from the cart. Further combining this with the data from the barcode scanner, we can identify several behaviors: normal placement, missed scanning, incorrect scanning, etc. In one embodiment, the shopping action types include one or any combination of the following: placing an item into the shopping cart after scanning the barcode, placing an item into the shopping cart after scanning the barcode and then replacing it with another item, placing an item into the shopping cart without scanning the barcode, and retrieving an item from the shopping cart. Specifically:

[0133] d1) If action_cls indicates that the product was placed in the vehicle without scanning the barcode, then the check will determine it as "missed scanning behavior";

[0134] d2) If action_cls is to scan the code and put it in the car, then check whether the number of codes n sent by the scanner is consistent with the num output by the model. If num is greater than n, then check and judge it as "missed scan behavior". If n is equal to num, then check and judge it as "normal behavior".

[0135] d3) If action_cls means scanning the barcode and then placing the item in the car, then the check and judgment should be "incorrectly scanned and placed";

[0136] d4) If action_cls is to take out goods from the vehicle, then check and judge it as "normally take out num goods" and report it.

[0137] As described above, in one embodiment, detecting whether a shopper is engaging in abnormal shopping behavior based on product barcode scanner data, the shopper's shopping action type, and the number of products in motion includes:

[0138] If the shopping action type is "adding items to the shopping cart without scanning the barcode", it is confirmed that the shopper has missed scanning the barcode.

[0139] If the shopping action type is scanning the barcode and then adding the item to the shopping cart, check whether the number of items in the barcode scanner data matches the number of items in the moving state; if the number of items in the moving state is greater than the number of items in the barcode scanner data, it is confirmed that the shopper has missed scanning.

[0140] If the shopping action type is "scan the code and then add the item to the shopping cart", it is confirmed that the shopper has accidentally scanned and added the wrong item.

[0141] In another embodiment, if it is determined that a shopper has engaged in abnormal shopping behavior, a reminder message can be displayed on the interactive display screen to remind the user that there is currently no barcode scanning or a barcode scanning error, and to assist the shopper in correcting this abnormal shopping behavior. For example, abnormal shopping behavior information (which may include: shopping cart icon, shopping cart location, and abnormal shopping behavior) can also be sent to the administrator's user terminal to remind the administrator to assist the shopper in correcting this abnormal shopping behavior.

[0142] In summary, the advantages of the abnormal shopping behavior detection method for shopping carts provided in this application include:

[0143] 1) By using a ranging sensor to determine the occurrence of an action, and then triggering the vision sensor to work when the action is confirmed, the effective video data of the shopping process is obtained, avoiding the vision sensor from constantly taking pictures and reducing power consumption.

[0144] 2) By mounting a wide-angle camera on the mobile shopping cart, video frames are captured from a wide angle, enabling comprehensive monitoring of the shopper's entire shopping process and the status of the goods during that process, overcoming the limitations of traditional loss prevention methods. The wide-angle camera was chosen because it allows for monitoring of the shopper's entire shopping process from a wide angle, which enables subsequent action and behavior recognition.

[0145] 3) A novel video action recognition algorithm was designed, which integrates image data and product bounding box time-series data and feeds them into a time-series transformer model containing a “BoundingBox-Space-Time” attention module.

[0146] 4) Based on computer vision technology, using deep learning, machine learning and artificial intelligence algorithms, a complete process and logical judgment are designed to analyze images captured by the camera and items in the shopping cart in real time, and to analyze abnormal behavior of shoppers.

[0147] This application also provides a method for detecting abnormal shopping behavior applied to a system, as described in the following embodiments. Since the principle behind this method is similar to that of the method for detecting abnormal shopping behavior applied to a device, the implementation of this method can be found in the implementation of the method for detecting abnormal shopping behavior applied to a device; repeated details will not be elaborated further.

[0148] Figure 2 is a flowchart illustrating the abnormal shopping behavior detection method for the shopping cart applied to the system in this embodiment of the application. As shown in Figure 2, the method includes the following steps:

[0149] Step 201: The barcode scanner acquires the product barcode data and sends the product barcode data to the abnormal shopping behavior detection device;

[0150] Step 202: The motion monitoring device in the basket area monitors motion data within the basket area and sends the motion data within the basket area to the abnormal shopping behavior detection device.

[0151] Step 203: When the abnormal shopping behavior detection device determines that a shopping action has occurred based on the product scanning data, it issues a first trigger shooting signal; when it determines that an action has occurred in the shopping cart basket area based on the action data in the basket area, it issues a second trigger shooting signal; when it determines that a shopping action has occurred based on the first trigger shooting signal and / or the second trigger shooting signal, it sends a trigger shooting command to the visual image data acquisition device.

[0152] Step 204: When the visual image data acquisition device receives the trigger shooting command, it acquires video frame image data of the shopper during the shopping process and sends the video frame image data to the abnormal shopping behavior detection device.

[0153] Step 205: The abnormal shopping behavior detection device determines the type of shopping action and the number of moving items based on the video frame image data of the shopper during the shopping process; based on the product barcode scanner data, the type of shopping action and the number of moving items, it detects whether the shopper is engaging in abnormal shopping behavior.

[0154] In one embodiment, determining the type of shopping action and the number of items in motion based on video frame image data during the shopper's shopping process includes:

[0155] Extract each frame image from the video frame image data;

[0156] Each frame of image is input into a pre-trained object detection algorithm model to obtain the bounding box data of all items in each frame of image; the object detection algorithm model is pre-trained using a sample dataset of the relationship between the frame images of the items and the corresponding bounding boxes of the objects.

[0157] The bounding box data of the target detection of the goods corresponding to all frame images are arranged in the temporal order of the video frames to form the product bounding box temporal data;

[0158] The video frame image data is fused with the product bounding box time series data to obtain the current fused data of the video frame image data and the product bounding box time series data;

[0159] The current fused data is input into the action recognition algorithm model to obtain the shopper's action type and the number of items in motion. The action recognition algorithm model is pre-trained and generated based on the relationship between historical fused data and the corresponding action type and the number of items in motion sample datasets.

[0160] In one embodiment, video frame image data and product bounding box time-series data are fused to obtain current fused data of video frame image data and product bounding box time-series data, including:

[0161] The current video frame image data is extracted and divided into a preset number of frame images;

[0162] Each frame of the image is divided into a fixed number of non-overlapping image blocks;

[0163] Each image patch is input into a linear network layer, and each image patch is converted into a feature vector of a preset dimension, thus obtaining the feature vectors of all image patches in each frame.

[0164] The bounding box data of all the objects detected in each frame of the image are concatenated with the feature vectors of all the image blocks in each frame of the image to obtain the feature vector of each frame of the image containing the bounding box data of the objects detected in the image.

[0165] The temporal data location information and the spatial location information of the image patch in each frame are concatenated with the feature vector containing the target detection bounding box data in each frame to obtain the current feature vector of each frame containing the target detection bounding box data, temporal data location information, and spatial location information of the image patch.

[0166] The current feature vector containing object detection bounding box data, temporal data location information, and spatial location information of image patches in each frame image is input into a pre-built Transformer encoder, which represents the current video frame image data as a matrix as the current fused data. The encoder includes a joint BoundingBox-Space-Time attention module, which is pre-trained and generated based on a sample dataset of the relationship between historical feature vectors containing object detection bounding box data, temporal data location information, and spatial location information of image patches and historical fused data.

[0167] In one embodiment, the current feature vector containing object detection bounding box data, temporal data location information, and spatial location information of image patches in each frame is input into a pre-built Transformer encoder. The current video frame image data is represented as a matrix as the current fusion data, including:

[0168] The target detection bounding box data, temporal data location information, and spatial location information of image patches in each frame are mapped to the projection space of the Query vector, Key vector, and Value vector in the transformer algorithm.

[0169] Determine the dot product between the query vector and all key vectors to obtain the attention score;

[0170] Divide the attention score by a preset scaling factor, apply the Softmax function, and convert the quotient of the attention score divided by the preset scaling factor into a probability distribution between 0 and 1 to obtain the normalized attention score.

[0171] The normalized attention scores are used to perform a weighted summation of the Value vectors to obtain the current fused data.

[0172] In one embodiment, the bounding box data of all items in each frame of an image are concatenated with the feature vectors of all image patches in each frame of the image to obtain a feature vector in each frame of the image containing the bounding box data of the items, including:

[0173] Determine the intersection-union ratio (IUU) of each bounding box with each image patch;

[0174] The bounding box with the larger intersection-union ratio of the image patch is concatenated with the feature vector of that image patch to obtain the feature vector containing the target detection bounding box data in each frame of the image.

[0175] In one embodiment, the motion monitoring device within the shopping basket area is a depth data acquisition device, used to acquire depth image data within the shopping cart basket area and send the depth image data to the abnormal shopping behavior detection device.

[0176] When it is determined from motion data within the shopping cart basket area that a shopping action has occurred within the shopping cart basket area, a second trigger shooting signal is issued, including: when it is determined from depth image data that a shopping action has occurred within the shopping cart basket area, a second trigger shooting signal is issued.

[0177] In one embodiment, when it is determined from depth image data that a shopping action has occurred within the shopping cart basket area, a second trigger shooting signal is issued, including:

[0178] Obtain the depth image data at the current moment;

[0179] Determine the average depth difference between the current depth image data and the previous depth image data;

[0180] If the average depth difference exceeds the preset average depth difference threshold, it is determined that a screen disturbance has occurred at the current moment, and the number of disturbed frames is incremented by 1;

[0181] When the value of the disturbance frame count exceeds the preset disturbance frame count threshold, it is determined that a shopping action has occurred in the shopping cart basket area, and a second trigger shooting signal is issued.

[0182] In one embodiment, the above-mentioned method for detecting abnormal shopping behavior in a shopping cart further includes:

[0183] Initialize the number of still frames to 0;

[0184] If the average depth difference is 0 or less than the preset depth value within the preset time period, it is determined that there is no screen disturbance at the current moment, and the number of still frames is incremented by 1.

[0185] When the value of the number of still frames equals the preset number of still frames threshold, the shopping action is determined to be over, and a signal to stop shooting is issued; the visual image data acquisition device is also used to stop shooting video frame image data when it receives the signal to stop shooting.

[0186] In one embodiment, the motion monitoring device within the shopping cart basket area is a distance data acquisition device, used to acquire the fluctuation of distance data within the shopping cart basket area and send the distance fluctuation data to the abnormal shopping behavior detection device.

[0187] When it is determined from the motion data within the shopping cart basket area that a shopping action has occurred within the shopping cart basket area, a second trigger shooting signal is issued, including: when it is determined from the fluctuation of distance data that a shopping action has occurred within the shopping cart basket area, a second trigger shooting signal is issued.

[0188] In one embodiment, the above-mentioned method for detecting abnormal shopping behavior in a shopping cart further includes:

[0189] When the distance data stops fluctuating, a signal to stop shooting is issued; the visual image data acquisition device is also used to stop capturing video frame image data when it receives the signal to stop shooting.

[0190] In one embodiment, the above-mentioned abnormal shopping behavior detection method for shopping carts further includes: when the visual image data acquisition device is detecting that it is capturing video frame image data, and upon receiving the first trigger shooting signal and / or the second trigger shooting signal again, the trigger shooting command is not sent to the visual image data acquisition device again.

[0191] In one embodiment, the shopping action type includes one or any combination of the following: adding goods to the shopping cart after scanning the barcode, adding goods to the shopping cart after scanning the barcode, adding goods to the shopping cart without scanning the barcode, and taking goods out of the shopping cart.

[0192] In one embodiment, detecting abnormal shopping behavior by a shopper based on barcode scanner data, the shopper's shopping action type, and the number of items in motion includes:

[0193] If the shopping action type is "adding items to the shopping cart without scanning the barcode", it is confirmed that the shopper has missed scanning the barcode.

[0194] If the shopping action type is scanning the barcode and then adding the item to the shopping cart, check whether the number of items in the barcode scanner data matches the number of items in the moving state; if the number of items in the moving state is greater than the number of items in the barcode scanner data, it is confirmed that the shopper has missed scanning.

[0195] If the shopping action type is "scan the code and then add the item to the shopping cart", it is confirmed that the shopper has accidentally scanned and added the wrong item.

[0196] This application also provides a device for detecting abnormal shopping behavior in a shopping cart, as described in the following embodiments. Since the principle behind this device is similar to the method for detecting abnormal shopping behavior applied to a device, the implementation of this device can refer to the implementation of the method for detecting abnormal shopping behavior applied to a device; repeated details will not be elaborated further.

[0197] Figure 3 is a schematic diagram of the abnormal shopping behavior detection device for shopping carts in an embodiment of this application. As shown in Figure 3, the device includes:

[0198] The data acquisition unit 041 is used to acquire the product scanning data obtained by the barcode scanner, and the motion data within the basket area monitored by the motion monitoring device within the basket area.

[0199] The first trigger shooting unit 042 is used to send out a first trigger shooting signal when it is determined that a shopping action has occurred based on the product scanning data;

[0200] The second trigger shooting unit 043 is used to issue a second trigger shooting signal when it is determined from the motion data within the shopping basket area that a shopping action has occurred within the shopping basket area of ​​the shopping cart.

[0201] The trigger shooting control unit 044 is used to send a trigger shooting command to the visual image data acquisition device when it is determined that a shopping action has occurred based on the first trigger shooting signal and / or the second trigger shooting signal; the visual image data acquisition device is used to collect video frame image data of the shopper during the shopping process when it receives the trigger shooting command, and send the video frame image data to the abnormal shopping behavior detection device.

[0202] Analysis unit 045 is used to determine the type of shopping action of the shopper and the number of items in motion based on video frame image data during the shopper's shopping process;

[0203] The detection unit 046 is used to detect whether a shopper is engaging in abnormal shopping behavior based on data from the product barcode scanner, the type of shopping action of the shopper, and the number of products in motion.

[0204] In one embodiment, the analysis unit is specifically used for:

[0205] Extract each frame image from the video frame image data;

[0206] Each frame of image is input into a pre-trained object detection algorithm model to obtain the bounding box data of all items in each frame of image; the object detection algorithm model is pre-trained using a sample dataset of the relationship between the frame images of the items and the corresponding bounding boxes of the objects.

[0207] The bounding box data of the target detection of the goods corresponding to all frame images are arranged in the temporal order of the video frames to form the product bounding box temporal data;

[0208] The video frame image data is fused with the product bounding box time series data to obtain the current fused data of the video frame image data and the product bounding box time series data;

[0209] The current fused data is input into the action recognition algorithm model to obtain the shopper's action type and the number of items in motion. The action recognition algorithm model is pre-trained and generated based on the relationship between historical fused data and the corresponding action type and the number of items in motion sample datasets.

[0210] In one embodiment, video frame image data and product bounding box time-series data are fused to obtain current fused data of video frame image data and product bounding box time-series data, including:

[0211] The current video frame image data is extracted and divided into a preset number of frame images;

[0212] Each frame of the image is divided into a fixed number of non-overlapping image blocks;

[0213] Each image patch is input into a linear network layer, and each image patch is converted into a feature vector of a preset dimension, thus obtaining the feature vectors of all image patches in each frame.

[0214] The bounding box data of all the objects detected in each frame of the image are concatenated with the feature vectors of all the image blocks in each frame of the image to obtain the feature vector of each frame of the image containing the bounding box data of the objects detected in the image.

[0215] The temporal data location information and the spatial location information of the image patch in each frame are concatenated with the feature vector containing the target detection bounding box data in each frame to obtain the current feature vector of each frame containing the target detection bounding box data, temporal data location information, and spatial location information of the image patch.

[0216] The current feature vector containing object detection bounding box data, temporal data location information, and spatial location information of image patches in each frame image is input into a pre-built Transformer encoder, which represents the current video frame image data as a matrix as the current fused data. The encoder includes a joint BoundingBox-Space-Time attention module, which is pre-trained and generated based on a sample dataset of the relationship between historical feature vectors containing object detection bounding box data, temporal data location information, and spatial location information of image patches and historical fused data.

[0217] In one embodiment, the current feature vector containing object detection bounding box data, temporal data location information, and spatial location information of image patches in each frame is input into a pre-built Transformer encoder. The current video frame image data is represented as a matrix as the current fusion data, including:

[0218] The target detection bounding box data, temporal data location information, and spatial location information of image patches in each frame are mapped to the projection space of the Query vector, Key vector, and Value vector in the transformer algorithm.

[0219] Determine the dot product between the query vector and all key vectors to obtain the attention score;

[0220] Divide the attention score by a preset scaling factor, apply the Softmax function, and convert the quotient of the attention score divided by the preset scaling factor into a probability distribution between 0 and 1 to obtain the normalized attention score.

[0221] The normalized attention scores are used to perform a weighted summation of the Value vectors to obtain the current fused data.

[0222] In one embodiment, the bounding box data of all items in each frame of an image are concatenated with the feature vectors of all image patches in each frame of the image to obtain a feature vector in each frame of the image containing the bounding box data of the items, including:

[0223] Determine the intersection-union ratio (IUU) of each bounding box with each image patch;

[0224] The bounding box with the larger intersection-union ratio of the image patch is concatenated with the feature vector of that image patch to obtain the feature vector containing the target detection bounding box data in each frame of the image.

[0225] In one embodiment, the motion monitoring device within the shopping basket area is a depth data acquisition device, used to acquire depth image data within the shopping cart basket area and send the depth image data to the abnormal shopping behavior detection device.

[0226] The second trigger shooting unit is specifically used to: issue a second trigger shooting signal when it is determined from the depth image data that a shopping action has occurred within the shopping cart basket area.

[0227] In one embodiment, the second triggering unit is specifically used for:

[0228] Obtain the depth image data at the current moment;

[0229] Determine the average depth difference between the current depth image data and the previous depth image data;

[0230] If the average depth difference exceeds the preset average depth difference threshold, it is determined that a screen disturbance has occurred at the current moment, and the number of disturbed frames is incremented by 1;

[0231] When the value of the disturbance frame count exceeds the preset disturbance frame count threshold, it is determined that a shopping action has occurred in the shopping cart basket area, and a second trigger shooting signal is issued.

[0232] In one embodiment, the above-mentioned abnormal shopping behavior detection device for shopping carts further includes:

[0233] The still frame initialization unit is used to initialize the number of still frames to 0.

[0234] The still frame processing unit is used to determine that there is no screen disturbance at the current moment if the average depth difference is 0 or less than the preset depth value within a preset time period, and then increment the still frame count by 1.

[0235] The first shutter-off trigger unit is used to determine the end of the shopping action and send a shutter-off trigger signal when the value of the number of still frames is equal to a preset number of still frames threshold; the visual image data acquisition device is also used to stop capturing video frame image data when it receives the shutter-off trigger signal.

[0236] In one embodiment, the motion monitoring device within the shopping cart basket area is a distance data acquisition device, used to acquire the fluctuation of distance data within the shopping cart basket area and send the distance fluctuation data to the abnormal shopping behavior detection device.

[0237] The second trigger shooting unit is also specifically used to: issue a second trigger shooting signal when it is determined, based on the fluctuation of distance data, that a shopping action has occurred within the shopping cart basket area.

[0238] In one embodiment, the above-mentioned abnormal shopping behavior detection device for shopping carts further includes:

[0239] The second shutter-off trigger unit is used to issue a shutter-off trigger signal when the distance data stops fluctuating; the visual image data acquisition device is also used to stop capturing video frame image data when it receives the shutter-off trigger signal.

[0240] In one embodiment, the abnormal shopping behavior detection device for the shopping cart further includes a trigger processing unit, configured to: when the visual image data acquisition device is detecting that it is capturing video frame image data, and upon receiving the first trigger shooting signal and / or the second trigger shooting signal again, not to repeatedly send the trigger shooting command to the visual image data acquisition device.

[0241] In one embodiment, the shopping action type includes one or any combination of the following: adding goods to the shopping cart after scanning the barcode, adding goods to the shopping cart after scanning the barcode, adding goods to the shopping cart without scanning the barcode, and taking goods out of the shopping cart.

[0242] In one embodiment, the detection unit is specifically used for:

[0243] If the shopping action type is "adding items to the shopping cart without scanning the barcode", it is confirmed that the shopper has missed scanning the barcode.

[0244] If the shopping action type is scanning the barcode and then adding the item to the shopping cart, check whether the number of items in the barcode scanner data matches the number of items in the moving state; if the number of items in the moving state is greater than the number of items in the barcode scanner data, it is confirmed that the shopper has missed scanning.

[0245] If the shopping action type is "scan the code and then add the item to the shopping cart", it is confirmed that the shopper has accidentally scanned and added the wrong item.

[0246] This application also provides an abnormal shopping behavior detection system, as described in the following embodiments. Since the principle behind this system's problem-solving is similar to that of the abnormal shopping behavior detection method applied to a device, the implementation of this system can refer to the implementation of the abnormal shopping behavior detection method applied to a device; repeated details will not be elaborated further.

[0247] Figure 4 is a schematic diagram of the abnormal shopping behavior detection system for shopping carts in this embodiment of the present application. As shown in Figure 4, the system includes:

[0248] The barcode scanner 01 is used to acquire product barcode scanning data and send the product barcode scanning data to the abnormal shopping behavior detection device.

[0249] The motion monitoring device 02 within the basket area is used to monitor motion data within the basket area and send the motion data within the basket area to the abnormal shopping behavior detection device.

[0250] The visual image data acquisition device 03 is used to acquire video frame image data of the shopper during the shopping process when a shooting command is received, and send the video frame image data to the abnormal shopping behavior detection device.

[0251] The abnormal shopping behavior detection device 04 is used to issue a first trigger shooting signal when a shopping action is determined to have occurred based on the product barcode scanning data; issue a second trigger shooting signal when a movement is determined to have occurred in the shopping cart basket area based on the movement data in the basket area; send a trigger shooting command to the visual image data acquisition device when a shopping action is determined to have occurred based on the first trigger shooting signal and / or the second trigger shooting signal; determine the type of shopping action of the shopper and the number of products in motion based on the video frame image data of the shopper during the shopping process; and detect whether the shopper has abnormal shopping behavior based on the product barcode scanner data, the type of shopping action of the shopper, and the number of products in motion.

[0252] In one embodiment, determining the type of shopping action and the number of items in motion based on video frame image data during the shopper's shopping process includes:

[0253] Extract each frame image from the video frame image data;

[0254] Each frame of image is input into a pre-trained object detection algorithm model to obtain the bounding box data of all items in each frame of image; the object detection algorithm model is pre-trained using a sample dataset of the relationship between the frame images of the items and the corresponding bounding boxes of the objects.

[0255] The bounding box data of the target detection of the goods corresponding to all frame images are arranged in the temporal order of the video frames to form the product bounding box temporal data;

[0256] The video frame image data is fused with the product bounding box time series data to obtain the current fused data of the video frame image data and the product bounding box time series data;

[0257] The current fused data is input into the action recognition algorithm model to obtain the shopper's action type and the number of items in motion. The action recognition algorithm model is pre-trained and generated based on the relationship between historical fused data and the corresponding action type and the number of items in motion sample datasets.

[0258] In one embodiment, video frame image data and product bounding box time-series data are fused to obtain current fused data of video frame image data and product bounding box time-series data, including:

[0259] The current video frame image data is extracted and divided into a preset number of frame images;

[0260] Each frame of the image is divided into a fixed number of non-overlapping image blocks;

[0261] Each image patch is input into a linear network layer, and each image patch is converted into a feature vector of a preset dimension, thus obtaining the feature vectors of all image patches in each frame.

[0262] The bounding box data of all the objects detected in each frame of the image are concatenated with the feature vectors of all the image blocks in each frame of the image to obtain the feature vector of each frame of the image containing the bounding box data of the objects detected in the image.

[0263] The temporal data location information and the spatial location information of the image patch in each frame are concatenated with the feature vector containing the target detection bounding box data in each frame to obtain the current feature vector of each frame containing the target detection bounding box data, temporal data location information, and spatial location information of the image patch.

[0264] The current feature vector containing object detection bounding box data, temporal data location information, and spatial location information of image patches in each frame image is input into a pre-built Transformer encoder, which represents the current video frame image data as a matrix as the current fused data. The encoder includes a joint BoundingBox-Space-Time attention module, which is pre-trained and generated based on a sample dataset of the relationship between historical feature vectors containing object detection bounding box data, temporal data location information, and spatial location information of image patches and historical fused data.

[0265] In one embodiment, the current feature vector containing object detection bounding box data, temporal data location information, and spatial location information of image patches in each frame is input into a pre-built Transformer encoder. The current video frame image data is represented as a matrix as the current fusion data, including:

[0266] The target detection bounding box data, temporal data location information, and spatial location information of image patches in each frame are mapped to the projection space of the Query vector, Key vector, and Value vector in the transformer algorithm.

[0267] Determine the dot product between the query vector and all key vectors to obtain the attention score;

[0268] Divide the attention score by a preset scaling factor, apply the Softmax function, and convert the quotient of the attention score divided by the preset scaling factor into a probability distribution between 0 and 1 to obtain the normalized attention score.

[0269] The normalized attention scores are used to perform a weighted summation of the Value vectors to obtain the current fused data.

[0270] In one embodiment, the bounding box data of all items in each frame of an image are concatenated with the feature vectors of all image patches in each frame of the image to obtain a feature vector in each frame of the image containing the bounding box data of the items, including:

[0271] Determine the intersection-union ratio (IUU) of each bounding box with each image patch;

[0272] The bounding box with the larger intersection-union ratio of the image patch is concatenated with the feature vector of that image patch to obtain the feature vector containing the target detection bounding box data in each frame of the image.

[0273] In one embodiment, the motion monitoring device within the shopping basket area is a depth data acquisition device, used to acquire depth image data within the shopping cart basket area and send the depth image data to the abnormal shopping behavior detection device.

[0274] When it is determined from motion data within the shopping cart basket area that a shopping action has occurred within the shopping cart basket area, a second trigger shooting signal is issued, including: when it is determined from depth image data that a shopping action has occurred within the shopping cart basket area, a second trigger shooting signal is issued.

[0275] In one embodiment, when it is determined from depth image data that a shopping action has occurred within the shopping cart basket area, a second trigger shooting signal is issued, including:

[0276] Obtain the depth image data at the current moment;

[0277] Determine the average depth difference between the current depth image data and the previous depth image data;

[0278] If the average depth difference exceeds the preset average depth difference threshold, it is determined that a screen disturbance has occurred at the current moment, and the number of disturbed frames is incremented by 1;

[0279] When the value of the disturbance frame count exceeds the preset disturbance frame count threshold, it is determined that a shopping action has occurred in the shopping cart basket area, and a second trigger shooting signal is issued.

[0280] In one embodiment, the above-mentioned abnormal shopping behavior detection device for shopping carts is further used for:

[0281] Initialize the number of still frames to 0;

[0282] If the average depth difference is 0 or less than the preset depth value within the preset time period, it is determined that there is no screen disturbance at the current moment, and the number of still frames is incremented by 1.

[0283] When the value of the number of still frames equals the preset number of still frames threshold, the shopping action is determined to be over, and a signal to stop shooting is issued; the visual image data acquisition device is also used to stop shooting video frame image data when it receives the signal to stop shooting.

[0284] In one embodiment, the motion monitoring device within the shopping cart basket area is a distance data acquisition device, used to acquire the fluctuation of distance data within the shopping cart basket area and send the distance fluctuation data to the abnormal shopping behavior detection device.

[0285] When it is determined from the motion data within the shopping cart basket area that a shopping action has occurred within the shopping cart basket area, a second trigger shooting signal is issued, including: when it is determined from the fluctuation of distance data that a shopping action has occurred within the shopping cart basket area, a second trigger shooting signal is issued.

[0286] In one embodiment, the above-mentioned abnormal shopping behavior detection device for shopping carts further includes:

[0287] When the distance data stops fluctuating, a signal to stop shooting is issued; the visual image data acquisition device is also used to stop capturing video frame image data when it receives the signal to stop shooting.

[0288] In one embodiment, the above-mentioned abnormal shopping behavior detection method for shopping carts further includes: when the visual image data acquisition device is detecting that it is capturing video frame image data, and upon receiving the first trigger shooting signal and / or the second trigger shooting signal again, the trigger shooting command is not sent to the visual image data acquisition device again.

[0289] In one embodiment, the shopping action type includes one or any combination of the following: adding goods to the shopping cart after scanning the barcode, adding goods to the shopping cart after scanning the barcode, adding goods to the shopping cart without scanning the barcode, and taking goods out of the shopping cart.

[0290] In one embodiment, detecting abnormal shopping behavior by a shopper based on barcode scanner data, the shopper's shopping action type, and the number of items in motion includes:

[0291] If the shopping action type is "adding items to the shopping cart without scanning the barcode", it is confirmed that the shopper has missed scanning the barcode.

[0292] If the shopping action type is scanning the barcode and then adding the item to the shopping cart, check whether the number of items in the barcode scanner data matches the number of items in the moving state; if the number of items in the moving state is greater than the number of items in the barcode scanner data, it is confirmed that the shopper has missed scanning.

[0293] If the shopping action type is "scan the code and then add the item to the shopping cart", it is confirmed that the shopper has accidentally scanned and added the wrong item.

[0294] This application also provides a shopping cart for detecting abnormal shopping behavior, as described in the following embodiments. Since the principle behind this system's problem-solving is similar to that of the abnormal shopping behavior detection method applied to a device, the implementation of this shopping cart can refer to the implementation of the abnormal shopping behavior detection method applied to a device; repeated details will not be elaborated further.

[0295] Figure 5 is a schematic diagram of the circuit structure of the shopping cart in this embodiment of the application. As shown in Figure 5, the shopping cart includes: the abnormal shopping behavior detection device 04 of the shopping cart as described above. The implementation of the abnormal shopping behavior detection device 04 is detailed in the above embodiment.

[0296] In summary, the abnormal shopping behavior detection scheme for shopping carts provided in this application embodiment achieves the following:

[0297] 1. By utilizing visual cameras and pure vision algorithms, without using additional expensive sensors, it can accurately detect whether a product is put into or taken out of the shopping cart, resulting in lower hardware costs and more flexible installation and deployment.

[0298] 2. Effectively prevents product loss. This not only improves the efficiency and accuracy of loss prevention but also reduces labor costs and error rates.

[0299] Based on the aforementioned inventive concept, as shown in FIG6, this application also proposes a computer device 500, including a memory 510, a processor 520, and a computer program 530 stored in the memory 510 and executable on the processor 520. When the processor 520 executes the computer program 530, it implements a method for detecting abnormal shopping behavior in a shopping cart.

[0300] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-described method for detecting abnormal shopping behavior in a shopping cart.

[0301] This application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the above-mentioned method for detecting abnormal shopping behavior in a shopping cart.

[0302] In this embodiment, the abnormal shopping behavior detection scheme for shopping carts involves: collecting product barcode scanning data from a barcode scanner and motion data within the basket area monitored by a motion monitoring device; issuing a first trigger shooting signal when a shopping action is determined to have occurred based on the product barcode scanning data; issuing a second trigger shooting signal when a shopping action is determined to have occurred within the shopping cart basket area based on the motion data within the basket area; and sending a trigger shooting command to a visual image data acquisition device when a shopping action is determined to have occurred based on the first and / or second trigger shooting signals. The visual image data acquisition device, upon receiving the trigger shooting command, collects video frame image data of the shopper's shopping process and sends the video frame image data to the abnormal shopping behavior detection device. Based on the video frame image data of the shopper's shopping process, it determines the type of the shopper's shopping action and the number of items in motion. Based on the product barcode scanner data, the type of the shopper's shopping action, and the number of items in motion, it detects whether the shopper is engaging in abnormal shopping behavior, thus efficiently and accurately detecting abnormal shopping behavior.

[0303] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0304] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more flowchart illustrations and / or one or more block diagrams.

[0305] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flowcharts and / or one or more block diagrams.

[0306] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more flowcharts and / or one or more block diagrams.

[0307] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of this application. It should be understood that the above descriptions are merely specific embodiments of this application and are not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this invention.

Claims

1. A method for detecting abnormal shopping behavior in a shopping cart, characterized in that, This method is applied to an abnormal shopping behavior detection device, wherein the abnormal shopping behavior detection method for the shopping cart includes: Collect product scanning data from the barcode scanner, and collect motion data within the basket area from the motion monitoring device within the basket area; When a shopping action is determined to have occurred based on the product scanning data, the first trigger shooting signal is issued; When it is determined, based on motion data within the shopping basket area, that a shopping action has occurred within the shopping cart basket area, a second trigger shooting signal is issued; When a shopping action is determined to have occurred based on a first trigger shooting signal and / or a second trigger shooting signal, a trigger shooting command is sent to the visual image data acquisition device; the visual image data acquisition device is used to acquire video frame image data of the shopper during the shopping process when it receives the trigger shooting command, and send the video frame image data to the abnormal shopping behavior detection device. Based on video frame data during the shopper's shopping process, determine the type of shopper's shopping actions and the number of items in motion; and Based on data from product barcode scanners, the types of shopping actions performed by shoppers, and the number of items in motion, the system detects whether shoppers are engaging in abnormal shopping behavior.

2. The method as described in claim 1, characterized in that, Based on video frame image data during the shopper's shopping process, determine the type of shopper's shopping actions and the number of items in motion, including: Extract each frame image from the video frame image data; Each frame of image is input into a pre-trained object detection algorithm model to obtain object detection bounding box data for all items in each frame of image; the object detection algorithm model is pre-trained using a sample dataset of the relationship between the frame images of the items and the corresponding object detection bounding boxes. The bounding box data of the target detection of the goods corresponding to all frame images are arranged in the temporal order of the video frames to form the product bounding box temporal data; The video frame image data is fused with the product bounding box time-series data to obtain the current fused data of the video frame image data and the product bounding box time-series data; and The current fused data is input into the action recognition algorithm model to obtain the shopper's action type and the number of items in motion; the action recognition algorithm model is pre-trained and generated based on a sample dataset that shows the relationship between historical fused data and the corresponding action type and the number of items in motion.

3. The method as described in claim 2, characterized in that, The video frame image data is fused with the product bounding box time-series data to obtain the current fused data of the video frame image data and the product bounding box time-series data, including: The current video frame image data is extracted and divided into a preset number of frame images; Each frame of the image is divided into a fixed number of non-overlapping image blocks; Each image patch is input into a linear network layer, and each image patch is converted into a feature vector of a preset dimension, thus obtaining the feature vectors of all image patches in each frame. The bounding box data of all the objects detected in each frame of the image are concatenated with the feature vectors of all the image blocks in each frame of the image to obtain the feature vector of each frame of the image containing the bounding box data of the objects detected in the image. The temporal data location information and spatial location information of image patches in each frame are concatenated with the feature vector containing target detection bounding box data in each frame to obtain the current feature vector of each frame containing target detection bounding box data, temporal data location information, and spatial location information of image patches; and The current feature vector containing object detection bounding box data, temporal data location information, and spatial location information of image patches in each frame image is input into a pre-built Transformer encoder, and the current video frame image data is represented as a matrix as the current fused data. The encoder includes a joint BoundingBox-Space-Time attention module, which is pre-trained and generated based on a sample dataset of the relationship between historical feature vectors containing object detection bounding box data, temporal data location information, and spatial location information of image patches and historical fused data.

4. The method as described in claim 3, characterized in that, The bounding box data of all items in each frame of the image are concatenated with the feature vectors of all image patches in each frame to obtain the feature vector of each frame containing the bounding box data of the items, including: Determine the intersection-union ratio (IoU) of each bounding box with each image patch; and The bounding box with the larger intersection-union ratio of the image patch is concatenated with the feature vector of that image patch to obtain the feature vector containing the target detection bounding box data in each frame of the image.

5. The method as described in claim 1, characterized in that, The motion monitoring device within the shopping cart basket area is a depth data acquisition device, used to acquire depth image data within the shopping cart basket area and send the depth image data to the abnormal shopping behavior detection device. When it is determined from motion data within the shopping cart basket area that a shopping action has occurred within the shopping cart basket area, a second trigger shooting signal is issued, including: when it is determined from depth image data that a shopping action has occurred within the shopping cart basket area, a second trigger shooting signal is issued.

6. The method as described in claim 5, characterized in that, When it is determined from depth image data that a shopping action has occurred within the shopping cart basket area, a second trigger signal is issued, including: Obtain the depth image data at the current moment; Determine the average depth difference between the current depth image data and the previous depth image data; If the average depth difference exceeds a preset average depth difference threshold, it is determined that a screen disturbance has occurred at the current moment, and the disturbance frame count is incremented by 1; and When the value of the disturbance frame count exceeds the preset disturbance frame count threshold, it is determined that a shopping action has occurred in the shopping cart basket area, and a second trigger shooting signal is issued.

7. The method as described in claim 5, characterized in that, Also includes: Initialize the number of still frames to 0; If the average depth difference is 0 or less than the preset depth value within the preset time period, it is determined that there is no screen disturbance at the current moment, and the number of still frames is incremented by 1. as well as When the value of the number of still frames equals the preset number of still frames threshold, the shopping action is determined to be over, and a signal to stop shooting is issued; the visual image data acquisition device is also used to stop shooting video frame image data when the signal to stop shooting is received.

8. The method as described in claim 1, characterized in that, The motion monitoring device within the shopping cart basket area is a distance data acquisition device, used to acquire the fluctuation of distance data within the shopping cart basket area and send the distance fluctuation data to the abnormal shopping behavior detection device. When it is determined from the motion data within the shopping cart basket area that a shopping action has occurred within the shopping cart basket area, a second trigger shooting signal is issued, including: when it is determined from the fluctuation of distance data that a shopping action has occurred within the shopping cart basket area, a second trigger shooting signal is issued.

9. The method as described in claim 8, characterized in that, Also includes: When the distance data stops fluctuating, a signal to stop shooting is issued; the visual image data acquisition device is also used to stop shooting video frame image data when it receives the signal to stop shooting.

10. The method as described in claim 1, characterized in that, Also includes: When the visual image data acquisition device is detected to be capturing video frame image data, and the first trigger capture signal and / or the second trigger capture signal are received again, the trigger capture command will not be sent to the visual image data acquisition device again.

11. The method as described in claim 1, characterized in that, Shopping action types include: adding items to the shopping cart after scanning the barcode, adding items to the shopping cart after scanning the barcode, adding items to the shopping cart without scanning the barcode, and taking items out of the shopping cart, or any combination thereof.

12. The method as described in claim 11, characterized in that, Based on data from product barcode scanners, the types of shopping actions performed by shoppers, and the number of items in motion, the system detects whether shoppers are engaging in abnormal shopping behavior, including: If the shopping action type is "adding items to the shopping cart without scanning the barcode", it is confirmed that the shopper has missed scanning the barcode. If the shopping action type is scanning a barcode and then adding the item to the shopping cart, then check if the number of items in the barcode scanner data matches the number of items in motion; if the number of items in motion is greater than the number of items in the barcode scanner data, it is confirmed that the shopper has missed scanning; and If the shopping action type is "scan the code and then add the item to the shopping cart", it is confirmed that the shopper has accidentally scanned and added the wrong item.

13. A method for detecting abnormal shopping behavior in a shopping cart, characterized in that, This method is applied to the system, and the abnormal shopping behavior detection method for the shopping cart includes: The barcode scanner acquires product barcode data and sends the product barcode data to the abnormal shopping behavior detection device; The motion monitoring device within the bicycle basket area monitors motion data within the bicycle basket area and sends the motion data within the bicycle basket area to the abnormal shopping behavior detection device; When the abnormal shopping behavior detection device determines that a shopping action has occurred based on the product scanning data, it issues a first trigger shooting signal; when it determines that a movement has occurred in the shopping cart basket area based on the movement data in the basket area, it issues a second trigger shooting signal; when it determines that a shopping action has occurred based on the first trigger shooting signal and / or the second trigger shooting signal, it sends a trigger shooting command to the visual image data acquisition device. Upon receiving the trigger shooting command, the visual image data acquisition device collects video frame image data of the shopper's shopping process and sends the video frame image data to the abnormal shopping behavior detection device; and The abnormal shopping behavior detection device determines the type of shopping action and the number of items in motion by using video frame image data during the shopping process; based on the data from the product barcode scanner, the type of shopping action, and the number of items in motion, it detects whether the shopper is engaging in abnormal shopping behavior.

14. A device for detecting abnormal shopping behavior in a shopping cart, characterized in that, include: The data acquisition unit is used to collect product scanning data obtained by the barcode scanner, as well as motion data within the basket area monitored by the motion monitoring device within the basket area. The first trigger shooting unit is used to issue a first trigger shooting signal when it is determined that a shopping action has occurred based on the product scanning data; The second trigger shooting unit is used to issue a second trigger shooting signal when it is determined from the motion data within the shopping basket area that a shopping action has occurred within the shopping basket area of ​​the shopping cart. The trigger shooting control unit is used to send a trigger shooting command to the visual image data acquisition device when a shopping action is determined to have occurred based on the first trigger shooting signal and / or the second trigger shooting signal; the visual image data acquisition device is used to collect video frame image data of the shopper during the shopping process when it receives the trigger shooting command, and send the video frame image data to the abnormal shopping behavior detection device. The analysis unit is used to determine the type of shopping action of the shopper and the number of items in motion based on video frame image data during the shopping process; as well as The detection unit is used to detect whether a shopper is engaging in abnormal shopping behavior based on data from the product scanner, the type of shopping action of the shopper, and the number of products in motion.

15. A system for detecting abnormal shopping behavior in a shopping cart, characterized in that, include: A barcode scanner is used to acquire product barcode data and send the product barcode data to an abnormal shopping behavior detection device. The motion monitoring device within the bicycle basket area is used to monitor motion data within the bicycle basket area and send the motion data within the bicycle basket area to the abnormal shopping behavior detection device. The visual image data acquisition device is used to acquire video frame image data of shoppers during the shopping process when a shooting command is received, and send the video frame image data to the abnormal shopping behavior detection device. as well as An abnormal shopping behavior detection device is used to issue a first trigger shooting signal when a shopping action is determined to have occurred based on product barcode scanning data; issue a second trigger shooting signal when a movement is determined to have occurred within the shopping cart basket area based on movement data within the basket area; send a trigger shooting command to a visual image data acquisition device when a shopping action is determined to have occurred based on the first and / or second trigger shooting signals; determine the type of shopping action and the number of moving products based on video frame image data during the shopper's shopping process; and detect whether the shopper is engaging in abnormal shopping behavior based on product barcode scanner data, the type of shopping action, and the number of moving products.

16. A shopping cart for detecting abnormal shopping behavior, characterized in that, include: The abnormal shopping behavior detection device for shopping carts as described in claim 15.

17. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 13.

18. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method of any one of claims 1 to 13.

19. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the method of any one of claims 1 to 13.

Images