Method, device and system for warning abnormal shopping behavior of shopping cart

Abstract

The invention discloses a warning method, device and system for abnormal shopping behaviors of a shopping cart. The method comprises the following steps: collecting video frame data and bar code information for operating commodities; intercepting video frame data with actions from the video frame data; reasoning the intercepted video frame data, and judging the action type of the action and the corresponding confidence coefficient; identifying the intercepted video frame data to obtain an identification result of the commodity; judging a shopping behavior type according to the action type, the bar code information and the identification result; if the shopping behavior type is an abnormal behavior, calculating the confidence coefficient of the abnormal behavior according to the confidence coefficient of the action type and the frame number of the intercepted video frame data; calculating a loss amount corresponding to the abnormal behavior; and when the loss amount exceeds an amount threshold and the confidence of the abnormal behavior is greater than a confidence threshold, generating first warning information. According to the invention, an abnormal shopping behavior can be warned, and the disturbance to a shopper is minimized.

Description

Technical Field

[0001] The present invention relates to the technical field of self-service shopping, and in particular, to a warning method, device, system and shopping cart for abnormal shopping behaviors of a shopping cart. Background Art

[0002] This section aims to provide background or context for embodiments of the present invention. The description herein is not admitted to be prior art merely because it is included in this section.

[0003] Currently, by applying technologies such as the Internet of Things, artificial intelligence, and sensors, shopping carts can not only automatically sense the information and location of goods, but also perform functions such as product recommendation and coupon use, forming intelligent shopping carts, providing consumers with personalized shopping experiences, and enhancing the convenience and efficiency of the entire shopping process. The application of intelligent shopping carts can also bring benefits of cost reduction and efficiency improvement to retailers, such as reducing the number of cashiers. At the same time, the data collected by intelligent shopping carts can also help retailers conduct precision marketing and improve sales performance. The market scale of the intelligent shopping cart industry is continuously growing and is expected to maintain stable growth in the next few years.

[0004] However, during the process of putting intelligent shopping carts into use in a supermarket environment, due to the characteristics of self-service use and settlement, some abnormal shopping behaviors will occur. Usually, intelligent shopping carts are equipped with weight sensors and visual AI algorithms to monitor the goods placed in the shopping cart and verify the unscanned goods. If an unpaid good is detected, the consumer will be reminded to scan the code or a reminder will be issued during settlement.

[0005] However, since weight sensors and visual AI algorithms often cannot cover complex shopping behaviors in reality during use, there will be false warnings and interceptions. Moreover, supermarkets need to make reasonable strategies that minimize the disruption to shoppers.

[0006] Therefore, there is a need for an effective warning solution for abnormal shopping behaviors of shopping carts currently. Summary of the Invention

[0007] An embodiment of the present invention provides a warning method for abnormal shopping behaviors of a shopping cart, which can give warnings for abnormal shopping behaviors and minimize the disruption to shoppers. The method is applied to a shopping cart device and includes:

[0008] Collecting video frame data of a shopper's operations on goods and barcode information of the goods during the process of using the shopping cart for shopping;

[0009] Intercepting the video frame data with actions on the goods from the video frame data;

[0010] Performing inference on the intercepted video frame data to determine the action type and corresponding confidence level of the actions that occur;

[0011] Identify the intercepted video frame data to obtain the identification result of the commodity;

[0012] Based on the action type, the barcode information, and the identification result, determine the shopping behavior type, where the shopping behavior type includes normal and abnormal behaviors;

[0013] If the shopping behavior type is an abnormal behavior, calculate the confidence level of the abnormal behavior based on the confidence level of the action type and the number of frames of the intercepted video frame data;

[0014] Calculate the loss amount corresponding to the abnormal behavior;

[0015] When the loss amount exceeds the amount threshold and the confidence level of the abnormal behavior is greater than the confidence level threshold, generate a first warning message.

[0016] An embodiment of the present invention provides a warning device for abnormal shopping behaviors in a shopping cart, which can give warnings for abnormal shopping behaviors and minimize the disturbance to shoppers. The device is applied to a shopping cart device and includes:

[0017] An acquisition module, configured to acquire video frame data of a shopper operating on a commodity during the shopping process using the shopping cart and the barcode information of the commodity;

[0018] A video frame data interception module, configured to intercept the video frame data with actions on the commodity from the video frame data;

[0019] An action confidence level calculation module, configured to reason about the intercepted video frame data to determine the action type and the corresponding confidence level of the action that occurs;

[0020] A commodity identification module, configured to identify the intercepted video frame data to obtain the identification result of the commodity;

[0021] A shopping behavior type judgment module, configured to determine the shopping behavior type based on the action type, the barcode information, and the identification result, where the shopping behavior type includes normal and abnormal behaviors;

[0022] A behavior confidence level calculation module, configured to calculate the confidence level of the abnormal behavior based on the confidence level of the action type and the number of frames of the intercepted video frame data if the shopping behavior type is an abnormal behavior;

[0023] A loss amount calculation module, configured to calculate the loss amount corresponding to the abnormal behavior;

[0024] A first warning message generation module, configured to generate a first warning message when the loss amount exceeds the amount threshold and the confidence level of the abnormal behavior is greater than the confidence level threshold.

[0025] An embodiment of the present invention provides a warning system for abnormal shopping behaviors of a shopping cart, including a shopping cart device corresponding to the aforementioned device and a background server. The shopping cart device is integrated with a vision sensor, a scanning device, and a display screen. Among them,

[0026] The vision sensor is used to: collect video frame data of a shopper operating on goods during the shopping process using the shopping cart;

[0027] The scanning device is used to: scan and obtain barcode information of goods;

[0028] The display screen is used to: display the first warning information;

[0029] The background server is used to: receive the first warning information sent by the shopping cart device.

[0030] An embodiment of the present invention also provides a computer device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the warning method for abnormal shopping behaviors of the above-mentioned shopping cart is implemented.

[0031] An embodiment of the present invention also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the warning method for abnormal shopping behaviors of the above-mentioned shopping cart is implemented.

[0032] An embodiment of the present invention also provides a computer program product. The computer program product includes a computer program. When the computer program is executed by a processor, the warning method for abnormal shopping behaviors of the above-mentioned shopping cart is implemented.

[0033] In an embodiment of the present invention, video frame data of a shopper's operations on goods during the shopping process using the shopping cart and barcode information of the goods are collected; video frame data with actions on the goods is intercepted from the video frame data; the intercepted video frame data is inferred to determine the action type and the corresponding confidence level of the actions that occur; the intercepted video frame data is recognized to obtain the recognition result of the goods; according to the action type, the barcode information, and the recognition result, the shopping behavior type is determined, and the shopping behavior type includes normal and abnormal behaviors; if the shopping behavior type is abnormal, according to the confidence level of the action type and the number of frames of the intercepted video frame data, the confidence level of the abnormal behavior is calculated; the loss amount corresponding to the abnormal behavior is calculated; when the loss amount exceeds the amount threshold and the confidence level of the abnormal behavior is greater than the confidence level threshold, a first warning message is generated. By calculating the confidence level of the abnormal behavior through the confidence level of the above action behavior and the number of frames of the intercepted video frame data to determine whether to generate a warning message, it can avoid the situation of giving false warnings due to the weight sensor and the visual AI algorithm being unable to cover complex shopping behaviors in reality. At the same time, by determining that a first warning message is generated only when the loss amount exceeds the amount threshold and the confidence level of the abnormal behavior is greater than the confidence level threshold, excessive warnings can be avoided, and the disturbance to shoppers can be minimized. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts. In the drawings:

[0035] Figure 1 It is a schematic flowchart of a warning method for abnormal shopping behavior of a shopping cart applied to a device in an embodiment of the present invention;

[0036] Figure 2 It is a schematic structural diagram of a warning device for abnormal shopping behavior of a shopping cart in an embodiment of the present invention;

[0037] Figure 3 It is another schematic structural diagram of a warning device for abnormal shopping behavior of a shopping cart in an embodiment of the present invention;

[0038] Figure 4 It is a schematic structural diagram of a warning system for abnormal shopping behavior of a shopping cart in an embodiment of the present invention;

[0039] Figure 5 It is a schematic structural diagram of a computer device in an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0040] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer and more understandable, the following further describes the embodiments of the present invention in detail with reference to the accompanying drawings. Herein, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but not to limit the present invention.

[0041] In the technical solutions of this application, the acquisition, storage, use, processing, etc. of data all comply with the relevant regulations of laws and regulations.

[0042] Figure 1 The following is a schematic flowchart of a warning method for abnormal shopping behavior of a shopping cart applied to a device in an embodiment of the present invention. It is applied to a shopping cart device, such as Figure 1 As shown, the method includes the following steps:

[0043] Step 101: Collect video frame data of a shopper's operations on goods during the shopping process using the shopping cart and barcode information of the goods;

[0044] Step 102: Intercept video frame data with actions on the goods from the video frame data;

[0045] Step 103: Infer the intercepted video frame data to determine the action type and corresponding confidence level of the actions that occurred;

[0046] Step 104: Identify the intercepted video frame data to obtain the identification result of the goods;

[0047] Step 105: Determine the shopping behavior type based on the action type, the barcode information, and the identification result. The shopping behavior type includes normal and abnormal behaviors;

[0048] Step 106: If the shopping behavior type is an abnormal behavior, calculate the confidence level of the abnormal behavior based on the confidence level of the action type and the number of frames of the intercepted video frame data;

[0049] Step 107: Calculate the loss amount corresponding to the abnormal behavior;

[0050] Step 108: Generate a first warning message when the loss amount exceeds the amount threshold and the confidence level of the abnormal behavior is greater than the confidence level threshold.

[0051] In an embodiment of the present invention, the shopping cart device is integrated with a vision sensor, a scanning device, and a display screen.

[0052] The vision sensor is used to: collect video frame data of a shopper's operations on goods during the shopping process using the shopping cart;

[0053] The scanning device is used to: scan and obtain barcode information of the goods;

[0054] The display screen is used for: displaying the first warning message;

[0055] In addition, the shopping cart device may be equipped with lights that can be lit. When the first warning message is displayed, the lights on the shopping cart are lit simultaneously.

[0056] The following is a detailed introduction to this method.

[0057] In step 101, video frame data of the actions of the shopper on the goods during the shopping process using the shopping cart and barcode information of the goods are collected;

[0058] During the shopping process of the shopper using the intelligent shopping cart, the vision sensor on the shopping cart device is activated to collect video frame data, and at the same time, the barcode information of the goods scanned by the shopper through the scanning device is obtained.

[0059] In step 102, the video frame data with actions on the goods is intercepted from the video frame data;

[0060] In one embodiment, intercepting the video frame data with actions on the goods from the video frame data includes:

[0061] According to the video frame data, it is judged whether there is disturbance in the shopping cart basket;

[0062] If so, it is determined that there is an action in the video frame data corresponding picture, and if not, it is determined that the video frame data corresponding picture is static;

[0063] The video frame data of the video segment corresponding to the picture from static to action occurrence and then to static is intercepted.

[0064] Specifically, in the embodiment of the present invention, the actions can be putting in, taking out, etc. It is necessary to interpret a complete video segment with actions of "putting in" or "taking out" and record the number of frames n of the video segment. The frame difference method, background subtraction method or optical flow method can be used for the images in the video frame data to calculate whether there is disturbance in the shopping cart basket. If there is pixel disturbance, it is considered that there is an action of putting in or taking out, otherwise it is static. Take the time points of the two static states before and after "static - disturbance - static" of the picture as the starting point and ending point of a video segment, and thus intercept the video segment.

[0065] In step 103, the intercepted video frame data is inferred to judge the action type and the corresponding confidence level of the action that occurs;

[0066] In the embodiment of the present invention, the action type is the type of actions such as putting in and taking out by the shopper on the goods.

[0067] In one embodiment, inferring the intercepted video frame data to determine the action type and corresponding confidence level of the occurring action includes:

[0068] Sending the intercepted video frame data to an action recognition model to obtain the action type and corresponding confidence level conf a , where the action recognition model is a classification model obtained through training.

[0069] Specifically, the action recognition model is obtained by training with a large amount of video segment data with label information using deep learning methods.

[0070] In step 104, recognizing the intercepted video frame data to obtain the recognition result of the commodity;

[0071] In one embodiment, recognizing the intercepted video frame data to obtain the recognition result of the commodity includes:

[0072] Detecting the intercepted video frame data to obtain the detection bounding box of the commodity;

[0073] According to the detection bounding box, cropping out the image of the commodity from the intercepted video frame data;

[0074] Recognizing the cropped image of the commodity to obtain the recognition result of the commodity.

[0075] Specifically, the intercepted video frame data can be input into a moving object detection model, and the detection bounding box of the commodity in the intercepted video frame data is output; among them, the moving object detection model is an object detection model obtained by training with a large amount of data sets with label information using deep learning methods.

[0076] According to the detection bounding box of the commodity, cropping out the image of the commodity from the intercepted video frame data and sending it into a commodity recognition model to output the recognition result of each frame of the commodity, specifically including the barcode information corresponding to the commodity. Among them, the commodity recognition model belongs to a classification model obtained by training with a large amount of data sets with label information using deep learning methods, and can directly recognize that the image belongs to a specific commodity, and then the corresponding barcode information can be queried from the database.

[0077] In step 105, judging the shopping behavior type according to the action type, the barcode information and the recognition result, where the shopping behavior type includes normal and abnormal behaviors;

[0078] In one embodiment, the abnormal behaviors include missed scanning behavior and mis-scanning behavior;

[0079] Judging the shopping behavior type according to the action type, the barcode information and the recognition result includes:

[0080] If the action type is "put in" and the barcode information is empty, determine that the shopping behavior type is a missed scanning behavior;

[0081] If the action type is "put in" and the barcode information is consistent with the barcode information in the recognition result, determine that the shopping behavior type is a normal behavior;

[0082] If the action type is "put in" and the barcode information is inconsistent with the barcode information in the recognition result, determine that the shopping behavior type is a mis-scanning behavior.

[0083] Specifically, if the result of the inference operation is "put in", first check whether there is barcode scanning information before this moment. If there is no barcode information before this moment, it is determined as a "missed scanning behavior"; if there is barcode information, check whether the barcode information sent by scanning is consistent with the barcode information of the recognized commodity. If they are consistent, it is determined as a "normal behavior", and if they are inconsistent, it is determined as a "mis-scanning behavior".

[0084] In step 106, if the shopping behavior type is an abnormal behavior, calculate the confidence level of the abnormal behavior according to the confidence level of the action type and the number of frames of the intercepted video frame data;

[0085] In an embodiment, the following formula is used to calculate the confidence level of the abnormal behavior according to the confidence level of the action type and the number of frames of the intercepted video frame data, including:

[0086]

[0087] where conf is the confidence level of the abnormal behavior, α and β are weight values, conf a is the confidence level of the action type, n is the number of frames of the intercepted video frame data, and fps is the frame rate at which the video frame data is obtained.

[0088] The meaning of the above formula is that the confidence level of an action being determined as a theft or damage behavior is equal to the weighted sum of the confidence level of the action type itself and the duration of the action. When an action (put in or take out) occurs, the higher the confidence level of the action type and the closer the number of frames it lasts to half of the fps, the higher the confidence level of the abnormal behavior.

[0089] Among them, the embodiment of the present invention believes that when the number of frames of the action lasts is closer to half of the fps, the confidence level of the abnormal behavior is the highest, because according to statistical data, a normal put-in or take-out action usually takes only 500 milliseconds to complete. If the action is too slow or too fast, it will deviate from half of the fps. At this time, the embodiment of the present invention uses the "normal distribution function" to restrict the degree of influence, that is, the slower and faster the action is ultimately, the lower the final confidence level will be.

[0090] In step 107, calculate the loss amount corresponding to the abnormal behavior;

[0091] In one embodiment, calculating the loss amount corresponding to the abnormal behavior includes:

[0092] When the abnormal behavior is a missed scan behavior, use the amount of the commodity in the recognition result as the loss amount;

[0093] When the abnormal behavior is a mis-scan behavior, use the difference between the amount of the commodity in the recognition result and the amount corresponding to the barcode information of the collected commodity as the loss amount.

[0094] Specifically, use the amount corresponding to the recognition result of the commodity placed for the missed scan behavior at this time as the amount of theft loss. For example, for a commodity that is not scanned but placed in the vehicle, the recognition result is "500ml Coke" with a barcode of "69000121222". After querying, the corresponding commodity price is "4.00", then the calculated loss amount is "4.00".

[0095] Use the difference between the amount corresponding to the recognition result of the commodity placed for the mis-scan behavior at this time and the amount corresponding to the barcode information scanned and input as the loss amount. For example, for a commodity placed in the vehicle, the recognition result is "1000ml Coke" with a barcode of "69000121222". After querying, the corresponding commodity price is "10.00", but after scanning with the barcode scanner, "69000121211" is input, and the corresponding amount is "3.00", then the loss amount at this time is 10.00 - 3.00 = 7.00.

[0096] In step 108, when the loss amount exceeds the amount threshold and the confidence level of the abnormal behavior is greater than the confidence level threshold, generate a first warning message.

[0097] Specifically, the generated first warning message is a preliminary warning. The methods of the preliminary warning include: directly displaying the signal of "putting something in without scanning" on the interactive screen of the shopping cart device; displaying a logo of a certain color and a certain shape on the interactive screen of the shopping cart device, such as displaying a yellow exclamation mark; or lighting up the lights on the shopping cart device and flashing at a certain color or a certain frequency to remind the shopper.

[0098] If at this time, one of the determined confidence level conf and the loss amount does not exceed the set threshold, upload the abnormal information of this shopper to the background server.

[0099] In one embodiment, the method further includes:

[0100] If the loss amount does not exceed the amount threshold, or the confidence level of the abnormal behavior is not greater than the confidence level threshold, the potential abnormal information is sent to the background server, so that when the background server determines that a warning is needed based on all the potential abnormal information of the shopper, a second warning information is generated. The potential abnormal information includes the shopper's identity identification, the confidence level of the abnormal behavior, the loss amount, and the intercepted video frame data.

[0101] If the second warning information is received, it is displayed through the display of the shopping cart.

[0102] Specifically, the background server receives the potential abnormal information uploaded by each shopping cart device in real time, and then takes the shopper's identity identification (such as ID) as an induction, calculates all the previous abnormal information of the shopper corresponding to this identity identification, and calculates the frequency of the shopper's abnormal behavior, the total loss amount, and the total confidence level.

[0103] In an embodiment, when the background server determines that a warning is needed based on all the potential abnormal information of the shopper, the second warning information generated includes:

[0104] Obtain all the abnormal information of the shopper corresponding to the shopper's identity identification in the potential abnormal information. The abnormal information includes the shopper's identity identification, the confidence level of the abnormal behavior, and the loss amount.

[0105] Based on all the abnormal information of this shopper, calculate the frequency of the shopper's abnormal behavior, the total loss amount, and the total confidence level.

[0106] If the frequency of the shopper's abnormal behavior exceeds the frequency threshold, and the total loss amount exceeds the loss threshold, and the total confidence level exceeds the total confidence level threshold, it is determined that a warning is needed, and a second warning information is generated.

[0107] Specifically, if the frequency of the shopper's abnormal behavior exceeds the frequency threshold, and the total loss amount exceeds the loss threshold, and the total confidence level exceeds the total confidence level threshold, then the shopper's current potential abnormal behavior is characterized, it is determined that a warning is needed, a second warning information is generated, and the generated second warning information is sent to the shopping cart device that the shopper is currently using. The shopping cart device issues the second warning information, and the warning method is as above to achieve a secondary warning.

[0108] In addition, the second warning information and the intercepted video frame data are also sent to the store clerk with verification functions in the store, and sent to the mobile device of the clerk.

[0109] In an embodiment, the method further includes:

[0110] The background server also sends the second warning information and the intercepted video frame data to the mobile device of the store clerk, so that the clerk can intercept the shopper when it is determined through the mobile device that the shopper needs to be warned.

[0111] Specifically, the store clerk who receives the shopper's theft and damage information checks the video segment image information of the theft and damage behavior to confirm whether the shopper actually has abnormal behavior.

[0112] After verification by the store clerk, when the shopper walks out of the store or returns the shopping cart device, intercept and check the shopper whose shopping cart device has a prompt sign or the light is lit, to achieve three interceptions.

[0113] An embodiment of the present invention also provides a warning device for abnormal shopping behavior of a shopping cart, as described in the following embodiment. Since the principle of solving problems by this device is similar to that of the warning method for abnormal shopping behavior of a shopping cart, the implementation of this device can refer to the implementation of the warning method for abnormal shopping behavior of a shopping cart, and the repeated parts will not be described again.

[0114] Figure 2 It is a schematic structural diagram of the warning device for abnormal shopping behavior of a shopping cart in an embodiment of the present invention, which is applied to a shopping cart device. The device includes:

[0115] An acquisition module 201, configured to acquire video frame data of a shopper's operation on a commodity and barcode information of the commodity during the process of using the shopping cart for shopping;

[0116] A video frame data interception module 202, configured to intercept video frame data with actions on the commodity from the video frame data;

[0117] An action confidence calculation module 203, configured to reason about the intercepted video frame data to judge the action type and the corresponding confidence of the action;

[0118] A commodity recognition module 204, configured to recognize the intercepted video frame data to obtain the recognition result of the commodity;

[0119] A shopping behavior type judgment module 205, configured to judge the shopping behavior type according to the action type, the barcode information and the recognition result, and the shopping behavior type includes normal and abnormal behaviors;

[0120] A behavior confidence calculation module 206, configured to calculate the confidence of the abnormal behavior according to the confidence of the action type and the number of frames of the intercepted video frame data if the shopping behavior type is abnormal;

[0121] A loss amount calculation module 207, configured to calculate the loss amount corresponding to the abnormal behavior;

[0122] The first warning information generation module 208 is configured to generate a first warning information when the loss amount exceeds the amount threshold and the confidence level of the abnormal behavior is greater than the confidence level threshold.

[0123] In one embodiment, the video frame data intercepting module is configured to:

[0124] Determine whether there is any disturbance in the basket of the shopping cart according to the video frame data;

[0125] If so, determine that there is an action in the corresponding picture of the video frame data, and if not, determine that the corresponding picture of the video frame data is static;

[0126] Intercept the video frame data of the video segment corresponding to the picture from static to action and then to static.

[0127] In one embodiment, the commodity recognition module is configured to:

[0128] Detect the intercepted video frame data to obtain the detection bounding box of the commodity;

[0129] According to the detection bounding box, crop the image of the commodity from the intercepted video frame data;

[0130] Identify the cropped image of the commodity to obtain the recognition result of the commodity.

[0131] In one embodiment, the abnormal behavior includes missed scanning behavior and mis-scanning behavior;

[0132] The shopping behavior type judgment module is configured to:

[0133] If the action type is putting in and the barcode information is empty, determine that the shopping behavior type is missed scanning behavior;

[0134] If the action type is putting in and the barcode information is consistent with the barcode information in the recognition result, determine that the shopping behavior type is normal behavior;

[0135] If the action type is putting in and the barcode information is inconsistent with the barcode information in the recognition result, determine that the shopping behavior type is mis-scanning behavior.

[0136] In one embodiment, the behavior confidence calculation module is configured to:

[0137] Adopt the following formula to calculate the confidence level of the abnormal behavior according to the confidence level of the action type and the number of frames of the intercepted video frame data, including:

[0138]

[0139] where conf is the confidence level of the abnormal behavior, α and β are weight values, confa Confidence for the action type, n is the number of frames of the intercepted video frame data, and fps is the frame rate at which the video frame data is captured.

[0140] In one embodiment, the loss amount calculation module is configured to:

[0141] When the abnormal behavior is the missed scanning behavior, take the amount of the commodity in the recognition result as the loss amount;

[0142] When the abnormal behavior is the wrong scanning behavior, take the difference between the amount of the commodity in the recognition result and the amount corresponding to the barcode information of the collected commodity as the loss amount.

[0143] See Figure 3 Another structural schematic diagram of the warning device for abnormal shopping behavior of the shopping cart in the embodiment of the present invention. The device further includes a second warning module 301, which is used for:

[0144] If the loss amount does not exceed the amount threshold, or the confidence of the abnormal behavior is not greater than the confidence threshold, send the potential abnormal information to the background server, so that when the background server determines that a warning is needed based on all the potential abnormal information of the shopper, generate a second warning information. The potential abnormal information includes the identity recognition of the shopper, the confidence of the abnormal behavior, the loss amount, and the intercepted video frame data;

[0145] If receiving the second warning information, display it through the display of the shopping cart.

[0146] In the embodiment of the present invention, a warning system for abnormal shopping behavior of a shopping cart is also provided, as described in the following embodiments. Since the principle of solving problems by this system is similar to that of the warning method for abnormal shopping behavior of a shopping cart, the implementation of this system can refer to the implementation of the warning method for abnormal shopping behavior of a shopping cart, and the repeated parts will not be described again.

[0147] Figure 4 Structural schematic diagram of the warning system for abnormal shopping behavior of the shopping cart in the embodiment of the present invention, as Figure 4 shown, the system includes: the shopping cart device 401 corresponding to the aforementioned device and the background server 402. The shopping cart device 401 is integrated with a vision sensor 4011, a scanning device 4012, and a display screen 4013, wherein,

[0148] The vision sensor 4011 is used for: collecting video frame data of the shopper operating on the commodity during the shopping process using the shopping cart;

[0149] The scanning device 4012 is used for: scanning to obtain the barcode information of the commodity;

[0150] The display screen 4013 is used for: displaying the first warning message;

[0151] The background server 402 is used for: receiving the first warning message sent by the shopping cart device.

[0152] In summary, the warning solution for abnormal shopping behavior of the shopping cart provided by the embodiments of the present invention can collect video frame data of the operations performed on the commodities during the shopping process of the shopper using the shopping cart and the barcode information of the commodities; intercept the video frame data with actions on the commodities from the video frame data; perform reasoning on the intercepted video frame data to judge the action type and the corresponding confidence level of the actions; identify the intercepted video frame data to obtain the identification result of the commodities; judge the shopping behavior type according to the action type, the barcode information and the identification result, and the shopping behavior type includes normal and abnormal behaviors; if the shopping behavior type is abnormal, calculate the confidence level of the abnormal behavior according to the confidence level of the action type and the number of frames of the intercepted video frame data; calculate the loss amount corresponding to the abnormal behavior; when the loss amount exceeds the amount threshold and the confidence level of the abnormal behavior is greater than the confidence level threshold, generate the first warning message. The idea of calculating the confidence level of the abnormal behavior through the confidence level of the above action behavior and the number of frames of the intercepted video frame data to judge whether to generate a warning message can avoid the situation of giving wrong warnings due to the inability of the weight sensor and the vision AI algorithm to cover complex shopping behaviors in reality. At the same time, by judging that the first warning message is generated only when the loss amount exceeds the amount threshold and the confidence level of the abnormal behavior is greater than the confidence level threshold, excessive warnings can be avoided, and the disturbance to the shopper can be minimized.

[0153] Based on the foregoing inventive concept, Figure 5 As a schematic structural diagram of a computer device in the embodiments of the present invention, the present invention also proposes a computer device 500, including a memory 510, a processor 520, and a computer program 530 stored on the memory 510 and executable on the processor 520. When the processor 520 executes the computer program 530, the warning method for abnormal shopping behavior of the shopping cart is implemented.

[0154] The embodiments of the present invention also provide a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the warning method for abnormal shopping behavior of the shopping cart is implemented.

[0155] The embodiments of the present invention also provide a computer program product, which includes a computer program. When the computer program is executed by a processor, the warning method for abnormal shopping behavior of the shopping cart is implemented.

[0156] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memory, CD-ROM, optical memory, etc.) that contain computer-usable program code.

[0157] The present invention is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to embodiments of the present invention. It should be understood that each flow and / or block in the flowchart and / or block diagram, and the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices generate means for implementing the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0158] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including instruction means that implement the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0159] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0160] The specific embodiments described above further elaborate on the objectives, technical solutions, and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not used to limit the protection scope of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for warning of abnormal shopping behavior in a shopping cart, characterized in that: Applied to a shopping cart device, the method comprises: Collecting video frame data of a shopper operating a product using the shopping cart and barcode information of the product; Intercepting video frame data in which an action on the commodity occurs from the video frame data; Reasoning on the captured video frame data to determine the action type and corresponding confidence level of the action; Identify the captured video frame data to obtain an identification result of the product; Determine the shopping behavior type according to the action type, the barcode information and the recognition result, wherein the shopping behavior type includes normal and abnormal behaviors; If the shopping behavior type is abnormal behavior, the confidence of the abnormal behavior is calculated according to the confidence of the action type and the number of frames of the intercepted video frame data; Calculate the amount of loss corresponding to abnormal behavior; When the loss amount exceeds the amount threshold and the confidence of the abnormal behavior is greater than the confidence threshold, a first warning message is generated.

2. The method according to claim 1, characterized in that Intercepting video frame data in which an action on the commodity occurs from the video frame data, including: Determining whether there is a disturbance in the basket of the shopping cart according to the video frame data; If so, it is determined that an action has occurred in the picture corresponding to the video frame data, and if not, it is determined that the picture corresponding to the video frame data is still; The video frame data of the video segment corresponding to the scene from stillness to action and then back to stillness is captured.

3. The method according to claim 1, characterized in that Identify the captured video frame data to obtain the identification result of the product, including: Detecting the captured video frame data to obtain a detection boundary box of the product; According to the detection boundary box, cropping the image of the product from the captured video frame data; The cropped image of the commodity is recognized to obtain a recognition result of the commodity.

4. The method according to claim 1, characterized in that The abnormal behavior includes missed scanning and wrong scanning; Judging the shopping behavior type according to the action type, the barcode information and the recognition result includes: If the action type is insert, and the barcode information is empty, the shopping behavior type is determined to be a missed scan behavior; If the action type is inserting, and the barcode information is consistent with the barcode information in the recognition result, the shopping behavior type is determined to be normal behavior; If the action type is inserting, and the barcode information is inconsistent with the barcode information in the recognition result, it is determined that the shopping behavior type is a wrong scanning behavior.

5. The method according to claim 1, characterized in that The confidence of abnormal behavior is calculated using the following formula based on the confidence of the action type and the number of frames of the captured video frame data, including: Among them, conf is the confidence of abnormal behavior, α and β are weight values, and conf a is the confidence of the action type, n is the number of frames of the captured video frame data, and fps is the frame rate when obtaining the video frame data.

6. The method according to claim 4, characterized in that Calculate the loss amount corresponding to abnormal behavior, including: When the abnormal behavior is a missed scan, the amount of the goods in the identification result is used as the loss amount; When the abnormal behavior is a wrong scanning behavior, the difference between the amount of the product in the recognition result and the amount corresponding to the barcode information of the collected product is used as the loss amount.

7. The method according to claim 1, characterized in that Also includes: If the loss amount does not exceed the amount threshold, or the confidence of the abnormal behavior is not greater than the confidence threshold, the potential abnormal information is sent to the backend server, so that the backend server generates a second warning message when determining that a warning is needed based on all the potential abnormal information of the shopper, the potential abnormal information includes the identity of the shopper, the confidence of the abnormal behavior, the loss amount, and the intercepted video frame data; If a second warning message is received, it is displayed on the display of the shopping cart.

8. The method according to claim 7, characterized in that When the backend server determines that a warning is required based on all potential abnormal information of the shopper, it generates a second warning message, including: Obtaining all abnormal information of the shopper corresponding to the shopper's identity identification in the potential abnormal information, wherein the abnormal information includes the shopper's identity identification, the confidence level of the abnormal behavior, and the amount of loss; Based on all abnormal information of the shopper, the frequency of the shopper's abnormal behavior, the total amount of loss, and the total confidence level are calculated; If the frequency of the shopper's abnormal behavior exceeds the frequency threshold, and the total loss amount exceeds the loss threshold, and the total confidence exceeds the total confidence threshold, it is determined that a warning is needed and a second warning message is generated.

9. The method according to claim 8, characterized in that Also includes: The backend server also sends the second warning information and the captured video frame data to the clerk's mobile device, so that the clerk can intercept the shopper when determining through the mobile device that the shopper needs a warning.

10. A warning device for abnormal shopping behavior in a shopping cart, characterized in that: Applied to a shopping cart device, the device comprises: A collection module, used to collect video frame data of a shopper operating a product in the process of shopping with the shopping cart and barcode information of the product; A video frame data interception module, used to intercept the video frame data in which an action occurs on the commodity from the video frame data; The action confidence calculation module is used to infer the captured video frame data and determine the action type and corresponding confidence of the action; A commodity identification module is used to identify the intercepted video frame data and obtain the identification result of the commodity; A shopping behavior type determination module, used to determine the shopping behavior type according to the action type, the barcode information and the recognition result, wherein the shopping behavior type includes normal and abnormal behaviors; A behavior confidence calculation module is used to calculate the confidence of the abnormal behavior according to the confidence of the action type and the number of frames of the intercepted video frame data if the shopping behavior type is abnormal behavior; A loss amount calculation module is used to calculate the loss amount corresponding to abnormal behavior; The first warning information generating module is used to generate the first warning information when the loss amount exceeds the amount threshold and the confidence of the abnormal behavior is greater than the confidence threshold.

11. A warning system for abnormal shopping behavior in a shopping cart, characterized in that: The device comprises a shopping cart device and a backend server corresponding to the device of claim 10, wherein the shopping cart device is integrated with a visual sensor, a scanning device and a display screen, wherein: The visual sensor is used to: collect video frame data of shoppers operating commodities in the process of shopping with the shopping cart; The scanning device is used to: scan and obtain the barcode information of the product; The display screen is used to: display the first warning information; The backend server is used to receive the first warning information sent by the shopping cart device.

12. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 9 is implemented.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

14. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images