Method and device for monitoring state in shopping cart

Abstract

The invention discloses a method and a device for monitoring a state in a shopping cart. The method comprises the following steps of: detecting whether an intentional shielding operation exists in the shopping cart for each frame of image data; detecting texture information: detecting edge texture information in each frame of image; a texture number determination step: determining an edge texture number according to the edge texture information; an intentional shielding judgment step: when the number of the edge textures is less than a preset threshold value of the number of the edge textures, determining that a preliminary result of intentional shielding exists in the shopping cart; performing intentional shielding secondary confirmation operation on the preliminary result; sequentially performing the steps of texture information detection, texture number determination and intentional shielding judgment on each frame of image in the basket area which is acquired at the preset acquisition frequency; and determining a final result of the intentional shielding state in the shopping cart when the accumulated intentional shielding frame number in the preset second reaches a preset frame number threshold value. The state in the shopping cart can be monitored in real time.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent shopping carts, and in particular, to a method and device for monitoring the internal state of a shopping cart. Background Art

[0002] This section aims to provide background or context for the embodiments of the present invention recited in the claims. The description herein is not admitted to be prior art merely by virtue of being included in this section.

[0003] With the rapid development of technologies such as the Internet of Things, artificial intelligence, big data analysis, mobile payment, and intelligent hardware, the retail industry is undergoing an intelligent and digital transformation in order to better meet the needs of consumers, enhance the shopping experience, and optimize the operational efficiency of merchants. In this wave of transformation, the innovation of the shopping settlement link is particularly crucial, and intelligent shopping carts emerge as the times require, becoming a major highlight in the supermarket environment. They are designed to provide a self-service shopping experience, enabling customers to bypass the checkout counter and avoid communicating with cashiers by simplifying the traditional shopping and payment processes, thereby significantly enhancing the convenience and pleasure of shopping. At the same time, for merchants, the configuration of cashiers can be reduced, thus reducing operating costs.

[0004] Generally speaking, intelligent shopping carts have a series of advanced functions, including automatic scanning and settlement, navigation and product positioning, item recognition and weighing, personalized recommendation and advertising, intelligent prevention of abnormal shopping behaviors, etc. (strategies). Usually, new intelligent shopping carts are configured with visual sensors to capture the shopping process of shoppers for subsequent inference and calculation to complete the functions of intelligent settlement and intelligent prevention of abnormal shopping behaviors.

[0005] However, during the use of shopping carts, situations that interfere with the visual image often occur, such as the camera being deliberately blocked, or the seat board being opened, a child sitting in the shopping cart, or the stacked goods blocking the field of view covered by the camera. When these situations occur, the functions of intelligent settlement and intelligent prevention of abnormal shopping behaviors face greater challenges. In addition, the boundary information of the shopping cart and whether there are few or basically full goods in the shopping cart will affect intelligent settlement and loss prevention. If these situations can be calculated and perceived in real time, it is beneficial for the built-in algorithms of intelligent shopping carts to achieve better effects. Summary of the Invention

[0006] Embodiments of the present invention provide a method for monitoring the internal state of a shopping cart to monitor different internal states of the shopping cart in real time. The method includes:

[0007] Obtaining video frame image data within the range of the shopping cart basket area collected at a first preset time interval;

[0008] For each frame of image data, perform the following operation to detect whether there is a deliberate occlusion state in the shopping cart:

[0009] Steps for texture information detection: For each frame of image data obtained, perform digital graphics processing to detect the edge texture information in the image;

[0010] Steps for determining the number of textures: Determine the number of edge textures according to the edge texture information in the image;

[0011] Steps for deliberate occlusion judgment: When the number of edge textures in the image is less than the preset threshold of the number of edge textures, determine a preliminary result that there may be a deliberately occluded state in the shopping cart;

[0012] When determining the preliminary result that there may be a deliberately occluded state in the shopping cart, perform the following operation for secondary confirmation of deliberate occlusion on the preliminary result:

[0013] Control to shorten the first preset time interval to the second preset time interval to achieve the preset acquisition frequency, and acquire video frame image data within the range of the shopping cart basket area;

[0014] Perform the above steps of texture information detection, steps for determining the number of textures, and steps for deliberate occlusion judgment on each frame of image acquired at the preset acquisition frequency in sequence;

[0015] When the number of frames with deliberate occlusion accumulated within the preset number of seconds reaches the preset frame threshold, determine the final result that there may be a deliberately occluded state in the shopping cart.

[0016] An embodiment of the present invention further provides a device for monitoring the state inside a shopping cart, which is used to monitor different states inside the shopping cart in real time. The device includes:

[0017] An acquisition unit, configured to acquire video frame image data within the range of the shopping cart basket area acquired at a first preset time interval;

[0018] A state monitoring unit, configured to perform the following operation for each frame of image data to detect whether there is a deliberate occlusion state in the shopping cart:

[0019] Steps for texture information detection: For each frame of image data obtained, perform digital graphics processing to detect the edge texture information in the image;

[0020] Steps for determining the number of textures: Determine the number of edge textures according to the edge texture information in the image;

[0021] Steps for deliberate occlusion judgment: When the number of edge textures in the image is less than the preset threshold of the number of edge textures, determine a preliminary result that there may be a deliberately occluded state in the shopping cart;

[0022] When determining the preliminary result that there may be a deliberately blocked state in the shopping cart, the following operation of secondary confirmation of deliberate occlusion is performed on the preliminary result:

[0023] Control to shorten the first preset time interval to the second preset time interval to achieve the preset acquisition frequency, and acquire the video frame image data within the range of the shopping cart basket area;

[0024] Perform the above steps of texture information detection, texture quantity determination, and deliberate occlusion judgment on each frame of image acquired at the preset acquisition frequency in sequence;

[0025] When the number of frames with deliberate occlusion accumulated within the preset number of seconds reaches the preset frame number threshold, determine the final result that there may be a deliberately blocked state in the shopping cart.

[0026] An embodiment of the present invention also provides a computer device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the method for monitoring the state inside the shopping cart is implemented.

[0027] An embodiment of the present invention also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the method for monitoring the state inside the shopping cart is implemented.

[0028] An embodiment of the present invention also provides a computer program product. The computer program product includes a computer program. When the computer program is executed by a processor, the method for monitoring the state inside the shopping cart is implemented.

[0029] In the embodiments of the present invention, the solution for monitoring the state inside the shopping cart is as follows: Obtain video frame image data within the range of the shopping cart basket area collected at a first preset time interval; for each frame of image data, perform the following operation to detect whether there is a deliberate occlusion state inside the shopping cart: The step of texture information detection: For each frame of image data obtained, perform digital graphics processing to detect the edge texture information in the image; The step of determining the texture quantity: Determine the edge texture quantity according to the edge texture information in the image; The step of judging deliberate occlusion: When the edge texture quantity in the image is less than the preset edge texture quantity threshold, determine a preliminary result that there may be a deliberately occluded state inside the shopping cart; When determining the preliminary result that there may be a deliberately occluded state inside the shopping cart, perform the following operation for secondary confirmation of deliberate occlusion on the preliminary result: Control to shorten the first preset time interval to a second preset time interval to reach the preset acquisition frequency, and collect video frame image data within the range of the shopping cart basket area; For each frame collected at the preset acquisition frequency, sequentially perform the above steps of texture information detection, the step of determining the texture quantity, and the step of judging deliberate occlusion; When the number of frames with deliberate occlusion accumulated within a preset number of seconds reaches the preset frame number threshold, determine the final result that there may be a deliberately occluded state inside the shopping cart. The embodiments of the present invention can monitor the state inside the shopping cart in real time. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings. In the drawings:

[0031] Figure 1 It is a flowchart of the method for monitoring the state inside the shopping cart in the embodiments of the present invention;

[0032] Figure 2 It is a flowchart of the method for detecting whether there is an unintentional occlusion state inside the shopping cart in the embodiments of the present invention;

[0033] Figure 3 It is a flowchart of the method for detecting the state of the commodity capacity inside the shopping cart in the embodiments of the present invention;

[0034] Figure 4 It is a flowchart of the method for detecting the state of the shopping cart basket boundary in the embodiments of the present invention;

[0035] Figure 5 It is a schematic diagram of the output result of the shopping cart basket boundary detection or segmentation model in the embodiments of the present invention;

[0036] Figure 6 Schematic structural diagram of the device for monitoring the in - vehicle state of the shopping cart in the embodiment of the present invention;

[0037] Figure 7 Schematic structural diagram of a computer device according to an embodiment of the present invention. Specific embodiments

[0038] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer and more understandable, the following further describes the embodiments of the present invention in detail with reference to the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but not to limit the present invention.

[0039] In the technical solution of this application, the acquisition, storage, use, processing, etc. of data all comply with the relevant regulations of laws and regulations.

[0040] The embodiment of the present invention provides a method for monitoring the in - vehicle state of a shopping cart. This method can perceive different states inside the shopping cart, and the method includes the following steps: obtaining video frame image data within the range of the shopping cart basket area collected at a first preset time interval; for each frame of image data, perform the following operations to detect different states inside the shopping cart: for each frame of image data, perform the following operations to detect intentional and / or unintentional occlusion states inside the shopping cart: according to the video frame image data, detect whether there is an intentional occlusion state inside the shopping cart; according to the video frame image data, detect whether there is an unintentional occlusion state inside the shopping cart; when it is detected that there is no intentional or unintentional occlusion state inside the shopping cart, for each frame of image data, perform the following operations to detect the state of the commodity capacity and / or the state of the basket boundary inside the shopping cart: according to the video frame image data, detect the state of the commodity capacity inside the shopping cart; according to the video frame image data, detect the state of the shopping cart basket boundary; wherein, the different states detected inside the shopping cart are used to adjust different strategies of the intelligent shopping cart. The following details this method.

[0041] Figure 1 Schematic flow diagram of the method for monitoring the in - vehicle state of the shopping cart in the embodiment of the present invention, as Figure 1 shown, this method includes the following steps:

[0042] Step 10: Obtain video frame image data within the range of the shopping cart basket area collected at a first preset time interval;

[0043] Step 20: For each frame of image data, perform the following operation to detect whether there is an intentional occlusion state inside the shopping cart:

[0044] Step 201: Step of texture information detection: For each frame of image data obtained, perform digital graphics processing to detect the edge texture information in the image;

[0045] Step 202: Step for determining the number of textures: Determine the number of edge textures according to the edge texture information in the image.

[0046] Step 203: Step for judging intentional occlusion: When the number of edge textures in the image is less than the preset threshold of the number of edge textures, determine a preliminary result that there may be an intentionally occluded state in the shopping cart.

[0047] Step 204: When determining the preliminary result that there may be an intentionally occluded state in the shopping cart, perform the following operation for secondary confirmation of intentional occlusion on the preliminary result:

[0048] Step 2041: Control to shorten the first preset time interval to the second preset time interval to achieve the preset acquisition frequency, and acquire video frame image data within the range of the shopping cart basket area.

[0049] Step 2042: For each frame of image acquired at the preset acquisition frequency, sequentially perform the above steps of texture information detection, step for determining the number of textures, and step for judging intentional occlusion.

[0050] Step 2043: When the number of frames with intentional occlusion accumulated within the preset number of seconds reaches the preset frame number threshold, determine the final result that there may be an intentionally occluded state in the shopping cart.

[0051] In the method for monitoring the state inside the shopping cart provided by the embodiment of the present invention, the shopping cart can be an intelligent shopping cart. When the method works: Acquire video frame image data within the range of the shopping cart basket area acquired at the first preset time interval; for each frame of image data, perform the following operation for detecting whether there is an intentional occlusion state inside the shopping cart: Step of texture information detection: For each frame of image data acquired, perform digital graphics processing to detect the edge texture information in the image; Step for determining the number of textures: Determine the number of edge textures according to the edge texture information in the image; Step for judging intentional occlusion: When the number of edge textures in the image is less than the preset threshold of the number of edge textures, determine a preliminary result that there may be an intentionally occluded state in the shopping cart; when determining the preliminary result that there may be an intentionally occluded state in the shopping cart, perform the following operation for secondary confirmation of intentional occlusion on the preliminary result: Control to shorten the first preset time interval to the second preset time interval to achieve the preset acquisition frequency, and acquire video frame image data within the range of the shopping cart basket area; for each frame of image acquired at the preset acquisition frequency, sequentially perform the above steps of texture information detection, step for determining the number of textures, and step for judging intentional occlusion; when the number of frames with intentional occlusion accumulated within the preset number of seconds reaches the preset frame number threshold, determine the final result that there may be an intentionally occluded state in the shopping cart. The embodiment of the present invention can monitor different states inside the shopping cart in real time. The following details the method for monitoring the state inside the shopping cart.

[0052] An embodiment of the present invention provides a method for monitoring the state inside a shopping cart. This method is a method for perceiving the state inside the shopping cart. Based on a shopping cart configured with a vision camera, a computing and interaction device (hereinafter referred to as an intelligent shopping cart), the intelligent shopping cart is given the ability to perceive and judge the situation inside the basket in real time. This ability includes that during the use of the intelligent shopping cart, images can be taken through the camera, and then a series of algorithms can judge whether there are intentional occlusions, unintentional occlusions, the capacity of goods in the basket, the basket boundary, etc. inside the basket of the intelligent shopping cart. By perceiving and judging these states and reporting them to the computer system of the intelligent shopping cart, the intelligent settlement and intelligent anti-abnormal shopping behavior algorithms adopt specific strategies according to the reported actual situation, and finally obtain better intelligent shopping results. Some states inside the basket may include:

[0053] 1) Whether the shooting screen of the built-in camera of the intelligent shopping cart tablet is occluded.

[0054] 2) Whether the child seat structure of the vehicle body part of the intelligent shopping cart is opened.

[0055] 3) Whether there are children in the child seat part of the shopping cart.

[0056] 4) Whether there are stacked goods in the basket area of the intelligent shopping cart that occlude the shooting screen of the camera.

[0057] 5) The content (capacity) of the goods inside the intelligent shopping cart.

[0058] 6) Information such as the edge position of the basket of the intelligent shopping cart.

[0059] The perception and judgment of the above several states in the embodiment of the present invention will have an auxiliary adjustment effect on the intelligent settlement algorithm and intelligent anti-theft algorithm built in the intelligent shopping cart. That is, the embodiment of the present invention can adjust different strategies according to different situations inside the basket, or give prompts to shoppers. That is, the embodiment of the present invention can endow the intelligent shopping device with the ability to perceive the state inside the basket and assist other algorithms inside the intelligent shopping cart to perform functions such as intelligent settlement and intelligent anti-abnormal shopping behavior, that is, the different states detected inside the intelligent shopping cart are used to adjust different strategies of the intelligent shopping cart (such as intelligent settlement or intelligent anti-abnormal shopping behavior, etc.). At the same time, the embodiment of the present invention perceives and discovers some states inside the basket of the intelligent shopping cart during the use process by integrating deep learning, machine learning, and computer vision algorithms.

[0060] In terms of hardware, the method for monitoring the state inside the shopping cart provided by the embodiment of the present invention may include:

[0061] 1) Visual sensor (image acquisition device): At least one camera with a large viewing angle, and the large viewing angle includes but is not limited to a large field of view angle, a fisheye camera, a panoramic camera, a stereo camera, a camera mounted at a high position, etc. The coverage range includes the area inside the shopping cart basket and a certain distance outside the shopping cart basket.

[0062] 2) Operation unit (a device for executing the method for monitoring the state inside the shopping cart provided by the embodiments of the present invention, that is, the device for monitoring the state inside the shopping cart described below): Receive the image data transmitted by the visual sensor, that is, obtain the video frame image data within the range of the shopping cart basket area collected by the image acquisition device at a first preset time interval, and call the algorithm built into the device (the method for monitoring the state inside the shopping cart provided by the embodiments of the present invention) for inference operation to obtain the state inside the shopping cart.

[0063] In terms of software, the method for monitoring the state inside the shopping cart provided by the embodiments of the present invention may include:

[0064] The embodiments of the present invention use a large-view camera to regularly obtain video frame data during the shopping process, which is processed by computer vision algorithms, including the judgment of intentional occlusion, the judgment of unintentional occlusion, the calculation of the capacity of goods inside the vehicle, the detection of the edge and state of the shopping cart, etc.

[0065] The general method for monitoring the state inside the shopping cart provided by the embodiments of the present invention is as follows:

[0066] 1. Obtain the picture information within the camera's field of view at intervals through the visual sensor;

[0067] 2. Use digital image processing algorithms to judge whether there is intentional occlusion according to the visual image;

[0068] 3. Use an identification model to judge whether there is unintentional occlusion according to the visual image;

[0069] 4. Use an identification model to judge the capacity of goods inside the vehicle according to the visual image;

[0070] 5. Use a detection and segmentation model to detect and delimit the boundary of the basket according to the visual image.

[0071] The specific process of the method for monitoring the state inside the shopping cart provided by the embodiments of the present invention is as follows:

[0072] 1. The visual sensor obtains visual image data at intervals, that is, step 10 above.

[0073] According to the set time interval value of low-frequency photographing, that is, the first preset time interval, for example, 5 seconds, obtain a visual image.

[0074] 2. Determine whether there is intentional occlusion based on the visual image, i.e., step 20 above.

[0075] For the case of intentional occlusion: Use the hand or other objects to directly cover the camera of the visual camera. The methods and steps for judging occlusion are as follows:

[0076] 1) First, for a single obtained visual image, perform digital graphics processing to detect the edge texture in the image, i.e., step 201 above. Among them, the edge texture detection algorithm uses the edge detection algorithm in digital graphics, including but not limited to the Canny edge detection algorithm, the edge detection algorithm of the Sobel operator, etc. These edge detection algorithms do not need to process the morphological information of the edge, but are only used to calculate and count the number of edge pixels.

[0077] 2) Count the number of edge textures, i.e., step 202 above.

[0078] 3) Make a judgment by comparing with a set corresponding threshold. When the number of edge textures in the picture is less than the threshold number, it is judged that the visual sensor may be intentionally occluded, i.e., step 203 above.

[0079] 4) If the situation in 3) occurs, further perform a secondary occlusion confirmation, i.e., step 204 above:

[0080] a) Activate the visual sensor to shorten the photographing interval time to the second preset time interval, such as shortening it to 16.67 milliseconds or 33.3 milliseconds, to reach the preset acquisition frequency, such as a shooting frequency of 60fps or 30fps, and perform high-frequency shooting, i.e., step 2041 above; that is, in one embodiment, the value range of the preset acquisition frequency can be 30fps - 120fps, preferably 60fps or 30fps, and the value of the second preset time interval can be the shortest shooting duration of the camera: 1 second / preset acquisition frequency, that is, the value range of the second preset time interval can be 8.33 milliseconds - 33.33 milliseconds, preferably 16.67 milliseconds or 33.3 milliseconds, which can further improve the accuracy of monitoring the intentionally occluded state.

[0081] b) Perform the judgments of 1), 2), and 3) on each frame of the image, that is, perform the above steps of texture information detection, texture quantity determination, and intentional occlusion judgment on each frame of the image collected at the preset acquisition frequency in sequence, i.e., step 2042 above.

[0082] c) When the cumulative number of frames with intentional occlusion reaches a certain threshold (preset frame number threshold, such as 30) within a preset number of seconds, such as within 1 second, there is a final result that the intelligent shopping cart may be in an intentionally occluded state, such as determining that the camera is intentionally occluded, i.e., step 2043 above.

[0083] 5) If it is determined in 4) that there is intentional occlusion, a signal of "there is intentional occlusion" is reported to the system, and at the same time, the shooting frequency of the vision camera is restored to the low frequency (for example, 5 seconds).

[0084] 3. Judge whether there is unintentional occlusion according to the visual image frame

[0085] If the situation of "intentional occlusion" is not judged in step 2, then start to judge whether there is the situation of "unintentional occlusion".

[0086] For the situation of unintentional occlusion: such as the child seat is opened, a child sits on the seat board and blocks the vision camera of the hardware device, and stacked goods block the vision camera of the hardware device. The methods and steps of judgment are as follows:

[0087] 1) First, for a single visual image obtained, it is sent into the "shopping cart state recognition model" to identify and judge whether it belongs to the situation of unintentional occlusion. Among them, the "shopping cart state recognition model" is a classification model, built based on a deep neural network model and trained by an image data set that has been manually screened and classified. This data set is defined to contain 7 categories, including "the seat board is opened", "a child sitting on the seat board blocks", "stacked goods block", "the basket is empty", "a small amount of goods in the basket (the goods in the shopping cart stack 0% to 30% of the basket capacity)", "a medium amount of goods in the basket (the goods in the shopping cart stack 30% - 60% of the basket capacity)", "a full amount of goods in the basket (the goods in the shopping cart stack to 60% - 100% of the shopping cart capacity)". The input of this shopping cart state recognition model can be historical video frame image data, and the output can include the unintentional occlusion state.

[0088] 2) If it is judged in step 1) that it is one of the three categories of unintentional occlusion types, such as "the seat board is opened", "a child sitting on the seat board blocks", "stacked goods block", and the confidence level (in the interval [0.6, 1.0]) reaches a certain threshold, then continue to judge the video frame images taken in the subsequent n times. When at least a certain proportion (such as 90%) of the video frame images are judged to be one of the three categories of unintentional occlusion types, such as "the seat board is opened", "a child sitting on the seat board blocks", "stacked goods block", it is considered that there is an "unintentional occlusion situation" in the basket at this time, and a signal of "there is unintentional occlusion" is reported to the system at this time (at this time, the video picture is blocked and the subsequent anti-abnormal shopping behavior algorithm logic judgment cannot be carried out, etc.).

[0089] As can be seen from the above, in one embodiment, such as Figure 2As shown, the above method for monitoring the status inside the shopping cart may further include the following step 30: For each frame of image data, perform the following operation to detect whether there is an unintentional occlusion state inside the shopping cart:

[0090] Step 301: Input a single-frame image into the shopping cart status recognition model to identify the result of whether it belongs to the unintentional occlusion state; the shopping cart status recognition model is pre-trained and generated according to the relationship samples between historical video frame image data and the unintentional occlusion state;

[0091] Step 302: If it is determined that the result is one of the unintentional occlusion types and the confidence level reaches the preset confidence threshold, continue to use the shopping cart status recognition model to identify each subsequent single-frame image. When at least a preset proportional number of single-frame images are all identified as one of the unintentional occlusion types, it is determined that there is an unintentional occlusion state inside the shopping cart at this time.

[0092] In specific implementation, the above implementation manner of detecting whether there is an unintentional occlusion state inside the shopping cart can improve the accuracy of detecting the unintentional occlusion state.

[0093] 4. Judge the commodity capacity situation inside the vehicle according to the visual image screen

[0094] If the situation of "intentional occlusion" is not judged in step 2 and the situation of "unintentional occlusion" is not judged in step 3, the capacity of the commodities in the basket at this moment can be reported to the system according to the set time interval value of low-frequency photographing (subsequent algorithms for preventing abnormal shopping behaviors can adjust the algorithm strategies according to the capacity of the commodities in the shopping cart basket). The judgment method is as follows:

[0095] 1) First, for the single visual image obtained, send it into the "shopping cart status recognition model" to identify and judge the status of the commodity capacity in the basket at this moment. The input of this shopping cart status recognition model can be historical video frame image data, and the output can include the commodity capacity status, as detailed in the introduction in the above "3. Judge whether there is an unintentional occlusion according to the visual image screen": "the basket is empty", "a small amount of commodities in the basket (the commodities in the shopping cart accumulate 0% to 30% of the basket capacity)", "a medium amount of commodities in the basket (the commodities in the shopping cart accumulate 30% - 60% of the basket capacity)", "a full amount of commodities in the basket (the commodities in the shopping cart accumulate to 60% - 100% of the shopping cart capacity)".

[0096] 2) Continuously identify and judge the video frame images taken n times, that is, repeat step 1) n times.

[0097] 3) For the recognition and judgment results of n times, sort the number of times of the results of the 4 possible judgments in the results, namely "empty basket", "basket with a small amount of goods", "basket with a medium amount of goods", and "basket full of goods", and determine the situation with the largest number of times as the current commodity capacity situation in the basket.

[0098] 4) Report the recognition and judgment results to the system (intelligent shopping cart system, which has intelligent shopping strategies inside, such as intelligent settlement or intelligent prevention of abnormal shopping behavior, etc.).

[0099] As can be seen from the above, in one embodiment, as Figure 3 shown, the above method for monitoring the status inside the shopping cart may further include the following step 40: when it is detected that there is no intentional occlusion state inside the shopping cart, for each frame of image data, perform the following operation to detect the commodity capacity status inside the shopping cart:

[0100] Step 401: Input each single-frame image into the shopping cart capacity recognition model to recognize the commodity capacity type inside the shopping cart for each single-frame image; the shopping cart capacity recognition model is pre-trained and generated according to the relationship samples between historical video frame image data and the status of commodity capacity.

[0101] Step 402: Sort the number of occurrences of the commodity capacity types inside the shopping cart for all single-frame images.

[0102] Step 403: Use the commodity capacity type with the largest number of occurrences as the current commodity capacity status inside the shopping cart.

[0103] Specifically in implementation, the above implementation method for detecting the status of the commodity capacity inside the shopping cart can improve the accuracy of detecting the status of the commodity capacity inside the shopping cart.

[0104] 5. Detect and delimit the basket boundary according to the visual image frame

[0105] If the situation of "intentional occlusion" is not judged in step 2 and the situation of "unintentional occlusion" is not judged in step 3, the boundary information of the basket in the current frame can be reported to the system according to the set time interval value of low-frequency photographing. The judgment method is:

[0106] 1) First, for the single visual image obtained, it is fed into the "basket boundary detection / segmentation model (basket boundary detection or segmentation model)", and several boundary coordinate points including the basket boundary or the basket area mask are output. Among them, the "basket boundary detection / segmentation model" is built using a deep neural network model and trained with a large number of image training sets of different vehicle models and baskets of different colors. The task of model training is defined as object detection or image segmentation. If the task is defined as object detection, then the training objective is to make the model output at least 6 coordinate points of the accurate basket boundary; if the task is defined as image segmentation, then the training objective is to make the model output the accurate basket area mask. The schematic diagram is as Figure 5 shown, Figure 5 This is a schematic diagram of the output result of the basket boundary detection or segmentation model in the embodiment of the present invention. Figure 5 In it, the 6 blue points are the basket boundary coordinate points, and when connected together, they can enclose the basket area within the red area; Figure 5 The green area part in it is a schematic diagram of the basket area mask output by the segmentation model.

[0107] 2) In order to obtain more accurate basket boundary coordinate points or basket area mask, the inference calculation is continuously performed on the video frame images taken n times, that is, step 1) is repeated n times.

[0108] 3) For the recognition and judgment results of n times, fusion is performed. If the output is the basket boundary coordinate points, the final result takes the average value of each coordinate point after removing the maximum and minimum values in the n times of results. If the output is the basket area mask, the area where the n times of results overlap and the overlap times is n is taken as the final result after the n times of results overlap.

[0109] 4) After the calculation and determination are completed, the information of the basket boundary is reported to the system.

[0110] As can be seen from the above, in one embodiment, as Figure 4 shown, the above method for monitoring the state inside the shopping cart may further include the following step 50: when it is detected that there is no intentional occlusion state inside the shopping cart, for each frame of image data, perform the following operation to detect the state of the shopping cart basket boundary:

[0111] Step 501: Input each single-frame image into the basket boundary detection or segmentation model, and output several boundary coordinate points including the basket boundary or the basket area mask; the basket boundary detection or segmentation model is pre-trained and generated according to the relationship samples of historical video frame image data and several boundary coordinate points including the basket boundary or the basket area mask;

[0112] Step 502: Fuse the output results corresponding to all single-frame images to obtain the state of the shopping cart basket boundary.

[0113] In specific implementation, the above implementation manner for detecting the state of the shopping cart basket boundary can improve the accuracy of detecting the state of the shopping cart basket boundary.

[0114] As can be seen from the above, in one embodiment, fusing the output results corresponding to all single-frame images to obtain the state of the shopping cart basket boundary includes:

[0115] If the output is several boundary coordinate points, for the coordinate values of the same coordinate point in all single-frame images, the average value after removing the maximum and minimum values is used as the final coordinate value of the coordinate point, and the final coordinate values of all coordinate points are used as the state of the final shopping cart basket boundary;

[0116] If the output is a basket area mask, after overlapping the basket area masks output by all single-frame images, the boundary of the area with an overlapping count of n is used as the state of the final intelligent shopping cart basket boundary, where n is the number of single-frame images included in the video frame image.

[0117] In specific implementation, the above implementation manner for fusing the output results corresponding to all single-frame images can improve the accuracy of monitoring the state of the shopping cart basket boundary.

[0118] As can be seen from the above, in one embodiment, the training task of the basket boundary detection or segmentation model is set as an object detection task or an image segmentation task; the method for monitoring the state inside the shopping cart further includes: if it is set as an object detection task, the training objective is to make the model output at least six coordinate points of the basket boundary that meet the first preset accuracy threshold; if it is set as an image segmentation task, the training objective is to make the model output a basket area mask that meets the second preset accuracy threshold.

[0119] In specific implementation, the above implementation manner for training the basket boundary detection or segmentation model can improve the recognition accuracy of the basket boundary detection or segmentation model.

[0120] In summary, the method for monitoring the state inside the shopping cart provided by the embodiments of the present invention achieves:

[0121] 1) Using only a visual sensor and a visual algorithm, it is possible to judge and calculate various complex states inside the shopping cart basket.

[0122] 2) Based on computer vision technology, using deep learning, machine learning, and artificial intelligence algorithms, it is possible to analyze sensor data in real time and perceive the shopping cart environment.

[0123] 3) Prioritize various complex states inside the shopping cart basket and design the overall algorithm according to the logic of "if there is occlusion, do not calculate further" (the meaning of prioritization is: occlusion has the highest priority, and intentional occlusion has the highest priority. When perceiving the state inside the cart, first judge the occlusion situation (first judge intentional occlusion, then non-intentional occlusion), and then judge the commodity capacity inside the cart, the boundary situation of the cart basket, etc.).

[0124] 4) For the case of "intentional occlusion", it can be judged by using efficient digital image processing calculations and the logic of effectively shooting high and low frequency images, without using complex deep learning algorithms, so as to reduce the development cost and operation power consumption.

[0125] 5) For "non-intentional occlusion" and "commodity capacity inside the cart basket", innovatively use the method of classification models, combined with strategies of multiple fusions, sorting, or taking the maximum value, to convert a single judgment problem into a more accurate decision-making problem, and better judgment results can be obtained.

[0126] In summary, the beneficial technical effects of the method for monitoring the state inside the shopping cart provided by the embodiments of the present invention are as follows:

[0127] 1) The embodiments of the present invention provide a method for monitoring the state inside the shopping cart, which endows the intelligent shopping device with the ability to perceive the state inside the cart basket. After obtaining the state inside the cart basket, the intelligent settlement algorithm and intelligent anti-theft algorithm of the intelligent shopping device can adjust different strategies according to different situations inside the cart basket to obtain better judgment results.

[0128] 2) The cart basket perception method provided by the embodiments of the present invention can seamlessly deploy the same shopping cart tablet to different ordinary shopping carts, upgrade the ordinary shopping cart to an intelligent shopping cart, that is, directly install the device for monitoring the state inside the shopping cart and the acquisition device in the embodiments of the present invention on the intelligent shopping cart, and most of the site survey processes are reduced.

[0129] The embodiments of the present invention also provide a device for monitoring the state inside the shopping cart, as described in the following embodiments. Since the principle of solving problems by this device is similar to that of the method for monitoring the state inside the shopping cart, the implementation of this device can refer to the implementation of the method for monitoring the state inside the shopping cart, and the repeated parts will not be elaborated.

[0130] Figure 6 It is a schematic structural diagram of the device for monitoring the state inside the shopping cart in the embodiments of the present invention, as Figure 6 shown. The device includes:

[0131] An acquisition unit 01, configured to acquire video frame image data within the range of the shopping cart basket area collected at a first preset time interval;

[0132] The status monitoring unit 02 is configured to perform the following operation for each frame of image data to detect whether there is a deliberate occlusion state in the shopping cart:

[0133] Steps of texture information detection: For each frame of image data obtained, perform digital graphics processing to detect the edge texture information in the image;

[0134] Steps of determining the texture quantity: Determine the edge texture quantity according to the edge texture information in the image;

[0135] Steps of deliberate occlusion judgment: When the edge texture quantity in the image is less than the preset edge texture quantity threshold, determine a preliminary result that there may be a deliberately occluded state in the shopping cart;

[0136] When determining the preliminary result that there may be a deliberately occluded state in the shopping cart, perform the following operation of secondary confirmation of deliberate occlusion on the preliminary result:

[0137] Control to shorten the first preset time interval to the second preset time interval to reach the preset acquisition frequency, and acquire video frame image data within the range of the shopping cart basket area;

[0138] Perform the above steps of texture information detection, steps of determining the texture quantity, and steps of deliberate occlusion judgment on each frame of image acquired at the preset acquisition frequency in sequence;

[0139] When the number of frames with deliberate occlusion accumulated within the preset number of seconds reaches the preset frame number threshold, determine the final result that there may be a deliberately occluded state in the shopping cart.

[0140] In one embodiment, the above device for monitoring the state inside the shopping cart may further include a non-deliberate occlusion state detection unit, which is configured to: For each frame of image data, perform the following operation to detect whether there is a non-deliberate occlusion state in the shopping cart:

[0141] Input a single-frame image into the shopping cart state recognition model to identify the result of whether it belongs to a non-deliberate occlusion state; the shopping cart state recognition model is pre-trained according to the relationship samples between historical video frame image data and non-deliberate occlusion states;

[0142] If the determined result is one of the non-deliberate occlusion types and the confidence level reaches the preset confidence level threshold, continue to use the shopping cart state recognition model to identify each subsequent single-frame image. When at least the preset proportional quantity of single-frame images are all identified as one of the non-deliberate occlusion types, determine that there is a non-deliberate occlusion state in the shopping cart at this time.

[0143] In one embodiment, the device for monitoring the status inside the shopping cart may further include a capacity status detection unit, which is configured to: when it is detected that there is no intentional occlusion state inside the shopping cart, for each frame of image data, perform the following operations to detect the commodity capacity status inside the shopping cart:

[0144] Input each single-frame image into the shopping cart capacity recognition model to identify the commodity capacity type of each single-frame image; the shopping cart capacity recognition model is pre-trained and generated according to the relationship samples between historical video frame image data and the status of commodity capacity;

[0145] Sort the occurrence times of the commodity capacity types of all single-frame images;

[0146] Take the commodity capacity type with the most occurrence times as the status of the commodity capacity inside the current shopping cart.

[0147] In one embodiment, the device for monitoring the status inside the shopping cart may further include a boundary status detection unit, which is configured to: when it is detected that there is no intentional occlusion state inside the shopping cart, for each frame of image data, perform the following operations to detect the status of the shopping cart basket boundary:

[0148] Input each single-frame image into the basket boundary detection or segmentation model, and output a number of boundary coordinate points including the basket boundary, or a basket area mask; the basket boundary detection or segmentation model is pre-trained and generated according to the relationship samples between historical video frame image data and a number of boundary coordinate points including the basket boundary, or a basket area mask;

[0149] Fuse the output results corresponding to all single-frame images to obtain the status of the shopping cart basket boundary.

[0150] In one embodiment, fusing the output results corresponding to all single-frame images to obtain the status of the shopping cart basket boundary includes:

[0151] If the output is a number of boundary coordinate points, for the coordinate values of the same coordinate point in all single-frame images, take the average value after removing the maximum and minimum values as the final coordinate value of this coordinate point, and take the final coordinate values of all coordinate points as the final status of the shopping cart basket boundary;

[0152] If the output is a basket area mask, after overlapping the basket area masks output by all single-frame images, take the area boundary with the overlapping times of n as the final status of the shopping cart basket boundary, where n is the number of single-frame images included in the video frame image.

[0153] In one embodiment, the training task of the shopping basket boundary detection or segmentation model is set as an object detection task or an image segmentation task; the method for monitoring the state inside the shopping cart further includes: if it is set as an object detection task, the training objective is to make the model output at least six coordinate points of the shopping basket boundary that meet the first preset accuracy threshold; if it is set as an image segmentation task, the training objective is to make the model output a shopping basket area mask that meets the second preset accuracy threshold.

[0154] Based on the foregoing inventive concept, Figure 7 The following is a schematic structural diagram of a computer device according to an embodiment of the present invention. As Figure 7 shown, the present invention also provides a computer device 600, including a memory 610, a processor 620, and a computer program 630 stored on the memory 610 and executable on the processor 620. When the processor 620 executes the computer program 630, the foregoing method for monitoring the state inside the shopping cart is implemented.

[0155] An embodiment of the present invention also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the foregoing method for monitoring the state inside the shopping cart is implemented.

[0156] An embodiment of the present invention also provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the foregoing method for monitoring the state inside the shopping cart is implemented.

[0157] In an embodiment of the present invention, a solution for monitoring the state inside a shopping cart includes: obtaining video frame image data within the range of the shopping cart basket area collected at a first preset time interval; for each frame of image data, performing the following operation to detect whether there is a deliberate occlusion state inside the shopping cart: a texture information detection step: for each frame of image data obtained, performing digital graphics processing to detect the edge texture information in the image; a texture quantity determination step: determining the quantity of edge textures according to the edge texture information in the image; a deliberate occlusion judgment step: when the quantity of edge textures in the image is less than a preset edge texture quantity threshold, determining a preliminary result that there may be a deliberately occluded state inside the shopping cart; when determining the preliminary result that there may be a deliberately occluded state inside the shopping cart, performing the following operation for secondary confirmation of deliberate occlusion on the preliminary result: controlling to shorten the first preset time interval to a second preset time interval to reach a preset acquisition frequency, and collecting video frame image data within the range of the shopping cart basket area; sequentially performing the above texture information detection step, texture quantity determination step, and deliberate occlusion judgment step on each frame collected at the preset acquisition frequency; when the number of frames with deliberate occlusion occurring within a preset number of seconds reaches a preset frame number threshold, determining a final result that there may be a deliberately occluded state inside the shopping cart. The embodiment of the present invention can monitor the state inside the shopping cart in real time.

[0158] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0159] The present invention is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate means for implementing the functions specified in one Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0160] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to operate in a particular manner, such that the instructions stored in the computer-readable memory produce a manufacture including instruction means embodying the functionality specified in the flowchart(s) Figure 1 or block(s) of one or more flowcharts and / or Figure 1 block(s).

[0161] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable apparatus provide steps for implementing the functionality specified in the flowchart(s) Figure 1 or block(s) of one or more flowcharts and / or Figure 1 block(s).

[0162] The specific embodiments described above further elaborate on the objectives, technical solutions, and beneficial effects of the present invention. It should be understood that the above are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A method for monitoring the status inside a shopping cart, characterized in that: include: Acquire video frame image data within the basket area of ​​the shopping cart collected at a first preset time interval; For each frame of image data, perform the following operations to detect whether there is intentional occlusion in the shopping cart: Texture information detection steps: for each frame of image data obtained, digital graphics processing is performed to detect edge texture information in the image; Steps for determining the number of textures: determining the number of edge textures according to edge texture information in the image; Deliberate occlusion judgment step: when the number of edge textures in the image is less than a preset edge texture number threshold, a preliminary result is determined that there is a possible intentional occlusion state in the shopping cart; When it is determined that there are preliminary results in the shopping cart that may be intentionally blocked, the preliminary results are subjected to the following intentionally blocked secondary confirmation operation: Controlling to shorten the first preset time interval to a second preset time interval to achieve a preset acquisition frequency, and to acquire video frame image data within the basket area of ​​the shopping cart; Each frame of the image acquired at the preset acquisition frequency is sequentially subjected to the above steps of texture information detection, texture quantity determination and intentional occlusion determination; When the number of frames in which intentional occlusion occurs cumulatively within the preset number of seconds reaches a preset frame number threshold, it is determined that there is a final result that the shopping cart may be in an intentional occlusion state.

2. The method according to claim 1, characterized in that The preset acquisition frequency has a value range of 30fps-120fps, and the second preset time interval has a value of the shortest shooting time of the camera: 1 second / preset acquisition frequency.

3. The method according to claim 1, characterized in that Also includes: For each frame of image data, perform the following operations to detect whether there is any unintentional occlusion in the shopping cart: Inputting a single frame image into a shopping cart state recognition model to identify whether it is an unintentional occlusion state; the shopping cart state recognition model is pre-trained and generated based on a sample of the relationship between historical video frame image data and unintentional occlusion states; If the result is determined to be one of the unintentional occlusion types and the confidence reaches a preset confidence threshold, continue to use the shopping cart state recognition model to identify each subsequent single-frame image. When at least a preset proportion of single-frame images are identified as one of the unintentional occlusion types, it is determined that an unintentional occlusion state exists in the shopping cart at this time.

4. The method according to claim 1, characterized in that Also includes: When it is detected that there is no intentional occlusion in the shopping cart, the following operation of detecting the capacity of the goods in the shopping cart is performed for each frame of image data: Input each single frame image into a shopping cart capacity recognition model to identify the type of commodity capacity in the shopping cart of each single frame image; the shopping cart capacity recognition model is pre-trained and generated based on a sample of the relationship between historical video frame image data and the state of commodity capacity; Sort the occurrence counts of the product capacity types in the shopping carts of all single-frame images; The product capacity type that appears most frequently is used as the status of the product capacity in the current shopping cart.

5. The method according to claim 1, characterized in that Also includes: When it is detected that there is no intentional occlusion state in the shopping cart, for each frame of image data, the following operation of detecting the state of the shopping cart basket boundary is performed: Input each single frame image into a vehicle basket boundary detection or segmentation model, and output a number of boundary coordinate points containing the vehicle basket boundary, or a vehicle basket area mask; the vehicle basket boundary detection or segmentation model is pre-trained and generated based on a relationship sample between historical video frame image data and a number of boundary coordinate points containing the vehicle basket boundary, or a vehicle basket area mask; The output results corresponding to all single-frame images are fused to obtain the state of the shopping cart basket boundary.

6. The method according to claim 5, characterized in that The output results corresponding to all single-frame images are fused to obtain the state of the shopping cart basket boundary, including: If the output is a number of boundary coordinate points, for the coordinate values ​​of the same coordinate point in all single-frame images, the average value after removing the maximum and minimum values ​​is used as the final coordinate value of the coordinate point, and the final coordinate values ​​of all coordinate points are used as the final state of the shopping cart basket boundary; If the output is a basket area mask, after overlapping the basket area masks output by all single-frame images, the region boundary with the number of overlaps n is taken as the final state of the shopping cart basket boundary, where n is the number of single-frame images contained in the video frame image.

7. The method according to claim 5, characterized in that The training task of the basket boundary detection or segmentation model is defined as a target detection task or an image segmentation task; the method for monitoring the interior status of a shopping cart also includes: if it is defined as a target detection task, the training goal is to allow the model to output at least six coordinate points of the basket boundary that meet a first preset accuracy threshold; if it is defined as an image segmentation task, the training goal is to allow the model to output a basket area mask that meets a second preset accuracy threshold.

8. A device for monitoring the status inside a shopping cart, characterized in that: include: An acquisition unit, configured to acquire video frame image data within a shopping cart basket area collected at a first preset time interval; The state monitoring unit is used to perform the following operations to detect whether there is an intentional occlusion state in the shopping cart for each frame of image data: Texture information detection steps: for each frame of image data obtained, digital graphics processing is performed to detect edge texture information in the image; Steps for determining the number of textures: determining the number of edge textures according to edge texture information in the image; Deliberate occlusion judgment step: when the number of edge textures in the image is less than a preset edge texture number threshold, a preliminary result is determined that there is a possible intentional occlusion state in the shopping cart; When it is determined that there are preliminary results in the shopping cart that may be intentionally blocked, the preliminary results are subjected to the following intentionally blocked secondary confirmation operation: Controlling to shorten the first preset time interval to a second preset time interval to achieve a preset acquisition frequency, and to acquire video frame image data within the basket area of ​​the shopping cart; Each frame of the image acquired at the preset acquisition frequency is sequentially subjected to the above steps of texture information detection, texture quantity determination and intentional occlusion determination; When the number of frames in which intentional occlusion occurs cumulatively within the preset number of seconds reaches a preset frame number threshold, it is determined that there is a final result that the shopping cart may be in an intentional occlusion state.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

11. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images