Shopping cart overflow rate detection method and device based on depth image

Abstract

The embodiment of the invention provides a shopping cart overflow rate detection method and device based on a depth image, and the method comprises the steps: obtaining a sample depth image and a sample grayscale image, carrying out the filtering, carrying out the splicing, and carrying out the training of a shopping cart target detection model according to the spliced image; obtaining a depth image to be identified and a gray level image to be identified; splicing the to-be-recognized depth image and the to-be-recognized grayscale image to obtain a to-be-recognized spliced image; inputting a to-be-identified spliced image into the trained detection model, and detecting to obtain the bottom height, the highest height and the cargo height of the shopping cart; the ratio of the first difference value of the cargo height and the bottom height to the second difference value of the highest height and the bottom height is calculated, the shopping cart overflow rate is obtained, due to the fact that no image is formed in the depth map when the cart is empty, the recognition effect of the grey-scale map is optimal when the cart is empty, and the problem of how to recognize the shopping cart overflow rate can be solved through the fused image.

Description

Technical Field

[0001] This application relates to the field of image recognition, and in particular to a method and apparatus for detecting shopping cart overflow rate based on depth images. Background Technology

[0002] With the rapid development of image recognition technology, its applications in daily life have become increasingly widespread. Image recognition technology is also used to capture images of shopping carts and identify their overflow rate, thus facilitating the optimization of the shopping experience. However, the inventors have found that most current methods for detecting shopping carts using RGB images often have poor recognition results and cannot quantify the percentage of cart overflow. Summary of the Invention

[0003] The purpose of this application is to provide a method and apparatus for detecting shopping cart overflow rate based on depth images, so as to solve the problem of how to identify the overflow rate of shopping carts. The specific technical solution is as follows: A first aspect of this application provides a shopping cart overflow rate detection method based on depth images, comprising: Acquire a sample depth image and a sample grayscale image, wherein the sample depth image and the sample grayscale image include a shopping cart image; identify the installation height based on the sample depth image, wherein the installation height is the installation height of the device that acquired the sample depth image; calculate the difference between the installation height and a preset human height to obtain a filtering threshold; filter the sample depth image according to the filtering threshold to obtain a filtered image; The filtered image and the sample grayscale image are concatenated to obtain a concatenated image; the shopping cart object detection model is trained based on the concatenated image to obtain the trained detection model; Acquire a depth image and a grayscale image to be identified; stitch the depth image and the grayscale image to be identified to obtain a stitched image to be identified; input the stitched image to be identified into the trained detection model to detect the bottom height, top height, and item height of the shopping cart; calculate the ratio of the first difference between the item height and the bottom height to the second difference between the top height and the bottom height to obtain the shopping cart overflow rate.

[0004] In one possible implementation, the identification of installation height based on the sample depth image includes: The installation height is obtained by identifying the average height of the central region of the sample depth image; The step of filtering the sample depth image according to the filtering threshold to obtain the filtered image includes: Based on the filtering threshold, images in the sample depth image that are higher than a preset human height are filtered out to obtain the filtered image.

[0005] In one possible implementation, the step of concatenating the filtered image and the sample grayscale image to obtain the concatenated image includes: The filtered image and the sample grayscale image are then stitched together vertically to obtain the stitched image. The step of training a shopping cart object detection model based on the stitched image to obtain a trained detection model includes: The shopping carts in the stitched image are labeled as a first category and a second category, respectively, wherein the first category is the shopping carts in the filtered image and the second category is the shopping carts in the sample grayscale image; The stitched image is input into the shopping cart target detection model to obtain the shopping cart detection result, wherein the shopping cart detection result is detected from the filtered image and / or from the sample grayscale image; the shopping cart target detection model is corrected according to the shopping cart detection result, and training continues until a preset iteration stopping condition is met to obtain the trained detection model.

[0006] In one possible implementation, the step of inputting the stitched image to be identified into the trained detection model to detect the bottom height, top height, and item height of the shopping cart includes: The stitched image to be identified is input into the trained detection model to detect the bottom height and top height of the shopping cart; The cargo height is obtained by using a tracking algorithm to identify the average depth of the central region of multiple depth images to be identified.

[0007] In one possible implementation, before calculating the ratio of the first difference between the height of the goods and the bottom height, and the second difference between the maximum height and the bottom height to obtain the shopping cart overflow rate, the method further includes: When the shopping cart is empty, the height of the goods is set to be the same as the bottom height; When the shopping cart is full, the height of the goods is set to be the same as the maximum height.

[0008] A second aspect of this application provides a shopping cart overflow rate detection device based on depth images, comprising: An image acquisition module is used to acquire a sample depth image and a sample grayscale image, wherein the sample depth image and the sample grayscale image include a shopping cart image; to identify the installation height based on the sample depth image, wherein the installation height is the installation height of the device acquiring the sample depth image; to calculate the difference between the installation height and a preset human height to obtain a filtering threshold; and to filter the sample depth image according to the filtering threshold to obtain a filtered image. The model training module is used to stitch the filtered image and the sample grayscale image together to obtain a stitched image; and to train a shopping cart object detection model based on the stitched image to obtain a trained detection model. The overflow recognition module is used to acquire a depth image and a grayscale image to be recognized; to stitch the depth image and the grayscale image to be recognized to obtain a stitched image to be recognized; to input the stitched image to be recognized into the trained detection model to detect the bottom height, top height and item height of the shopping cart; and to calculate the ratio of the first difference between the item height and the bottom height to the second difference between the top height and the bottom height to obtain the shopping cart overflow rate.

[0009] In one possible implementation, the image acquisition module is specifically used to identify the average height of the central region of the sample depth image to obtain the installation height; and to filter out images in the sample depth image that are higher than a preset human height according to the filtering threshold to obtain the filtered image.

[0010] In one possible implementation, the model training module is specifically used to concatenate the filtered image and the sample grayscale image vertically to obtain the concatenated image; to label the shopping carts in the concatenated image as a first category and a second category, wherein the first category is the shopping carts in the filtered image and the second category is the shopping carts in the sample grayscale image; to input the concatenated image into a shopping cart target detection model to obtain shopping cart detection results, wherein the shopping cart detection results are detected from the filtered image and / or from the sample grayscale image; to correct the shopping cart target detection model based on the shopping cart detection results and continue training until a preset iteration stopping condition is met to obtain the trained detection model.

[0011] In one possible implementation, the overflow recognition module is specifically used to input the stitched image to be recognized into the trained detection model to detect the bottom height and top height of the shopping cart; and to obtain the height of the goods by recognizing the average depth of the central region of multiple depth images to be recognized through a tracking algorithm.

[0012] In one possible implementation, the device further includes: The height correction module is used to set the height of the goods to be the same as the bottom height when the shopping cart is empty, and to set the height of the goods to be the same as the maximum height when the shopping cart is full.

[0013] Another aspect of the application embodiments also provides an electronic device, including: Memory, used to store computer programs; When the processor executes the program stored in memory, it implements any of the above-mentioned depth image-based shopping cart overflow rate detection methods.

[0014] In another aspect of the embodiments of this application, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements any of the above-described depth image-based shopping cart overflow rate detection methods.

[0015] In another aspect of the embodiments of this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the above-described depth image-based shopping cart overflow rate detection methods.

[0016] Beneficial effects of the embodiments in this application: This application provides a method and apparatus for detecting shopping cart overflow rate based on depth images. The method includes: acquiring a sample depth image and a sample grayscale image, wherein the sample depth image and the sample grayscale image include a shopping cart image; identifying the installation height based on the sample depth image, wherein the installation height is the installation height of the device acquiring the sample depth image; calculating the difference between the installation height and a preset human height to obtain a filtering threshold; filtering the sample depth image according to the filtering threshold to obtain a filtered image; and comparing the filtered image with the sample depth image. Grayscale images are stitched together to obtain a stitched image; a shopping cart object detection model is trained based on the stitched image to obtain a trained detection model; a depth image and a grayscale image to be identified are acquired; the depth image and the grayscale image to be identified are stitched together to obtain a stitched image to be identified; the stitched image to be identified is input into the trained detection model to detect the bottom height, top height, and item height of the shopping cart; the ratio of the first difference between the item height and the bottom height to the second difference between the top height and the bottom height is calculated to obtain the shopping cart overflow rate. The solution in this application embodiment can acquire sample depth images and sample grayscale images, thereby training a model using the sample depth images and sample grayscale images. The trained model can then identify the bottom height, top height, and item height of the shopping cart, and calculate the ratio of the first difference between the item height and the bottom height and the second difference between the top height and the bottom height to obtain the shopping cart overflow rate. This achieves the calculation of the shopping cart overflow rate. Since the depth image is not imaged when the cart is empty, the grayscale image has the best recognition effect when the cart is empty. By fusing images, the problem of how to identify the shopping cart overflow rate can be solved.

[0017] Of course, implementing any product or method of this application does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0019] Figure 1 A schematic flowchart of a shopping cart overflow rate detection method based on depth images provided in an embodiment of this application; Figure 2 A schematic diagram of a spliced ​​image provided in an embodiment of this application; Figure 3 A schematic diagram of a shopping cart provided in an embodiment of this application; Figure 4 A schematic diagram of a shopping cart overflow rate detection method based on depth images provided in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0020] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of this application.

[0021] The first aspect of this application provides a shopping cart overflow rate detection method based on depth images, see [link to relevant documentation]. Figure 1 , Figure 1 A flowchart illustrating a depth image-based shopping cart overflow rate detection method provided in this application embodiment includes: Step S11: Acquire a sample depth image and a sample grayscale image, wherein the sample depth image and the sample grayscale image include a shopping cart image; identify the installation height based on the sample depth image, wherein the installation height is the installation height of the device that acquired the sample depth image; calculate the difference between the installation height and a preset human height to obtain a filtering threshold; filter the sample depth image based on the filtering threshold to obtain a filtered image; Step S12: The filtered image and the sample grayscale image are stitched together to obtain a stitched image; the shopping cart target detection model is trained based on the stitched image to obtain a trained detection model; Step S13: Obtain the depth image and grayscale image to be identified; stitch the depth image and grayscale image to be identified to obtain the stitched image to be identified; input the stitched image to be identified into the trained detection model to detect the bottom height, top height and cargo height of the shopping cart; calculate the ratio of the first difference between the cargo height and the bottom height to the second difference between the top height and the bottom height to obtain the shopping cart overflow rate.

[0022] Corresponding to step S11 above, the depth image in this embodiment is an image storing three-dimensional depth feature information. It obtains the distance information from each point in the scene to the camera using stereo vision technology, and is typically represented in grayscale form, with brightness reflecting distance. A grayscale image is an image where each pixel has only one sampled color. Such images are usually displayed as grayscale ranging from the darkest black to the brightest white, although theoretically this sampling can be any shade of color, even different colors at different brightness levels. In this embodiment, the camera acquiring the image can be mounted on top facing the ground. This camera can integrate a depth camera and a grayscale camera, acquiring images from top to bottom, thus simultaneously acquiring both depth and grayscale images. The sample depth image and sample grayscale image in this embodiment can be pre-acquired images, which may include a shopping cart, such as one used by customers in a supermarket or similar setting. When identifying the installation height based on the sample depth image, the installation height is the installation height of the device acquiring the sample depth image, which is also the camera's installation height. Then, the difference between the installation height and the preset human height is calculated to obtain the filtering threshold. The sample depth image is then filtered according to the filtering threshold to obtain the filtered image.

[0023] Corresponding to step S12 above, the filtered image and the sample grayscale image are stitched together to obtain a stitched image. This stitching can be done by stitching the filtered image and the sample grayscale image vertically or horizontally. Then, the shopping cart object detection model is trained based on the stitched image to obtain a trained detection model. In one example, the stitched image can be input into the shopping cart object detection model to obtain a shopping cart detection result; the shopping cart object detection model is then corrected based on the shopping cart detection result, and training continues until a preset iteration stopping condition is met to obtain the trained detection model.

[0024] Corresponding to step S13 above, the depth image and grayscale image to be identified are acquired. When the depth image and grayscale image to be identified are stitched together to obtain the stitched image to be identified, the stitching can be performed in the same way as described above. The stitched image to be identified is input into the trained detection model to detect the bottom height, top height, and item height of the shopping cart. Then, the ratio of the first difference between the item height and the bottom height to the second difference between the top height and the bottom height is calculated to obtain the shopping cart overflow rate. In one example, when detecting item height, multiple depth images and grayscale images to be identified can be acquired, and then the average item height can be calculated, thereby using this average value to calculate the overflow rate. In this application, a depth image and a grayscale image to be identified are acquired, and then the depth image and the grayscale image to be identified are stitched together to obtain a stitched image to be identified. Finally, the model is trained and identified based on the stitched image. This method can make full use of the advantages of depth images and grayscale images for identification. Since depth images are not formed when the shopping cart is empty, and grayscale images have the best recognition effect when the shopping cart is empty, the fusion of the two can avoid the problem of the shopping cart not being able to be identified when it is empty, and can identify the shopping cart overflow rate.

[0025] As can be seen, the method of this application embodiment can acquire sample depth images and sample grayscale images, thereby training the model using the sample depth images and sample grayscale images. The trained model can then identify the bottom height, top height, and item height of the shopping cart, and calculate the ratio of the first difference between the item height and the bottom height and the second difference between the top height and the bottom height to obtain the shopping cart overflow rate. Since the depth image is not imaged when the cart is empty, the grayscale image has the best recognition effect when the cart is empty. The problem of how to identify the shopping cart overflow rate can be solved by fusing the images.

[0026] In one possible implementation, the identification of the installation height based on the sample depth image includes: identifying the average height of the central region of the sample depth image to obtain the installation height; the filtering of the sample depth image according to the filtering threshold to obtain the filtered image includes: filtering out images in the sample depth image that are higher than a preset human height according to the filtering threshold to obtain the filtered image. Specifically, the solution of this application can perform adaptive height filtering. Specifically, identifying the average height of the central region of the sample depth image can be achieved by calculating the average value of a 20*20 area at the center point of the image as the installation height H. When filtering out images in the sample depth image that are higher than the preset human height according to the filtering threshold, data with depth values ​​greater than the installation height can be set as the installation height, and invalid depth values ​​can be removed. In one example, to address the issue of different colors for people at different installation heights, the height can be based on people. Considering that the average height of Chinese people is around 160cm-180cm, an average of 170cm can be taken, and the preset human height can be denoted as P. The height from the equipment to the ground where the person stands is the installation height. The filtering steps are as follows: First, calculate the average value of a 20x20 area around the center of the image as the installation height H (the center area of ​​the image is unobstructed, representing normal ground). Data with depth values ​​greater than the installation height are set as the installation height, and invalid depth values ​​are removed. Data with depth values ​​less than HP, specifically the depth value from the device to the top of the head, are set as HP, and invalid data from the top of the head is removed. The depth values ​​from the feet to the top of the head are mapped to 0-255 and rendered with color. This results in a stable human feature: red for the top of the head, blue for the feet, and light red for the shoulders (for adults). This stable human feature can be obtained at any height, significantly improving the accuracy of neural network object detection.

[0027] In one possible implementation, the step of stitching the filtered image and the sample grayscale image together to obtain a stitched image includes: stitching the filtered image and the sample grayscale image vertically together to obtain the stitched image; the step of training a shopping cart target detection model based on the stitched image to obtain a trained detection model includes: labeling shopping carts in the stitched image as a first class and a second class, wherein the first class is the shopping cart in the filtered image and the second class is the shopping cart in the sample grayscale image; inputting the stitched image into the shopping cart target detection model to obtain a shopping cart detection result, wherein the shopping cart detection result is detected from the filtered image and / or from the sample grayscale image; correcting the shopping cart target detection model based on the shopping cart detection result and continuing training until a preset iteration stopping condition is met to obtain the trained detection model.

[0028] In this embodiment, the filtered image and the sample grayscale image are stitched together vertically to obtain the stitched image. Shopping cart target detection primarily combines depth map and grayscale image detection. After the above filtering process, the depth map and grayscale image are stitched together vertically to form a single image. The shopping carts in the stitched image are labeled as a first category and a second category, where the first category refers to the shopping carts in the filtered image, and the second category refers to the shopping carts in the sample grayscale image. See the example provided. Figure 2 You can label the shopping cart categories in the image as category 0 (depth image) + category 1 (grayscale image) and then stitch them together. Figure 2 The top image is a depth image, and the bottom image is a grayscale image. Then, the stitched image is input into a shopping cart object detection model to obtain a shopping cart detection result. This shopping cart detection result is obtained from the filtered image and / or from the sample grayscale image. The shopping cart object detection model is then corrected based on the shopping cart detection result, and training continues until a preset iteration stopping condition is met to obtain the trained detection model. Specifically, the stitched image is used for model training. After training, the model infers, and during inference, detection is considered successful if either the depth image or the grayscale image is detected. This application's solution, by combining depth and grayscale images for shopping cart detection, can be applied to various scenarios: 1. When the grayscale image imaging is poor, it cannot be identified, requiring detection using the depth image. 2. When the shopping cart is empty, the depth image imaging is almost nonexistent, requiring detection using the grayscale image.

[0029] In one possible implementation, the step of inputting the stitched image to be identified into the trained detection model to detect the bottom height, maximum height, and item height of the shopping cart includes: inputting the stitched image to be identified into the trained detection model to detect the bottom height and maximum height of the shopping cart; and using a tracking algorithm to identify the average depth of the central regions of multiple depth images to be identified, thereby obtaining the item height. In one possible implementation, before calculating the ratio of the first difference between the item height and the bottom height, and the second difference between the maximum height and the bottom height to obtain the shopping cart overflow rate, the method further includes: when the shopping cart is empty, setting the item height to be the same as the bottom height; and when the shopping cart is full, setting the item height to be the same as the maximum height. When obtaining the item height by identifying the average depth of the central regions of multiple depth images to be identified using a tracking algorithm, the depth map of the corresponding shopping cart can be found, and a 20*20 region depth value can be taken in the middle of the detection box. The average depth value is then used as the item height. See one example. Figure 3 First, measure the height P1 from the ground to the bottom of the shopping cart and the maximum height P2 (the bottom height of each shopping cart is fixed for each supermarket). After detecting the shopping cart, find the corresponding depth map. Take a 20*20 area in the middle of the detection frame and calculate the depth value. Calculate the average depth value to get the height P3 of the shopping cart in a single detection image. The height of the shopping cart items can then be calculated: P = P3 - P1. Based on the detection of multiple frames and the height P of the shopping cart items, use a tracking algorithm to calculate the average value P0 of the shopping cart item height P over 20 frames. Finally, a highly stable shopping cart item height P0 can be obtained. For abnormal data handling, when the cart is empty, the depth map has no image, and the detected P1 is 0. In this case, P0 needs to be set to the minimum P1 value, i.e., P0 = P1. When the shopping cart is full, P0 is set to P2, i.e., P0 = P2. Finally, the overflow rate rate can be calculated as rate = (P0 - P1) / (P2 - P1). A second aspect of this application provides a shopping cart overflow rate detection device based on depth images, see [link to relevant documentation]. Figure 4 , Figure 4 A schematic diagram of a depth image-based shopping cart overflow rate detection method provided in this application embodiment includes: Image acquisition module 401 is used to acquire a sample depth image and a sample grayscale image, wherein the sample depth image and the sample grayscale image include a shopping cart image; identify the installation height based on the sample depth image, wherein the installation height is the installation height of the device acquiring the sample depth image; calculate the difference between the installation height and a preset human height to obtain a filtering threshold; and filter the sample depth image according to the filtering threshold to obtain a filtered image. The model training module 402 is used to stitch the filtered image and the sample grayscale image together to obtain a stitched image; and to train a shopping cart target detection model based on the stitched image to obtain a trained detection model. The overflow recognition module 403 is used to acquire a depth image to be recognized and a grayscale image to be recognized; to stitch the depth image to be recognized and the grayscale image to be recognized to obtain a stitched image to be recognized; to input the stitched image to be recognized into the trained detection model to detect the bottom height, the top height and the height of the goods in the shopping cart; and to calculate the ratio of the first difference between the height of the goods and the bottom height to the second difference between the top height and the bottom height to obtain the shopping cart overflow rate.

[0030] In one possible implementation, the image acquisition module is specifically used to identify the average height of the central region of the sample depth image to obtain the installation height; and to filter out images in the sample depth image that are higher than a preset human height according to the filtering threshold to obtain the filtered image.

[0031] In one possible implementation, the model training module is specifically used to concatenate the filtered image and the sample grayscale image vertically to obtain the concatenated image; to label the shopping carts in the concatenated image as a first category and a second category, wherein the first category is the shopping carts in the filtered image and the second category is the shopping carts in the sample grayscale image; to input the concatenated image into a shopping cart target detection model to obtain shopping cart detection results, wherein the shopping cart detection results are detected from the filtered image and / or from the sample grayscale image; to correct the shopping cart target detection model based on the shopping cart detection results and continue training until a preset iteration stopping condition is met to obtain the trained detection model.

[0032] In one possible implementation, the overflow recognition module is specifically used to input the stitched image to be recognized into the trained detection model to detect the bottom height and top height of the shopping cart; and to obtain the height of the goods by recognizing the average depth of the central region of multiple depth images to be recognized through a tracking algorithm.

[0033] In one possible implementation, the device further includes: The height correction module is used to set the height of the goods to be the same as the bottom height when the shopping cart is empty, and to set the height of the goods to be the same as the maximum height when the shopping cart is full.

[0034] As can be seen, the apparatus of this application embodiment can acquire sample depth images and sample grayscale images, thereby training a model using the sample depth images and sample grayscale images. The trained model can then identify the bottom height, top height, and cargo height of the shopping cart, and calculate the ratio of the first difference between the cargo height and the bottom height and the second difference between the top height and the bottom height to obtain the shopping cart overflow rate. Since the depth image is not imaged when the cart is empty, the grayscale image has the best recognition effect when the cart is empty. The problem of how to identify the shopping cart overflow rate can be solved by fusing the images.

[0035] In another aspect of the embodiments of this application, an electronic device is also provided, see [link to relevant documentation]. Figure 5 ,include: Memory 501 is used to store computer programs; Processor 502, when executing a program stored in memory, implements: Acquire a sample depth image and a sample grayscale image, wherein the sample depth image and the sample grayscale image include a shopping cart image; identify the installation height based on the sample depth image, wherein the installation height is the installation height of the device that acquired the sample depth image; calculate the difference between the installation height and a preset human height to obtain a filtering threshold; filter the sample depth image according to the filtering threshold to obtain a filtered image; The filtered image and the sample grayscale image are concatenated to obtain a concatenated image; the shopping cart object detection model is trained based on the concatenated image to obtain the trained detection model; Acquire a depth image and a grayscale image to be identified; stitch the depth image and the grayscale image to be identified to obtain a stitched image to be identified; input the stitched image to be identified into the trained detection model to detect the bottom height, top height, and item height of the shopping cart; calculate the ratio of the first difference between the item height and the bottom height to the second difference between the top height and the bottom height to obtain the shopping cart overflow rate.

[0036] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0037] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0038] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0039] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0040] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements any of the above-described depth image-based shopping cart overflow rate detection methods.

[0041] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, enables the computer to implement any of the above-described depth image-based shopping cart overflow rate detection methods.

[0042] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a solid-state drive (SSD), etc.

[0043] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0044] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. For related parts, please refer to the description of the method embodiments.

[0045] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. A method for detecting shopping cart overflow rate based on depth images, characterized in that, include: Acquire a sample depth image and a sample grayscale image, wherein the sample depth image and the sample grayscale image include a shopping cart image; identify the installation height based on the sample depth image, wherein the installation height is the installation height of the device that acquired the sample depth image; calculate the difference between the installation height and a preset human height to obtain a filtering threshold; filter the sample depth image according to the filtering threshold to obtain a filtered image; The filtered image and the sample grayscale image are concatenated to obtain a concatenated image; the shopping cart object detection model is trained based on the concatenated image to obtain the trained detection model; Acquire a depth image and a grayscale image to be identified; stitch the depth image and the grayscale image to be identified to obtain a stitched image to be identified; input the stitched image to be identified into the trained detection model to detect the bottom height, top height, and item height of the shopping cart; calculate the ratio of the first difference between the item height and the bottom height to the second difference between the top height and the bottom height to obtain the shopping cart overflow rate.

2. The method according to claim 1, characterized in that, The identification of installation height based on the sample depth image includes: The installation height is obtained by identifying the average height of the central region of the sample depth image; The step of filtering the sample depth image according to the filtering threshold to obtain the filtered image includes: Based on the filtering threshold, images in the sample depth image that are higher than a preset human height are filtered out to obtain the filtered image.

3. The method according to claim 1, characterized in that, The step of concatenating the filtered image and the sample grayscale image to obtain the concatenated image includes: The filtered image and the sample grayscale image are then stitched together vertically to obtain the stitched image. The step of training a shopping cart object detection model based on the stitched image to obtain a trained detection model includes: The shopping carts in the stitched image are labeled as a first category and a second category, respectively, wherein the first category is the shopping carts in the filtered image and the second category is the shopping carts in the sample grayscale image; The stitched image is input into the shopping cart target detection model to obtain the shopping cart detection result, wherein the shopping cart detection result is detected from the filtered image and / or from the sample grayscale image; the shopping cart target detection model is corrected according to the shopping cart detection result, and training continues until a preset iteration stopping condition is met to obtain the trained detection model.

4. The method according to claim 1, characterized in that, The step of inputting the stitched image to be identified into the trained detection model to detect the bottom height, top height, and item height of the shopping cart includes: The stitched image to be identified is input into the trained detection model to detect the bottom height and top height of the shopping cart; The cargo height is obtained by using a tracking algorithm to identify the average depth of the central region of multiple depth images to be identified.

5. The method according to claim 4, characterized in that, Before calculating the ratio of the first difference between the height of the goods and the bottom height, and the second difference between the maximum height and the bottom height to obtain the shopping cart overflow rate, the method further includes: When the shopping cart is empty, the height of the goods is set to be the same as the bottom height; When the shopping cart is full, the height of the goods is set to be the same as the maximum height.

6. A shopping cart overflow rate detection device based on depth images, characterized in that, include: An image acquisition module is used to acquire a sample depth image and a sample grayscale image, wherein the sample depth image and the sample grayscale image include a shopping cart image; to identify the installation height based on the sample depth image, wherein the installation height is the installation height of the device acquiring the sample depth image; to calculate the difference between the installation height and a preset human height to obtain a filtering threshold; and to filter the sample depth image according to the filtering threshold to obtain a filtered image. The model training module is used to stitch the filtered image and the sample grayscale image together to obtain a stitched image; and to train a shopping cart object detection model based on the stitched image to obtain a trained detection model. The overflow recognition module is used to acquire a depth image and a grayscale image to be recognized; to stitch the depth image and the grayscale image to be recognized to obtain a stitched image to be recognized; to input the stitched image to be recognized into the trained detection model to detect the bottom height, top height and item height of the shopping cart; and to calculate the ratio of the first difference between the item height and the bottom height to the second difference between the top height and the bottom height to obtain the shopping cart overflow rate.

7. The apparatus according to claim 6, characterized in that, The image acquisition module is specifically used to identify the average height of the central region of the sample depth image to obtain the installation height; and to filter out images in the sample depth image that are higher than a preset human height according to the filtering threshold to obtain the filtered image.

8. The apparatus according to claim 6, characterized in that, The model training module is specifically used to concatenate the filtered image and the sample grayscale image to obtain the concatenated image; to label the shopping carts in the concatenated image as a first category and a second category, wherein the first category is the shopping carts in the filtered image and the second category is the shopping carts in the sample grayscale image; to input the concatenated image into the shopping cart target detection model to obtain the shopping cart detection result, wherein the shopping cart detection result is detected from the filtered image and / or from the sample grayscale image; to correct the shopping cart target detection model based on the shopping cart detection result and continue training until a preset iteration stopping condition is met to obtain the trained detection model.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the method described in any one of claims 1-5.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium contains a computer program that, when executed by a processor, implements the method described in any one of claims 1-5.

Images