Deficiency detection method for shelf articles, deficiency detection device, deficiency detection robot, computer apparatus, storage medium and computer program

Abstract

To provide a deficiency detection method, device and robot for shelf articles, which improve an accuracy ratio of a deficiency detection result.SOLUTION: A deficiency detection method for shelf articles, includes the steps of: acquiring a plurality of shelf RGB images and a plurality of shelf depth images; executing a deficiency region detection step on the shelf RGB images and the shelf depth images to acquire a free region having all position information and depth information in the shelf; connecting the respective shelf RGB images together to create a perfect shelf image, and merging articles, price tags, shelf cell detection frames and all the free regions to deduplicate the merged data, and mapping the deduplicated data on the perfect shelf image; generating a grid region corresponding to each article in the perfect shelf image, and dividing all the free regions in the shelf on the basis of the grid region; and determining a deficiency detection result for each article on the basis of the free region corresponding to each article, the position information and the depth information for the grid region.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to the technical field of electronic price tags, and in particular to a method, device and robot for detecting shortages of products on shelves. [Background technology]

[0002] This section is intended to provide a background or context for the claimed embodiments of the invention. The statements herein are included in this section but are not admitted as prior art.

[0003] With the continuous development of artificial intelligence technology, the supermarket industry is also moving towards intelligence and digitalization. Through the application of AI technology, supermarkets can better meet customer needs, improve operational efficiency, improve user experience, and further expand the market influence of supermarkets. Through the application of artificial intelligence algorithms, supermarkets can analyze and predict consumer needs in real time, and more precisely formulate marketing strategies to meet consumer needs, leading to improved sales and customer satisfaction. Supermarkets can further use AI technology to automate inventory management, reduce inventory costs and shelf dwell rates, and ensure that goods are quickly replenished when needed to meet customers' shopping needs.

[0004] Smart shelves are a new type of smart retail tool that has smart analysis and automated management functions and can meet various needs of supermarkets, such as display analysis and product recognition. By monitoring and analyzing data on products on shelves in real time, smart shelves can improve shelf utilization, optimize inventory management, and improve customers' shopping experience, thereby improving supermarket operating profits. Smart shelves have already become an important part of smart retail in supermarkets and will also be an important trend in the future development of supermarkets.

[0005] In smart shelves, out-of-stock detection, such as out-of-stock area detection and out-of-stock rate detection, is an important part, so timely detection, discovery, and analysis of out-of-stock situations on shelves are beneficial for improving business operators' operating profits and increasing shoppers' motivation to shop. Summary of the Invention [Problem to be solved by the invention]

[0006] Conventional solutions for detecting missing areas and existing technical problems: Conventional solutions for detecting missing areas involve first performing image stitching and then performing detection and recognition, which heavily relies on the effect of image stitching, or directly determining the missing area using the depth information of the depth image. All of these solutions have the problem of low accuracy in detecting missing areas. [Means for solving the problem]

[0007] An embodiment of the present invention provides a shelf product shortage detection method for improving the accuracy of shelf product shortage detection, the shelf product shortage detection method comprising: acquiring a plurality of shelf RGB images and a plurality of shelf depth images; Inputting each shelf RGB image into an object detection model pre-trained by a deep learning method, detecting all products to obtain a product detection frame, detecting all electronic price tags to obtain a price tag detection frame, and detecting all shelf cells to obtain a shelf cell detection frame; acquiring a first empty area corresponding to each shelf RGB image based on position information of the product detection frame and the shelf cell detection frame corresponding to each shelf RGB image; determining a first average depth value of pixel points in the area within each price tag detection frame based on a price tag detection frame corresponding to each shelf RGB image and a corresponding shelf depth image; determining a second average depth value of pixel points in each first empty area based on a first empty area corresponding to each shelf RGB image and a corresponding shelf depth image; and acquiring a fourth empty area corresponding to each shelf RGB image based on pixel points within a set interval whose depth values ​​are determined based on the first average depth value and the second average depth value; Merging the first empty area and the fourth empty area corresponding to each shelf RGB image to obtain an empty area having all position information and depth information within the shelf; performing a missing area detection step including the steps of: stitching together each shelf RGB image to create a complete shelf image, and merging and de-duplicating the product detection frames, price tag detection frames, shelf cell detection frames and all free space to map to the complete shelf image; A step of determining a placement area of ​​each product for a complete shelf image, generating a grid area corresponding to each product, and dividing all free space in the shelf based on the grid area corresponding to each product to obtain a free space corresponding to each product, wherein the free space corresponding to each product has position information and depth information; and determining, based on the position information and depth information of the free area corresponding to each product, the volume of the free area corresponding to each product as a free area detection result.

[0008] The embodiment of the present invention further provides a shelf product shortage detection device for improving the accuracy of shelf product shortage detection, the shelf product shortage detection device comprising: an acquisition unit for acquiring a plurality of shelf RGB images and a plurality of shelf depth images; Inputting each shelf RGB image into an object detection model pre-trained by a deep learning method, detecting all products to obtain a product detection frame, detecting all electronic price tags to obtain a price tag detection frame, and detecting all shelf cells to obtain a shelf cell detection frame; acquiring a first empty area corresponding to each shelf RGB image based on position information of the product detection frame and the shelf cell detection frame corresponding to each shelf RGB image; determining a first average depth value of pixel points in the area within each price tag detection frame based on a price tag detection frame corresponding to each shelf RGB image and a corresponding shelf depth image; determining a second average depth value of pixel points in each first empty area based on a first empty area corresponding to each shelf RGB image and a corresponding shelf depth image; and acquiring a fourth empty area corresponding to each shelf RGB image based on pixel points within a set interval whose depth values ​​are determined based on the first average depth value and the second average depth value; a missing area detection unit for executing a missing area detection step for each shelf RGB image and each shelf depth image, the step including: merging a first empty area and a fourth empty area corresponding to each shelf RGB image, and acquiring an empty area having all position information and depth information within the shelf; An image stitching unit to stitch together each shelf RGB image to create a complete shelf image, and then merge and de-duplicate the product detection frames, price tag detection frames, shelf cell detection frames and all free space to map them to the complete shelf image. a space partitioning unit for determining a placement area of ​​each product for a complete shelf image, generating a grid area corresponding to each product, and partitioning all space areas in the shelf based on the grid area corresponding to each product to obtain a space area corresponding to each product, the space partitioning unit having position information and depth information for both the space area and the grid area corresponding to each product; and a stockout detection unit for determining, based on position information and depth information of the free area corresponding to each product, a volume of a stockout volume of the free area corresponding to each product as a stockout detection result.

[0009] An embodiment of the present invention further provides a computer device including a memory, a processor, and a computer program stored in the memory and executable by the processor, wherein the above-mentioned method for detecting shortage of a shelf product is realized when the computer program is executed by the processor.

[0010] An embodiment of the present invention further provides a computer-readable storage medium storing a computer program for implementing the above-described method for detecting a shortage of a shelf item when executed by a processor.

[0011] An embodiment of the present invention further provides a computer program for implementing the above-described method for detecting a shortage of a shelf item when executed by a processor. Effect of the Invention

[0012] The beneficial technical effects of the shelf product shortage detection means according to the embodiment of the present invention are as follows.

[0013] First, in an embodiment of the present invention, each shelf RGB image is input into an object detection model pre-trained by a deep learning method, and all products are detected to obtain product detection frames, all electronic price tags are detected to obtain price tag detection frames, and all shelf cells are detected to obtain shelf cell detection frames, so as to accurately obtain each detection frame and lay a foundation for subsequent detection of out-of-stock areas.

[0014] Next, compared with the technical means in the prior art that directly uses the depth information of the depth image to obtain the available space, resulting in a low accuracy in detecting out-of-stock areas, an embodiment of the present invention can obtain a more accurate out-of-stock area by inversely estimating the available space using a grid area and product detection frame based on a shelf visual RGB image and a shelf depth image including depth information.

[0015] Next, compared with technical means that first perform image stitching and then perform detection and recognition, which relies too much on the effect of image stitching and therefore has a low accuracy rate for detecting missing areas, an embodiment of the present invention performs image stitching after detecting empty areas, and when determining the empty area, combines information on the price tag position, product position, visual empty area, and depth value, ultimately obtaining a more accurate empty area and further improving the accuracy rate of missing area detection results.

[0016] As described above, the shelf product shortage detection means according to the embodiment of the present invention improves the accuracy of the shortage detection result. [Brief description of the drawings]

[0017] In order to more clearly describe the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below. Obviously, the drawings in the following description are only some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings without making any inventive efforts. In the drawings,

[0018] [Figure 1] 2 is a schematic flowchart of a method for detecting a shortage of a shelf product in an embodiment of the present invention. [Diagram 2] FIG. 13 is a schematic diagram showing a robot imaging a shelf in an embodiment of the present invention. [Diagram 3] FIG. 2 is a schematic diagram of a shelf structure and display according to an embodiment of the present invention. [Figure 4] FIG. 2 is a schematic diagram illustrating the principle of detecting a shortage of a shelf product in an embodiment of the present invention. [Diagram 5] FIG. 13 is a schematic diagram illustrating the principle of detecting a shortage of a shelf item in another embodiment of the present invention. [Figure 6] FIG. 2 is a three-dimensional effect diagram of a shelf in an embodiment of the present invention. [Figure 7] 1 is a schematic diagram showing the structure of a shelf product shortage detection device according to an embodiment of the present invention; [Figure 8] 1 is a schematic diagram showing the structure of a shelf stock shortage detection robot according to an embodiment of the present invention; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] In order to make the objectives, technical solutions and advantages of the embodiments of the present invention clearer, the embodiments of the present invention will be described in more detail below with reference to the drawings, in which the exemplary embodiments of the present invention and the description thereof are for interpreting the present invention, but are not intended to limit the present invention.

[0020] In order to better understand the embodiments of the present invention, first, relevant nouns of the embodiments of the present invention are introduced.

[0021] Product display surface, i.e. product detection frame: Like product detection frame 31 in FIG. 3, this is a product surface that is located in the front row on the shelf and is visible to the naked eye.

[0022] Price tag: As in the price tag detection frame 21 in FIG. 3, this is an identifier for identifying product information in a supermarket, such as the product name, price, and the like.

[0023] Grid: An area jointly occupied by several products of the same type, such as grid area 22 in FIG.

[0024] Shelf: As in the shelf detection frame 23 in FIG. 3, this particularly refers to a partition board on each level of a shelf on which products and price tags are placed.

[0025] Void: An area that is clearly characterized as having no products placed therein, such as void area 24 in FIG.

[0026] In order to accurately measure the out-of-stock rate of each product on the shelf in two aspects, visually and in actual depth, an embodiment of the present invention proposes a shelf product out-of-stock detection means, which includes detecting the location of an empty area on a specific shelf where no product is placed, and calculating the ratio of the empty area to a single SKU product area. The shelf product out-of-stock detection means will be described in detail below.

[0027] FIG. 1 is a schematic flow chart of a shelf product shortage detection method according to an embodiment of the present invention. As shown in FIG. 1, the method includes: Step 101 of acquiring a plurality of shelf RGB images and a plurality of shelf depth images; Step 1021: inputting each shelf RGB image into an object detection model pre-trained by a deep learning method, detecting all products to obtain product detection frames, detecting all electronic price tags to obtain price tag detection frames, and detecting all shelf cells to obtain shelf cell detection frames; A step 1022 of acquiring a first empty area corresponding to each shelf RGB image based on position information of the product detection frame and the shelf cell detection frame corresponding to each shelf RGB image; a step 1023 of determining a first average depth value of pixel points in an area within each price tag detection frame based on a price tag detection frame and a corresponding shelf depth image corresponding to each shelf RGB image, determining a second average depth value of pixel points in each first empty area based on a first empty area and a corresponding shelf depth image corresponding to each shelf RGB image, and acquiring a fourth empty area corresponding to each shelf RGB image based on pixel points within a set interval whose depth values ​​are determined based on the first average depth value and the second average depth value; a step 1024 of merging the first empty area and the fourth empty area corresponding to each shelf RGB image to obtain empty areas having position information and depth information for all areas within the shelf; a step 102 of detecting a missing area including the above, i.e., a step of "detecting a missing area" in FIG. 4 is executed for each shelf RGB image and each shelf depth image; Stitching each shelf RGB image together to create a complete shelf image, merging and de-duplicating the product detection frames, price tag detection frames, shelf cell detection frames and all free space, and mapping them to the complete shelf image, i.e., step 103 of stitching pictures in FIG. 4; A step of determining the placement area of ​​each product for the complete shelf image, generating a grid area corresponding to each product, and partitioning all the free space in the shelf based on the grid area corresponding to each product to obtain the free space corresponding to each product, where the free space corresponding to each product has position information and depth information, i.e., the step of partitioning free space in FIG. 4; and step 105 of determining the volume of the stockout volume of the free area corresponding to each product as a stockout detection result based on the position information and depth information of the free area corresponding to each product, i.e., calculating the stockout volume product.

[0028] The beneficial technical effects of the method for detecting shortage of shelf items according to the embodiment of the present invention are as follows:

[0029] First, in an embodiment of the present invention, each shelf RGB image is input into an object detection model pre-trained by a deep learning method, and all products are detected to obtain product detection frames, all electronic price tags are detected to obtain price tag detection frames, and all shelf cells are detected to obtain shelf cell detection frames, so as to accurately obtain each detection frame and lay a foundation for subsequent detection of out-of-stock areas.

[0030] Next, compared with the technical means in the prior art that directly uses the depth information of the depth image to obtain the available space, resulting in a low accuracy in detecting out-of-stock areas, an embodiment of the present invention can obtain a more accurate out-of-stock area by inversely estimating the available space using a grid area and product detection frame based on a shelf visual RGB image and a shelf depth image including depth information.

[0031] Next, compared with technical means that first perform image stitching and then perform detection and recognition, which relies too much on the effect of image stitching and therefore has a low accuracy rate for detecting missing areas, an embodiment of the present invention performs image stitching after detecting the empty area, and when determining the empty area, combines information on the price tag position, product position, visual empty area, and depth value, thereby ultimately obtaining a more accurate empty area and further improving the accuracy rate of the missing area detection result.

[0032] As described above, the accuracy of the detection result of shortage of a shelf product can be improved by the method for detecting shortage of a shelf product according to the embodiment of the present invention, which will be described in detail below with reference to FIGS.

[0033] 1. As shown in FIG. 2, a device or system for detection, (1) The wheeled robot (the internal structure diagram is shown in Figure 8) includes two parts: a chassis and a mechanical body, as shown in Figure 2. (2) The chassis is a motion module, including motors, batteries, wheels, computer software hardware, etc. (3) The machine body is an imaging system (imaging device 11 shown in FIG. 8) and a calculation system (shelf product shortage detection device 12 shown in FIG. 8), which correspond to the human eye and brain. The imaging system includes several vision cameras (111), a pair of depth cameras (112), and a supplementary light lamp, where the vision cameras can capture shelf RGB images, and the depth cameras can obtain shelf depth images, using hardware that can obtain image depth information such as, but not limited to, a Time of Flight (ToF) camera or a structured light camera. The calculation system includes computer software hardware and an algorithm processing module, and the algorithm processing module includes an algorithm model (for example, the embodiments of the present invention refer to an object detection model, a feature extraction model, a character recognition model, etc.) and a necessary database, etc.

[0034] 2. As shown in FIG. 5, the method steps of defining the stockout area and calculating the stockout rate are as follows:

[0035] (1) The robot captures and acquires an image in front of the shelf.

[0036] As shown in FIG. 1, the imaging system of the robot 1 includes multiple cameras arranged vertically and can capture images of the top, middle, and bottom of the shelf 2, and the robot needs to capture only a portion of the shelf at a time and capture the rest of the entire shelf with some overlapping of the images to the left or right, and finally stitch together the captured images to reconstruct the entire image of the shelf.

[0037] Here, the vision camera (camera) can capture an RGB image of the shelf, and the depth camera can capture an image including depth information.

[0038] (2) Detecting products in a visual shelf image.

[0039] For the acquired RGB image, a. Use the object detection model of the algorithm processing module in the computing system to detect all products 3 (as shown in FIG. 2) and obtain a detection frame, i.e., input each shelf RGB image into the object detection model pre-trained by a deep learning method to detect all products and obtain a product detection frame such as “31” in FIG. 3; b. Extract a small image of the product area based on the coordinates, width, and height of the detection frame using image processing methods; Here, the object detection model can be trained using a deep learning method, i.e., the collected scene image data is used to annotate the product, and then a deep network model is trained and verified based on the idea of ​​maximum likelihood, where the deep network model includes, but is not limited to, a deep convolutional network model and a visual VIT-like model.

[0040] Recognize products in shelf visual images (optional).

[0041] (3) Detect the image cropped in step (2).

[0042] a. The small images are fed into a feature extraction model to extract features. b. Match the features extracted in the previous step with features in a product feature library prepared in advance in the algorithm processing module, calculate the Euclidean distance between the features, and determine the result corresponding to the minimum value as the product recognition result with the highest reliability; Here, the feature extraction model is trained using a deep learning method, i.e., the collected scene image data is used to annotate and classify products, and then a deep network model is trained and verified.

[0043] As can be seen from the above, in one embodiment, the above shelf product shortage detection method includes: A step of extracting a small image of the product area based on position information of the product detection frame; A step of inputting the cut-out small image of the product area into a feature extraction model that has been previously trained and generated using a deep learning method to extract features; The method may further include a step of matching the extracted features with features in a pre-prepared product feature library, calculating a Euclidean distance between the extracted features and the features in the pre-prepared product feature library, and determining a result corresponding to the smallest Euclidean distance as the most reliable product recognition result.

[0044] When specifically implemented, the above embodiment realizes product recognition, and specifically determines which products are out of stock through subsequent identification of the recognized products. Of course, the product recognition result can also be used to adjust the grid, so as to further improve the accuracy of out-of-stock detection. For details, please refer to the description of the embodiment below.

[0045] (4) Detecting price tags in the shelf visual image.

[0046] For the acquired RGB image, a. use an object detection model of an algorithm processing module in the computing system to detect all EPLs, i.e., to detect all EPLs and obtain a price tag detection frame, such as the detection frame of “21” in FIG. 3; b. Extract a small image of the price tag area based on the coordinates, width, and height of the detection frame using an image processing method; Here, the object detection model is trained using a deep learning method, i.e., the price tag is annotated using collected scene image data, and then a deep network model is trained and verified based on the idea of ​​maximum likelihood, and the deep network model includes, but is not limited to, a deep convolutional network model and a visual VIT-like model.

[0047] (5) Recognize the detected price tag. a. The small image detected and cut out in step (4) is fed into a character recognition model or an OCR algorithm model to recognize the character or character code on the price tag; b. Search for information that matches the character or character code recognized in the previous step from the "product information library" prepared in advance in the algorithm processing module, and use the searched corresponding product information as the price tag recognition result (optional); Here, the character recognition model can be trained using a deep learning method, i.e., character codes are annotated using collected scene image data, and then a deep network model is trained and verified based on the idea of ​​maximum likelihood, and the deep network model includes, but is not limited to, a deep convolutional network model and a visual VIT-like model.

[0048] In one embodiment, the above shelf product shortage detection method includes: A step of extracting a small image of the price tag area based on position information of the price tag detection frame; inputting the cut-out small image of the price tag area into a character recognition model that has been previously trained using a deep learning method to recognize character information on the price tag; The method may further include a step of searching for price tag character information that matches the recognized price tag character information from a pre-prepared product information library, and setting product information corresponding to the searched matching price tag character information as the price tag recognition result.

[0049] When specifically implemented, the above embodiment can recognize price tags on shelves, and subsequently bind the price tags to corresponding products based on the price tag recognition results, and can quickly determine which specific products are out of stock.

[0050] (6) Segment the shelves in the shelf visual image. Method 1: Based on the object detection model trained by the deep learning method, the RGB image is fed into the network model to directly output the shelf detection frame, that is, all shelves are detected, and a shelf detection frame such as “23” in Figure 3 is obtained.

[0051] Method 2: Fitting is performed based on the position of the price tag detection in step (4), i.e., price tags in the same row are connected, and the average horizontal line is regarded as the shelf line.

[0052] (7) Divide the cells. Method 1: Limit the height of one grid based on the position of the shelf, treat the range between two shelves as one cell, and detect all the above shelves to obtain a shelf detection frame. Objective 1 is to realize Method 1 based on the shelf detection frame, and objective 2 can also be applied to the following step “(13) Stitching images together”. For details, see the description of the examples below.

[0053] Method 2: Based on the object detection model trained by the deep learning method, the RGB image is fed into the network model to directly output the detection frame of the shelf cell area, that is, all shelf cells are detected to obtain the shelf cell detection frame; The final cell area is shown as "25" in FIG.

[0054] (8) Based on the cell and the product, the free space is inversely estimated to obtain free space 1 (first free space), that is, step 1022 above. Using an image processing method, the masks of the cell and the product detection frame are calculated, and the result of subtracting the product mask from the cell mask is taken as the empty area 1, such as the gray area in Figure 3.

[0055] (9) Detect cavities in the shelf visual image and obtain empty area 2 (second empty area) (optional).

[0056] In order to prevent mistakes or omissions in the free space obtained in step (8), the obtained RGB image is Using the object detection model of the algorithm processing module in the computing system, detect all vacant product positions in the image, and obtain a detection frame of vacant positions as shown by "24" in Figure 3; Here, the object detection model can be trained using a deep learning method, in which the collected scene image data is used to annotate the exact vacant position, and then a deep network model is trained and verified based on the idea of ​​maximum likelihood (the deep network model includes, but is not limited to, a deep convolutional network model and a visual VIT-like model).

[0057] (10) Merge and combine free space 1 and free space 2 (optional).

[0058] For free space 1 and free space 2, take the union of the two areas to obtain free space 3 (the third free space), As can be seen from the above, in one embodiment, the above shelf product shortage detection method includes: Detecting cavities in each shelf RGB image using the object detection model to obtain a second empty area; and merging the first empty area and the second empty area to obtain a third empty area corresponding to each shelf RGB image. Then, the subsequent step of merging the first empty space area and the fourth empty space area corresponding to each shelf RGB image to obtain all empty space within the shelf may include merging the third empty space area and the fourth empty space area corresponding to each shelf RGB image to obtain all empty space within the shelf.

[0059] In specific implementation, the object detection model can be used to detect cavities in each shelf RGB image, and subsequently improve the accuracy of missing item detection.

[0060] (11) Match the free space range and depth value.

[0061] For the acquired depth image, All price tag detection frames in the RGB image are overlaid on the depth image, an average depth value x1 of the pixel points in the area within all price tag detection frames is calculated, all price tag detection frame areas are regarded as the frontmost display surface, and all depth values ​​are corrected to x1; similarly, for empty area 3 obtained from the RGB image, a corresponding depth image is overlaid, an average depth value x2 of all pixel points in the area is calculated, all empty areas are regarded as the bottom of the innermost shelf, and all depth values ​​are corrected to x2; Within a cell, pixels whose depth values ​​are between x1 and x2 are not in the front row or in a sold-out area, but can be regarded as an area where the front row is sold and the back row is exposed. In this case, one interval value [x1+a, x2-b] is set, and the pixel points within this interval are also regarded as a missing item area, missing item area 4 is obtained, and the depth value is the average value within the area.

[0062] As can be seen from the above, in one embodiment, the step of determining a first average depth value of pixel points in the area within each price tag detection frame based on the price tag detection frame corresponding to each shelf RGB image and the corresponding shelf depth image may include overlaying all price tag detection frames in each RGB image with the corresponding shelf depth image and calculating a first average depth value of pixel points in the area within all price tag detection frames; Determining a second average depth value of pixel points within each first empty area based on the first empty area corresponding to each shelf RGB image and the corresponding shelf depth image may include overlaying the first empty area corresponding to each shelf RGB image onto the corresponding shelf depth image and calculating a second average depth value of all pixel points within the first empty area.

[0063] Specifically, when this method is implemented, the free space calculated in the RGB image is overlaid on the depth image, and based on the idea of ​​running statistics within the area, a more accurate depth value of the area can be obtained, which is advantageous for subsequently determining unsold free space, and improves the accuracy of out-of-stock detection results.

[0064] (12) Merge free space.

[0065] The empty area 1, empty area 3, and empty area 4 (fourth empty area) are combined into all empty areas in the shelf, and each empty area has detection frame position information and depth information.

[0066] (13) Stitching images together.

[0067] For RGB images, The images captured in step (1) are stitched together to form one complete aisle or one complete shelf, and during the image stitching process, the price tags, items, shelves, cells, and free space captured in the previous steps are merged, deduplicated, and mapped into the stitched image.

[0068] As can be seen from the above, in one embodiment, the above shelf product shortage detection method may further include inputting each shelf RGB image into an object detection model pre-trained by a deep learning method to detect all shelf boards and obtain a shelf board detection frame; The step of stitching together each shelf RGB image to create a complete shelf image, merging and deduplicating the product detection frames, price tag detection frames, shelf cell detection frames, and all open areas, and mapping to the complete shelf image may include stitching together each shelf RGB image to create a complete shelf image, merging and deduplicating the product detection frames, price tag detection frames, shelf board detection frames, shelf cell detection frames, and all open areas, and mapping to the complete shelf image.

[0069] In concrete implementation, when determining the free space, the price tag position, the product position, the shelf position, the visual free space, and the depth value information are combined, and finally a more accurate free space can be obtained.

[0070] (14) Generate a grid.

[0071] For the stitched image, a placement area for each product, i.e., a product grid area, can be defined.

[0072] Method 1, a. Define a grid based on the location of the price tag in the image. For example, if a price tag is placed at the bottom left of a product in a supermarket, the width of one grid will be the left edge of the first price tag from left to right, and the width of one grid will be the left edge of the next price tag from left to right. If there is no price tag on a shelf, the width of one grid will be the two ends of the shelf. b. Limit the grid height based on shelf position or cell area.

[0073] Method 2, Previously, we trained an object detection model to detect grids based on deep learning methods. The specific method is as follows. a) collecting a plurality of images to form an image dataset; b) The possible placement area of ​​each product in each image is taken as a grid of the product, which is manually defined and annotated as location label information, and finally a label information set is formed; c) Build a deep neural network model, and use the above image dataset and the corresponding label information set to train the network model based on regression ideas to obtain a grid detection model; Given the current RGB image, feed it into the trained grid detection network model to directly output a detection window of possible grid regions on the shelf; The grid is bound to the nearest price tag, and the grid area represents a specific product placement area, or the number of each product in the grid is counted, and the product with the highest number is positioned in the product placement area represented by the grid.

[0074] (15) Adjust the grid (optional).

[0075] There may be some deviation between the generated grid and the actual product position arrangement. To eliminate the deviation, the generated grid is adjusted based on the product recognition results so that the left and right boundaries of the grid include all corresponding products near the current position, and finally a more accurate product arrangement area is obtained, such as the central grid area “22” in Figure 3.

[0076] As can be seen from the above, in one embodiment, the above shelf product shortage detection method includes: The method may further include adjusting the generated grid area based on the product recognition result, adjusting left and right boundaries of the grid area so as to include all corresponding products near the current position, and obtaining adjusted grid areas corresponding to each product; The step of dividing all the free space in the shelf based on the grid area corresponding to each product to obtain the free space corresponding to each product includes dividing all the free space in the shelf based on the adjusted grid area corresponding to each product to obtain the free space corresponding to each product.

[0077] In specific implementation, the results of product recognition can be used to adjust the grid generated by the price tags to obtain a more accurate product sales area, which is more accurate when subsequently calculating the out-of-stock rate.

[0078] (16) Partition free space.

[0079] Considering that some free regions span multiple grids, we partition the free regions based on grid area, to the effect in FIG.

[0080] (17) Remove tiny void spaces.

[0081] Considering that gaps will exist when products are actually placed, the free space estimated based on step (8) may include long and narrow free spaces (strip-shaped free spaces with a length-to-width ratio of greater than 5) or relatively small free spaces separated in step (16). Therefore, here, a morphological method in image processing is used to directly remove the small free spaces.

[0082] As can be seen from the above, in one embodiment, the above shelf product shortage detection method may further include a step of using a morphological method in image processing to remove a first empty space corresponding to each shelf RGB image or a strip-shaped empty space having a length-to-width ratio greater than 5 relative to the empty space corresponding to each product.

[0083] In a concrete implementation, by removing the elongated regions, the free area can be determined more accurately, and the accuracy is high.

[0084] (18) Calculate the volume of the grid volume and the volume of the missing part.

[0085] For each divided grid, the width of the grid is the width w of the grid volume, the height of the grid is the height h of the grid volume, the value of x2-x1 in step (11) is the depth d, and the product w×h×d is the volume V0 of the grid volume. Similarly, for the free areas within the grid, calculate the volume V1 of each free volume based on its width, height, and depth information. If there are multiple free areas, multiple free volumes are formed, and the corresponding volumes are V2, V3, V4..., and the volume of the final merged free volume is Vs.

[0086] In real scenes, the cell area is often higher than the product, so there may be vacant spaces that do not actually fit in the vacant area estimated in step (8). Therefore, for V0 and Vs, a discrete value of [0, 1] scale is multiplied in the dimension of height h, and the higher the position, the lower the weight, thereby weakening the influence of the top of the grid volume and vacant volume, and finally obtaining more accurate V0 and Vs.

[0087] As can be seen from the above, in one embodiment, the above shelf product shortage detection method may further include a step of multiplying the volume of the grid area volume and the volume of the free space shortage volume by a discrete scale value ranging from 0 to 1 in the height dimension so as to weaken the influence of the upper parts of the grid area volume and the free space shortage volume, the discrete scale value being smaller for higher positions.

[0088] In specific implementation, by multiplying the grid area and the stockout area by a discrete scale value from 0 to 1, the natural and unrealistic empty area can be weakened, which is beneficial to improving the accuracy of the subsequent calculation of the stockout rate.

[0089] (19) Determine information about the stockout area.

[0090] According to the description of step (14), each grid corresponds to a different product. As shown in FIG. 6, there is a corresponding grid volume and a stockout volume in each grid, the stockout rate=Vs / V0, the stockout area is an empty area, and based on the position, width, height, and depth, a two-dimensional or three-dimensional display can be performed on the stitched image. Finally, it is possible to notify that the stockout rate of a product with the product name xxx is xx%, and the location is xth layer xth row on the shelf diagram, or to directly draw the area.

[0091] As can be seen from the above, in one embodiment, the grid area corresponding to each product has position information and depth information, and the method for detecting shortage of shelf products includes: determining a volume of a grid area corresponding to each product based on position information and depth information of the grid area corresponding to each product; The method may further include a step of determining a stock-out rate for each product based on the volume of the grid area volume corresponding to each product and the volume of the free area stock-out volume, that is, a step of "calculating the stock-out rate" in FIG.

[0092] In concrete implementation, compared with the prior art's solution of allocating products with the same attributes near the price tag to the price tag based on the idea of ​​allocation and determining whether the product is out of stock based on the number or area threshold, the embodiment of the present invention calculates the out-of-stock rate using a volume-based method, so as to more accurately calculate the proportion of sold out and non-sold out products in the out-of-stock state, and obtain a more accurate out-of-stock rate. The embodiment of the present invention inversely estimates the free area using a grid area and a product detection frame, so as to obtain a more accurate area than directly using the depth information of a depth image to obtain the free area, and the calculation is simple, and the robustness is higher and the model training cost is lower than the method of directly calculating the out-of-stock area based on deep learning.

[0093] As described above, an embodiment of the present invention can obtain the accurate vacant product positions on the shelf and the accurate actual grid area of ​​each product, thereby calculating the accurate product out-of-stock rate and presenting a more visually appropriate product out-of-stock situation or product placement saturation situation.

[0094] The embodiment of the present invention further provides a shelf product shortage detection device as described in the following embodiment. The principle by which the device solves the problem is similar to that of the shelf product shortage detection method, so the implementation of the device can refer to the implementation of the shelf product shortage detection method and will not be described again.

[0095] FIG. 7 is a structural schematic diagram of a shelf product shortage detection device 12 according to an embodiment of the present invention. As shown in FIG. 7, the device includes: an acquisition unit 121 for acquiring a plurality of shelf RGB images and a plurality of shelf depth images; Inputting each shelf RGB image into an object detection model pre-trained by a deep learning method, detecting all products to obtain a product detection frame, detecting all electronic price tags to obtain a price tag detection frame, and detecting all shelf cells to obtain a shelf cell detection frame; acquiring a first empty area corresponding to each shelf RGB image based on position information of the product detection frame and the shelf cell detection frame corresponding to each shelf RGB image; determining a first average depth value of pixel points in the area within each price tag detection frame based on a price tag detection frame corresponding to each shelf RGB image and a corresponding shelf depth image; determining a second average depth value of pixel points in each first empty area based on a first empty area corresponding to each shelf RGB image and a corresponding shelf depth image; and acquiring a fourth empty area corresponding to each shelf RGB image based on pixel points within a set interval whose depth values ​​are determined based on the first average depth value and the second average depth value; Merging the first empty area and the fourth empty area corresponding to each shelf RGB image to obtain an empty area having all position information and depth information within the shelf; a missing area detection unit 122 that executes a missing area detection step including the steps of: an image stitching unit 123 for stitching together each shelf RGB image to create a complete shelf image, and merging and de-duplicating the product detection frames, the price tag detection frames, the shelf cell detection frames and all free space to map to the complete shelf image; a space partitioning unit 124 for determining a placement area of ​​each product for a complete shelf image, generating a grid area corresponding to each product, and partitioning all empty areas in the shelf based on the grid area corresponding to each product to obtain an empty area corresponding to each product, the empty area corresponding to each product having position information and depth information; and a stockout detection unit 125 for determining, based on the position information and depth information of the free area corresponding to each product, the volume of the free area stockout volume corresponding to each product as a stockout detection result. In one embodiment, the shelf product shortage detection device includes: A step of extracting a small image of the product area based on position information of the product detection frame; A step of inputting the cut-out small image of the product area into a feature extraction model that has been previously trained and generated using a deep learning method to extract features; The method may further include a product recognition unit used in the steps of: matching the extracted features with features in a pre-prepared product feature library; calculating the Euclidean distance between the extracted features and the features in the pre-prepared product feature library; and determining a result corresponding to the smallest Euclidean distance as the most reliable product recognition result.

[0096] In one embodiment, the above-mentioned shelf product shortage detection device may further include an adjustment unit used in the steps of adjusting the generated grid area based on the product recognition result, adjusting the left and right boundaries of the grid area so as to include all corresponding products near the current location, and obtaining adjusted grid areas corresponding to each product.

[0097] The space partition unit is specifically used for partitioning all the space areas in the shelf based on the adjusted grid area corresponding to each product, to obtain the space area corresponding to each product.

[0098] In one embodiment, the shelf product shortage detection device includes: A step of extracting a small image of the price tag area based on position information of the price tag detection frame; inputting the cut-out small image of the price tag area into a character recognition model that has been previously trained using a deep learning method to recognize character information on the price tag; The method may further include a product recognition unit used in the steps of searching for price tag character information that matches the recognized price tag character information from a pre-prepared product information library, and setting product information corresponding to the searched matching price tag character information as the price tag recognition result.

[0099] In one embodiment, the shelf product shortage detection device includes: a cavity detection unit for detecting a cavity in each shelf RGB image using the object detection model to obtain a second empty area; and a merging processing unit for merging the first empty area and the second empty area to obtain a third empty area corresponding to each shelf RGB image; The step of merging the first empty space area and the fourth empty space area corresponding to each shelf RGB image to obtain all empty space within the shelf includes merging the third empty space area and the fourth empty space area corresponding to each shelf RGB image to obtain all empty space within the shelf.

[0100] In one embodiment, the above-mentioned shelf product shortage detection device may further include a strip-shaped empty space removal unit used to remove a first empty space corresponding to each shelf RGB image or a strip-shaped empty space having a length-to-width ratio greater than 5 relative to the empty space corresponding to each product, using a morphological method in image processing.

[0101] In one embodiment, the step of determining a first average depth value of pixel points in an area within each price tag detection frame based on the price tag detection frame corresponding to each shelf RGB image and the corresponding shelf depth image may include overlaying all price tag detection frames in each RGB image with the corresponding shelf depth image and calculating a first average depth value of pixel points in the area within all price tag detection frames; The step of determining a second average depth value of pixel points in each first empty area based on the first empty area corresponding to each shelf RGB image and the corresponding shelf depth image may include overlaying the first empty area corresponding to each shelf RGB image onto the corresponding shelf depth image and calculating the second average depth value of all pixel points in the first empty area.

[0102] In one embodiment, the grid area corresponding to each product has position information and depth information, and the missing item detection unit further comprises: determining a volume of a grid area corresponding to each product based on position information and depth information of the grid area corresponding to each product; and determining a stock-out rate for each product based on the volume of the grid area volume and the volume of the free area stock-out volume corresponding to each product.

[0103] In one embodiment, the above-mentioned shelf product shortage detection device may further include a discrete processing unit for multiplying the volume of the grid area volume and the volume of the free space shortage volume by a discrete scale value from 0 to 1 in the height dimension to weaken the influence of the upper part of the grid area volume and the free space shortage volume, wherein the higher the position, the smaller the discrete scale value.

[0104] In one embodiment, the above-mentioned shelf product shortage detection method may further include a shelf detection unit, for inputting each shelf RGB image into an object detection model pre-trained by a deep learning method, to detect all shelves and obtain a shelf detection frame.

[0105] Specifically, the image stitching unit is used to stitch together each shelf RGB image to create a complete shelf image, and merge and eliminate duplicates of the product detection frames, price tag detection frames, shelf board detection frames, shelf cell detection frames and all free areas to map them into the complete shelf image.

[0106] The embodiments of the present invention further provide a shelf product shortage detection robot as described in the following embodiments. Since the principle of the robot's problem solving is similar to that of the shelf product shortage detection method, the implementation of the robot can refer to the implementation of the shelf product shortage detection method, and will not be described again.

[0107] FIG. 8 is a structural schematic diagram of a shelf product shortage detection robot according to an embodiment of the present invention. As shown in FIG. 8, the robot: The imaging device 11 includes a vision camera 111 for capturing a plurality of shelf RGB images and a depth camera 112 for capturing a plurality of shelf depth images, and the imaging device realizes a step of capturing a picture in FIG. The shelf product shortage detection device 12 described above is used to obtain a shortage detection result for each product based on a plurality of shelf RGB images and a plurality of shelf depth images.

[0108] An embodiment of the present invention further provides a computer device including a memory, a processor, and a computer program stored in the memory and executable by the processor, wherein the above-mentioned method for detecting shortage of a shelf product is realized when the computer program is executed by the processor.

[0109] An embodiment of the present invention further provides a computer-readable storage medium storing a computer program for implementing the above-described method for detecting a shortage of a shelf item when executed by a processor.

[0110] An embodiment of the present invention further provides a computer program for implementing the above-described method for detecting a shortage of a shelf item when executed by a processor.

[0111] The advantageous technical effects of the shelf product shortage detection means according to the embodiment of the present invention are as follows.

[0112] First, in this embodiment of the present invention, each shelf RGB image is input into an object detection model pre-trained by a deep learning method, and all products are detected to obtain product detection frames, all electronic price tags are detected to obtain price tag detection frames, and all shelf cells are detected to obtain shelf cell detection frames, so as to accurately obtain each detection frame and lay the foundation for subsequent detection of out-of-stock areas.

[0113] Next, compared with the technical means in the prior art that directly uses the depth information of the depth image to obtain the available space, resulting in a low accuracy in detecting out-of-stock areas, an embodiment of the present invention can obtain a more accurate out-of-stock area by inversely estimating the available space using a grid area and product detection frame based on a shelf visual RGB image and a shelf depth image including depth information.

[0114] Next, compared with technical means that first perform image stitching and then perform detection and recognition, which relies too much on the effect of image stitching and therefore has a low accuracy rate for detecting missing areas, an embodiment of the present invention performs image stitching after detecting the empty area, and when determining the empty area, combines information on the price tag position, product position, visual empty area, and depth value, thereby ultimately obtaining a more accurate empty area and further improving the accuracy rate of the missing area detection result.

[0115] As described above, the shelf product shortage detection means according to the embodiment of the present invention improves the accuracy of the shortage detection results.

[0116] Those skilled in the art should understand that the embodiments of the present invention may be provided as a method, a system, or a computer program product. Therefore, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. And, the present invention may take the form of a computer program product embodied in one or more computer usable storage media (including, but not limited to, magnetic disk memory, CD-ROM, optical memory, etc.) containing computer usable program code.

[0117] The present invention has been described with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to the embodiments of the present invention. It should be understood that each flow and / or block of the flowcharts and / or block diagrams, and combinations of flows and / or blocks of the flowcharts and / or block diagrams, can be realized by computer program instructions. These computer program instructions can be provided to a general-purpose computer, a special-purpose computer, an embedded processor, or a processor of other programmable data processing devices to generate an apparatus for implementing a function specified in one or more flows of the flowcharts and / or one or more blocks of the block diagrams by the instructions being executed by a processor of the computer or other programmable data processing devices, thereby generating an apparatus.

[0118] These computer program instructions may be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a particular manner, thereby producing an article of manufacture including an instruction apparatus that implements the functions specified in one or more flows of the flowcharts and / or one or more blocks of the block diagrams with the instructions stored in the computer-readable memory.

[0119] These computer program instructions can be loaded into a computer or other programmable data processing device, thereby causing the computer or other programmable device to execute a series of operational steps to generate a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more flows of the flowcharts and / or one or more blocks of the block diagrams.

[0120] The specific embodiments described above further explain the objectives, technical solutions and beneficial effects of the present invention. However, it should be understood that the above are only specific embodiments of the present invention and are not intended to limit the scope of the present invention. Any amendments, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention should be included within the scope of the present invention.

Claims

1. acquiring a plurality of shelf RGB images and a plurality of shelf depth images; Inputting each shelf RGB image into an object detection model pre-trained by a deep learning method, detecting all products to obtain a product detection frame, detecting all electronic price tags to obtain a price tag detection frame, and detecting all shelf cells to obtain a shelf cell detection frame; acquiring a first empty area corresponding to each shelf RGB image based on position information of the product detection frame and the shelf cell detection frame corresponding to each shelf RGB image; determining a first average depth value of pixel points in the area within each price tag detection frame based on a price tag detection frame corresponding to each shelf RGB image and a corresponding shelf depth image; determining a second average depth value of pixel points in each first empty area based on a first empty area corresponding to each shelf RGB image and a corresponding shelf depth image; and acquiring a fourth empty area corresponding to each shelf RGB image based on pixel points within a set interval whose depth values ​​are determined based on the first average depth value and the second average depth value; Merging the first empty area and the fourth empty area corresponding to each shelf RGB image to obtain an empty area having position information and depth information of all areas in the shelf; A step of detecting a missing area including the following for each shelf RGB image and each shelf depth image; stitching together each shelf RGB image to create a complete shelf image, merging and de-duplicating the product detection frames, price tag detection frames, shelf cell detection frames and all free space, and mapping them to the complete shelf image; A step of determining a placement area of ​​each product for a complete shelf image, generating a grid area corresponding to each product, and dividing all free space in the shelf based on the grid area corresponding to each product to obtain a free space corresponding to each product, wherein the free space corresponding to each product has position information and depth information; determining a volume of a stockout volume of the free area corresponding to each product as a stockout detection result based on the position information and depth information of the free area corresponding to each product. A method for detecting a shortage of a shelf product.

2. A step of extracting a small image of the product area based on position information of the product detection frame; A step of inputting the cut-out small image of the product area into a feature extraction model that has been previously trained and generated using a deep learning method to extract features; The method further includes a step of: matching the extracted features with features in a product feature library prepared in advance; calculating a Euclidean distance between the extracted features and the features in the product feature library prepared in advance; and determining a result corresponding to the minimum value of the Euclidean distance as a product recognition result with the highest reliability.

2. The method for detecting a shortage of a shelf product according to claim 1.

3. The method further includes adjusting the generated grid area based on the product recognition result, adjusting left and right boundaries of the grid area so as to include all corresponding products near the current position, and obtaining adjusted grid areas corresponding to each product; The step of dividing all the free space areas in the shelf based on the grid area corresponding to each product to obtain the free space areas corresponding to each product includes dividing all the free space areas in the shelf based on the adjusted grid area corresponding to each product to obtain the free space areas corresponding to each product.

3. The method for detecting a shortage of a shelf product according to claim 2.

4. A step of extracting a small image of the price tag area based on position information of the price tag detection frame; inputting the cut-out small image of the price tag area into a character recognition model that has been previously trained using a deep learning method to recognize character information on the price tag; The method further includes a step of searching for price tag character information that matches the recognized price tag character information from a product information library prepared in advance, and determining the product information corresponding to the searched matching price tag character information as a recognition result of the price tag.

2. The method for detecting a shortage of a shelf product according to claim 1.

5. Detecting cavities in each shelf RGB image using the object detection model to obtain a second empty area; and fusing and merging the first empty area and the second empty area to obtain a third empty area corresponding to each shelf RGB image. The step of merging the first empty area and the fourth empty area corresponding to each shelf RGB image to obtain all empty areas in the shelf includes merging the third empty area and the fourth empty area corresponding to each shelf RGB image to obtain all empty areas in the shelf.

2. The method for detecting a shortage of a shelf product according to claim 1.

6. and removing a first empty area corresponding to each shelf RGB image or a strip-like empty area having a length to width ratio greater than 5 relative to an empty area corresponding to each product using a morphological method in image processing.

2. The method for detecting a shortage of a shelf product according to claim 1.

7. The step of determining a first average depth value of pixel points in an area within each price tag detection frame based on the price tag detection frame corresponding to each shelf RGB image and the corresponding shelf depth image includes overlaying all price tag detection frames in each RGB image on the corresponding shelf depth image and calculating a first average depth value of pixel points in the area within all price tag detection frames; The step of determining a second average depth value of pixel points in each first empty area based on the first empty area corresponding to each shelf RGB image and the corresponding shelf depth image includes overlaying the first empty area corresponding to each shelf RGB image on the corresponding shelf depth image and calculating the second average depth value of all pixel points in the first empty area.

2. The method for detecting a shortage of a shelf product according to claim 1.

8. The grid area corresponding to each product has position information and depth information, determining a volume of a grid area corresponding to each product based on position information and depth information of the grid area corresponding to each product; determining a stock-out rate for each product based on a volume of the grid area volume corresponding to each product and a volume of the free area stock-out volume.

2. The method for detecting a shortage of a shelf product according to claim 1.

9. The method further includes a step of multiplying the volume of the grid area volume and the volume of the free space shortage volume by a discrete scale value ranging from 0 to 1 in a height dimension so as to weaken the influence of the upper part of the grid area volume and the free space shortage volume, the discrete scale value being smaller at a higher position; The method for detecting a shortage of a shelf product according to claim 8.

10. The method further includes inputting each shelf RGB image into an object detection model pre-trained by a deep learning method to detect all shelves and obtain a shelf detection frame; The step of stitching each shelf RGB image to create a complete shelf image, merging and de-duplicating the product detection frames, the price tag detection frames, the cell detection frames, and all open areas, and mapping to the complete shelf image includes stitching each shelf RGB image to create a complete shelf image, merging and de-duplicating the product detection frames, the price tag detection frames, the shelf board detection frames, the shelf cell detection frames, and all open areas, and mapping to the complete shelf image.

2. The method for detecting a shortage of a shelf product according to claim 1.

11. An acquisition unit for acquiring a plurality of shelf RGB images and a plurality of shelf depth images; Inputting each shelf RGB image into an object detection model pre-trained by a deep learning method, detecting all products to obtain a product detection frame, detecting all electronic price tags to obtain a price tag detection frame, and detecting all shelf cells to obtain a shelf cell detection frame; acquiring a first empty area corresponding to each shelf RGB image based on position information of the product detection frame and the shelf cell detection frame corresponding to each shelf RGB image; determining a first average depth value of pixel points in the area within each price tag detection frame based on a price tag detection frame corresponding to each shelf RGB image and a corresponding shelf depth image; determining a second average depth value of pixel points in each first empty area based on a first empty area corresponding to each shelf RGB image and a corresponding shelf depth image; and acquiring a fourth empty area corresponding to each shelf RGB image based on pixel points within a set interval whose depth values ​​are determined based on the first average depth value and the second average depth value; Merging the first empty area and the fourth empty area corresponding to each shelf RGB image to obtain an empty area having position information and depth information of all areas in the shelf; A missing area detection unit for performing a missing area detection step including the steps of: An image stitching unit to stitch together each shelf RGB image to create a complete shelf image, and to merge and de-duplicate the product detection frames, price tag detection frames, shelf cell detection frames and all free space to map them to the complete shelf image. a space partitioning unit for determining a placement area of ​​each product for a complete shelf image, generating a grid area corresponding to each product, and partitioning all space areas in the shelf based on the grid area corresponding to each product to obtain a space area corresponding to each product, the space partitioning unit having position information and depth information for each of the space areas and the grid area corresponding to each product; a stockout detection unit for determining a volume of a stockout volume of the free area corresponding to each product as a stockout detection result based on position information and depth information of the free area corresponding to each product. A shelf stockout detection device characterized by the above.

12. An imaging device including a vision camera for capturing a plurality of shelf RGB images and a depth camera for capturing a plurality of shelf depth images; and the shelf product shortage detection device according to claim 11 for acquiring a shortage detection result for each product based on a plurality of shelf RGB images and a plurality of shelf depth images. A robot for detecting missing items on shelves.

13. A computing device including a memory, a processor, and a computer program stored in the memory and executable by the processor, Execution of the computer program by the processor implements the method according to any one of claims 1 to 10.

1. A computer device comprising:

14. A computer program is stored which, when executed by a processor, implements the method according to any one of claims 1 to 10. A computer-readable storage medium comprising:

15. A computer program product, which when executed by a processor, implements the method according to any one of claims 1 to 10.

Images