Display detection method, device and equipment of goods shelf and storage medium

Abstract

The invention provides a shelf display detection method and device, equipment and a storage medium, and belongs to the technical field of image processing. The shelf display detection method comprises the steps of obtaining two-dimensional image data of a to-be-detected shelf from a front orientation; determining coordinate information and category information of each commodity in the two-dimensional image data; based on the coordinate information of each commodity, hierarchical information of each commodity is determined, and the hierarchical information comprises a hierarchical sequence, an in-layer arrangement sequence and an in-layer stacking sequence; based on the category information and the hierarchy information of each commodity, display information of the shelf is determined, and the display information comprises a mapping relation between each commodity and the category information and the hierarchy information; and detecting the display state of the goods shelf according to a predefined display standard based on the display information. According to the invention, the category and coordinate information of the commodity can be accurately identified and effectively associated, and then the display information of the shelf is obtained for display detection, so that the accuracy and management efficiency of shelf display detection are improved.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to a shelf display detection method, device, equipment and storage medium. Background Art

[0002] As technology develops, retailers are increasingly adopting an online-offline integrated business model, the so-called “omnichannel retail”, to provide a seamless shopping experience.

[0003] However, existing shelf detection methods rely on manual inspections or simple image processing technology, making it difficult to accurately identify the categories and specific coordinates of goods, and fail to effectively associate the categories and coordinate information of goods, which affects the accuracy of shelf display detection and management efficiency. Summary of the invention

[0004] The purpose of the embodiments of the present disclosure is to provide a shelf display detection method, device, equipment and storage medium, aiming to solve the technical problem of how to improve the accuracy and management efficiency of shelf display detection.

[0005] In order to achieve the above-mentioned objectives, in a first aspect, an embodiment of the present disclosure provides a method for detecting a display of a shelf, comprising: acquiring two-dimensional image data of a shelf to be detected from a frontal orientation; determining coordinate information and category information of each commodity in the two-dimensional image data; determining hierarchical information of each commodity based on the coordinate information of each commodity, wherein the hierarchical information includes a hierarchical sequence, an arrangement sequence within a layer, and a stacking sequence within a layer; determining display information of the shelf based on the category information and hierarchical information of each commodity, wherein the display information includes a mapping relationship between each commodity and the category information and the hierarchical information; based on the display information, detecting the display status of the shelf according to a predefined display standard.

[0006] In some embodiments, determining the coordinate information and category information of each commodity in the two-dimensional image data includes: determining the coordinate information of each commodity based on the two-dimensional image data using a preset commodity detection model; cropping the image data of each commodity from the two-dimensional image data based on the coordinate information of each commodity; and determining the category information of each commodity based on the image data of each commodity using a preset commodity classification model.

[0007] In some embodiments, the training set of the preset product detection model includes at least one two-dimensional image data training set with coordinate information of each product and a SKU-110K retail product data set; the training set of the preset product classification model includes at least two image data training sets of each product with category information marked at different rotation angles.

[0008] In some embodiments, the coordinate information includes the horizontal coordinates and vertical coordinates of at least two diagonal points of the bounding box where each product is located; based on the coordinate information of each product, the hierarchical information of each product is determined, including: based on the vertical coordinates of at least two diagonal points of the bounding box where each product is located, the hierarchical sequence of each product is determined; based on the horizontal coordinates and vertical coordinates of at least two diagonal points of the bounding box where each product is located, the intra-layer arrangement sequence and the intra-layer stacking sequence of each product are determined respectively.

[0009] In some embodiments, the hierarchical sequence of each commodity is determined based on the vertical coordinates of at least two diagonal points of the bounding box where each commodity is located, including: determining the projection coordinate information of each commodity on a reference coordinate system based on the vertical coordinates of the upper left corner and the lower right corner of the bounding box where each commodity is located; determining the hierarchical sequence of each commodity based on the projection coordinate information of each commodity; wherein the reference coordinate system is a coordinate system with the upper left corner of the shelf as the origin.

[0010] In some embodiments, based on the horizontal coordinates and vertical coordinates of at least two diagonal points of the bounding box where each product is located, the in-layer arrangement sequence and the in-layer stacking sequence of each product are determined respectively, including: based on the horizontal coordinates of the upper left corner and the horizontal coordinates of the lower right corner of the bounding box where each product is located, determining the in-layer arrangement sequence of each product; based on the vertical coordinates of the upper left corner and the vertical coordinates of the lower right corner of the bounding box where each product is located, determining the in-layer stacking sequence of each product.

[0011] In some embodiments, the display status of the shelf includes at least one of the following: normal display, wrong display, and missing display.

[0012] In a second aspect, an embodiment of the present disclosure provides a display detection device for a shelf, comprising: an acquisition unit, for acquiring two-dimensional image data of a shelf to be detected from a frontal orientation; a determination unit, for determining coordinate information and category information of each commodity in the two-dimensional image data; based on the coordinate information of each commodity, determining the hierarchical information of each commodity, wherein the hierarchical information includes a hierarchical sequence, an arrangement sequence within a layer, and a stacking sequence within a layer; and for determining the display information of the shelf based on the category information and hierarchical information of each commodity, wherein the display information includes a mapping relationship between each commodity and the category information and the hierarchical information; a judgment unit, for displaying information, and detecting the display status of the shelf according to a predefined display standard.

[0013] In a third aspect, an embodiment of the present disclosure provides an electronic device, comprising: a memory; and a processor, wherein the processor is configured to execute the shelf display detection method provided in the first aspect or any embodiment of the first aspect.

[0014] In a fourth aspect, an embodiment of the present disclosure provides a machine-readable storage medium having instructions stored thereon, the instructions being used to enable a machine to execute the shelf display detection method provided in the first aspect or any one of the embodiments of the first aspect.

[0015] Through the above technical solution, the disclosed embodiment can accurately identify and effectively associate the category and coordinate information of the goods, successfully obtain the display information of the shelf, and then perform display detection based on the display information, thereby improving the accuracy and management efficiency of shelf display detection.

[0016] Other features and advantages of the embodiments of the present disclosure will be described in detail in the subsequent detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings are used to provide a further understanding of the embodiments of the present disclosure and constitute a part of the specification. Together with the following specific embodiments, they are used to explain the embodiments of the present disclosure but do not constitute a limitation on the embodiments of the present disclosure. In the accompanying drawings:

[0018] Figure 1 It is a flow chart provided according to the first embodiment of the shelf display detection method disclosed in the present invention;

[0019] Figure 2 It is a flow chart provided according to the second embodiment of the shelf display detection method disclosed in the present invention;

[0020] Figure 3 It is a flow chart provided according to the third embodiment of the shelf display detection method disclosed in the present invention;

[0021] Figure 4 This is a schematic diagram of a process for determining the hierarchical sequence of each commodity provided in the third embodiment of the present disclosure;

[0022] Figure 5 This is a schematic diagram of a process for determining the intra-layer arrangement sequence and the intra-layer stacking sequence of each commodity provided in the third embodiment of the present disclosure;

[0023] Figure 6 It is a structural schematic diagram of a shelf display detection device provided according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0024] The specific implementation of the embodiment of the present disclosure is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described here is only used to illustrate and explain the embodiment of the present disclosure, and is not used to limit the embodiment of the present disclosure.

[0025] With the rise of e-commerce and changes in consumer shopping habits, retailers need to quickly adapt to the new market environment, improve operational efficiency, reduce errors and lower costs. Given the current urgent need for efficiency and accuracy in the retail industry, especially in display inspection, inventory management, and customer experience, the use of computer vision technology to achieve automated product detection and identification has become a hot topic. This technology has become an indispensable tool for modern retail by providing real-time and accurate product information processing capabilities, reducing omissions in manual inspections, and improving store operations. However, some current technologies have special requirements for image acquisition equipment and acquisition methods, so the flexibility, scalability and accuracy of a single method when applied to actual scenarios are limited.

[0026] For example, patent CN113743382A uses mobile devices to collect real images of shelves, uses transfer learning methods to improve general target detection and image classification models, and uses K-means methods to cluster shelf products based on the product detection results to stratify shelf products, ultimately achieving shelf product detection. However, this method requires manual or semi-automatic processing before product detection and identification.

[0027] For another example, patent CN116109992A obtains product information by positioning and globally monitoring the shelves, and then classifies and segments the products by row, and combines OCR information, image and color features to identify the products. However, the OCR text detection and recognition process has high requirements for image clarity, so this method has high requirements for the performance of the image acquisition device and the image acquisition quality.

[0028] Therefore, the existing shelf detection methods rely on manual inspections or simple image processing technology, which makes it difficult to accurately identify the categories and specific coordinates of goods, and fail to effectively associate the categories and coordinate information of goods, thereby affecting the accuracy of shelf display detection and management efficiency.

[0029] Based on this, in the first aspect, the present disclosure provides a shelf display detection method, referring to Figure 1 As shown, Figure 1 It is a flow chart provided according to the first embodiment of the shelf display detection method disclosed in the present invention.

[0030] like Figure 1 As shown, the shelf display detection method includes steps S101 to S105.

[0031] Step S101, acquiring two-dimensional image data of the shelf to be inspected from the front.

[0032] In the embodiments of the present disclosure, the specific source of shelf image acquisition is not limited, the sampling method is flexible, and it can be used in combination with a variety of hardware. It can adapt to different scenarios and different types of shelves such as large supermarkets and retail convenience stores, and has wide applicability to a variety of commercial scenarios.

[0033] For example, the front image of the shelf can be automatically or manually captured by a camera, which needs to have a high-resolution camera to obtain high-definition images of the goods on the shelf. The camera should be able to work under different lighting conditions and can capture clear and complete shelf images by adjusting the focus and angle.

[0034] The disclosed embodiment requires that the shelf be photographed completely and clearly without including other useless shelves. Therefore, after acquiring the shelf image data, there is no need to manually or semi-automatically frame the shelf detection area in advance.

[0035] Step S102, determining the coordinate information and category information of each commodity in the two-dimensional image data.

[0036] The preset commodity detection model and the preset commodity classification model are used to determine the coordinate information and category information of each commodity in the two-dimensional image data. The coordinate information includes the horizontal coordinates and vertical coordinates of at least two diagonal points of the boundary box where each commodity is located, and the category information is the category to which each commodity belongs.

[0037] It should be noted that all commodity detection and identification in the embodiments of the present disclosure only consider the first row of the shelf image, without considering the depth of the shelf and the commodity.

[0038] Step S103: determining the level information of each commodity based on the coordinate information of each commodity.

[0039] The hierarchical information may include a hierarchical sequence, an intra-layer arrangement sequence, and an intra-layer stacking sequence.

[0040] Specifically, the hierarchical sequence indicates the vertical hierarchical order of the commodities on the shelf; the intra-layer arrangement sequence indicates the left-right arrangement order of the commodities in the same layer; and the intra-layer stacking sequence indicates the top-bottom stacking order of the commodities in the same layer.

[0041] Step S104, determining shelf display information based on the category information and level information of each commodity.

[0042] The display information includes the mapping relationship between each product and the category information and level information.

[0043] Specifically, on the basis of obtaining the coordinate information and category information of a single product, the coordinate information of all products on the shelf is used for lateral projection to obtain the basic hierarchical structure of the shelf product display, which is calibrated in combination with the standard shelf display information to finally obtain the mapping relationship between each product on the shelf and the category information and hierarchical information.

[0044] Step S105 , based on the display information and according to predefined display standards, the display status of the shelf is detected.

[0045] In this step, the merchandise arrangement can be compared in layers according to the differentiated merchandise and its corresponding shelf layers. On each layer, the actual merchandise arrangement is compared with the standard display arrangement, where the standard display arrangement specifies the merchandise categories to be placed and their order coordinates.

[0046] Specifically, the predefined display standards may include: standard commodity categories, standard intra-layer sequences, standard arrangement sequences, and standard stacking sequences.

[0047] In some embodiments, the display status of the shelf includes at least one of the following: normal display, display error, and display missing. Specifically, if it is identified that the actual display arrangement of the shelf contains goods that are not required to be placed in the standard display arrangement (extra display), then the display of the goods is determined to be wrong; if the category and order coordinates of the goods required in the standard display arrangement do not appear in the actual display of the goods, or the order coordinates do not meet the requirements of the standard display arrangement, then it is determined to be missing.

[0048] Among them, normal display may include: category information of commodities in the same level sequence is equal to the standard commodity category, the arrangement sequence within each layer of the commodity is equal to the standard arrangement sequence, and the stacking sequence within each layer of the commodity is equal to the standard stacking sequence.

[0049] Display missing may include at least one of the following: the number of category information of commodities in the same level sequence is less than the number of standard commodity categories; the maximum intra-layer arrangement sequence of commodities in the same level sequence is less than the maximum standard arrangement sequence; the intra-layer stacking sequence of at least one commodity in the same level sequence is less than the standard stacking sequence.

[0050] Display errors may include at least one of the following: the number of category information of products in the same level sequence is greater than the number of standard product categories; the maximum intra-layer arrangement sequence of products in the same level sequence is greater than the maximum standard arrangement sequence; the intra-layer stacking sequence of at least one product in the same level sequence is greater than the standard stacking sequence.

[0051] For example, if the category information of commodities in the same level sequence is equal to the standard commodity category, but the maximum intra-level arrangement sequence of commodities in the same level sequence is greater than the maximum standard arrangement sequence, the display state of the detected shelf is a display error.

[0052] For another example, if the number of category information of commodities in the same level sequence is less than the number of standard commodity categories, and the stacking sequence of two commodities in the same level is less than the standard stacking sequence, the display status of the detected shelf is display missing.

[0053] The disclosed embodiment first uses a camera to obtain images of shelves and commodities, and then performs target detection of commodities to obtain images and coordinates of individual commodities. Feature extraction is then performed on individual commodity images, and the images are compared with the data in the feature database to achieve commodity classification. Based on the coordinate information and category information of the commodities, the shelf hierarchy structure and its correspondence with the commodities are determined in combination with the standard display information of the shelves, and the shelf display information is obtained. After comparison with the standard display information, compliance judgment of the shelf display is achieved.

[0054] The shelf display detection method provided by the embodiment of the present disclosure can accurately identify and effectively associate the category and coordinate information of the goods, successfully obtain the display information of the shelf, and then perform display detection based on the display information, thereby improving the accuracy and management efficiency of shelf display detection.

[0055] In the first aspect, based on the first embodiment, the second embodiment of the present disclosure provides a shelf display detection method, referring to Figure 2 As shown, Figure 2 It is a flow chart provided according to the second embodiment of the shelf display detection method disclosed in the present invention.

[0056] In a possible implementation, Figure 2 As shown, determining the coordinate information and category information of each commodity in the two-dimensional image data includes steps S201 to S203.

[0057] Step S201 : Based on the two-dimensional image data, the coordinate information of each commodity is determined using a preset commodity detection model.

[0058] In some embodiments, the training set of the preset commodity detection model includes at least one two-dimensional image data training set with coordinate information of each commodity annotated and a SKU-110K retail commodity data set.

[0059] For example, a dataset consisting of real shelf images containing products is used to annotate the products in the images and train a product detection model. The product detection model is used to infer the collected images and obtain the coordinate information of all products in the collected images. The model can handle shelf products of different sizes, shapes, and arrangements, and is robust to the orientation, rotation, and partial occlusion of the products.

[0060] In a feasible implementation manner, the preset commodity detection model is trained using the following steps.

[0061] (1) Based on the YOLOv8 pre-trained model, the SKU110K dataset is first used for incremental training to improve the performance of the model in shelf product detection.

[0062] (2) Collect real convenience store shelf images, manually annotate products, and build a convenience store shelf image dataset. Combine this with a small portion of the SKU110K dataset to form a training dataset. Use this data to continue training and further improve the accuracy of the model on convenience store shelf product images.

[0063] Among them, YOLOv8 includes a series of real-time target detection models, whose main features include advanced Backbone and Neck architectures, optimized trade-offs between accuracy and speed, and a variety of pre-trained models to meet various tasks and performance requirements. The SKU-110K dataset is a collection of retail shelf images designed to support research on target detection tasks. The dataset contains more than 110,000 unique stock keeping unit (SKU) categories, which often look similar or even identical and are placed closely together.

[0064] The commodity detection model disclosed in this paper is fine-tuned and trained based on the YOLOv8 pre-trained model. YOLOv8 demonstrates excellent performance in the case of rotation, direction change and partial occlusion through special network structure design and the introduction of SEAM attention module.

[0065] Based on the pre-trained model, this paper combines public data sets with real store shelves and product images for incremental training and tuning. It is more targeted in small target detection and dense product detection, and is more suitable for product detection on shelves.

[0066] It should be noted that small object detection aims to accurately identify and locate objects that are small in size or occupy fewer pixels in the image. For example, in the COCO dataset, small objects are defined as objects whose bounding box resolution is less than 32 pixels × 32 pixels. Compared with large objects, small objects lack sufficient visual features, which makes them easily disturbed by background noise, thus affecting detection accuracy.

[0067] Dense object detection focuses on the presence of a large number of objects in an image, even overlapping. This scenario places higher demands on the resolution and processing mechanisms of the detection algorithm, especially when distinguishing and locating objects that are close to each other or partially occluded. The items on the shelf occupy smaller pixels than the entire shelf image, and the items are densely arranged and may overlap with each other, which belongs to this type of object detection task.

[0068] Step S202 : based on the coordinate information of each commodity, image data of each commodity is obtained by cutting out from the two-dimensional image data.

[0069] In some embodiments, the method may specifically include: determining a bounding box of each commodity based on the coordinate information of each commodity; and cropping the image data of each commodity from the two-dimensional image data based on the bounding box of each commodity.

[0070] The detected products are cropped out from the original collected images to facilitate subsequent input into the product classification model to extract features and compare them with the standard product feature data in the feature library to achieve automatic classification of the product images.

[0071] Step S203: Based on the image data of each commodity, the category information of each commodity is determined using a preset commodity classification model.

[0072] In some embodiments, the training set of the preset product classification model includes at least two training sets of image data of each product with labeled category information at different rotation angles.

[0073] For example, the labeled product images in the above shelf image dataset are cropped and labeled to form a product image dataset to train the product classification model. Through multiple evaluations and optimizations, the product classification model can better capture the detailed features of the product images to improve the classification accuracy and generalization ability of the model.

[0074] The disclosed embodiment uses a deep learning model to extract features of a single product image, including low-level features such as color, edge, texture, etc., as well as deep abstract features of the image, to obtain a 1*512-dimensional feature vector. This vector is then used to perform cosine similarity calculations with the feature vectors of all images in the feature library, and the image classification with the highest similarity is selected as the product category result of the image.

[0075] The present invention cuts the shelf image after commodity detection and directly uses a single commodity image for image feature extraction, thereby minimizing the impact of the surrounding environment, background and other commodities while retaining the commodity image features. At the same time, a commodity feature database is established and continuously enriched to ensure the comprehensiveness and accuracy of commodity classification.

[0076] In the first aspect, based on the first embodiment, the third embodiment of the present disclosure provides a shelf display detection method, referring to Figure 3 As shown, Figure 3 It is a flow chart provided according to the third embodiment of the shelf display detection method disclosed in the present invention.

[0077] In a possible implementation, Figure 3As shown, the coordinate information may include the horizontal coordinates and vertical coordinates of at least two diagonal points of the boundary box where each product is located, such as Figure 3 As shown, based on the coordinate information of each commodity, the level information of each commodity is determined, including steps S301 to S302.

[0078] Step S301 : determining the hierarchical sequence of each product based on the ordinates of at least two diagonal points of the boundary box where each product is located.

[0079] This step distinguishes which shelf level the product is located on by analyzing the vertical position of the product.

[0080] Step S302 : determining the intra-layer arrangement sequence and intra-layer stacking sequence of each product based on the horizontal coordinates and vertical coordinates of at least two diagonal points of the boundary box where each product is located.

[0081] This step involves analyzing the horizontal and vertical positions of the products to determine their relative position and stacking within the same layer.

[0082] A bounding box is defined as the smallest rectangle that contains an object and is usually used to locate and identify objects in an image. For a rectangular bounding box, the coordinates of the four vertices are usually used to define it, but in many cases, only the coordinates of the two diagonal points are sufficient to determine its edge position. However, different frameworks and applications may have different definitions, so when using coordinate information, it is necessary to determine it based on the specific situation. For example, the horizontal and vertical coordinates of the center point of the bounding box where the product is located can also be used to analyze the hierarchical sequence of the product.

[0083] In some embodiments, the horizontal coordinates and vertical coordinates of at least two diagonal points of the bounding box where each product is located may include: the horizontal coordinates and vertical coordinates of the upper left corner and the lower right corner, or the horizontal coordinates and vertical coordinates of the lower left corner and the upper right corner.

[0084] In a possible implementation, Figure 4 As shown, step S301 may include steps S401 to S402.

[0085] Step S401 : determining the projection coordinate information of each product on the reference coordinate system based on the ordinate of the upper left corner and the ordinate of the lower right corner of the boundary box where each product is located.

[0086] The reference coordinate system is a coordinate system with the upper left corner of the shelf as the origin.

[0087] Step S402: determining the hierarchical sequence of each commodity based on the projection coordinate information of each commodity.

[0088] For example, assume that the product coordinate information (x1, y1, x2, y2) is the coordinate of the upper left corner and lower right corner of the product. Take the upper left corner of the shelf image as the coordinate origin, and use the ordinate (y1, y2) in the coordinate information (x1, y1, x2, y2) of all the products on the shelf to be tested to project on the Y axis.

[0089] Since the goods are placed on different layers of the shelves, the vertical coordinates of the goods on the same layer will be in the same range. The shelf levels to which the goods belong are divided according to the different intervals divided by the coordinate projection on the Y-axis.

[0090] In another possible implementation, Figure 5 As shown, step S302 may include steps S501 to S502.

[0091] Step S501 : determining the intra-layer arrangement sequence of each product based on the horizontal coordinate of the upper left corner and the horizontal coordinate of the lower right corner of the boundary box where each product is located.

[0092] Step S502 : determining the stacking sequence of each product in the layer based on the ordinate of the upper left corner and the ordinate of the lower right corner of the boundary box where each product is located.

[0093] For example, assuming that the product coordinate information (x1, y1, x2, y2) is the coordinates of the upper left corner and the lower right corner of the product, the left-right arrangement order of the products is determined according to the horizontal coordinate (x1, x2) in the product coordinate information (x1, y1, x2, y2); the stacking arrangement is determined according to the vertical coordinate (y1, y2).

[0094] The disclosed embodiments improve the efficiency and accuracy of display detection and enhance the real-time monitoring capability of commodity placement by automatically analyzing the coordinate information and hierarchical sequence of commodities.

[0095] Based on this, in a second aspect, the present disclosure provides a shelf display detection device, referring to Figure 6 As shown, Figure 6 It is a structural schematic diagram of a shelf display detection device provided according to an embodiment of the present disclosure.

[0096] like Figure 6 As shown, the shelf display detection device 100 may include an acquisition unit 110 , a determination unit 120 and a judgment unit 130 .

[0097] The acquisition unit 110 is used to acquire two-dimensional image data of the shelf to be inspected from the front.

[0098] The determination unit 120 is used to determine the coordinate information and category information of each commodity in the two-dimensional image data; determine the level information of each commodity based on the coordinate information of each commodity; and determine the display information of the shelf based on the category information and level information of each commodity.

[0099] The hierarchical information includes hierarchical sequence, intra-layer arrangement sequence and intra-layer stacking sequence; the display information includes the mapping relationship between each commodity and the category information and hierarchical information.

[0100] The judging unit 130 is used for displaying information and detecting the display status of the shelf according to a predefined display standard.

[0101] The shelf display detection device provided in the embodiment of the present disclosure adopts the shelf display detection method in the above embodiment, which can solve the technical problems in the background technology.

[0102] The beneficial effects of the shelf display detection device provided in the present disclosure are the same as the beneficial effects of the shelf display detection method provided in the above-mentioned embodiment, and other technical features in the shelf display detection device are the same as the features disclosed in the shelf display detection method, which will not be repeated here.

[0103] Based on this, in a third aspect, an embodiment of the present disclosure provides an electronic device, which includes: a memory; and a processor, wherein the processor is configured to execute the shelf display detection method provided in the above embodiment.

[0104] In a typical configuration, a device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0105] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0106] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0107] The electronic device provided by the embodiment of the present disclosure adopts the shelf display detection method in the above embodiment, which can solve the technical problems in the background technology.

[0108] The beneficial effects of the electronic device provided by the embodiment of the present disclosure are the same as the beneficial effects of the shelf display detection method provided by the above embodiment, and other technical features in the device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0109] Based on this, in a fourth aspect, an embodiment of the present disclosure provides a machine-readable storage medium, on which instructions are stored, and the instructions are used to enable a machine to execute the shelf display detection method provided in the above-mentioned first aspect or any embodiment of the first aspect.

[0110] The beneficial effects of the machine-readable storage medium provided in the present disclosure are the same as the beneficial effects of the shelf display detection method provided in the above-mentioned embodiment, and will not be elaborated here.

[0111] The embodiment of the present disclosure also provides a computer program product, including a computer program, which implements the steps of the shelf display detection method as described above when executed by a processor.

[0112] The beneficial effects of the computer program product provided by the embodiments of the present disclosure are the same as the beneficial effects of the shelf display detection method provided by the above embodiments, and are not elaborated here.

[0113] Those skilled in the art should understand that the embodiments of the present disclosure can be provided as a method, a system, or a computer program product. Therefore, the present disclosure can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present disclosure can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0114] The present disclosure is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, as well as the combination of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to the processors of general-purpose computers, special-purpose computers, embedded processors, or other programmable data processing devices to generate a machine, such that the instructions executed by the processors of the computer or other programmable data processing devices generate means for implementing the functions specified in one Figure 1 flow or multiple flows and / or blocks Figure 1 or multiple blocks.

[0115] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including instruction means that implement the functions specified in one Figure 1 flow or multiple flows and / or blocks Figure 1 or multiple blocks.

[0116] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one Figure 1 flow or multiple flows and / or blocks Figure 1 or multiple blocks.

[0117] It should be noted that although expressions such as "first" and "second" are used in this document to describe different modules, steps, and data of the embodiments of the present disclosure, the expressions "first" and "second" are only used to distinguish between different modules, steps, and data, and do not represent a specific order or importance. In fact, the expressions "first" and "second" can be used interchangeably.

[0118] Although operations are described in a particular order in the drawings, this should not be understood as requiring that the operations be performed in the particular order shown or in serial order, or that all shown operations be performed to achieve desired results. In certain circumstances, multitasking and parallel processing may be advantageous.

[0119] The acquisition, transmission, storage, use, and processing of data in the embodiments of the present disclosure are in compliance with the relevant provisions of national laws and regulations.

[0120] It should be noted that in the embodiments of the present disclosure, certain software, components, models and other existing solutions in the industry may be mentioned, which should be regarded as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of the present disclosure, but it does not mean that the applicant has or will necessarily use the solution.

[0121] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0122] The above are only embodiments of the present disclosure and are not intended to limit the present disclosure. For those skilled in the art, the present disclosure may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present disclosure shall be included in the scope of the claims of the present disclosure.

Claims

1. A method for detecting shelf display, characterized in that: include: Acquire two-dimensional image data of the shelf to be inspected from the front side; Determining coordinate information and category information of each commodity in the two-dimensional image data; Based on the coordinate information of each commodity, determining the hierarchical information of each commodity, wherein the hierarchical information includes a hierarchical sequence, an arrangement sequence within a layer, and a stacking sequence within a layer; Based on the category information and level information of each commodity, determining the display information of the shelf, wherein the display information includes a mapping relationship between each commodity and the category information and the level information; Based on the display information and according to predefined display standards, the display status of the shelf is detected.

2. The shelf display detection method according to claim 1, characterized in that: The determining the coordinate information and category information of each commodity in the two-dimensional image data includes: Based on the two-dimensional image data, using a preset commodity detection model, determining the coordinate information of each commodity; Based on the coordinate information of each commodity, clipping the two-dimensional image data to obtain image data of each commodity; Based on the image data of each commodity, the category information of each commodity is determined using a preset commodity classification model.

3. The shelf display detection method according to claim 2, characterized in that: The training set of the preset commodity detection model includes at least one two-dimensional image data training set with coordinate information of each commodity and a SKU-110K retail commodity data set; the training set of the preset commodity classification model includes at least two image data training sets of each commodity with category information marked at different rotation angles.

4. The shelf display detection method according to claim 1, characterized in that: The coordinate information includes the horizontal coordinates and the vertical coordinates of at least two diagonal points of the boundary box where each product is located; The step of determining the level information of each commodity based on the coordinate information of each commodity includes: Determine the hierarchical sequence of each product based on the ordinates of at least two diagonal points of the bounding box where each product is located; Based on the horizontal coordinates and the vertical coordinates of at least two diagonal points of the boundary box where each product is located, the intra-layer arrangement sequence and the intra-layer stacking sequence of each product are determined respectively.

5. The shelf display detection method according to claim 4, characterized in that: The step of determining the hierarchical sequence of each commodity based on the ordinate coordinates of at least two diagonal points of the bounding box where each commodity is located comprises: Determine the projection coordinate information of each commodity on the reference coordinate system based on the ordinate of the upper left corner and the ordinate of the lower right corner of the boundary box where each commodity is located; Determine the hierarchical sequence of each commodity based on the projection coordinate information of each commodity; The reference coordinate system is a coordinate system with the upper left corner of the shelf as the origin.

6. The shelf display detection method according to claim 4, characterized in that: Based on the horizontal coordinates and the vertical coordinates of at least two diagonal points of the boundary box where each product is located, respectively determining the intra-layer arrangement sequence and the intra-layer stacking sequence of each product, including: Determine the intra-layer arrangement sequence of each product based on the horizontal coordinate of the upper left corner and the horizontal coordinate of the lower right corner of the boundary box where each product is located; Based on the ordinate of the upper left corner and the ordinate of the lower right corner of the bounding box where each product is located, the in-layer stacking sequence of each product is determined.

7. The shelf display detection method according to claim 1, characterized in that: The display status of the shelf includes at least one of the following: normal display, wrong display, and missing display.

8. A shelf display detection device, characterized in that: include: An acquisition unit, used for acquiring two-dimensional image data of the shelf to be inspected from the front; a determination unit, used to determine coordinate information and category information of each commodity in the two-dimensional image data; Based on the coordinate information of each commodity, determining the level information of each commodity, wherein the level information includes a level sequence, an arrangement sequence within a layer, and a stacking sequence within a layer; and determining the display information of the shelf based on the category information and the level information of each commodity, wherein the display information includes a mapping relationship between each commodity and the category information and the level information; The judging unit is used for the display information to detect the display status of the shelf according to a predefined display standard.

9. An electronic device, characterized in that: The electronic device comprises: Memory; and A processor, wherein the processor is configured to execute the shelf display detection method according to any one of claims 1 to 7.

10. A machine-readable storage medium, characterized in that: The machine-readable storage medium stores instructions for causing the machine to execute the shelf display detection method described in any one of claims 1 to 7.

Images