Method and apparatus for generating shelf display areas

Abstract

The application discloses a method and device for generating a shelf display shelf area, and the method comprises the following steps: obtaining a shelf image to be detected and recognized; inputting the shelf image to be detected and recognized into a pre-trained shelf detection model to detect the shelf image to be detected and recognized and obtain a shelf display shelf area bounding box; the shelf detection model is pre-trained according to a historical shelf image data set and a corresponding shelf area bounding box label data set sample; and outputting the detected shelf display shelf area bounding box information. The shelf detection model can be used to directly detect possible shelf areas on a shelf image, so that the efficiency and accuracy of the shelf area generation are improved.

Description

Technical Field

[0001] This invention relates to the field of intelligent shelving technology, and in particular to a method and apparatus for generating shelving display areas. Background Technology

[0002] This section is intended to provide background or context for the embodiments of the invention set forth in the claims. The description herein is not an admission that it is prior art simply because it is included in this section.

[0003] With the continuous development of artificial intelligence (AI) technology, the supermarket industry is also constantly transforming towards intelligence and digitalization. The application of AI technology enables supermarkets to better meet customer needs, improve operational efficiency, enhance user experience, and further expand their market influence. Through the use of AI algorithms, supermarkets can analyze and predict consumer demand in real time, more accurately formulate marketing strategies to meet consumer needs, thereby increasing sales and customer satisfaction. Supermarkets can also use AI technology to automate inventory management, reduce inventory costs and shelf space, and quickly replenish goods when necessary, ensuring that customer shopping needs are met.

[0004] Smart shelves are a new type of intelligent retail tool with intelligent analysis and automated management functions, capable of meeting various needs of supermarkets and hypermarkets, such as shelf layout analysis and product identification. By monitoring and analyzing products on the shelves in real time, smart shelves can improve shelf utilization, optimize inventory management, and enhance the customer shopping experience, thereby improving the operational efficiency of supermarkets and hypermarkets. Smart shelves have become an important component of intelligent retail in supermarkets and hypermarkets and represent a significant trend in the future development of supermarkets and hypermarkets.

[0005] In smart shelves, shelf display areas are a crucial component of smart shelf management, as they are used for subsequent display supervision and out-of-stock management. However, generating accurate display areas for multiple products on each shelf is currently a challenging problem in smart shelf systems. Common methods include manually configuring these areas in the backend system, which is highly inefficient, and manually configured areas are generally fixed, difficult to maintain, and often do not match the actual products on the shelves. Another method is to use the position of each price tag on the shelf to help generate display areas, but the actual placement of price tags may deviate from the actual products, or missing display areas may occur due to missed price tags, resulting in inaccurate display areas. Summary of the Invention

[0006] This invention provides a method for generating shelf display area, which can efficiently and accurately generate shelf display area. The method includes:

[0007] Acquire images of the shelves to be detected and identified;

[0008] The shelf image to be detected and identified is input into the pre-trained shelf detection model, and the shelf image to be detected and identified is used to obtain the bounding box of the shelf display shelf area; the shelf detection model is pre-trained and generated based on the historical shelf image dataset and the corresponding shelf area bounding box label dataset samples.

[0009] Output the bounding box information of the detected shelf display area.

[0010] This invention also provides an apparatus for generating shelf display area, for efficiently and accurately generating shelf display area, the apparatus comprising:

[0011] The acquisition unit is used to acquire images of the shelves to be detected and identified.

[0012] The detection unit is used to input the shelf image to be detected and identified into the pre-trained shelf detection model, and to detect the shelf image to be detected and identify to obtain the shelf region bounding box; the shelf detection model is pre-trained and generated based on the historical shelf image dataset and the corresponding shelf region bounding box label dataset samples.

[0013] The generation unit is used to output the bounding box information of the detected grid area.

[0014] This invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the above-described method for generating shelf display areas.

[0015] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for generating shelf display areas.

[0016] This invention also provides a computer program product, which includes a computer program that, when executed by a processor, implements the above-described method for generating shelf display areas.

[0017] In this embodiment of the invention, the method for generating shelf display grid areas differs from existing technologies that rely on manual division in a backend system (which is inefficient) or use the position of each price tag on the shelf to generate grids (which can lead to inaccuracies). This method involves: acquiring a shelf image to be detected; inputting the shelf image into a pre-trained grid detection model to detect the shelf image and obtain the bounding boxes of the shelf display grid areas; the grid detection model being pre-trained based on historical shelf image datasets and corresponding grid area bounding box label datasets; and outputting the detected shelf display grid area bounding box information. This invention allows the grid detection model to directly detect possible grid areas on a shelf image, improving the efficiency and accuracy of grid area generation. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings:

[0019] Figure 1 This is a flowchart illustrating the method for generating shelf display areas in an embodiment of the present invention.

[0020] Figure 2 This is a schematic diagram of the marked grid area in an embodiment of the present invention. Figure 1 ;

[0021] Figure 3 This is a schematic diagram of the marked grid area in an embodiment of the present invention. Figure 2 ;

[0022] Figure 4 This is a schematic diagram of the marked grid area in an embodiment of the present invention. Figure 3 ;

[0023] Figure 5 This is a schematic diagram illustrating the location of the goods in an embodiment of the present invention;

[0024] Figure 6a This is a schematic diagram of the grid boundary frame before correction and adjustment in an embodiment of the present invention;

[0025] Figure 6b This is a schematic diagram showing the modified and adjusted grid boundary frame in an embodiment of the present invention;

[0026] Figure 7 This is a schematic diagram of the device for generating shelf display area in an embodiment of the present invention. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0028] To better understand this invention, the terms used in the embodiments of this invention will first be introduced.

[0029] Product bounding box: A rectangular frame that perfectly encloses the product display area, such as... Figure 6b The "1" in the text.

[0030] Merchandise shelf area: In a supermarket, when the shelves are full, the area occupied by the same product.

[0031] Shelf boundary frame: such as Figure 6b The "2" in the text.

[0032] Shelf Image Dataset: A collection of multiple images, each taken of the surface of a shelf.

[0033] Product Image Dataset: A collection of multiple images, each a small image of a single product.

[0034] Shelf Boundary Box Label Dataset: A collection of information describing the boundaries of multiple rectangular boxes in multiple images. Each image is taken of the shelf surface, and each rectangle represents the extent of each product shelf area.

[0035] Product bounding box label dataset: A collection of information describing the boundaries of multiple rectangles in multiple images. Each image is taken of the shelf surface, and each rectangle represents the range of each product shelf.

[0036] Product category label dataset: A collection describing category information for multiple product images.

[0037] In view of the technical problems existing in the prior art, embodiments of the present invention provide a solution for generating shelf display area, which can obtain accurate product shelf space in intelligent shelves and digital display systems. The solution for generating shelf display area is described in detail below.

[0038] Figure 1 This is a flowchart illustrating the method for generating shelf display areas in an embodiment of the present invention, as shown below. Figure 1 As shown, the method includes the following steps:

[0039] Step 101: Obtain the image of the shelf to be detected and identified;

[0040] Step 102: Input the shelf image to be detected and identified into the pre-trained shelf detection model to detect the shelf image and obtain the bounding box of the shelf display area; the shelf detection model is pre-trained and generated based on the historical shelf image dataset and the corresponding shelf area bounding box label dataset samples.

[0041] Step 103: Output the bounding box information of the detected shelf display area.

[0042] The method for generating shelf display area provided in this embodiment of the invention operates as follows: acquiring a shelf image to be detected and identified; inputting the shelf image to be detected and identified into a pre-trained shelf detection model to detect the shelf image and obtain the shelf display area bounding box; the shelf detection model is pre-trained and generated based on historical shelf image datasets and corresponding shelf area bounding box label dataset samples; and outputting the detected shelf display area bounding box information.

[0043] Compared to existing technologies that involve inefficient manual division in a backend system or rely on the position of each price tag on the shelf to generate shelf compartments, resulting in inaccurate compartment generation, the method for generating shelf display compartment areas provided in this invention can directly detect possible compartment areas on a shelf image using a compartment detection model, thereby improving the efficiency and accuracy of compartment area generation. A detailed description follows.

[0044] This invention provides a method for generating shelf display area, which can obtain accurate product shelf space in smart shelves and digital display systems. The detailed steps are as follows:

[0045] 1. Capture shelf images

[0046] Acquire the shelf images to be detected and identified (historical shelf images can be used for modeling), and collect them to form a shelf image dataset and a product image dataset.

[0047] 2. Train to obtain the grid detection model

[0048] Each image contains labeled product grids to form a grid bounding box label dataset. This dataset is then used to train a grid detection model. The specific method is as follows:

[0049] 1) In the image dataset, for each image, the possible placement area for each product is defined as the shelf for that product, and the location label information is manually defined and marked, which finally forms a shelf bounding box label information set;

[0050] Specific, illustrative methods for defining product placement areas are as follows: For a certain SKU product, such as... Figure 2For goods A and B, the areas where each type of goods is grouped and arranged together are marked as two separate sheds:

[0051] Specifically, if B shorts the stock alone, such as... Figure 3 As shown, a possible shelf area is marked manually based on experience and by referring to the product information and shelf information on both sides nearby.

[0052] In a more unusual scenario, several items are listed side-by-side as sold-out, such as... Figure 4 As shown, several possible grid areas were manually marked based on experience;

[0053] Build a deep neural network model, in which the backbone network used by the deep neural network model can be, but is not limited to, CNN-type models or VIT-type models, and the specific detection method can be, but is not limited to, one-stage YOLO series detection methods or two-stage Faster-RCNN methods.

[0054] 2) Using the above-mentioned shelf image dataset and corresponding shelf bounding box label dataset, the network model is trained based on the idea of ​​regression and the gradient backpropagation optimization method to obtain the shelf detection model, which can predict all possible shelf bounding boxes on the shelf.

[0055] As can be seen from the above, in one embodiment, the method for generating shelf display area may further include: pre-training a shelf detection model according to the following method:

[0056] Acquire historical shelf images and collect them to form a historical shelf image dataset;

[0057] For each image in the shelf image dataset, the possible placement area of ​​each product in the image is defined as the shelf cell for that product. The location label information of the shelf cell is marked, and finally the shelf cell bounding box label dataset is formed.

[0058] Build a deep neural network model;

[0059] Using the shelf image dataset and the corresponding shelf bounding box label dataset, the deep neural network model is trained based on the idea of ​​regression and the gradient backpropagation optimization method to obtain the shelf detection model, which is used to predict the bounding boxes of all possible shelf areas on the shelf.

[0060] 3. Train to obtain a product detection model

[0061] 1) Label the location of the product in each image (e.g., Figure 5 The bounding boxes of the products are used to form a product bounding box label dataset;

[0062] 2) Train a product detection model using the above shelf image dataset and product bounding box label dataset to obtain the product detection model. This model can predict all possible product bounding boxes on the shelf. The detailed method is the same as described in the previous steps.

[0063] As can be seen from the above, in one embodiment, the method for generating shelf display areas may further include: pre-training a product detection model according to the following method:

[0064] Acquire historical shelf images and collect them to form a product image dataset;

[0065] For each image in the product image dataset, the location of each product in the image is labeled to form a product bounding box label dataset;

[0066] Using the product image dataset and the corresponding product bounding box label dataset, a pre-built deep neural network model is trained to obtain a product detection model, which is used to predict all possible product bounding boxes on the shelf.

[0067] 4. Train to obtain a product recognition model

[0068] 1) For the product image dataset, each product image is labeled as a separate category, ultimately forming a product category label dataset;

[0069] 2) Build a deep neural network model. The backbone network used in this model can be, but is not limited to, CNN-type models or VIT-type models. A classifier is connected after the backbone network model.

[0070] 3) Using the product image dataset and product category label dataset, the network model is trained based on the idea of ​​maximum likelihood and the loss function of metric learning. Two models can be trained here: a product category recognition model and a product feature description recognition model. The input of the product category recognition model can be an image (a small image of the product), and the output can be a product category. The input of the product feature description recognition model can be an image (a small image of the product), and the output can be an image feature description.

[0071] 4) The final model can predict the possible category or image feature description for each input image. That is, the product category recognition model outputs the product category, and the product category recognition model outputs the feature description.

[0072] As can be seen from the above, in one embodiment, the method for generating shelf display areas may further include: pre-training the product category recognition model according to the following method:

[0073] For a product image dataset collected in advance based on historical shelf images, each product image is labeled as a separate product category, ultimately forming a product category label dataset;

[0074] Build a deep neural network model;

[0075] Using a product image dataset and a product category label dataset, a deep neural network model is trained based on the idea of ​​maximum likelihood and the loss function of metric learning to obtain the product category recognition model.

[0076] In practice, the category identified by the product category recognition model can be used to identify the information corresponding to the generated grid area. Alternatively, as described in the following example, the identified product information can be used to adjust the grid area to further improve the accuracy of grid area generation.

[0077] 5. Inspect the shed / grid.

[0078] After training the shelf detection model described above, the model is deployed in the system and can wait for the input image to be inferred. Specifically, the shelf detection model trained in the previous steps is used to infer and predict the real-time input image to obtain the placement area of ​​each product on the shelf, i.e. the bounding box of the shelf area.

[0079] 6. Inspect the goods.

[0080] After training the product detection model described above, the model is deployed in the system and can wait for the input image to be used for inference. Specifically, the product detection model is used to detect each product on the shelf and obtain the bounding box of each product's location.

[0081] 7. Identify or classify goods.

[0082] Based on the product bounding box in step 6, crop out a small image of each product from the large image;

[0083] Each product thumbnail is identified or categorized; only one method needs to be selected. The specific method is as follows:

[0084] a) Identification:

[0085] Using the product category recognition model trained in step 4, each cropped small image is fed into the product category recognition model, and the model's classifier directly outputs the specific category. Finally, each product's bounding box has a product category information.

[0086] b) Classification:

[0087] 1) Using the product feature description recognition model trained in step 4, each cropped small image is fed into the product feature description recognition model, and the backbone of the model outputs the feature description of the image.

[0088] 2) For each product on the shelf, compare its corresponding feature description with the feature descriptions of nearby products. Specifically, calculate, but not limited to, the Euclidean distance or cosine distance between feature vectors (this step can be skipped for products whose categories have already been confirmed). Generally, the feature descriptions of the same type of product have a high degree of similarity, so the same type of product can be classified or labeled into a separate category, such as product A, B, C, etc. Finally, label the bounding box of each product on the shelf as the classification result.

[0089] As can be seen from the above, in one embodiment, the method for generating shelf display area may further include:

[0090] A pre-trained product detection model is used to detect the shelf images to be identified and obtain product bounding boxes; the product detection model is pre-trained based on historical shelf image datasets and corresponding product bounding box label datasets.

[0091] Cut out a small image of each individual product from the bounding box of each detected product;

[0092] Each product's thumbnail is input into a pre-established product category recognition model to obtain the product category recognition result corresponding to the bounding box of each product; or, each product's thumbnail is input into a pre-established product feature description recognition model to obtain the classification result corresponding to the bounding box of each product.

[0093] As can be seen from the above, in one embodiment, inputting the small image of each product into a pre-established product feature description and recognition model for classification to obtain the product category corresponding to the bounding box of each product may include:

[0094] Each product's thumbnail is input into a pre-established product feature description recognition model to obtain the feature description of each product's thumbnail; the product feature description recognition model is used to predict the feature description of the input image.

[0095] For each product on the shelf, compare the feature description corresponding to that product with the feature description corresponding to the products near that product;

[0096] Products with a similarity higher than a preset threshold are classified as the same type of product. The same type of product is categorized or tagged into a separate category. Finally, the bounding box of each product on the shelf is tagged with the corresponding category as the classification result.

[0097] 8. Adjust and correct the shed grid.

[0098] Since the bounding boxes of the display shelves generated in step 5 may deviate from the actual placement of the goods, the results of the goods identification or classification in step 7 can be used to correct and adjust the bounding boxes, such as... Figure 6a Adjust to Figure 6b The effect.

[0099] The specific method is as follows:

[0100] 1) Determine the quantity of each SKU (Stock Keeping Unit) in each shelf. The SKU with the highest quantity is designated as the product information for that shelf. For example... Figure 6a The product information corresponding to shelf 21 in the diagram is A. Figure 6a The product information corresponding to shelf 22 in the diagram is B;

[0101] 2) Expand each shed outwards: Based on the shed (e.g., Figure 6a Find the product information corresponding to grid 21 in the grid, and locate the product (belonging to the grid) that is furthest from the left boundary of the grid. Figure 6a In section 21, the left-side bounding box of the item is used as the left-side border of the shelf, i.e., as shown in section 21. Figure 6b The left side of the shed 21 shown expands outward; locate the shed (e.g. Figure 6a If the item that is furthest away from the right boundary of grid 21 is found, the right boundary of that item is taken as the right border of the grid. If no item is found (for example, if all items outside the right boundary of grid 21 are item B, then it is considered that no item belonging to the grid has been found), the grid border is temporarily kept still. Record the change in the grid's outer edge.

[0102] 3) Shrink each shed: For each shed (e.g. Figure 6a In the shed 21), determine the sheds on both sides (e.g. Figure 6b If there is a record of outward expansion of grid 22 (e.g., grid 21 expanded outward on the left, but not on the right), and if the adjacent grid (grid 22) has expanded outward (as can be seen from step "2" above, the left boundary of grid 22 has expanded outward) and intersects with the current grid (the right boundary of grid 21 intersects with the left boundary of grid 22), then the current grid ( Figure 6a The right boundary frame of grid 21 in the middle is shrunk inward until it borders the adjacent grid without intersecting. After shrunk, it looks like... Figure 6b As shown.

[0103] As can be seen from the above, in one embodiment, the method for generating shelf display area may further include: adjusting the boundary frame of the shelf area according to the product category identification or classification results, so as to eliminate the deviation between the generated shelf area boundary frame and the actual product placement.

[0104] In practice, the boundary boxes of the grid area are corrected and adjusted based on the product category identification or classification results. This can eliminate the deviation between the generated grid area boundary boxes and the actual product placement, and further improve the accuracy of grid area generation.

[0105] As can be seen from the above, in one embodiment, adjusting the boundary box of the grid area based on the product category identification or classification results may include:

[0106] Determine the quantity of each type of product within each shelf area, and designate the product information with the highest quantity as the product information corresponding to that shelf area;

[0107] Perform an expansion operation on each shelf: Based on the product information corresponding to the current shelf, find the product that is furthest from the current shelf outside its left boundary, and use the left boundary frame of the product that is furthest from the current shelf outside its left boundary as the left border of the shelf; Based on the product information corresponding to the current shelf, find the product that is furthest from the current shelf outside its right boundary, and use the right boundary frame of the product that is furthest from the current shelf outside its right boundary as the right border of the shelf; If no product is found, keep the current shelf border unchanged for now; Record the expansion change of the current shelf.

[0108] Perform a shrinking operation on each grid cell: For each current grid cell, determine whether there are any expansion change records for the grid cells on both sides of the current grid cell. If it is determined from the expansion change records that the grid cells next to the current grid cell have expanded and intersected with the current grid cell, then shrink the boundary box of the current grid cell until it borders the adjacent grid cells and does not intersect with them, thus obtaining the result of correcting and adjusting the boundary box of the grid cell area.

[0109] In practice, the above-mentioned method of correcting and adjusting the boundary box of the grid area based on the product category identification or classification results can further improve the accuracy of grid area generation.

[0110] 9. Output grid results

[0111] The final output shelf style is a rectangular box. Each detected and trimmed bounding box is formatted with certain information, such as the coordinates and width and height of the bounding box at the top left corner or center point in the image: (x, y, w, h), or the coordinates of two diagonal points: (x1, y1, x2, y2). All shelf bounding box information in the shelf image is output to the subsequent cells.

[0112] As can be seen from the above, in one embodiment, outputting the detected shelf display area boundary box information may include:

[0113] The detected grid area boundary box information is converted into grid boundary box information in a preset format; wherein, the detected grid area boundary box is a rectangle.

[0114] Output the grid boundary information in the preset format to the subsequent use unit.

[0115] In practice, the detected grid area boundary box information is converted into grid boundary box information in a preset format, and the preset format grid boundary box information is output to the subsequent user unit, which can facilitate the subsequent user unit to quickly obtain grid information.

[0116] In summary, the beneficial technical effects of the embodiments of the present invention are:

[0117] 1) The method of using a pre-trained rack detection model to directly detect possible rack areas on a shelf image avoids the inefficiency, difficulty in maintenance, and inflexibility of manually dividing racks on the system. It also avoids the potential bias of directly using price tags to generate racks, such as missing racks due to missed price tags, thus improving the efficiency and accuracy of rack generation.

[0118] 2) Adjusting and correcting the shelf range using the results of product identification or classification on the shelf to obtain a more accurate shelf area, further improving the accuracy of shelf generation.

[0119] This invention also provides an apparatus for generating shelf display area, as described in the following embodiments. Since the principle by which this apparatus solves the problem is similar to the method for generating shelf display area, the implementation of this apparatus can refer to the implementation of the method for generating shelf display area; repeated details will not be repeated.

[0120] Figure 7 This is a schematic diagram of the device for generating shelf display area in an embodiment of the present invention, as shown below. Figure 7 As shown, the device includes:

[0121] Acquisition unit 01 is used to acquire the shelf image to be detected and identified;

[0122] The detection unit 02 is used to input the shelf image to be detected and identified into the pre-trained shelf detection model, and to detect the shelf image to be detected and identify to obtain the shelf region bounding box; the shelf detection model is pre-trained and generated based on the historical shelf image dataset and the corresponding shelf region bounding box label dataset samples.

[0123] Generation unit 03 is used to output the bounding box information of the detected grid area.

[0124] In one embodiment, the apparatus for generating shelf display area may further include a shelf detection model training unit for pre-training a shelf detection model according to the following method:

[0125] Acquire historical shelf images and collect them to form a historical shelf image dataset;

[0126] For each image in the shelf image dataset, the possible placement area of ​​each product in the image is defined as the shelf cell for that product. The location label information of the shelf cell is marked, and finally the shelf cell bounding box label dataset is formed.

[0127] Build a deep neural network model;

[0128] Using the shelf image dataset and the corresponding shelf bounding box label dataset, the deep neural network model is trained based on the idea of ​​regression and the gradient backpropagation optimization method to obtain the shelf detection model, which is used to predict the bounding boxes of all possible shelf areas on the shelf.

[0129] In one embodiment, the apparatus for generating shelf display areas may further include:

[0130] The product detection unit is used to detect the shelf image to be identified using a pre-trained product detection model to obtain the product bounding box; the product detection model is pre-trained based on historical shelf image datasets and corresponding product bounding box label dataset samples.

[0131] The cropping unit is used to crop out a small image of a single product from the bounding box of each detected product;

[0132] The product category identification or classification processing unit is used to input the small image of each product into a pre-established product category identification model to identify the product category corresponding to the bounding box of each product; or, input the small image of each product into a pre-established product feature description identification model to classify the product and obtain the classification result corresponding to the bounding box of each product.

[0133] In one embodiment, the apparatus for generating shelf display areas may further include a product detection model training unit for pre-training a product detection model according to the following method:

[0134] Acquire historical shelf images and collect them to form a product image dataset;

[0135] For each image in the product image dataset, the location of each product in the image is labeled to form a product bounding box label dataset;

[0136] Using the product image dataset and the corresponding product bounding box label dataset, a pre-built deep neural network model is trained to obtain a product detection model, which is used to predict all possible product bounding boxes on the shelf.

[0137] In one embodiment, the apparatus for generating shelf display areas may further include a product category recognition unit, used to pre-train the product category recognition model according to the following method:

[0138] For a product image dataset collected in advance based on historical shelf images, each product image is labeled as a separate product category, ultimately forming a product category label dataset;

[0139] Build a deep neural network model;

[0140] Using a product image dataset and a product category label dataset, a deep neural network model is trained based on the idea of ​​maximum likelihood and the loss function of metric learning to obtain the product category recognition model.

[0141] In one embodiment, the product category identification or classification processing unit can be specifically used for:

[0142] Each product's thumbnail is input into a pre-established product feature description recognition model to obtain the feature description of each product's thumbnail; the product feature description recognition model is used to predict the feature description of the input image.

[0143] For each product on the shelf, compare the feature description corresponding to that product with the feature description corresponding to the products near that product;

[0144] Products with a similarity higher than a preset threshold are classified as the same type of product. The same type of product is categorized or tagged into a separate category. Finally, the bounding box of each product on the shelf is tagged with the corresponding category as the classification result.

[0145] In one embodiment, the apparatus for generating shelf display area may further include: an adjustment unit, used to correct and adjust the shelf area boundary frame according to the product category identification or classification results, so as to eliminate the deviation between the generated shelf area boundary frame and the actual product placement.

[0146] In one embodiment, the adjustment unit is specifically used for:

[0147] Determine the quantity of each type of product within each shelf area, and designate the product information with the highest quantity as the product information corresponding to that shelf area;

[0148] Perform an expansion operation on each shelf: Based on the product information corresponding to the current shelf, find the product that is furthest from the current shelf outside its left boundary, and use the left boundary frame of the product that is furthest from the current shelf outside its left boundary as the left border of the shelf; Based on the product information corresponding to the current shelf, find the product that is furthest from the current shelf outside its right boundary, and use the right boundary frame of the product that is furthest from the current shelf outside its right boundary as the right border of the shelf; If no product is found, keep the current shelf border unchanged for now; Record the expansion change of the current shelf.

[0149] Perform a shrinking operation on each grid cell: For each current grid cell, determine whether there are any expansion change records for the grid cells on both sides of the current grid cell. If it is determined from the expansion change records that the grid cells next to the current grid cell have expanded and intersected with the current grid cell, then shrink the boundary box of the current grid cell until it borders the adjacent grid cells and does not intersect with them, thus obtaining the result of correcting and adjusting the boundary box of the grid cell area.

[0150] In one embodiment, the generation unit is specifically used for:

[0151] The detected grid area boundary box information is converted into grid boundary box information in a preset format; wherein, the detected grid area boundary box is a rectangle.

[0152] Output the grid boundary information in the preset format to the subsequent use unit.

[0153] This invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the above-described method for generating shelf display areas.

[0154] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for generating shelf display areas.

[0155] This invention also provides a computer program product, which includes a computer program that, when executed by a processor, implements the above-described method for generating shelf display areas.

[0156] In this embodiment of the invention, the method for generating shelf display grid areas differs from existing technologies that rely on manual division in a backend system (which is inefficient) or use the position of each price tag on the shelf to generate grids (which can lead to inaccuracies). This method involves: acquiring a shelf image to be detected; inputting the shelf image into a pre-trained grid detection model to detect the shelf image and obtain the bounding boxes of the shelf display grid areas; the grid detection model being pre-trained based on historical shelf image datasets and corresponding grid area bounding box label datasets; and outputting the detected shelf display grid area bounding box information. This invention allows the grid detection model to directly detect possible grid areas on a shelf image, improving the efficiency and accuracy of grid area generation.

[0157] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0158] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0159] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0160] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0161] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method of generating a shelf display bay area, characterized by, The method comprises the following steps: obtaining a shelf image to be detected and recognized; inputting the shelf image to be detected and recognized into a pre-trained and generated shelf compartment detection model to detect the shelf image to be detected and recognized to obtain a shelf display compartment region bounding box; the shelf compartment detection model is generated according to historical shelf image data sets and corresponding shelf compartment region bounding box label data set samples by pre-training; outputting the shelf display compartment region bounding box information obtained by detection; inputting a small image of each commodity into a pre-established commodity category recognition model to recognize the commodity category recognition result corresponding to the bounding box of each commodity; or inputting the small image of each commodity into a pre-established commodity feature description recognition model to classify to obtain the classification result corresponding to the bounding box of each commodity; according to the commodity category recognition result or the classification result, the shelf compartment region bounding box is corrected and adjusted, which comprises: determining the number of each kind of commodity in each shelf compartment, and setting the commodity information of the commodity with the largest number as the commodity information corresponding to the shelf compartment; according to the commodity information corresponding to the shelf compartment, an outward expansion operation is performed on each shelf compartment: according to the commodity information corresponding to the current shelf compartment, the commodity farthest from the current shelf compartment outside the left boundary of the current shelf compartment belonging to the current shelf compartment is found, and the left boundary box of the commodity farthest from the current shelf compartment outside the left boundary of the current shelf compartment belonging to the current shelf compartment is taken as the left boundary box of the shelf compartment. If not found, the current shelf compartment boundary box is temporarily kept unchanged.

2. The method of claim 1, wherein, The method further comprises pre-training the shelf compartment detection model by the following method: obtaining historical shelf images to collect historical shelf image data sets; for each image in the shelf image data set, the possible placement region of each commodity in the image is set as the shelf of the commodity, the position label information of the shelf is labeled, and finally a shelf bounding box label data set is formed; a deep neural network model is built; the deep neural network model is trained based on the regression idea and the gradient back propagation optimization method by using the shelf image data set and the corresponding shelf bounding box label data set, and a shelf compartment detection model is obtained, which is used to predict all possible shelf compartment region bounding boxes on the shelf.

3. The method of claim 1, wherein, The method further comprises: detecting the shelf image to be detected and recognized by using the pre-trained and generated commodity detection model to obtain a commodity bounding box; the commodity detection model is generated by pre-training according to historical shelf image data sets and corresponding commodity bounding box label data set samples; a small image of each commodity is cropped from each commodity bounding box detected.

4. The method of claim 3, wherein, The method further comprises: the commodity detection model is obtained by pre-training by the following method: obtaining historical shelf images to collect commodity image data sets; for each image in the commodity image data set, the position of each commodity in the image is labeled to form a commodity bounding box label data set; the deep neural network model is trained by using the commodity image data set and the corresponding commodity bounding box label data set to obtain a commodity detection model, which is used to predict all possible commodity bounding boxes on the shelf.

5. The method of claim 3, wherein, The method further comprises pre-training the commodity category recognition model by the following method: For a commodity image dataset collected in advance according to historical shelf images, each kind of commodity image is labeled as a separate commodity category, and finally a commodity category label dataset is formed; A deep neural network model is built; The deep neural network model is trained based on the commodity image dataset and the commodity category label dataset, the idea of maximum likelihood and the loss function of metric learning, and the commodity category recognition model is obtained.

6. The method of claim 3, wherein, Each small image of a commodity is input into a pre-established commodity feature description recognition model to obtain the commodity category corresponding to each commodity bounding box, including: Each small image of a commodity is input into a pre-established commodity feature description recognition model to obtain the feature description of each small image of the commodity; the commodity feature description recognition model is used to predict the feature description of the input image; For each commodity on the shelf, the feature description of the commodity is compared with the feature descriptions of the commodities near the commodity; Commodities with a similarity higher than a preset threshold are regarded as the same kind of commodities, and the same kind of commodities are classified or labeled as a separate category, and finally each commodity bounding box on the shelf is labeled with a corresponding category label as a classification result.

7. The method of claim 1, wherein, The outer expansion operation on each shelf further includes: finding the farthest commodity from the current shelf outside the right boundary of the current shelf according to the commodity information corresponding to the current shelf, and taking the right bounding box of the farthest commodity from the current shelf outside the right boundary of the current shelf as the right bounding box of the shelf; if not found, the current shelf bounding box is temporarily kept unchanged; and recording the outer expansion change record of the current shelf; According to the commodity category recognition result or the classification result, the shelf region bounding box is corrected and adjusted, and further includes: performing an inner shrinking operation on each shelf: for each current shelf, it is determined whether the shelf on both sides of the current shelf has an outer expansion change record, if it is determined according to the outer expansion change record that the shelf on the side of the current shelf has been expanded and intersects with the current shelf, the current shelf bounding box is shrunk inward until it borders with the shelf on the side and has no intersection, to obtain a result of correcting and adjusting the shelf region bounding box.

8. The method of claim 1, wherein, The shelf display shelf region bounding box information detected is output, including: The shelf region bounding box information detected is converted into shelf bounding box information in a preset format; wherein the shelf region bounding box information detected is a rectangular box; The shelf bounding box information in the preset format is output to a subsequent use unit.

9. An apparatus for generating a shelf display bay area, characterized by, It includes: An acquisition unit is configured to acquire a shelf image to be detected and recognized; A detection unit is configured to input the shelf image to be detected and recognized into a pre-trained shelf detection model to detect the shelf image to be detected and recognized to obtain shelf region bounding box information; The shelf detection model is pre-trained according to a historical shelf image dataset and a corresponding shelf region bounding box label dataset sample; A generation unit is configured to output the shelf region bounding box information detected. The commodity category recognition or classification processing unit is configured to input the small image of each commodity into a pre-established commodity category recognition model to obtain a commodity category recognition result corresponding to the boundary box of each commodity; or input the small image of each commodity into a pre-established commodity feature description recognition model to obtain a classification result corresponding to the boundary box of each commodity. The adjusting unit is configured to correct and adjust the shelf area boundary box according to the commodity category recognition result or the classification result, which includes: determining the number of each kind of commodity in each shelf area, and setting the commodity information of the commodity with the largest number as the commodity information corresponding to the shelf. According to the commodity information corresponding to the shelf, an outward expansion operation is performed on each shelf: according to the commodity information corresponding to the current shelf, the commodity farthest from the current shelf outside the left boundary of the current shelf belonging to the current shelf is found, and the left boundary box of the commodity farthest from the current shelf outside the left boundary of the current shelf belonging to the current shelf is taken as the left boundary box of the shelf; if not found, the boundary box of the current shelf is temporarily kept unchanged.

10. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the method in any one of claims 1 to 8.

11. A computer readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the method in any one of claims 1 to 8.

12. A computer program product, characterised in that, The computer program product includes a computer program, and the computer program is executed by the processor to implement the method in any one of claims 1 to 8.

Images