Intelligent supermarket shelf checking method and device based on image recognition

Abstract

The invention discloses an intelligent supermarket shelf checking method and device based on image recognition, and relates to the technical field of artificial intelligence. The method comprises the following steps: carrying out image acquisition by using mobile data acquisition equipment to obtain original image data, and carrying out image preprocessing; generating a plurality of detection images containing commodity bounding boxes by using the commodity recognition model; performing feature extraction by using an image feature extraction model, comparing to-be-matched image features with a commodity image feature database, determining the SKU of each commodity, and identifying unknown commodities; if the unknown commodity exists, using a meta-heuristic optimization algorithm to finely adjust the image feature extraction model, returning to the feature comparison step, and determining the commodity type and the SKU of the unknown commodity; and performing instance segmentation and counting on the identified commodities, and generating a digital inventory report. The problems that in the prior art, recognition robustness is insufficient, adaptability to new commodities is poor, counting precision is limited, and a closed-loop learning mechanism is lacked are solved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to an intelligent supermarket shelf inventory method and apparatus based on image recognition. Background Technology

[0002] As a core format of modern retail, the accuracy and efficiency of supermarket inventory management directly impact operating costs and customer satisfaction. Traditional shelf inventory methods rely primarily on manual labor, where store clerks use handheld recording devices to count the quantity of goods on the shelves and verify the stock keeping unit (SKU). This method has several drawbacks: First, it is inefficient and labor-intensive; large supermarkets offer a wide variety of goods, making manual inventory counting time-consuming and labor-intensive. Second, accuracy is difficult to guarantee; manual operation is prone to errors due to fatigue, negligence, or unfamiliarity with the products, leading to inaccurate inventory data and consequently, stockouts or overstocking. Finally, the data lacks real-time responsiveness; manual inventory counting is typically conducted outside of business hours, failing to support real-time replenishment decisions and dynamic pricing.

[0003] To address the aforementioned issues, several automated inventory management solutions based on image recognition have emerged in the existing technology. These solutions typically use fixed cameras or mobile robots to capture images of the shelves, and then automatically identify the goods using object detection and image recognition technologies. However, these existing technologies still have the following shortcomings:

[0004] 1) Insufficient robustness in recognition: The complex environment of supermarkets, uneven lighting, dense arrangement of goods, similar packaging, and occlusion can all seriously affect the recognition accuracy of a single model.

[0005] 2) Poor adaptability to new products: Supermarket products are updated rapidly, and promotional activities are frequent, constantly introducing new SKUs. Once traditional recognition models are trained, they are often powerless against new products that have never been seen before (i.e., "unknown products"). It is necessary to manually collect data, label and train the model again, which results in high maintenance costs and the system cannot evolve on its own.

[0006] 3) Limited counting accuracy: Simple target detection can only provide the location box of the product. For products that are closely attached or slightly occluded, it is difficult to accurately distinguish individual products, resulting in inaccurate counting.

[0007] 4) Lack of closed-loop learning mechanism: Most existing systems remain in the open-loop process of "identification-reporting", failing to transform the "unknown" or "uncertain" samples encountered in the identification process into opportunities to improve the system's capabilities, and lacking an adaptive and self-learning closed-loop mechanism.

[0008] Therefore, there is an urgent need for an intelligent inventory method that can efficiently, accurately, and adaptively complete supermarket shelf inventory and continuously optimize itself. Summary of the Invention

[0009] This invention provides an intelligent supermarket shelf inventory method and device based on image recognition, which solves the problems of insufficient recognition robustness, poor adaptability to new products, limited counting accuracy, and lack of closed-loop learning mechanism in the existing technology.

[0010] In a first aspect, embodiments of the present invention provide an intelligent supermarket shelf inventory method based on image recognition, the method comprising:

[0011] Use at least one mobile data acquisition device to capture images of supermarket shelves, obtain raw image data containing information about the goods on the shelves, and perform image preprocessing to obtain preprocessed image data.

[0012] Using a product recognition model based on object detection algorithms, the location of the product is identified from the preprocessed image data, and several detection images containing the product bounding boxes are generated.

[0013] Using an image feature extraction model, features are extracted from the regions within the bounding boxes of goods in several detection images. The extracted features of the images to be matched are compared with a pre-set database of goods image features to determine the unique SKU of each goods and to identify unknown goods.

[0014] If there are unknown products, the image feature extraction model is fine-tuned using a metaheuristic optimization algorithm based on the image features of the unknown products to be matched. Based on the fine-tuned image feature extraction model, the feature comparison step is returned to determine the product category and unique SKU of the unknown products.

[0015] The identified products are segmented to distinguish different product instances of the same SKU for counting, and combined with the shelf hierarchy information, a digital inventory report containing product type, quantity, SKU and shelf location information is generated.

[0016] The technical solution provided in this application has at least the following beneficial effects:

[0017] By employing a coarse-fine cascaded target detection strategy combined with image preprocessing, the accuracy and robustness of product positioning in complex supermarket environments are effectively improved, while ensuring processing speed. The introduction of instance segmentation technology generates pixel-level product masks, enabling precise individual differentiation and counting even when products are closely packed or slightly occluded, far exceeding the accuracy of traditional bounding box methods. Furthermore, an adaptive learning loop based on metaheuristic optimization algorithms is introduced. When encountering unknown products, the model can proactively and intelligently fine-tune its parameters instead of passively waiting for manual intervention, possessing a "self-evolution" capability. This allows for rapid adaptation to supermarket SKU updates, significantly reducing model maintenance costs and extending its lifespan. Unknown problems discovered during inventory checks are transformed into opportunities for model optimization, achieving a complete closed loop of "data collection - identification - problem discovery - self-optimization - re-identification," making the entire inventory process increasingly intelligent and accurate.

[0018] In one optional implementation, at least one mobile data acquisition device is used to acquire images of supermarket shelves, obtain raw image data containing information about the goods on the shelves, and perform image preprocessing to obtain preprocessed image data, including:

[0019] An autonomous navigation inventory robot equipped with an RGB high-definition camera, a structured light depth camera, and a 2D LiDAR is used as a mobile data acquisition device.

[0020] Based on a pre-built supermarket map and LiDAR SLAM technology, it autonomously plans the optimal inventory route including all supermarket shelves and moves at a constant speed along the center line of the shelf aisle.

[0021] During the journey, mobile data acquisition devices are used to capture images of supermarket shelves, collecting continuous frames of raw image data containing information about the goods on the shelves and corresponding depth maps;

[0022] The raw image data and corresponding depth map of each frame are uploaded to the backend server in real time;

[0023] On the backend server, based on the depth map, the original image data undergoes image preprocessing, including image denoising, illumination compensation and enhancement, geometric correction, and background segmentation, to obtain preprocessed image data.

[0024] In one alternative implementation, the product recognition model includes a first object detection sub-model built on the YOLOv5s algorithm and a second object detection sub-model built on the Cascade R-CNN algorithm.

[0025] In one optional implementation, a product recognition model based on an object detection algorithm is used to identify the location of the product from the preprocessed image data, generating several detection images containing the product's bounding box, including:

[0026] The preprocessed image data is input into the first target detection sub-model of the product recognition model for coarse detection to obtain the product areas of several products on the current shelf. The product areas are then cropped to obtain the corresponding product area images.

[0027] The product region images of several products are input into the second object detection sub-model of the product recognition model for fine detection, so as to obtain the product bounding box and position of each product;

[0028] Based on the product bounding box and position, the product region image is cropped to obtain several detection images containing the product bounding box.

[0029] In one optional implementation, an image feature extraction model is used to extract features from the regions within the bounding boxes of goods in several detection images. The extracted features of the images to be matched are compared with a pre-set product image feature database to determine the unique SKU of each product and to identify unknown products, including:

[0030] Several detection images are input into a pre-built image feature extraction model to extract features from the regions within the bounding boxes of the goods in the detection images, thereby obtaining the image features of the goods to be matched.

[0031] The features of the image to be matched are compared with the preset product image feature database. The approximate nearest neighbor search algorithm is used for fast retrieval to obtain K product image features that are similar to the features of the image to be matched, where K is the preset number of products to be retrieved.

[0032] Calculate the cosine similarity between the features of the image to be matched and the features of K product images. If the cosine similarity is greater than the cosine similarity threshold, the product category and unique SKU corresponding to the features of the image to be matched are determined; otherwise, the product is identified as an unknown product.

[0033] In one alternative implementation, the image feature extraction model is built based on a ResNet-50 algorithm trained with Triplet Loss;

[0034] The approximate nearest neighbor search algorithm is the FAISS algorithm.

[0035] In one alternative implementation, the metaheuristic optimization algorithm is the ITSOA algorithm.

[0036] In one optional implementation, if an unknown product exists, a metaheuristic optimization algorithm is used to fine-tune the image feature extraction model based on the image features to be matched for the unknown product. Based on the fine-tuned image feature extraction model, a feature comparison step is returned to determine the product category and unique SKU of the unknown product, including:

[0037] If there are unknown products, extract the image features corresponding to the unknown products to be matched;

[0038] The image features of unknown products to be matched are defined as the planets to be explored by the ITSOA algorithm, and the product image features of all SKUs in the product feature database are defined as the corresponding stars.

[0039] The model parameters of the image feature extraction model are encoded into vectors of orbits in the ITSOA algorithm, and several initial orbits are initialized using the Tent chaotic mapping sequence.

[0040] The initial orbit is applied to the planet to obtain the optimized first image features of the unknown product to be matched, and the cosine similarity between the optimized first image features to be matched and the product image features of several stars is obtained to obtain the initial similarity set.

[0041] Based on the initial similarity set, the initial fitness value of the corresponding initial orbital is generated using a preset fitness function, and the orbital with the best fitness value is taken as the global optimal orbital.

[0042] Based on the fitness values ​​of several initial orbits, elite orbits are selected, and based on the selected elite orbits, elite retention behavior, random perturbation behavior, and orbit crossing behavior are performed to perform orbit iterative optimization and obtain several updated orbits.

[0043] The updated orbit is applied to the planet to obtain the optimized second image features to be matched for the unknown commodity, and the cosine similarity between the optimized second image features to be matched and the commodity image features of several stars is obtained to obtain the updated similarity set.

[0044] Based on the updated similarity set, the fitness function is used to generate the updated fitness values ​​of the corresponding updated orbitals, and the orbit with the best fitness value is updated as the global optimal orbital.

[0045] Repeat the above orbit iteration optimization steps until the number of iterations reaches the maximum number of iterations or the fitness value of the global optimal orbit meets the requirements, then terminate the iteration and decode the vector of the global optimal orbit to obtain the optimal model parameters.

[0046] Based on the optimal model parameters, update the initial model parameters of the image feature extraction model, and input the image features of several commodities to be matched for fine-tuning to obtain the fine-tuned image feature extraction model.

[0047] Based on the fine-tuned image feature extraction model, the feature comparison step is returned to determine the product category and unique SKU of the unknown product.

[0048] In one optional implementation, the identified products are segmented to distinguish different product instances of the same SKU for counting. Combined with shelf hierarchy information, a digital inventory report is generated, including product type, quantity, SKU, and shelf location information, comprising:

[0049] The detected images, including several identified products, are input into a pre-built instance segmentation model to generate a corresponding pixel-level mask for each product.

[0050] The pixel-level mask of each product is associated with the corresponding SKU. By counting the number of all pixel-level masks corresponding to each SKU, the number of SKUs in the current shelf corresponding to the detection image is obtained.

[0051] Based on the location of each product and the preset shelf structure information, calculate the vertical coordinates of the centroid of the pixel-level mask of each product in the detection image, and map the product to the specific shelf level according to the vertical coordinates and coordinate range.

[0052] The current shelf number, hierarchical structure information, and product shelf level are used as the corresponding shelf location information. The inventory results of each product's product type, quantity, SKU, and shelf location information are summarized into a structured digital inventory report.

[0053] Secondly, embodiments of the present invention provide an intelligent supermarket shelf inventory device based on image recognition, used to implement an intelligent supermarket shelf inventory method. The device includes:

[0054] The data acquisition unit is used to acquire images of supermarket shelves using at least one mobile data acquisition device, obtain raw image data containing information about the goods on the shelves, and perform image preprocessing to obtain preprocessed image data.

[0055] The product recognition unit is used to identify the location of the product from the preprocessed image data using a product recognition model based on object detection algorithm, and generate several detection images containing the product bounding box.

[0056] The SKU determination unit is used to extract features from the regions within the bounding boxes of goods in several detection images using an image feature extraction model, compare the extracted image features to be matched with a preset product image feature database, determine the unique SKU of each product, and identify unknown products.

[0057] The unknown product processing unit is used to, if an unknown product exists, fine-tune the image feature extraction model using a metaheuristic optimization algorithm based on the image features to be matched of the unknown product, and return the feature comparison step based on the fine-tuned image feature extraction model to determine the product type and unique SKU of the unknown product.

[0058] The merchandise inventory unit is used to segment identified merchandise into instances to distinguish different instances of the same SKU for counting, and, in conjunction with shelf hierarchy information, to generate a digital inventory report containing information on merchandise type, quantity, SKU, and shelf location.

[0059] A third aspect of this invention provides an electronic device, which includes:

[0060] At least one processor; and a memory communicatively connected to the at least one processor; wherein,

[0061] The memory stores instructions that can be executed by at least one processor, such that the at least one processor can perform the method proposed in the first aspect of the present invention.

[0062] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in the first aspect of the present invention. Attached Figure Description

[0063] Figure 1 This is a schematic diagram of the electronic device structure of the hardware operating environment involved in the embodiments of the present invention;

[0064] Figure 2 This is a flowchart illustrating the steps of an intelligent supermarket shelf inventory method based on image recognition, as provided in an embodiment of the present invention.

[0065] Figure 3 This is a functional unit diagram of an intelligent supermarket shelf inventory device based on image recognition provided in an embodiment of the present invention. Detailed Implementation

[0066] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0067] The present invention will be further described below with reference to the accompanying drawings.

[0068] Reference Figure 1 , Figure 1 This is a schematic diagram of the electronic device structure of the hardware operating environment involved in the embodiments of the present invention.

[0069] like Figure 1 As shown, the electronic device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be a high-speed random access memory (RAM) or a stable non-volatile memory (NVM), such as a disk drive. The memory 1005 may also optionally be a storage device independent of the aforementioned processor 1001.

[0070] Those skilled in the art will understand that Figure 1 The structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0071] like Figure 1 As shown, the memory 1005, which serves as a storage medium, may include an operating system, a data storage module, a network communication module, a user interface module, and an electronic program for an image recognition-based intelligent supermarket shelf inventory device.

[0072] exist Figure 1In the electronic device shown, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the electronic device of the present invention can be set in the electronic device. The electronic device calls the electronic program of the intelligent supermarket shelf inventory device based on image recognition stored in the memory 1005 through the processor 1001, and executes the intelligent supermarket shelf inventory method based on image recognition provided in the embodiment of the present invention.

[0073] Reference Figure 2 The present invention provides an image recognition-based intelligent supermarket shelf inventory method, the method comprising:

[0074] S201: Using at least one mobile data acquisition device, images of supermarket shelves are acquired to obtain raw image data containing information about the goods on the shelves, and image preprocessing is performed to obtain preprocessed image data.

[0075] S202: Using a product recognition model based on object detection algorithm, identify the location of the product from the preprocessed image data and generate several detection images containing the product bounding box;

[0076] S203: Using an image feature extraction model, extract features from the regions within the bounding boxes of goods in several detection images, compare the extracted features of the images to be matched with a preset product image feature database, determine the unique SKU of each product, and identify unknown products.

[0077] S204: If there are unknown products, the image feature extraction model is fine-tuned using a metaheuristic optimization algorithm based on the image features to be matched of the unknown products. Based on the fine-tuned image feature extraction model, the feature comparison step is returned to determine the product category and unique SKU of the unknown products.

[0078] S205: Segment the identified products to distinguish different product instances of the same SKU for counting, and combine the shelf hierarchy information to generate a digital inventory report containing product type, quantity, SKU and shelf location information.

[0079] The technical solution provided in this application has at least the following beneficial effects:

[0080] By employing a coarse-fine cascaded target detection strategy combined with image preprocessing, the accuracy and robustness of product positioning in complex supermarket environments are effectively improved, while ensuring processing speed. The introduction of instance segmentation technology generates pixel-level product masks, enabling precise individual differentiation and counting even when products are closely packed or slightly occluded, far exceeding the accuracy of traditional bounding box methods. Furthermore, an adaptive learning loop based on metaheuristic optimization algorithms is introduced. When encountering unknown products, the model can proactively and intelligently fine-tune its parameters instead of passively waiting for manual intervention, possessing a "self-evolution" capability. This allows for rapid adaptation to supermarket SKU updates, significantly reducing model maintenance costs and extending its lifespan. Unknown problems discovered during inventory checks are transformed into opportunities for model optimization, achieving a complete closed loop of "data collection - identification - problem discovery - self-optimization - re-identification," making the entire inventory process increasingly intelligent and accurate.

[0081] In one optional implementation, at least one mobile data acquisition device is used to acquire images of supermarket shelves, obtain raw image data containing information about the goods on the shelves, and perform image preprocessing to obtain preprocessed image data, including:

[0082] S2011: An autonomous navigation inventory robot equipped with an RGB high-definition camera, a structured light depth camera, and a 2D LiDAR is used as a mobile data acquisition device.

[0083] S2012: Based on a pre-built supermarket map and LiDAR Simultaneous Localization and Mapping (SLAM) technology, it autonomously plans the optimal inventory route including all supermarket shelves and travels at a constant speed along the center line of the shelf aisle.

[0084] S2013: During the driving process, use a mobile data acquisition device to acquire images of supermarket shelves, and collect continuous frames of raw image data containing shelf product information and corresponding depth maps;

[0085] In this embodiment, the RGB camera is triggered once every 0.5 meters or per second in a direction perpendicular to the shelf to capture a high-resolution (e.g., 4K) image of the front of the shelf, forming a continuous image sequence covering the entire shelf surface. At the same time, the depth camera synchronously acquires the corresponding depth map.

[0086] When the autonomous navigation inventory robot reaches the preset "key inspection points" at either end or in the middle of the shelf, it will pause briefly for 1-2 seconds. During this time, the gimbal will drive the RGB camera to capture images from multiple angles.

[0087] Shoot from the front: Obtain the most standard front view of the shelf;

[0088] 45° angle shooting: Shoot from the top left and top right at two 45° angles to obtain side information of the product packaging, which is crucial for distinguishing products that differ only in front design;

[0089] Close-up shots: Using optical zoom, zoom in on the middle and lower parts of the shelf to capture details such as barcodes, production dates, and small fonts;

[0090] S2014: Upload the raw image data and corresponding depth map of each frame to the backend server in real time;

[0091] In this embodiment, the original image data of each frame, the corresponding depth map, and the precise pose information (including coordinates and orientation) from the lidar and robot odometry are packaged into a data packet and uploaded to the backend server in real time via 5G or Wi-Fi network.

[0092] S2015: On the background server, based on the depth map, the original image data is preprocessed by image denoising, illumination compensation and enhancement, geometric correction and background segmentation to obtain preprocessed image data.

[0093] In this embodiment, the preprocessing includes:

[0094] Image denoising: To address salt-and-pepper noise and Gaussian noise that may occur in low-light environments, a non-local mean filtering algorithm is adopted. This algorithm searches for similar pixel blocks in the image and performs a weighted average, which can effectively denoise while preserving the edge and texture details of the product to the greatest extent.

[0095] Illumination compensation and enhancement: To overcome the problem of uneven supermarket lighting (such as light spots caused by overhead spotlights), adaptive gamma correction is adopted. First, the image is decomposed into illumination component and reflection component using the Retinex algorithm; then, homomorphic filtering is applied to the illumination component to compress the dynamic range; finally, it is merged with the reflection component and adaptive gamma correction is performed, thereby brightening the dark areas while suppressing overexposed areas.

[0096] Geometric correction (perspective transformation): Using depth information obtained from the depth map and camera intrinsic parameters, the pose of the shelf plane relative to the camera is accurately calculated. By solving the perspective transformation matrix, the tilted shelf image is "straightened" to generate a standard front view. Compared with the traditional feature point matching method, using depth information for correction is more robust and accurate.

[0097] Background segmentation: By using the synchronously acquired depth map and setting a depth threshold, the foreground product area can be easily separated from the background such as the back panel of the shelf and the wall, generating a binary foreground mask for subsequent processing to eliminate background interference, making object detection and feature extraction more focused.

[0098] In one alternative implementation, the product recognition model includes a first object detection sub-model built on the YOLOv5s algorithm and a second object detection sub-model built on the Cascade Region-based Convolutional Neural Network (Cascade R-CNN) algorithm.

[0099] In one optional implementation, a product recognition model based on an object detection algorithm is used to identify the location of the product from the preprocessed image data, generating several detection images containing the product's bounding box, including:

[0100] S2021: Input the preprocessed image data into the first target detection sub-model of the product recognition model for coarse detection, obtain the product areas of several products on the current shelf, and crop the product areas to obtain the corresponding product area images;

[0101] In this embodiment, the first object detection sub-model only needs to identify the broad category of "products" and quickly delineate the "product area" (e.g., a shelf) in the image that may contain multiple products.

[0102] S2022: Input the product region images of several products into the second object detection sub-model of the product recognition model for fine detection, and obtain the product bounding box and position of each product;

[0103] In this embodiment, the second target detection sub-model performs a detailed analysis of the cropped area, which can accurately distinguish each closely arranged product and output its high-precision bounding box coordinates and position.

[0104] S2023: Based on the product bounding box and position, crop the product region image to obtain several detection images containing the product bounding box;

[0105] In this embodiment, this "coarse-fine" cascade strategy ensures both processing speed across the entire map and positioning accuracy for individual items, effectively balancing efficiency and performance.

[0106] In one optional implementation, an image feature extraction model is used to extract features from the regions within the bounding boxes of goods in several detection images. The extracted features of the images to be matched are compared with a pre-set product image feature database to determine the unique SKU of each product and to identify unknown products, including:

[0107] S2031: Input several detection images into a pre-built image feature extraction model, extract features from the region within the bounding box of the goods in the detection images, and obtain the image features of the goods to be matched;

[0108] In this embodiment, the goal of the image feature extraction model is not classification, but to map the image to a high-dimensional (e.g., 256-dimensional) feature space, in which the feature vectors of products of the same SKU are as close as possible, and the feature vectors of different SKUs are as far apart as possible.

[0109] S2032: Compare the features of the image to be matched with the preset product image feature database, and use the approximate nearest neighbor search algorithm to quickly retrieve K product image features that are similar to the features of the image to be matched, where K is the preset number of products to be retrieved;

[0110] S2033: Calculate the cosine similarity between the features of the image to be matched and the features of K product images. If the cosine similarity is greater than the cosine similarity threshold, then determine the product category and unique SKU for the product corresponding to the features of the image to be matched; otherwise, identify it as an unknown product.

[0111] In this embodiment, a threshold (e.g., 0.85) is set. If the highest similarity is ≥0.85, it is directly identified as the corresponding SKU and the identification is successful. If the highest similarity is <0.85, it is identified as an "unknown product".

[0112] In one alternative implementation, the image feature extraction model is built based on a 50-layer residual network (ResNet-50) algorithm trained with Triplet Loss.

[0113] The approximate nearest neighbor search algorithm is the Facebook AI SimilaritySearch (FAISS) algorithm.

[0114] In one alternative implementation, the metaheuristic optimization algorithm is the Improved TransitSearch Optimization Algorithm (ITSOA).

[0115] In one optional implementation, if an unknown product exists, a metaheuristic optimization algorithm is used to fine-tune the image feature extraction model based on the image features to be matched for the unknown product. Based on the fine-tuned image feature extraction model, a feature comparison step is returned to determine the product category and unique SKU of the unknown product, including:

[0116] S2041: If there are unknown products, extract the image features to be matched corresponding to the unknown products;

[0117] S2042: Define the image features of unknown products to be matched as the planets to be explored by the ITSOA algorithm, and define the product image features of all SKUs in the product feature database as the corresponding stars.

[0118] S2043: Encode the model parameters of the image feature extraction model into a vector of orbits for the ITSOA algorithm, and initialize M initial orbits using the Tent chaotic mapping sequence, where M is the total number of orbits;

[0119] The formula is:

[0120]

[0121] In the formula, Let i be the i-th initial orbit; Let i be the i-th chaotic variable; represents the upper and lower bounds of the search space; i is the orbital indicator.

[0122]

[0123] In the formula, Let i be the (i+1)th chaotic variable; compared with random initialization, chaotic initialization can ensure that the population is evenly distributed in the solution space, thus enhancing diversity.

[0124] S2044: Apply the initial orbit to the planet to obtain the optimized first matching image features of the unknown product, and obtain the cosine similarity between the optimized first matching image features and the product image features of several stars to obtain the initial similarity set.

[0125] In this embodiment, through the initial orbit The candidate image feature extraction model trained with the corresponding model parameters extracts the optimized first image features to be matched for the unknown product.

[0126] S2045: Based on the initial similarity set, use the preset fitness function to generate the initial fitness value of the corresponding initial orbital, and take the orbital with the best fitness value as the global optimal orbital.

[0127] The formula is:

[0128]

[0129] In the formula, Let be the fitness value of the i-th orbit in iteration number t; Let be the set of similarities for iteration number t; The maximum cosine similarity; denoted as similarity variance; t is the current iteration number; i is the orbital indicator; this function encourages optimization results to not only be highly similar to a certain class, but also significantly different from other classes;

[0130] S2046: Based on the fitness values ​​of several initial orbits, select elite orbits, and based on the selected elite orbits, perform elite retention behavior, random perturbation behavior, and orbit crossing behavior to perform orbit iterative optimization and obtain several updated orbits.

[0131] The formula is:

[0132]

[0133] In the formula, For elite orbital groups; This is the set of fitness values ​​for the orbital; Select composite functions for elite tracks, including sorting functions. descending order and index ;

[0134]

[0135] In the formula, The i-th track updated for iteration number t+1 obtained by performing elite preservation behavior;

[0136]

[0137] In the formula, The i-th orbit updated with iteration number t+1 to implement the Levy aircraft mechanism and convergence factor improved stochastic perturbation behavior; For the elite orbital of iteration number t; Let b be a random number distributed by Levy; b is the Levy step size, and b∈[1,2]. This is element-wise multiplication; This is the step size parameter; The convergence factor;

[0138]

[0139] In the formula, These are the maximum and minimum values ​​of the convergence factor; , To adjust the parameters; The function is the hyperbolic tangent; T is the maximum number of iterations.

[0140]

[0141] In the formula, The i-th track is updated for iteration number t+1 obtained from performing the track crossing behavior; Let be any two elite tracks selected from the set of elite tracks at iteration number t; The globally optimal trajectory for iteration number t; is a randomly generated floating-point number uniformly distributed within the interval [0, 1]; convergence factor. The step size for approaching the optimal solution was controlled: in the early stages of iteration. Larger step size facilitates global exploration and later iterations. Approaching 0, with a small step size, it is conducive to detailed local development;

[0142] S2047: Apply the updated orbit to the planet to obtain the optimized second image features to be matched for the unknown commodity, and obtain the cosine similarity between the optimized second image features to be matched and the commodity image features of several stars to obtain the updated similarity set.

[0143] S2048: Based on the updated similarity set, use the fitness function to generate the updated fitness values ​​of the corresponding updated orbitals, and update the orbital with the best fitness value as the globally optimal orbital;

[0144] S2049: Repeat the above orbit iteration optimization steps until the number of iterations reaches the maximum number of iterations or the fitness value of the global optimal orbit meets the requirements, then terminate the iteration and decode the vector of the global optimal orbit to obtain the optimal model parameters.

[0145] In this embodiment, the model parameters include the initial weights and biases of the fully connected layers of the image feature extraction model (ResNet-50), the initial weights of the last convolutional block, and the learnable parameters of the batch normalization layer.

[0146] S20410: Based on the optimal model parameters, update the initial model parameters of the image feature extraction model, and input the image features of several commodities to be matched for fine-tuning to obtain the fine-tuned image feature extraction model.

[0147] S20411: Based on the fine-tuned image feature extraction model, return the feature comparison step to determine the product category and unique SKU of the unknown product;

[0148] In one optional implementation, based on the fine-tuned image feature extraction model, a feature comparison step is returned to determine the product category and unique SKU of the unknown product, including:

[0149] S204111: Based on the fine-tuned image feature extraction model, extract the latest image features to be matched for unknown products;

[0150] S204112: Compare the latest image features of the unknown product to be matched with the preset product image feature database, and use the approximate nearest neighbor search algorithm to quickly retrieve K product image features that are similar to the latest image features of the unknown product to be matched, where K is the preset number of products to be retrieved.

[0151] S204113: Calculate the latest cosine similarity between the latest image feature to be matched of the unknown product and the K product image features. If the latest cosine similarity is greater than the cosine similarity threshold, then determine the product category and unique SKU for the unknown product corresponding to the latest image feature to be matched.

[0152] S204114: If the latest cosine similarity is less than or equal to the cosine similarity threshold several times, then it is officially determined that this is a brand new product category, and a brand new, unique SKU is set.

[0153] S204115: Use the latest image features to be matched as the product image features of this new SKU and store them in the product image feature database.

[0154] In one optional implementation, the identified products are segmented to distinguish different product instances of the same SKU for counting. Combined with shelf hierarchy information, a digital inventory report is generated, including product type, quantity, SKU, and shelf location information, comprising:

[0155] S2051: Input the detection image containing several identified products into the pre-built instance segmentation model to generate a corresponding pixel-level mask for each product;

[0156] In this embodiment, the instance segmentation model generates an accurate pixel-level mask for each detected product instance, which can effectively distinguish products even if they are slightly occluded or closely attached.

[0157] S2052: Associate the pixel-level mask of each product with the corresponding SKU, and obtain the number of SKUs on the current shelf corresponding to the detection image by counting the number of all pixel-level masks corresponding to each SKU;

[0158] S2053: Based on the location of each product and the preset shelf structure information (such as the height of each layer and the starting position), calculate the vertical coordinates of the centroid of the pixel-level mask of each product in the detection image, and map the product to the specific product shelf layer according to the vertical coordinates and the coordinate range.

[0159] S2054: Use the current shelf number, hierarchical structure information, and product shelf level as the corresponding shelf location information (e.g., shelf A01-03, second level), and summarize the inventory results of each product's product type, quantity, SKU, and shelf location information into a structured digital inventory report.

[0160] In this embodiment, a structured JSON or XML format digital inventory report is obtained. Through the API interface, the report can be automatically pushed to the supermarket's WMS or ERP system to trigger replenishment alerts or update inventory data.

[0161] This invention also provides an intelligent supermarket shelf inventory device based on image recognition, referring to... Figure 3 The diagram shows a functional unit diagram of an intelligent supermarket shelf inventory device 300 based on image recognition according to the present invention. The device may include the following units:

[0162] The data acquisition unit 301 is used to acquire images of supermarket shelves using at least one mobile data acquisition device, obtain raw image data containing information about the goods on the shelves, and perform image preprocessing to obtain preprocessed image data.

[0163] The product recognition unit 302 is used to identify the location of the product from the preprocessed image data using a product recognition model based on the object detection algorithm, and generate several detection images containing the product bounding box.

[0164] The SKU determination unit 303 is used to extract features from the region within the bounding box of the product in several detection images using an image feature extraction model, compare the extracted image features to be matched with a preset product image feature database, determine the unique SKU of each product, and identify unknown products.

[0165] The unknown product processing unit 305 is used to, if there is an unknown product, fine-tune the image feature extraction model using a metaheuristic optimization algorithm based on the image features to be matched of the unknown product, and return to the feature comparison step based on the fine-tuned image feature extraction model to determine the product type and unique SKU of the unknown product.

[0166] The inventory unit 304 is used to segment the identified products into instances to distinguish different product instances of the same SKU for counting, and to generate a digital inventory report containing product type, quantity, SKU and shelf location information by combining the shelf hierarchy information.

[0167] Based on the same inventive concept, another embodiment of the present invention provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus.

[0168] Memory, used to store computer programs;

[0169] When the processor executes the program stored in the memory, it implements the image recognition-based intelligent supermarket shelf inventory method of the present invention.

[0170] The communication bus mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of representation, only one thick line is used in the diagram, but this does not indicate that there is only one bus or one type of bus. The communication interface is used for communication between the aforementioned terminal and other devices. The memory can include Random Access Memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory can also be at least one storage device located remotely from the aforementioned processor.

[0171] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0172] Furthermore, to achieve the above objectives, embodiments of the present invention also propose a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the image recognition-based intelligent supermarket shelf inventory method of the embodiments of the present invention.

[0173] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, embodiments of the present invention can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of the present invention can take the form of computer program products implemented on one or more computer-usable hardware devices (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0174] The embodiments of the present invention are described with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (apparatus), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0175] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0176] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0177] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. "And / or" indicates that either one or both can be chosen. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes the element.

[0178] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An image recognition-based intelligent supermarket shelf inventory method, characterized in that, The method comprises: using at least one mobile data acquisition device to collect images of supermarket shelves, obtain raw image data containing shelf commodity information, and perform image preprocessing to obtain preprocessed image data; using a commodity recognition model based on a target detection algorithm to identify the positions of commodities from the preprocessed image data, and generating a plurality of detection images containing commodity bounding boxes; using an image feature extraction model to extract features from the regions within the commodity bounding boxes in the plurality of detection images, comparing the extracted image features to be matched with a pre-set commodity image feature database, determining the unique SKU of each commodity, and identifying unknown commodities; if there are unknown commodities, using a meta-heuristic optimization algorithm to fine-tune the image feature extraction model based on the image features to be matched of the unknown commodities, and returning to the feature comparison step based on the fine-tuned image feature extraction model to determine the commodity type and unique SKU of the unknown commodities; performing instance segmentation on the identified commodities to distinguish different commodity instances of the same SKU for counting, and generating a digital inventory report containing commodity type, quantity, SKU, and shelf location information in combination with the hierarchical structure information of the shelves. 2.The image recognition based intelligent supermarket shelf inventory method according to claim 1, characterized in that, using at least one mobile data acquisition device to collect images of supermarket shelves, obtain raw image data containing shelf commodity information, and perform image preprocessing to obtain preprocessed image data, comprising: using an autonomous navigation inventory robot equipped with an RGB high-definition camera, a structured light depth camera, and a 2D laser radar as a mobile data acquisition device; based on a pre-constructed supermarket map and laser radar SLAM technology, autonomously planning an optimal inventory path including all supermarket shelves, and driving at a uniform speed along the center line of the shelf channel; during driving, using the mobile data acquisition device to collect images of supermarket shelves, collecting continuous frames of raw image data containing shelf commodity information and corresponding depth maps; uploading the raw image data and corresponding depth maps of each frame to a background server in real time; in the background server, performing image preprocessing of image denoising, illumination compensation and enhancement, geometric correction, and background segmentation on the raw image data based on the depth map to obtain preprocessed image data. 3.The image recognition based intelligent supermarket shelf inventory method of claim 2, wherein, The commodity recognition model comprises a first target detection sub-model based on the YOLOv5s algorithm and a second target detection sub-model based on the Cascade R-CNN algorithm.

4. The image recognition based intelligent supermarket shelf inventory method according to claim 3, characterized in that, using a commodity recognition model based on a target detection algorithm to identify the positions of commodities from the preprocessed image data, and generating a plurality of detection images containing commodity bounding boxes, comprising: inputting the preprocessed image data into the first target detection sub-model of the commodity recognition model for coarse detection to obtain the commodity regions of a plurality of commodities of the current shelf, and cropping the commodity regions to obtain corresponding commodity region images; inputting the commodity region images of the plurality of commodities into the second target detection sub-model of the commodity recognition model for fine detection to obtain the commodity bounding boxes and positions of each commodity; According to the commodity bounding box and the position, the commodity region image is cropped to obtain a plurality of detection images containing the commodity bounding box.

5. The image recognition based intelligent supermarket shelf inventory method according to claim 4, characterized in that, Using an image feature extraction model, the region within the commodity bounding box in the plurality of detection images is extracted, and the extracted matching image features are compared with a preset commodity image feature database to determine the unique SKU of each commodity and identify unknown commodities, including: The plurality of detection images are input into a pre-constructed image feature extraction model to extract features of the region within the commodity bounding box in the detection image, obtaining the matching image features of the commodity; The matching image features are compared with the preset commodity image feature database, and a fast search is performed using an approximate nearest neighbor search algorithm to obtain K commodity image features similar to the matching image features, K being a preset number of search commodities; The cosine similarity of the matching image features and the K commodity image features is calculated, and if the cosine similarity is greater than a cosine similarity threshold, the commodity corresponding to the matching image features is determined as a commodity type and a unique SKU, otherwise it is identified as an unknown commodity. 6.The image recognition based intelligent supermarket shelf inventory method of claim 5, wherein, The image feature extraction model is constructed based on a ResNet-50 algorithm trained by Triplet Loss; The approximate nearest neighbor search algorithm is a FAISS algorithm. 7.The image recognition based intelligent supermarket shelf inventory method of claim 6, wherein, The meta-heuristic optimization algorithm is an ITSOA algorithm. 8.The image recognition based intelligent supermarket shelf inventory method of claim 7, wherein, If there is an unknown commodity, the image feature extraction model is fine-tuned using a meta-heuristic optimization algorithm based on the matching image features of the unknown commodity, and based on the fine-tuned image feature extraction model, return to the feature comparison step to determine the commodity type and unique SKU of the unknown commodity, including: If there is an unknown commodity, the matching image features corresponding to the unknown commodity are extracted; The matching image features of the unknown commodity are defined as planets to be explored in the ITSOA algorithm, and the commodity image features of all SKUs in the commodity feature database are defined as a plurality of stars; The model parameters of the image feature extraction model are encoded as vectors of orbits in the ITSOA algorithm, and a Tent chaotic mapping sequence is used to initialize a plurality of initial orbits; The initial orbits are applied to the planets to obtain the first optimized matching image features of the unknown commodity, and the cosine similarity of the first optimized matching image features and the commodity image features of the plurality of stars is obtained to obtain an initial similarity set; According to the initial similarity set, a preset fitness function is used to generate an initial fitness value corresponding to the initial orbits, and the orbit with the optimal fitness value is taken as the global optimal orbit; According to the fitness values of the plurality of initial orbits, elite orbits are selected, and based on the selected elite orbits, elite retention behavior, random disturbance behavior and orbit crossover behavior are performed for orbit iterative optimization to obtain a plurality of updated orbits; The updated orbits are applied to the planets to obtain the second optimized matching image features of the unknown commodity, and the cosine similarity of the second optimized matching image features and the commodity image features of the plurality of stars is obtained to obtain an updated similarity set; According to the updated similarity set, an updated fitness value of a corresponding updated track is generated using a fitness function, and the track with the optimal fitness value is updated as a global optimal track; The track iteration optimization step is repeated until the number of iterations reaches a maximum number of iterations or the fitness value of the global optimal track meets the requirement, the iteration is terminated, and the vector of the global optimal track is decoded to obtain optimal model parameters; According to the optimal model parameters, the initial model parameters of the image feature extraction model are updated, and the image features of several commodities to be matched are input for fine-tuning to obtain a fine-tuned image feature extraction model; Based on the fine-tuned image feature extraction model, return to the feature comparison step to determine the product category and unique SKU of the unknown product. 9.The image recognition based intelligent supermarket shelf inventory method of claim 8, wherein, Instance segmentation is performed on the recognized products to distinguish different product instances of the same SKU for counting, and the hierarchical structure information of the shelf is combined to generate a digital inventory report containing product category, quantity, SKU, and shelf location information, including: Input the detection image including several recognized products into the pre-constructed instance segmentation model to generate a corresponding pixel-level mask for each product; Associate the pixel-level mask of each product with the corresponding SKU, and obtain the quantity of each SKU in the current shelf corresponding to the detection image by counting the number of all pixel-level masks corresponding to each SKU; According to the position of each product and the preset shelf structure information, calculate the vertical coordinate of the centroid of each product's pixel-level mask in the detection image, and map the product to a specific product shelf level according to the vertical coordinate and the coordinate range; The number of the current shelf, the hierarchical structure information, and the product shelf level are used as the corresponding shelf location information, and the inventory results of the product category, quantity, SKU, and shelf location information of each product are summarized into a structured digital inventory report.

10. An image recognition based intelligent supermarket shelf inventory device for implementing the intelligent supermarket shelf inventory method of any one of claims 1-9, characterized in that, The device comprises: A data acquisition unit for using at least one mobile data acquisition device to perform image acquisition on supermarket shelves, obtaining raw image data containing shelf product information, and performing image preprocessing to obtain preprocessed image data A product recognition unit for using a product recognition model based on a target detection algorithm to identify the location of products from the preprocessed image data, and generating a plurality of detection images containing product bounding boxes; A SKU determination unit for using an image feature extraction model to extract features from the regions within the product bounding boxes in a plurality of detection images, comparing the extracted image features to be matched with a pre-set product image feature database, determining the unique SKU of each product, and recognizing unknown products; An unknown product processing unit for fine-tuning the image feature extraction model using a meta-heuristic optimization algorithm based on the image features to be matched of unknown products, and returning to the feature comparison step based on the fine-tuned image feature extraction model to determine the product category and unique SKU of the unknown product. Instance segmentation is performed on the recognized products to distinguish different product instances of the same SKU for counting, and the hierarchical structure information of the shelf is combined to generate a digital inventory report containing product category, quantity, SKU, and shelf location information, including: Input the detection image including several recognized products into the pre-constructed instance segmentation model to generate a corresponding pixel-level mask for each product; Associate the pixel-level mask of each product with the corresponding SKU, and obtain the quantity of each SKU in the current shelf corresponding to the detection image by counting the number of all pixel-level masks corresponding to each SKU; According to the position of each product and the preset shelf structure information, calculate the vertical coordinate of the centroid of each product's pixel-level mask in the detection image, and map the product to a specific product shelf level according to the vertical coordinate and the coordinate range; The number of the current shelf, the hierarchical structure information, and the product shelf level are used as the corresponding shelf location information, and the inventory results of the product category, quantity, SKU, and shelf location information of each product are summarized into a structured digital inventory report. A commodity inventory unit is used for instance segmentation on the recognized commodities to distinguish different commodity instances of the same SKU for counting, and generate a digital inventory report containing commodity category, quantity, SKU and shelf location information in combination with the shelf hierarchy information.

Images