Commodity display management method, device and system

Abstract

The invention discloses a commodity display management method, device and system, and the method is applied to a digital twinborn server, and comprises the steps: constructing a digital twinborn model of shelf commodities according to a store shelf digital map, the identification information and position information of electronic price tags on a shelf guide rail, and the commodity information matched with each electronic price tag; according to a preset period, sending a shooting instruction to a shelf camera server, so that the shelf camera server controls a camera to shoot a shelf image after receiving the shooting instruction, and obtaining a commodity recognition result through a commodity recognition model; the commodity identification model is obtained by training according to a plurality of commodity images of each commodity and commodity images of similar commodities of each commodity; receiving a commodity identification result fed back by the shelf camera server, and matching the commodity identification result with the digital twin model of the shelf commodity; if matching succeeds, determining that the commodity information is correct; otherwise, determining that the commodity information is wrong. The method is high in precision and efficiency, low in cost and good in adaptability.

Description

Technical Field

[0001] This invention relates to the field of communication technology, and in particular to a method, apparatus and system for managing merchandise display. Background Technology

[0002] This section is intended to provide background or context for the embodiments of the invention set forth in the claims. The description herein is not an admission that it is prior art simply because it is included in this section.

[0003] In the retail and warehouse management sectors, merchandise positioning and display management have always been key to improving operational efficiency and enhancing customer experience. Existing merchandise positioning solutions include the following:

[0004] I. RFID-based Store Positioning and Display Management System

[0005] Existing RFID technology is widely used in the retail industry, especially in inventory management and product location. RFID tags are attached to goods, and data is collected through RFID readers. These systems can achieve product tracking, location, and inventory management to a certain extent, and are particularly suitable for large-scale retail environments. However, these systems still face some challenges:

[0006] High cost: RFID system deployment costs are high, especially for large-scale deployments, resulting in high overall expenses. RFID systems require specialized reading and writing equipment, increasing hardware procurement and maintenance costs.

[0007] Environmental interference: In complex retail environments, RFID signals are easily interfered with by metal shelves and other equipment, resulting in unstable signals and inability to guarantee positioning accuracy.

[0008] Positioning accuracy: Generally, manual handheld reader and writer is required for identification and data collection, and the identification accuracy may reach sub-meter level; non-manual positioning accuracy based on fixed antenna can generally only reach 3-5 meters, and the accuracy fluctuates greatly with environmental changes, making it difficult to achieve an automated and stable quantitative solution.

[0009] II. Indoor Positioning System Based on Wi-Fi and Bluetooth

[0010] Some stores use Wi-Fi or Bluetooth positioning technology to track merchandise and customers. These systems calculate the distance between devices and base stations or between devices by analyzing signal data such as signal strength index (RSSI) or phase difference, thereby achieving indoor positioning. While Wi-Fi and Bluetooth positioning technologies have wide applications, they also have the following limitations:

[0011] Multipath effect and interference issues: In complex indoor environments, factors such as signal reflection and refraction can affect the accuracy of positioning, leading to systematic errors.

[0012] Strong network dependence: Wi-Fi and Bluetooth positioning require a stable network environment; otherwise, positioning accuracy and system stability will be affected. Due to this need for long-term connectivity, positioning typically requires higher power consumption.

[0013] Low positioning accuracy: The positioning accuracy of Wi-Fi and Bluetooth is usually between 2 and 5 meters, which is still not ideal for refined management and precise control of shelf display.

[0014] III. Vision Recognition-Based Shelf Management System

[0015] Some existing technologies employ computer vision technology, using cameras to monitor and manage shelves in real time. These systems utilize image processing algorithms to identify products on the shelves, thereby performing tasks such as inventory management and product display optimization. However, these technological solutions also have the following drawbacks:

[0016] High processing complexity: Computer vision technology requires powerful computing resources, especially in large-scale stores, where real-time processing of large amounts of image data brings a significant computational burden.

[0017] Sensitive to light and angle: Visual recognition systems have high requirements for changes in ambient light, angle, and camera configuration, and are easily affected by changes in the actual scene, leading to a decrease in recognition accuracy.

[0018] Reliance on high-definition cameras and high-resolution images: For some low-cost or low-quality cameras, the image quality may be insufficient, affecting the effectiveness of shelf display management.

[0019] High deployment and maintenance costs: Shelf cameras require extensive manual annotation to train the visual recognition model, resulting in high deployment costs. Furthermore, battery maintenance and the computing power requirements of the equipment make maintenance costs very high.

[0020] Unable to achieve full-scenario deployment: Shelf cameras have significant limitations in terms of comprehensive deployment across the entire retail environment, especially when mobile asset tracking is required.

[0021] In summary, while existing technologies have made some explorations and attempts in store positioning and shelf management, these technologies generally suffer from problems such as insufficient accuracy, strong environmental dependence, high cost, and high reliance on employees. Moreover, most of the identification systems lack self-verification and learning processes, and the positioning accuracy gradually decreases as goods move randomly within the store over time. Ultimately, a large amount of manual intervention is still needed to maintain accuracy indicators to meet the business needs of supermarkets. Summary of the Invention

[0022] This invention provides a product display management method applied to a digital twin server, which offers high accuracy and efficiency, low cost, and good adaptability, including:

[0023] Read the digital map of store shelves from the map server, the digital map of shelves including shelf information;

[0024] Obtain the identification and location information of electronic price tags on the shelf guide rails;

[0025] Read the product information that matches each electronic shelf label from the electronic shelf label server, and construct a digital twin model of the shelf products based on the digital map of the store shelves, the identification and location information of the electronic shelf labels on the shelf rails, and the product information that matches each electronic shelf label.

[0026] According to a preset cycle, a shooting command is sent to the shelf camera server so that the shelf camera server controls the camera to shoot images of the shelf after receiving the shooting command, and obtains product recognition results through the product recognition model; the product recognition model is trained based on multiple product images of each product and product images of similar products of each product.

[0027] The system receives the product recognition results from the shelf camera server and matches them with the digital twin model of the products on the shelf. If the match is successful, the product information is confirmed to be correct; otherwise, the product information is confirmed to be incorrect.

[0028] This invention provides a merchandise display management device applied to a digital twin server, which features high accuracy and efficiency, low cost, and good adaptability, comprising:

[0029] The store shelf digital map reading module is used to read the store shelf digital map from the map server, the shelf digital map including shelf information;

[0030] The guide rail recognition module is used to obtain the identification and location information of the electronic price tags on the shelf guide rails;

[0031] The digital twin model building module is used to read the product information that matches each electronic shelf tag from the electronic shelf tag server, and build a digital twin model of the shelf products based on the digital map of the store shelves, the identification and location information of the electronic shelf tags on the shelf rails, and the product information that matches each electronic shelf tag.

[0032] The shooting instruction sending module is used to send shooting instructions to the shelf camera server according to a preset period, so that the shelf camera server controls the camera to shoot shelf images after receiving the shooting instructions, and obtains product recognition results through the product recognition model; the product recognition model is trained based on multiple product images of each product and product images of similar products of each product.

[0033] The matching module receives the product recognition results from the shelf camera server and matches them with the digital twin model of the products on the shelf. If the match is successful, the product information is confirmed to be correct; otherwise, the product information is confirmed to be incorrect.

[0034] This invention provides a merchandise display management system that is highly accurate and efficient, low in cost, and adaptable, including: a map server, a shelf camera server, an electronic price tag server, and a digital twin server that implements the aforementioned devices;

[0035] The map server is used to store digital maps of store shelves;

[0036] The shelf camera server is used to control the camera to capture images of the shelf after receiving a shooting command, and after obtaining the product recognition results through the product recognition model, it feeds them back to the digital twin server.

[0037] An electronic shelf label server is used to store product information that matches each electronic shelf label.

[0038] This invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the above-described merchandise display management method.

[0039] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described merchandise display management method.

[0040] This invention also provides a computer program product, which includes a computer program that, when executed by a processor, implements the above-described merchandise display management method.

[0041] In this embodiment of the invention, the digital twin server can read the digital map of the store shelves from the map server, the digital map of the shelves including shelf information; obtain the identification information and location information of the electronic price tags on the shelf rails; read the product information matching each electronic price tag from the electronic price tag server, and construct a digital twin model of the shelf products based on the store shelf digital map, the identification information and location information of the electronic price tags on the shelf rails, and the product information matching each electronic price tag; send shooting instructions to the shelf camera server according to a preset cycle, so that the shelf camera server controls the camera to shoot shelf images after receiving the shooting instructions, and obtains product recognition results through a product recognition model; the product recognition model is trained based on multiple product images of each product and product images of similar products of each product; receive the product recognition results fed back by the shelf camera server, and match the product recognition results with the digital twin model of the shelf products; if the match is successful, the product information is determined to be correct; otherwise, the product information is determined to be incorrect. Through the above steps, based on the multimodal data fusion of the shelf camera and the rails, combined with historical records and real-time recognition results, a high-precision 3D positioning closed loop is established. Automated verification and optimization mechanisms ensure the system maintains high stability and accuracy in dynamic environments. It possesses self-learning capabilities, automatically optimizing the product recognition model and digital twin model to adapt to the dynamic adjustments required for different products. Real-time anomaly detection reduces manual intervention while improving the efficiency and accuracy of product display, lowering overall costs, and making it widely applicable to various retail formats. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings:

[0043] Figure 1 This is a flowchart of the merchandise display management method in an embodiment of the present invention;

[0044] Figure 2 This is a schematic diagram illustrating the principle of merchandise display management in an embodiment of the present invention;

[0045] Figure 3 This is a flowchart illustrating the training process of the product recognition model in an embodiment of the present invention.

[0046] Figure 4 This is a schematic diagram of the structure of the merchandise display management system in an embodiment of the present invention;

[0047] Figure 5 This is a schematic diagram of the structure of the merchandise display management system in an embodiment of the present invention;

[0048] Figure 6 This is a schematic diagram of a computer device in an embodiment of the present invention. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0050] Figure 1 This is a flowchart of a product display management method in an embodiment of the present invention, applied to a digital twin server, including:

[0051] Step 101: Read the digital map of store shelves from the map server, wherein the digital map of shelves includes shelf information;

[0052] Step 102: Obtain the identification and location information of the electronic price tags on the shelf guide rails;

[0053] Step 103: Read the product information that matches each electronic shelf label from the electronic shelf label server, and construct a digital twin model of the shelf products based on the digital map of the store shelves, the identification and location information of the electronic shelf labels on the shelf rails, and the product information that matches each electronic shelf label.

[0054] Step 104: According to a preset cycle, a shooting command is sent to the shelf camera server so that the shelf camera server controls the camera to shoot shelf images after receiving the shooting command, and obtains product recognition results through the product recognition model; the product recognition model is trained based on multiple product images of each product and product images of similar products of each product.

[0055] Step 105: Receive the product recognition results from the shelf camera server and match the product recognition results with the digital twin model of the products on the shelf; if the match is successful, the product information is confirmed to be correct; otherwise, the product information is confirmed to be incorrect.

[0056] Compared to existing technologies, this system establishes a high-precision 3D positioning closed loop by fusing multimodal data from shelf cameras and guide rails, combining historical records and real-time recognition results. An automated verification and optimization mechanism ensures the system maintains high stability and accuracy in dynamic environments. It possesses self-learning capabilities, automatically optimizing the product recognition model and digital twin model to adapt to the dynamic adjustment needs of different products. It can detect anomalies in real time, reducing manual intervention while improving the efficiency and accuracy of product display, lowering overall costs, and making it widely applicable to various retail formats.

[0057] Figure 2This is a schematic diagram of the merchandise display management principle in an embodiment of the present invention. The overall system requires a map server, an electronic price tag server, a digital twin server, a user system, a shelf camera server, cameras, a base station (electronic price tag / shelf camera controller), and a guide rail. The map server, electronic price tag server, user system, and shelf camera server can all be independent servers or different parts of the same server. The electronic price tags can communicate with each server via narrowband. In one case, the camera includes a price tag communication unit. When the shelf camera server sends instructions to the camera, it does so via narrowband communication. When the camera transmits images back, it does so via broadband transmission. The base station can control the electronic price tags and shelf cameras. Therefore, the base station can act as a controller for the electronic price tags and shelf cameras. The following describes the process in conjunction with steps 101-105.

[0058] In step 101, the digital map of the store shelves from the map server is read, and the digital map of the shelves includes shelf information;

[0059] Specifically, the digital shelf map includes the store's area information, the aisle information for each area, the shelf information for each aisle, the layer information for each shelf, and the location information for each layer;

[0060] This information is closely linked to the 3D position of the shelf camera, enabling the shelf camera to perform precise monitoring based on the shelf number it monitors and the location information of the shelf camera.

[0061] Map servers can support functions including, but not limited to, web-based map editing, mobile terminal interaction, and location information configuration.

[0062] In step 102, the identification information and location information of the electronic price tags on the shelf guide rail are obtained;

[0063] Each electronic shelf label is mounted on a guide rail with multiple sequentially arranged wireless tags (e.g., NFC tags). After being installed and bound to a specific shelf and its tiered guide rails, the wireless tags contain guide rail information and tag number sequence information. The precise location (shelf location information) can be obtained by reading each wireless tag. The identification information of the electronic shelf label can be its ID.

[0064] The binding methods include, but are not limited to: 1) using a mobile terminal to sequentially read all the wireless tags on the same guide rail to generate a guide rail ID, thus forming a binding relationship between the wireless tags and the guide rail ID; 2) using a mobile terminal to scan the shelf ID, and then sequentially scan the guide rail ID of each layer to form a binding relationship between the shelf and the guide rail ID.

[0065] When moving or reinstalling electronic shelf labels, changes in surrounding wireless signals are detected, allowing specific electronic shelf labels or shelf-level electronic shelf labels to read their new locations and update the digital map of the store shelves.

[0066] When a product is re-attached to a price tag, a location reading process is triggered to obtain the new location information.

[0067] The electronic tags have NFC read / write capabilities, enabling precise positioning when used with the guide rail, while simultaneously displaying product information such as prices. By binding with products on the shelf, real-time, high-precision 3D positioning of the merchandise display is achieved.

[0068] When the electronic shelf label is installed on the guide rail, its built-in NFC read / write module can read the position information in the NFC tag chip of the guide rail, which serves as the position information of the electronic shelf label. The electronic shelf label management server stores the tag information of the electronic shelf label.

[0069] In step 103, the product information matching each electronic shelf label in the electronic shelf label server is read, and a digital twin model of the shelf products is constructed based on the digital map of the store shelves, the identification and location information of the electronic shelf labels on the shelf rails, and the product information matching each electronic shelf label.

[0070] The electronic shelf label management server stores product information that matches each electronic shelf label;

[0071] Based on the store's regional information, the aisle information of each region, the shelf information of each aisle, the layer information of each shelf, the identification and location information of the electronic price tags on the shelf rails, and the product information matching each electronic price tag, a digital twin model of the shelf products can be constructed.

[0072] The digital twin model can be sent to a map server for storage and display.

[0073] In step 104, a shooting command is sent to the shelf camera server according to a preset cycle, so that the shelf camera server controls the camera to shoot shelf images after receiving the shooting command, and obtains product recognition results through the product recognition model; the product recognition model is trained based on multiple product images of each product and product images of similar products of each product.

[0074] In the above embodiment, a camera is installed on the shelf. The camera's task is to periodically or continuously capture images of the shelf display it monitors and transmit the captured images back to the shelf camera server. The shelf camera server obtains the product recognition result through a product recognition model, which is obtained through training.

[0075] The shelf camera server can obtain the binding relationship between the camera and the shelf it photographs. In other words, the shelf camera server can identify which specific shelf the camera is photographing. The ways to obtain the binding relationship include, but are not limited to: recognizing the QR code information of the shelf in the image captured by the camera to obtain the shelf ID; recognizing the specific flashing sequence or page cutting content of the price tag in the image captured by the camera to obtain the shelf ID; and pre-binding the camera and the shelf ID.

[0076] In step 105, the product recognition result fed back by the shelf camera server is received, and the product recognition result is matched with the digital twin model of the product on the shelf; if the match is successful, the product information is determined to be correct; otherwise, the product information is determined to be incorrect.

[0077] Because the digital twin model includes complete information such as the identification and location information of electronic price tags on the store shelf rails and the product information matched with each electronic price tag, the product identification results can be matched with the digital twin model to determine whether the product information is incorrect.

[0078] Through the above steps, continuous monitoring of price tag and product positions is achieved to ensure consistency with records. Real-time verification via shelf cameras generates dynamic adjustment suggestions. Incorrect product information includes two scenarios:

[0079] The first scenario involves missing product information, indicating an out-of-stock situation. In this case, the product identification result is "no product detected." For example, if no product shape or packaging features are detected in the product area corresponding to an electronic shelf tag, that area is marked as out of stock. By combining the product information matched with each electronic shelf tag in the digital twin model, it is immediately possible to determine which product is out of stock and trigger a restocking reminder to the user's system. Because the location information of the electronic shelf tags in the digital twin model is precise, out-of-stock alarms can be pinpointed to specific shelves and specific products, facilitating timely restocking by employees.

[0080] The second type is misplaced merchandise. In this case, the merchandise recognition result shows that the merchandise has been identified, but the identified merchandise does not match the merchandise information matched with each electronic price tag in the digital twin model. Further anomaly confirmation is needed: for example, if a bag of snacks is detected in the beverage area, and its shape and packaging determine that it belongs to the snack category and should not be in that location, then this is confirmed as an incorrect display. This anomaly is recorded and marked, and the system prompts the store staff to correct the merchandise placement.

[0081] If product information is found to be incorrect, the system notifies the user to correct it by illuminating a light, switching the label page, or directly displaying a modification suggestion on the electronic shelf tag. Through these steps, product feature acquisition and model training are automatically completed. Using a point-and-click approach, the shelf camera server can pinpoint the location and characteristics of new products, completing the adaptation and learning of new products without manual intervention. The point-and-click approach refers to a method that allows the camera to quickly and accurately locate new products and obtain their relevant characteristics. The electronic shelf tag can perform location verification with the shelf camera server. By binding a shelf tag to the product to be verified and using a point-and-click approach to illuminate or switch the label page of a specific electronic shelf tag, the shelf camera server can perform location verification based on the identified shelf tag features. Automatic closed-loop verification can promptly detect and correct positioning errors, ensuring stability and accuracy.

[0082] In one embodiment, the product recognition model employs a contrastive learning loss function during training. This contrastive learning loss function enables the image recognition model to map the features of different product images of the same product to similar vector spaces, while mapping the features of product images of different products to different vector spaces.

[0083] Contrastive learning is a self-supervised learning method whose core objective is to guide a product recognition model to learn discriminative feature representations by constructing comparative relationships between similar and dissimilar samples. In product image recognition scenarios, the design of the contrastive learning loss function follows this logic:

[0084] Positive sample pairs (same product): force the product recognition model to map different image features of the same product to close positions in the vector space (i.e., small feature distance).

[0085] Negative sample pairs (different products): force the product recognition model to map the image features of different products to different locations in the vector space (i.e., large feature distance).

[0086] The advantages of contrastive learning in product image recognition include:

[0087] Enhanced feature judgment: Different perspectives, lighting, and background images of the same product are mapped to similar feature spaces, while the features of different products are significantly distinguished, improving the accuracy of cross-scene recognition.

[0088] Reduced reliance on labeling: No need for extensive manual labeling of product categories; training can be performed simply by constructing positive and negative sample pairs (such as different images of the same product as positive samples, and images of different products as negative samples), thus reducing data labeling costs.

[0089] Improved generalization ability: The features learned by the product recognition model focus more on the essential attributes of the product (such as shape and texture) rather than surface noise (such as background and shooting angle), and have better generalization ability for new products or unseen images.

[0090] Figure 3 This is a flowchart illustrating the training process of the product recognition model in one embodiment of the present invention. In one embodiment, the training steps of the product recognition model include:

[0091] Step 301: Collect multiple product images of each product under different angles and lighting conditions, as well as product images of similar products for each product;

[0092] Product images can be preprocessed, including image scaling, cropping, and normalization, to make them meet the input requirements of the product recognition model.

[0093] Step 302: Label all product images to obtain the initial training dataset;

[0094] In one embodiment, after labeling all product images to obtain an initial training dataset, the process includes:

[0095] Collect multimodal data for each product and similar products. The multimodal data includes video introduction data, category tag data in the supply chain system, and sales data, inventory data, and specification data of similar products.

[0096] Product images and multimodal data are associated and labeled.

[0097] The aforementioned association annotations can provide richer feature dimensions for the product recognition model, enabling it to learn new product features from multiple perspectives and improve its ability to recognize complex products.

[0098] Step 303: Determine the architecture of the product recognition model and initialize the deep learning model architecture;

[0099] Step 304: Construct the contrastive learning loss function;

[0100] Step 305: Use the initial training dataset to train the product recognition model in multiple rounds. During the training process, continuously adjust the parameters of the product recognition model to optimize the contrastive learning loss function.

[0101] After training, it can process new product images. Of course, if there are new products each time, it needs to be retrained using the data of the new products.

[0102] In one embodiment, the method further includes:

[0103] Once the product information is confirmed to be correct, a positive reward signal is generated; if the product information is confirmed to be incorrect, a negative reward signal is generated.

[0104] The reward signal is sent to the shelf camera server, so that the shelf camera server can adjust the parameters of the product recognition model based on the reward signal using a reinforcement learning algorithm.

[0105] Specifically, the reward signal is generated based on metrics such as the model's accuracy and false positive rate in identifying product information in real-world scenarios. When the model accurately identifies a new product (for example, by determining whether a product is new through image recognition and obtaining verification from the new product identifier during actual on-site verification), a positive reward, such as +10 points, is given.

[0106] If the model makes a misclassification (e.g., identifying a non-new product as a new product, or a new product as a non-new product), a negative reward, such as -5 points, is given. More complex reward functions can also be designed based on the specific values ​​of the recognition accuracy and misclassification rate; for example, a positive reward could be given for every 1% increase in accuracy, and a negative reward could be given for every 1% increase in the misclassification rate.

[0107] In one embodiment, the method further includes:

[0108] When a new product appears on the shelf, the system receives a new recognition instruction from the user system and forwards it to the shelf camera server. The shelf camera server then controls the camera to capture an image of the product after receiving the new recognition instruction, and performs transfer learning training on the pre-trained product recognition model corresponding to the new product based on the captured product image.

[0109] In the transfer learning training, the parameters of the pre-trained product recognition model corresponding to the new product are used as the initial values, and the training is carried out based on the captured product image and the annotation corresponding to the new product.

[0110] When encountering new product categories or large-scale packaging changes, the system utilizes a pre-trained product recognition model corresponding to the new product for transfer learning. The parameters of the pre-trained model are used as initial values ​​and fine-tuned for the new product data. Simultaneously, the system automatically adjusts the range of fine-tuned parameters and hyperparameters such as the learning rate, improving the convergence speed and performance of the product recognition model on new data while reducing training time and data requirements.

[0111] The above process requires no manual annotation and is a form of unsupervised learning (or self-supervised learning) mechanism. As more image samples are collected during the sales process, the accuracy of the product recognition model in identifying new products will gradually improve. Furthermore, if the product recognition model makes a misjudgment during operation, it can be corrected using the correct product identity provided by ESL (i.e., self-calibration), thereby continuously optimizing the product recognition model parameters.

[0112] In one embodiment, the method further includes:

[0113] If the accuracy of labeling new products does not meet the accuracy requirements during transfer learning training, the shelf camera server sends a labeling request to the user system. After receiving the labeling feedback from the user system, the shelf camera server trains the product recognition model based on the captured product images and the corresponding labels for the new products. The labeling request includes the product images.

[0114] In the above embodiments, the shelf camera server not only passively collects image samples during the sales process, but also expands the database by actively learning strategies to select more valuable samples. When encountering images of new products that cannot be accurately identified, the shelf camera server analyzes the uncertainty of the image features, selects a subset of samples from images with high uncertainty, and automatically sends annotation requests to store clerks or professionals in the user system. The requests include the image and existing relevant product information, guiding the annotators to complete the annotations more efficiently. These high-quality annotated samples, which have been manually verified, are preferentially added to the training set, accelerating the product recognition model's learning and optimization of new products.

[0115] The user system can provide customer demand information and inventory status, further optimizing product display and arrangement management. It can implement, but is not limited to, business functions such as optimal route navigation, 3D display maps, sales heatmaps, out-of-stock status and duration, providing retail customers and consumers with a more intuitive and real-time store management and shopping experience.

[0116] In one embodiment, the method further includes:

[0117] If a product on the shelf cannot be photographed due to obstruction within a preset time period, an error message indicating that it cannot be photographed will be sent to the user's system.

[0118] In real-world scenarios, special situations may arise, such as obstructed camera view. For example, a customer standing in front of a shelf to retrieve an item might temporarily obscure the camera's view, or an item might fall into an abnormal position. To address these issues, an anomaly detection and alert mechanism is implemented. When persistent uncertainty in recognition is detected (e.g., a location being obstructed for multiple frames), an anomaly is identified, and the system alerts store staff to inspect the scene. This process ensures that the accuracy of the digital twin model of the shelf is maintained even under less-than-ideal conditions, with human-machine collaboration jointly guaranteeing the quality of display management.

[0119] In one embodiment, the method further includes:

[0120] When replenishing goods, record the replenishment time. If it is determined that the goods are out of stock due to incorrect product information, record the out-of-stock time. The time between the replenishment time and the out-of-stock time shall be used as the time when the goods are sold.

[0121] Based on the product's sales time, the shelf location of the product, and a digital map of the store's shelves, a product sales heat map is generated and sent to the user system, enabling the user system to make intelligent recommendations on product placement based on the product sales heat map.

[0122] In addition to basic data such as restocking time and out-of-stock time, environmental factors data are incorporated, such as real-time temperature, humidity, and customer traffic density within the store. The potential relationship between environmental factors and product sales time is analyzed. For example, during hot summer months, the time from restocking to selling cold drinks may be shorter. These relationships can be used to weight and optimize the product sales heatmap, making the sales trends reflected in the heatmap more closely reflect the actual scenario.

[0123] Data such as shipping times and transportation durations from the upstream supply chain can be incorporated. By analyzing the entire process from supplier shipment to product replenishment and sale on store shelves, potential delays in the supply chain can be identified, thereby optimizing supply chain management. Additionally, supply chain efficiency-related indicators can be added to the heat map, providing users with more comprehensive product sales information.

[0124] Alternatively, a dynamic time slicing approach can be used. The time after a product is listed is divided into multiple short time periods, and changes in the product's status (such as whether customers browse or touch it) are analyzed within each period. Based on the frequency and pattern of product status changes within different time periods, the actual time when a product is noticed and purchased by a customer can be estimated more accurately. This makes the record of product sales time more consistent with actual sales behavior, thereby optimizing the time accuracy of the product sales heatmap.

[0125] Time series forecasting models can be built based on historical data on product replenishment times, stockout times, and sales times. These models can predict replenishment times, stockout times, and potential sales time ranges for various products during different seasons and promotional periods. Using these forecasts, product sales heatmaps can be adjusted and early warnings provided in advance. For example, if a promotional event predicts that a certain type of product will sell faster, that area can be highlighted on the heatmap in advance, and users can be provided with corresponding suggestions for adjusting product placement to allow for proactive preparation.

[0126] The following methods can be used when presenting a heatmap:

[0127] (1) Multi-dimensional heatmap: Building upon existing heatmaps generated based on product sales time and shelf location, this feature adds more dimensions of information. For example, color can be used to distinguish product categories, heatmap transparency can represent profit margins, and graphic size can indicate sales volume. This allows users to obtain multiple key information points simultaneously on a single heatmap, enabling a more intuitive analysis of the sales performance and profitability of different products in different locations, and providing a more comprehensive and intelligent recommendation basis for product placement.

[0128] (2) Interactive Heatmap: Develop an interactive product sales heatmap. Users can click on different areas of the heatmap in the user system to obtain detailed product sales data, including specific product names, restocking times, out-of-stock times, sales times, and sales trends. Simultaneously, users can zoom, rotate, and perform other operations to observe the relationship between store shelf layout and product sales from different angles, enhancing users' ability to explore data and making intelligent recommendations more targeted and personalized.

[0129] In addition, the digital twin model of the goods on the shelf in this embodiment of the invention can be used to form a corresponding product display diagram, which can be compared with the pre-existing product display diagram. When changes occur, the diagram can be automatically uploaded to the user system so that the user system can respond quickly.

[0130] Figure 4 This is a schematic diagram of the structure of a merchandise display management device in an embodiment of the present invention, applied to a digital twin server, including:

[0131] The store shelf digital map reading module 401 is used to read the store shelf digital map from the map server, wherein the shelf digital map includes shelf information;

[0132] The guide rail recognition module 402 is used to obtain the identification information and position information of the electronic price tag on the shelf guide rail;

[0133] The digital twin model building module 403 is used to read the product information matching each electronic shelf label from the electronic shelf label server, and build a digital twin model of the shelf products based on the digital map of the store shelves, the identification and location information of the electronic shelf labels on the shelf rails, and the product information matching each electronic shelf label.

[0134] The shooting instruction sending module 404 is used to send shooting instructions to the shelf camera server according to a preset period, so that the shelf camera server controls the camera to shoot shelf images after receiving the shooting instructions, and obtains product recognition results through the product recognition model; the product recognition model is trained based on multiple product images of each product and product images of similar products of each product.

[0135] The matching module 405 is used to receive the product recognition results fed back by the shelf camera server, and match the product recognition results with the digital twin model of the products on the shelf; if the match is successful, the product information is determined to be correct; otherwise, the product information is determined to be incorrect.

[0136] In one embodiment, the product recognition model employs a contrastive learning loss function during training. This contrastive learning loss function enables the image recognition model to map the features of different product images of the same product to similar vector spaces, while mapping the features of product images of different products to different vector spaces.

[0137] In one embodiment, the training steps of the product recognition model include:

[0138] Collect multiple product images of each product from different angles and under different lighting conditions, as well as product images of similar products for each product;

[0139] All product images were labeled to obtain the initial training dataset;

[0140] Determine the architecture of the product recognition model and initialize the deep learning model architecture;

[0141] Construct a contrastive learning loss function;

[0142] The product recognition model is trained multiple times using the initial training dataset. During the training process, the parameters of the product recognition model are continuously adjusted to optimize the contrastive learning loss function.

[0143] In one embodiment, after labeling all product images to obtain an initial training dataset, the process includes:

[0144] Collect multimodal data for each product and similar products. The multimodal data includes video introduction data, category tag data in the supply chain system, and sales data, inventory data, and specification data of similar products.

[0145] Product images and multimodal data are associated and labeled.

[0146] In one embodiment, the apparatus further includes a reward signal generation module, used for:

[0147] Once the product information is confirmed to be correct, a positive reward signal is generated; if the product information is confirmed to be incorrect, a negative reward signal is generated.

[0148] The reward signal is sent to the shelf camera server, so that the shelf camera server can adjust the parameters of the product recognition model based on the reward signal using a reinforcement learning algorithm.

[0149] In one embodiment, the device further includes a new identification module for:

[0150] When a new product appears on the shelf, the system receives a new recognition instruction from the user system and forwards it to the shelf camera server. The shelf camera server then controls the camera to capture an image of the product after receiving the new recognition instruction, and performs transfer learning training on the pre-trained product recognition model corresponding to the new product based on the captured product image.

[0151] In the transfer learning training, the parameters of the pre-trained product recognition model corresponding to the new product are used as the initial values, and the product recognition model is trained based on the captured product image and the annotation corresponding to the new product.

[0152] In one embodiment, if the accuracy of labeling new products does not meet the accuracy requirements during transfer learning training, the shelf camera server sends a labeling request to the user system. After receiving the labeling feedback from the user system, the shelf camera server trains the product recognition model based on the captured product images and the corresponding labels for the new products. The labeling request includes the product images.

[0153] In one embodiment, the device further includes a reminder module for:

[0154] If a product on the shelf cannot be photographed due to obstruction within a preset time period, an error message indicating that it cannot be photographed will be sent to the user's system.

[0155] Figure 5 This is a schematic diagram of the structure of a merchandise display management system in an embodiment of the present invention. The system includes a map server 501, a shelf camera server 502, an electronic price tag server 503, and a digital twin server 504 that implements the aforementioned devices.

[0156] The map server 501 is used to store digital maps of store shelves;

[0157] The shelf camera server 502 is used to control the camera to capture images of the shelf after receiving a shooting instruction, and to obtain the product recognition results through the product recognition model and then feed them back to the digital twin server.

[0158] Electronic shelf label server 503 is used to store product information that matches each electronic shelf label.

[0159] In one embodiment, the system further includes a user system 505, used for:

[0160] When a new product appears on the shelf, a new identification instruction is sent to the digital twin server, which then forwards the new identification instruction to the shelf camera server.

[0161] The methods, apparatus, and systems proposed in the embodiments of the present invention have the following beneficial effects:

[0162] 3D Positioning Closed-Loop Management: Based on multimodal data fusion from shelf cameras and guide rails integrating NFC tag chips, combined with historical records and real-time recognition results, a high-precision 3D positioning closed loop is established. Automated verification and optimization mechanisms ensure that the system maintains high stability and high accuracy in dynamic environments.

[0163] Self-learning and dynamic adaptation: It has self-learning capabilities and can automatically optimize the product recognition model and digital twin model to adapt to the dynamic adjustment needs of different products.

[0164] Intelligent anomaly detection and optimization: It can detect anomalies in real time and generate optimization suggestions, reducing manual intervention while improving the efficiency and accuracy of product display.

[0165] The solution of this invention can be widely applied to various retail formats:

[0166] Applicable scenarios: This invention can be deployed in various locations requiring sophisticated shelf management, including large chain supermarkets, convenience stores, pharmacies, and even clothing and electronics retail. Its advantages are particularly pronounced in retail environments with a wide variety of goods and strict display requirements.

[0167] Improve replenishment and picking efficiency: Real-time out-of-stock monitoring allows store staff to replenish stock promptly, preventing product shortages and improving shelf availability. Simultaneously, accurate display location information guides pickers to quickly locate products, increasing order fulfillment efficiency.

[0168] Data-driven display decisions: Continuously collect data on the quantity of each product on the shelves, such as stockout frequency and inventory turnover rate. This data can be used to calculate display performance indicators (such as sales per unit shelf space), providing managers with a basis for evaluating merchandise display strategies. Brands can also use these indicators to make bidding or leasing decisions for display space, placing products in higher-value locations to increase sales.

[0169] Reduce labor costs and enhance automation: Traditional shelf management requires a significant investment of manpower for inspection and record-keeping. This solution automates the entire process from data collection to analysis and early warning, significantly reducing the need for manual intervention. In particular, through its self-learning mechanism, the system can automatically adapt to new products and changes without the need for manual re-labeling and retraining of the model, resulting in a significant reduction in long-term operation and maintenance costs. Retailers can then allocate manpower to higher-value services and management, thereby improving overall operational efficiency.

[0170] This invention also provides a computer device. Figure 6This is a schematic diagram of a computer device in an embodiment of the present invention. The computer device 600 includes a memory 610, a processor 620, and a computer program 630 stored in the memory 610 and executable on the processor 620. When the processor 620 executes the computer program 630, it implements the above-mentioned merchandise display management method.

[0171] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described merchandise display management method.

[0172] This invention also provides a computer program product, which includes a computer program that, when executed by a processor, implements the above-described merchandise display management method.

[0173] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0174] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0175] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0176] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0177] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for managing merchandise display, characterized in that, Applications in digital twin servers include: Read the digital map of store shelves from the map server, the digital map of shelves including shelf information; Obtain the identification and location information of electronic price tags on the shelf guide rails; Read the product information that matches each electronic shelf label from the electronic shelf label server, and construct a digital twin model of the shelf products based on the digital map of the store shelves, the identification and location information of the electronic shelf labels on the shelf rails, and the product information that matches each electronic shelf label. According to a preset cycle, a shooting command is sent to the shelf camera server so that the shelf camera server controls the camera to shoot images of the shelf after receiving the shooting command, and obtains product recognition results through the product recognition model; the product recognition model is trained based on multiple product images of each product and product images of similar products of each product. The system receives the product recognition results from the shelf camera server and matches them with the digital twin model of the products on the shelf. If the match is successful, the product information is confirmed to be correct; otherwise, the product information is confirmed to be incorrect.

2. The method as described in claim 1, characterized in that, During the training process, the product recognition model employs a contrastive learning loss function, which enables the image recognition model to map the features of different product images of the same product to similar vector spaces, while mapping the features of product images of different products to different vector spaces.

3. The method as described in claim 2, characterized in that, The training steps for the product recognition model include: Collect multiple product images of each product from different angles and under different lighting conditions, as well as product images of similar products for each product; All product images were labeled to obtain the initial training dataset; Determine the architecture of the product recognition model and initialize the deep learning model architecture; Construct a contrastive learning loss function; The product recognition model is trained multiple times using the initial training dataset. During the training process, the parameters of the product recognition model are continuously adjusted to optimize the contrastive learning loss function.

4. The method as described in claim 3, characterized in that, After labeling all product images to obtain the initial training dataset, the following steps are taken: Collect multimodal data for each product and similar products. The multimodal data includes video introduction data, category tag data in the supply chain system, and sales data, inventory data, and specification data of similar products. Product images and multimodal data are associated and labeled.

5. The method as described in claim 1, characterized in that, Also includes: Once the product information is confirmed to be correct, a positive reward signal is generated; if the product information is confirmed to be incorrect, a negative reward signal is generated. The reward signal is sent to the shelf camera server, so that the shelf camera server can adjust the parameters of the product recognition model based on the reward signal using a reinforcement learning algorithm.

6. The method as described in claim 1, characterized in that, Also includes: When a new product appears on the shelf, the system receives a new recognition instruction from the user system and forwards it to the shelf camera server. The shelf camera server then controls the camera to capture an image of the product after receiving the new recognition instruction, and performs transfer learning training on the pre-trained product recognition model corresponding to the new product based on the captured product image. In the transfer learning training, the parameters of the pre-trained product recognition model corresponding to the new product are used as the initial values, and the training is carried out based on the captured product image and the annotation corresponding to the new product.

7. The method as described in claim 6, characterized in that, Also includes: If the accuracy of labeling new products does not meet the accuracy requirements during transfer learning training, the shelf camera server sends a labeling request to the user system. After receiving the labeling feedback from the user system, the shelf camera server trains the product recognition model based on the captured product images and the corresponding labels for the new products. The labeling request includes the product images.

8. The method as described in claim 1, characterized in that, Also includes: If a product on the shelf cannot be photographed due to obstruction within a preset time period, an error message indicating that it cannot be photographed will be sent to the user's system.

9. A merchandise display management device, characterized in that, Applications in digital twin servers include: The store shelf digital map reading module is used to read the store shelf digital map from the map server, the shelf digital map including shelf information; The guide rail recognition module is used to obtain the identification and location information of the electronic price tags on the shelf guide rails; The digital twin server construction module is used to read the product information matched with each electronic shelf tag in the electronic shelf tag server, and construct a digital twin model of the shelf products based on the digital map of the store shelves, the identification and location information of the electronic shelf tags on the shelf rails, and the product information matched with each electronic shelf tag. The shooting instruction sending module is used to send shooting instructions to the shelf camera server according to a preset period, so that the shelf camera server controls the camera to shoot shelf images after receiving the shooting instructions, and obtains product recognition results through the product recognition model; the product recognition model is trained based on multiple product images of each product and product images of similar products of each product. The matching module receives the product recognition results from the shelf camera server and matches them with the digital twin model of the products on the shelf. If the match is successful, the product information is confirmed to be correct; otherwise, the product information is confirmed to be incorrect.

10. A merchandise display management system, characterized in that, include: Map server, shelf camera server, electronic price tag server, and digital twin server corresponding to the device described in claim 9; The map server is used to store digital maps of store shelves; The shelf camera server is used to control the camera to capture images of the shelf after receiving a shooting command, and after obtaining the product recognition results through the product recognition model, it feeds them back to the digital twin server. An electronic shelf label server is used to store product information that matches each electronic shelf label.

11. The system as claimed in claim 10, characterized in that, It also includes a user system, used for: When a new product appears on the shelf, a new identification instruction is sent to the digital twin server, which then forwards the new identification instruction to the shelf camera server.

12. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method according to any one of claims 1 to 6.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method of any one of claims 1 to 6.

14. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the method of any one of claims 1 to 6.

Images