Commodity identification method and system based on positioning information, shopping cart, and processor

Abstract

The present application discloses a commodity identification method and system based on positioning information, a shopping cart, and a processor. The method comprises: once a target shopping cart has entered a shelf aisle, the target shopping cart is connected to a nearest sensor and obtains location information of the connected sensor as preliminary positioning information of the shopping cart; a control device sends wake-up information to an acquisition device in the aisle on the basis of a shelf aisle identifier that is determined on the basis of the preliminary positioning information; the acquisition device captures a picture of a shelf that the acquisition device faces, and sends the picture and the location of the acquisition device to a processor; when the processor identifies that the shopping cart is present in the shelf picture, the processor uses the location information of the acquisition device corresponding to the picture as final positioning information; the target shopping cart determines commodity set information within a final positioning range; and on the basis of the commodity set information, commodity identification within a limited range is carried out on a target commodity.

Description

Location-based product identification methods, systems, shopping carts, and processors

[0001] Related applications

[0002] This application claims priority to Chinese Patent Application No. 202410564757.7, filed on May 8, 2024, and incorporates the entire contents of the aforementioned patent application as part of this application. Technical Field

[0003] This application relates to the field of self-service shopping technology, and in particular to a product identification method, system, shopping cart, and processor based on location information. Background Technology

[0004] This section is intended to provide background or context for the embodiments of this application set forth in the claims. The description herein is not an admission that it is prior art simply because it is included in this section.

[0005] In recent years, with the continuous development of technologies such as the Internet of Things, artificial intelligence, big data analytics, mobile payment, and smart hardware, the supermarket industry has been transforming towards intelligence and digitalization in order to better meet customer needs, improve the customer shopping experience, and optimize merchant operations. Smart shopping carts have emerged in this context. The main purpose of smart shopping carts is to simplify the traditional steps of shopping and payment, eliminating the need for customers to interact with cashiers at the checkout counter, thus enhancing the shopping experience. For merchants, this also reduces the need for human cashiers, thereby reducing operating costs.

[0006] Generally, smart shopping carts have a range of advanced features, including automatic scanning and checkout, navigation and product location, item recognition and weighing, personalized recommendations and advertising, data analysis and inventory management. Self-service scanning and checkout is one of its core functions. It utilizes smart hardware and algorithms to intelligently detect and recognize products or barcodes, update the shopping list on a tablet in real time, and finally, the shopper checks the list and proceeds to checkout.

[0007] During the shopping process, the identification of goods in the shopping cart is the core of self-service shopping. Accurately identifying the items put in or taken out is a basic requirement. Only by effectively identifying the goods can the efficiency of shopping checkout be improved and abnormal shopping behavior be truly prevented.

[0008] However, in actual supermarket scenarios, due to the large variety of products, diverse product angles, similar product packaging, and lighting conditions, it is often difficult to achieve high accuracy in product recognition. Even with a large amount of training data to train the machine learning model, it is still difficult to achieve accurate product recognition in the case of a large number of product categories.

[0009] Therefore, in the current supermarket shopping process, the accuracy and efficiency of shopping cart product recognition are both low. Summary of the Invention

[0010] This application provides a product recognition method based on location information. This method is used in a system to assist the shopping cart in recognizing target products within a limited range based on the shopping cart's location information, thereby improving the accuracy and speed of product recognition. The method includes:

[0011] When the target shopping cart enters the aisle through the end of the shelf aisle, it connects to the nearest sensor and obtains the location information of the connected sensor as the initial positioning information of the target shopping cart. The initial positioning information is then sent to the data acquisition and control device. The initial positioning information includes the shelf aisle identifier determined by the unique identifier of the connected sensor.

[0012] When the data collection and control device determines that the target shopping cart is entering the shelf aisle, it sends a wake-up message to the data collection device in that shelf aisle according to the shelf aisle identifier; the data collection device is deployed on at least one shelf in the supermarket, and each data collection device is pre-configured with location information;

[0013] After receiving the wake-up information, the data acquisition device begins to periodically or continuously capture images of the shelf it faces, and sends the shelf image data and the location information of the data acquisition device itself to the processor.

[0014] After receiving the shelf image data and the location information of the acquisition device itself, the processor identifies whether the target shopping cart exists in the shelf image. When the target shopping cart is identified in the shelf image, the processor sends the location information of the acquisition device corresponding to the shelf image containing the target shopping cart as the final location information of the target shopping cart to the target shopping cart.

[0015] The target shopping cart uses the final location information as its location information within the supermarket to determine the set of products within a preset range of the target shopping cart's location; based on the set of products within the preset range, it performs product identification within a limited range for the target products.

[0016] This application provides a product identification method based on location information. This method is used with a shopping cart to assist in identifying target products within a limited range based on the shopping cart's location information, thereby improving the accuracy and speed of product identification. The method includes:

[0017] Upon entering the aisle via the end of the shelf, the shopping cart connects to the nearest sensor and acquires the location information of the connected sensor as the initial location information of the target shopping cart. This initial location information is then sent to a data acquisition and control device. The initial location information includes the shelf aisle identifier determined by the unique identifier of the connected sensor. The data acquisition and control device, upon determining that the target shopping cart has entered the shelf aisle, sends a wake-up message to the data acquisition device within that aisle based on the shelf aisle identifier. The data acquisition device is deployed on at least one shelf in the supermarket, and each device is pre-configured with location information. Upon receiving the wake-up message, the data acquisition device begins to periodically or continuously capture shelf images of itself facing the shelf and sends the shelf image data and its own location information to a processor. Upon receiving the shelf image data and the location information of the data acquisition device, the processor identifies whether the target shopping cart is present in the shelf image. If the target shopping cart is identified, the processor sends the location information of the data acquisition device corresponding to the shelf image containing the target shopping cart as the final location information of the target shopping cart to the target shopping cart.

[0018] The final location information is used as the location information of the target shopping cart in the supermarket to determine the set of goods within a preset range of the target shopping cart's location; based on the set of goods within the preset range, the target goods are identified within a limited range.

[0019] This application provides a product recognition method based on location information. The method is used by a processor to assist the shopping cart in recognizing target products within a limited range based on the shopping cart's location information, thereby improving the accuracy and speed of product recognition. The method includes:

[0020] The system receives shelf image data and the location information of the acquisition device itself. The shelf image data and the location information of the acquisition device are sent by the acquisition device, which, upon receiving a wake-up message, begins periodically or continuously capturing shelf image data facing the shelf. The wake-up message is sent by the acquisition control device, which, upon determining that the target shopping cart is entering the shelf aisle, sends a wake-up message to the acquisition device within that aisle based on the shelf aisle identifier. The acquisition devices are deployed on at least one shelf in the supermarket, and each acquisition device is pre-configured with location information. The shelf aisle identifier is included in the preliminary positioning information sent by the target shopping cart. When the target shopping cart enters the aisle through the end of the shelf aisle, it connects to the nearest sensor, obtains the location information of the connected sensor as the target shopping cart's preliminary positioning information, and sends this preliminary positioning information to the acquisition control device. The preliminary positioning information includes the shelf aisle identifier determined based on the unique identifier of the connected sensor.

[0021] After receiving the shelf image data and the location information of the data acquisition device itself, it identifies whether the target shopping cart exists in the shelf image;

[0022] When a target shopping cart is identified in a shelf image, the location information of the acquisition device corresponding to the shelf image containing the target shopping cart is sent to the target shopping cart as the final location information of the target shopping cart. The target shopping cart uses the final location information as its location information in the supermarket to determine the set of goods within a preset range of the target shopping cart's location. Based on the set of goods within the preset range, the target goods are identified within a limited range.

[0023] This application provides a product recognition system based on location information, used to assist the shopping cart in recognizing target products within a limited range based on the shopping cart's location information, thereby improving the accuracy and speed of product recognition. The system includes: a sensor, a target shopping cart, a data acquisition and control device, a data acquisition device, and a processor; wherein:

[0024] At least one sensor is installed in the shelf aisle, and each sensor has a unique identifier.

[0025] The target shopping cart is used to connect to the nearest sensor when it enters the aisle through the end of the shelf aisle, and obtain the location information of the connected sensor as the initial positioning information of the target shopping cart. The initial positioning information is then sent to the data acquisition and control device. The initial positioning information includes the shelf aisle identifier determined by the unique identifier of the connected sensor.

[0026] A data collection and control device is used to send a wake-up message to the data collection device in the shelf aisle based on the shelf aisle identifier when the target shopping cart is determined to be entering the shelf aisle; the data collection device is deployed on at least one shelf in the supermarket, and each data collection device is pre-configured with location information;

[0027] The data acquisition device is used to start taking pictures of the shelf facing it at regular intervals or continuously after receiving the wake-up information, and send the shelf picture data and the location information of the data acquisition device itself to the processor.

[0028] The processor is used to identify whether a target shopping cart exists in the shelf image after receiving shelf image data and the location information of the acquisition device itself; when the target shopping cart is identified in the shelf image, the processor sends the location information of the acquisition device corresponding to the shelf image containing the target shopping cart as the final location information of the target shopping cart to the target shopping cart.

[0029] The target shopping cart uses the final location information as its location information within the supermarket to determine the set of products within a preset range of the target shopping cart's location; based on the set of products within the preset range, it performs product identification within a limited range for the target products.

[0030] This application provides a shopping cart for product recognition based on location information. This cart uses the location information of the shopping cart to assist in the limited-range recognition of target products, thereby improving the accuracy and speed of product recognition. The shopping cart includes:

[0031] A preliminary positioning unit is used to connect with the nearest sensor when entering the aisle via the end of the shelf aisle, acquire the location information of the connected sensor as the preliminary positioning information of the target shopping cart, and send the preliminary positioning information to a data acquisition and control device. The preliminary positioning information includes: the shelf aisle identifier determined by the unique identifier of the connected sensor. The data acquisition and control device is used to send a wake-up message to the data acquisition device in the shelf aisle based on the shelf aisle identifier when it determines that the target shopping cart has entered the shelf aisle. The data acquisition device is deployed on at least one shelf in the supermarket, and each data acquisition device is pre-configured with location information. After receiving the wake-up message, the data acquisition device is used to start taking pictures of the shelf facing it at regular intervals or continuously, and send the shelf picture data and the location information of the data acquisition device itself to the processor. After receiving the shelf picture data and the location information of the data acquisition device itself, the processor is used to identify whether the target shopping cart is in the shelf picture. When the target shopping cart is identified in the shelf picture, the processor sends the location information of the data acquisition device corresponding to the shelf picture containing the target shopping cart as the final positioning information of the target shopping cart to the target shopping cart.

[0032] The restricted range identification unit is used to use the final location information as the location information of the target shopping cart in the supermarket, determine the set of goods information within a preset range of the target shopping cart's location, and identify the target goods within a restricted range based on the set of goods information within the preset range.

[0033] This application provides a product recognition processor based on location information, used to assist the shopping cart in recognizing target products within a limited range based on the shopping cart's location information, thereby improving the accuracy and speed of product recognition in the shopping cart. The processor includes:

[0034] A receiving unit is used to receive shelf image data and the location information of the acquisition device itself. The shelf image data and the location information of the acquisition device are sent by the acquisition device, which, upon receiving a wake-up message, begins to periodically or continuously capture shelf image data facing the shelf. The wake-up message is sent by a acquisition control device, which, upon determining that a target shopping cart is entering a shelf aisle, sends a wake-up message to the acquisition device within that aisle based on the shelf aisle identifier. The acquisition devices are deployed on at least one shelf in the supermarket, and each acquisition device is pre-configured with location information. The shelf aisle identifier is included in the preliminary positioning information sent by the target shopping cart. When the target shopping cart enters the aisle through the end of the shelf aisle, it connects to the nearest sensor, obtains the location information of the connected sensor as the preliminary positioning information of the target shopping cart, and sends the preliminary positioning information to the acquisition control device. The preliminary positioning information includes: the shelf aisle identifier determined based on the unique identifier of the connected sensor.

[0035] The shopping cart recognition unit is used to identify whether a target shopping cart exists in the shelf image after receiving shelf image data and the location information of the acquisition device itself.

[0036] The final positioning unit is used to send the location information of the acquisition device corresponding to the shelf image containing the target shopping cart as the final positioning information of the target shopping cart when the target shopping cart is identified in the shelf image. The target shopping cart uses the final positioning information as its positioning information in the supermarket to determine the set of goods information within a preset range of the target shopping cart's positioning location. Based on the set of goods information within the preset range, the target goods are identified within a limited range.

[0037] This application also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the above-described product identification method based on location information.

[0038] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described product identification method based on location information.

[0039] This application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the above-described product identification method based on location information.

[0040] In this embodiment, the product identification scheme based on location information, compared with existing technologies where the accuracy and efficiency of shopping cart product identification in supermarkets are both low, achieves the following improvements: When the target shopping cart enters the aisle through the end of the shelf aisle, it connects to the nearest sensor, acquires the location information of the connected sensor as the initial location information of the target shopping cart, and sends this initial location information to the data acquisition and control device; the initial location information includes: the shelf aisle identifier determined by the unique identifier of the connected sensor; when the data acquisition and control device determines that the target shopping cart has entered the shelf aisle, it sends a wake-up message to the data acquisition device in that shelf aisle based on the shelf aisle identifier; the data acquisition devices are deployed on at least one shelf in the supermarket, and each data acquisition device is pre-configured with location information; after receiving the wake-up message, the data acquisition device begins to periodically or continuously capture images of the target shopping cart. The data acquisition device collects image data of the shelves facing the shelves and sends the shelf image data and the location information of the acquisition device itself to the processor. After receiving the shelf image data and the location information of the acquisition device, the processor identifies whether a target shopping cart exists in the shelf image. If a target shopping cart is identified in the shelf image, the processor sends the location information of the acquisition device corresponding to the shelf image containing the target shopping cart as the final location information of the target shopping cart. The target shopping cart uses the final location information as its location information in the supermarket and determines the set of goods within a preset range of the target shopping cart's location. Based on the set of goods within the preset range, the processor performs limited-range product recognition of the target goods. This allows the shopping cart to perform limited-range product recognition of the target goods based on its location information, improving the accuracy and speed of product recognition in the shopping cart. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings:

[0042] Figure 1 is a flowchart illustrating the method for location-based assisted shopping cart item recognition applied to the system in this embodiment of the application;

[0043] Figure 2 is a schematic diagram of the process of tracking the shopping cart using the shopping cart icon in an embodiment of this application;

[0044] Figure 3 is a schematic diagram of the process of tracking a shopping cart according to the new multi-target tracking algorithm in an embodiment of this application;

[0045] Figure 4 is a schematic diagram of the process of dynamically expanding the target detection box to track the shopping cart in an embodiment of this application;

[0046] Figure 5 is a flowchart illustrating the method for assisting shopping cart product identification based on location information in an embodiment of this application.

[0047] Figure 6 is a flowchart illustrating the method for assisting shopping cart item recognition based on location information applied to a processor in an embodiment of this application;

[0048] Figure 7 is a schematic diagram of the system for assisted shopping cart product identification based on location information in an embodiment of this application;

[0049] Figure 8 is a schematic diagram of the shopping cart structure based on location information-assisted shopping cart product identification in an embodiment of this application;

[0050] Figure 9 is a schematic diagram of the processor for assisting shopping cart product recognition based on location information in an embodiment of this application. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the embodiments of this application will be further described in detail below with reference to the accompanying drawings. Here, the illustrative embodiments and descriptions of this application are used to explain this application, but are not intended to limit this application.

[0052] The acquisition, storage, use, and processing of data in this application comply with relevant laws and regulations.

[0053] The existing product identification methods in supermarket environments and their technical problems are as follows:

[0054] In existing supermarket shopping processes, some solutions for product identification and comparison directly identify or search within the entire range of SKU product categories. The drawbacks of this approach are: selecting the category with the highest calculated probability from a large number of categories is very difficult; and the presence of many interfering factors, such as the similarity between the packaging of a beverage and a dairy product, increases the probability of misidentification. This not only affects the accuracy of product identification but also reduces its speed.

[0055] In existing supermarket shopping processes, another approach to product identification and comparison involves training a classification model to first filter and confirm the category of the object to be identified, such as initially identifying it as "beverages." Then, within the filtered category, such as "beverages," a second level of fine-grained identification is performed. This approach has several problems: First, the classification model has a certain probability of error in classifying the category. For example, if "drinking water" is identified as "liquor," the second-level fine-grained identification will definitely be incorrect. Second, training a category classification model is difficult because some products cannot be correctly categorized from their packaging, making it difficult to establish standardized screening criteria. Third, collecting and organizing the model's training data incurs significant manual costs.

[0056] To address the aforementioned technical problems, this application proposes a product recognition scheme based on location information. This scheme is a location-assisted product recognition solution for shopping carts, aiming to optimize the product recognition process in the shopping cart. By using location-determined range information, it reduces the range or number of categories or information to be compared during product classification or retrieval, thereby improving the accuracy and speed of product recognition. The location-based product recognition scheme will be described in detail below.

[0057] Figure 1 is a flowchart illustrating the method for location-based assisted shopping cart item recognition applied to the system in this application embodiment. As shown in Figure 1, the method includes the following steps:

[0058] Step 101: When the target shopping cart enters the aisle through the end of the shelf aisle, it connects to the nearest sensor, obtains the location information of the connected sensor as the initial location information of the target shopping cart, and sends the initial location information to the data acquisition and control device; the initial location information includes: the shelf aisle identifier determined by the unique identifier of the connected sensor.

[0059] Step 102: When the target shopping cart enters the shelf aisle, the data collection and control device sends a wake-up message to the data collection device in that shelf aisle according to the shelf aisle identifier; the data collection device is deployed on at least one shelf in the supermarket, and each data collection device is pre-configured with location information;

[0060] Step 103: After receiving the wake-up message, the data acquisition device begins to periodically or continuously capture images of the shelf it faces, and sends the shelf image data and the location information of the data acquisition device itself to the processor.

[0061] Step 104: After receiving the shelf image data and the location information of the acquisition device itself, the processor identifies whether there is a target shopping cart in the shelf image; when it identifies that there is a target shopping cart in the shelf image, it sends the location information of the acquisition device corresponding to the shelf image containing the target shopping cart as the final location information of the target shopping cart to the target shopping cart.

[0062] Step 105: The target shopping cart uses the final location information as the target shopping cart's location information in the supermarket, and determines the set of goods within the preset range of the target shopping cart's location; based on the set of goods within the preset range, the target goods are identified within a limited range.

[0063] In this embodiment of the application, the method for assisting shopping cart product recognition based on positioning information operates as follows: When a target shopping cart enters the aisle through the end of the shelf aisle, it connects to the nearest sensor, acquires the location information of the connected sensor as the initial positioning information of the target shopping cart, and sends the initial positioning information to the acquisition control device; the initial positioning information includes: the shelf aisle identifier determined by the unique identifier of the connected sensor; when the acquisition control device determines that the target shopping cart has entered the shelf aisle, it sends a wake-up message to the acquisition device in the shelf aisle according to the shelf aisle identifier; the acquisition device is deployed on at least one shelf in the supermarket, and each acquisition device is pre-configured with location information; after receiving the wake-up message, the acquisition device begins to periodically or continuously capture shelf image data of the shelf facing the acquisition device, and sends the shelf image data and the location information of the acquisition device itself to the processor; the processor receives... After obtaining the shelf image data and the location information of the acquisition device itself, the system identifies whether a target shopping cart exists in the shelf image. If a target shopping cart is identified, the system sends the location information of the acquisition device corresponding to the shelf image containing the target shopping cart as the final location information of the target shopping cart. The target shopping cart uses this final location information as its location within the supermarket, determining the set of goods within a preset range of its location. Based on this set of goods, the system performs limited-range product identification of the target goods. Compared to existing technologies with low accuracy and efficiency in supermarket product identification, the method for location-based assisted shopping cart product identification provided in this application can improve the accuracy and speed of product identification by using the shopping cart's location information to assist in limited-range product identification. A detailed description follows.

[0064] The main process of product identification in a location-based shopping cart proposed in this application includes: configuring hardware and system, locating the shopping cart, determining the product set or category, and limiting the scope of product identification. The specific process or steps are as follows:

[0065] I. Deployment

[0066] 1) Deploy sensors

[0067] In specific implementations, the sensor in this application embodiment can be an anchor point, which has the advantages of low cost and convenient positioning. Of course, the sensor can also be an infrared or ultrasonic sensor, etc.

[0068] Positioning anchor points are deployed at both ends of each shelf aisle or inside the shelf in the supermarket. The anchor points at different ends of the same aisle are configured as "paired" (that is, if a first anchor point is configured at one end of an aisle, a second anchor point is also configured at the other end of the aisle corresponding to that end. The first anchor point and the second anchor point match each other and are each other's paired anchor points).

[0069] 2) Deploy visual sensors (the acquisition devices mentioned below)

[0070] Visual sensors are deployed in the scene, and each visual sensor can cover at least one shelf that it is facing (specifically, each visual sensor can capture images of at least one small shelf that it is facing, for example, it can capture a panoramic image of a small shelf at once; or it can capture an image of a part of the small shelf it is facing at once, and then stitch the images of each part of the small shelf it is facing to obtain the corresponding panoramic image; or it can only capture an image of a part of the small shelf it is facing; furthermore, it can also stitch the images of multiple small shelves to obtain an image of a large shelf), and position information is configured for each visual sensor.

[0071] 3) Deploy the visual algorithm inference service (the processor mentioned below)

[0072] The service, which can be deployed at the edge or in the cloud, includes an algorithm inference module and a communication module.

[0073] In practical implementation, the data acquisition control device (also known as the data acquisition controller) in this application embodiment can be a communication base station or an edge service, generally deployed locally in the supermarket / store. Deploying locally in the supermarket / store allows the data acquisition control device to quickly receive connections and send information, avoiding excessively high information transmission latency and slow control and wake-up speed of the data acquisition device (camera). The processor can be a cloud server or an edge service, allowing for flexible deployment.

[0074] 2. Locate the shopping cart, i.e., steps 101 to 104 above.

[0075] The specific method for locating the shopping cart's spatial position within the supermarket environment is as follows:

[0076] 1) First, perform rough anchor point positioning:

[0077] a) When the shopping cart enters the aisle from the end of the shelf, it will connect with the nearest anchor point;

[0078] b) The shopping cart obtains information about the anchor points it connects to as positioning information (Note: If the shopping cart connects to an anchor point of a certain channel for the first time (e.g., within a preset time period), it is determined to be entering; if it connects to the same anchor point or the "paired" anchor point of the same channel again after a preset time period threshold, such as n seconds, it is determined to be leaving); that is, in one embodiment, when the acquisition control device determines that the target shopping cart has entered the shelf channel when it first connects to the sensor (anchor point) of the preset shelf channel within a preset time period; the method for assisting shopping cart product identification based on positioning information may also include: if the shopping cart first connects to the anchor point at one end of the preset shelf channel, the acquisition control device determines that the shopping cart has entered the channel; if it connects to the same anchor point or the corresponding paired anchor point of the same shelf channel again after a preset time period threshold, the shopping cart is determined to be leaving the channel;

[0079] c) The shopping cart sends the connected anchor point information to the AI ​​vision system (the information can be understood as: the shopping cart with ID number (identifier) ​​xxx has been connected to the anchor point with ID number (identifier) ​​yyy and has entered / left the zzz shelf aisle).

[0080] 2) AI vision system for precise positioning:

[0081] a) The AI ​​vision system (AI vision system acquisition and control device) receives the anchor point connection information sent by the shopping cart.

[0082] b) If the system receives the message "Shopping cart with ID number xxx has entered the zzz aisle," the AI ​​vision system's acquisition and control device will wake up and notify all vision sensors within the zzz aisle to start working. If the system receives the message "Shopping cart with ID number xxx has left the zzz aisle," the AI ​​vision system, after verification, will notify all vision sensors within the zzz aisle to stop working if it determines that there is no shopping cart in any of the vision sensors (acquisition devices) within the aisle. In one embodiment, the acquisition and control device sends a stop-work notification to all acquisition devices within the aisle when it determines that the target shopping cart has left the shelf aisle.

[0083] c) Visual sensor (acquisition device): After receiving the wake-up information from the AI ​​system (the acquisition and control device of the AI ​​vision system), it begins to take pictures at regular intervals or continuously, and sends the image data (shelf image data) and the camera's own position information (the position information of the acquisition device itself) to the algorithm processing module (processor).

[0084] d) Algorithm Processing Module (Processor): Upon receiving image information, the module uses a detection algorithm to check if a shopping cart exists in the shelf image. If a shopping cart is detected in the shelf image, the module uses the location information of the visual sensor corresponding to the shelf image from which the shopping cart was detected as the location point information of the shopping cart (the final location information of the target shopping cart).

[0085] To avoid situations where multiple shopping carts appear simultaneously in the same aisle, making it difficult to determine which cart's location information accurately corresponds to, two methods for matching location information to shopping carts are provided here:

[0086] i. Identify the ID identifier on each shopping cart. When a shopping cart is detected in the image, continue to detect and identify the numbered ID identifier mounted on the shopping cart body; (if this method is used, the prerequisite is that different numbered ID identifiers are already affixed to the shopping cart). This scheme is the first method of matching location information with shopping carts.

[0087] ii. Track shopping carts entering the aisle. When the AI ​​system receives the message "Shopping cart with ID xxx enters aisle zzz," it tracks the shopping cart detected in the sequence of images (multiple shelf images with different time markers), using the shopping cart's ID as the tracking target. This is scheme two, which matches location information with the shopping cart.

[0088] Specifically, the shopping cart target tracking algorithm used is innovatively designed as an auto-deepsort multi-target tracking algorithm, and the specific process is as follows:

[0089] 1. Use the object detection bounding box in the shopping cart and the image within the bounding box as input for object tracking.

[0090] 2. Establish a new starting trajectory point and ID for each shopping cart that newly enters the shelf aisle, and continuously update its trajectory based on the tracking status.

[0091] 3. Specifically, when it is confirmed that there is only one shopping cart in the aisle, only the detection bounding box of the shopping cart target is used as the basis for trajectory tracking, and the image inside the box is not used as the basis for trajectory tracking. That is, only the Intersection over Union (IoU) value between the predicted target detection bounding boxes in different frames and the target detection bounding box in the current frame is judged. If the IoU value is greater than the threshold s, the targets are connected as the trajectory of the same target.

[0092] 4. Considering that the shopping cart may move at a relatively fast speed while the visual sensor acquires images at a low frequency, and that the coverage of the visual sensor between different shelves may not be completely seamless, in order to avoid the trajectory being interrupted during target tracking, the target tracking algorithm provided in this application dynamically expands the width and height of the input target detection box, that is, expands it from the original (w,h) to (a*w,b*h), with a and b configured to values ​​greater than 1. Furthermore, the lower the frequency of the camera (acquisition device) taking pictures and the closer the target shopping cart is to the edge of the image, the larger the values ​​of a and b are.

[0093] 5. Specifically, when it is confirmed that there is more than one shopping cart in the aisle, the image within each target detection box is automatically processed additionally, i.e., the corresponding image feature vector is extracted. During trajectory matching, the Intersection over Union (IoU) value between the predicted target detection boxes in different frames and the current target detection box, as well as the feature similarity score between frames, are calculated. The weighted sum of these values ​​is used as the final trajectory connection determination. The specific trajectory connection determination formula can be:

[0094] Match_score=alph*iou+beta*featscore.}

[0095] In the above formula, alph means: the weight pre-configured for the intersection-union (IoU) degree; beta means: the weight pre-configured for the feature similarity degree. The weight setting is used to adjust whether more emphasis is placed on IoU or feature similarity.

[0096] As can be seen from the above, in steps 101-104, locating the shopping cart's position within the supermarket and obtaining the shopping cart's location information can include:

[0097] When the target shopping cart (in this embodiment, the target shopping cart can refer to the shopping cart to be located and whose products are to be identified) enters the aisle through the shelf aisle end, the target shopping cart connects to the nearest anchor point; the location information of the connected anchor point is obtained as the initial location information of the target shopping cart; the initial location information is sent to the acquisition and control device; the initial location information includes: determining the aisle identifier based on the identifier of the connected anchor point; the anchor point is deployed in the shelf aisle; at this time, the initial location information is coarse location information obtained based on the anchor point, and this coarse location information can be used as the location information of the target shopping cart in the supermarket. In order to further improve the positioning accuracy, and thus further improve the accuracy of product identification in the shopping cart, the initial location information can be sent to an auxiliary positioning system (which can be the AI ​​vision system mentioned below). The auxiliary positioning system feeds back the final precise positioning information to the shopping cart based on the initial location information. The shopping cart performs product identification based on the final precise positioning information. That is, the shopping cart interacts with the auxiliary positioning system to obtain the final precise positioning information, which can further improve the accuracy of product identification in the shopping cart. The assisted positioning system may include: a data acquisition and control device (which may be the control part of an AI vision system), a data acquisition device (which may be a vision sensor), and a processor (which may be the algorithm processing module mentioned above, i.e., the algorithm inference service mentioned above). In this embodiment:

[0098] When the acquisition and control device determines that the target shopping cart has entered the shelf aisle, it sends a wake-up message to all visual sensors in that aisle according to the shelf aisle markings. The visual sensors are deployed on each shelf in the shopping scene, and each visual sensor can cover the shelf it is facing. Each visual sensor is pre-configured with location information.

[0099] After receiving the wake-up message, all the acquisition devices in the channel begin to take pictures of the shelves at regular intervals or continuously, and send the shelf picture data and the location information of the acquisition devices themselves to the processor.

[0100] After receiving the shelf image data and the location information of the acquisition device itself, the processor identifies whether the target shopping cart is in the shelf image. When the target shopping cart is identified as being in the shelf image, the processor sends the location information of the acquisition device corresponding to the shelf image containing the target shopping cart as the final location information of the target shopping cart to the target shopping cart, so that the target shopping cart can determine the product category information or product set information within the preset range of the final location information.

[0101] In specific implementation, this application proposes a scheme of using anchor point coarse positioning and AI vision system fine positioning to accurately locate the shopping cart, thereby improving the accuracy of subsequent shopping cart product recognition.

[0102] In specific implementation, in this embodiment of the application, the location information of the acquisition device (AI vision sensor) corresponding to the shelf image data of the target shopping cart can be the location information of the acquisition device that is predetermined and stored when the acquisition device is deployed (for example, the acquisition device with a preset number is located on the edge of the shelf in the 4th row and 5th column of shelf number 1. Of course, for greater accuracy, it can also include the coordinate position information of the acquisition device on the shelf, or the spatial position information of the acquisition device in the entire supermarket).

[0103] As described above, in one embodiment, each target shopping cart is pre-configured with a unique shopping cart identifier; as shown in Figure 2, when a target shopping cart is identified as being present in a shelf image, the location information of the acquisition device corresponding to the shelf image containing the target shopping cart is sent to the target shopping cart as the final location information of the target shopping cart, including:

[0104] Step 201: When the shelf image contains the target shopping cart, identify the unique shopping cart identifier on the target shopping cart;

[0105] Step 202: Send the location information of the acquisition device corresponding to the shelf image containing the target shopping cart as the final location information of the target shopping cart to the target shopping cart corresponding to the unique shopping cart identifier.

[0106] In practice, the first approach, which matches location information with shopping carts, uses a unique shopping cart identifier to locate the cart. This avoids situations where multiple shopping carts appear in the same aisle simultaneously, making it impossible to determine which cart to send the location information to, thereby improving the accuracy of product recognition.

[0107] In specific implementation, in this embodiment of the application, before locating the shopping cart by identifying a unique shopping cart identifier, the processor performs target tracking or trajectory tracking of the shopping cart at the moment it enters the shelf aisle. This avoids the situation where the shopping cart is not tracked in time due to the identifier being temporarily obscured, thus achieving accurate and timely tracking of each shopping cart and improving the accuracy of product identification.

[0108] As described above, in one embodiment, when a target shopping cart is identified as being contained in a shelf image, the location information of the acquisition device corresponding to the shelf image containing the target shopping cart is sent to the target shopping cart as the final location information of the target shopping cart. This includes: when a target shopping cart is identified as being contained in a shelf image, target tracking is performed on each shopping cart entering the aisle according to the following target tracking algorithm to obtain the final location information of each target shopping cart:

[0109] The detection bounding box of the target shopping cart is used as the basis for trajectory tracking to obtain the trajectory identifier corresponding to each target shopping cart;

[0110] The location information of the data acquisition device corresponding to the shelf image containing the target shopping cart is sent as the final location information of the target shopping cart to the target shopping cart corresponding to the trajectory marker.

[0111] In practical implementation, in Scheme 2, which matches location information with shopping carts, when there is only one shopping cart in the shelf aisle, the detection bounding box of the target shopping cart can be used as the basis for trajectory tracking. This accurately sends the final location information to the target shopping cart corresponding to the trajectory identifier, improving the accuracy of shopping cart positioning and thus improving the accuracy of product recognition. Specifically, the Intersection over Union (IoU) value between the predicted target detection bounding boxes in different frame shelf images and the target detection bounding box in the current frame is determined. If the IoU value is greater than a threshold s, the target detection bounding boxes are connected to form the trajectory of the same target shopping cart.

[0112] As can be seen from the above, in one embodiment, as shown in Figure 3, when the shelf image is identified to contain the target shopping cart, the location information of the acquisition device corresponding to the shelf image containing the target shopping cart is sent to the target shopping cart as the final location information of the target shopping cart. This includes: when the shelf image is identified to contain the target shopping cart, each shopping cart entering the aisle is tracked according to the following multi-target tracking algorithm to obtain the final location information of each target shopping cart:

[0113] Step 301: When it is confirmed that there are more than two shopping carts in the shelf aisle, extract the image feature vector corresponding to the shopping cart image within the detection box of the target shopping cart;

[0114] Step 302: Based on the detection box of the target shopping cart and its corresponding image feature vector, obtain the trajectory identifier corresponding to each target shopping cart; this step is the process of matching multiple shopping carts and multiple trajectories. To correctly match the trajectories, the implementation method of the trajectory connection judgment formula mentioned above can be used.

[0115] Step 303: Send the location information of the acquisition device corresponding to the shelf image containing the target shopping cart as the final location information of the target shopping cart to the target shopping cart corresponding to the trajectory identifier.

[0116] In specific implementation, in Scheme 2, which matches the location information with the shopping cart, the auto-deepsort multi-target tracking algorithm tracks the shopping cart based on the detection box of the target shopping cart and its corresponding image feature vector. This avoids the situation where multiple shopping carts appear in the same channel at the same time, making it impossible to determine which shopping cart to send the location information to. This improves the accuracy of shopping cart positioning and thus improves the accuracy of product recognition.

[0117] As can be seen from the above, in one embodiment, as shown in Figure 4, when the shelf image is identified to contain the target shopping cart, the location information of the acquisition device corresponding to the shelf image containing the target shopping cart is sent to the target shopping cart as the final location information of the target shopping cart. This includes: when the shelf image is identified to contain the target shopping cart, target tracking is performed on each shopping cart entering the aisle according to the following method to obtain the final location information of each target shopping cart:

[0118] Step 401: Based on the shooting frequency of the acquisition device, dynamically expand the width and height of the detection box of the target shopping cart; specifically, it can be expanded from the original (w,h) to (a*w,b*h), with a and b configured as values ​​greater than 1, where a is the value for expanding the width of the detection box and b is the value for expanding the height of the detection box. The lower the shooting frequency of the camera (acquisition device) and the closer the target shopping cart is to the edge of the image, the larger the values ​​of a and b are.

[0119] Step 402: Use the dynamically enlarged detection box as the basis for trajectory tracking to obtain the trajectory identifier corresponding to each target shopping cart;

[0120] Step 403: Send the location information of the acquisition device corresponding to the shelf image containing the target shopping cart as the final location information of the target shopping cart to the target shopping cart corresponding to the trajectory marker.

[0121] In specific implementation, this application proposes a new auto-deepsort target tracking algorithm to ensure better shopping cart target tracking performance under different shelf vision sensor coverage ranges and different shooting frequencies.

[0122] 3. Determine the product information within the vicinity of the shopping cart.

[0123] Using the shopping cart location information from the previous step, confirm the product information near that location, including product categories or product collections. The specific method is as follows:

[0124] 1) Obtain product information for the shelves covered by each camera (visual sensor) location. This product information comes from the supermarket's shelf display system or from the algorithm inference module's daily periodic identification and updates.

[0125] Among them, the supermarket's shelf display system records the product categories or detailed product display information set for each aisle or shelf;

[0126] The algorithm inference module acquires images uploaded daily by the visual sensors and detects and identifies all products in the images. The detection and identification results are then summarized to categorize product information, or to extract specific information about each product on each shelf (such as SKU code, barcode, category information, etc.), and recorded in the product set 's'. Finally, this information is stored in relation to each camera.

[0127] In practice, the above-mentioned preset range can be adjusted according to actual needs by changing the angle of the vision sensor to cover the shelf range, the width between two shelves, etc., making it flexible and convenient.

[0128] 2) When the visual positioning information of the shopping cart is determined, the product information (product set s) of the shelf covered by the camera at this moment is sent to the corresponding shopping cart.

[0129] In specific implementation, this application proposes to accurately determine the range of the product set based on the location information, thereby improving the accuracy and speed of shopping cart product recognition.

[0130] Fourth, when the shopping cart needs to identify products, the product identification is limited to a specific range of target products. This "fourth" together with the "third" mentioned above constitutes step 105.

[0131] When it is necessary to identify goods placed inside or removed from a vehicle, different range restriction methods can be used for goods identification.

[0132] Product identification can utilize two methods: classification models and retrieval calculations. Different product identification processes can be employed when product set information is obtained. Specific usage methods and instructions are as follows:

[0133] a) If a classification and identification method is used for product identification

[0134] The system identifies the category to which each product in the product set belongs. If a product is classified into a certain category, the recognition module in the shopping cart calls the classification model of that category to identify the target product and outputs the classification result.

[0135] {For example: If the product set 's' contains "6 bottles of Coke - beverage", "8 bottles of Sprite - beverage", "2 bottles of Fanta - beverage", etc., then the shopping cart is located in the beverage section. In this case, the shopping cart will automatically select the "beverage" product recognition model from several pre-set product recognition models for different categories and identify the target product.}

[0136] {The product recognition and classification model extracted above was trained using deep learning methods on an artificial neural network model. Specifically, according to the designed scheme, several category-level classification models (sub-models) need to be trained; that is, each category corresponds to one sub-model. For example, the "beverage" product recognition model was trained entirely using beverage-related image data on the artificial neural network model.}

[0137] As can be seen from the above, in one embodiment, the limited-range product identification of the target product based on the product set information within a preset range includes: identifying products placed inside or taken out of the vehicle according to the following classification and identification method:

[0138] Retrieve information on a set of goods within a preset range;

[0139] Determine the category to which each product in the product collection belongs;

[0140] If each product belongs to the same category, the pre-trained classification sub-model corresponding to that category is called to identify the target product, and the identification result is obtained after limiting the target product range.

[0141] In practice, a classification and recognition method is used to identify items placed in or taken out of the shopping cart within a limited range. That is, when the shopping cart needs to identify items, the classification and recognition method is used to identify items within a limited range, which can improve the accuracy of shopping cart item identification.

[0142] b) If a retrieval and identification method is used for product identification

[0143] 1) Establish a small product feature retrieval library: First, the product recognition module in the shopping cart selects features corresponding to products in the product set from the full feature retrieval library to form a small product feature retrieval library A (product feature retrieval sub-library); at the same time, the product recognition module in the shopping cart counts the category to which each product in the product set belongs. If it counts a certain category, it selects all features corresponding to that category from the full feature retrieval library to form a category-level small feature library B (category-level sub-feature library).

[0144] 2) Extracting features of the target product: The shopping cart recognition module extracts the image corresponding to the target product as a feature vector;

[0145] 3) Retrieval and Recognition: First, the shopping cart's recognition module performs a retrieval calculation between the target product's feature vector and a small product feature retrieval database, that is, it calculates the cosine distance between the feature vectors, and finally selects the result with the smallest distance as the product recognition result.

[0146] To avoid the failure to identify or the error caused by the target product not being in the located product set s (e.g., the product has just been temporarily put on the shelf or the target product belongs to the adjacent shelf), the unidentifiable or low-confidence targets are further searched and calculated again using a small category-level feature library (category-level sub-feature library). When the confidence of the identification exceeds the preset threshold, the result is taken as the final identification result.

[0147] {A concrete example: If product set s explicitly contains only "Coca-Cola 2L bottle", "Sprite 300ml can", "Fanta 300ml can", "Pulse 500ml bottle", ... "Coca-Cola 200ml can", then feature values ​​corresponding to these products are selected only from the full feature library to form a feature sub-library A. Simultaneously, feature values ​​for all "beverages" are selected from the full feature library to form feature sub-library B. When retrieving and identifying a target product, calculations are first performed in feature sub-library A. If the highest confidence result reaches the threshold, the identification result is used as the output. If the highest confidence result does not reach the threshold, it means the target product may not be in feature sub-library A, i.e., it may not be in product set s, and may come from another shelf. Therefore, the target product is then searched and calculated a second time in feature sub-library B. If the result of this search calculation reaches the threshold, it is used as the final output.}

[0148] As can be seen from the above, in one embodiment, the limited-range product identification of the target product based on the product set information within a preset range includes: identifying products placed inside or removed from the vehicle according to the following retrieval and identification method:

[0149] Retrieve information on a set of goods within a preset range;

[0150] Features corresponding to products in the product set are selected from the full feature retrieval database to form a product feature retrieval sub-database;

[0151] Extract features from the target product image to obtain the target product feature vector;

[0152] The target product feature vector is retrieved from the product feature retrieval sub-database to obtain the identification result after limiting the target product range.

[0153] In practice, a retrieval and identification method is used to identify items placed in or taken out of the shopping cart within a limited range. That is, when the shopping cart needs to identify items, the retrieval and identification method is used to identify items within a limited range, which can improve the accuracy of shopping cart item identification.

[0154] As can be seen from the above, in one embodiment, if the identification result after limiting the target product range is that the product cannot be identified or the identification is incorrect, the method further includes:

[0155] Determine the category to which each product in the product collection belongs;

[0156] If each product belongs to the same category, all features corresponding to that category are selected from the full feature retrieval library to form a category-level sub-feature library;

[0157] The target product feature vector is retrieved from the category-level sub-feature library to obtain the final identification result after limiting the target product range.

[0158] In practice, when the shopping cart needs to identify products, in order to avoid the failure to identify or the identification error that occurs because the target product of the shopping cart is not in the located product set, a small feature library at the category level is used to perform another search and calculation. When the confidence level of the identification exceeds the preset threshold, the result is taken as the final identification result, which can improve the accuracy of shopping cart product identification.

[0159] To better understand the embodiments of this application, the names involved in the implementation of this application will be explained below.

[0160] Smart shopping cart: A shopping cart equipped with smart hardware that has network connectivity and communication capabilities.

[0161] Location Anchor: A small piece of hardware with Bluetooth connectivity, used to be deployed within the shelf aisle and can connect to shopping carts passing through its range.

[0162] AI vision system (the assisted positioning system mentioned above): includes a vision camera sensor and a server for taking pictures, and uses an algorithm module (processor) to detect the shopping cart, analyze and process the location of the shopping cart, analyze and process the products on the shelf covered by the AI ​​camera, and can communicate with the shopping cart.

[0163] In summary, the beneficial effects of the location-based shopping cart item recognition method provided in this application are:

[0164] 1) A scheme using anchor point coarse positioning and AI vision system fine positioning is proposed to locate the shopping cart and improve the accuracy of shopping cart positioning;

[0165] 2) A method for determining the range of product sets based on location information is proposed, which improves the accuracy of shopping cart product recognition;

[0166] 3) A method for product identification within a limited range is proposed, that is, when the shopping cart needs to identify products, product identification is performed within a limited range, which improves the accuracy of shopping cart product identification;

[0167] 4) A new auto-deepsort target tracking algorithm is proposed to ensure better shopping cart target tracking performance under different shelf vision sensor coverage ranges and shooting frequencies.

[0168] In summary, the method for shopping cart product recognition based on location information provided in this application reduces the range or number of categories to be compared during product classification or retrieval, thereby improving the accuracy and speed of shopping cart product recognition.

[0169] This application also provides a method for location-based product recognition in a shopping cart, as described in the following embodiments. Since the principle behind this method is similar to that of the location-based product recognition method for a system, its implementation can be found in the implementation of the location-based product recognition method for a system; repeated details will not be elaborated further.

[0170] Figure 5 is a flowchart illustrating the method for location-based assisted shopping cart product identification applied in an embodiment of this application. As shown in Figure 5, the method includes the following steps:

[0171] Step 501: Upon entering the aisle via the end of the shelf aisle, the cart connects to the nearest sensor to obtain its location information as the initial location information of the target shopping cart. This initial location information is then sent to a data acquisition and control device. The initial location information includes the shelf aisle identifier determined by the unique identifier of the connected sensor. The data acquisition and control device, upon determining that the target shopping cart has entered the shelf aisle, sends a wake-up message to the data acquisition device within that aisle based on the shelf aisle identifier. Each data acquisition device is deployed on at least one shelf in the supermarket, and each device is pre-configured with location information. Upon receiving the wake-up message, the data acquisition device begins periodically or continuously capturing shelf images of itself facing the shelf, and sends the shelf image data and its own location information to a processor. Upon receiving the shelf image data and the location information of the data acquisition device, the processor identifies whether the target shopping cart is present in the shelf image. When the target shopping cart is identified in the shelf image, the processor sends the location information of the data acquisition device corresponding to the shelf image containing the target shopping cart as the final location information of the target shopping cart to the target shopping cart.

[0172] Step 502: Use the final location information as the location information of the target shopping cart in the supermarket, determine the set of goods within the preset range of the target shopping cart's location; and perform limited-range product identification on the target goods based on the set of goods within the preset range.

[0173] In one embodiment, when the data acquisition and control device determines that the target shopping cart has entered the shelf aisle for the first time within a preset time period, it determines that the target shopping cart has entered the shelf aisle.

[0174] In one embodiment, when the data acquisition control device determines that the target shopping cart has left the shelf aisle, it sends a stop-work notification message to all data acquisition devices in that aisle.

[0175] In one embodiment, sensors are deployed at both ends of a shelf aisle, wherein sensors at different ends of the same aisle are configured as paired sensors.

[0176] In location-based shopping cart product recognition, if the shopping cart is connected to the sensor at one end of a preset shelf aisle for the first time, the data acquisition and control device determines that the shopping cart is entering the aisle. If the preset time period threshold is exceeded, the cart is connected to the same sensor or the corresponding paired sensor in the same shelf aisle again to determine that the shopping cart is leaving the aisle.

[0177] In one embodiment, each target shopping cart is pre-configured with a unique shopping cart identifier; when a target shopping cart is identified in a shelf image, the location information of the acquisition device corresponding to the shelf image containing the target shopping cart is sent to the target shopping cart as the final location information of the target shopping cart, including:

[0178] When a target shopping cart is identified in a shelf image, the unique shopping cart identifier on the target shopping cart is identified;

[0179] The location information of the data acquisition device corresponding to the shelf image of the target shopping cart is sent as the final location information of the target shopping cart to the target shopping cart corresponding to the unique shopping cart identifier.

[0180] In one embodiment, when a target shopping cart is identified in a shelf image, the location information of the acquisition device corresponding to the shelf image containing the target shopping cart is sent to the target shopping cart as the final location information of the target shopping cart. This includes: when a target shopping cart is identified in a shelf image, target tracking is performed on each shopping cart entering the aisle according to the following target tracking algorithm to obtain the final location information of each target shopping cart:

[0181] The detection bounding box of the target shopping cart is used as the basis for trajectory tracking to obtain the trajectory identifier corresponding to each target shopping cart;

[0182] The location information of the data acquisition device corresponding to the shelf image of the target shopping cart is sent as the final location information of the target shopping cart to the target shopping cart corresponding to the trajectory marker.

[0183] In one embodiment, when a target shopping cart is identified in a shelf image, the location information of the acquisition device corresponding to the shelf image containing the target shopping cart is sent to the target shopping cart as the final location information of the target shopping cart. This includes: when a target shopping cart is identified in a shelf image, target tracking is performed on each shopping cart entering the aisle according to the following multi-target tracking algorithm to obtain the final location information of each target shopping cart:

[0184] When it is confirmed that there are more than two shopping carts in the shelf aisle, extract the image feature vector corresponding to the shopping cart image within the detection box of the target shopping cart;

[0185] Based on the detection bounding box of the target shopping cart and its corresponding image feature vector, the trajectory identifier corresponding to each target shopping cart is obtained;

[0186] The location information of the data acquisition device corresponding to the shelf image containing the target shopping cart is sent as the final location information of the target shopping cart to the target shopping cart corresponding to the trajectory marker.

[0187] In one embodiment, when a target shopping cart is identified in a shelf image, the location information of the acquisition device corresponding to the shelf image containing the target shopping cart is sent to the target shopping cart as the final location information of the target shopping cart. This includes: when a target shopping cart is identified in a shelf image, target tracking is performed on each shopping cart entering the aisle according to the following method to obtain the final location information of each target shopping cart:

[0188] Based on the shooting frequency of the acquisition device, the width and height of the detection frame of the target shopping cart are dynamically expanded;

[0189] The dynamically expanded detection box is used as the basis for trajectory tracking to obtain the trajectory identifier corresponding to each target shopping cart.

[0190] The location information of the data acquisition device corresponding to the shelf image containing the target shopping cart is sent as the final location information of the target shopping cart to the target shopping cart corresponding to the trajectory marker.

[0191] In one embodiment, based on a pre-defined range of product set information, the target product is identified within a limited scope, including: identifying products placed inside or removed from the vehicle according to the following classification and identification method:

[0192] Retrieve information on a set of goods within a preset range;

[0193] Determine the category to which each product in the product collection belongs;

[0194] If each product belongs to the same category, the pre-trained classification sub-model corresponding to that category is called to identify the target product, and the identification result is obtained after limiting the target product range.

[0195] In one embodiment, based on a pre-defined range of product set information, the target product is identified within a limited scope, including: identifying products placed inside or removed from the vehicle using the following retrieval and identification method:

[0196] Retrieve information on a set of goods within a preset range;

[0197] Features corresponding to products in the product set are selected from the full feature retrieval database to form a product feature retrieval sub-database;

[0198] Extract features from the target product image to obtain the target product feature vector;

[0199] The target product feature vector is retrieved from the product feature retrieval sub-database to obtain the identification result after limiting the target product range.

[0200] In one embodiment, if the identification result after limiting the target product range is that the product cannot be identified or the identification is incorrect, the method further includes:

[0201] Determine the category to which each product in the product collection belongs;

[0202] If each product belongs to the same category, all features corresponding to that category are selected from the full feature retrieval library to form a category-level sub-feature library;

[0203] The target product feature vector is retrieved from the category-level sub-feature library to obtain the final identification result after limiting the target product range.

[0204] This application also provides a method for location-based shopping cart item recognition applied to a processor, as described in the following embodiments. Since the principle behind this method is similar to that of the location-based shopping cart item recognition method applied to a system, the implementation of this method can refer to the implementation of the location-based shopping cart item recognition method applied to a processor; repeated details will not be elaborated further.

[0205] Figure 6 is a flowchart illustrating the method for location-based assisted shopping cart item recognition applied to a processor in an embodiment of this application. As shown in Figure 6, the method includes the following steps:

[0206] Step 601: Receive shelf image data and the location information of the acquisition device itself; The shelf image data and the location information of the acquisition device itself are sent by the acquisition device. After receiving the wake-up message, the acquisition device is used to start taking shelf image data of its facing shelf at regular intervals or continuously; The wake-up message is sent by the acquisition control device. When the acquisition control device determines that the target shopping cart has entered the shelf aisle, it sends the wake-up message to the acquisition device in the shelf aisle according to the shelf aisle identifier. The acquisition device is deployed on at least one shelf in the supermarket, and each acquisition device is pre-configured with location information; The shelf aisle identifier is included in the preliminary positioning location information sent by the target shopping cart. When the target shopping cart enters the aisle through the end of the shelf aisle, it connects to the nearest sensor, obtains the location information of the connected sensor as the preliminary positioning location information of the target shopping cart, and sends the preliminary positioning location information to the acquisition control device. The preliminary positioning location information includes: the shelf aisle identifier of the shelf aisle entered according to the unique identifier of the connected sensor;

[0207] Step 602: After receiving the shelf image data and the location information of the acquisition device itself, identify whether the target shopping cart exists in the shelf image;

[0208] Step 603: When the target shopping cart is identified in the shelf image, the location information of the acquisition device corresponding to the shelf image containing the target shopping cart is sent to the target shopping cart as the final location information of the target shopping cart; the target shopping cart uses the final location information as the location information of the target shopping cart in the supermarket, and determines the set of goods information within the preset range of the target shopping cart's location; based on the set of goods information within the preset range, the target goods are identified within a limited range.

[0209] In one embodiment, when the data acquisition and control device determines that the target shopping cart has entered the shelf aisle for the first time within a preset time period, it determines that the target shopping cart has entered the shelf aisle.

[0210] In one embodiment, when the data acquisition control device determines that the target shopping cart has left the shelf aisle, it sends a stop-work notification message to all data acquisition devices in that aisle.

[0211] In one embodiment, sensors are deployed at both ends of a shelf aisle, wherein sensors at different ends of the same aisle are configured as paired sensors.

[0212] When using location information to assist in the identification of goods in a shopping cart, if the shopping cart is connected to the sensor at one end of a preset shelf aisle for the first time, the data collection and control device determines that the shopping cart is entering the aisle. If the preset time period threshold is exceeded, the cart is connected to the same sensor or the corresponding paired sensor in the same shelf aisle again to determine that the shopping cart is leaving the aisle.

[0213] In one embodiment, each target shopping cart is pre-configured with a unique shopping cart identifier; when a target shopping cart is identified in a shelf image, the location information of the acquisition device corresponding to the shelf image containing the target shopping cart is sent to the target shopping cart as the final location information of the target shopping cart, including:

[0214] When a target shopping cart is identified in a shelf image, the unique shopping cart identifier on the target shopping cart is identified;

[0215] The location information of the data acquisition device corresponding to the shelf image of the target shopping cart is sent as the final location information of the target shopping cart to the target shopping cart corresponding to the unique shopping cart identifier.

[0216] In one embodiment, when a target shopping cart is identified in a shelf image, the location information of the acquisition device corresponding to the shelf image containing the target shopping cart is sent to the target shopping cart as the final location information of the target shopping cart. This includes: when a target shopping cart is identified in a shelf image, target tracking is performed on each shopping cart entering the aisle according to the following target tracking algorithm to obtain the final location information of each target shopping cart:

[0217] The detection bounding box of the target shopping cart is used as the basis for trajectory tracking to obtain the trajectory identifier corresponding to each target shopping cart;

[0218] The location information of the data acquisition device corresponding to the shelf image of the target shopping cart is sent as the final location information of the target shopping cart to the target shopping cart corresponding to the trajectory marker.

[0219] In one embodiment, when a target shopping cart is identified in a shelf image, the location information of the acquisition device corresponding to the shelf image containing the target shopping cart is sent to the target shopping cart as the final location information of the target shopping cart. This includes: when a target shopping cart is identified in a shelf image, target tracking is performed on each shopping cart entering the aisle according to the following multi-target tracking algorithm to obtain the final location information of each target shopping cart:

[0220] When it is confirmed that there are more than two shopping carts in the shelf aisle, extract the image feature vector corresponding to the shopping cart image within the detection box of the target shopping cart;

[0221] Based on the detection bounding box of the target shopping cart and its corresponding image feature vector, the trajectory identifier corresponding to each target shopping cart is obtained;

[0222] The location information of the data acquisition device corresponding to the shelf image containing the target shopping cart is sent as the final location information of the target shopping cart to the target shopping cart corresponding to the trajectory marker.

[0223] In one embodiment, when a target shopping cart is identified in a shelf image, the location information of the acquisition device corresponding to the shelf image containing the target shopping cart is sent to the target shopping cart as the final location information of the target shopping cart. This includes: when a target shopping cart is identified in a shelf image, target tracking is performed on each shopping cart entering the aisle according to the following method to obtain the final location information of each target shopping cart:

[0224] Based on the shooting frequency of the acquisition device, the width and height of the detection frame of the target shopping cart are dynamically expanded;

[0225] The dynamically expanded detection box is used as the basis for trajectory tracking to obtain the trajectory identifier corresponding to each target shopping cart.

[0226] The location information of the data acquisition device corresponding to the shelf image containing the target shopping cart is sent as the final location information of the target shopping cart to the target shopping cart corresponding to the trajectory marker.

[0227] In one embodiment, based on a pre-defined range of product set information, the target product is identified within a limited scope, including: identifying products placed inside or removed from the vehicle according to the following classification and identification method:

[0228] Retrieve information on a set of goods within a preset range;

[0229] Determine the category to which each product in the product collection belongs;

[0230] If each product belongs to the same category, the pre-trained classification sub-model corresponding to that category is called to identify the target product, and the identification result is obtained after limiting the target product range.

[0231] In one embodiment, based on a pre-defined range of product set information, the target product is identified within a limited scope, including: identifying products placed inside or removed from the vehicle using the following retrieval and identification method:

[0232] Retrieve information on a set of goods within a preset range;

[0233] Features corresponding to products in the product set are selected from the full feature retrieval database to form a product feature retrieval sub-database;

[0234] Extract features from the target product image to obtain the target product feature vector;

[0235] The target product feature vector is retrieved from the product feature retrieval sub-database to obtain the identification result after limiting the target product range.

[0236] In one embodiment, if the identification result after limiting the target product range is that the product cannot be identified or the identification is incorrect, the method further includes:

[0237] Determine the category to which each product in the product collection belongs;

[0238] If each product belongs to the same category, all features corresponding to that category are selected from the full feature retrieval library to form a category-level sub-feature library;

[0239] The target product feature vector is retrieved from the category-level sub-feature library to obtain the final identification result after limiting the target product range.

[0240] This application also provides a system for assisting shopping cart item recognition based on location information, as described in the following embodiments. Since the principle behind this system's problem-solving is similar to the method for assisting shopping cart item recognition based on location information applied to a system, the implementation of this system can refer to the implementation of the method for assisting shopping cart item recognition based on location information applied to a processor; repeated details will not be elaborated further.

[0241] Figure 7 is a schematic diagram of the system for assisting shopping cart product recognition based on location information in an embodiment of this application. As shown in Figure 7, the system includes: a sensor 01, a target shopping cart 02, a data acquisition and control device 03, a data acquisition device 04, and a processor 05; wherein:

[0242] At least one sensor 01 is installed in the shelf aisle, and each sensor has a unique identifier.

[0243] The target shopping cart 02 is used to connect to the nearest sensor when entering the aisle through the end of the shelf aisle, obtain the location information of the connected sensor as the initial positioning information of the target shopping cart, and send the initial positioning information to the data acquisition and control device. The initial positioning information includes: the shelf aisle identifier determined by the unique identifier of the connected sensor; the final positioning information is used as the positioning information of the target shopping cart in the supermarket, and the set of goods information within the preset range of the target shopping cart's positioning location is determined; based on the set of goods information within the preset range, the target goods are identified within a limited range.

[0244] The data acquisition and control device 03 is used to send a wake-up message to the data acquisition device in the shelf aisle according to the shelf aisle identifier when the target shopping cart enters the shelf aisle; the data acquisition device is deployed on at least one shelf in the supermarket, and each data acquisition device is pre-configured with location information;

[0245] The data acquisition device 04 is used to start taking pictures of the shelf facing it at regular intervals or continuously after receiving the wake-up information, and send the shelf picture data and the location information of the data acquisition device itself to the processor.

[0246] The processor 05 is used to identify whether a target shopping cart exists in the shelf image after receiving the shelf image data and the location information of the acquisition device itself; when the target shopping cart is identified in the shelf image, the processor sends the location information of the acquisition device corresponding to the shelf image containing the target shopping cart as the final location information of the target shopping cart to the target shopping cart.

[0247] In one embodiment, the data acquisition control device is further configured to determine that the target shopping cart has entered the shelf aisle when the sensor first connects to the preset shelf aisle within a preset time period.

[0248] In one embodiment, the data acquisition control device is also used to send a stop-work notification message to all data acquisition devices in the aisle when it is determined that the target shopping cart has left the shelf aisle.

[0249] In one embodiment, sensors are deployed at both ends of a shelf aisle, wherein sensors at different ends of the same aisle are configured as paired sensors.

[0250] If the shopping cart is connected to the sensor at one end of the preset shelf aisle for the first time, the data collection and control device is also used to: determine that the shopping cart is entering the aisle, and when the preset time period threshold is exceeded, reconnect to the same sensor or the corresponding paired sensor in the same shelf aisle to determine that the shopping cart is leaving the aisle.

[0251] In one embodiment, each target shopping cart is pre-configured with a unique shopping cart identifier; when a target shopping cart is identified in a shelf image, the location information of the acquisition device corresponding to the shelf image containing the target shopping cart is sent to the target shopping cart as the final location information of the target shopping cart, including:

[0252] When a target shopping cart is identified in a shelf image, the unique shopping cart identifier on the target shopping cart is identified;

[0253] The location information of the data acquisition device corresponding to the shelf image of the target shopping cart is sent as the final location information of the target shopping cart to the target shopping cart corresponding to the unique shopping cart identifier.

[0254] In one embodiment, when a target shopping cart is identified in a shelf image, the location information of the acquisition device corresponding to the shelf image containing the target shopping cart is sent to the target shopping cart as the final location information of the target shopping cart. This includes: when a target shopping cart is identified in a shelf image, target tracking is performed on each shopping cart entering the aisle according to the following target tracking algorithm to obtain the final location information of each target shopping cart:

[0255] The detection bounding box of the target shopping cart is used as the basis for trajectory tracking to obtain the trajectory identifier corresponding to each target shopping cart;

[0256] The location information of the data acquisition device corresponding to the shelf image of the target shopping cart is sent as the final location information of the target shopping cart to the target shopping cart corresponding to the trajectory marker.

[0257] In one embodiment, when a target shopping cart is identified in a shelf image, the location information of the acquisition device corresponding to the shelf image containing the target shopping cart is sent to the target shopping cart as the final location information of the target shopping cart. This includes: when a target shopping cart is identified in a shelf image, target tracking is performed on each shopping cart entering the aisle according to the following multi-target tracking algorithm to obtain the final location information of each target shopping cart:

[0258] When it is confirmed that there are more than two shopping carts in the shelf aisle, extract the image feature vector corresponding to the shopping cart image within the detection box of the target shopping cart;

[0259] Based on the detection bounding box of the target shopping cart and its corresponding image feature vector, the trajectory identifier corresponding to each target shopping cart is obtained;

[0260] The location information of the data acquisition device corresponding to the shelf image containing the target shopping cart is sent as the final location information of the target shopping cart to the target shopping cart corresponding to the trajectory marker.

[0261] In one embodiment, when a target shopping cart is identified in a shelf image, the location information of the acquisition device corresponding to the shelf image containing the target shopping cart is sent to the target shopping cart as the final location information of the target shopping cart. This includes: when a target shopping cart is identified in a shelf image, target tracking is performed on each shopping cart entering the aisle according to the following method to obtain the final location information of each target shopping cart:

[0262] Based on the shooting frequency of the acquisition device, the width and height of the detection frame of the target shopping cart are dynamically expanded;

[0263] The dynamically expanded detection box is used as the basis for trajectory tracking to obtain the trajectory identifier corresponding to each target shopping cart.

[0264] The location information of the data acquisition device corresponding to the shelf image containing the target shopping cart is sent as the final location information of the target shopping cart to the target shopping cart corresponding to the trajectory marker.

[0265] In one embodiment, based on a pre-defined range of product set information, the target product is identified within a limited scope, including: identifying products placed inside or removed from the vehicle according to the following classification and identification method:

[0266] Retrieve information on a set of goods within a preset range;

[0267] Determine the category to which each product in the product collection belongs;

[0268] If each product belongs to the same category, the pre-trained classification sub-model corresponding to that category is called to identify the target product, and the identification result is obtained after limiting the target product range.

[0269] In one embodiment, based on a pre-defined range of product set information, the target product is identified within a limited scope, including: identifying products placed inside or removed from the vehicle using the following retrieval and identification method:

[0270] Retrieve information on a set of goods within a preset range;

[0271] Features corresponding to products in the product set are selected from the full feature retrieval database to form a product feature retrieval sub-database;

[0272] Extract features from the target product image to obtain the target product feature vector;

[0273] The target product feature vector is retrieved from the product feature retrieval sub-database to obtain the identification result after limiting the target product range.

[0274] In one embodiment, if the identification result after limiting the target product range is that the product cannot be identified or the identification is incorrect, the method further includes:

[0275] Determine the category to which each product in the product collection belongs;

[0276] If each product belongs to the same category, all features corresponding to that category are selected from the full feature retrieval library to form a category-level sub-feature library;

[0277] The target product feature vector is retrieved from the category-level sub-feature library to obtain the final identification result after limiting the target product range.

[0278] This application also provides a shopping cart for product recognition based on location information, as described in the following embodiments. Since the principle behind this shopping cart's solution is similar to the location-based product recognition method applied to the system, its implementation can refer to the implementation of the location-based product recognition method applied to the processor; repeated details will not be elaborated further.

[0279] Figure 8 is a schematic diagram of the shopping cart structure based on location information-assisted shopping cart product recognition in an embodiment of this application. As shown in Figure 8, the shopping cart includes:

[0280] The preliminary positioning unit 021 is used to connect with the nearest sensor when entering the aisle through the end of the shelf aisle, obtain the position information of the connected sensor as the preliminary positioning position information of the target shopping cart, and send the preliminary positioning position information to a data acquisition and control device. The preliminary positioning position information includes: the shelf aisle identifier determined by the unique identifier of the connected sensor. The data acquisition and control device is used to send a wake-up message to the data acquisition device in the shelf aisle according to the shelf aisle identifier when it determines that the target shopping cart has entered the shelf aisle. The data acquisition device is deployed on at least one shelf in the supermarket, and each data acquisition device is pre-configured with position information. After receiving the wake-up message, the data acquisition device is used to start taking pictures of the shelf facing it at regular intervals or continuously, and send the shelf picture data and the position information of the data acquisition device itself to the processor. After receiving the shelf picture data and the position information of the data acquisition device itself, the processor is used to identify whether the target shopping cart is in the shelf picture. When the target shopping cart is identified in the shelf picture, the processor sends the position information of the data acquisition device corresponding to the shelf picture with the target shopping cart as the final positioning position information of the target shopping cart to the target shopping cart.

[0281] The restricted range identification unit 022 is used to use the final location information as the location information of the target shopping cart in the supermarket, determine the set of goods information within the preset range of the target shopping cart's location, and identify the target goods within the restricted range based on the set of goods information within the preset range.

[0282] In one embodiment, when the data acquisition and control device determines that the target shopping cart has entered the shelf aisle for the first time within a preset time period, it determines that the target shopping cart has entered the shelf aisle.

[0283] In one embodiment, when the data acquisition control device determines that the target shopping cart has left the shelf aisle, it sends a stop-work notification message to all data acquisition devices in that aisle.

[0284] In one embodiment, sensors are deployed at both ends of a shelf aisle, wherein sensors at different ends of the same aisle are configured as paired sensors.

[0285] In location-based shopping cart product recognition, if the shopping cart is connected to the sensor at one end of a preset shelf aisle for the first time, the data acquisition and control device determines that the shopping cart is entering the aisle. If the preset time period threshold is exceeded, the cart is connected to the same sensor or the corresponding paired sensor in the same shelf aisle again to determine that the shopping cart is leaving the aisle.

[0286] In one embodiment, each target shopping cart is pre-configured with a unique shopping cart identifier; when a target shopping cart is identified in a shelf image, the location information of the acquisition device corresponding to the shelf image containing the target shopping cart is sent to the target shopping cart as the final location information of the target shopping cart, including:

[0287] When a target shopping cart is identified in a shelf image, the unique shopping cart identifier on the target shopping cart is identified;

[0288] The location information of the data acquisition device corresponding to the shelf image of the target shopping cart is sent as the final location information of the target shopping cart to the target shopping cart corresponding to the unique shopping cart identifier.

[0289] In one embodiment, when a target shopping cart is identified in a shelf image, the location information of the acquisition device corresponding to the shelf image containing the target shopping cart is sent to the target shopping cart as the final location information of the target shopping cart. This includes: when a target shopping cart is identified in a shelf image, target tracking is performed on each shopping cart entering the aisle according to the following target tracking algorithm to obtain the final location information of each target shopping cart:

[0290] The detection bounding box of the target shopping cart is used as the basis for trajectory tracking to obtain the trajectory identifier corresponding to each target shopping cart;

[0291] The location information of the data acquisition device corresponding to the shelf image of the target shopping cart is sent as the final location information of the target shopping cart to the target shopping cart corresponding to the trajectory marker.

[0292] In one embodiment, when a target shopping cart is identified in a shelf image, the location information of the acquisition device corresponding to the shelf image containing the target shopping cart is sent to the target shopping cart as the final location information of the target shopping cart. This includes: when a target shopping cart is identified in a shelf image, target tracking is performed on each shopping cart entering the aisle according to the following multi-target tracking algorithm to obtain the final location information of each target shopping cart:

[0293] When it is confirmed that there are more than two shopping carts in the shelf aisle, extract the image feature vector corresponding to the shopping cart image within the detection box of the target shopping cart;

[0294] Based on the detection bounding box of the target shopping cart and its corresponding image feature vector, the trajectory identifier corresponding to each target shopping cart is obtained;

[0295] The location information of the data acquisition device corresponding to the shelf image containing the target shopping cart is sent as the final location information of the target shopping cart to the target shopping cart corresponding to the trajectory marker.

[0296] In one embodiment, when a target shopping cart is identified in a shelf image, the location information of the acquisition device corresponding to the shelf image containing the target shopping cart is sent to the target shopping cart as the final location information of the target shopping cart. This includes: when a target shopping cart is identified in a shelf image, target tracking is performed on each shopping cart entering the aisle according to the following method to obtain the final location information of each target shopping cart:

[0297] Based on the shooting frequency of the acquisition device, the width and height of the detection frame of the target shopping cart are dynamically expanded;

[0298] The dynamically expanded detection box is used as the basis for trajectory tracking to obtain the trajectory identifier corresponding to each target shopping cart.

[0299] The location information of the data acquisition device corresponding to the shelf image containing the target shopping cart is sent as the final location information of the target shopping cart to the target shopping cart corresponding to the trajectory marker.

[0300] In one embodiment, based on a pre-defined range of product set information, the target product is identified within a limited scope, including: identifying products placed inside or removed from the vehicle according to the following classification and identification method:

[0301] Retrieve information on a set of goods within a preset range;

[0302] Determine the category to which each product in the product collection belongs;

[0303] If each product belongs to the same category, the pre-trained classification sub-model corresponding to that category is called to identify the target product, and the identification result is obtained after limiting the target product range.

[0304] In one embodiment, based on a pre-defined range of product set information, the target product is identified within a limited scope, including: identifying products placed inside or removed from the vehicle using the following retrieval and identification method:

[0305] Retrieve information on a set of goods within a preset range;

[0306] Features corresponding to products in the product set are selected from the full feature retrieval database to form a product feature retrieval sub-database;

[0307] Extract features from the target product image to obtain the target product feature vector;

[0308] The target product feature vector is retrieved from the product feature retrieval sub-database to obtain the identification result after limiting the target product range.

[0309] In one embodiment, if the identification result after limiting the target product range is that the product cannot be identified or the identification is incorrect, the method further includes:

[0310] Determine the category to which each product in the product collection belongs;

[0311] If each product belongs to the same category, all features corresponding to that category are selected from the full feature retrieval library to form a category-level sub-feature library;

[0312] The target product feature vector is retrieved from the category-level sub-feature library to obtain the final identification result after limiting the target product range.

[0313] This application also provides a processor for location-based shopping cart item recognition, as described in the following embodiments. Since the principle behind this processor's problem-solving is similar to the location-based shopping cart item recognition method applied to the system, the implementation of this processor can refer to the implementation of the location-based shopping cart item recognition method applied to the processor; repeated details will not be elaborated further.

[0314] Figure 9 is a schematic diagram of the processor for location-based shopping cart product recognition in an embodiment of this application. As shown in Figure 9, the processor includes:

[0315] The receiving unit 051 is used to receive shelf image data and the location information of the acquisition device itself. The shelf image data and the location information of the acquisition device are sent by the acquisition device, which is used to start taking shelf image data of itself facing the shelf at regular intervals or continuously after receiving the wake-up information. The wake-up information is sent by the acquisition control device, which is used to send the wake-up information to the acquisition device in the shelf aisle according to the shelf aisle identifier when it determines that the target shopping cart has entered the shelf aisle. The acquisition device is deployed on at least one shelf in the supermarket, and each acquisition device is pre-configured with location information. The shelf aisle identifier is included in the preliminary positioning location information sent by the target shopping cart. When the target shopping cart enters the aisle through the end of the shelf aisle, it connects to the nearest sensor, obtains the location information of the connected sensor as the preliminary positioning location information of the target shopping cart, and sends the preliminary positioning location information to the acquisition control device. The preliminary positioning location information includes: the shelf aisle identifier of the shelf aisle determined according to the unique identifier of the connected sensor.

[0316] The shopping cart recognition unit 052 is used to identify whether a target shopping cart exists in the shelf image after receiving shelf image data and the location information of the acquisition device itself.

[0317] The final positioning unit 053 is used to send the location information of the acquisition device corresponding to the shelf image containing the target shopping cart as the final positioning information of the target shopping cart when the target shopping cart is identified in the shelf image. The target shopping cart uses the final positioning information as the positioning information of the target shopping cart in the supermarket to determine the set of goods information within the preset range of the target shopping cart's positioning location. Based on the set of goods information within the preset range, the target goods are identified within a limited range.

[0318] In one embodiment, when the data acquisition and control device determines that the target shopping cart has entered the shelf aisle for the first time within a preset time period, it determines that the target shopping cart has entered the shelf aisle.

[0319] In one embodiment, when the data acquisition control device determines that the target shopping cart has left the shelf aisle, it sends a stop-work notification message to all data acquisition devices in that aisle.

[0320] In one embodiment, sensors are deployed at both ends of a shelf aisle, wherein sensors at different ends of the same aisle are configured as paired sensors.

[0321] In location-based shopping cart product recognition, if the shopping cart is connected to the sensor at one end of a preset shelf aisle for the first time, the data acquisition and control device determines that the shopping cart is entering the aisle. If the preset time period threshold is exceeded, the cart is connected to the same sensor or the corresponding paired sensor in the same shelf aisle again to determine that the shopping cart is leaving the aisle.

[0322] In one embodiment, each target shopping cart is pre-configured with a unique shopping cart identifier; when a target shopping cart is identified in a shelf image, the location information of the acquisition device corresponding to the shelf image containing the target shopping cart is sent to the target shopping cart as the final location information of the target shopping cart, including:

[0323] When a target shopping cart is identified in a shelf image, the unique shopping cart identifier on the target shopping cart is identified;

[0324] The location information of the data acquisition device corresponding to the shelf image of the target shopping cart is sent as the final location information of the target shopping cart to the target shopping cart corresponding to the unique shopping cart identifier.

[0325] In one embodiment, when a target shopping cart is identified in a shelf image, the location information of the acquisition device corresponding to the shelf image containing the target shopping cart is sent to the target shopping cart as the final location information of the target shopping cart. This includes: when a target shopping cart is identified in a shelf image, target tracking is performed on each shopping cart entering the aisle according to the following target tracking algorithm to obtain the final location information of each target shopping cart:

[0326] The detection bounding box of the target shopping cart is used as the basis for trajectory tracking to obtain the trajectory identifier corresponding to each target shopping cart;

[0327] The location information of the data acquisition device corresponding to the shelf image of the target shopping cart is sent as the final location information of the target shopping cart to the target shopping cart corresponding to the trajectory marker.

[0328] In one embodiment, when a target shopping cart is identified in a shelf image, the location information of the acquisition device corresponding to the shelf image containing the target shopping cart is sent to the target shopping cart as the final location information of the target shopping cart. This includes: when a target shopping cart is identified in a shelf image, target tracking is performed on each shopping cart entering the aisle according to the following multi-target tracking algorithm to obtain the final location information of each target shopping cart:

[0329] When it is confirmed that there are more than two shopping carts in the shelf aisle, extract the image feature vector corresponding to the shopping cart image within the detection box of the target shopping cart;

[0330] Based on the detection bounding box of the target shopping cart and its corresponding image feature vector, the trajectory identifier corresponding to each target shopping cart is obtained;

[0331] The location information of the data acquisition device corresponding to the shelf image containing the target shopping cart is sent as the final location information of the target shopping cart to the target shopping cart corresponding to the trajectory marker.

[0332] In one embodiment, when a target shopping cart is identified in a shelf image, the location information of the acquisition device corresponding to the shelf image containing the target shopping cart is sent to the target shopping cart as the final location information of the target shopping cart. This includes: when a target shopping cart is identified in a shelf image, target tracking is performed on each shopping cart entering the aisle according to the following method to obtain the final location information of each target shopping cart:

[0333] Based on the shooting frequency of the acquisition device, the width and height of the detection frame of the target shopping cart are dynamically expanded;

[0334] The dynamically expanded detection box is used as the basis for trajectory tracking to obtain the trajectory identifier corresponding to each target shopping cart.

[0335] The location information of the data acquisition device corresponding to the shelf image containing the target shopping cart is sent as the final location information of the target shopping cart to the target shopping cart corresponding to the trajectory marker.

[0336] In one embodiment, based on a pre-defined range of product set information, the target product is identified within a limited scope, including: identifying products placed inside or removed from the vehicle according to the following classification and identification method:

[0337] Retrieve information on a set of goods within a preset range;

[0338] Determine the category to which each product in the product collection belongs;

[0339] If each product belongs to the same category, the pre-trained classification sub-model corresponding to that category is called to identify the target product, and the identification result is obtained after limiting the target product range.

[0340] In one embodiment, based on a pre-defined range of product set information, the target product is identified within a limited scope, including: identifying products placed inside or removed from the vehicle using the following retrieval and identification method:

[0341] Retrieve information on a set of goods within a preset range;

[0342] Features corresponding to products in the product set are selected from the full feature retrieval database to form a product feature retrieval sub-database;

[0343] Extract features from the target product image to obtain the target product feature vector;

[0344] The target product feature vector is retrieved from the product feature retrieval sub-database to obtain the identification result after limiting the target product range.

[0345] In one embodiment, if the identification result after limiting the target product range is that the product cannot be identified or the identification is incorrect, the method further includes:

[0346] Determine the category to which each product in the product collection belongs;

[0347] If each product belongs to the same category, all features corresponding to that category are selected from the full feature retrieval library to form a category-level sub-feature library;

[0348] The target product feature vector is retrieved from the category-level sub-feature library to obtain the final identification result after limiting the target product range.

[0349] This application also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the above-described method for assisting shopping cart item recognition based on location information.

[0350] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for assisting shopping cart item recognition based on location information.

[0351] This application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the above-described method for assisting shopping cart item recognition based on location information.

[0352] In this embodiment, the shopping cart product recognition scheme based on location information, compared with existing technologies that suffer from low accuracy and efficiency in supermarket product recognition, achieves the following improvements: When the target shopping cart enters the aisle via the end of the shelf aisle, it connects to the nearest sensor, acquires the location information of the connected sensor as the target shopping cart's preliminary location information, and sends this preliminary location information to the data acquisition and control device. The preliminary location information includes the shelf aisle identifier determined by the unique identifier of the connected sensor. When the data acquisition and control device determines that the target shopping cart has entered the shelf aisle, it sends a wake-up message to the data acquisition device within that shelf aisle based on the shelf aisle identifier. The data acquisition devices are deployed on at least one shelf in the supermarket, and each device is pre-configured with location information. Upon receiving the wake-up message, the data acquisition device begins to take photos periodically or continuously. The device captures images of the shelves facing the supermarket, and sends the image data along with the device's own location information to a processor. Upon receiving the images, the processor identifies whether a target shopping cart exists within the shelf image. If a target shopping cart is detected, the processor sends the location information of the corresponding device within the shelf image as the final location information for the target shopping cart. The target shopping cart uses this final location information as its position within the supermarket, determining the set of products within a preset range of its location. Based on this product set, the system performs limited-range product identification of the target products. This system, using the shopping cart's location information, assists in limited-range product identification of the target products, improving both the accuracy and speed of product recognition.

[0353] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0354] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more flowchart illustrations and / or one or more block diagrams.

[0355] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flowcharts and / or one or more block diagrams.

[0356] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more flowcharts and / or one or more block diagrams.

[0357] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of this application. It should be understood that the above descriptions are merely specific embodiments of this application and are not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

A product identification method based on location information, characterized in that, This method is applied to a system, and the method includes: When the target shopping cart enters the aisle through the end of the shelf aisle, it connects to the nearest sensor and obtains the location information of the connected sensor as the initial location information of the target shopping cart. The initial location information is then sent to the data acquisition and control device. The initial location information includes the shelf aisle identifier determined by the unique identifier of the connected sensor. When the data collection and control device determines that the target shopping cart has entered the shelf aisle, it sends a wake-up message to the data collection device in that shelf aisle according to the shelf aisle identifier; the data collection device is deployed on at least one shelf in the supermarket, and each data collection device is pre-configured with location information; After receiving the wake-up message, the data acquisition device begins to periodically or continuously capture images of the shelf it faces, and sends the shelf image data and the location information of the data acquisition device itself to the processor; and After receiving the shelf image data and the location information of the acquisition device itself, the processor identifies whether the target shopping cart exists in the shelf image. When the target shopping cart is identified in the shelf image, the processor sends the location information of the acquisition device corresponding to the shelf image containing the target shopping cart as the final location information of the target shopping cart to the target shopping cart. The target shopping cart uses the final location information as its location information within the supermarket to determine the set of products within a preset range of the target shopping cart's location; based on the set of products within the preset range, it performs product identification within a limited range for the target products. The method as described in claim 1, characterized in that, When the data collection and control device first connects to the sensor of the preset shelf aisle within a preset time period, it determines that the target shopping cart has entered the shelf aisle. The method as described in claim 1, characterized in that, Also includes: When the data collection and control device determines that the target shopping cart has left the shelf aisle, it sends a stop-work notification to all data collection devices in that aisle. The method as described in claim 3, characterized in that, Sensors are deployed at both ends of the shelving aisle, with sensors at different ends of the same aisle configured as paired sensors. The product identification method based on location information further includes: if the shopping cart is connected to the sensor at one end of the preset shelf aisle for the first time, the data acquisition and control device determines that the shopping cart is entering the aisle; if the preset time period threshold is exceeded, the shopping cart is connected to the same sensor or the corresponding paired sensor in the same shelf aisle again to determine that the shopping cart is leaving the aisle. The method as described in claim 1, characterized in that, Each target shopping cart is pre-configured with a unique shopping cart identifier; when a target shopping cart is detected in a shelf image, the location information of the acquisition device corresponding to the shelf image containing the target shopping cart is sent to the target shopping cart as the final location information, including: When a target shopping cart is identified in a shelf image, the unique shopping cart identifier on the target shopping cart is identified; The location information of the data acquisition device corresponding to the shelf image of the target shopping cart is sent as the final location information of the target shopping cart to the target shopping cart corresponding to the unique shopping cart identifier. The method as described in claim 1, characterized in that, When a target shopping cart is identified in a shelf image, the location information of the acquisition device corresponding to the shelf image containing the target shopping cart is sent to the target shopping cart as its final location information. This includes: when a target shopping cart is identified in a shelf image, target tracking is performed on each shopping cart entering the aisle according to the following target tracking algorithm to obtain the final location information of each target shopping cart: Using the bounding boxes of the target shopping carts as the basis for trajectory tracking, the trajectory identifier corresponding to each target shopping cart is obtained; and The location information of the data acquisition device corresponding to the shelf image of the target shopping cart is sent as the final location information of the target shopping cart to the target shopping cart corresponding to the trajectory marker. The method as described in claim 1, characterized in that, When a target shopping cart is identified in a shelf image, the location information of the acquisition device corresponding to the shelf image containing the target shopping cart is sent to the target shopping cart as its final location information. This includes: when a target shopping cart is identified in a shelf image, each shopping cart entering the aisle is tracked according to the following multi-target tracking algorithm to obtain the final location information of each target shopping cart: When it is confirmed that there are more than two shopping carts in the shelf aisle, extract the image feature vector corresponding to the shopping cart image within the detection box of the target shopping cart; Based on the detection bounding box of the target shopping cart and its corresponding image feature vector, the trajectory identifier for each target shopping cart is obtained; and The location information of the data acquisition device corresponding to the shelf image containing the target shopping cart is sent as the final location information of the target shopping cart to the target shopping cart corresponding to the trajectory marker. The method as described in claim 1, characterized in that, When a target shopping cart is identified in a shelf image, the location information of the acquisition device corresponding to the shelf image containing the target shopping cart is sent to the target shopping cart as its final location information. This includes: when a target shopping cart is identified in a shelf image, target tracking is performed on each shopping cart entering the aisle using the following method to obtain the final location information of each target shopping cart: Based on the shooting frequency of the acquisition device, the width and height of the detection frame of the target shopping cart are dynamically expanded; The dynamically expanded detection box is used as the basis for trajectory tracking to obtain the trajectory identifier corresponding to each target shopping cart; and The location information of the data acquisition device corresponding to the shelf image containing the target shopping cart is sent as the final location information of the target shopping cart to the target shopping cart corresponding to the trajectory marker. The method as described in claim 1, characterized in that, Based on the product set information within a preset range, the target product is identified within a limited range, including: identifying products placed inside or removed from the vehicle according to the following classification and identification method: Retrieve information on a set of goods within a preset range; Determine the category to which each product in the product collection belongs; and If each product belongs to the same category, the pre-trained classification sub-model corresponding to that category is called to identify the target product, and the identification result is obtained after limiting the target product range. The method as described in claim 1, characterized in that, Based on the product set information within a preset range, the target product is identified within a limited range, including: identifying products placed inside or removed from the vehicle using the following retrieval and identification method: Retrieve information on a set of goods within a preset range; Features corresponding to products in the product set are selected from the full feature retrieval database to form a product feature retrieval sub-database; Extract features from the target product image to obtain the target product feature vector; and The target product feature vector is retrieved from the product feature retrieval sub-database to obtain the identification result after limiting the target product range. The method as described in claim 10, characterized in that, If the identification result after limiting the target product range is that the product cannot be identified or the identification is incorrect, the method further includes: Determine the category to which each product in the product collection belongs; If each product belongs to the same category, all features corresponding to that category are selected from the full feature retrieval database to form a category-level sub-feature database; and The target product feature vector is retrieved from the category-level sub-feature library to obtain the final identification result after limiting the target product range. A product identification method based on location information, characterized in that, This method is applied to a shopping cart and includes: Upon entering the aisle via the end of the shelf, the cart connects to the nearest sensor, acquires the location information of the connected sensor as the initial location information of the target shopping cart, and sends this initial location information to a data acquisition and control device. The initial location information includes: the shelf aisle identifier determined by the unique identifier of the connected sensor. The data acquisition and control device, upon determining that the target shopping cart has entered the shelf aisle, sends a wake-up message to the acquisition device within that shelf aisle based on the shelf aisle identifier. The acquisition device is deployed on at least one shelf in the supermarket, and each acquisition device is pre-configured with location information. Upon receiving the wake-up message, the acquisition device begins to periodically or continuously capture shelf images of itself facing the shelf, and sends the shelf image data and its own location information to a processor. The processor, upon receiving the shelf image data and the acquisition device's own location information, identifies whether the target shopping cart is present in the shelf image. When the target shopping cart is identified, the processor sends the location information of the acquisition device corresponding to the shelf image containing the target shopping cart as the final location information of the target shopping cart to the target shopping cart. The final location information is used as the location information of the target shopping cart in the supermarket to determine the set of goods within a preset range of the target shopping cart's location; based on the set of goods within the preset range, the target goods are identified within a limited range. A product identification method based on location information, characterized in that, This method is applied to a processor, and the method includes: The system receives shelf image data and the location information of the acquisition device itself. The shelf image data and the location information of the acquisition device are sent by the acquisition device, which, upon receiving a wake-up message, begins periodically or continuously capturing shelf image data facing the shelf. The wake-up message is sent by the acquisition control device, which, upon determining that a target shopping cart has entered the shelf aisle, sends a wake-up message to the acquisition device within that aisle based on the shelf aisle identifier. The acquisition devices are deployed on at least one shelf in the supermarket, and each acquisition device is pre-configured with location information. The shelf aisle identifier is included in the preliminary positioning information sent by the target shopping cart. When the target shopping cart enters the aisle via the end of the shelf aisle, it connects to the nearest sensor, obtains the location information of the connected sensor as the target shopping cart's preliminary positioning information, and sends this preliminary positioning information to the acquisition control device. The preliminary positioning information includes: the shelf aisle identifier determined based on the unique identifier of the connected sensor. After receiving the shelf image data and the location information of the data acquisition device itself, it identifies whether a target shopping cart exists in the shelf image; and When a target shopping cart is identified in a shelf image, the location information of the acquisition device corresponding to the shelf image containing the target shopping cart is sent to the target shopping cart as the final location information of the target shopping cart. The target shopping cart uses the final location information as its location information in the supermarket to determine the set of goods within a preset range of the target shopping cart's location. Based on the set of goods within the preset range, the target goods are identified within a limited range. A product identification system based on location information, characterized in that, include: Sensors, target shopping cart, data collection and control device, data collection device, processor; among which: At least one sensor is installed in the shelf aisle, and each sensor has a unique identifier. The target shopping cart, upon entering the aisle via the end of the shelf aisle, connects to the nearest sensor to acquire the location information of the connected sensor as the initial positioning information of the target shopping cart, and sends this initial positioning information to the data acquisition and control device. The initial positioning information includes: the shelf aisle identifier determined by the unique identifier of the connected sensor; the final positioning information is used as the target shopping cart's positioning information in the supermarket to determine the set of goods within a preset range of the target shopping cart's positioning location; and based on the set of goods within the preset range, the target goods are identified within a limited range. A data acquisition and control device is used to send a wake-up message to the data acquisition device in the shelf aisle based on the shelf aisle identifier when the target shopping cart is determined to enter the shelf aisle; the data acquisition device is deployed on at least one shelf in the supermarket, and each data acquisition device is pre-configured with location information; The data acquisition device, upon receiving the wake-up information, begins periodically or continuously capturing images of the shelf it faces, and sends the shelf image data and the location information of the acquisition device itself to the processor; and The processor is used to identify whether a target shopping cart exists in the shelf image after receiving the shelf image data and the location information of the acquisition device itself; when the target shopping cart is identified in the shelf image, the processor sends the location information of the acquisition device corresponding to the shelf image containing the target shopping cart as the final location information of the target shopping cart to the target shopping cart. A shopping cart for product recognition based on location information, characterized in that, include: A preliminary positioning unit is used to connect with the nearest sensor when entering the aisle via the end of the shelf aisle, acquire the position information of the connected sensor as the preliminary positioning information of the target shopping cart, and send the preliminary positioning information to a data acquisition and control device. The preliminary positioning information includes: the shelf aisle identifier determined by the unique identifier of the connected sensor. The data acquisition and control device is used to send a wake-up message to the data acquisition device in the shelf aisle based on the shelf aisle identifier when it determines that the target shopping cart has entered the shelf aisle. The data acquisition device is deployed on at least one shelf in the supermarket, and each data acquisition device is pre-configured with position information. After receiving the wake-up message, the data acquisition device is used to start taking pictures of the shelf facing it at regular intervals or continuously, and send the shelf picture data and the position information of the data acquisition device itself to a processor. After receiving the shelf picture data and the position information of the data acquisition device itself, the processor is used to identify whether the target shopping cart is in the shelf picture. When the target shopping cart is identified in the shelf picture, the processor sends the position information of the data acquisition device corresponding to the shelf picture containing the target shopping cart as the final positioning information of the target shopping cart to the target shopping cart. The restricted range identification unit is used to use the final location information as the location information of the target shopping cart in the supermarket, determine the set of goods information within a preset range of the target shopping cart's location, and identify the target goods within a restricted range based on the set of goods information within the preset range. A processor for product recognition based on location information, characterized in that, include: The receiving unit is used to receive shelf image data and the location information of the acquisition device itself. The shelf image data and the location information of the acquisition device itself are sent by the acquisition device. The acquisition device is used to start taking shelf image data of its facing shelf at regular intervals or continuously after receiving the wake-up information. The wake-up information is sent by the acquisition control device. The acquisition control device is used to send the wake-up information to the acquisition device in the shelf aisle according to the shelf aisle identifier when it determines that the target shopping cart has entered the shelf aisle. The acquisition device is deployed on at least one shelf in the supermarket, and each acquisition device is pre-configured with location information. The shelf aisle identifier is included in the preliminary positioning location information sent by the target shopping cart. The target shopping cart is used to connect to the nearest sensor when it enters the aisle through the end of the shelf aisle, obtain the location information of the connected sensor as the preliminary positioning location information of the target shopping cart, and send the preliminary positioning location information to the acquisition control device. The preliminary positioning location information includes: the shelf aisle identifier of the entry into the shelf aisle determined according to the unique identifier of the connected sensor. The shopping cart recognition unit is used to identify whether a target shopping cart exists in the shelf image after receiving shelf image data and the location information of the acquisition device itself; and The final positioning unit is used to send the location information of the acquisition device corresponding to the shelf image containing the target shopping cart as the final positioning information of the target shopping cart when the target shopping cart is identified in the shelf image. The target shopping cart uses the final positioning information as its positioning information in the supermarket to determine the set of goods information within a preset range of the target shopping cart's positioning location. Based on the set of goods information within the preset range, the target goods are identified within a limited range. A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, it implements the method of any one of claims 1 to 13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method of any one of claims 1 to 13. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the method of any one of claims 1 to 13.

Images