Method for identifying goods based on visual recognition and related device

Abstract

The application provides a commodity identification method based on visual recognition and a related device thereof. The method comprises: extracting query features of a cargo entry image, calculating similarities between the query features and features corresponding to each commodity category in a commodity database, screening commodity categories with similarities reaching a first threshold from the commodity database as candidate commodity categories, and sorting the candidate commodity categories from high to low to generate a candidate commodity category queue. When the candidate commodity category queue does not belong to an easily-confused category library, the commodity information of the cargo is queried from the commodity database according to a first candidate commodity category in the candidate commodity category queue. The application is based on image preprocessing, feature matching, easily-confused rechecking and incremental updating, thereby significantly reducing the misidentification rate and improving the identification accuracy.

Description

Technical Field

[0001] This application relates to the field of image recognition technology, and in particular to a product recognition method and related equipment based on visual recognition. Background Technology

[0002] In checkout scenarios such as fresh food supermarkets, community fresh food stores, or farmers' markets, there are many types of non-barcode products such as vegetables and fruits with subtle differences in appearance. They are often placed at multiple angles, obstructed, subject to changes in lighting, and new products are quickly put on the shelves. This means that the checkout process must not only quickly identify the product category but also complete the pricing and settlement simultaneously.

[0003] Existing product identification and pricing methods typically involve manual weighing and labeling followed by barcode scanning for payment, or deploying a general image recognition model at the cash register to automatically extract the SKU and unit price after recognizing fruit and vegetable images. For cases where recognition is uncertain, manual verification by the cashier is often required. While these methods can reduce manual data entry to some extent, they still tend to follow a "fixed model first, then direct recognition" approach, relying heavily on pre-trained models to cover all products at once and completing matching and pricing locally or in the cloud.

[0004] When faced with long-tail SKUs, new product launches, and easily confused products with highly similar appearances, traditional methods are difficult to identify in a timely and accurate manner. This often results in problems such as "inaccurate identification, delayed updates, and frequent manual intervention," which in turn affects checkout efficiency and increases the risk of settlement disputes. Summary of the Invention

[0005] In view of this, this application provides a visual recognition-based commodity identification method and related equipment, which improves the accuracy of identifying priced commodities.

[0006] On the one hand, embodiments of this application provide a product identification method based on visual recognition, the method comprising: Extract query features from the images of goods entered, and calculate the similarity between the query features and the features corresponding to each product category in the preset product database; Product categories whose similarity reaches a preset first threshold are selected from the product database as candidate product categories, and the candidate product categories are sorted from high to low according to the similarity to generate a candidate product category queue. When the candidate product category queue does not belong to the preset easily confused product category library, the product information of the product is queried from the product database according to the first candidate product category in the candidate product category queue.

[0007] In an optional implementation, the step of extracting query features from the product entry image and calculating the similarity between the query features and the features corresponding to each product category in a preset product database includes: Acquire multiple frames of recorded images of the goods, and perform noise reduction and color space conversion on the recorded images to obtain images to be recognized with uniform image size; The image to be identified is subjected to subject region detection and target region segmentation, and the product region in the image to be identified is extracted as the region to be identified. Based on preset visual and semantic feature extraction rules, visual features are extracted and semantic mapping is performed on the region to be identified to obtain the query features corresponding to the goods. Calculate the similarity between the query features and the features corresponding to each product category in the preset product database.

[0008] In an optional implementation, when none of the similarities reach the first threshold, the method further includes: Product categories whose similarity reaches a preset second threshold are selected from the product database as candidate product categories, and the candidate product categories are sorted from high to low according to the similarity to generate a queue of categories to be determined; wherein, the second threshold is less than the first threshold; Based on the preset number of candidates and the order of the queue of categories to be determined, the categories of goods to be determined are sent to the preset target address; Based on the received first feedback information, a first target product category is determined from the queue of categories to be determined, so as to query the product information of the product from the product database based on the first target product category.

[0009] In an optional implementation, after generating the candidate product category queue, the method further includes: When the first candidate product category belongs to the easily confused product category library, based on the easily confused product category library and the first candidate product category, a second candidate product category belonging to the easily confused product category library is selected from the candidate product category queue, and the difference between the similarity corresponding to the second candidate product category and the similarity corresponding to the first candidate product category is calculated to obtain the similarity score difference. When the similarity difference is less than a preset third threshold, the second candidate product category and the first candidate product category are used as a confirmation list, and the confirmation list is sent to a preset target address. Based on the received second feedback information, a second target product category is determined from the list to be confirmed, so as to query the product information of the product from the product database according to the second target product category; When the similarity difference reaches a preset third threshold, the product information of the goods is queried from the product database according to the first candidate product category in the candidate product category queue.

[0010] In an optional implementation, the method further includes: According to the preset sample format, the product information, the entered image, and the received manual selection event are combined to form an incremental learning sample, and the product category corresponding to the incremental learning sample is labeled. According to a preset time period, incremental learning samples corresponding to each product category are obtained, and statistical calculations of the features of the incremental learning samples are performed to update the features corresponding to the product category.

[0011] In an optional implementation, the method for obtaining the second feedback information further includes: According to the preset event group format, the second feedback information, the first candidate product type, the second candidate product type, and the similarity difference are combined into a confusion event pair; The occurrence frequency of the confusion event pairs is counted according to a preset time window. When the occurrence frequency reaches a preset frequency threshold, the easily confused category library is updated according to the confusion event pairs. Calculate the average or minimum similarity difference among the confused event pairs, and update the third threshold using the average or minimum similarity difference.

[0012] On one hand, embodiments of this application provide a product identification device based on visual recognition, the device comprising: The similarity calculation module is used to extract query features from the product entry images and calculate the similarity between the query features and the features corresponding to each product category in the preset product database. The category filtering module is used to filter out product categories whose similarity reaches a preset first threshold from the product database as candidate product categories, and sort the candidate product categories from high to low according to the similarity to generate a candidate product category queue. The product query module is used to query the product information of the goods from the product database based on the first candidate product category in the candidate product category queue when the candidate product category queue does not belong to the preset easily confused product category library.

[0013] In summary, this application includes at least the following beneficial technical effects: 1. By performing noise reduction, color space conversion, main region detection, and target segmentation on the images of the goods, and then combining visual features and semantic mapping to calculate similarity, and filtering candidate product categories according to thresholds, it is possible to reduce misjudgments caused by background interference and image differences, and improve the adaptability to different shooting conditions and different product appearances.

[0014] 2. By using a database of easily confused product categories, similarity difference judgment, and manual feedback confirmation mechanism, when multiple highly similar products are difficult to distinguish automatically, the confused objects are first screened out and then handed over to manual confirmation to avoid direct misjudgment as similar products, thereby improving the reliability of the recognition results.

[0015] 3. The product information, entered images, and manually selected events are combined to form incremental learning samples, and the product features are updated periodically. At the same time, the easily confused category library and the third threshold can be dynamically updated based on confusion events, so that the recognition model can be continuously corrected and optimized with actual application. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart of a product identification method based on visual recognition provided in an embodiment of this application; Figure 2 This is a functional block diagram of a product identification device based on visual recognition provided in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0018] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0019] It should be noted that the message processing solution provided in this application requires special explanation of the following two points: 1. The relevant data (such as query features, etc.) involved in the message processing process in this application. When the above embodiments of this application are applied to specific products or technologies, permission or consent from the target audience is required, and the collection, use, and processing of relevant data must comply with the relevant laws, regulations, and standards of the region, conforming to the principles of legality, legitimacy, and necessity, and not involving the acquisition of data types prohibited or restricted by laws and regulations. In some optional embodiments, the relevant data involved in the embodiments of this application is obtained after separate authorization from the target audience. In addition, when obtaining separate authorization from the target audience, the purpose of the relevant data is explained to the target audience.

[0020] 2. It is understood that in this application, the term "at least one" refers to one or more, and "multiple" means two or more; for example, "at least one notification method" means one, two, or more notification methods. The terms "first," "second," etc., are used to distinguish identical or similar items with essentially the same function. It should be understood that there is no logical or temporal dependency between "first," "second," and "nth," nor is there any limitation on the quantity or execution order.

[0021] like Figure 1 The diagram shows a flowchart of a visual recognition-based product identification method provided in an embodiment of this application. The visual recognition-based product identification method provided in this embodiment includes the following steps.

[0022] Step S1: Extract the query features of the product entry image and calculate the similarity between the query features and the features corresponding to each product category in the preset product database.

[0023] First, the system needs to acquire multiple frames of the product images. These images refer to the original images of the product captured by the cash register camera during each recognition process. Depending on the application scenario, there are two main sources of these images: The first is the initial image capture when a new product is first placed on the shelf. At this time, the cashier slowly rotates or flips the product according to the on-screen prompts, causing the camera to record a short video of 3 to 5 seconds from multiple angles such as the front, side, and top. The system extracts multiple keyframes from this video, each keyframe constituting one image. The second is real-time recognition capture during daily checkout. When a customer places a product with pre-recorded features (such as a Fuji apple) under the camera, the system automatically triggers a photo or short video recording, acquiring one or more frames of the current product as the image. The reason for acquiring multiple frames rather than a single frame is that the product placement angle, lighting conditions, and occlusion in the actual checkout environment are random. A single frame might capture an atypical appearance of the product (such as the stem side of an apple), causing the extracted features to deviate from the product's common visual form. Multiple frames of images cover different perspectives and poses, providing more comprehensive raw data for subsequent fusion processing.

[0024] After acquiring multiple frames of input images, the system performs preprocessing operations on each frame, including denoising and color space conversion, ultimately obtaining a uniform image size for recognition. The purpose of denoising is to eliminate granular noise generated by the camera sensor under low light or high gain conditions, as well as coding artifacts introduced during compression and transmission. Common denoising methods include Gaussian filtering or median filtering. These filters smooth the image by calculating a weighted average of the pixel's neighborhood, while preserving the texture edges of the product surface as much as possible. Color space conversion transforms the camera's default output RGB (red, green, blue) format into a color space more suitable for subsequent feature extraction. For example, RGB can be converted to HSV (hue, saturation, brightness) space, or the RGB components can be directly normalized. The reason for this conversion is that the same product from different batches or origins may differ in overall brightness or contrast (for example, Fuji apples are brighter red in summer and darker in winter). Color space conversion can reduce the interference of brightness information on feature extraction, allowing the model to focus more on relatively stable color attributes such as hue and saturation. After denoising and color space conversion, the system scales all images to a fixed size, such as 224×224 pixels. This size is the standard input requirement for lightweight feature extraction networks (such as MobileNetV3). The fixed size ensures that the subsequent neural network can process each image with a fixed receptive field and stride, avoiding feature misalignment caused by inconsistent sizes.

[0025] The preprocessed image to be identified contains both the main product and background areas, such as the checkout counter, the cashier's fingers, or the product packaging. These background areas do not help in identifying the product type; instead, they introduce noise and interfere with the semantic purity of the feature vector. Therefore, the system needs to perform main region detection and target region segmentation on the image to be identified, extracting the product area as the region to be identified. The goal of main region detection is to locate the approximate bounding box of the product in the image. Since fresh produce has irregular shapes and may stack, simple rectangular detection boxes often cannot accurately cut out the product outline. Therefore, further target region segmentation is required, i.e., using edge detection or graph cut algorithms to separate product pixels from background pixels, generating a binary mask. The area inside the mask is the product area, and the area outside the mask is the background. The system then extracts the product pixel area based on this mask and sets the remaining pixels to a uniform background color or directly crops them out. The feature extraction network should only learn the visual information of the product itself, not the checkout counter texture or finger shape. For example, when identifying Chinese yam, if the image contains the cashier's white gloves, the network might incorrectly identify "white" as a characteristic associated with yam, leading to confusion when identifying other white products. By segmenting the main body region, the system ensures the purity of the image content fed into the feature extraction network, thereby improving the generalization ability of the features.

[0026] After obtaining the region to be identified, which contains only the main body of the goods, the system performs visual feature extraction and semantic mapping on the region to be identified according to preset visual and semantic feature extraction rules to obtain the query features corresponding to the goods. The visual and semantic feature extraction rules are the forward propagation process of a lightweight deep learning network that has been pre-trained on a large number of general images and can map any input image to a fixed-dimensional semantic space. Specifically, the system loads a MobileNetV3 or ShuffleNetV2 model with the last classification layer removed, inputs the image of the region to be identified (a pixel tensor of size 224×224×3) into the network, and performs operations such as convolution, batch normalization, activation function, and pooling layer by layer. Finally, a dense vector (i.e., the query feature) is output from the penultimate layer of the network. The query feature is usually a 512-dimensional or 128-dimensional vector, where the value in each dimension is a floating-point number, such as a sequence of [0.85, 0.20, 0.45, …, -0.12]. This application uses the penultimate layer's classification output instead of the last layer because the penultimate layer stores the highest-level semantic abstract features of the image, which have not yet been compressed into class probabilities using Softmax. Therefore, it retains richer geometric and texture information, enabling it to be used for open-set recognition—that is, identifying new varieties not found in the training set. The position of the query features in high-dimensional space represents the visual semantic connotation of the product. The closer the features are in space, the more visually similar the two products are. For example, the query features for Fuji apples are almost oriented in the same direction as the features of Fuji apples in the database, while they have a certain angle with the vector of Gala apples.

[0027] Finally, the system calculates the similarity between the query features and the features corresponding to each product category in the pre-set product database. The product database is a locally stored table, with each row recording information for a product category (SKU), including the product name, unit price, and the corresponding support feature vector. The support feature vector is a standard feature vector calculated from multiple frames of images when the product is first entered, or the mean of the features updated in subsequent incremental learning. To quantify the similarity between the query vector and each support vector (i.e., the features corresponding to each product category in the product database), the system uses cosine similarity as a metric. Cosine similarity focuses on the direction of two vectors rather than their length, thus ignoring changes in vector magnitude caused by differences in light intensity and camera gain, comparing only whether the semantic orientation is consistent. For each product category, the system calculates the similarity using the formula... similarity in, To query features, The closer the feature calculation result for each product category in the product database is to 1, the higher the visual similarity between the query product and the products in the database. For example, the similarity calculated between the query vector of a Fuji apple and the support vector of Fuji apples in the database may reach 0.994, while the similarity calculated between the query vector and the support vector of Gala apples in the database may only be 0.91.

[0028] Step S2: Select product categories whose similarity reaches a preset first threshold from the product database as candidate product categories, and sort the candidate product categories from high to low according to the similarity to generate a candidate product category queue.

[0029] First, product categories that reach a preset first threshold are selected from all similarity scores and treated as candidate product categories. The first threshold is a high confidence level, for example, set to 0.95 (cosine similarity). This high threshold is chosen because when the similarity between a query vector and a support vector exceeds 0.95, it means the angle between the two vectors in the high-dimensional semantic space is extremely small, and their directions are almost identical. In a real-world checkout environment, this level of similarity usually only occurs between different individuals of the same product type. For example, if a query vector for a Fuji apple and the support vectors for Fuji apples in the database have a similarity of 0.994, which is much higher than 0.95, the system can confidently confirm that the current product is a Fuji apple. Conversely, if the similarity is below 0.95, even if the highest score reaches 0.94, there is still a risk of confusion because different varieties (such as Fuji and Gala) may have scores as low as 0.91, and therefore automatic confirmation is not recommended. After selecting candidate product categories, the system sorts these categories in descending order of similarity, generating a queue of candidate product categories. The first category in this queue, i.e., the one with the highest similarity, is called the first candidate product category.

[0030] When the system finds that all similarity scores have failed to reach the first threshold, it means that the visual features of the product to be identified do not have a high degree of consistency with the support vectors of any product in the database. This situation frequently occurs in actual checkouts, due to reasons such as: the product being partially obscured, abnormal lighting conditions, or the product itself being similar in appearance to multiple varieties in the database (e.g., a fruit whose shape is somewhere between a green apple and a green pear). In this case, the system cannot simply refuse to identify the product, because directly returning "cannot be identified" would interrupt the checkout process and increase customer waiting time. Therefore, this application performs the following medium confidence processing.

[0031] First, the system filters out all product categories from the product database whose similarity reaches a preset second threshold, designating these categories as candidate product categories. The second threshold is a threshold lower than the first threshold, for example, set to 0.80. This threshold represents the lowest similarity level the system considers "potentially correct." Any product category with a similarity below 0.80 means that the query vector and support vector are far apart in semantic space, making it difficult to classify them as the same variety even by human observation (for example, the similarity between apples and oranges may only be 0.3), and therefore they are directly excluded. After filtering out the candidate product categories, the system sorts them in descending order of similarity, generating a queue of categories to be determined. The length of this queue may range from 0 to tens, depending on the size of the product database and the visual uniqueness of the current product. Due to the limited display area of ​​the cash register screen and the need for cashiers to quickly focus their attention, the system cannot display the entire queue of categories to be determined. Therefore, based on a preset number of candidates (e.g., 3 or 5) and the order of the queue, the system retrieves the first N candidate product categories from the head of the queue and sends these categories to a preset target address. In the deployment environment of a community chain supermarket, the preset target address is the screen display buffer of the current cash register. Data sent to this address is formatted as an interactive candidate list pop-up, with each candidate item displaying the product name and unit price, sometimes accompanied by a thumbnail. This is to limit manual confirmation to the most likely items, avoiding cashiers having to search through an excessively long list, thus keeping each manual confirmation time within a few seconds. Then, the system waits to receive initial feedback. This initial feedback refers to the cashier's selection result on the candidate list pop-up. The cashier visually observes the physical product and clicks the correct option from the N candidates displayed on the screen. The system captures this click event and encapsulates it as initial feedback, which includes the identifier of the product category selected by the cashier. Based on the received initial feedback, the system determines the first target product category from the queue of categories to be determined, i.e., the correct product confirmed by the cashier. Afterwards, the system queries the product database for the corresponding product information based on this first target product category and completes the pricing.

[0032] Through this tiered processing, step S2 ensures that even when the algorithm is uncertain, the system can still complete the identification with brief human intervention. This avoids settlement disputes caused by algorithmic misjudgments and is more efficient than completely manually searching for product items. For example, when a customer brings a bag of yams that look between iron yam and white yam to be weighed, the system might give them similarity scores of 0.89 and 0.87 respectively, neither reaching the first threshold of 0.95, but both exceeding the second threshold of 0.80. The system sends these two types of yams as the first two in the queue of products to be identified to the screen. The cashier can then select the correct variety with a glance at the product, requiring only one click. Without this tiered mechanism, the system would either automatically select one incorrectly or directly refuse to identify and force re-entry. The former would lead to disputes, while the latter would significantly reduce checkout efficiency.

[0033] Step S3: When the candidate product category queue does not belong to the preset easily confused product category library, query the product information of the product from the product database according to the first candidate product category in the candidate product category queue.

[0034] After generating and sorting the candidate product category queue, the system needs to determine whether the queue belongs to the preset easily confused product category library. The judgment rule adopted is a composite logic that includes querying the easily confused product category library, calculating similarity difference, and comparing difference thresholds. The introduction of this rule stems from a long-standing problem in fresh food supermarket checkouts: some different varieties of products are highly similar in appearance, such as iron yam and white yam, red onion and purple onion, and their visual feature vectors are very close in high-dimensional semantic space. Even if the similarity between the query vector and the support vector of a certain variety exceeds the first threshold, misidentification may still occur because the support vector of another variety is also close to the threshold. Traditional single threshold methods cannot handle such fine-grained differentiation scenarios. Therefore, step S3 pre-records frequently manually corrected confusion pairs in the history of easily confused product categories through the easily confused product category library and performs additional verification before automatic confirmation, thereby eliminating settlement errors while maintaining high efficiency.

[0035] When the candidate product category queue does not belong to the easily confused product category library, the system directly queries the corresponding product information from the product database based on the first candidate product category in the queue. Not belonging to the easily confused product category library includes the following two situations: First, the first candidate product category does not belong to any entry in the easily confused product category library; second, the first candidate product category belongs to the easily confused product category library, but the similarity difference reaches or exceeds the third threshold in the subsequent difference judgment. In the first case, the system assumes that the product corresponding to the current identification result has no known easily confused objects, and therefore can be safely and automatically confirmed. For example, when a customer brings a regular potato to be weighed, the system calculates that the product with the highest similarity is "potato," with a similarity of 0.98. Since there is no record of "potato" in the easily confused product category library, the system directly queries the unit price based on the "potato" SKU and completes the pricing. In the second case, although the first candidate product belongs to the easily confused product category library, after difference judgment, it is confirmed that its distinction from easily confused objects is sufficiently obvious, and the system also automatically confirms it.

[0036] Before querying product information based on the first candidate product category in the candidate product category queue, the system must execute a specific judgment process for the easily confused category rule. First, the system checks whether the first candidate product category belongs to a preset easily confused category library. The easily confused category library is a data table stored locally on the cash register, where each row records a product pair that has been proven to be frequently confused, such as "Iron Yam - White Jade Yam" and "Red Fuji Apple - Gala Apple". This library is built from manual correction events in historical cash registers, and its update mechanism will be explained later. If the first candidate product category is not in this library, the system considers it not to belong to the easily confused category library and directly proceeds to the product information query step. If the first candidate product category is in the library, the system needs to filter out the second candidate product category belonging to the same easily confused pair from the candidate product category queue based on the easily confused category library and the first candidate product category. Specifically, the system reads another variety that is paired with the first candidate product category from the easily confused category library and then checks whether that variety appears in the candidate product category queue. Since the candidate queue is already sorted from highest to lowest similarity, the second candidate product category is usually located in the second position or later in the queue. After finding the second candidate product category, the system calculates the difference between the similarity of the second candidate product category and the similarity of the first candidate product category. This difference is called the similarity difference. It is calculated by subtracting the second candidate similarity from the first candidate similarity, resulting in a positive number.

[0037] After obtaining the similarity difference, the system compares it with a preset third threshold. The third threshold is a pre-set small value, such as 0.05 (i.e., 5 percentage points). The rationale for setting the third threshold is that when the similarity of two easily confused items is very close, the query vector is roughly located in the middle region of the two support vectors in the high-dimensional semantic space. In this case, even if the similarity of the first candidate is slightly higher than that of the second candidate, it is not enough to guarantee the correct identification result. Conversely, if the similarity of the first candidate is significantly higher than that of the second candidate, for example, more than 0.05 higher, it indicates that the query vector is clearly biased towards the first candidate item, and the system can rely on automatic identification. Therefore, when the similarity difference is less than the preset third threshold, the system determines that the current identification result is in the ambiguous zone and cannot be automatically confirmed. At this time, the system combines the second and first candidate item types into a confirmation list and sends this list to a preset target address, namely the display buffer of the cash register screen. A simplified confirmation window will pop up on the screen, containing only the two candidate buttons. Each button displays the item name and unit price, sometimes with a thumbnail. This design is more efficient than displaying a long list because cashiers only need to choose one of two options and glance at the item to make a decision. The system then waits to receive a second feedback signal from the outside. This second feedback signal is the event signal generated after the cashier clicks the selected button in the pending confirmation list. Based on the received second feedback signal, the system determines the second target product category selected by the cashier from the pending confirmation list, and then retrieves the product information from the product database based on this second target product category and completes the pricing.

[0038] When the similarity difference reaches or exceeds the preset third threshold, even if the first candidate product category belongs to the easily confused category library, the system considers the visual characteristics of the first candidate product to be far stronger than its easily confused counterpart, thus requiring no manual intervention. At this point, the system directly queries product information based on the first candidate product category in the candidate product category queue. For example, when identifying iron yam, the system calculates a similarity of 0.96 for iron yam and 0.89 for white yam, with a difference of 0.07, which is greater than the third threshold of 0.05. Therefore, the system automatically adopts the result for iron yam. In another scenario, a yam with an appearance between the two might generate similarities of 0.92 and 0.90, with a difference of 0.02, less than 0.05. In this case, the system will display a confirmation list for the cashier to choose from.

[0039] In addition to the online identification and decision-making process described above, this method also includes offline or periodic model updates and maintenance of the easily confused database. Based on a preset sample format, the system combines the product information generated during each identification process, the entered images, and the received manual selection events into an incremental learning sample, and labels the product category corresponding to this incremental learning sample. The so-called manual selection events include first feedback information (candidate list selection in medium-confidence scenarios) and second feedback information (two-choice confirmation in easily confused scenarios), as well as implicit confirmation from the cashier after automatic system identification without modification. Each incremental learning sample essentially associates a specific image feature vector with a specific product category label. The system does not update the model immediately after each sample generation. Instead, according to a preset time period, such as after closing each day, it acquires all the incremental learning samples accumulated for each product category within that time period and performs statistical calculations on the features of these samples to update the features corresponding to that product category. Specifically, the statistical calculation method is to calculate the arithmetic mean of the feature vectors of all samples for the same product category to obtain a new support feature vector, and then use this new vector to replace the original feature vector of that category in the product database. In this way, the system continuously absorbs newly emerging individual differences, such as the color changes between summer and winter Fuji apples, making the supporting feature vectors increasingly closer to the visual distribution center of the variety in the real environment, thereby gradually improving the recognition accuracy.

[0040] Regarding the maintenance of the easily confused product category library, when the system receives the second feedback information—that is, when the cashier makes a selection in the easily confused pending confirmation list—a dedicated update process is triggered. Specifically, the system, according to a preset event group format, combines the final confirmed product category indicated by the second feedback information, the first candidate product category, the second candidate product category, and the calculated similarity difference at that time into a confusion event pair. For example, in one event, the first candidate is iron yam, the second candidate is white jade yam, and the cashier ultimately selects white jade yam. The system records a confusion pair (iron yam, white jade yam) and the similarity difference value of 0.02 for that event. The system counts the occurrences of the same confusion event pair within a preset time window (e.g., 7 consecutive days or a cumulative total of 100 identifications). When this occurrence reaches a preset frequency threshold, such as a cumulative total of 5 times, the system updates the easily confused product category library based on this confusion event pair. The update method is to add the confusion pair to the library (if it does not already exist) or increase its weight. Simultaneously, the system needs to calculate the average or minimum similarity difference of the confusion pair across all similarity differences in historical records, and use this average or minimum difference to update the third threshold. For example, if the difference in similarity between iron yam and white yam is generally around 0.03, lower than the original 0.05, the system lowers the third threshold to 0.03, making it more likely to prompt manual confirmation when encountering similar differences in the future, thus avoiding automatic misjudgment. Conversely, if the difference in similarity between a confusion pair is consistently large (e.g., above 0.10) in long-term records, the system can raise the third threshold or remove the pair from the database. Through this adaptive adjustment, the easily confused product category database and the third threshold can dynamically adapt to the actual product distribution and cashier habits of each store, continuously optimizing the human-machine collaboration strategy.

[0041] The following describes the relevant apparatus of the product recognition scheme based on visual recognition provided in the embodiments of this application.

[0042] It should be noted that, in the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program with a predetermined function, which works together with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0043] Please see Figure 2 This is a functional block diagram of a vision-based product recognition device provided in an embodiment of this application. This vision-based product recognition device 2 can be used to perform the functions described in this application. Figure 1The corresponding steps in the vision-based product recognition method provided in the embodiment. Specifically, the vision-based product recognition device 2 may include: The similarity calculation module 21 is used to extract the query features of the product input image and calculate the similarity between the query features and the features corresponding to each product category in the preset product database. The category filtering module 22 is used to filter out product categories whose similarity reaches a preset first threshold from the product database as candidate product categories, and sort the candidate product categories from high to low according to the similarity to generate a candidate product category queue. The product query module 23 is used to query the product information of the goods from the product database according to the first candidate product category in the candidate product category queue when the candidate product category queue does not belong to the preset easily confused product category library.

[0044] In one possible implementation, the similarity calculation module 21 is also used to perform the following operations: Acquire multiple frames of recorded images of the goods, and perform noise reduction and color space conversion on the recorded images to obtain images to be recognized with uniform image size; The image to be identified is subjected to subject region detection and target region segmentation, and the product region in the image to be identified is extracted as the region to be identified. Based on preset visual and semantic feature extraction rules, visual features are extracted and semantic mapping is performed on the region to be identified to obtain the query features corresponding to the goods. Calculate the similarity between the query features and the features corresponding to each product category in the preset product database.

[0045] In one possible implementation, the category filtering module 22 is also used to perform the following operations: Product categories whose similarity reaches a preset second threshold are selected from the product database as candidate product categories, and the candidate product categories are sorted from high to low according to the similarity to generate a queue of categories to be determined; wherein, the second threshold is less than the first threshold; Based on the preset number of candidates and the order of the queue of categories to be determined, the categories of goods to be determined are sent to the preset target address; Based on the received first feedback information, a first target product category is determined from the queue of categories to be determined, so as to query the product information of the product from the product database based on the first target product category.

[0046] In one possible implementation, the visual recognition-based product recognition device 2 further includes a confusion judgment module 24, which is also used to perform the following operations: When the first candidate product category belongs to the easily confused product category library, based on the easily confused product category library and the first candidate product category, a second candidate product category belonging to the easily confused product category library is selected from the candidate product category queue, and the difference between the similarity corresponding to the second candidate product category and the similarity corresponding to the first candidate product category is calculated to obtain the similarity score difference. When the similarity difference is less than a preset third threshold, the second candidate product category and the first candidate product category are used as a confirmation list, and the confirmation list is sent to a preset target address. Based on the received second feedback information, a second target product category is determined from the list to be confirmed, so as to query the product information of the product from the product database according to the second target product category; When the similarity difference reaches a preset third threshold, the product information of the goods is queried from the product database according to the first candidate product category in the candidate product category queue.

[0047] In one possible implementation, the visual recognition-based product recognition device 2 further includes a data update module 25, which is also used to perform the following operations: According to the preset sample format, the product information, the entered image, and the received manual selection event are combined to form an incremental learning sample, and the product category corresponding to the incremental learning sample is labeled. According to a preset time period, incremental learning samples corresponding to each product category are obtained, and statistical calculations of the features of the incremental learning samples are performed to update the features corresponding to the product category.

[0048] In one possible implementation, the data update module 25 is further configured to perform the following operations: According to the preset event group format, the second feedback information, the first candidate product type, the second candidate product type, and the similarity difference are combined into a confusion event pair; The occurrence frequency of the confusion event pairs is counted according to a preset time window. When the occurrence frequency reaches a preset frequency threshold, the easily confused category library is updated according to the confusion event pairs. Calculate the average or minimum similarity difference among the confused event pairs, and update the third threshold using the average or minimum similarity difference.

[0049] It should be understood that the various variations and specific embodiments of the methods provided in the above embodiments are also applicable to the visual recognition-based product identification device of this embodiment. Through the foregoing detailed description of the visual recognition-based product identification method, those skilled in the art can clearly understand the implementation method of the visual recognition-based product identification device of this embodiment. For the sake of brevity, it will not be described in detail here.

[0050] Please see Figure 3 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. The computer device 3 is used to execute the steps performed by the computer device in the aforementioned method embodiments. The computer device 3 may include one or more devices (e.g., a server, node, terminal device, etc.) or internal components (e.g., a chip, software module, or hardware module). The computer device may include at least one processor 31 and a communication interface 32. Further optionally, the computer device may also include at least one memory 33 and a bus 34. Additionally, the processor 31, communication interface 32, and memory 33 are connected via the bus 34. Wherein: (1) The processor 31 is a module that performs arithmetic and / or logical operations. Specifically, it may be one or a combination of processing modules such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor unit (MPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a complex programmable logic device (CPLD), a coprocessor (to assist the central processing unit in completing corresponding processing and applications), and a micro controller unit (MCU).

[0051] (2) The communication interface 32 can be used to provide information input or output to at least one processor 31. And / or, the communication interface 32 can be used to receive data sent from outside and / or send data to outside, and can be a wired link interface including such as an Ethernet cable, or a wireless link interface (Wi-Fi, Bluetooth, general wireless transmission, vehicle short-range communication technology and other short-range wireless communication technologies, etc.). The communication interface 32 can serve as a network interface.

[0052] (3) The memory 33 is used to provide storage space, in which data such as the operating system and computer programs (including program instructions) can be stored. The memory 33 can be one or a combination of random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or compact disc read-only memory (CD-ROM), etc.

[0053] In specific implementation, processor 31 executes the following steps by running the computer program stored in memory 33: Extract query features from the images of goods entered, and calculate the similarity between the query features and the features corresponding to each product category in the preset product database; Product categories whose similarity reaches a preset first threshold are selected from the product database as candidate product categories, and the candidate product categories are sorted from high to low according to the similarity to generate a candidate product category queue. When the candidate product category queue does not belong to the preset easily confused product category library, the product information of the product is queried from the product database according to the first candidate product category in the candidate product category queue.

[0054] In one possible implementation, processor 31 is also used to perform the following operations: Acquire multiple frames of recorded images of the goods, and perform noise reduction and color space conversion on the recorded images to obtain images to be recognized with uniform image size; The image to be identified is subjected to subject region detection and target region segmentation, and the product region in the image to be identified is extracted as the region to be identified. Based on preset visual and semantic feature extraction rules, visual features are extracted and semantic mapping is performed on the region to be identified to obtain the query features corresponding to the goods. Calculate the similarity between the query features and the features corresponding to each product category in the preset product database.

[0055] In one possible implementation, processor 31 is also used to perform the following operations: Product categories whose similarity reaches a preset second threshold are selected from the product database as candidate product categories, and the candidate product categories are sorted from high to low according to the similarity to generate a queue of categories to be determined; wherein, the second threshold is less than the first threshold; Based on the preset number of candidates and the order of the queue of categories to be determined, the categories of goods to be determined are sent to the preset target address; Based on the received first feedback information, a first target product category is determined from the queue of categories to be determined, so as to query the product information of the product from the product database based on the first target product category.

[0056] In one possible implementation, processor 31 is also used to perform the following operations: When the first candidate product category belongs to the easily confused product category library, based on the easily confused product category library and the first candidate product category, a second candidate product category belonging to the easily confused product category library is selected from the candidate product category queue, and the difference between the similarity corresponding to the second candidate product category and the similarity corresponding to the first candidate product category is calculated to obtain the similarity score difference. When the similarity difference is less than a preset third threshold, the second candidate product category and the first candidate product category are used as a confirmation list, and the confirmation list is sent to a preset target address. Based on the received second feedback information, a second target product category is determined from the list to be confirmed, so as to query the product information of the product from the product database according to the second target product category; When the similarity difference reaches a preset third threshold, the product information of the goods is queried from the product database according to the first candidate product category in the candidate product category queue.

[0057] In one possible implementation, processor 31 is also used to perform the following operations: According to the preset sample format, the product information, the entered image, and the received manual selection event are combined to form an incremental learning sample, and the product category corresponding to the incremental learning sample is labeled. According to a preset time period, incremental learning samples corresponding to each product category are obtained, and statistical calculations of the features of the incremental learning samples are performed to update the features corresponding to the product category.

[0058] In one possible implementation, processor 31 is also used to perform the following operations: According to the preset event group format, the second feedback information, the first candidate product type, the second candidate product type, and the similarity difference are combined into a confusion event pair; The occurrence frequency of the confusion event pairs is counted according to a preset time window. When the occurrence frequency reaches a preset frequency threshold, the easily confused category library is updated according to the confusion event pairs. Calculate the average or minimum similarity difference among the confused event pairs, and update the third threshold using the average or minimum similarity difference.

[0059] In one possible implementation, processor 31 is also used to perform the following operations: The similarity calculation module is used to extract query features from the product entry images and calculate the similarity between the query features and the features corresponding to each product category in the preset product database. The category filtering module is used to filter out product categories whose similarity reaches a preset first threshold from the product database as candidate product categories, and sort the candidate product categories from high to low according to the similarity to generate a candidate product category queue. The product query module is used to query the product information of the goods from the product database based on the first candidate product category in the candidate product category queue when the candidate product category queue does not belong to the preset easily confused product category library.

[0060] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially as a computer program product, which includes one or more computer programs. When the computer program is loaded and executed on a computer device, it generates, in whole or in part, the processes or functions described in the embodiments of this application; the computer device can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer program can be stored in or transmitted through a computer-readable storage medium; the computer program can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium accessible to the computer device or a data processing device such as a server or data center that integrates one or more available media; wherein, the available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0061] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, any equivalent variations made in accordance with the claims of this application shall still fall within the scope of this application.

Claims

1. A product recognition method based on visual recognition, characterized in that, The method includes: Extract query features from the images of goods entered, and calculate the similarity between the query features and the features corresponding to each product category in the preset product database; Product categories whose similarity reaches a preset first threshold are selected from the product database as candidate product categories, and the candidate product categories are sorted from high to low according to the similarity to generate a candidate product category queue. When the candidate product category queue does not belong to the preset easily confused product category library, the product information of the product is queried from the product database according to the first candidate product category in the candidate product category queue.

2. The product identification method based on visual recognition according to claim 1, characterized in that, The process of extracting query features from the product entry images and calculating the similarity between the query features and the features corresponding to each product category in the preset product database includes: Acquire multiple frames of recorded images of the goods, and perform noise reduction and color space conversion on the recorded images to obtain images to be recognized with uniform image size; The image to be identified is subjected to subject region detection and target region segmentation, and the product region in the image to be identified is extracted as the region to be identified. Based on preset visual and semantic feature extraction rules, visual features are extracted and semantic mapping is performed on the region to be identified to obtain the query features corresponding to the goods. Calculate the similarity between the query features and the features corresponding to each product category in the preset product database.

3. The product identification method based on visual recognition according to claim 1, characterized in that, When none of the similarities reach the first threshold, the method further includes: Product categories whose similarity reaches a preset second threshold are selected from the product database as candidate product categories, and the candidate product categories are sorted from high to low according to the similarity to generate a queue of categories to be determined; wherein, the second threshold is less than the first threshold; Based on the preset number of candidates and the order of the queue of categories to be determined, the categories of goods to be determined are sent to the preset target address; Based on the received first feedback information, a first target product category is determined from the queue of categories to be determined, so as to query the product information of the product from the product database based on the first target product category.

4. The product identification method based on visual recognition according to claim 1, characterized in that, After generating the candidate product category queue, the method further includes: When the first candidate product category belongs to the easily confused product category library, based on the easily confused product category library and the first candidate product category, a second candidate product category belonging to the easily confused product category library is selected from the candidate product category queue, and the difference between the similarity corresponding to the second candidate product category and the similarity corresponding to the first candidate product category is calculated to obtain the similarity score difference. When the similarity difference is less than a preset third threshold, the second candidate product category and the first candidate product category are used as a confirmation list, and the confirmation list is sent to a preset target address. Based on the received second feedback information, a second target product category is determined from the list to be confirmed, so as to query the product information of the product from the product database according to the second target product category; When the similarity difference reaches a preset third threshold, the product information of the goods is queried from the product database according to the first candidate product category in the candidate product category queue.

5. The product identification method based on visual recognition according to claim 1, characterized in that, The method further includes: According to the preset sample format, the product information, the entered image, and the received manual selection event are combined to form an incremental learning sample, and the product category corresponding to the incremental learning sample is labeled. According to a preset time period, incremental learning samples corresponding to each product category are obtained, and statistical calculations of the features of the incremental learning samples are performed to update the features corresponding to the product category.

6. The product identification method based on visual recognition according to claim 4, characterized in that, The method for obtaining the second feedback information further includes: According to the preset event group format, the second feedback information, the first candidate product type, the second candidate product type, and the similarity difference are combined into a confusion event pair; The occurrence frequency of the confusion event pairs is counted according to a preset time window. When the occurrence frequency reaches a preset frequency threshold, the easily confused category library is updated according to the confusion event pairs. Calculate the average or minimum similarity difference among the confused event pairs, and update the third threshold using the average or minimum similarity difference.

7. A product identification device based on visual recognition, applied to the product identification method based on visual recognition as described in claim 1, characterized in that, The device includes: The similarity calculation module is used to extract query features from the product entry images and calculate the similarity between the query features and the features corresponding to each product category in the preset product database. The category filtering module is used to filter out product categories whose similarity reaches a preset first threshold from the product database as candidate product categories, and sort the candidate product categories from high to low according to the similarity to generate a candidate product category queue. The product query module is used to query the product information of the goods from the product database based on the first candidate product category in the candidate product category queue when the candidate product category queue does not belong to the preset easily confused product category library.

8. A computer device, characterized in that, include: Memory and processor: A memory, wherein one or more computer programs are stored; A processor is configured to load one or more computer programs to implement the visual recognition-based product identification method as described in any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the visual recognition-based commodity recognition method according to any one of claims 1 to 6.

Images