Product identification methods, devices, equipment, media and procedures

Abstract

This invention relates to the field of intelligent computer vision technology, providing a product recognition method, apparatus, device, medium, and program product. The method includes acquiring a set of region images corresponding to a shelf image and a set of text nouns corresponding to product text information; mapping the region images in the region image set and the text nouns in the text noun set to a shared semantic space based on a semantic alignment network; correcting the semantic distance between the region images and text nouns through pre-constructed retail prior constraints to obtain a distance matrix; and performing global bipartite graph matching on the region image set and text noun set based on the distance matrix, allowing some region images or some text nouns to be unmatched during the matching process, thereby obtaining the product recognition result for the shelf image. This invention corrects the semantic distance based on retail prior constraints in the shared semantic space and utilizes global bipartite graph matching that allows unmatched states for optimal one-to-one allocation, achieving open category recognition of shelf products.

Description

Technical Field

[0001] This invention relates to the field of intelligent computer vision technology, and in particular to a product recognition method, apparatus, equipment, medium, and program product. Background Technology

[0002] In smart retail operations, real-time identification and management of shelf merchandise is one of the core requirements. On the one hand, retail stores have a large number of merchandise, and new products are frequently added to the shelves. Traditional object recognition models rely on a large amount of manually labeled product image data for training, and the recognition accuracy drops significantly when faced with unlabeled new products. On the other hand, stores need to regularly check shelf displays and conduct dynamic inventory counts, and manual operations are inefficient and prone to errors.

[0003] Currently, product identification typically employs detection mechanisms based on open lexicons. However, these mechanisms primarily rely on basic cross-modal semantic similarity metrics. In smart retail scenarios, there are often numerous products from the same product series that are highly similar in appearance or have extremely similar textual descriptions. Relying solely on semantic dimensions for measurement easily leads to confusion between similar products, resulting in a significant drop in recognition accuracy. Secondly, when filtering and assigning targets, current matching strategies struggle to coordinate overall correspondences in the complex environment of densely packed multi-category products on retail shelves. This easily leads to multiple products competing for the same label, or the omission or incorrect selection of irrelevant products. Furthermore, traditional methods typically assign all detected image targets to existing text labels when outputting the final result. However, in dynamic retail operations, new products without timely description text entries or products with labels that are out of stock often appear on shelves. Therefore, these methods cannot truly adapt to the open-category and frequently changing real-world retail scenarios. Summary of the Invention

[0004] This invention provides a product identification method, apparatus, equipment, medium, and program product to solve the technical problems existing in current product identification methods.

[0005] This invention provides a product identification method, comprising the following steps: Obtain shelf images and product text information; Obtain the region image set corresponding to the shelf image and the text noun set corresponding to the product text information; Based on the semantic alignment network, the regional images in the regional image set and the text nouns in the text noun set are mapped to a shared semantic space, and the semantic distance between the regional images and the text nouns is corrected by a pre-constructed retail prior constraint to obtain a distance matrix; Based on the distance matrix, global bipartite graph matching is performed on the set of regional images and the set of text nouns. During the matching process, some of the regional images or some of the text nouns are allowed to be in an unmatched state, so as to obtain the product recognition result of the shelf image.

[0006] According to the product recognition method provided by the present invention, the pre-constructed retail prior constraints include product location layout constraints and product physical size constraints; the step of correcting the semantic distance between the region image and the text noun through the pre-constructed retail prior constraints to obtain a distance matrix includes: Extract the visual feature vector of the region image and the text feature vector of the text noun, and determine the initial semantic distance between the visual feature vector and the text feature vector; Based on the product location layout constraints and the spatial location information of the region image, the layout penalty distance between the region image and the text noun is determined; Based on the physical size constraints of the product and the size feature information of the region image, the size penalty distance between the region image and the text noun is determined; Based on the initial semantic distance, the layout penalty distance, and the size penalty distance, the corrected distance between the region image and the text noun is determined; A distance matrix is ​​obtained based on the corrected distances of each data pair in the region image set and the text noun set.

[0007] According to the product recognition method provided by the present invention, the step of extracting the visual feature vector of the region image and the text feature vector of the text noun includes: The region image is input into a pre-trained visual coding network to extract the visual feature vector of the region image; The text nouns are input into a pre-trained text encoding network to extract the text feature vectors of the text nouns.

[0008] According to the product recognition method provided by the present invention, the step of performing global bipartite graph matching on the region image set and the text noun set based on the distance matrix, and allowing some region images or some text nouns to be in an unmatched state during the matching process, to obtain the product recognition result of the shelf image, includes: The region image set and the text noun set are respectively used as two node sets of a bipartite graph, and the connection weights between corresponding nodes are determined based on the distance matrix; A global mapping assignment is performed on the bipartite graph to obtain a mapping relationship that includes matching node pairs, unmatched region image nodes, and unmatched text noun nodes; wherein, when the connection weight between nodes is greater than a preset weight threshold, the corresponding node is determined to be in an unmatched state. The unmatched region image nodes are marked as unknown objects, and the unmatched text noun nodes are marked as not displayed. Based on the matching node pairs, the region image nodes marked as unknown objects, and the text noun nodes marked as not displayed, the product recognition result of the shelf image is obtained.

[0009] According to the product recognition method provided by the present invention, after obtaining the mapping relationship including matching node pairs, unmatched region image nodes, and unmatched text noun nodes, the method further includes: Obtain the reference text information associated with the region image nodes in the matching node pair; Determine the text similarity between the reference text information and the text noun nodes in the matching node pair; Based on the text similarity and the connection weights of the matching node pairs, a comprehensive confidence score for the matching node pairs is determined. When the overall confidence score is not greater than a preset score threshold, the matching node pair is marked as pending review.

[0010] According to the product identification method provided by the present invention, after marking the unmatched region image nodes as unknown objects, the method further includes: Obtain the supplementary text information corresponding to the image nodes in the unmatched region; Incremental alignment samples are constructed based on the region images corresponding to the unmatched region image nodes and the supplementary text information. The semantic alignment network is updated based on the incremental alignment samples.

[0011] According to the product identification method provided by the present invention, the region images in the region image set are obtained in the following manner: The visual feature map of the shelf image is extracted, and the visual feature map is subjected to feature response enhancement processing to obtain an enhanced feature map; The enhanced feature map is subjected to instance segmentation processing to obtain at least one instance mask and the bounding box corresponding to the instance mask; Obtain the coordinate information of the bounding box in the shelf image, and map the coordinate information into spatial location information; Based on the instance mask, the bounding box corresponding to the instance mask, and the spatial location information, the corresponding region image is extracted from the shelf image.

[0012] According to the product identification method provided by the present invention, the text nouns in the text noun set are obtained in the following manner: Semantic features are extracted from the product text information to obtain at least one product entity name and product attribute information matching the product entity name; Based on the dependency relationship between the product entity name and the product attribute information, and a preset text reconstruction template, the corresponding text name is generated.

[0013] The present invention also provides a product identification device, comprising: The first acquisition module is used to acquire shelf images and product text information; The second acquisition module is used to acquire the regional image set corresponding to the shelf image and the text noun set corresponding to the product text information; The semantic alignment module is used to map the regional images in the regional image set and the text nouns in the text noun set to a shared semantic space based on the semantic alignment network, and to correct the semantic distance between the regional images and the text nouns through a pre-constructed retail prior constraint to obtain a distance matrix; The recognition module is used to perform global bipartite graph matching on the set of regional images and the set of text nouns based on the distance matrix, and allows some of the regional images or some of the text nouns to be in an unmatched state during the matching process, so as to obtain the product recognition result of the shelf image.

[0014] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the program to implement the product identification method as described above.

[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the commodity identification method as described above.

[0016] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements any of the above-described product identification methods.

[0017] The product identification method, apparatus, device, medium, and program products provided by this invention first acquire the set of regional images corresponding to shelf images and the set of text nouns corresponding to product text information. Based on this, a semantic alignment network maps the regional images and text nouns to a shared semantic space, and a distance matrix is ​​obtained by correcting the semantic distance between the two through pre-constructed retail prior constraints. This introduction of retail prior constraints improves the distinguishability between similar products and enhances the accuracy of cross-modal feature alignment. Finally, global bipartite graph matching is performed on the regional image set and text noun set based on this distance matrix. During the matching process, some regional images or some text nouns are allowed to remain unmatched, achieving optimal one-to-one mapping of products under a global perspective. This not only avoids local matching conflicts and allocation chaos in densely displayed environments but also adapts to the frequent occurrence of new product launches and stockouts in dynamic retail operations, ensuring high accuracy and robustness of product identification in open category scenarios. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the present invention or the conventional ones, the drawings used in the embodiments or conventional descriptions will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the product identification method provided in an embodiment of the present invention.

[0020] Figure 2 This is a schematic diagram of the product identification device provided in an embodiment of the present invention.

[0021] Figure 3 This is a schematic diagram of the structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0023] It should be noted that in the description of this invention, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. The terms "upper," "lower," etc., indicating orientation or positional relationships based on the orientation or positional relationships shown in the accompanying drawings, are only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the system or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0024] The terms "first," "second," etc., used in this invention are used to distinguish similar objects, not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class, without limiting the number of objects; for example, a first object can be one or more. Furthermore, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0025] Figure 1 This is a flowchart illustrating the product identification method provided in an embodiment of the present invention, as shown below. Figure 1 As shown, the method includes the following steps 110, 120, 130 and 140.

[0026] Step 110: Obtain the shelf image and product text information.

[0027] Shelf images refer to image data obtained by photographing shelves in a retail store. These can be static images captured by cameras fixed to the shelf area, images manually taken by users using mobile devices, or images automatically collected by autonomous inspection equipment within the shelf area. Shelf images can cover a single row of shelves or panoramic images covering multiple rows of shelves; this invention does not limit this. In some embodiments, shelf images may include tilted shots, images with obstructions, or images affected by changes in lighting, to adapt to the complex shooting conditions in a real retail store.

[0028] Product text information refers to the textual descriptions of products on store shelves, including but not limited to: store inventory lists, price tag text extracted from shelf price tags using optical character recognition technology, title text of product detail pages on e-commerce platforms, and product names, specifications, and other descriptive information entered by store staff when new products are listed. Product text information can originate from a single channel or be a fusion of text data from multiple channels.

[0029] Step 120: Obtain the set of regional images corresponding to the shelf image and the set of text nouns corresponding to the product text information.

[0030] A region image set refers to the collection of images from which candidate regions for individual product instances are extracted after processing a shelf image. Each region image in the set corresponds to a candidate product instance region in the shelf image, carrying visual information about that instance and its spatial location on the shelf. The region images in the set do not carry predefined product category labels to support the extraction of region images for any product on the shelf, such as new products not present in the training set.

[0031] A text noun set refers to a collection of text descriptions of products extracted in a standardized form after semantic processing of product text information. Each text noun in the text noun set corresponds to a semantic tag of a product, and usually includes a combination of attributes such as the product's brand, category, specifications, and packaging.

[0032] In this embodiment, a set of regional images can be obtained based on instance-level region decomposition of shelf images, and a set of text nouns can be obtained based on semantic unit extraction of product text information.

[0033] Step 130: Based on the semantic alignment network, map the regional images in the regional image set and the text nouns in the text noun set to the shared semantic space, and correct the semantic distance between the regional images and the text nouns through a pre-constructed retail prior constraint to obtain a distance matrix.

[0034] It should be understood that semantic alignment networks refer to neural network models that can jointly encode image information and text information and map them to the same semantic vector space, so that in this shared semantic space, the feature vectors of semantically related regions are closer to the feature vectors of text nouns, while the feature vectors of semantically unrelated regions are farther away from the feature vectors of text nouns.

[0035] Specifically, through the semantic alignment network, both region images and text nouns are represented as fixed-dimensional feature vectors in a unified semantic space, thereby quantifying the semantic correlation between region images and text nouns through the distance metric between vectors.

[0036] It should be noted that retail prior constraints refer to the constraint relationships constructed using known product display patterns and physical characteristics in retail business scenarios. These constraints are used to correct the visual-text semantic distance and introduce constraint information at the business logic level to improve the differentiation accuracy between products with similar appearances.

[0037] A distance matrix is ​​a data structure that records the semantic distance between each region image in a region image set and each text noun in a text noun set, after being corrected by retail prior constraints. The rows of the distance matrix correspond to each region image in the region image set, and the columns correspond to each text noun in the text noun set. Each element in the matrix represents the corrected distance between the corresponding region image and the text noun.

[0038] Step 140: Based on the distance matrix, perform global bipartite graph matching on the set of regional images and the set of text nouns, and allow some of the regional images or some of the text nouns to be in an unmatched state during the matching process, so as to obtain the product recognition result of the shelf image.

[0039] Here, global bipartite graph matching refers to the process of taking the region image set and the text noun set as two node sets in a bipartite graph, using the modified semantic distance in the distance matrix as the connection weight between each node pair, and using a global optimal allocation algorithm to uniformly solve the one-to-one mapping scheme with the optimal connection weight between the two node sets in the whole scope.

[0040] It should be noted that, unlike the greedy local matching method that compares one by one, global bipartite graph matching solves the optimal mapping relationship uniformly within the entire region image and the entire text nouns. This can effectively avoid the overall matching deviation caused by local optima and is suitable for situations where multiple products are densely displayed on shelves.

[0041] In this embodiment, during the matching process, some image regions or some text terms are allowed to remain unmatched, i.e., empty matching is allowed. For image regions that fail to match any text terms, they are marked as unknown objects, indicating that there are products in that shelf area that have not yet been registered in the product text information; for text terms that fail to match any image region, they are marked as not displayed, indicating that the corresponding product has not been detected in the current shelf image.

[0042] Based on this, the empty matching mechanism eliminates the need to pre-cover all possible product categories in the product identification process, thus supporting product identification needs under open categories.

[0043] Ultimately, the product identification results include, but are not limited to: the spatial location information of each matched product on the shelf and its corresponding semantic label; the image location information of areas marked as unknown objects; and a list of text nouns marked as not displayed.

[0044] The product recognition method provided in this invention first acquires the set of regional images corresponding to the shelf images and the set of text nouns corresponding to the product text information. Based on this, a semantic alignment network maps the regional images and text nouns to a shared semantic space. A pre-constructed retail prior constraint corrects the semantic distance between the two to obtain a distance matrix. This introduction of retail prior constraints improves the distinguishability between similar products and enhances the accuracy of cross-modal feature alignment. Finally, based on this distance matrix, global bipartite graph matching is performed on the regional image set and text noun set. During the matching process, some regional images or some text nouns are allowed to remain unmatched, achieving optimal one-to-one mapping of products under a global perspective. This not only avoids local matching conflicts and allocation chaos in densely displayed environments but also adapts to the frequent occurrence of new product launches and stockouts in dynamic retail operations, ensuring high accuracy and robustness of product recognition in open category scenarios.

[0045] In some embodiments, the pre-built retail prior constraints include product location layout constraints and product physical size constraints; the step of correcting the semantic distance between the region image and the text noun using the pre-built retail prior constraints to obtain a distance matrix includes: Extract the visual feature vector of the region image and the text feature vector of the text noun, and determine the initial semantic distance between the visual feature vector and the text feature vector; Based on the product location layout constraints and the spatial location information of the region image, the layout penalty distance between the region image and the text noun is determined; Based on the physical size constraints of the product and the size feature information of the region image, the size penalty distance between the region image and the text noun is determined; Based on the initial semantic distance, the layout penalty distance, and the size penalty distance, the corrected distance between the region image and the text noun is determined; A distance matrix is ​​obtained based on the corrected distances of each data pair in the region image set and the text noun set.

[0046] Here, the visual feature vector refers to the dense vector representation of the semantic content of the region image obtained after inputting the region image into the image encoding part of the semantic alignment network and performing encoding processing. The visual feature vector and the text feature vector reside in the same shared semantic space. The text feature vector refers to the dense vector representation of the semantic content of the text noun obtained after inputting the text noun into the text encoding part of the semantic alignment network and performing encoding processing.

[0047] Initial semantic distance refers to the distance between the visual feature vector and the text feature vector of a specific region image and text noun pair in a shared semantic space.

[0048] In some embodiments, the initial semantic distance can be calculated using cosine distance, which is 1 minus the normalized inner product of the visual feature vector and the text feature vector. The smaller the initial semantic distance, the closer the image of the region and the text noun are in the shared semantic space, and the higher the probability of a match between them.

[0049] In this embodiment, the pre-constructed retail prior constraints include product location layout constraints and product physical size constraints.

[0050] The product location layout constraint is a pre-defined constraint based on the patterns of retail display operations, establishing a priori association between product categories and shelf locations. This constraint can be implemented as a location-category association matrix. In this matrix, rows correspond to shelf levels (e.g., levels 1 to N), columns correspond to product category sets (e.g., carbonated beverages, bottled water, puffed snacks, dairy products, daily necessities, etc.), and matrix elements represent the priori association weight of a product category displayed at a corresponding shelf location, ranging from [0, 1]. When the priori association weight is close to 1, it indicates that the product category is highly likely to be displayed at that shelf location; when the priori association weight is close to 0, it indicates that the product category is almost never displayed at that shelf location. The location-category association matrix can be initialized using historical inventory data from retail stores or standard display specifications and can be dynamically updated based on the specific store's display structure.

[0051] Spatial location information refers to the relative position of the product instance corresponding to the region image on the shelf, such as the structured shelf coordinate description of the third cell on the left side of the second shelf.

[0052] In this embodiment, for a given pair of regional images and text terms, the prior association weight of the category in the shelf position corresponding to the current regional image is first queried from the location-category association matrix based on the category attribute information contained in the text term. Then, the layout penalty distance is defined as the difference between 1 and the prior association weight. When the relative position information of the product instance corresponding to the regional image on the shelf is consistent with the high-probability display position of the category corresponding to the text term, the prior association weight is larger, the layout penalty distance is smaller, and the contribution to the increment of the correction distance is small. When the two are inconsistent, the prior association weight is smaller, the layout penalty distance is larger, and a more significant penalty is imposed on the correction distance.

[0053] The physical dimension constraints of a product are linked to the product descriptions in the text noun set. This includes geometric parameters such as the approximate height, width, or volume of the product packaging in physical space. For example, a 200ml carton of milk is approximately 15cm high, and a 500ml bottle of mineral water is approximately 20cm high. These physical dimension constraints can be extracted from product detail pages on e-commerce platforms, product standard specifications databases, or product packaging labeling information, and are maintained in the product information management system during new product entry.

[0054] Size feature information is geometric measurement information extracted from the instance mask or minimum bounding rectangle corresponding to the region image, including the pixel height and pixel width of the region image in the shelf image coordinate system.

[0055] For a given pair of region images and text terms, the size feature information of the region image is compared with the reference height in the physical size constraint of the product corresponding to the text term. The size penalty distance is defined as the normalized absolute difference between the two. When the size features of the region image match the prior height of the packaging size of the text term, the size penalty distance approaches 0; when the difference is significant, the size penalty distance approaches 1, imposing a more significant penalty on the correction distance.

[0056] After obtaining the initial semantic distance, layout penalty distance, and size penalty distance, the corrected distance can be calculated using the following weighted summation method: Corrected distance = Initial semantic distance + λ1 × Layout penalty distance + λ2 × Size penalty distance; Wherein, λ1 and λ2 are preset weight hyperparameters, which are used to adjust the weight ratio of layout prior constraints and commodity physical size constraints in the correction distance calculation, respectively. They can be adjusted according to the reliability of the prior information of display rules in specific retail scenarios. The specific values ​​of λ1 and λ2 are not limited in this embodiment of the invention.

[0057] Finally, the above processing procedure is performed on each region image in the region image set and each text noun in the text noun set to obtain the corrected distances corresponding to all region images and text nouns. These distances are then arranged in a matrix according to the row-column correspondence to obtain the distance matrix.

[0058] The product recognition method provided in this invention introduces layout penalty distance and size penalty distance on the basis of initial semantic distance. It incorporates the prior knowledge of product display location and packaging physical size in the retail scenario as soft constraints into the distance metric, thereby achieving effective differentiation of similar-looking products that are difficult to distinguish when relying solely on visual-text semantic similarity. This reduces the false matching rate. At the same time, since the above-mentioned prior constraints are superimposed in the form of penalty terms rather than mandatory constraints, they do not affect the open matching capability for products of unknown categories.

[0059] In some embodiments, extracting the visual feature vector of the region image and the text feature vector of the text noun includes: The region image is input into a pre-trained visual coding network to extract the visual feature vector of the region image; The text nouns are input into a pre-trained text encoding network to extract the text feature vectors of the text nouns.

[0060] In this embodiment, the region image is input into a pre-trained visual coding network, which extracts and encodes features from the region image and outputs the corresponding visual feature vector.

[0061] Visual coding networks are neural networks that transform image inputs into dense semantic feature vectors. Their specific structure is not limited and can be convolutional neural networks, visual Transformer networks, or visual coding branches trained based on the vision-language pre-training paradigm.

[0062] Visual encoding networks need to be pre-trained before use. The training data is unrestricted and can include general image-text pairs or domain data consisting of retail product images and product description text. This enhances the visual encoding network's ability to encode retail-specific visual features such as product packaging colors, brand logos, and material textures, thereby enabling the extracted visual feature vectors to more accurately represent the semantic information of the product.

[0063] Similarly, text nouns are input into a pre-trained text encoding network, which performs semantic encoding on the text nouns and outputs the corresponding text feature vectors.

[0064] Text encoding networks are neural networks that transform text input into dense semantic feature vectors. Their specific structure is not restricted; they can be language encoders based on the Transformer architecture or text encoding branches trained based on the vision-language pre-training paradigm.

[0065] In one implementation, the visual encoding network and the text encoding network can employ corresponding visual encoding branches and text encoding branches from the same visual-language pre-trained model, respectively. This ensures that the two types of feature vectors reside in the same shared semantic space, guaranteeing the effectiveness of subsequent semantic distance calculations. The text encoding network also requires pre-training, and its training data can include various text corpora such as product titles, price tags, and store lists from the retail sector, to improve the semantic encoding capability for textual descriptions in retail scenarios.

[0066] The product recognition method provided in this invention independently encodes region images and text terms using pre-trained visual encoding networks and text encoding networks, mapping information from two heterogeneous modalities to the same shared semantic space, enabling effective comparison of semantic information between the visual and text modalities through feature vectors.

[0067] In some embodiments, the step of performing global bipartite graph matching on the region image set and the text noun set based on the distance matrix, and allowing some region images or some text nouns to be in an unmatched state during the matching process, to obtain the product recognition result of the shelf image, includes: The region image set and the text noun set are respectively used as two node sets of a bipartite graph, and the connection weights between corresponding nodes are determined based on the distance matrix; A global mapping assignment is performed on the bipartite graph to obtain a mapping relationship that includes matching node pairs, unmatched region image nodes, and unmatched text noun nodes; wherein, when the connection weight between nodes is greater than a preset weight threshold, the corresponding node is determined to be in an unmatched state. The unmatched region image nodes are marked as unknown objects, and the unmatched text noun nodes are marked as not displayed. Based on the matching node pairs, the region image nodes marked as unknown objects, and the text noun nodes marked as not displayed, the product recognition result of the shelf image is obtained.

[0068] Specifically, each region image in the region image set is used as a binary segment. Figure 1 The set of nodes on one side is the set of each text noun in the set of text nouns as the set of nodes on the other side of the bipartite graph. There are no internal connections between the two sets of nodes, but there are candidate connecting edges between the nodes on both sides.

[0069] For any edge connecting any region image node in the region image set and any text noun node in the text noun set, the corrected distance value of the corresponding element in the distance matrix is ​​used as the connection weight of the edge. The larger the corrected distance, the higher the matching cost of the connection, that is, the less likely the corresponding node pair is to be a correct matching relationship.

[0070] After constructing the weighted bipartite graph, it is globally mapped and assigned. The optimal one-to-one matching scheme that minimizes the total matching cost is solved globally. Each region image node is matched with at most one text noun node, and each text noun node is matched with at most one region image node.

[0071] During the global mapping assignment process, some nodes are allowed to remain unmatched in the final mapping relationship. That is, when the connection weight of a node pair is greater than a preset weight threshold, it indicates that the corrected distance between the data pairs exceeds the acceptable matching threshold, and the two do not have sufficient semantic matching possibility. In this case, the corresponding nodes are judged to be unmatched, and no matching relationship is established between the node pairs. Here, the preset weight threshold can be adjusted according to the actual scenario.

[0072] After the global mapping assignment is completed, the resulting mapping relationship contains three types of results: node pairs that have established a matching relationship, region image nodes that have not established a matching relationship with any text noun nodes, and text noun nodes that have not established a matching relationship with any region image nodes.

[0073] In this embodiment, an improved Hungarian algorithm can be used to implement the above global mapping allocation process, that is, allowing node pairs whose cost exceeds a preset weight threshold not to establish a matching relationship.

[0074] It should be understood that for unmatched area image nodes, it indicates that there are visually visible product instances in the shelf image, but there is no text noun in the current product text information that matches its semantics. The product may be a new product that has not yet been entered into the product information system, a temporarily listed product, or other unknown products. In this case, such area images will be marked as unknown objects, and the recognition results can prompt the operator to supplement the text information of the corresponding product.

[0075] In one example, after marking the unmatched region image nodes as an unknown object state, the method further includes: Obtain the supplementary text information corresponding to the image nodes in the unmatched region; Incremental alignment samples are constructed based on the region images corresponding to the unmatched region image nodes and the supplementary text information. The semantic alignment network is updated based on the incremental alignment samples.

[0076] It should be understood that supplementary text information refers to textual information used to describe the product category or attribute corresponding to the image in that area. Supplementary text information can be obtained in various ways, such as: users entering the product's name, specifications, and other descriptive information via a terminal; or by scanning the product's packaging barcode and querying a product database to obtain the corresponding product description information; or by querying an e-commerce platform's product information database and retrieving the corresponding product description information based on image similarity. The specific method of acquisition is not limited.

[0077] Incremental alignment samples are image-text pairing samples consisting of the region image corresponding to the unmatched region image node and supplementary text information. They are used to supplement the semantic alignment network with the alignment relationship of the unknown category of goods in the visual-semantic shared space.

[0078] Specifically, the semantic alignment network is updated based on the constructed incremental alignment samples. In one implementation, the same contrastive learning loss function as the original training phase of the semantic alignment network can be used. Each image-text pair in the incremental alignment sample set is used as a positive sample, and the remaining image-text pairs in the current batch are used as negative samples. This fine-tunes the parameters of the visual encoding network and the text encoding network of the semantic alignment network, reducing the distance between the visual features of the new product area image and the text features of the corresponding product text noun in the shared semantic space. This allows the updated semantic alignment network to correctly identify the new product in the next shelf image processing. After the update, the standardized text nouns generated from the corresponding supplementary text information are synchronously stored in the system's product text noun database for use in subsequent recognition tasks.

[0079] Similarly, for unmatched text noun nodes, it indicates that the product is recorded in the product text information, but no area image matching its visual semantics is found in the shelf image. The product may not be displayed on the current shelf. Such text nouns are marked as not displayed, and the corresponding product name can be further recorded in the out-of-stock warning list to prompt the shelf to replenish stock.

[0080] Finally, the identification information of the matching node pairs, the regional image nodes marked as unknown objects, and the text noun nodes marked as not displayed are all combined to obtain the complete identification result of the goods in the shelf image.

[0081] In one example, the recognition results include, but are not limited to: for matching node pairs, the spatial location information of the product in the shelf image and its corresponding text noun node; for region image nodes with unknown object status, the spatial location information of the product instance in the shelf image and the unknown object marker; for text noun nodes with no display status, the corresponding product name and out-of-stock warning marker.

[0082] The product identification method provided in this invention constructs a weighted bipartite graph by combining a set of regional images and a set of text nouns, and performs global mapping allocation based on a distance matrix that allows for unmatched states. This achieves globally optimal identification and matching of shelf products, avoiding the global suboptimal problem caused by greedy local matching. At the same time, it automatically identifies unknown and out-of-stock products by using a semantic tagging mechanism for unmatched states, providing structured and complete data support for operational scenarios such as retail store shelf display inspection, dynamic inventory counting, and out-of-stock warning.

[0083] In some embodiments, after obtaining the mapping relationship including matching node pairs, unmatched region image nodes, and unmatched text noun nodes, the method further includes: Obtain the reference text information associated with the region image nodes in the matching node pair; Determine the text similarity between the reference text information and the text noun nodes in the matching node pair; Based on the text similarity and the connection weights of the matching node pairs, a comprehensive confidence score for the matching node pairs is determined. When the overall confidence score is not greater than a preset score threshold, the matching node pair is marked as pending review.

[0084] In this embodiment, for a pair of matching nodes that have established a matching relationship, reference text information spatially associated with the region image in the pair of matching nodes is further obtained.

[0085] Here, reference text information refers to textual content that provides a description of the product and is spatially adjacent to the image of the area in the shelf image. Its specific form is not limited. For example, it can be the price tag text located below or near the product area, which can be obtained by OCR processing of the shelf image; or it can be the real-time display content of the electronic price tag at the corresponding position on the shelf, etc.

[0086] Text similarity is a quantitative indicator that measures the degree to which reference text information and text nouns match at the literal description level or semantic level. The value range is [0, 1], and the larger the value, the closer the two are.

[0087] In one implementation, reference text information and text nouns can be input into a pre-trained text encoding network to extract their text feature vectors, and the cosine similarity between the two text feature vectors can be calculated to obtain the text similarity.

[0088] The comprehensive confidence score is an indicator that comprehensively and quantitatively evaluates the reliability of a single matching node pair by combining cross-modal visual-semantic matching results and price tag text verification results. The value range is [0,1], and the larger the value, the more reliable the matching node pair is.

[0089] In this embodiment, the connection weights determined based on the distance matrix are transformed into cross-modal semantic similarity, that is, cross-modal semantic similarity is defined as the difference between 1 and the connection weights. Based on this, the text similarity and cross-modal semantic similarity are weighted and summed to obtain a comprehensive confidence score.

[0090] When the overall confidence score of the matched node pair is higher than the preset score threshold, it indicates that the matching result has high reliability after joint verification of two dimensions: cross-modal semantic similarity and price tag text consistency. It can be directly output and used for downstream retail operation tasks such as shelf display inspection, dynamic inventory counting, and price tag verification.

[0091] When the overall confidence score of the matching node pair is not greater than the preset score threshold, it indicates that the matching result has a large degree of uncertainty. The corresponding matching node pair is marked as pending review and enters the manual review queue. Store operations personnel will manually confirm the correspondence between the image and the text noun in this area. The confirmation result can be further used to optimize the training data of the product recognition model.

[0092] The product recognition method provided in this embodiment of the invention further obtains reference text information associated with the spatial location of the matching region after global bipartite graph matching is completed, performs text-level consistency verification with the corresponding text nouns, constructs a comprehensive confidence score by combining cross-modal semantic similarity, and filters out low-confidence matching results that require manual review by using a preset scoring threshold, thus realizing dual quality control of cross-modal visual-semantic matching and text cross-validation.

[0093] In some embodiments, the region images in the region image set are obtained in the following manner: The visual feature map of the shelf image is extracted, and the visual feature map is subjected to feature response enhancement processing to obtain an enhanced feature map; The enhanced feature map is subjected to instance segmentation processing to obtain at least one instance mask and the bounding box corresponding to the instance mask; Obtain the coordinate information of the bounding box in the shelf image, and map the coordinate information into spatial location information; Based on the instance mask, the bounding box corresponding to the instance mask, and the spatial location information, the corresponding region image is extracted from the shelf image.

[0094] Visual feature maps are multi-scale feature representations obtained by performing feature extraction operations on shelf images. They contain everything from shallow, low-level visual information such as color and texture to deep, high-level semantic information such as shape and semantics.

[0095] In one implementation, a backbone feature extraction network, such as a convolutional neural network or a visual transformer, can be used to process the shelf image to obtain a visual feature map containing multiple levels. The specific backbone feature extraction network structure is not limited.

[0096] Feature response enhancement processing refers to the process of selectively enhancing the visual feature response weights of specific types of goods based on visual feature maps.

[0097] In one implementation, a retail-specific attention module can be introduced into the output feature map of the backbone feature extraction network. This attention module combines channel attention and spatial attention mechanisms to adaptively adjust the feature weights of different channels and different spatial locations in the feature map, thereby enhancing the feature response to the areas where special-shaped products (such as small-sized products, slender products, and transparently packaged products) are located, while suppressing feature interference from non-target areas such as shelf backgrounds and shelf shelves, resulting in an enhanced feature map.

[0098] Instance segmentation is a process of performing pixel-level contour detection on each independent product instance in the enhanced feature map, and generating instance masks and bounding boxes for each product instance.

[0099] It should be noted that the instance segmentation process in this embodiment directly generates instance masks without category labels, that is, it only segments and locates the pixel-level contours of each product instance without predicting the category of each product instance. This makes the construction of the region image set completely independent of a preset set of product categories, enabling region extraction for any category of products on the shelf, thus supporting shelf product recognition in open category scenarios.

[0100] In one implementation, an instance segmentation network can be used to perform instance segmentation processing on the enhanced feature map.

[0101] Coordinate information is the position description of the bounding box in the shelf image pixel coordinate system, directly output by instance segmentation processing. It is usually represented by the coordinates of the top-left corner, width, and height of the bounding box. Spatial location information is the result of converting the coordinate information from the shelf image pixel coordinate system into a structured coordinate description of the shelf with retail business semantics.

[0102] In this embodiment, the shelf image can be directly cropped based on the bounding box to obtain a rectangular image containing the product instance, which serves as the visual content of the region image. Alternatively, after cropping the rectangular image based on the bounding box, an instance mask can be used to further suppress the background of the cropped image. This involves filling pixels outside the mask range with the average pixel value to reduce the interference of adjacent product areas within the same bounding box on the visual features of the region image.

[0103] The product recognition method provided in this invention enhances the visual perception of products with special shapes by extracting visual feature maps from shelf images and introducing feature response enhancement processing. It employs an instance segmentation processing method that directly generates instance masks without category labels, ensuring that the construction of the region image set is completely independent of a preset product category set. This supports the extraction of candidate regions for any category of products on the shelf, enabling product region detection for open category scenarios. Simultaneously, by acquiring and storing the structured spatial location information corresponding to each region image, it provides the necessary structured coordinate basis for subsequent layout prior constraints.

[0104] In some embodiments, the text nouns in the text noun set are obtained in the following manner: Semantic features are extracted from the product text information to obtain at least one product entity name and product attribute information matching the product entity name; Based on the dependency relationship between the product entity name and the product attribute information, and a preset text reconstruction template, the corresponding text name is generated.

[0105] Before semantic feature extraction, text normalization preprocessing is performed on the product text information to eliminate the interference of synonymous expressions with different forms on the subsequent extraction results. Text normalization preprocessing includes: standardizing specification expressions, that is, normalizing different representations of measurement units; and removing redundant promotional words, that is, filtering promotional rhetoric words in the product text information that are irrelevant to the core semantics of the product, so as to reduce the interference of such words on the semantic feature extraction results.

[0106] After text normalization preprocessing, semantic features are extracted from the product text information to determine product entity nouns and product attribute information.

[0107] Here, the product entity noun is the main word in the product text information that represents the core category or product name of the product, that is, the most basic semantic unit of the product at the category level. Product attribute information consists of attribute words in the product text information that further define and describe the product entity noun, and are used to depict the specific characteristics of the product, including but not limited to: brand, specifications, flavor, packaging form, etc.

[0108] In one implementation, a retail-specific named entity recognizer built on a joint structure of lightweight Transformer and Conditional Random Field can be used to perform the above semantic feature extraction. This recognizer is jointly trained on three types of heterogeneous corpora: product titles, price tag texts, and store lists, to enhance its adaptability to diverse text expressions in retail scenarios.

[0109] It should be understood that the dependency relationship between product entity names and product attribute information refers to the semantic modification and being modified relationship between each attribute information and the product entity name. That is, each attribute information is semantically subordinate to and modifies the product entity name, jointly and completely describing the product. The dependency relationship can be determined by performing semantic role labeling on the extraction results, identifying the semantic modification relationship of each attribute information on the product entity name, and expressing it in the form of an attribute-core word dependency structure.

[0110] In this embodiment, the preset text reconstruction template is a pre-designed rule template that combines each product attribute information and product entity nouns into a standardized text description string according to a fixed field order and splicing rules.

[0111] In one implementation, the text reconstruction template can be constructed by concatenating fields in the order of "brand-category-flavor-specification-packaging". It should be noted that when some attribute fields are missing in the product text information, the corresponding fields can be omitted in the standardized text description, and the reconstruction should be based on the actual extractable attribute information.

[0112] By using a pre-set text reconstruction template to standardize and reconstruct product text information, it is possible to ensure that the same product description from different sources and in different forms of expression is normalized into the same standardized text description.

[0113] Finally, each standardized text description serves as a text noun, and all standardized text descriptions constitute a text noun set.

[0114] The product recognition method provided in this invention sequentially extracts semantic features from product text information and generates standardized text descriptions based on preset text reconstruction templates, thereby enabling the text input in the cross-modal semantic matching process to have higher semantic consistency and discriminability.

[0115] The product identification device provided in the embodiments of the present invention is described below. The product identification device described below and the product identification method described above can be referred to in correspondence.

[0116] The product identification device of this invention, such as Figure 2 As shown, it includes the following modules: The first acquisition module 210 is used to acquire shelf images and product text information; The second acquisition module 220 is used to acquire the regional image set corresponding to the shelf image and the text noun set corresponding to the product text information; The semantic alignment module 230 is used to map the regional images in the regional image set and the text nouns in the text noun set to a shared semantic space based on the semantic alignment network, and to correct the semantic distance between the regional images and the text nouns through a pre-constructed retail prior constraint to obtain a distance matrix; The recognition module 240 is used to perform global bipartite graph matching on the set of regional images and the set of text nouns based on the distance matrix, and allows some of the regional images or some of the text nouns to be in an unmatched state during the matching process, so as to obtain the product recognition result of the shelf image.

[0117] The product recognition device provided in this invention first acquires the set of regional images corresponding to the shelf images and the set of text nouns corresponding to the product text information. Based on this, a semantic alignment network maps the regional images and text nouns to a shared semantic space. A distance matrix is ​​obtained by correcting the semantic distance between the two through pre-constructed retail prior constraints, introducing multi-dimensional physical and display auxiliary information. This expands the distinguishability between products with extremely similar appearances or descriptions and improves the accuracy of cross-modal feature alignment. Finally, global bipartite graph matching is performed on the regional image set and text noun set based on this distance matrix. During the matching process, some regional images or some text nouns are allowed to remain unmatched, achieving optimal one-to-one mapping of products under a global perspective. This not only avoids local matching conflicts and allocation chaos in densely displayed environments but also adapts to the frequent occurrence of new product launches and stockouts in dynamic retail operations, ensuring high accuracy and robustness of product recognition in open category scenarios.

[0118] Figure 3 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 3 As shown, the electronic device may include: a processor 310, a communications interface 320, a memory 330, and a communication bus 340, wherein the processor 310, the communications interface 320, and the memory 330 communicate with each other via the communication bus 340. The processor 310 can call logical instructions in the memory 330 to execute a product identification method, which includes: Obtain shelf images and product text information; Obtain the region image set corresponding to the shelf image and the text noun set corresponding to the product text information; Based on the semantic alignment network, the regional images in the regional image set and the text nouns in the text noun set are mapped to a shared semantic space, and the semantic distance between the regional images and the text nouns is corrected by a pre-constructed retail prior constraint to obtain a distance matrix; Based on the distance matrix, global bipartite graph matching is performed on the set of regional images and the set of text nouns. During the matching process, some of the regional images or some of the text nouns are allowed to be in an unmatched state, so as to obtain the product recognition result of the shelf image.

[0119] Furthermore, the logical instructions in the aforementioned memory 330 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the conventional, or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc., each of which can store program code.

[0120] On the other hand, embodiments of the present invention also provide a computer program product, the computer program product including a computer program, the computer program being able to be stored on a non-transitory computer-readable storage medium, and when the computer program is executed by a processor, the computer is able to execute the product identification method provided in each of the above embodiments, the method including: Obtain shelf images and product text information; Obtain the region image set corresponding to the shelf image and the text noun set corresponding to the product text information; Based on the semantic alignment network, the regional images in the regional image set and the text nouns in the text noun set are mapped to a shared semantic space, and the semantic distance between the regional images and the text nouns is corrected by a pre-constructed retail prior constraint to obtain a distance matrix; Based on the distance matrix, global bipartite graph matching is performed on the set of regional images and the set of text nouns. During the matching process, some of the regional images or some of the text nouns are allowed to be in an unmatched state, so as to obtain the product recognition result of the shelf image.

[0121] In another aspect, embodiments of the present invention also provide a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the product identification method provided in each of the above embodiments, the method comprising: Obtain shelf images and product text information; Obtain the region image set corresponding to the shelf image and the text noun set corresponding to the product text information; Based on the semantic alignment network, the regional images in the regional image set and the text nouns in the text noun set are mapped to a shared semantic space, and the semantic distance between the regional images and the text nouns is corrected by a pre-constructed retail prior constraint to obtain a distance matrix; Based on the distance matrix, global bipartite graph matching is performed on the set of regional images and the set of text nouns. During the matching process, some of the regional images or some of the text nouns are allowed to be in an unmatched state, so as to obtain the product recognition result of the shelf image.

[0122] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0123] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the conventional, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0124] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in each of the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of each embodiment of the present invention.

Claims

1. A product identification method, characterized in that, include: Obtain shelf images and product text information; Obtain the region image set corresponding to the shelf image and the text noun set corresponding to the product text information; Based on the semantic alignment network, the regional images in the regional image set and the text nouns in the text noun set are mapped to a shared semantic space, and the semantic distance between the regional images and the text nouns is corrected by a pre-constructed retail prior constraint to obtain a distance matrix; Based on the distance matrix, global bipartite graph matching is performed on the set of regional images and the set of text nouns. During the matching process, some of the regional images or some of the text nouns are allowed to be in an unmatched state, so as to obtain the product recognition result of the shelf image.

2. The product identification method according to claim 1, characterized in that, The pre-constructed retail prior constraints include product location layout constraints and product physical size constraints; The semantic distance between the region image and the text noun is corrected by pre-constructed retail prior constraints to obtain a distance matrix, including: Extract the visual feature vector of the region image and the text feature vector of the text noun, and determine the initial semantic distance between the visual feature vector and the text feature vector; Based on the product location layout constraints and the spatial location information of the region image, the layout penalty distance between the region image and the text noun is determined; Based on the physical size constraints of the product and the size feature information of the region image, the size penalty distance between the region image and the text noun is determined; Based on the initial semantic distance, the layout penalty distance, and the size penalty distance, the corrected distance between the region image and the text noun is determined; A distance matrix is ​​obtained based on the corrected distances of each data pair in the region image set and the text noun set.

3. The product identification method according to claim 2, characterized in that, The extraction of the visual feature vector of the region image and the text feature vector of the text noun includes: The region image is input into a pre-trained visual coding network to extract the visual feature vector of the region image; The text nouns are input into a pre-trained text encoding network to extract the text feature vectors of the text nouns.

4. The product identification method according to claim 1, characterized in that, The step involves performing global bipartite graph matching on the region image set and the text noun set based on the distance matrix, allowing some region images or some text nouns to remain unmatched during the matching process, to obtain the product recognition result of the shelf image, including: The region image set and the text noun set are respectively used as two node sets of a bipartite graph, and the connection weights between corresponding nodes are determined based on the distance matrix; A global mapping assignment is performed on the bipartite graph to obtain a mapping relationship that includes matching node pairs, unmatched region image nodes, and unmatched text noun nodes; wherein, when the connection weight between nodes is greater than a preset weight threshold, the corresponding node is determined to be in an unmatched state; The unmatched region image nodes are marked as unknown objects, and the unmatched text noun nodes are marked as not displayed. Based on the matching node pairs, the region image nodes marked as unknown objects, and the text noun nodes marked as not displayed, the product recognition result of the shelf image is obtained.

5. The product identification method according to claim 4, characterized in that, After obtaining the mapping relationship containing matching node pairs, unmatched region image nodes, and unmatched text noun nodes, the process further includes: Obtain the reference text information associated with the region image nodes in the matching node pair; Determine the text similarity between the reference text information and the text noun nodes in the matching node pair; Based on the text similarity and the connection weights of the matching node pairs, a comprehensive confidence score for the matching node pairs is determined. When the overall confidence score is not greater than a preset score threshold, the matching node pair is marked as pending review.

6. The product identification method according to claim 4, characterized in that, After marking the unmatched region image nodes as unknown objects, the method further includes: Obtain the supplementary text information corresponding to the image nodes in the unmatched region; Incremental alignment samples are constructed based on the region images corresponding to the unmatched region image nodes and the supplementary text information. The semantic alignment network is updated based on the incremental alignment samples.

7. The product identification method according to claim 1, characterized in that, The region images in the region image set are obtained in the following way: The visual feature map of the shelf image is extracted, and the visual feature map is subjected to feature response enhancement processing to obtain an enhanced feature map; The enhanced feature map is subjected to instance segmentation processing to obtain at least one instance mask and the bounding box corresponding to the instance mask; Obtain the coordinate information of the bounding box in the shelf image, and map the coordinate information into spatial location information; Based on the instance mask, the bounding box corresponding to the instance mask, and the spatial location information, the corresponding region image is extracted from the shelf image.

8. The product identification method according to claim 1, characterized in that, The text nouns in the text noun set were obtained in the following way: Semantic features are extracted from the product text information to obtain at least one product entity name and product attribute information matching the product entity name; Based on the dependency relationship between the product entity name and the product attribute information, and a preset text reconstruction template, the corresponding text name is generated.

9. A product identification device, characterized in that, include: The first acquisition module is used to acquire shelf images and product text information; The second acquisition module is used to acquire the regional image set corresponding to the shelf image and the text noun set corresponding to the product text information; The semantic alignment module is used to map the regional images in the regional image set and the text nouns in the text noun set to a shared semantic space based on the semantic alignment network, and to correct the semantic distance between the regional images and the text nouns through a pre-constructed retail prior constraint to obtain a distance matrix; The recognition module is used to perform global bipartite graph matching on the set of regional images and the set of text nouns based on the distance matrix, and allows some of the regional images or some of the text nouns to be in an unmatched state during the matching process, so as to obtain the product recognition result of the shelf image.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the commodity identification method as described in any one of claims 1 to 8.

11. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the commodity identification method as described in any one of claims 1 to 8.

12. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the commodity identification method as described in any one of claims 1 to 8.

Images