Commodity recognition method and device based on spatial visual semantic understanding and electronic equipment

Abstract

The present disclosure provides a commodity identification method and device based on spatial visual semantic understanding and electronic equipment, relates to the field of artificial intelligence, and particularly relates to the fields of deep learning, natural language processing and large models. The specific implementation scheme is: based on a quantification model, processing semantic feature vectors of each commodity in a commodity library to be recommended to obtain a semantic identifier corresponding to each commodity; based on the semantic identifier corresponding to each commodity, determining a semantic identifier set corresponding to a scene picture in a scene restoration dataset; wherein the semantic identifier set includes a semantic identifier of at least one commodity in the scene picture; based on the scene picture in the scene restoration dataset and the semantic identifier set corresponding thereto, training a scene restoration model; wherein the scene restoration model is used to determine a semantic identifier of at least one recommended commodity corresponding to a target scene picture based on the target scene picture. The present disclosure can improve the efficiency and accuracy of commodity identification in a complex scene.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence, particularly to the fields of deep learning, natural language processing, and large models. Background Technology

[0002] With the development of artificial intelligence technology and e-commerce platform search and recommendation systems, the market demand for image-based product recognition and recommendation is continuously growing, especially in the field of scene-based displays for home furnishings and decoration. Reconstructing and recommending products based on scene images has become a key means to improve user experience and promote conversion rates. However, most existing product recognition technologies are based on shallow matching of visual features, lacking a deep understanding of scene spatial layout and semantic relationships, resulting in low recognition accuracy. Therefore, how to improve the efficiency and accuracy of product recognition in complex scenes has become an urgent problem to be solved. Summary of the Invention

[0003] This disclosure provides a product recognition method, apparatus, and electronic device based on spatial visual semantic understanding.

[0004] According to one aspect of this disclosure, a product recognition method based on spatial visual semantic understanding is provided, comprising: The semantic feature vector of each product in the product library to be recommended is processed based on the quantization model to obtain the semantic identifier corresponding to each product. Based on the semantic identifier corresponding to each product, a set of semantic identifiers corresponding to the scene images in the scene reconstruction dataset is determined; wherein, the set of semantic identifiers includes the semantic identifiers of at least one product in the scene images; A scene reconstruction model is trained based on the scene images in the scene reconstruction dataset and their corresponding semantic identifier set; wherein, the scene reconstruction model is used to determine the semantic identifier of at least one recommended product corresponding to the target scene image based on the target scene image.

[0005] According to another aspect of this disclosure, a product recognition method based on spatial visual semantic understanding is provided, comprising: Based on the target scene image, a scene reconstruction model is used to determine the semantic identifier of at least one recommended product corresponding to the target scene image; wherein, the scene reconstruction model is trained based on the scene images and their corresponding semantic identifier sets in the scene reconstruction dataset, and the semantic identifier set includes the semantic identifier of at least one product in the scene image; Based on the semantic identifier of the at least one recommended product, the product identifier of the at least one recommended product is determined.

[0006] According to another aspect of this disclosure, a product recognition device based on spatial visual semantic understanding is provided, comprising: The semantic identifier determination module is used to process the semantic feature vector of each product in the product library to be recommended based on the quantization model, so as to obtain the semantic identifier corresponding to each product. The semantic identifier set determination module is used to determine the semantic identifier set corresponding to the scene image in the scene reconstruction dataset based on the semantic identifier corresponding to each product; wherein, the semantic identifier set includes the semantic identifier of at least one product in the scene image; The scene reconstruction model training module is used to train a scene reconstruction model based on scene images in the scene reconstruction dataset and their corresponding semantic identifier sets; wherein, the scene reconstruction model is used to determine the semantic identifier of at least one recommended product corresponding to the target scene image based on the target scene image.

[0007] According to another aspect of this disclosure, a product recognition device based on spatial visual semantic understanding is provided, comprising: The semantic identifier determination module is used to determine the semantic identifier of at least one recommended product corresponding to the target scene image based on the target scene image and using a scene reconstruction model; wherein, the scene reconstruction model is trained based on scene images and their corresponding semantic identifier sets in the scene reconstruction dataset, and the semantic identifier set includes the semantic identifier of at least one product in the scene image; The product identifier determination module is used to determine the product identifier of the at least one recommended product based on the semantic identifier of the at least one recommended product.

[0008] According to another aspect of this disclosure, an electronic device is provided, comprising: At least one processor; and The memory is communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform any of the methods described in the present disclosure.

[0009] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform any of the methods according to embodiments of this disclosure.

[0010] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements any of the methods according to embodiments of this disclosure.

[0011] According to embodiments of this disclosure, by processing the semantic feature vector of each product in the recommended product library using a quantization model, a semantic identifier corresponding to each product is obtained. This effectively compresses high-dimensional continuous semantic features into low-dimensional discrete semantic identifiers, thereby reducing data storage and computational complexity. Furthermore, by determining the set of semantic identifiers corresponding to scene images in the scene reconstruction dataset based on the semantic identifiers corresponding to each product, and training the scene reconstruction model based on the scene images and their corresponding semantic identifier sets in the scene reconstruction dataset, product recognition can be achieved by determining the semantic identifiers of recommended products based on target scene images, without needing to perform high-dimensional vector similarity calculations. This improves the efficiency of product recognition while ensuring its accuracy.

[0012] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0013] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein: Figure 1 This is a flowchart illustrating a product recognition method based on spatial visual semantic understanding according to an embodiment of the present disclosure; Figure 2 This is a flowchart illustrating a product recognition method based on spatial visual semantic understanding according to another embodiment of the present disclosure; Figure 3 This is a schematic diagram of a product recognition method based on spatial visual semantic understanding, which is an application example. Figure 4 This is a schematic block diagram of a product recognition device based on spatial visual semantic understanding provided in an embodiment of this disclosure; Figure 5 This is a schematic block diagram of a product recognition device based on spatial visual semantic understanding provided in another embodiment of this disclosure; Figure 6 This is a schematic block diagram of the quantization model training module in an embodiment of this disclosure; Figure 7 This is a schematic block diagram of a quantization model training module in another embodiment of this disclosure; Figure 8 This is a schematic block diagram of a product recognition device based on spatial visual semantic understanding, provided in another embodiment of this disclosure; Figure 9 This is a schematic block diagram of a product recognition device based on spatial visual semantic understanding, provided in another embodiment of this disclosure; Figure 10This is a block diagram of an electronic device used to implement the product recognition method based on spatial visual semantic understanding according to the embodiments of this disclosure. Detailed Implementation

[0014] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0015] Current mainstream product recognition solutions primarily employ a combination of image feature extraction and vector retrieval techniques. This involves first using deep neural networks to extract corresponding visual features from individual product images and scene images containing products, transforming the originally uncalculated two-dimensional image data into a high-dimensional continuous vector representation capable of mathematical calculations. Then, by calculating similarity metrics such as cosine similarity between vectors, the system searches a product database for the most similar product, thus completing product recognition and matching. Furthermore, for complex application scenarios with densely packed products and strong background interference, object detection or semantic segmentation techniques are often combined. This involves first locating the regions containing each candidate product within the complete scene image, cropping the region for each candidate product, and then performing feature extraction and matching on each cropped region separately. This reduces the negative impact of background interference on the recognition results.

[0016] The following problems exist in the related technologies: First, the product features extracted by traditional product recognition technology are high-dimensional vectors. With the development of the e-commerce industry, the existing large-scale product databases can store millions or even tens of millions of products. Storing the high-dimensional vectors of all products will generate extremely high storage costs. At the same time, when carrying out full-database product retrieval and matching, it is necessary to complete the similarity calculation of the high-dimensional vector of each product, which greatly limits the overall retrieval efficiency.

[0017] Second, the high-dimensional vectors used in traditional retrieval technologies lack a structured discrete semantic system. For new products that have not participated in model training, it is difficult to directly generate feature vectors that fit the current retrieval framework, resulting in poor recommendation performance for new products and a poor cold start effect.

[0018] Third, scene images and pure product images have natural differences in overall visual structure. Scene images contain a lot of background content and irrelevant interference. When the features of the two types of images are directly placed in the same vector space for similarity matching, semantic offset problems are likely to occur during feature mapping, which directly affects the accuracy of the final product matching.

[0019] This disclosure utilizes a quantization model to compress high-dimensional continuous semantic feature vectors into low-dimensional discrete semantic identifiers, constructing a structured product semantic system. Based on this, by establishing a mapping relationship between scene images and product semantic identifier sets and training a scene reconstruction model, the model can directly parse product combinations with spatial layout and semantic association from target scene images, rather than relying solely on shallow visual feature matching. This significantly reduces data storage costs and retrieval computation complexity, improves retrieval efficiency in large-scale product databases, effectively solves the cold start problem for new products, and overcomes semantic offset defects caused by background interference. Therefore, in complex home furnishing and decoration scenario-based display fields, it achieves higher accuracy and stronger robustness in product recognition and recommendation.

[0020] Figure 1 A schematic diagram of a product recognition method based on spatial visual semantic understanding according to an embodiment of this disclosure is shown. This method can be applied to a product recognition device based on spatial visual semantic understanding, which can be deployed in an electronic device. The electronic device is, for example, a single-machine or multi-machine terminal, server, or other processing device. In some possible implementations, the method can also be implemented by a processor calling computer-readable instructions stored in memory. Figure 1 As shown, the method may include the following steps S110 to S130.

[0021] S110. Based on the quantization model, process the semantic feature vector of each product in the recommended product library to obtain the semantic identifier corresponding to each product.

[0022] In this embodiment of the disclosure, the quantization model is a model that maps high-dimensional continuous vectors to low-dimensional discrete identifiers. Its purpose is to significantly reduce the dimensionality and storage overhead of data while preserving the original semantic information.

[0023] Optionally, the quantization model can use vector quantization (VQ), product quantization (PQ), or other clustering algorithms to divide the continuous semantic feature space into several discrete subspaces or clusters, each subspace or cluster corresponding to a unique semantic identifier.

[0024] Optionally, the semantic feature vectors in the product library to be recommended can be obtained by encoding the product images through a pre-trained visual encoder, or by jointly encoding the product images and text descriptions through a multimodal large model.

[0025] Optionally, the semantic identifier can be an integer index or a specific encoded string used to uniquely represent the position of the product in a discrete semantic space.

[0026] Optionally, the index can be organized using an inverted index or a graph retrieval method to support efficient search queries. Specifically, the semantic identifier corresponding to each product is used as the key, and the unique identifier of the product (such as product ID) is used as the value, establishing a mapping relationship. When it is necessary to retrieve products related to a specific semantic identifier, the index can be used to quickly locate all sets of products with the same or similar semantic identifiers, avoiding traversing and calculating similarity in the entire high-dimensional vector space, thereby significantly reducing search latency.

[0027] Optionally, when constructing an inverted index or graph index structure, semantic tags can be organized hierarchically or stored hierarchically. For example, based on the generality or popularity of semantic tags, frequently occurring semantic tags can be stored in a cache, while infrequently occurring semantic tags can be stored in conventional storage media.

[0028] S120. Based on the semantic identifier corresponding to each product, determine the set of semantic identifiers corresponding to the scene images in the scene reconstruction dataset; wherein, the set of semantic identifiers includes the semantic identifier of at least one product in the scene image.

[0029] In this embodiment of the disclosure, the scene reconstruction dataset contains multiple scene images and labeled product information for each image. For each scene image in the dataset, one or more products contained therein are first identified, and then the semantic identifiers generated in step S110 for these products are queried. These semantic identifiers are combined into a semantic identifier set, which serves as the semantic identifier set corresponding to that scene image.

[0030] Optionally, identifying products contained in scene images can be achieved through manual annotation, or by automatically extracting product regions from scene images using pre-trained object detection models or instance segmentation models, and then matching products in the product recommendation library based on the image features of the product regions, thereby determining the products contained in the scene images and their corresponding semantic identifiers.

[0031] Optionally, the order of elements in the semantic tag set can be associated with the spatial position of the product in the scene image, such as being arranged from left to right or from top to bottom, or sorted according to the importance weight of the product in the scene, thus implying certain spatial layout information in the semantic tags.

[0032] Optionally, location codes or relative positional relationship descriptions can be introduced and combined or merged with the semantic identifiers of the product to form a composite identifier set containing "semantic + location" information.

[0033] Optionally, when determining the set of semantic identifiers corresponding to the scene images, the initially determined semantic identifiers can be verified or corrected by combining the contextual information of the scene or global semantic features.

[0034] Optionally, the scene images can be processed at multiple scales to generate scene image copies at different resolutions, and the semantic tag set corresponding to the scene images at each resolution can be determined. The multi-scale semantic tag set can be used as training input so that the trained scene reconstruction model can adapt to target scene images of different resolutions or shooting distances, thereby enhancing the model's adaptability in real-world application scenarios.

[0035] Optionally, data augmentation processing can be performed on the semantic tag set, such as generating multiple variants by random masking, random replacement, or random shuffling, to increase the diversity of training data and thereby improve the generalization ability and robustness of the scene reconstruction model.

[0036] S130. Based on the scene images in the scene reconstruction dataset and their corresponding semantic identifier set, a scene reconstruction model is trained; wherein, the scene reconstruction model is used to determine the semantic identifier of at least one recommended product corresponding to the target scene image based on the target scene image.

[0037] Optionally, to improve training efficiency, the scene reconstruction model can adopt a pre-training-fine-tuning strategy, that is, first pre-train the scene reconstruction model on a general scene dataset, and then fine-tune it on a specific scene reconstruction dataset to adapt to the scene features of a specific domain (such as home furnishing and decoration).

[0038] Optionally, the scene reconstruction model can adopt an encoder-decoder architecture, where the encoder is used to extract visual features of the target scene image, and the decoder is used to predict the corresponding set of semantic labels based on the visual features.

[0039] Optionally, the scene reconstruction model can introduce an attention mechanism, enabling the model to focus on key areas in the scene image that are relevant to product recognition, while ignoring background noise interference, thereby further improving the model's recognition accuracy.

[0040] Optionally, different weights can be assigned to semantic identifiers of different frequencies, so that the model pays more attention to rare but important semantic features of goods, thereby balancing the overall recognition performance. Optionally, the semantic identifiers output by the scene reconstruction model can include confidence information. The system can sort the predicted semantic identifiers according to the confidence level and select the top K semantic identifiers with the highest confidence level as the final query conditions.

[0041] Optionally, to provide richer recommendation results, in addition to precisely matching semantic tags, the proximity relationship between semantic tags can also be considered. For example, in a discrete semantic space, some semantic tags may be semantically similar (such as "modern minimalist style sofa" and "Nordic style sofa"). The system can retrieve products corresponding to other tags that are close to the target semantic tag in the semantic space, thereby increasing the diversity of recommendations.

[0042] Optionally, if the target scene image contains multiple different regions or focal points, the scene reconstruction model can analyze the different regions separately to obtain multiple local semantic label sets, and then combine these local results for global retrieval to achieve more refined scene-based recommendations.

[0043] As can be seen from this embodiment, by using a quantization model to compress high-dimensional continuous semantic feature vectors into low-dimensional discrete semantic identifiers, a structured product semantic system is constructed, significantly reducing data storage costs and retrieval computational complexity. Based on this, by establishing a mapping relationship between scene images and product semantic identifier sets and using this mapping to train the scene reconstruction model, the model can directly parse semantically related products from target scene images without performing high-dimensional vector similarity calculations, improving product recognition efficiency. Furthermore, this recognition process does not rely solely on shallow visual feature matching, effectively solving the cold-start problem for new products and overcoming semantic shift defects caused by background interference. Thus, in complex home furnishing and decoration scenario-based display fields, higher accuracy and stronger robustness in product recognition and recommendation are achieved.

[0044] In some embodiments, the product recognition method based on spatial visual semantic understanding further includes: Based on the image of each product in the full product database, a vector generation model is used to generate the semantic feature vector corresponding to each product in the full product database; the full product database includes a library of products to be recommended. A quantization model is trained based on the semantic feature vector corresponding to each product in the full product database.

[0045] In this embodiment of the disclosure, when training the quantization model, the semantic feature vector corresponding to each product in the full product database can be used as input. The parameters of the quantization model are iteratively updated by objective functions such as minimizing quantization error or maximizing intra-cluster similarity until the model converges. Specifically, the training process of the quantization model aims to find an optimal codebook or cluster centers, such that each input continuous vector can be mapped to the nearest discrete identifier, while ensuring that the mapped discrete identifier can retain the semantic information of the original vector to the greatest extent.

[0046] Optionally, the vector generation model can be a pre-trained convolutional neural network (CNN) or the visual encoder part of a multimodal large model. This vector generation model is pre-trained on large-scale image data and has powerful feature extraction capabilities, enabling it to map product images into a high-dimensional semantic space to obtain vector representations that can characterize attributes such as product appearance, material, and style.

[0047] Optionally, the vector generation model can be fine-tuned to better suit the product feature extraction needs of specific domains (such as home furnishings and decoration).

[0048] Optionally, as new products are added to the full product database, the quantization model can be retrained or fine-tuned periodically to maintain the accuracy of the semantic labeling system and ensure consistently good recognition performance.

[0049] Optionally, for multimodal scenarios, the semantic labeling space can be further constrained by combining the textual description information of the products and using a cross-modal alignment loss function, so that it not only reflects visual similarity but also semantic relevance, thereby providing a richer semantic foundation for subsequent scene reconstruction and product recommendation.

[0050] According to the above embodiments, by extracting semantic feature vectors based on a vector generation model using a full product database, and using these high-dimensional continuous vectors to train a quantization model, a structured and compact discrete semantic labeling system covering the entire product database can be constructed, laying a solid data foundation for accurate product identification and recommendation in complex scenarios.

[0051] In some embodiments, the product recognition method based on spatial visual semantic understanding further includes: Based on the image of each product in the product library to be recommended, a vector generation model is used to generate the semantic feature vector of each product in the product library to be recommended.

[0052] Optionally, the specific architecture and parameters of the vector generation model can be the same as the model used to generate semantic feature vectors for all products in the aforementioned embodiments, or they can be adjusted according to the characteristics of the product library to be recommended. For example, if the product library to be recommended mainly contains a specific category (such as furniture), a visual encoder finely tuned on the data of that category can be used to extract more discriminative fine-grained features.

[0053] Optionally, for newly added products to the product library, feature vectors can be extracted directly using a vector generation model, and corresponding semantic identifiers can be obtained through a quantization model. These identifiers are then added to the existing index structure. This incremental processing approach enables the system to flexibly respond to dynamic updates to the product library, ensuring the real-time performance and accuracy of the recommendation results.

[0054] According to the above embodiments, by generating semantic feature vectors based on images from the product library to be recommended using a vector generation model, fine-grained visual semantic information of the products to be recommended can be accurately captured. Combined with a quantization model, these vectors are mapped to standard discrete semantic identifiers. This process not only achieves efficient conversion from raw image data to structured semantic indexes, but also ensures that new products can be seamlessly integrated into the existing semantic system without reconstructing the overall model architecture. The resulting set of semantic identifiers retains the core attribute features of the products and has extremely high retrieval efficiency, providing high-quality data support for rapid and accurate product matching and recommendation in complex scenarios.

[0055] In some embodiments, the vector generation model is obtained by fine-tuning the multimodal vector model based on an industry product database; the full product database includes an industry product database, which includes a product database to be recommended.

[0056] In this embodiment of the disclosure, the industry product library may contain product data from specific vertical fields (such as home furnishings, home appliances, and clothing). By fine-tuning the multimodal vector model on the industry product library, the model can be made to focus more on capturing the key visual attributes and semantic details of products in that field, such as the material texture of furniture, the functional form of home appliances, or the tailoring style of clothing, thereby generating more distinctive and domain-adaptive semantic feature vectors.

[0057] Optionally, the multimodal vector model can be a vision-language pre-trained model based on the Transformer architecture, which can process image and text inputs simultaneously. During fine-tuning, product images and their corresponding titles, descriptions, attribute tags, and other multimodal information from an industry product library can be used as training samples. By minimizing the distance between image features and text features in the shared semantic space, the model can learn more robust and fine-grained cross-modal representation capabilities.

[0058] Optionally, the fine-tuning process can employ full parameter fine-tuning or parameter-efficient fine-tuning strategies to balance computational resource consumption and model performance improvement.

[0059] According to the above embodiments, by fine-tuning the multimodal vector model based on the industry product library, it is possible to obtain vector generation capabilities optimized for specific fields. This makes the generated semantic feature vectors more reflective of the subtle differences and deep semantics of industry products, thereby providing a more powerful feature expression foundation for building a high-precision discrete semantic identification system and realizing accurate scenario-based product recommendations.

[0060] In some embodiments, the quantization model is based on a multi-level residual quantization structure; the semantic identifier includes multiple discrete encoded values, which correspond to multiple quantization layers in the multi-level residual quantization structure.

[0061] In this embodiment, the multi-level residual quantization structure decomposes the original high-dimensional semantic feature vector into multiple levels of discrete encoding by quantizing the residual layer by layer. Specifically, in the first level of quantization, the model finds the codebook vector that is closest to the input vector and calculates the residual vector between the two. In each subsequent level of quantization, the model quantizes the residual vector generated in the previous level again, thereby gradually approximating the fine features of the original vector. This hierarchical quantization method enables semantic identifiers to represent subtle differences in products with higher precision. For example, in a home furnishing scenario, it can more accurately distinguish similar products with different material textures or color shades.

[0062] Optionally, each quantization layer can have its own codebook, and the size of the codebook can be adjusted according to the amount of information carried by that layer. For example, the first few quantization layers are used to capture the main semantic categories and overall style of the product, while the later quantization layers are used to capture local detailed features.

[0063] Optionally, when generating semantic identifiers, the index values ​​obtained from quantization at each level can be concatenated together to form a composite semantic identifier. This composite identifier not only retains global semantic information but also contains rich detailed information, thereby enabling finer-grained matching during the retrieval stage.

[0064] Optionally, when training a multi-level residual quantization model, an end-to-end training approach can be adopted to jointly optimize the parameters of each level of quantizer in order to minimize the error between the final vector and the original vector.

[0065] Optionally, different weight coefficients can be assigned to different levels of quantization results. For example, higher weights can be assigned to the first few quantization layers responsible for capturing global semantics to ensure the accuracy of the main features; while lower weights can be assigned to the later quantization layers responsible for capturing details to allow for some error tolerance.

[0066] According to the above embodiments, by adopting a quantization model based on a multi-level residual quantization structure, a discrete identification system with hierarchical semantic expression capability is constructed. This system not only has significant advantages in storage efficiency but also achieves a breakthrough in the precision of semantic expression. This enables the system to accurately capture subtle differences between products while maintaining efficient retrieval, providing strong technical support for high-precision product identification and personalized recommendation in complex scenarios.

[0067] In some embodiments, the quantization model includes a residual quantization variational autoencoder (RQ-VAE) model.

[0068] In this embodiment, the Residual Quantization Variational Autoencoder (RQ-VAE) model combines the probabilistic generation capability of Variational Autoencoders (VAEs) with the discrete representation advantages of Residual Quantization. During the encoding stage, the input product semantic feature vector is first mapped to the latent space by the encoder, resulting in a continuous feature distribution. Subsequently, a multi-level residual quantization module progressively transforms this continuous feature into a discrete codebook index sequence. During the decoding stage, the model reconstructs the feature vector based on these discrete indices and jointly optimizes the model parameters using reconstruction loss and quantization loss.

[0069] Alternatively, in order to overcome the problem of low codebook utilization, the RQ-VAE model can adopt a codebook reset strategy to ensure that all codebook vectors can be fully trained and used, and avoid the decline in representation ability caused by the failure of some codebooks.

[0070] According to the above embodiments, by introducing a Residual Quantization Variational Autoencoder (RQ-VAE) model, the flexibility of probabilistic generation models can be combined with the discrete characteristics of vector quantization. Specifically, the RQ-VAE model maps the input product semantic feature vector to a probability distribution in the latent space through an encoder, and uses a multi-level residual quantizer to discretize the sampled latent variables. Then, the original features are reconstructed through a decoder, and the model parameters are optimized by minimizing the reconstruction error and KL divergence during training, thereby learning a smoother and more continuous latent semantic space distribution.

[0071] In some embodiments, the product recognition method based on spatial visual semantic understanding further includes: Construct and store the mapping relationship between the product identifier and semantic identifier of each product in the product library to be recommended; wherein, the mapping relationship is used to determine the product identifier of at least one recommended product based on the semantic identifier of at least one recommended product in the target scene image.

[0072] Optionally, the mapping relationship can be stored in an inverted index table or a key-value pair database, so that the corresponding product identifier can be quickly retrieved based on the semantic identifier during the retrieval stage. Specifically, the system can use the unique product identifier of each product as the value and its corresponding one or more semantic identifiers as the key to establish a direct link from the semantic space to the product space. After the scene reconstruction model outputs the set of semantic identifiers of the target scene image, the retrieval module can directly use this mapping relationship to obtain all matching product identifiers in batches, thereby avoiding time-consuming similarity calculations in high-dimensional vector space and greatly improving retrieval efficiency.

[0073] Optionally, considering that a product may correspond to multiple semantic identifiers (e.g., different levels of encoding obtained through multi-level residual quantization, or different feature representations generated by multi-view images), a one-to-many structure can be adopted in the mapping relationship, that is, one product identifier is associated with multiple semantic identifiers. During retrieval, as long as the target semantic identifier set contains any semantic identifier associated with the product, the product can be recalled.

[0074] Optionally, the mapping relationship can be preprocessed, such as counting the number of products corresponding to each semantic tag, and performing special weight adjustments or filtering on semantic tags that are too popular or too unpopular, in order to balance the diversity and accuracy of the recommendation results.

[0075] Optionally, the mapping relationship can be updated in sync with changes to the product library. When a new product is added to the product library to be recommended, the system generates its semantic identifier and immediately updates the mapping relationship. When a product is removed from the shelves or its information changes, the system removes or updates the corresponding entry from the mapping relationship.

[0076] Optionally, the mapping relationship can be updated using either incremental updates or full reconstruction. For incremental updates, the system only regenerates semantic identifiers for products that have changed (such as newly added, modified, or deleted products) and updates the corresponding entries in the index accordingly. This method has low computational overhead and is suitable for scenarios with frequent, small-scale product changes. For full reconstruction, when the product library undergoes large-scale changes or the semantic identifier system changes due to retraining of the quantization model, the system can re-traverse the entire product library to be recommended, regenerate the semantic identifiers of all products, and construct a completely new mapping relationship to ensure the consistency and integrity of the index data.

[0077] Optionally, to improve the query efficiency of mapping relationships, semantic tags can be compressed or stored in segments. For example, semantic tags can be grouped and stored according to their corresponding product categories or attribute categories to form multiple sub-indexes. During retrieval, the relevant sub-index range is first determined based on the preliminary classification results of the target scene image, and then precise matching is performed within this range, thereby further narrowing the search space and improving retrieval speed.

[0078] According to the above embodiments, by constructing and maintaining an efficient mapping relationship between product identifiers and semantic identifiers, a rapid mapping from abstract semantic space to specific product entities is achieved. This not only significantly reduces the computational complexity of the online retrieval stage, but also significantly improves the response speed and concurrent processing capability of product recommendations. This enables the system to quickly and accurately locate a set of products that highly match the target scenario from a large product database, providing a solid technical guarantee for a real-time, scenario-based shopping experience.

[0079] Figure 2A schematic diagram of a product recognition method based on spatial visual semantic understanding, according to another embodiment of this disclosure, is shown. This method can be applied to a product recognition device based on spatial visual semantic understanding, which can be deployed in an electronic device. The electronic device is, for example, a single-machine or multi-machine terminal, server, or other processing device. In some possible implementations, the method can also be implemented by a processor calling computer-readable instructions stored in memory. Figure 2 As shown, the method may include the following steps S210 to S220.

[0080] S210. Based on the target scene image, a scene reconstruction model is used to determine the semantic identifier of at least one recommended product corresponding to the target scene image; wherein, the scene reconstruction model is trained based on the scene images and their corresponding semantic identifier set in the scene reconstruction dataset, and the semantic identifier set includes the semantic identifier of at least one product in the scene image.

[0081] In this embodiment, the scene reconstruction model can be a trained deep learning model, such as a convolutional neural network (CNN), a visual model based on the Transformer architecture, or a multimodal large model. This model can understand the spatial layout, object relationships, and contextual semantics in the target scene image, thereby inferring the possible products in the scene and their corresponding semantic identifiers.

[0082] Optionally, the scene reconstruction model can employ a deep learning-based image segmentation, object detection, or instance segmentation architecture. For example, the model can first perform semantic segmentation on the target scene image to identify the product region to which each pixel in the image belongs; then, for each identified product region, its visual feature vector is extracted and converted into a corresponding semantic label using a quantization model or classifier. Furthermore, the model can also incorporate contextual information and utilize an attention mechanism to capture the spatial relationships and semantic associations between products, thereby improving the accuracy of semantic label prediction.

[0083] Optionally, the scene reconstruction dataset may contain a large number of real-world or synthetic scene images labeled with product semantic identifiers. During training, the scene reconstruction model learns to extract visual features from the input scene images and predicts a set of semantic identifiers that match the products in the scene. These semantic identifiers may be discrete encoded values ​​generated by the quantization model in the aforementioned embodiments, or they may be other forms of semantic representation.

[0084] Optionally, to improve the accuracy of the scene reconstruction model, diverse scene types and product combinations can be introduced into the training data, covering different lighting conditions, shooting angles, and occlusion situations. Furthermore, data augmentation techniques, such as random cropping, rotation, and color jitter, can be used to increase the diversity of the training data and improve the model's generalization ability.

[0085] S220. Based on the semantic identifier of at least one recommended product, determine at least one recommended product in the product library to be recommended.

[0086] In this embodiment of the disclosure, based on the semantic identifier determined in step S210, the system can quickly find matching products from the mapping relationship between product identifiers and semantic identifiers.

[0087] Optionally, the mapping relationship between product identifiers and semantic identifiers can be pre-defined or automatically generated by discretizing the product feature vectors using a quantization model.

[0088] Optionally, when determining recommended products, a strategy of exact matching or approximate matching can be adopted. For exact matching, only products with completely matching semantic identifiers are recalled; for approximate matching, the similarity or distance between semantic identifiers can be considered to recall semantically similar products, thereby increasing the diversity and coverage of the recommendation results.

[0089] Optionally, if step S210 outputs multiple semantic identifiers, the system can retrieve each semantic identifier separately, and then merge, deduplicate, and sort the retrieval results. The sorting strategy can be determined based on a combination of factors such as the confidence level of semantic matching, the popularity of the product, sales volume, and user preferences to generate the final recommendation list.

[0090] Optionally, to further improve the recommendation effect, the ranking can be re-adjusted based on the user's personalized information. For example, the ranking weight of candidate products can be adjusted according to the user's browsing history and purchasing behavior, so that the recommendation results are more in line with the user's personal preferences.

[0091] Optionally, if the output semantic identifier contains multiple levels or types of encoding (such as global semantic identifiers and local detail identifiers), a multi-level filtering strategy can be adopted during retrieval. First, a batch of candidate products is recalled using the global semantic identifier to narrow the search scope; then, the candidate products are re-ranked or filtered using the local detail identifier to improve the relevance and accuracy of the recommendation results.

[0092] According to the above embodiments, by first using a scene reconstruction model to extract semantic tags from the target scene image, and then performing efficient retrieval in the product database based on the semantic tags, an end-to-end process from visual perception to product recommendation is realized. This fully utilizes the advantages of discrete semantic tags in storage and retrieval, significantly reduces computational complexity, improves response speed, and ensures a high degree of relevance between the recommendation results and the scene content, providing users with a more convenient and accurate shopping experience.

[0093] In some embodiments, the scene reconstruction model is trained using any of the methods in the product recognition method based on spatial visual semantic understanding provided in the above-described embodiment of the present disclosure. In this embodiment, the scene reconstruction model can be trained using the semantic tagging system constructed in the foregoing embodiments. During the training phase, scene images are input into the scene reconstruction model, and the model outputs a set of predicted semantic tags. Simultaneously, using real product semantic tags from the labeled data as ground truth, a loss function (such as cross-entropy loss or contrastive loss) is calculated between the predicted result and the ground truth, and the parameters of the scene reconstruction model are updated through a backpropagation algorithm.

[0094] Optionally, the scene reconstruction model can also be trained using a multi-task learning approach. In addition to the main task of predicting semantic labels, auxiliary tasks can be introduced, such as product category classification, attribute recognition, or bounding box regression. These auxiliary tasks can provide additional supervision signals, helping the model better understand local features and semantic information in the image, thereby improving the accuracy of semantic label prediction in the main task.

[0095] According to the above embodiments, by training with discrete semantic identifiers, the scene reconstruction model can learn the mapping relationship from complex scene visual features to a structured semantic space. This training method not only avoids the difficulty of regressing continuous high-dimensional vectors in traditional methods, but also enhances the model's ability to perceive semantic boundaries through discretization constraints, thereby enabling more accurate parsing of corresponding product semantic identifiers from target scene images in practical applications.

[0096] To facilitate understanding of the above technical solutions, a specific application example is provided below. Figure 3 This is a schematic diagram illustrating an example of the application. (For example...) Figure 3As shown in the example, the product recognition method based on spatial visual semantic understanding is divided into a product semantic identifier construction stage and a scene reconstruction model training and inference stage. The specific processing flow of the product semantic identifier construction stage is as follows: After preparing all the product data, firstly, the Qwen2.5-VL model, which has been fine-tuned, is used to generate corresponding vector representations (embeddings) for each product image. This vector is used to carry all the visual feature information of the product. After all the feature vectors of the products have been generated, this set of vectors is input into the RQ-VAE model for training to build a codebook based on semantic encoding. Next, using the trained RQ-VAE model, the feature vector of each product in the recommended product library is quantized and encoded to obtain the corresponding semantic ID. This semantic ID serves as the discrete representation of the corresponding product in a unified semantic space. The training and inference process of the scene reconstruction model is as follows: After obtaining the semantic IDs of all products, a scene reconstruction dataset is constructed using the semantic IDs and scene images. In this dataset, each scene image corresponds to several product semantic ID labels. Then, the Qwen3-VL model is trained using the constructed scene reconstruction dataset, allowing the model to learn to establish a mapping relationship from scene images to the set of semantic IDs. The trained model is the final scene reconstruction model. After the model training is complete, when the user inputs a scene image that needs to be reconstructed, the scene reconstruction model will directly output the semantic ID of the corresponding product in the image. Finally, based on the obtained semantic ID, a mapping query is performed in the product library to obtain the corresponding product ID, and the final recommendation result (reconstruction result) is output.

[0097] Specifically, in this application example, the generation of product feature vectors is a preprocessing step, and its main function is to provide standardized continuous feature representations for subsequent semantic discretization encoding.

[0098] The specific processing flow is as follows: Input all product images from the full product data into the finely tuned Qwen2.5-VL model. The model will encode the input images and output a fixed-dimensional vector representation. This vector contains various visual feature information such as the product's color, structure, shape, and style. After this step is completed, all products in the full product data will have corresponding feature vector representations.

[0099] In this application example, RQ-VAE semantic encoding is the core step in transforming continuous feature vectors into discrete semantic IDs.

[0100] This step, based on a multi-level residual quantization structure, maps the input embedding layer by layer to different levels of codebook indices. Each quantization layer outputs a discrete encoded value, and the combination of multiple levels of encoded indices forms a fixed-length semantic ID. For example, semantic ID = (c1, c2, c3, …, cn), where each ci (i = 1, 2, 3, …, n) corresponds to the encoding result of each quantization layer. Each product has a unique semantic ID. Through this step, the transformation from continuous vectors to discrete semantic identifiers is realized, and a complete product semantic encoding library is constructed.

[0101] In this application example, the training process for the scene reconstruction model is as follows: First, a scene reconstruction dataset (approximately 30,000 scene images) is constructed. Each scene composite image contains a set of semantic IDs for the corresponding products within the scene. The input to this scene reconstruction dataset is a single scene composite image, and the output is the set of semantic IDs for the products contained in that composite image. The scene reconstruction model is then fine-tuned through supervised training to ensure that the model's output is consistent with the real semantic IDs. This allows the model to learn the correspondence between scene images and product semantic identifiers. After training, the model possesses the ability to directly generate multiple semantic IDs from complex scene images.

[0102] In this application example, the scene reconstruction model inference process consists of two parts: The first part is the reasoning from scene images to semantic IDs. During the reasoning process, the model receives scene images input by the user, extracts the semantic information of the scene images, and then generates multiple semantic IDs. Each semantic ID corresponds to a product entity in the scene image. The output of the model is a discrete token sequence, rather than a continuous vector, which eliminates the vector similarity search step in existing technologies and realizes the direct generation of product semantic identifiers.

[0103] The second part involves semantic ID mapping and product restoration. After obtaining the semantic ID predicted by the model, the system determines the corresponding target product by querying a pre-established semantic ID-product ID mapping table. If the prediction result contains multiple semantic IDs, the system performs lookup operations on each ID separately, then aggregates the results, and finally outputs a list of matched products as the recommendation result.

[0104] Through the complete process described above, end-to-end automated processing from complex scene images to specific product recommendations can be achieved. This method abandons the high computational overhead of existing technologies based on continuous vector similarity retrieval, and instead uses discrete semantic tags to achieve accurate matching, improving retrieval efficiency in scenarios with large-scale product databases and effectively solving the alignment problem between visual features and product semantics. In addition, because semantic tags have a fixed encoding structure and clear semantic boundaries, the system can more stably handle complex scenarios such as occlusion, lighting changes, and the coexistence of multiple objects, significantly improving the robustness and accuracy of product recognition, and providing strong technical support for applications such as smart retail, home furnishing matching, and personalized recommendations.

[0105] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0106] Figure 4 This is a schematic block diagram of a product recognition device 400 based on spatial visual semantic understanding provided in one embodiment of this disclosure. Figure 4 As shown, the product recognition device 400 based on spatial visual semantic understanding includes: The semantic identifier determination module 410 is used to process the semantic feature vector of each product in the product library to be recommended based on the quantization model to obtain the semantic identifier corresponding to each product. The semantic identifier set determination module 420 is used to determine the semantic identifier set corresponding to the scene image in the scene reconstruction dataset based on the semantic identifier corresponding to each product; wherein, the semantic identifier set includes the semantic identifier of at least one product in the scene image; The scene reconstruction model training module 430 is used to train a scene reconstruction model based on scene images in the scene reconstruction dataset and their corresponding semantic identifier sets; wherein, the scene reconstruction model is used to determine the semantic identifier of at least one recommended product corresponding to the target scene image based on the target scene image.

[0107] In some embodiments, such as Figure 5 As shown, the product recognition device 400 based on spatial visual semantic understanding also includes: Quantization model training module 440, such as Figure 6 As shown, it includes: The full-product vector generation unit 441 is used to generate semantic feature vectors corresponding to each product in the full-product database based on the image of each product in the full-product database and using a vector generation model; wherein, the full-product database includes a library of products to be recommended; The quantization model training unit 442 is used to train a quantization model based on the semantic feature vector corresponding to each product in the full product database.

[0108] In some embodiments, such as Figure 7 As shown, the quantization model training module 440 also includes: The product vector generation unit 443 is used to generate semantic feature vectors for each product in the product library based on the image of each product in the product library, using a vector generation model.

[0109] In some embodiments, the vector generation model is obtained by fine-tuning the multimodal vector model based on an industry product database; the full product database includes an industry product database, which includes a product database to be recommended.

[0110] In some embodiments, the quantization model is based on a multi-level residual quantization structure; the semantic identifier includes multiple discrete encoded values, which correspond to multiple quantization layers in the multi-level residual quantization structure.

[0111] In some embodiments, the quantization model includes a residual quantization variational autoencoder (RQ-VAE) model.

[0112] In some embodiments, such as Figure 8 As shown, the product recognition device 400 based on spatial visual semantic understanding also includes: The mapping relationship construction module 450 is used to construct and store the mapping relationship between the product identifier and semantic identifier of each product in the product library to be recommended; wherein, the mapping relationship is used to determine the product identifier of at least one recommended product based on the semantic identifier of at least one recommended product in the target scene image.

[0113] Figure 9 This is a schematic block diagram of a product recognition device 900 based on spatial visual semantic understanding, provided in another embodiment of this disclosure. Figure 9 As shown, the product recognition device 900 based on spatial visual semantic understanding includes: The semantic identifier determination module 910 is used to determine the semantic identifier of at least one recommended product corresponding to the target scene image based on the target scene image and using a scene reconstruction model; wherein, the scene reconstruction model is trained based on the scene images and their corresponding semantic identifier sets in the scene reconstruction dataset, and the semantic identifier set includes the semantic identifier of at least one product in the scene image; The product identifier determination module 920 is used to determine the product identifier of at least one recommended product based on the semantic identifier of at least one recommended product.

[0114] In some embodiments, the scene reconstruction model is trained using any one of the methods described in the above-disclosed embodiments of the product recognition method based on spatial visual semantic understanding.

[0115] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0116] Figure 10 A schematic block diagram of an example electronic device 1000 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0117] like Figure 10 As shown, the electronic device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1002 or a computer program loaded from a storage unit 1008 into a random access memory (RAM) 1003. The RAM 1003 may also store various programs and data required for the operation of the electronic device 1000. The computing unit 1001, ROM 1002, and RAM 1003 are interconnected via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0118] Multiple components in electronic device 1000 are connected to I / O interface 1005, including: input unit 1006, such as keyboard, mouse, etc.; output unit 1007, such as various types of displays, speakers, etc.; storage unit 1008, such as disk, optical disk, etc.; and communication unit 1009, such as network card, modem, wireless transceiver, etc. Communication unit 1009 allows device 1000 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0119] The computing unit 1001 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1001 performs the various methods and processes described above, such as a product recognition method based on spatial visual semantic understanding. For example, in some embodiments, the product recognition method based on spatial visual semantic understanding can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 1008. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 1000 via ROM 1002 and / or communication unit 1009. When the computer program is loaded into RAM 1003 and executed by the computing unit 1001, one or more steps of the product recognition method based on spatial visual semantic understanding described above can be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured in any other suitable manner (e.g., by means of firmware) to perform a product recognition method based on spatial visual semantic understanding.

[0120] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0121] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0122] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0123] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0124] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0125] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0126] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0127] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A product recognition method based on spatial visual semantic understanding, characterized in that, include: The semantic feature vector of each product in the product library to be recommended is processed based on the quantization model to obtain the semantic identifier corresponding to each product. Based on the semantic identifier corresponding to each product, a set of semantic identifiers corresponding to the scene images in the scene reconstruction dataset is determined; wherein, the set of semantic identifiers includes the semantic identifiers of at least one product in the scene images; A scene reconstruction model is trained based on the scene images in the scene reconstruction dataset and their corresponding semantic identifier set; wherein, the scene reconstruction model is used to determine the semantic identifier of at least one recommended product corresponding to the target scene image based on the target scene image.

2. The method according to claim 1, characterized in that, Also includes: Based on the image of each product in the full product database, a vector generation model is used to generate a semantic feature vector corresponding to each product in the full product database; wherein, the full product database includes the product library to be recommended; The quantization model is trained based on the semantic feature vector corresponding to each product in the full product database.

3. The method according to claim 2, characterized in that, Also includes: Based on the image of each product in the product library to be recommended, the semantic feature vector of each product in the product library to be recommended is generated using the vector generation model.

4. The method according to claim 2 or 3, characterized in that, The vector generation model is obtained by fine-tuning the multimodal vector model based on the industry product library; the full product database includes the industry product library, and the industry product library includes the product library to be recommended.

5. The method according to any one of claims 1-3, characterized in that, The quantization model is based on a multi-level residual quantization structure; the semantic identifier includes multiple discrete encoded values, which correspond to multiple quantization layers in the multi-level residual quantization structure.

6. The method according to claim 5, characterized in that, The quantization model includes the residual quantization variational autoencoder (RQ-VAE) model.

7. The method according to any one of claims 1-3, characterized in that, Also includes: Construct and store the mapping relationship between the product identifier and semantic identifier of each product in the product library to be recommended; wherein, the mapping relationship is used to determine the product identifier of the at least one recommended product based on the semantic identifier of the at least one recommended product in the target scene image.

8. A product recognition method based on spatial visual semantic understanding, characterized in that, include: Based on the target scene image, a scene reconstruction model is used to determine the semantic identifier of at least one recommended product corresponding to the target scene image; wherein, the scene reconstruction model is trained based on the scene images and their corresponding semantic identifier sets in the scene reconstruction dataset, and the semantic identifier set includes the semantic identifier of at least one product in the scene image; Based on the semantic identifier of the at least one recommended product, the product identifier of the at least one recommended product is determined.

9. The method according to claim 8, characterized in that, The scene reconstruction model is trained based on the method described in any one of claims 1-7.

10. A product recognition device based on spatial visual semantic understanding, characterized in that, include: The semantic identifier determination module is used to process the semantic feature vector of each product in the product library to be recommended based on the quantization model, so as to obtain the semantic identifier corresponding to each product. The semantic identifier set determination module is used to determine the semantic identifier set corresponding to the scene image in the scene reconstruction dataset based on the semantic identifier corresponding to each product; wherein, the semantic identifier set includes the semantic identifier of at least one product in the scene image; The scene reconstruction model training module is used to train a scene reconstruction model based on scene images in the scene reconstruction dataset and their corresponding semantic identifier sets; wherein, the scene reconstruction model is used to determine the semantic identifier of at least one recommended product corresponding to the target scene image based on the target scene image.

11. A product recognition device based on spatial visual semantic understanding, characterized in that, include: The semantic identifier determination module is used to determine the semantic identifier of at least one recommended product corresponding to the target scene image based on the target scene image and using a scene reconstruction model; wherein, the scene reconstruction model is trained based on scene images and their corresponding semantic identifier sets in the scene reconstruction dataset, and the semantic identifier set includes the semantic identifier of at least one product in the scene image; The product identifier determination module is used to determine the product identifier of the at least one recommended product based on the semantic identifier of the at least one recommended product.

12. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the method of any one of claims 1-9.

13. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-9.

Images