Article attribute recognition model training method and article attribute recognition method and device

Abstract

The invention discloses a training method of an article attribute recognition model, and an article attribute recognition method and device, and relates to the technical field of computers. A specific embodiment of the model training method comprises the following steps: respectively carrying out feature coding processing on obtained main picture information and title information of an article to obtain a picture coding vector and a title text coding vector; performing feature enhancement processing on the picture coding vector based on a memory network, and performing feature fusion on the picture coding vector after feature enhancement processing and the title text coding vector to generate a multi-modal attribute feature; according to the multi-modal attribute features, training by using a weak supervised learning strategy to obtain an article attribute recognition model; the sample labeling cost can be reduced, and the training efficiency and the accuracy of the recognition result can be improved. A specific embodiment of the identification method comprises the following steps: inputting main image information and title information of a target article into an article attribute identification model to obtain an attribute identification result of the target article, and displaying the attribute identification result to a user; and the identification accuracy is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a training method for an object attribute recognition model, an object attribute recognition method and a device. Background Art

[0002] In order to make the item obtain more search traffic, in addition to configuring some basic information related to the item itself, some attribute information unrelated to the item will also be configured in the attribute position, resulting in a lower matching degree between the overall basic information of the item and the item.

[0003] In the process of implementing the present invention, the inventors found that there are at least the following problems in the related art:

[0004] The sample annotation cost used in the current object attribute recognition model training is high, the training efficiency is low, and the accuracy of the recognition results is poor. Summary of the invention

[0005] In view of this, the training method, item attribute recognition method and device of the object attribute recognition model provided by the embodiments of the present invention can use a small amount of samples for weak supervision training, and enhance the features of the image coding vector based on the memory network, thereby reducing the cost of sample annotation and improving the training efficiency and the accuracy of the recognition results.

[0006] To achieve the above object, according to one aspect of an embodiment of the present invention, a method for training an item attribute recognition model is provided, comprising:

[0007] Perform feature encoding processing on the main image information and title information of the acquired items respectively to obtain the image encoding vector and the title text encoding vector;

[0008] Based on the pre-built memory network, the image coding vector is enhanced, and the enhanced image coding vector and the title text coding vector are fused to generate multimodal attribute features.

[0009] The object attribute recognition model is trained using weakly supervised learning strategies based on multimodal attribute features.

[0010] Optionally, feature encoding is performed on the main image information and title information of the acquired item to obtain an image encoding vector and a title text encoding vector, including:

[0011] An image encoder is used to convert the main image information of the item into an image encoding vector; or a word embedding or transformer model is used to convert the title information of the item into a title text encoding vector.

[0012] Optionally, the memory network includes a dynamic memory network and a static memory network; and the image encoding vector is subjected to feature enhancement processing based on the pre-built memory network, including:

[0013] In response to the number of samples being zero, feature enhancement processing is performed on the image coding vector based on historical image features in the dynamic memory network; or, in response to the number of samples being less than a preset threshold, feature enhancement processing is performed on the image coding vector based on historical image features in the static memory network.

[0014] Optionally, the image encoding vector after feature enhancement processing and the title text encoding vector are feature fused to generate multimodal attribute features, including:

[0015] Determine the correlation between the image encoding vector after feature enhancement and the title text encoding vector based on the multi-head self-attention mechanism;

[0016] According to the correlation relationship, the feature-enhanced image encoding vector and the title text encoding vector are fused to generate multimodal attribute features.

[0017] Optionally, before performing feature enhancement processing on the image encoding vector based on the pre-built memory network, the process further includes:

[0018] The memory of the pre-constructed memory network is updated based on the picture coding vector to obtain an updated memory network, so as to optimize the picture coding vector according to the updated memory network.

[0019] Optionally, feature enhancement processing is performed on the image encoding vector based on a pre-built memory network, including:

[0020] Based on the cross-attention mechanism, the image coding vector and the historical image features stored in the memory network are weighted and summed to obtain the weighted image coding vector, so as to perform feature fusion based on the weighted image coding vector and the title text coding vector.

[0021] According to a second aspect of an embodiment of the present invention, there is provided a method for identifying an attribute of an item, comprising:

[0022] Get the main image information and title information of the target item;

[0023] Inputting the main image information and the title information into an item attribute recognition model, and the item attribute recognition model outputs an attribute recognition result of the target item; wherein the item attribute recognition model is obtained by any method in the first aspect of the embodiment of the present invention;

[0024] The attribute recognition results of the item are displayed to the user.

[0025] According to a third aspect of an embodiment of the present invention, there is provided a training device for an item attribute recognition model, comprising:

[0026] A feature encoding module is used to perform feature encoding processing on the main image information and title information of the acquired item, respectively, to obtain an image encoding vector and a title text encoding vector;

[0027] A feature fusion module is used to perform feature enhancement processing on the image coding vector based on a pre-built memory network, and to perform feature fusion on the image coding vector after feature enhancement processing and the title text coding vector to generate multi-modal attribute features;

[0028] The model training module is used to train the object attribute recognition model based on multimodal attribute features using a weakly supervised learning strategy.

[0029] According to a fourth aspect of an embodiment of the present invention, there is provided an object attribute recognition device, comprising:

[0030] An information acquisition module is used to obtain the main image information and title information of the target item;

[0031] A model recognition module, used to input the main image information and the title information into an item attribute recognition model, and the item attribute recognition model outputs an attribute recognition result of the item; wherein the item attribute recognition model is obtained by any method of the first aspect of the embodiment of the present invention;

[0032] The result display module is used to display the attribute recognition results of the item to the user.

[0033] According to a fifth aspect of an embodiment of the present invention, there is provided an electronic device, including:

[0034] one or more processors;

[0035] a storage device for storing one or more programs,

[0036] When one or more programs are executed by one or more processors, the one or more processors implement the method of any of the above embodiments.

[0037] According to a sixth aspect of an embodiment of the present invention, there is provided a computer-readable medium on which a computer program is stored, and when the program is executed by a processor, the method of any of the above embodiments is implemented.

[0038] According to a seventh aspect of an embodiment of the present invention, there is provided a computer program product, including a computer program, which implements the method of any of the above embodiments when the computer program is executed by a processor.

[0039] An embodiment of the above invention has the following advantages or beneficial effects: by performing feature encoding processing on the main image information and title information of the acquired item respectively, a picture encoding vector and a title text encoding vector are obtained; based on the pre-constructed memory network, feature enhancement processing is performed on the picture encoding vector, and the feature enhancement processed picture encoding vector and title text encoding vector are feature fused to generate multimodal attribute features; according to the multimodal attribute features, a weakly supervised learning strategy is used to train an item attribute recognition model; thereby, a small number of samples can be used for weakly supervised training, and feature enhancement is performed on the picture encoding vector based on the memory network, which reduces the cost of sample annotation, improves training efficiency and the accuracy of recognition results. By acquiring the main image information and title information of the target item; inputting the main image information and title information into the item attribute recognition model, the item attribute recognition model outputs the attribute recognition result of the target item; wherein the item attribute recognition model is obtained by any method in the first aspect of the embodiment of the present invention; the item attribute recognition result is displayed to the user; and the accuracy of item attribute recognition and user experience are improved.

[0040] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0042] Figure 1 is a schematic diagram of the main process of the training method of the object attribute recognition model according to an embodiment of the present invention;

[0043] Figure 2 is a schematic diagram of a method for training an item attribute recognition model according to an optional embodiment of the present invention;

[0044] Figure 3 is a schematic diagram of an object attribute recognition method according to an embodiment of the present invention;

[0045] Figure 4 is a schematic diagram of main modules of a training device for an object attribute recognition model according to an embodiment of the present invention;

[0046] Figure 5 is a schematic diagram of main modules of an object attribute recognition device according to an embodiment of the present invention;

[0047] Figure 6 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;

[0048] Figure 7 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0049] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0050] It should be noted that the acquisition, storage and application of personal information involved in the embodiments of the present invention are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0051] In order to make the item gain more search traffic, in addition to configuring some basic information related to the item itself, some attribute information unrelated to the item is also configured in the attribute position, resulting in a low matching degree between the overall item basic information and the item. The current item attribute recognition model training has a high sample labeling cost, low training efficiency, and poor recognition result accuracy.

[0052] In view of this, according to a first aspect of an embodiment of the present invention, a method for training an object attribute recognition model is provided.

[0053] Figure 1 FIG. 1 is a schematic diagram of the main process of the method for training an item attribute recognition model according to an embodiment of the present invention. Figure 1 As shown, the training method of the object attribute recognition model according to the embodiment of the present invention mainly includes the following steps S101 to S103.

[0054] Step S101, feature encoding processing is performed on the main image information and title information of the acquired item respectively to obtain an image encoding vector and a title text encoding vector.

[0055] In the field of e-commerce, the main image information of an item refers to the main image displayed on the e-commerce platform, which can intuitively present the appearance, color, material and other characteristics of the item. The title information is a text description of the item, which is used to briefly summarize the core characteristics of the product, including key information such as brand, model, function, etc., such as "185L double-door refrigerator for small household use, air-cooled and frost-free" and so on.

[0056] When encoding the features of the main image information, you can use a convolutional neural network to extract features. Through the combination of multiple convolutional layers and pooling layers, you can gradually extract high-order features of the image, and finally generate a picture encoding vector. This picture encoding vector represents important visual features in the image, such as edges, textures, etc. In addition, you can also use pre-trained image classification models for image encoding. These models can be used directly to extract features of the image and generate corresponding picture encoding vectors. The picture encoding vector is a high-dimensional vector obtained by the above encoding method. It represents the visual features of the image and can be used to compare or match with other objects or visual information.

[0057] To encode the features of the title information, we can map each word to a vector space with a fixed dimension based on word embedding technology to capture the semantic relationship between words, thereby generating a title text encoding vector. We can also use a natural language processing model based on the Transformer architecture to process the title information. These models can capture the contextual information of the text through the deep self-attention mechanism and generate richer title text encoding vectors. The title text encoding vector is a vector that represents the semantic information of the title after encoding, and can be used to capture key information in the text.

[0058] Step S102, based on the pre-built memory network, feature enhancement processing is performed on the image coding vector, and feature fusion is performed on the image coding vector after feature enhancement processing and the title text coding vector to generate multimodal attribute features.

[0059] The pre-built memory network is a neural network structure used to enhance the model's processing capabilities. It can capture and store historical information to enhance the model's feature expression capabilities in specific tasks. When processing image encoding vectors, the memory network can store historical information or contextual information related to the image, helping the model to more accurately capture and emphasize the key features of the image during the encoding process. The memory network can be constructed by introducing memory units (such as long short-term memory networks or gated recurrent units, etc.), which can continuously update their stored memory content during the training process, thereby providing data basis for subsequent feature extraction or feature fusion.

[0060] Feature enhancement refers to improving the expressiveness of features through additional information or network structures based on the original features, so that the model can capture richer context or semantic information. Under the action of the memory network, the image encoding vector can enhance its expressiveness of key elements of the image by interacting with the historical information in the memory unit. For example, when processing product images, the memory network may enhance the features of the current image through historical information of similar images, so that the image encoding vector can more accurately represent the details or semantic information of the image. Feature enhancement can be achieved by introducing a memory module into the original network architecture, or by dynamically selecting important historical information through the attention mechanism, thereby updating and enhancing the image encoding vector.

[0061] Feature fusion is performed between the image encoding vector and the title text encoding vector after feature enhancement, that is, multimodal information is combined to more comprehensively characterize the attributes of the item. Feature fusion can be achieved in many ways, such as splicing fusion and weighted fusion. Splicing fusion is to directly splice the image encoding vector and the text encoding vector together according to the dimension to form a new high-dimensional feature vector. Weighted fusion is to assign different weights according to the importance of the image encoding vector and the title text encoding vector before splicing, so that important features can get more attention in the fusion process. Through feature fusion, the model can use the feature information of the image and the text at the same time to generate more expressive multimodal attribute features. Multimodal attribute features refer to feature representations that combine multiple different types of data. They not only contain the visual information of the image, but also integrate the semantic information of the text, so as to more comprehensively describe the attributes of the item. Multimodal features can consider multiple information sources at the same time, thereby improving the accuracy and effect of item attribute recognition. Multimodal attribute features generated by feature enhancement through memory network and fusion of image and text features can effectively capture the details of the item in different modalities and provide powerful feature representation for subsequent machine learning models.

[0062] Step S103, training an object attribute recognition model using a weakly supervised learning strategy based on multimodal attribute features.

[0063] The weakly supervised learning strategy is a method of using partially labeled or low-quality labeled data for model training, which can reduce the dependence on a large amount of high-quality labeled data. Unlike fully supervised learning, weakly supervised learning can handle situations where labeled data is scarce, inaccurate, or incomplete, and effectively learn the ability to classify or recognize on incompletely labeled data. When training an item attribute recognition model, weakly supervised learning can be implemented in a variety of ways. For example, a small amount of accurately labeled data can be combined with a large amount of unlabeled data for semi-supervised learning. The model is initially trained on labeled data, and then self-learns on unlabeled data to gradually improve recognition capabilities. Pseudo-labeling methods can also be used, that is, preliminary predictions are made on unlabeled data, and these prediction results are used as temporary labels for training to gradually optimize the model. In addition, weakly supervised learning can also be learned through multiple instances. The model can learn based on instances that contain some targets, further improving performance on weakly labeled data sets.

[0064] The item attribute recognition model can automatically identify the key information and attributes of an item by analyzing the multimodal attribute features of the item. During the training process, the model can learn based on the multimodal attribute features, and gradually understand and capture the various attributes of the item, such as color, material, brand, type, etc., through a weak supervision strategy, so that the model can accurately extract and identify the core attributes of the item from the given multimodal input. Through the weak supervision learning strategy, the model can continuously iterate and optimize in the case of sparse or incomplete annotations, and ultimately learn high-quality item attribute recognition capabilities.

[0065] The embodiment of the present invention performs feature encoding processing on the main image information and title information of the acquired item respectively to obtain a picture encoding vector and a title text encoding vector; performs feature enhancement processing on the picture encoding vector based on a pre-constructed memory network, performs feature fusion on the picture encoding vector and the title text encoding vector after the feature enhancement processing to generate multimodal attribute features; and obtains an item attribute recognition model by training using a weakly supervised learning strategy according to the multimodal attribute features; thereby, a small amount of samples can be used for weakly supervised training, and the picture encoding vector is feature enhanced based on the memory network, which reduces the cost of sample annotation and improves the training efficiency and the accuracy of the recognition result.

[0066] Optionally, feature encoding processing is performed on the main image information and title information of the acquired item respectively to obtain an image encoding vector and a title text encoding vector, including: using an image encoder to convert the main image information of the item into an image encoding vector; or, using a word embedding or transformer model to convert the title information of the item into a title text encoding vector.

[0067] After obtaining the main image information and title information of the item, the information is first feature encoded to convert it into a numerical vector form that the model can understand. For the main image information, an image encoder can be used for processing, where the image encoder can be a convolutional neural network or other pre-trained deep learning models, such as ResNet (Residual Network), VGG (Visual Geometry Group), etc. These image encoders extract low-level, medium-level and high-level features of the image through multiple convolutional layers, and finally convert the main image information of the item into an image encoding vector. This image encoding vector is a high-dimensional numerical vector that contains the visual information of the image, such as color, texture, shape, etc., and can effectively represent the key content in the image. The title information of an item usually exists in the form of text. In order to convert this text information into a numerical vector that can be processed by the model, word embedding technology or a transformer-based model can be used. Word embedding technology can map each word to a vector space of fixed dimension, capture the semantic relationship between words, and convert the entire title information into a title text encoding vector; transformer models can also be used. These models capture the contextual relationship in the title through the self-attention mechanism, thereby generating a more complex and semantically informative text encoding vector. This method is more advantageous than simple word embedding and can handle the complex semantics and contextual information of long texts.

[0068] The embodiment of the present invention processes the main image information through an image encoder, and the generated image encoding vector can represent the visual features of the object, and processes the title information through a word embedding or transformer model, and the generated title text encoding vector contains the semantic features of the title. These two encoding vectors can provide rich feature representations for subsequent multimodal tasks, such as classification and recognition.

[0069] Optionally, the memory network includes a dynamic memory network and a static memory network; feature enhancement processing is performed on the image coding vector based on a pre-constructed memory network, including: in response to the number of samples being zero, feature enhancement processing is performed on the image coding vector based on historical image features in the dynamic memory network; or, in response to the number of samples being less than a preset number threshold, feature enhancement processing is performed on the image coding vector based on historical image features in the static memory network.

[0070] When performing feature enhancement processing on the image coding vector, the dynamic memory network and the static memory network are used to enhance the features under different sample numbers to deal with the situation where the features are incomplete due to insufficient or missing sample numbers. When the number of processed samples is zero, the dynamic memory network is called for feature enhancement. The dynamic memory network is mainly used to capture the continuously updated historical image features and can self-learn and adjust according to the previously processed image features. In this case, the historical image features in the dynamic memory network will be activated and applied to the current image coding vector, thereby helping the model to make up for the key information that may be lost due to the lack of sample data. This enhancement processing enables the model to rely on historical feature information to generate a relatively complete image coding vector in the absence of real-time data. If the number of samples is less than the preset number threshold, processing is performed based on the static memory network. The static memory network stores fixed historical image features. These features will not be updated with time or the addition of new data. They are more focused on managing long-term accumulated and stable historical image features. In this case, the historical image features in the static memory network will be called to supplement and strengthen the current image coding vector. Especially when the number of samples is insufficient, the static memory network can help provide additional information to improve the richness and accuracy of feature expression.

[0071] The embodiment of the present invention uses such a feature enhancement mechanism. When the model processes the picture coding vector, no matter the number of samples is zero or small, it can use the features in the historical memory network to complete or enhance the feature expression of the picture, thereby improving the performance of the model in the case of data scarcity. The enhanced picture coding vector can more accurately represent the core information of the picture.

[0072] Figure 2 FIG. 1 is a schematic diagram of a method for training an object attribute recognition model according to an optional embodiment of the present invention. Figure 2 As shown in the figure, the image encoding vector after feature enhancement and the title text encoding vector are fused to generate multimodal attribute features, including:

[0073] Step S201, determining the correlation between the image encoding vector after feature enhancement processing and the title text encoding vector based on the multi-head self-attention mechanism.

[0074] Step S202: perform feature fusion on the image encoding vector after feature enhancement and the title text encoding vector according to the association relationship to generate multimodal attribute features.

[0075] When the image encoding vector and the title text encoding vector after feature enhancement are subjected to feature fusion, the association relationship between them can be determined based on the multi-head self-attention mechanism. In step S201, the multi-head self-attention mechanism is a mechanism for capturing the relationship between the elements in the sequence. The input data is processed in parallel by multiple "attention heads" to capture the association of different dimensions, so that the model can fully understand the semantic relationship between the image and the text. Specifically, the image encoding vector and the title text encoding vector after feature enhancement are used as the input of the multi-head self-attention mechanism. For each attention head, the correlation between the image encoding vector and the title text encoding vector is calculated by the query, key and value (Query, Key, Value) mechanism. For example, the query vector comes from the image encoding vector, the key vector and the value vector come from the title text encoding vector, and the attention weights between the image and the text are obtained by calculating the dot product of the query vector and the key vector and performing normalization processing. These weights represent the association strength between each image encoding vector and the title text encoding vector. Through the parallel calculation of multiple attention heads, the association information at different semantic levels can be obtained.

[0076] After determining the correlation between the image coding vector and the title text coding vector, feature fusion is performed. In step S202, the feature fusion process can selectively weight and combine the image coding vector and the title text coding vector according to the attention weight. Specifically, based on different attention weights, the model can assign different weights to the key features in the image coding vector and the title text coding vector, so that in the feature fusion process, the features with strong correlation can contribute more information to the final fusion features. Through such weighted fusion, the image coding vector and the title text coding vector are effectively combined to generate a new multimodal attribute feature vector. The generated multimodal attribute feature vector is a fusion representation of the visual features of the image and the semantic features of the text. It not only contains the visual information of the object, but also combines the semantic information carried by the text title, which can better describe the attributes of the object. The introduction of the multi-head self-attention mechanism makes full use of the fine-grained associations between each modality in the feature fusion process, thereby improving the accuracy of model training.

[0077] Optionally, before performing feature enhancement processing on the image coding vector based on the pre-constructed memory network, it also includes: updating the memory of the pre-constructed memory network based on the image coding vector to obtain an updated memory network, so as to optimize the image coding vector according to the updated memory network.

[0078] Update the memory of the pre-built memory network so that the memory network can effectively store and manage the latest feature information. First, take the current image encoding vector as input to update the memory of the memory network, which includes multiple memory cells that are used to store previously processed image features and related context information. When updating the memory, different strategies can be adopted, such as incremental update or weighted update. The incremental update method is to directly add the new image encoding vector to the memory network to form a new memory item; the weighted update is to calculate the weight value based on the similarity between the current image encoding vector and the stored memory item to adjust the influence of each memory item. After the memory is updated, the current image encoding vector can be optimized based on the updated memory network. The historical feature information in the memory network will provide a reference for the current image encoding vector, thereby helping it to better characterize the features of the current item. For example, by performing similarity matching between the current image coding vector and the historical image features in the memory network, the memory items most relevant to the current image coding vector can be extracted. These memory items can provide supplementary information for the current image coding vector. Especially when there are few current samples or noise, the accuracy of the current image coding vector can be significantly improved by borrowing the knowledge of historical image features.

[0079] Through the above-mentioned update and optimization process, the embodiment of the present invention enables the memory network to not only dynamically adapt to changes in new data, but also make full use of historical information in the feature enhancement processing stage, thereby enhancing the model's ability to understand the current task. By combining the updated memory network, the image encoding vector can be significantly optimized, thereby improving the overall performance of the model.

[0080] Optionally, feature enhancement processing is performed on the image coding vector based on a pre-constructed memory network, including: weighted summing the image coding vector with historical image features stored in the memory network based on a cross-attention mechanism to obtain a weighted image coding vector, and performing feature fusion based on the weighted image coding vector and the title text coding vector.

[0081] The current image encoding vector is used as the query vector, and the historical image features stored in the memory network are used as the key vector and value vector. The similarity between the query vector and the key vector is calculated, which can be achieved through dot product or other similarity measurement methods. The result of this step is to generate attention weights, which represent the degree of association between the current image encoding vector and each historical image feature; by normalizing these attention weights, a probability distribution can be obtained, which is used to characterize the importance of each historical image feature in enhancing the current image encoding. The attention weights are used to weight the sum of the historical image features and the current image encoding vector. The influence of each historical image feature will be adjusted according to its degree of association with the current image encoding vector, thereby forming a weighted image encoding vector. The weighted image encoding vector combines the advantages of the current image features and the historical image features, making the representation of the current image richer and more accurate. After obtaining the weighted image encoding vector, it can be feature fused with the title text encoding vector.

[0082] By utilizing the cross-attention mechanism, the embodiment of the present invention enables the current image encoding vector to not only absorb the useful information of historical image features, but also effectively integrate with text features, thereby ultimately improving the performance of the model in multimodal learning tasks and ensuring the dynamics and adaptability of feature enhancement, so that the model can still maintain efficient feature extraction and expression capabilities when facing different types of input data.

[0083] According to a second aspect of an embodiment of the present invention, a method for identifying attributes of an object is provided.

[0084] Figure 3 Schematic diagram of the main process of the object attribute recognition method according to an embodiment of the present invention; Figure 3 As shown, the object attribute recognition method according to the embodiment of the present invention mainly includes the following steps S301 to S303.

[0085] Step S301, obtaining the main image information and title information of the target item.

[0086] Step S302, input the main image information and the title information into the item attribute recognition model, and the item attribute recognition model outputs the attribute recognition result of the target item; wherein the item attribute recognition model is obtained by any method in the first aspect of the embodiment of the present invention.

[0087] Step S303: Display the attribute recognition result of the item to the user.

[0088] By obtaining the main image information and title information of the target item from the database or the network, this information may include the picture of the product and the text description related to the product. The main image information and title information obtained are respectively input into the item attribute recognition model, which can automatically identify and output various attributes of the target item, which may include the category, color, size, material, brand and other key information of the item. The output result of the model is the attribute recognition result of the target item. After being processed by the multi-layer neural network, the attribute result can be presented in a structured form (such as a label or attribute value pair). The generated item attribute recognition result is displayed to the user. The display method can be to present the recognized attributes in the front-end user interface in the form of text or visualization, so that the user can intuitively view and understand the key attributes of the item. Through this series of steps, the user can quickly understand the detailed information of the item and make a decision, which improves the accuracy of item attribute recognition and user experience.

[0089] The embodiment of the present invention obtains the main image information and title information of the target item; inputs the main image information and title information into an item attribute recognition model, and the item attribute recognition model outputs an attribute recognition result of the target item; wherein the item attribute recognition model is obtained by any method in the first aspect of the embodiment of the present invention; and displays the attribute recognition result of the item to the user; thereby improving the accuracy of item attribute recognition and user experience.

[0090] According to a preferred embodiment of the present invention, a training method for an object attribute recognition model mainly includes ImageEncoder (image encoder), TextEncoder (text encoder), Cross-modal (image and text multimodal feature fusion) and a memory network.

[0091] ImageEncoder is used to encode the main image information of the item. This embodiment uses the currently open source pre-trained model VIT (Vision Transformer). The core process of this model includes the following four parts: image patch processing (making patches), image patch embedding (patch embedding) and position encoding, encoder (Transformer), multi-layer perceptron (MLP) classification processing, etc. For the TextEncoder part, the first 6 layers of a 12-layer pre-trained Transformer are used to generate the text vector representation of the item title. Only the first 6 layers are used for efficient and effective text encoding processing.

[0092] The Cross-modal (multi-modal feature fusion of images and text) part is mainly used to fuse the image and text features obtained from ImageEncoder and TextEncoder. In order to achieve this fusion, this paper uses a 6-layer Transformer with a multi-head attention mechanism (i.e., the last 6 layers of the aforementioned 12-layer model). The solution has been verified that the ability of multi-head self-attention can achieve full and effective fusion of image and text features, where the title text encoding vector is used as the input of the query (Q), and the image encoding vector is used as the input of the key (K) and value (V). Through this multi-head self-attention mechanism, the correlation between image and text features can be effectively captured, and the full fusion of image and text features can be achieved.

[0093] The memory network (Mermory) provides an effective mechanism to explicitly accumulate and recall knowledge, and improve performance by utilizing relevant historical information. Specifically, an image input x is converted into a feature space, that is, the image completes the vector representation x->v through the above-mentioned ImageEncoder; then v is used to update the memory of the memory network, and a memory M with attribute item classification is constructed, where L is the memory length of each attribute value category, and v is stored in a "slot" of M. Specifically, given the attribute value classification label y∈[1, C] (C represents the number of specific attribute values) of the input image, find the sub-memory M_y corresponding to the attribute value category y. Assuming that the i-th row is used to store this category, then M_yi=v; when updating M, if all rows of M_y are occupied by image features, the probability P_v of predicting each attribute value classification will be calculated for each v, and the entropy value Ent(D) can be calculated based on P_v, where: The information entropy calculation formula is as follows:

[0094]

[0095] If the predicted entropy of v is small, replace the row with the largest entropy in M_y with v. In other words, store the sample with lower predicted entropy in memory M. It is believed that the smaller the information entropy, the less uncertainty the information contained in the sample, and the richer the feature information of a certain attribute value category, so it should be retained in the memory network. Among them, two memory networks are established, M_D represents the dynamic memory network, which refers to the running test phase; and M_S represents the static memory network, which usually runs in the few-shot training phase.

[0096] In addition, a ReadOut mechanism is also included. Given the updated M and the test feature v, this embodiment uses the Cross-attention mechanism to calculate the sample memory network output. C_m = ReadOut(v, M), where in the Cross-attention calculation: Q is represented by v, and K and V are represented by M. The calculation process is introduced by taking the calculation of the y_th row in C_m as an example. The result of the y_th row is calculated by v as Q, K and V as M, and the formula is as follows:

[0097]

[0098] Where: ωq, ωk, ωv and ωo represent the transfer matrices of Q, K, V and output respectively, where: Intuitively, Cm_y is a weighted combination of image features in My, where the weights are based on the cosine similarity between the test and memory image features, and the sample-adaptive classifier Cm is produced from image features rather than from text features that produce the text classifier C.

[0099] During model training, the complete training sample consists of four parts of input data. The construction example is shown in Table 1 below, where Input1 is the title information of the product, Input2 is the main image information of the product, Input_pos is the positive sample attribute information of this sample, Input_neg is the negative sample attribute information of this sample, Label_pos is the positive sample Lable=1 of this sample, and Label_neg is the negative sample Lable=0 of this sample.

[0100] Table 1 Training data

[0101]

[0102] Use Cross Entropy as the similarity loss function, where H y (y) is the symbolic representation of the cross entropy loss function, which is used to measure the difference between the model's predicted results and the actual results; yi′ is the model's true label, that is, the expected output, which is a vector, usually each item is 0 or 1, indicating the actual situation of the current category; yi is the model's predicted probability output value, indicating the model's prediction confidence for each category; log(yi) takes the logarithm of the predicted probability, expressed as a natural logarithm.

[0103]

[0104] This embodiment solves the problem that positive samples are difficult to collect and have high labeling costs. It uses the idea of ​​weak supervision learning to perform zero-training and few-sample learning. At the same time, in order to improve the recognition accuracy and break through the bottleneck of the pre-training model, it combines the memory network to improve the recognition accuracy.

[0105] According to a third aspect of an embodiment of the present invention, a training device for an object attribute recognition model is provided.

[0106] Figure 4 is a schematic diagram of the main modules of the training device for the object attribute recognition model according to an embodiment of the present invention, such as Figure 4 As shown, a training device 400 for an item attribute recognition model includes:

[0107] The feature encoding module 401 is used to perform feature encoding processing on the main image information and title information of the acquired item, respectively, to obtain a picture encoding vector and a title text encoding vector;

[0108] The feature fusion module 402 is used to perform feature enhancement processing on the image coding vector based on the pre-built memory network, and perform feature fusion on the image coding vector after feature enhancement processing and the title text coding vector to generate multi-modal attribute features;

[0109] The model training module 403 is used to train an object attribute recognition model using a weakly supervised learning strategy according to multimodal attribute features.

[0110] Optionally, the feature encoding module 401 is used to:

[0111] An image encoder is used to convert the main image information of the item into an image encoding vector; or a word embedding or transformer model is used to convert the title information of the item into a title text encoding vector.

[0112] Optionally, the memory network includes a dynamic memory network and a static memory network; the feature fusion module 402 is used to:

[0113] In response to the number of samples being zero, feature enhancement processing is performed on the image coding vector based on historical image features in the dynamic memory network; or, in response to the number of samples being less than a preset threshold, feature enhancement processing is performed on the image coding vector based on historical image features in the static memory network.

[0114] Optionally, the feature fusion module 402 is used to:

[0115] Determine the correlation between the image encoding vector after feature enhancement and the title text encoding vector based on the multi-head self-attention mechanism;

[0116] According to the correlation relationship, the feature-enhanced image encoding vector and the title text encoding vector are fused to generate multimodal attribute features.

[0117] Optionally, the training device 400 further includes a memory update module, and the memory update module is used to:

[0118] The memory of the pre-constructed memory network is updated based on the picture coding vector to obtain an updated memory network, so as to optimize the picture coding vector according to the updated memory network.

[0119] Optionally, the feature fusion module 402 is used to:

[0120] Based on the cross-attention mechanism, the image coding vector and the historical image features stored in the memory network are weighted and summed to obtain the weighted image coding vector, so as to perform feature fusion based on the weighted image coding vector and the title text coding vector.

[0121] It should be noted that the specific implementation content of the training device for the item attribute recognition model in the embodiment of the present invention has been described in detail in the above training method for the item attribute recognition model, so the repeated content will not be described again here.

[0122] According to a fourth aspect of an embodiment of the present invention, a device for identifying attributes of an object is provided.

[0123] Figure 5 is a schematic diagram of the main modules of the device for identifying the attributes of an item according to an embodiment of the present invention. Figure 5 As shown, an object attribute recognition device 500 includes:

[0124] Information acquisition module 501, used to acquire main image information and title information of the target item;

[0125] A model recognition module 502, used to input the main image information and the title information into an item attribute recognition model, and the item attribute recognition model outputs an attribute recognition result of the item; wherein the item attribute recognition model is obtained by any method of claims 1 to 6;

[0126] The result display module 503 is used to display the attribute recognition result of the item to the user.

[0127] It should be noted that the specific implementation content of the device for identifying item attributes in the embodiment of the present invention has been described in detail in the above method for identifying item attributes, so the repeated content will not be described again here.

[0128] According to a fifth aspect of an embodiment of the present invention, there is provided an electronic device, including:

[0129] one or more processors;

[0130] a storage device for storing one or more programs,

[0131] When one or more programs are executed by one or more processors, the one or more processors implement the method provided by the first aspect of the embodiment of the present invention and / or the second aspect of the embodiment of the present invention.

[0132] According to a sixth aspect of an embodiment of the present invention, a computer-readable medium is provided, on which a computer program is stored. When the program is executed by a processor, the method provided by the first aspect of the embodiment of the present invention and / or the second aspect of the embodiment of the present invention is implemented.

[0133] According to a seventh aspect of an embodiment of the present invention, a computer program product is provided, including a computer program, which, when executed by a processor, implements the method provided by the first aspect of the embodiment of the present invention and / or the second aspect of the embodiment of the present invention.

[0134] Figure 6 An exemplary system architecture 600 to which the method for training an item attribute recognition model or a device for training an item attribute recognition model according to an embodiment of the present invention can be applied is shown; an exemplary system architecture 600 to which the method for training an item attribute recognition model or a device for training an item attribute recognition model according to an embodiment of the present invention can be applied is shown.

[0135] like Figure 6 As shown, system architecture 600 may include terminal devices 601, 602, 603, network 604 and server 605. Network 604 is used to provide a medium for communication links between terminal devices 601, 602, 603 and server 605. Network 604 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0136] Users can use terminal devices 601, 602, and 603 to interact with server 605 through network 604 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 601, 602, and 603, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only examples).

[0137] The terminal devices 601 , 602 , and 603 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers, etc.

[0138] The server 605 may be a server that provides various services, such as a backend management server (only an example) that provides support for shopping websites browsed by users using the terminal devices 601, 602, and 603. The backend management server may analyze and process the received data such as the model training request, and feed back the processing results (such as the item attribute recognition model - only an example) to the terminal device.

[0139] It should be noted that the training method for the item attribute recognition model provided in the embodiment of the present invention is generally run by the server 605 , and accordingly, the training device for the item attribute recognition model is generally set in the server 605 .

[0140] It should be noted that the item attribute recognition method provided in the embodiment of the present invention is generally executed by the server 605 , and accordingly, the item attribute recognition device is generally disposed in the server 605 .

[0141] It should be understood that Figure 6 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0142] Reference below Figure 7 , which shows a schematic diagram of the structure of a computer system 700 of a terminal device suitable for implementing an embodiment of the present invention. Figure 7 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0143] like Figure 7 As shown, the computer system 700 includes a central processing unit (CPU) 701, which can execute various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage part 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the system 700 are also stored. The CPU 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0144] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, a mouse, etc.; an output section 707 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, a modem, etc. The communication section 709 runs a communication process via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 710 as needed, so that a computer program read therefrom is installed into the storage section 708 as needed.

[0145] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for running the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 709, and / or installed from the removable medium 711. When the computer program is run by the central processing unit (CPU) 701, the above-mentioned functions defined in the system of the present invention are run.

[0146] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0147] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually run basically in parallel, and they can sometimes run in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that runs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0148] The modules involved in the embodiments of the present invention may be implemented by software or hardware. The modules described may also be arranged in a processor, for example, they may be described as: a processor including a feature encoding module, a feature fusion module and a model training module. The names of these modules do not, in some cases, constitute limitations on the modules themselves. For example, the training module may also be described as "a module for training an object attribute recognition model using a weakly supervised learning strategy based on multimodal attribute features". Alternatively, a processor includes an information acquisition module, a model recognition module and a result display module. The names of these modules do not, in some cases, constitute limitations on the modules themselves. For example, the information acquisition module may also be described as "a module for obtaining the main image information and title information of the target object".

[0149] As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently and not be assembled into the device. The above computer-readable medium carries one or more programs. When the above one or more programs are executed by a device, the device includes: performing feature encoding processing on the main image information and title information of the acquired item respectively to obtain a picture encoding vector and a title text encoding vector; performing feature enhancement processing on the picture encoding vector based on a pre-built memory network, and performing feature fusion on the picture encoding vector and the title text encoding vector after feature enhancement processing to generate multimodal attribute features; and training an item attribute recognition model using a weakly supervised learning strategy according to the multimodal attribute features. Alternatively, the device implements the following method: obtaining the main image information and title information of the target item; inputting the main image information and title information into the item attribute recognition model, and the item attribute recognition model outputs the attribute recognition result of the target item; wherein the item attribute recognition model is obtained by any method in the first aspect of the embodiment of the present invention; and displaying the attribute recognition result of the item to the user.

[0150] The computer program product provided by the embodiment of the present invention includes a computer program, which, when executed by a processor, implements the training method of the item attribute recognition model in the embodiment of the first aspect of the present invention and / or the item attribute recognition method in the embodiment of the second aspect of the present invention.

[0151] According to the technical solution of the embodiment of the present invention, the following advantages or beneficial effects are achieved: by performing feature encoding processing on the main image information and title information of the acquired item respectively, a picture encoding vector and a title text encoding vector are obtained; based on the pre-constructed memory network, feature enhancement processing is performed on the picture encoding vector, and feature fusion is performed on the picture encoding vector and the title text encoding vector after feature enhancement processing to generate multimodal attribute features; according to the multimodal attribute features, a weakly supervised learning strategy is used to train an item attribute recognition model; thereby, a small number of samples can be used for weakly supervised training, and feature enhancement is performed on the picture encoding vector based on the memory network, which reduces the cost of sample annotation, improves training efficiency and the accuracy of recognition results. By acquiring the main image information and title information of the target item; inputting the main image information and title information into the item attribute recognition model, the item attribute recognition model outputs the attribute recognition result of the target item; wherein the item attribute recognition model is obtained by any method in the first aspect of the embodiment of the present invention; the item attribute recognition result is displayed to the user; and the accuracy of item attribute recognition and user experience are improved.

[0152] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

[0153] It should be noted that the acquisition, storage and application of user personal information involved in the technical solution of the present disclosure are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

Claims

1. A method for training an object attribute recognition model, characterized in that: include: Perform feature encoding processing on the main image information and title information of the acquired items respectively to obtain the image encoding vector and the title text encoding vector; Performing feature enhancement processing on the image encoding vector based on a pre-built memory network, and performing feature fusion on the image encoding vector after feature enhancement processing and the title text encoding vector to generate multimodal attribute features; An object attribute recognition model is obtained by training using a weakly supervised learning strategy based on the multimodal attribute features.

2. The method according to claim 1, characterized in that The main image information and title information of the acquired items are respectively subjected to feature encoding processing to obtain the image encoding vector and the title text encoding vector, including: An image encoder is used to convert the main image information of the item into an image encoding vector; or a word embedding or transformer model is used to convert the title information of the item into a title text encoding vector.

3. The method according to claim 1, characterized in that The memory network includes a dynamic memory network and a static memory network; performing feature enhancement processing on the image encoding vector based on the pre-built memory network includes: In response to the number of samples being zero, feature enhancement processing is performed on the picture coding vector based on historical picture features in the dynamic memory network; or, in response to the number of samples being less than a preset number threshold, feature enhancement processing is performed on the picture coding vector based on historical picture features in the static memory network.

4. The method according to claim 1, characterized in that: The image encoding vector after feature enhancement processing and the title text encoding vector are subjected to feature fusion to generate multimodal attribute features, including: Determine the correlation between the image encoding vector after the feature enhancement processing and the title text encoding vector based on the multi-head self-attention mechanism; According to the association relationship, the image encoding vector after the feature enhancement processing and the title text encoding vector are feature fused to generate multimodal attribute features.

5. The method according to claim 1, characterized in that Before performing feature enhancement processing on the image encoding vector based on the pre-built memory network, the method further includes: The memory of the pre-constructed memory network is updated based on the picture coding vector to obtain an updated memory network, so as to optimize the picture coding vector according to the updated memory network.

6. The method according to claim 1 or 5, characterized in that: Performing feature enhancement processing on the image encoding vector based on a pre-built memory network, including: Based on the cross attention mechanism, the picture coding vector and the historical picture features stored in the memory network are weightedly summed to obtain a weighted picture coding vector, so as to perform feature fusion according to the weighted picture coding vector and the title text coding vector.

7. A method for identifying object attributes, characterized in that: include: Get the main image information and title information of the target item; Inputting the main image information and title information into an item attribute recognition model, the item attribute recognition model outputting an attribute recognition result of the target item; wherein the item attribute recognition model is obtained by any one of the methods described in claims 1 to 6; The attribute recognition result of the item is displayed to the user.

8. A training device for an object attribute recognition model, characterized in that: include: A feature encoding module is used to perform feature encoding processing on the main image information and title information of the acquired item, respectively, to obtain an image encoding vector and a title text encoding vector; A feature fusion module, used to perform feature enhancement processing on the image coding vector based on a pre-built memory network, and perform feature fusion on the image coding vector after feature enhancement processing and the title text coding vector to generate multimodal attribute features; The model training module is used to train an object attribute recognition model using a weakly supervised learning strategy according to the multimodal attribute features.

9. An object attribute recognition device, characterized in that: include: An information acquisition module is used to obtain the main image information and title information of the target item; A model recognition module, used for inputting the main image information and title information into an item attribute recognition model, wherein the item attribute recognition model outputs an attribute recognition result of the item; wherein the item attribute recognition model is obtained by any one of the methods of claims 1 to 6; The result display module is used to display the attribute recognition result of the item to the user.

10. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

11. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images