Cross-modal commodity classification method and device, equipment, medium and product thereof

Abstract

The application discloses a cross-modal commodity classification method and device, equipment, medium and product. The method comprises the following steps: obtaining a picture-text splicing vector of a commodity object, wherein the picture-text splicing vector is spliced by a picture encoding vector of a commodity picture and a text encoding vector of a commodity title; performing multi-level encoding on the picture-text splicing vector based on a multi-head attention mechanism, realizing first feature interaction between the picture encoding vector and the text encoding vector, and obtaining a first picture-text fusion vector; performing second feature interaction between the first picture-text fusion vector and a label encoding vector representing a commodity label of the commodity object in a preset category tree label structure based on the multi-head attention mechanism, and obtaining a second picture-text fusion vector; and performing classification according to the second picture-text fusion vector, and determining a classification label of the commodity object according to a classification result. The picture-text fusion vector is obtained by deep interaction of features of multiple modes, and classification is performed according to the picture-text fusion vector, so that the classification accuracy can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of e-commerce information, in particular to a cross-modal commodity classification method and a corresponding device, computer equipment, computer readable storage medium, and computer program product. BACKGROUND

[0002] With the rise of cross-border e-commerce mode, more and more overseas merchants join the e-commerce industry. Due to different sources of goods, it poses a considerable challenge on how to better integrate commodity information to provide a better shopping experience for users. One of the basic technologies to improve the user shopping experience lies in the correct classification of commodity objects, and various downstream services are realized on the basis of classification.

[0003] Early commodity classification algorithms are mostly based on the extraction of features from the title of the commodity for classification. However, the features that can be extracted from the title of the commodity are relatively single, and with the increase of commodity categories, the accuracy of classification will decrease. With the development of deep learning, an algorithm for classifying commodities by splicing text and picture features has emerged. However, there is information blocking between different modalities, and simple feature splicing cannot well integrate multi-modal information.

[0004] In addition, with the gradual enrichment of commodity categories, the label system of the commodity is becoming more and more complex. The label system of the commodity can assist the classification of the commodity to some extent. However, the traditional algorithm fails to well combine the information of the three modalities, and therefore it is necessary to explore separately. SUMMARY

[0005] The primary purpose of the present application is to solve at least one of the above problems and provide a cross-modal commodity classification method and a corresponding device, computer equipment, computer readable storage medium, and computer program product.

[0006] To meet the various purposes of the present application, the present application adopts the following technical solutions:

[0007] A cross-modal commodity classification method provided for one of the purposes of the present application includes the following steps:

[0008] Obtaining a picture-text splicing vector of a commodity object, the picture-text splicing vector being spliced from a picture encoding vector of a commodity picture and a text encoding vector of a commodity title;

[0009] Based on a multi-head attention mechanism, the picture-text splicing vector is encoded at multiple levels to realize the first feature interaction between the picture encoding vector and the text encoding vector, and obtain a first picture-text fusion vector;

[0010] The first image-text fusion vector is subjected to a second feature interaction with a label encoding vector representing a product label of the product object in a preset category tree label structure based on a multi-head attention mechanism, to obtain a second image-text fusion vector;

[0011] The product object is classified according to the second image-text fusion vector, and a classification label of the product object is determined according to a classification result.

[0012] In a deepened embodiment, the image-text splicing vector of the product object is obtained, including the following steps:

[0013] An image feature extraction model is used to extract corresponding picture feature information from the product picture of the product object;

[0014] The picture feature information is unfolded into the picture encoding vector represented by a single-row vector;

[0015] The product title is converted into a text encoding vector, which is formed by fusing a text embedding vector, a position embedding vector and a sentence embedding vector of the product title;

[0016] The picture encoding vector and the text encoding vector are spliced into the image-text splicing vector.

[0017] In a specific embodiment, the image feature extraction model is used to extract corresponding picture feature information from the product picture of the product object, including the following steps:

[0018] The product picture is subjected to a format preprocessing;

[0019] An image feature extraction model is used to extract picture feature information from the product picture of the product object;

[0020] The picture feature information is subjected to average pooling;

[0021] The pooled picture feature information is activated and output.

[0022] In a specific embodiment, the product title is converted into a text encoding vector, including the following steps:

[0023] The product title is subjected to word segmentation;

[0024] A text feature extraction model is used to extract the text feature information from the word-segmented product title encoding information;

[0025] The text feature information is activated and output.

[0026] In the deepened embodiment, the image-text splicing vector is encoded based on a multi-head attention mechanism to realize first feature interaction between the picture encoding vector and the text encoding vector, and obtain a first image-text fusion vector. The encoding process of each level includes the following steps:

[0027] An encoder corresponding to the current level is used to construct a query vector, a key vector and a value vector for each row vector in the current input image-text splicing vector;

[0028] The value vectors corresponding to all row vectors are weighted and summed, and the weight of each value vector is the normalized score of the dot product of its corresponding key vector and the query vectors corresponding to other row vectors;

[0029] The weighted sum result is multiplied by a conversion weight matrix to obtain its corresponding intermediate information matrix;

[0030] The intermediate information matrix is subjected to feature extraction by a multi-layer perceptron to obtain an image-text interaction vector, which is used as the image-text splicing vector required by the next encoder, or is directly output as the first image-text fusion vector in the last encoder.

[0031] In the deepened embodiment, the first image-text fusion vector is subjected to second feature interaction with a label encoding vector representing the product label of the product object in the preset category tree label structure based on a multi-head attention mechanism to obtain a second image-text fusion vector, including the following steps:

[0032] The label encoding vector representing the product label of the product object in the preset category tree label structure is obtained;

[0033] Based on the first image-text fusion vector, a query vector, a key vector and a value vector corresponding to each row vector are constructed;

[0034] The dot product of each row vector with the value vector is calculated to update each value vector;

[0035] According to the updated value vector, the updated value vectors corresponding to all row vectors are weighted and summed, and the weight of each updated value vector is the normalized score of the dot product of its corresponding key vector and the query vectors corresponding to other row vectors;

[0036] The weighted sum result is multiplied by a conversion weight matrix to obtain its corresponding second image-text fusion vector.

[0037] In the deepened embodiment, classification is performed according to the second image-text fusion vector, and the classification label of the product object is determined according to the classification result, including the following steps:

[0038] The second image-text fusion vector is fully connected and output to a classification space;

[0039] According to the classification space, a probability corresponding to each classification label is calculated to obtain a classification result;

[0040] According to the classification result, a classification label with the maximum probability is determined;

[0041] The representation text corresponding to the classification label with the maximum probability is extracted and output.

[0042] A cross-modal commodity classification device is provided for one of the purposes of the present application, comprising: a vector acquisition module, a first interaction module, a second interaction module, and a classification processing module, wherein the vector acquisition module is configured to acquire a picture-text splicing vector of a commodity object, the picture-text splicing vector being spliced from a picture encoding vector of a commodity picture and a text encoding vector of a commodity title; the first interaction module is configured to perform multi-level encoding on the picture-text splicing vector based on a multi-head attention mechanism, realize first feature interaction between the picture encoding vector and the text encoding vector, and obtain a first picture-text fusion vector; the second interaction module is configured to perform second feature interaction between the first picture-text fusion vector and a label encoding vector representing a commodity label of the commodity object in a preset category tree label structure based on the multi-head attention mechanism, and obtain a second picture-text fusion vector; and the classification processing module is configured to perform classification according to the second picture-text fusion vector, and determine a classification label of the commodity object according to a classification result.

[0043] In a deepened embodiment, the vector acquisition module comprises: a picture extraction submodule, configured to extract corresponding picture feature information from a commodity picture of the commodity object by using an image feature extraction model; a picture encoding submodule, configured to expand the picture feature information into the picture encoding vector represented by a single-row vector; a text encoding submodule, configured to convert the commodity title into a text encoding vector, the text encoding vector being fused from a text embedding vector, a position embedding vector, and a sentence embedding vector of the commodity title; and a picture-text splicing submodule, configured to splice the picture encoding vector and the text encoding vector into the picture-text splicing vector.

[0044] In a specific embodiment, the picture extraction submodule comprises: a picture processing unit, configured to perform format preprocessing on the commodity picture; a picture feature extraction unit, configured to extract picture feature information from the commodity picture of the commodity object by using an image feature extraction model; a picture pooling processing unit, configured to perform average pooling on the picture feature information; and a picture activation output unit, configured to output the picture feature information after being activated.

[0045] In a specific embodiment, the text encoding submodule includes: a text segmentation unit configured to segment the product title; a text feature extraction unit configured to extract the text feature information from the segmented product title encoding information using a text feature extraction model; and a text activation output unit configured to output the activated text feature information.

[0046] In a deepened embodiment, the first interaction module includes: a first construction submodule configured to construct a query vector, a key vector, and a value vector for each row vector in the current input image-text splicing vector using an encoder corresponding to the current level; a first operation submodule configured to perform weighted summation on the value vectors corresponding to all row vectors, the weight of each value vector being a normalized score of the dot product of its corresponding key vector and the query vectors corresponding to other row vectors; a first conversion submodule configured to multiply the weighted summation result by a conversion weight matrix to obtain a corresponding intermediate information matrix; and a first perception submodule configured to perform feature extraction on the intermediate information matrix through a multi-layer perceptron to obtain an image-text interaction vector, which is used as the image-text splicing vector required by the next encoder, or is directly output as the first image-text fusion vector in the last encoder.

[0047] In a deepened embodiment, the second interaction module includes: a distribution acquisition submodule configured to acquire a label encoding vector representing the product label of the product object in the preset category tree label structure; a second construction submodule configured to construct a query vector, a key vector, and a value vector corresponding to each row vector based on the first image-text fusion vector; an interaction operation submodule configured to update each value vector by performing dot product operation on the value vectors corresponding to each row vector using the label encoding vector; a second operation submodule configured to perform weighted summation on the updated value vectors corresponding to all row vectors, the weight of each updated value vector being a normalized score of the dot product of its corresponding key vector and the query vectors corresponding to other row vectors; and a second conversion submodule configured to multiply the weighted summation result by a conversion weight matrix to obtain a corresponding second image-text fusion vector.

[0048] In a deepened embodiment, the classification processing module includes: a classification mapping submodule configured to fully connect the second image-text fusion vector to a classification space; a probability calculation submodule configured to calculate the probability corresponding to each classification label based on the classification space to obtain a classification result; a probability optimization submodule configured to determine the classification label with the largest probability based on the classification result; and a label output submodule configured to extract and output the representation text corresponding to the classification label with the largest probability.

[0049] A computer device provided to adapt to one of the purposes of the present application, comprising a central processor and a memory, the central processor is used to call a computer program stored in the memory to execute the steps of the cross-modal commodity classification method described in the present application.

[0050] A computer readable storage medium provided to adapt to another purpose of the present application, which stores a computer program implemented according to the cross-modal commodity classification method in the form of computer readable instructions, when the computer program is called and run by a computer, the steps included in the method are executed.

[0051] A computer program product provided to adapt to another purpose of the present application, comprising computer program / instructions, which when executed by a processor, implement the steps of the method described in any one of the embodiments of the present application.

[0052] The advantages of the present application over the prior art are as follows:

[0053] The present application respectively models different modal information of the commodity object such as commodity title, commodity picture, commodity label, etc. to obtain corresponding features, and these features are further interactively fused together based on a multi-head attention mechanism to finally obtain a picture-text fusion vector that comprehensively integrates deep semantic information of various modal information, realize comprehensive representation learning of the commodity object, and accordingly perform commodity classification to obtain a corresponding classification result.

[0054] The modeling of each modal information used in the present application can be decoupled and independent of each other. Compared with the case in the prior art where only text (mainly commodity title and sometimes including label) and picture are divided into two categories for processing, the network structure of different modalities can be flexibly defined according to the characteristics of various modal information, different scale modal feature fusion is supported, and multi-modal information can be processed more flexibly.

[0055] In addition, in the present application, the information of different modalities is fused twice, the first time is to fuse the features of the commodity title and the commodity picture, and the second time is to fuse the features of the first fusion and the commodity label, wherein the multi-head attention mechanism is applied for sufficient feature interaction in both fusions, thereby reducing the information blockage of different modal feature fusion, more rich commodity information can be extracted, more rich feature information can be obtained, and thus the accuracy of commodity classification can be improved. BRIEF DESCRIPTION OF DRAWINGS

[0056] The above and / or additional aspects and advantages of the present application will become apparent and more readily appreciated from the following description, taken in conjunction with the following drawings, in which:

[0057] Figure 1 A flowchart of a typical embodiment of the cross-modal commodity classification method of the present application;

[0058] Figure 2 A schematic diagram of a principle of a model network architecture employed in the present application;

[0059] Figure 3 A schematic diagram of a flow of a process of acquiring a picture-text splicing vector in an embodiment of the present application;

[0060] Figure 4 A schematic diagram of a flow of a process of extracting picture feature information in an embodiment of the present application;

[0061] Figure 5 A schematic diagram of a flow of a process of extracting a text encoding vector in an embodiment of the present application;

[0062] Figure 6 A schematic diagram of a flow of a first feature interaction implementation process in an embodiment of the present application;

[0063] Figure 7 A schematic diagram of a flow of a second feature interaction implementation process in an embodiment of the present application;

[0064] Figure 8 A schematic diagram of a flow of determining a classification label of a commodity object in an embodiment of the present application;

[0065] Figure 9 A principle block diagram of a cross-modal commodity classification apparatus of the present application;

[0066] Figure 10 A structural schematic diagram of a computer device employed in the present application. DETAILED DESCRIPTION

[0067] Embodiments of the present application are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar notations represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by reference to the drawings are exemplary only, and are for the purpose of explanation only, and are not to be construed as limiting the present application.

[0068] It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. As used herein, "connected," "coupled," and / or "coupling," can include both direct connections and / or indirect connections (i.e., via one or more other elements). As used herein, "connection" or "coupling" can include a wireless connection or a wireless coupling. As used herein, the term "and / or" comprises all of the associated listed items, one or more of the associated listed items, and all combinations of the associated listed items.

[0069] It will be further understood that, as used herein, the terms "comprises" and / or "comprising," while they can be used in the context of compositions of matter, preferably, are not used in a restrictive sense, for example, to denote the presence of stated features, integers, steps or components thereof, but to denote the presence at least the stated features, integers, steps or components thereof. As used herein, unless otherwise defined, all terms, including technical and scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the specification and relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

[0070] Those skilled in the art will understand that, as used herein, the terms "client," "terminal," and "terminal device" include both devices that are solely wireless signal receivers and devices that have both receiving and transmitting hardware that can communicate bi-directionally over a bi-directional communication link. Such devices can include cellular or other communication devices with single-line or multiple-line displays, or no display, Personal Communications Service (PCS) devices that can combine a voice and / or data processor, a PDA that can include a radio frequency receiver and a pager, Internet and / or Intranet access, a Web browser, a calendar, and / or a GPS receiver, a conventional laptop and / or palmtop computer and / or other devices that have a radio frequency receiver. As used herein, the terms "client," "terminal," and "terminal device" can be portable, transportable, mounted in a vehicle (aeronautical, maritime, and / or land), or adapted and / or configured for local and / or distributed operation on Earth and / or any other location in space. As used herein, the terms "client," "terminal," and "terminal device" can also be a communication terminal, an Internet terminal, a music / video playing terminal, such as a PDA, a Mobile Internet Device (MID), and / or a mobile phone with music / video playing function, a smart television, a set-top box, and / or the like.

[0071] As used herein, the terms "server," "client," "service node," and the like refer to hardware that has the equivalent capability of a personal computer, i.e., an electronic device having a central processing unit (including an arithmetic unit and a controller), a memory, an input device, and an output device, and the like necessary components disclosed by the Von Neumann principle. A computer program is stored in the memory, the central processing unit loads the program stored in the external memory into the memory and runs it, executes the instructions in the program, and interacts with the input and output devices, thereby completing a specific function.

[0072] It should be noted that the concept of "server" in the present application can also be extended to the case of a server cluster. According to the principle of network deployment understood by those skilled in the art, the servers should be logically divided, and in physical space, these servers can be independent of each other but can be called through an interface, or can be integrated into a physical computer or a computer cluster. Those skilled in the art should understand this variation and should not be restricted by the implementation of the network deployment of the present application.

[0073] One or more technical features of the present application, unless explicitly specified, can be deployed on a server for implementation and accessed by a client remotely calling an online service interface provided by the server, or can be directly deployed and run on a client for implementation and access.

[0074] The neural network model referred to or possibly referred to in the present application, unless explicitly specified, can be deployed on a remote server and remotely called by a client, or can be deployed on a client with sufficient device capability for direct calling. In some embodiments, when it is run on a client, its corresponding intelligence can be obtained through transfer learning, so as to reduce the requirement for client hardware running resources and avoid excessive occupation of client hardware running resources.

[0075] The various data involved in the present application, unless explicitly specified, can be remotely stored on a server or stored on a local terminal device, as long as it is suitable for being called by the technical solutions of the present application.

[0076] Those skilled in the art should know that the various methods of the present application, although based on the same concept and described to present commonality among them, are independently executable unless otherwise specified. Similarly, for each embodiment disclosed in the present application, it is based on the same inventive concept, so the same concept is understood to be equivalent, and although the concept is expressed differently, it is only for the convenience of appropriate transformation of the concept.

[0077] Unless it is explicitly stated that the embodiments disclosed in the present application are mutually exclusive, the technical features involved in each embodiment can be combined flexibly to construct new embodiments, as long as such combination does not deviate from the spirit of the present application and can meet the needs of the prior art or solve some aspects of the deficiencies in the prior art. For this variation, those skilled in the art should know.

[0078] The cross-modal commodity classification method of the present application can be programmed as a computer program product, deployed in a client or a server for running and implementation, for example, in the e-commerce platform application scenario of the present application, it is generally deployed in the server for implementation, thereby the interface opened after the running of the computer program product can be accessed, the process of the computer program product is interacted with the human-computer interaction through the graphical user interface to execute the method.

[0079] An exemplary application scenario of the present application is the application in the e-commerce platform based on independent stations, each independent station is a merchant instance of the e-commerce platform, has an independent access domain name, and is responsible for publishing and updating the goods by the actual owner.

[0080] The merchant instance of the independent station publishes each commodity online, and the e-commerce platform obtains the commodity information related to the commodity, and then constructs the corresponding commodity object for data storage. The commodity information mainly includes the commodity title, the commodity picture, the commodity label marked according to the hierarchical structure of the category tree of the e-commerce platform, etc.

[0081] In order to adapt to different tasks to classify the commodity object, in order to serve the needs of different downstream tasks such as search, sorting and promotion of the commodity object, when implementing the technical solution of the present application, the commodity title, the commodity picture and the commodity label of the commodity object can be collected; the commodity picture can be the main picture containing the product appearance of the commodity object.

[0082] Please refer to Figure 1 the flowchart, and combine the network architecture diagram shown in Figure 2 , the cross-modal commodity classification method of the present application in its typical embodiment includes the following steps:

[0083] Step S1100, obtaining the graphic-text splicing vector of the commodity object, the graphic-text splicing vector is spliced by the picture encoding vector of the commodity picture and the text encoding vector of the commodity title:

[0084] The commodity information of the commodity object is stored in the commodity database corresponding to each merchant instance, or can be actively submitted by the user, the commodity information mainly includes the corresponding commodity title, the commodity picture and the commodity label of the commodity object, and the commodity label is generated based on the label structure of the category tree of the e-commerce platform to which the merchant instance belongs, and is composed of the hierarchical node labels of each level.

[0085] In this step, the commodity picture and the commodity title of the commodity object are obtained first, and the encoding vectors of the two, i.e. the picture encoding vector and the text encoding vector, are obtained, and the two encoding vectors are simply spliced into the graphic-text splicing vector.

[0086] The picture encoding vector is feature information extracted from the product picture through deep semantic learning. A variety of pre-trained image feature extraction models can be used to extract deep semantic features of the product picture to obtain the corresponding picture encoding vector, thereby realizing the representation of the deep semantic information of the product picture. The image feature extraction model generally includes a neural network model based on CNN suitable for deep semantic feature extraction of pictures, such as Resnet, EfficientNet, etc., which can be flexibly modeled and selected by those skilled in the art.

[0087] The text encoding vector is feature information extracted from the product title through deep semantic learning. A variety of pre-trained text feature extraction models can be used to extract deep semantic features of the product title to obtain the corresponding text encoding vector, thereby realizing the representation of the deep semantic information of the product title. The text feature extraction model generally includes a model based on statistics or based on RNN suitable for deep semantic feature extraction of text, such as the Bert model, which is currently the best neural network model capable of processing text timing information and can be used for text extraction in this application. Similarly, the Electra model can obtain the same or similar effect as the Bert model with lower parameter amount, so it is also recommended to use.

[0088] As for the label encoding vector corresponding to the product label, it will also be used as a feature information corresponding to an independent modality in this application, and a label feature extraction model based on a Bi-LSTM bidirectional RNN network architecture is recommended to be used for processing. Since such a bidirectional short-long memory model can examine context information from two directions, it can better represent the features of the product label. Specifically, a pre-trained Bi-TreeLSTM model can be used to extract the label encoding vector corresponding to the product label of the product object in this application. Similarly, any pre-trained basic neural network model suitable for representing learning of product objects can be theoretically applied to the model for extracting the label encoding vector of the product label in this application, such as Text-RCNN, Bert, etc., which can be flexibly modeled by those skilled in the art.

[0089] It can be seen that the information corresponding to each modality required for the implementation of the commodity classification can be encoded by different models respectively to obtain the corresponding encoding vectors. These corresponding models can use various mature models known at present, especially pre-trained models, or can be trained to convergence by the person skilled in the art. Regardless of the model used, as long as it can be used in the technical solution of the present application to realize representation learning of information corresponding to each modality, such as commodity title, commodity picture, commodity label, etc., and obtain the corresponding encoding vector.

[0090] In this step, for a commodity object, the picture encoding vector corresponding to the commodity picture thereof and the text encoding vector corresponding to the commodity title thereof are simply spliced on a unified scale to obtain the picture-text splicing vector. It should be noted that the picture-text splicing vector is only a simple connection of the two encoding vectors, and the interaction of the two encoding vectors has not been realized. Therefore, the picture-text splicing vector has not realized the deep information fusion between the commodity picture and the commodity title.

[0091] The specific encoding process of the encoding vector corresponding to each modality will be further disclosed in subsequent embodiments, which will not be described here.

[0092] Step S1200, multi-level encoding of the picture-text splicing vector based on a multi-head attention mechanism, realizing the first feature interaction of the picture encoding vector and the text encoding vector, and obtaining a first picture-text fusion vector:

[0093] The picture-text splicing vector is input into a neural network model based on a multi-head attention mechanism for encoding to obtain a corresponding first picture-text fusion vector.

[0094] The neural network model based on the multi-head attention mechanism used here is preferably a neural network model based on the encoding path in the Transformer, for example, the BERT model can be preferred, which internally uses the encoding path in the Transformer; for example, it also includes the encoding path in the Vision Transformer (ViT) which is more suitable for processing picture-text information. In addition, other variant models that can achieve the same function of the present application can also be replaced.

[0095] According to the principle of the Transformer, the encoding path of the Transformer includes a plurality of encoders with the same structure and principle. Each encoder includes a self-attention layer, which performs feature interaction based on the picture-text splicing vector input therein, and transmits the result after the interaction to a multi-layer perceptron of the encoder for high-level semantic extraction and output.

[0096] The neural network model based on the multi-head attention mechanism used in the present application is pre-trained to be enabled after convergence. Since the training process of models such as Transformer, Bert, and ViT is known to those skilled in the art, the training process will not be described in detail. The neural network model is suitable for performing feature-level encoding operations on the image-text splicing vector, so that the picture feature vector and the text feature vector contained in the image-text splicing vector realize deep interaction at the feature level, thereby realizing deep fusion of the product picture and its product title at the deep semantic level. The principle will be further disclosed in subsequent embodiments, which will not be described here.

[0097] During the step-by-step encoding process of the image-text splicing vector by the neural network model, the multi-head attention mechanism always plays a role, so that the picture encoding vector and the text encoding vector in the image-text splicing vector are deeply interacted by fully referring to the relevant context information during the encoding process, and finally the first image-text fusion vector fully considering the context information is obtained.

[0098] The first image-text fusion vector is the product of the picture encoding vector and the text encoding vector after sufficient feature interaction operation at the semantic level, therefore, from the perspective of representation learning, the first image-text fusion vector successfully and uniformly represents the associated information between the product picture and the product title of the product object.

[0099] Step S1300, based on the multi-head attention mechanism, the first image-text fusion vector and the label encoding vector representing the product label of the product object in the preset category tree label structure are subjected to a second feature interaction to obtain a second image-text fusion vector:

[0100] Similarly, the same principle as the self-attention layer in the previous step encoder can be used to further realize the feature interaction between the first image-text fusion vector and the label encoding vector representing the product label of the product object in the preset category tree label structure by means of a self-attention layer based on the multi-head attention mechanism.

[0101] As mentioned earlier, the label encoding vector is a representation of the product label to which the product object belongs in the preset category tree label structure, therefore, after the label encoding vector and the first image-text fusion vector are sent into the self-attention layer, feature interaction can be performed.

[0102] The self-attention layer added in this step is not different from the self-attention layer in the encoder of the previous step in structure and principle. In fact, the self-attention layer can be obtained by removing the feedforward neural network from the last level of the encoder of the neural network model. As can be seen, in a typical case, only the Bert model needs to be simply modified to realize the neural network model required by the present application, and the training principle and process thereof are basically the same as the Bert prototype. Those skilled in the art can construct and train the model according to the disclosure herein.

[0103] In the process of feature interaction between the first image-text fusion vector and the label encoding vector by the self-attention layer provided in this step, the multi-head attention mechanism still plays a role, and in the feature interaction process, sufficient reference is made to the relevant context information, realizing the deep interaction between the first image-text fusion vector and the label encoding vector in the semantic aspect, and finally obtaining the second image-text fusion vector which fully considers the context information.

[0104] Similarly, the second image-text fusion vector is the product of the sufficient feature interaction operation between the picture encoding vector, the text encoding vector, and the label encoding vector in the semantic level. Therefore, from the perspective of representation learning, the second image-text fusion vector successfully and uniformly represents the association information between the product picture, the product title, and the product label of the product object.

[0105] Step S1400, classifying according to the second image-text fusion vector, and determining the classification label of the product object according to the classification result:

[0106] The second image-text fusion vector will first pass through the fully connected layer and be mapped to the classification space, and then the multi-classifier constructed by Softmax is used to perform probability calculation on the classification space to obtain the probability of the second image-text fusion vector mapping to each classification label, i.e., the classification result. According to the classification probability of each classification label in this classification result, the classification label corresponding to the product object can be determined. Usually, the classification label with the largest classification probability is taken as the classification label corresponding to the product object. At this point, the representation text corresponding to the classification label can be output.

[0107] As for the classification system to which the classification label belongs, it should not be confused with the concept of product label in the hierarchical structure of the category tree in the e-commerce platform, but mainly depends on the downstream task served by the technical solution of the present application. For example, in the exemplary application scenario of the present application for classifying a product object into a certain type of product, there are multiple classification labels in the classification system, respectively indicating that the product object is {normal product; dangerous product; suspicious product; non-sellable product}. Based on such a classification system, if a certain product object has the maximum classification probability of the classification label of "normal product", it can be determined that the classification label corresponding to "normal product" is the classification label corresponding to the final classification result.

[0108] The neural network model involved in the present application needs to be pre-trained. Those skilled in the art can understand that the neural network model can be gradually trained to converge by feeding corresponding training samples into the neural network model used in the present application, and the neural network model can be put into use after it reaches convergence. Since the neural network model used in the present application is implemented based on the basic architecture of the Transformer encoding path, and the training process of an architecture such as the BERT model is understood by those skilled in the art, the present application focuses on the encoding process before training, and does not need to emphasize the specific training process of the model. Those skilled in the art can train the neural network model required by the present application on the basis of the technical solution disclosed in the present application.

[0109] In one embodiment, in order to guide the neural network model used in the present application to acquire classification ability, a classification task identifier can be added to the text encoding vector or the picture encoding vector. When the BERT model is used in the present application to extract text feature information, according to the principle of the BERT model, a [CLS] identifier can be added to the text encoding vector to achieve this. Similarly, the identifier can also be added to the picture encoding vector. Correspondingly, in the prediction stage, the classification task identifier can also be added when encoding the picture or the text.

[0110] As can be known from the disclosure of the typical embodiment, the present application at least embodies the following advantages:

[0111] The present application models different modal information such as product title, product picture and product label of a product object to obtain corresponding features. These features are then interactively fused together based on the multi-head attention mechanism for multiple times to finally obtain a picture-text fusion vector that comprehensively integrates deep semantic information of various modal information, realize comprehensive representation learning of the product object, and accordingly classify the product to obtain a corresponding classification result.

[0112] The modeling of each modality information utilized in the present application can be decoupled and independent from each other. Compared with the case of only processing text (mainly the title of the commodity and sometimes including the label) and pictures in the traditional technology, the present application can adapt to the characteristics of various modal information and flexibly customize the network structure of different modalities, support the fusion of modal features of different scales, and more flexibly process multi-modal information.

[0113] In addition, in the present application, the information of different modalities is fused twice. The first time is to fuse the features of the title and the picture of the commodity, and the second time is to fuse the features of the first fusion and the features of the label. The multi-head attention mechanism is applied for sufficient feature interaction in both fusions, thereby reducing the information blockage of different modal feature fusion, extracting more abundant commodity information, obtaining more abundant feature information, and thus helping to improve the accuracy of commodity classification.

[0114] Please refer to Figure 3 In a deepened embodiment, step S1100 of obtaining the picture-text splicing vector of the commodity object includes the following steps:

[0115] Step S1110, the image feature extraction model is used to extract the corresponding picture feature information from the commodity picture of the commodity object:

[0116] In the present embodiment, after the commodity picture is preprocessed regularly, the image feature extraction model based on the Resnet network architecture is used to extract the features of the commodity picture of the commodity object. Resnet50, Resnet101, IResNet, ResNeXt101, and other network architectures based on residual convolutional neural network, and newer EfficienNet, etc. can be used. In practice, the application of these architectures can achieve good results. Of course, the image feature extraction model realized by other network architectures with excellent effects can also be used in the present embodiment. The image feature extraction model can extract the corresponding picture feature information from the commodity picture.

[0117] Step S1120, the picture feature information is unfolded into the picture encoding vector represented by a single row vector:

[0118] The image feature information can be compressed into a two-dimensional matrix. In order to adapt to the needs of the neural network model based on the multi-head attention mechanism of the present application, the two-dimensional matrix can be reduced in dimension. Specifically, the two-dimensional matrix is unfolded in a one-dimensional vector, that is, the row vectors in the two-dimensional matrix are sequentially spliced to obtain a single row vector, that is, a picture encoding vector. It is not difficult to understand that, assuming that the two-dimensional matrix is of size N*D, the picture encoding vector obtained after unfolding will be of size 1*(N*D). The row vector dimension (N*D) corresponding to this size is uniformly specified in advance with the text feature vector in the present application, so that the two can be directly spliced subsequently.

[0119] Step S1130, converting the product title into a text encoding vector fused by a text embedding vector, a position embedding vector and a segment embedding vector of the product title:

[0120] In the present embodiment, the Electra model can be considered as a text feature extraction model for feature extraction of the product title of the product object. The Electra model is trained based on a generative adversarial model. Like Bert, it is trained using MLM (Masked Language Model), so it can be replaced with Bert in the present application. To meet the needs of the text feature extraction model, three vectors are encoded from the product title, namely a text embedding vector (Token Embedding) for representing each word, a position embedding vector (Position Embedding) for representing the position information of each word, and a segment embedding vector (Segment Embedding) for representing the information for distinguishing between sentences. Then, the text feature extraction model extracts text features according to these embedding vectors, and finally obtains the corresponding text encoding vector. Those skilled in the art can implement it flexibly.

[0121] Step S1140, splicing the picture encoding vector and the text encoding vector into the graphic-text splicing vector:

[0122] On the basis of having obtained the picture encoding vector and the text encoding vector of the product object, only a simple feature splicing of the two is needed to obtain the graphic-text splicing vector. Since the size of the picture encoding vector and the size of the text encoding vector have been unified, the finally obtained graphic-text splicing vector is a 2*(N*D) size vector.

[0123] The embodiment discloses the principle of obtaining the product picture, the encoding vector of the product title through two different paths respectively, and finally splicing the two into the same picture-text splicing vector, realizing the preprocessing before the first feature interaction. According to the embodiment, it can be seen that a plurality of pre-training models can be adopted to realize the preparation of the picture-text splicing vector, so that a large amount of training cost does not need to be additionally invested, and preparation for subsequent feature interaction can be made.

[0124] Please refer to Figure 4 In a specific embodiment, step S1110 adopts an image feature extraction model to extract corresponding picture feature information from the product picture of the product object, including the following steps:

[0125] Step S1111 performs format preprocessing on the product picture:

[0126] The preprocessing of the product picture is mainly to adapt to the needs of the input of the picture feature extraction model, and to process the product picture to help the model to process. For example, the product picture is subjected to size scaling, object recognition, image standardization processing, etc. For this, those skilled in the art can flexibly implement as needed.

[0127] Step S1112 adopts an image feature extraction model to extract picture feature information from the product picture of the product object:

[0128] Referring to the foregoing embodiments, the preselected image feature extraction model, such as the aforementioned EfficientNet model, can extract corresponding picture feature information from the preprocessed product picture.

[0129] Step S1113 performs average pooling on the picture feature information:

[0130] Since the picture feature information extracted by the image feature extraction model is usually multi-channel data, a pooling layer can be used to perform average pooling on the multi-channel picture feature information, and pool it into single-channel picture feature information.

[0131] Step S1114 outputs the pooled picture feature information after activation:

[0132] Finally, the single-channel picture feature information is activated by applying an activation function, the semantic information extraction is completed, and the final picture feature information is obtained.

[0133] The embodiment gives the extraction process of the picture feature information of the commodity picture. The process realizes the conversion from the commodity picture to the picture feature information, and lays a foundation for subsequent preparation of the picture coding vector. As can be seen, the process is flexible and can be customized to adapt to the modal information of the picture information. Therefore, those skilled in the art can flexibly design the corresponding implementation process according to the principle disclosed in the embodiment, adapt to the specific selection of the picture feature extraction model, and as long as the picture feature information represented by a two-dimensional matrix can be obtained for dimension reduction processing to prepare for constructing the picture coding vector.

[0134] In summary of the foregoing embodiments, the following variations can be made to obtain corresponding variant embodiments: first, the commodity picture is divided into a plurality of equal-sized graph elements, and the picture feature information is extracted for each graph element by the image feature extraction model. Similarly, after average pooling and activation of the picture feature information, the picture feature information corresponding to each graph element is converted into a single-row vector, and then the single-row vectors are sequentially connected to construct the picture coding vector. And so on, those skilled in the art can flexibly modify.

[0135] Please refer to Figure 5 In a specific embodiment, step S1130 of converting the commodity title into a text coding vector includes the following steps:

[0136] Step S1131, tokenizing the commodity title:

[0137] In this step, the commodity title of the commodity is first processed. Because the commodity title of the commodity is generally short, and some special symbols play a key role in semantic expression, no preprocessing such as stop word removal is performed on the title, and any synonym model such as N-gram, WordPrice, and Word2Vec is directly used to tokenize the title.

[0138] Step S1132, extracting the text feature information from the tokenized commodity title coding information using a text feature extraction model:

[0139] In the previous step, the commodity title has been tokenized. According to a pre-set dictionary, each token is converted into a corresponding text embedding vector. Similarly, according to the position coding of the token, a position embedding vector is obtained, and a sentence embedding vector that distinguishes different sentences is coded. Then, these vectors are added and fused, and then transmitted to the text feature extraction model for deep semantic information extraction, and the corresponding text feature vector is obtained.

[0140] As described previously, the text feature extraction model used for feature extraction of the abstract text can be implemented using a pre-training model including Bert, Electra, etc. Those skilled in the art can flexibly implement it.

[0141] Taking the Bert model as an example, the three embedding vectors, i.e., the text embedding vector, the position embedding vector, and the sentence embedding vector, will be fused, and on the basis of the fusion, the Bert model will perform corresponding encoding, and the same is also based on the multi-head attention mechanism, and finally the text feature information corresponding to the product title is obtained.

[0142] Step S1133, output the text feature information after activation:

[0143] Further, the text feature information is applied to an activation function for feature activation to obtain the final text feature information.

[0144] In this embodiment, the encoding process of the product title of the product object is further disclosed, and the Electra model or the Bert model is recommended to realize text encoding, which fully utilizes the excellent achievements in the prior art, so that the encoding efficiency is higher, and the obtained text encoding vector is more accurate for understanding the deep semantics of the abstract text.

[0145] Please refer to Figure 6 In a deepened embodiment, step S1200, based on the multi-head attention mechanism, the image-text splicing vector is encoded in multiple levels, realizing the first feature interaction between the picture encoding vector and the text encoding vector, and obtaining a first image-text fusion vector, and the encoding process of each level includes the following steps:

[0146] Step S1210, using the encoder corresponding to the current level to construct a query vector, a key vector and a value vector for each row vector in the current input image-text splicing vector:

[0147] As described above, in this application, a neural network model based on a multi-head attention mechanism is recommended to be used, therefore, in this embodiment, a Bert model or a model composed of a multi-layer Transformer encoding path is used to construct the neural network model required by this application, therefore, the neural network model used in this application actually inherits the inherent encoding architecture of the traditional Transformer model, therefore, the encoder used in this embodiment is the encoder in the inherent architecture of the Transformer.

[0148] In the inherent encoding architecture of the Transformer, a plurality of encoders are included, each of which adopts the same structure, including a self-attention layer and a multi-layer perceptron (MLP), and the self-attention layer realizes the multi-head attention mechanism.

[0149] According to this, the self-attention layer of each encoder performs deep feature interaction on the image-text splicing vector received by it based on the algorithm fixed by the multi-head attention mechanism, and then transmits the result obtained after the feature interaction to its multi-layer perceptron for feature extraction, and the multi-layer perceptron provides the feature extraction result to the next encoder, and so on. The self-attention layer of the subsequent encoder takes the output of the multi-layer perceptron of the previous encoder as input and processes according to the same logic. The output of the last encoder can be output as the first image-text fusion vector.

[0150] According to this, in this step, the self-attention layer in each encoder can construct a corresponding query vector (Q), key vector (K), and value vector (V) for each row vector according to the inherent architecture principle after receiving the input of a vector. The construction of Q, K, and V vectors is the basic principle of the multi-head attention mechanism and is the basic means to realize deep interaction at the feature level.

[0151] Step S1220, weight and sum all value vectors corresponding to the row vectors, and the weight of each value vector is the normalized score of the dot product of its corresponding key vector and the query vector corresponding to other row vectors:

[0152] According to the inherent algorithm of the self-attention mechanism, for each row vector, the self-attention layer first calculates the dot product of its key vector and the query vector corresponding to other row vectors, and then normalizes these dot products using the Softmax function, so that the sum of the multiple dot products is 1. The normalized values of these dot products are used as the weights corresponding to each row vector. On this basis, multiply and add all value vectors corresponding to the row vectors and their corresponding weights to realize weighted summation and obtain a summation matrix after weighted summation.

[0153] Step S1230, multiply the weighted summation result by the conversion weight matrix to obtain the corresponding intermediate information matrix:

[0154] In order to standardize the output scale, multiply the weighted summation result, i.e., the summation matrix, by a conversion weight matrix to obtain an intermediate information matrix with the same scale as the original image-text splicing vector. It can be understood that after the operation between Q, K, and V, deep semantic interaction between each row vector has been realized in the intermediate information matrix.

[0155] Step S1240, feature extraction is performed on the intermediate information matrix through a multilayer perceptron to obtain a graph-text interaction vector as a graph-text splicing vector required by a next encoder or directly output as a first graph-text fusion vector in a last encoder:

[0156] The intermediate information matrix is output to a multilayer perceptron of a feedforward neural network for high-level semantic feature extraction to obtain a graph-text interaction vector, which is then output to a next encoder. For the next encoder, the graph-text interaction vector is also regarded as a graph-text splicing vector of a first encoder to perform the same operation. If the encoder outputting the graph-text interaction information is a last encoder, the graph-text interaction information is directly output as the first graph-text fusion vector.

[0157] In this embodiment, it is disclosed in detail how to combine the multi-head attention mechanism to realize the deep interaction between the product picture and the product title of the product object, thereby realizing deep semantic fusion. It can be seen that the present application effectively utilizes the existing technical architecture and obtains the expected result, which plays a key role in the multi-modal feature interaction fusion of the product object in the e-commerce platform and lays a solid technical foundation for realizing classification according to the graph and text information of the product object.

[0158] Please refer to Figure 7 In a deepened embodiment, according to the previous embodiment, a self-attention layer is used to further perform feature interaction between the first graph-text fusion vector and the label encoding vector. Therefore, step S1300, based on the multi-head attention mechanism, the first graph-text fusion vector and the label encoding vector representing the product label of the product object in the preset category tree label structure are subjected to a second feature interaction to obtain a second graph-text fusion vector, including the following steps:

[0159] Step S1310, obtaining the label encoding vector representing the product label of the product object in the preset category tree label structure:

[0160] As mentioned earlier, the pre-trained Bi-TreeLSTM can be used to realize the representation learning of the product label of the product object to obtain the corresponding label encoding vector, which represents the probability distribution of the mapping of the product label of the product object to the labels at each level of the category tree of the e-commerce platform, thereby realizing the semantic representation of the product label of the product object.

[0161] Step S1320, based on the first graph-text fusion vector, constructing a query vector, a key vector and a value vector corresponding to each row vector thereof:

[0162] Similarly, the self-attention layer used in this step can construct a corresponding query vector (Q), key vector (K) and value vector (V) for each row vector according to the inherent architecture principle after receiving the input of the first image-text fusion vector.

[0163] In step S1330, the label encoding vector is respectively multiplied with the value vector corresponding to each row quantity to update each value vector.

[0164] In order to realize the feature interaction between the label encoding vector and the first image-text fusion vector, the dot product of the label encoding vector and each value vector corresponding to the first image-text fusion vector is calculated to update the value vector. The label encoding vector represents the probability distribution of the product label, so through the dot product, the semantic information hidden in the product label is deeply embedded in the value vector, realizing the deep semantic interaction between the label encoding vector and the first image-text fusion vector.

[0165] In step S1340, the updated value vectors corresponding to all row vectors are weighted and summed, and the weight of each updated value vector is the normalized score of the dot product of its corresponding key vector and the query vector corresponding to other row vectors.

[0166] On the basis of updating the value vector, the inherent algorithm of the self-attention mechanism can be used. For each row vector, the dot product of its key vector and the query vector corresponding to other row vectors is calculated, and then the Softmax function is used to normalize these dot products, so that the sum of the dot products is 1. The normalized values of these dot products are used as the weights of each row vector. On this basis, the value vectors corresponding to all row vectors are multiplied by their corresponding weights and then added to realize weighted summation and obtain a weighted summation matrix.

[0167] In step S1350, the weighted summation result is multiplied by a conversion weight matrix to obtain a corresponding second image-text fusion vector.

[0168] In order to standardize the output scale, the weighted summation result, i.e. the summation matrix, is multiplied by a conversion weight matrix to obtain a matrix with the same scale as the original first image-text fusion vector. The matrix is the second image-text fusion vector.

[0169] It can be understood that by using the self-attention layer to realize the deep semantic interaction between the first image-text fusion vector and the label encoding vector, the second image-text fusion vector obtained has realized the deep semantic interaction between the multiple modal information, i.e. the product picture, product title and product label, which can be used for classification required by downstream tasks.

[0170] Please refer to Figure 8In a deepened embodiment, step S1400, classifying according to the second image-text fusion vector, determining the classification label of the commodity object according to the classification result, comprises the following steps:

[0171] Step S1410, fully connecting the second image-text fusion vector and outputting to a classification space:

[0172] The second image-text fusion vector is finally linearly converted by a fully connected layer and mapped to a classification space, so as to subsequently calculate the classification probability of each classification label in the classification space by a classifier.

[0173] Step S1420, calculating the probability of each classification label according to the classification space to obtain a classification result:

[0174] Here, a simple classifier constructed by a Softmax is used to normalize and calculate the probability of each classification label in the classification space, thereby obtaining the probability of each classification label. The probabilities of all classification labels add up to 1.

[0175] Step S1430, determining the classification label with the largest probability according to the classification result:

[0176] As can be easily understood, the classification label with the largest probability in the classification result obtained by probability calculation is the classification label corresponding to the classification to which the commodity object belongs.

[0177] Step S1440, extracting the representation text corresponding to the classification label with the largest probability and outputting:

[0178] According to the classification label, the corresponding mapping dictionary can be found to determine the representation text corresponding to the classification label.

[0179] According to the different downstream tasks, the representation text can be adaptively output. For example, for a classification task of identifying non-sellable products, when the commodity object is identified as a non-sellable product, a notification message that the commodity object belongs to the representation text is directly output, or only this representation text is added as the attribute data of the corresponding commodity object for subsequent processing. And the like, those skilled in the art can flexibly implement.

[0180] It can be understood in combination with the present embodiment that the present application can obtain more deep semantic information related to the commodity object by classifying on the basis of the second image-text fusion vector obtained by deep interaction of the commodity object in the feature level of the commodity picture, the commodity title and the commodity label, and classifying the commodity object according to the semantic information, thereby obtaining a more accurate classification result, and more suitable for serving other downstream tasks in the e-commerce platform such as commodity recommendation, commodity search, commodity safety attribute identification and the like related to the classification of the commodity object.

[0181] According to the principle of the present application, the neural network model called by the present application is more easily trained to converge because the representation learning effect is better based on the deep interaction of the image-text fusion vector. In practice, the training speed is significantly better than the case where the image encoding vector and the text encoding vector are not deep-interactive. Because the two kinds of information are interwoven at the semantic level, the loss of individual fine-grained information in one kind of information will not affect the recognition effect during classification.

[0182] Please refer to Figure 9 , a cross-modal commodity classification device provided for one of the purposes of the present application, which is a functional embodiment of the cross-modal commodity classification method of the present application. The device includes a vector acquisition module 1100, a first interaction module 1200, a second interaction module 1300, and a classification processing module 1400. The vector acquisition module 1100 is used to acquire an image-text splicing vector of a commodity object, which is spliced from an image encoding vector of a commodity image and a text encoding vector of a commodity title. The first interaction module 1200 is used to perform multi-level encoding on the image-text splicing vector based on a multi-head attention mechanism, to realize the first feature interaction between the image encoding vector and the text encoding vector, and to obtain a first image-text fusion vector. The second interaction module 1300 is used to perform a second feature interaction between the first image-text fusion vector and a label encoding vector representing a commodity label of the commodity object in a preset category tree label structure based on a multi-head attention mechanism, to obtain a second image-text fusion vector. The classification processing module 1400 is used to classify according to the second image-text fusion vector, and to determine the classification label of the commodity object according to the classification result.

[0183] In a deepened embodiment, the vector acquisition module 1100 includes a picture extraction submodule for extracting corresponding picture feature information from the commodity image of the commodity object using an image feature extraction model; a picture encoding submodule for expanding the picture feature information into the image encoding vector represented by a single-row vector; a text encoding submodule for converting the commodity title into a text encoding vector, which is fused from a text embedding vector, a position embedding vector, and a sentence embedding vector of the commodity title; and an image-text splicing submodule for splicing the image encoding vector and the text encoding vector into the image-text splicing vector.

[0184] In the embodied embodiment, the picture extraction submodule comprises: a picture processing unit configured to perform format preprocessing on the product picture; a picture feature extraction unit configured to extract picture feature information from the product picture of the product object by using an image feature extraction model; a picture pooling processing unit configured to perform average pooling on the picture feature information; and a picture activation output unit configured to output the pooled picture feature information after activation.

[0185] In the embodied embodiment, the text encoding submodule comprises: a text segmentation unit configured to perform text segmentation on the product title; a text feature extraction unit configured to extract the text feature information from the segmented product title encoding information by using a text feature extraction model; and a text activation output unit configured to output the text feature information after activation.

[0186] In the deepened embodiment, the first interaction module 1200 comprises: a first construction submodule configured to construct a query vector, a key vector and a value vector for each row vector in the current input image-text splicing vector by using an encoder corresponding to a current level; a first operation submodule configured to perform weighted summation on the value vectors corresponding to all row vectors, and the weight of each value vector is a normalized score of the dot product of the key vector corresponding to the value vector and the query vectors corresponding to other row vectors; a first conversion submodule configured to obtain an intermediate information matrix corresponding to the weighted summation result by multiplying the weighted summation result by a conversion weight matrix; and a first perception submodule configured to perform feature extraction on the intermediate information matrix by using a multilayer perceptron to obtain an image-text interaction vector, which is used as an image-text splicing vector required by a next encoder, or is directly output as a first image-text fusion vector in the last encoder.

[0187] In the deepened embodiment, the second interaction module 1300 comprises: a distribution acquisition submodule configured to acquire a label encoding vector representing the product label of the product object in a preset category tree label structure; a second construction submodule configured to construct a query vector, a key vector and a value vector corresponding to each row vector based on the first image-text fusion vector; an interaction operation submodule configured to update each value vector by performing dot product operation on the value vectors corresponding to each row vector with the label encoding vector; a second operation submodule configured to perform weighted summation on the updated value vectors corresponding to all row vectors, and the weight of each updated value vector is a normalized score of the dot product of the key vector corresponding to the updated value vector and the query vectors corresponding to other row vectors; and a second conversion submodule configured to obtain a second image-text fusion vector corresponding to the weighted summation result by multiplying the weighted summation result by a conversion weight matrix.

[0188] In a further embodiment, the classification processing module 1400 includes: a classification mapping submodule, used to output the second image-text fusion vector to the classification space via a fully connected method; a probability calculation submodule, used to calculate the probability corresponding to each classification label according to the classification space to obtain the classification result; a probability optimization submodule, used to determine the classification label with the highest probability according to the classification result; and a label output submodule, used to extract and output the representation text corresponding to the classification label with the highest probability.

[0189] To address the aforementioned technical problems, embodiments of this application also provide computer equipment. For example... Figure 10 The diagram shows the internal structure of a computer device. The computer device includes a processor, a computer-readable storage medium, a memory, and a network interface connected via a system bus. The computer-readable storage medium stores an operating system, a database, and computer-readable instructions. The database may store a sequence of control information. When executed by the processor, the computer-readable instructions enable the processor to implement a cross-modal product classification method. The processor provides computing and control capabilities to support the operation of the entire computer device. The memory stores computer-readable instructions, which, when executed by the processor, enable the processor to execute the cross-modal product classification method of this application. The network interface of the computer device is used for communication with a terminal. Those skilled in the art will understand that… Figure 10 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0190] In this embodiment, the processor is used to execute... Figure 9 The system includes the specific functions of each module and its sub-modules, and the memory stores the program code and various data required to execute these modules or sub-modules. A network interface is used for data transmission between the user terminal and the server. In this embodiment, the memory stores the program code and data required to execute all modules / sub-modules in the cross-modal commodity sorting device of this application, and the server can call the server's program code and data to execute the functions of all sub-modules.

[0191] This application also provides a storage medium storing computer-readable instructions, which, when executed by one or more processors, cause the one or more processors to perform the steps of the cross-modal product classification method of any embodiment of this application.

[0192] The application also provides a computer program product comprising computer programs / instructions which, when executed by one or more processors, implement the steps of the method according to any of the embodiments of the application.

[0193] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments of the application can be completed by a computer program instructing relevant hardware, and the computer program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the above-mentioned embodiments of the method. The storage medium can be a computer-readable storage medium such as a magnetic disc, an optical disc, a read-only memory (ROM), or a random access memory (RAM).

[0194] In summary, the application can perform deep semantic interaction at the feature level according to the product picture, product title, and product label of the product object, deepen the fusion of multiple modal information, and classify accordingly. The classification result of the product object can be obtained with higher accuracy, and the application is suitable for classifying product objects in an e-commerce platform and has a broad application prospect.

[0195] Those skilled in the art can understand that the steps, measures, and schemes in the various operations, methods, and processes discussed in the application can be alternated, changed, combined, or deleted. Further, other steps, measures, and schemes in the various operations, methods, and processes discussed in the application can also be alternated, changed, rearranged, decomposed, combined, or deleted. Further, the steps, measures, and schemes in the prior art with the various operations, methods, and processes disclosed in the application can also be alternated, changed, rearranged, decomposed, combined, or deleted.

[0196] The above only describes some embodiments of the application. It should be noted that those skilled in the art can make several improvements and refinements without departing from the principles of the application, and these improvements and refinements should also be considered as falling within the scope of protection of the application.

Claims

1. A cross-modal product classification method, characterized in that, Includes the following steps: Obtaining the image-text spliced ​​vector of a product object includes: extracting corresponding image feature information from the product image of the product object using an image feature extraction model; expanding the image feature information into an image encoding vector represented by a single-row vector; converting the product title into a text encoding vector, which is formed by fusing the text embedding vector, position embedding vector, and sentence embedding vector of the product title; and splicing the image encoding vector and the text encoding vector together to form the image-text spliced ​​vector; the image-text spliced ​​vector is formed by splicing the image encoding vector of the product image and the text encoding vector of the product title. The image-text splicing vector is encoded in multiple levels based on a multi-head attention mechanism to achieve the first feature interaction between the image encoding vector and the text encoding vector, thereby obtaining the first image-text fusion vector. Based on the multi-head attention mechanism, the first image-text fusion vector is subjected to a second feature interaction with the tag encoding vector representing the product tag of the product object in the preset category tree tag structure to obtain the second image-text fusion vector; The product object is classified according to the second image-text fusion vector, and the classification label is determined based on the classification result.

2. The cross-modal product classification method according to claim 1, characterized in that, The image feature extraction model is used to extract relevant image feature information from the product image of the product object, including the following steps: The product images are pre-processed and formatted. An image feature extraction model is used to extract image feature information from the product image of the product object; The image feature information is subjected to average pooling; The pooled image feature information is activated and then output.

3. The cross-modal product classification method according to claim 1, characterized in that, Converting the product title into a text-encoded vector includes the following steps: The product title is segmented into words; The text feature information is extracted from the word-segmented product title encoding information using a text feature extraction model; The text feature information is activated and then output.

4. The cross-modal product classification method according to claim 1, characterized in that, The image-text fusion vector is encoded at multiple levels based on a multi-head attention mechanism, achieving the first feature interaction between the image encoding vector and the text encoding vector to obtain the first image-text fusion vector. The encoding process at each level includes the following steps: The query vector, key vector, and value vector are constructed using each row vector in the currently input image and text splicing vector corresponding to the encoder at the current level. The value vectors corresponding to all row vectors are weighted and summed. The weight of each value vector is the normalized score of the dot product of its corresponding key vector and the query vectors corresponding to other row vectors. Multiply the weighted sum by the transformation weight matrix to obtain the corresponding intermediate information matrix; The intermediate information matrix is ​​processed by a multilayer perceptron to extract features and obtain a text-image interaction vector. This vector serves as the text-image stitching vector required by the next encoder, or it can be directly output as the first text-image fusion vector in the last encoder.

5. The cross-modal product classification method according to claim 1, characterized in that, Based on a multi-head attention mechanism, the first image-text fusion vector is subjected to a second feature interaction with the tag encoding vector representing the product tag of the product object in the preset category tree tag structure to obtain the second image-text fusion vector, including the following steps: Obtain the tag encoding vector representing the product tag of the product object in the preset category tree tag structure; Based on the first image-text fusion vector, construct the query vector, key vector, and value vector corresponding to each row vector; The value vectors are updated by taking the dot product of the label encoding vector with the value vectors corresponding to each row. Based on the updated value vector, the updated value vectors corresponding to all row vectors are weighted and summed. The weight of each updated value vector is the normalized score of the dot product of its corresponding key vector and the query vector corresponding to other row vectors. Multiply the weighted sum by the transformation weight matrix to obtain the corresponding second image-text fusion vector.

6. The cross-modal product classification method according to claim 1, characterized in that, Classification is performed based on the second image-text fusion vector, and the classification label of the product object is determined based on the classification result, including the following steps: The second image-text fusion vector is output to the classification space via a fully connected layer. The probability corresponding to each category label is calculated based on the category space to obtain the classification result; Determine the category label with the highest probability based on the classification results; Extract and output the representation text corresponding to the category label with the highest probability.

7. A cross-modal commodity sorting device, characterized in that, include: The vector acquisition module is used to acquire the image and text splicing vector of the product object. The image and text splicing vector is formed by splicing the image encoding vector of the product image and the text encoding vector of the product title. The initial interaction module is used to perform multi-level encoding on the image-text splicing vector based on a multi-head attention mechanism, so as to realize the first feature interaction between the image encoding vector and the text encoding vector and obtain the first image-text fusion vector. The secondary interaction module, based on the multi-head attention mechanism, performs a second feature interaction between the first image-text fusion vector and the tag encoding vector representing the product tag in the preset category tree tag structure to obtain the second image-text fusion vector. The classification processing module is used to classify the product based on the second image-text fusion vector and determine the classification label of the product object based on the classification result. The vector acquisition module further includes: an image extraction submodule, used to extract corresponding image feature information from the product image of the product object using an image feature extraction model; an image encoding submodule, used to expand the image feature information into an image encoding vector represented by a single-row vector; a text encoding submodule, used to convert the product title into a text encoding vector, which is formed by fusing the text embedding vector, position embedding vector, and sentence embedding vector of the product title; and an image-text splicing submodule, used to splice the image encoding vector and the text encoding vector into the image-text splicing vector.

8. A computer device comprising a central processing unit and a memory, characterized in that, The central processing unit is used to invoke and run a computer program stored in the memory to perform the steps of the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, It stores, in the form of computer-readable instructions, a computer program implemented according to any one of claims 1 to 6, which, when invoked by a computer, executes the steps included in the corresponding method.

10. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method described in any one of claims 1 to 6.

Images