Image-text fusion feature information generation method and device, equipment, medium and product

Abstract

The embodiment of the invention discloses an image-text fusion feature information generation method and device, equipment, a medium and a product. A specific embodiment of the method comprises the steps of performing feature extraction processing on an article image, article title information and article attribute setting information of a target article to obtain image feature information, title feature information and first attribute feature information; the image feature information and the title feature information are input into a first image-text feature fusion model, first image-text fusion feature information is obtained, and the first image-text feature fusion model is an image-text feature fusion model based on a first attention mechanism; and inputting the first image-text fusion feature information and the first attribute feature information into a second image-text feature fusion model to obtain second image-text fusion feature information, the second image-text feature fusion model being an image-text feature fusion model based on a second attention mechanism. The implementation mode is related to artificial intelligence, and the accuracy of representing key information by the image-text fusion feature vector can be improved.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of artificial intelligence technology, and specifically to methods, devices, equipment, media and products for generating image-text fusion feature information. Background Art

[0002] Multimodal feature fusion is to extract features of two or more modal information and fuse them to achieve information supplementation and capture of key information. At present, when multimodal feature fusion is performed on the object image and text information representing the attributes of the same object, the method usually adopted is: feature fusion based on feature vector splicing.

[0003] However, the inventors have discovered that when the above-mentioned method is used to perform multimodal feature fusion on the object image and text information representing the object attributes of the same object, the following technical problems often occur: different texts in the text information have different degrees of association with the object image, and the feature vector of the image and the feature vector of the text are fused by directly splicing, resulting in low accuracy of the fused feature vector in representing key information in the text information.

[0004] The above information disclosed in this Background section is only for enhancement of understanding of the background of the inventive concept and therefore it may contain information that does not form the prior art that is already known in this country to a person of ordinary skill in the art. Summary of the invention

[0005] The content of this disclosure is used to introduce concepts in a brief form, which will be described in detail in the detailed implementation section below. The content of this disclosure is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to limit the scope of the technical solution claimed for protection.

[0006] Some embodiments of the present disclosure propose methods, devices, electronic devices, computer-readable media and program products for generating image-text fusion feature information to solve the technical problems mentioned in the above background technology section.

[0007] In a first aspect, some embodiments of the present disclosure provide a method for generating image-text fusion feature information, the method comprising: performing feature extraction processing on an item image, item title information and item attribute setting information of a target item to obtain image feature information, title feature information and first attribute feature information; inputting the image feature information and the title feature information into a first image-text feature fusion model to obtain first image-text fusion feature information, wherein the first image-text feature fusion model is included in a pre-trained multimodal feature fusion model, and the first image-text feature fusion model is an image-text feature fusion model based on a first attention mechanism; inputting the first image-text fusion feature information and the first attribute feature information into a second image-text feature fusion model to obtain second image-text fusion feature information, wherein the second image-text feature fusion model is included in the multimodal feature fusion model, and the second image-text feature fusion model is an image-text feature fusion model based on a second attention mechanism.

[0008] Optionally, the method further includes: performing feature extraction processing on the item attribute filled-in information to obtain second attribute feature information; and performing proofreading processing on the second image-text fusion feature information and the second attribute feature information to obtain item proofreading result information.

[0009] Optionally, the above-mentioned feature extraction processing is performed on the item image, item title information and item attribute setting information of the target item to obtain image feature information, title feature information and first attribute feature information, including: determining the preset category information and item attribute setting information corresponding to the above-mentioned target item as the target item attribute information; inputting the above-mentioned target item attribute information into a pre-trained attribute feature extraction model to obtain the first attribute feature information.

[0010] Optionally, the above-mentioned inputting the above-mentioned first image-text fusion feature information and the above-mentioned first attribute feature information into the second image-text feature fusion model to obtain the second image-text fusion feature information includes: using the feature weight generation sub-model included in the above-mentioned second image-text feature fusion model to generate a feature weight information set corresponding to the above-mentioned first image-text fusion feature information according to the above-mentioned first image-text fusion feature information and the above-mentioned first attribute feature information; using the weighted summation sub-model included in the above-mentioned second image-text feature fusion model to perform weighted summation processing on the above-mentioned first image-text fusion feature information according to the above-mentioned feature weight information set to obtain the second image-text fusion feature information.

[0011] Optionally, the multimodal feature fusion model is pre-trained through the following steps: obtaining a sample, wherein the sample includes a sample item image, sample item title information, and sample item attribute filling information; performing the following training steps on the sample: performing feature extraction processing on the sample item image, sample item title information, and item attribute setting information included in the sample to obtain sample image feature information, sample title feature information, and first attribute feature information; inputting the sample image feature information and the sample title feature information into an initial first image-text feature fusion model to obtain sample first image-text fusion feature information, wherein the initial first image-text feature fusion model is included in the initial multimodal feature fusion model; inputting the sample first image-text fusion feature information and the first attribute feature information into the initial A second image-text feature fusion model is used to obtain the sample's second image-text fusion feature information, wherein the above-mentioned initial second image-text feature fusion model is included in the above-mentioned initial multimodal feature fusion model; feature extraction processing is performed on the sample item attribute filling information to obtain the sample's second attribute feature information; the sample's second image-text fusion feature information is compared with the sample's second attribute feature information; according to the comparison result, it is determined whether the initial multimodal feature fusion model has achieved a preset optimization goal; in response to determining that the initial multimodal feature fusion model has not achieved the above-mentioned optimization goal, the network parameters of the initial multimodal feature fusion model are adjusted, and the adjusted initial multimodal feature fusion model is used as the initial multimodal feature fusion model, and the item sample is reacquired as a sample, and the above-mentioned training steps are performed again.

[0012] Optionally, the training step further includes: in response to determining that the initial multimodal feature fusion model achieves the optimization target, determining the initial multimodal feature fusion model as a trained multimodal feature fusion model.

[0013] Optionally, after performing feature extraction processing on the sample item image, sample item title information and item attribute setting information included in the sample to obtain sample image feature information, sample title feature information and first attribute feature information, the above-mentioned training step also includes: performing regularization processing on the sample title feature information to obtain target sample title feature information as the sample title feature information.

[0014] In a second aspect, some embodiments of the present disclosure provide a device for generating image-text fusion feature information, the device comprising: a feature extraction unit, configured to perform feature extraction processing on an object image, object title information and object attribute setting information of a target object, to obtain image feature information, title feature information and first attribute feature information; a first input unit, configured to input the image feature information and the title feature information into a first image-text feature fusion model, to obtain first image-text fusion feature information, wherein the first image-text feature fusion model is included in a pre-trained multimodal feature fusion model, and the first image-text feature fusion model is an image-text feature fusion model based on a first attention mechanism; a second input unit, configured to input the first image-text fusion feature information and the first attribute feature information into a second image-text feature fusion model, to obtain second image-text fusion feature information, wherein the second image-text feature fusion model is included in the multimodal feature fusion model, and the second image-text feature fusion model is an image-text feature fusion model based on a second attention mechanism.

[0015] Optionally, the image-text fusion feature information generating device may further include a second feature extraction unit and a proofreading unit. The second feature extraction unit is configured to perform feature extraction processing on the item attribute filled-in information to obtain second attribute feature information. The proofreading unit is configured to perform proofreading processing on the second image-text fusion feature information and the second attribute feature information to obtain item proofreading result information.

[0016] Optionally, the feature extraction unit can be further configured to: determine the preset category information and item attribute setting information corresponding to the above-mentioned target item as the target item attribute information; input the above-mentioned target item attribute information into a pre-trained attribute feature extraction model to obtain first attribute feature information.

[0017] Optionally, the second input unit can be further configured to: utilize the feature weight generation sub-model included in the above-mentioned second image-text feature fusion model to generate a feature weight information set corresponding to the above-mentioned first image-text fusion feature information according to the above-mentioned first image-text fusion feature information and the above-mentioned first attribute feature information; utilize the weighted summation sub-model included in the above-mentioned second image-text feature fusion model to perform weighted summation processing on the above-mentioned first image-text fusion feature information according to the above-mentioned feature weight information set to obtain the second image-text fusion feature information.

[0018] Optionally, the multimodal feature fusion model is pre-trained through the following steps: obtaining a sample, wherein the sample includes a sample item image, sample item title information, and sample item attribute filling information; performing the following training steps on the sample: performing feature extraction processing on the sample item image, sample item title information, and item attribute setting information included in the sample to obtain sample image feature information, sample title feature information, and first attribute feature information; inputting the sample image feature information and the sample title feature information into an initial first image-text feature fusion model to obtain sample first image-text fusion feature information, wherein the initial first image-text feature fusion model is included in the initial multimodal feature fusion model; inputting the sample first image-text fusion feature information and the first attribute feature information into the initial A second image-text feature fusion model is used to obtain the sample's second image-text fusion feature information, wherein the above-mentioned initial second image-text feature fusion model is included in the above-mentioned initial multimodal feature fusion model; feature extraction processing is performed on the sample item attribute filling information to obtain the sample's second attribute feature information; the sample's second image-text fusion feature information is compared with the sample's second attribute feature information; according to the comparison result, it is determined whether the initial multimodal feature fusion model has achieved a preset optimization goal; in response to determining that the initial multimodal feature fusion model has not achieved the above-mentioned optimization goal, the network parameters of the initial multimodal feature fusion model are adjusted, and the adjusted initial multimodal feature fusion model is used as the initial multimodal feature fusion model, and the item sample is reacquired as a sample, and the above-mentioned training steps are performed again.

[0019] Optionally, the training step further includes: in response to determining that the initial multimodal feature fusion model achieves the optimization target, determining the initial multimodal feature fusion model as a trained multimodal feature fusion model.

[0020] Optionally, after performing feature extraction processing on the sample item image, sample item title information and item attribute setting information included in the sample to obtain sample image feature information, sample title feature information and first attribute feature information, the above-mentioned training step also includes: performing regularization processing on the sample title feature information to obtain target sample title feature information as the sample title feature information.

[0021] In a third aspect, some embodiments of the present disclosure provide an electronic device comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner of the above-mentioned first aspect.

[0022] In a fourth aspect, some embodiments of the present disclosure provide a computer-readable medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method described in any implementation manner of the first aspect.

[0023] In a fifth aspect, some embodiments of the present disclosure provide a computer program product, including a computer program, which implements the method described in any implementation manner of the above-mentioned first aspect when executed by a processor.

[0024] The above-mentioned various embodiments of the present disclosure have the following beneficial effects: the accuracy of the key information represented by the image-text fusion feature vector can be improved through the image-text fusion feature information generation method of some embodiments of the present disclosure. Specifically, the reason for the low accuracy of the key information represented by the fused feature vector in the text information is that the different texts in the text information have different degrees of association with the object image, and the feature vector of the image is directly fused with the feature vector of the text, resulting in the low accuracy of the key information represented by the fused feature vector in the text information. Based on this, the image-text fusion feature information generation method of some embodiments of the present disclosure firstly performs feature extraction processing on the object image, the object title information and the object attribute setting information of the target object to obtain image feature information, title feature information and first attribute feature information. Thus, the image feature vector, the title feature vector and the feature vector of the set object attribute of the target object can be obtained, so as to generate the image-text fusion feature vector. Secondly, the above-mentioned image feature information and the above-mentioned title feature information are input into the first image-text feature fusion model to obtain the first image-text fusion feature information. Wherein, the above-mentioned first image-text feature fusion model is included in the pre-trained multimodal feature fusion model. The first image-text feature fusion model is an image-text feature fusion model based on the first attention mechanism. Thus, the features of the item image and the features of the item title information can be fused based on the attention mechanism, so as to obtain the key feature information of the fused image and the title. Finally, the first image-text fusion feature information and the first attribute feature information are input into the second image-text feature fusion model to obtain the second image-text fusion feature information. Among them, the second image-text feature fusion model is included in the multimodal feature fusion model. The second image-text feature fusion model is an image-text feature fusion model based on the second attention mechanism. Thus, the features of the set item attributes can be fused with the first image-text fusion feature information, so as to obtain feature information focusing on more attributes. Also, when the item image and the text information representing the item attributes of the same item are fused with multimodal features, the features of the item image, the item title information and the item attribute setting information are fused based on the attention mechanism, so as to obtain feature information focusing on more attributes, so as to improve the accuracy of the key information represented by the image-text fusion feature vector. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0026] Figure 1 is a schematic diagram of an application scenario of the method for generating feature information of image-text fusion in some embodiments of the present disclosure;

[0027] Figure 2 is a flow chart of some embodiments of the method for generating feature information of image-text fusion according to the present disclosure;

[0028] Figure 3 is a schematic diagram of a network structure of a first image-text feature fusion model according to some embodiments of the present disclosure;

[0029] Figure 4 is a flow chart of other embodiments of the method for generating feature information of image-text fusion according to the present disclosure;

[0030] Figure 5 It is a structural schematic diagram of some embodiments of the device for generating feature information of image-text fusion according to the present disclosure;

[0031] Figure 6 It is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0032] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0033] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure can be combined with each other.

[0034] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0035] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0036] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0037] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0038] Figure 1 It is a schematic diagram of an application scenario of the method for generating feature information of image-text fusion in some embodiments of the present disclosure.

[0039] exist Figure 1 In the application scenario of , first, the computing device 101 can perform feature extraction processing on the object image 102, the object title information 103 and the object attribute information 104 of the target object to obtain image feature information 105, title feature information 106 and first attribute feature information 107. Then, the computing device 101 can input the above image feature information 105 and the above title feature information 106 into the first image-text feature fusion model 1081 to obtain the first image-text fusion feature information 109. Among them, the above first image-text feature fusion model 1081 is included in the pre-trained multimodal feature fusion model 108. The above first image-text feature fusion model 1081 is an image-text feature fusion model based on the first attention mechanism. Finally, the computing device 101 can input the above first image-text fusion feature information 109 and the above first attribute feature information 107 into the second image-text feature fusion model 1082 to obtain the second image-text fusion feature information 110. Among them, the above second image-text feature fusion model 1082 is included in the above multimodal feature fusion model 108. The above-mentioned second image-text feature fusion model 1082 is an image-text feature fusion model based on the second attention mechanism.

[0040] It should be noted that the computing device 101 can be hardware or software. When the computing device is hardware, it can be implemented as a distributed cluster consisting of multiple servers or terminal devices, or as a single server or a single terminal device. When the computing device is embodied as software, it can be installed in the hardware devices listed above. It can be implemented as multiple software or software modules for providing distributed services, or as a single software or software module. No specific limitation is made here.

[0041] It should be understood that Figure 1 The number of computing devices in the embodiment is only illustrative. Any number of computing devices may be provided according to implementation requirements.

[0042] Continue to refer Figure 2 , shows a process 200 of some embodiments of the method for generating feature information of image-text fusion according to the present disclosure. The method for generating feature information of image-text fusion includes the following steps:

[0043] Step 201, performing feature extraction processing on the object image, object title information and object attribute setting information of the target object to obtain image feature information, title feature information and first attribute feature information.

[0044] In some embodiments, the execution subject (eg Figure 1The computing device 101 shown in the figure can perform feature extraction processing on the object image, the object title information and the object attribute setting information of the target object to obtain image feature information, title feature information and first attribute feature information. Among them, the above-mentioned target object can be an object that needs to be circulated on the network platform. The above-mentioned object image can be an image representing the object that needs to be published on the network platform. The above-mentioned object title information can be a title representing the object that needs to be published on the network platform. The above-mentioned object attribute setting information can be information on different attributes of the same object. For example, when the above-mentioned target object is a refrigerator, the above-mentioned object title information can be "double-door two-door household small-sized refrigerator air-cooled and frost-free;", and the above-mentioned object attribute setting information can be "double door; three doors; double doors;". In practice, the above-mentioned execution subject can input the object image, the object title information and the object attribute setting information into a pre-trained feature extraction model to obtain image feature information, title feature information and first attribute feature information. Among them, the above-mentioned feature extraction model is a neural network with the object image, the object title information and the object attribute setting information as input and the image feature information, the title feature information and the first attribute feature information as output. The feature extraction model includes an image feature extraction submodel, a title feature extraction submodel and an attribute feature extraction submodel. The image feature extraction submodel may be a convolutional neural network that takes an item image as input and outputs image feature information. The image feature extraction submodel may also be a Transformer encoding model that takes an item image as input and outputs image feature information. The image feature information may include but is not limited to an image feature vector. The image feature vector may be a feature vector of an item image. The title feature extraction submodel may be a convolutional neural network that takes item title information as input and outputs title feature information. The title feature information may include but is not limited to a title feature vector. The title feature vector sequence may be a feature vector of item title information. The attribute feature extraction submodel may be a TF-IDF (Term Frequency–Inverse Document Frequency) algorithm. The first attribute feature information may include but is not limited to a first attribute feature vector. The first attribute feature vector may be a feature vector of item attribute setting information.

[0045] In some optional implementations of some embodiments, the execution subject may perform feature extraction processing on the object image, the object title information, and the object attribute setting information of the target object through the following steps to obtain the image feature information, the title feature information, and the first attribute feature information:

[0046] In the first step, the preset category information and item attribute setting information corresponding to the target item are determined as the target item attribute information. The preset category information may be information of the category to which the target item belongs. The preset category information may include but is not limited to an item category identifier. The item category identifier may be a unique identifier of the item category. The item category may be but is not limited to one of the following: home appliances, digital products.

[0047] In the second step, the target item attribute information is input into a pre-trained attribute feature extraction model to obtain first attribute feature information. The attribute feature extraction model may be a neural network that takes the target item attribute information as input and the first attribute feature information as output. For example, the attribute feature extraction model may be a Transformer encoding model trained based on hundreds of millions of target item attribute information.

[0048] Therefore, considering that the same attribute has different meanings in different categories, adding preset category information when extracting features from item attribute setting information can improve the speed and accuracy of feature extraction.

[0049] Step 202: input the image feature information and the title feature information into a first image-text feature fusion model to obtain first image-text fusion feature information.

[0050] In some embodiments, the execution entity may input the image feature information and the title feature information into a first image-text feature fusion model to obtain first image-text fusion feature information. The first image-text feature fusion model may be included in a pre-trained multimodal feature fusion model. The first image-text feature fusion model may be an image-text feature fusion model based on a first attention mechanism. The first attention mechanism may be a multi-head attention mechanism or a self-attention mechanism. The image-text feature fusion model may be a neural network for fusing image feature vectors and text feature vectors. For example, the image-text feature fusion model may be a Transformer encoding model.

[0051] As an example, when the first attention mechanism is a multi-head attention mechanism and the image-text feature fusion model is a Transformer encoding model, the network structure of the first image-text feature fusion model can be as follows: Figure 3As shown. In practice, first, the above-mentioned execution entity can determine the image feature vector included in the above-mentioned image feature information as the Key vector and the Value vector in the multi-head attention mechanism. Then, the title feature vector included in the above-mentioned title feature information is determined as the Query vector in the multi-head attention mechanism. After that, the Query vector, the Key vector and the Value vector are input into the first image-text feature fusion model to obtain the first image-text fusion feature vector as the first image-text fusion feature information. Among them, the vector of each dimension in the first image-text fusion feature vector corresponds to the text in the item title information.

[0052] Step 203: input the first image-text fusion feature information and the first attribute feature information into a second image-text feature fusion model to obtain second image-text fusion feature information.

[0053] In some embodiments, the execution entity may input the first image-text fusion feature information and the first attribute feature information into a second image-text feature fusion model to obtain second image-text fusion feature information. The second image-text feature fusion model may be included in the multimodal feature fusion model. The second image-text feature fusion model may be used to integrate item attribute setting information into the first image-text fusion feature information. The second image-text feature fusion model may be an image-text feature fusion model based on a second attention mechanism. The second attention mechanism may be a multi-head attention mechanism or a self-attention mechanism.

[0054] As an example, when the second attention mechanism is a multi-head attention mechanism and the image-text feature fusion model can be a Transformer encoding model, the execution subject can determine the first attribute feature vector included in the first attribute feature information as the Key vector and Value vector in the multi-head attention mechanism. Then, the first image-text fusion feature information is determined as the Query vector in the multi-head attention mechanism. After that, the Query vector, the Key vector and the Value vector are input into the second image-text feature fusion model to obtain the second image-text fusion feature vector as the second image-text fusion feature information.

[0055] Optionally, the above execution entity may further perform the following steps:

[0056] The first step is to perform feature extraction processing on the item attribute filling information to obtain second attribute feature information. The item attribute filling information may be pre-filled information characterizing the attributes of the item. For example, the item attribute filling information may be information characterizing the attributes of the item pre-filled by the merchant. The second attribute feature information may characterize the features of the item attribute filling information. When the item attribute filling information is the same as the item attribute setting information, the first attribute feature information may be the same as the second attribute feature information. In practice, the execution entity may perform feature extraction processing on the item attribute filling information through a preset text feature extraction algorithm to obtain a second attribute feature vector as the second attribute feature information. The preset text feature extraction algorithm may be, but is not limited to, one of the following: a bag of words model, a text feature extraction model based on a convolutional neural network.

[0057] In the second step, the second image-text fusion feature information and the second attribute feature information are proofread to obtain the item proofreading result information. In practice, first, the execution subject may determine the cosine similarity between the second image-text fusion feature information and the second attribute feature information as the matching similarity. Secondly, in response to determining that the matching similarity satisfies the preset similarity condition, the preset proofreading pass information is determined as the item proofreading result information. Among them, the preset similarity condition may be that the matching similarity is greater than or equal to the preset similarity. The preset similarity may be a preset similarity. The preset proofreading pass information may be preset information representing that the item image and item title information have been proofread. For example, the preset proofreading pass information may be "1". Then, in response to determining that the matching similarity does not meet the preset similarity condition, the preset proofreading failure information is determined as the item proofreading result information. Among them, the preset proofreading failure information may be preset information representing that the item image and item title information have not been proofread. For example, the preset proofreading failure information may be "0".

[0058] Optionally, the multimodal feature fusion model is pre-trained through the following steps:

[0059] The first step is to obtain samples, wherein the samples include sample item images, sample item title information, and sample item attribute information.

[0060] In the second step, the following training steps are performed on the sample:

[0061] In the first training step, the sample item image, the sample item title information and the item attribute setting information included in the sample are subjected to feature extraction processing to obtain sample image feature information, sample title feature information and first attribute feature information. In practice, the execution subject may input the sample item image, the sample item title information and the item attribute setting information into the feature extraction model to obtain sample image feature information, sample title feature information and first attribute feature information.

[0062] In the second training step, the sample image feature information and the sample title feature information are input into the initial first image-text feature fusion model to obtain the sample first image-text fusion feature information. The initial first image-text feature fusion model may be included in the initial multimodal feature fusion model.

[0063] In the third training step, the sample first image-text fusion feature information and the first attribute feature information are input into the second image-text feature fusion model included in the initial multimodal feature fusion model to obtain the sample second image-text fusion feature information. The initial second image-text feature fusion model can be included in the initial multimodal feature fusion model.

[0064] The fourth training step is to perform feature extraction processing on the filled-in information of the sample item attributes to obtain the sample attribute feature information. In practice, the above-mentioned execution subject can perform feature extraction processing on the filled-in information of the sample item attributes through the above-mentioned preset text feature extraction algorithm to obtain the sample attribute feature information.

[0065] The fifth training step is to compare the sample second image-text fusion feature information with the sample second attribute feature information. Here, the comparison method can be a method of determining the similarity between the sample second image-text fusion feature information and the sample second attribute feature information.

[0066] In the sixth training step, it is determined whether the initial multimodal feature fusion model reaches a preset optimization target based on the comparison result. The comparison result may include similarity. The above optimization target may be that the determined similarity is greater than or equal to a preset similarity. There is no limitation on the setting of the preset similarity.

[0067] The seventh training step, in response to determining that the initial multimodal feature fusion model does not achieve the above optimization goal, adjust the network parameters of the initial multimodal feature fusion model, and use the adjusted initial multimodal feature fusion model as the initial multimodal feature fusion model, reacquire the item sample as the sample, and perform the above training step again. The above item sample can be a sample of the corresponding item. As an example, the network parameters of the initial multimodal feature fusion model can be adjusted by using a back propagation algorithm (BP algorithm) and a gradient descent method (such as a random mini-batch gradient descent algorithm).

[0068] Optionally, the above training step further includes:

[0069] In an eighth training step, in response to determining that the initial multimodal feature fusion model achieves the above-mentioned optimization goal, the initial multimodal feature fusion model is determined as the trained multimodal feature fusion model.

[0070] Optionally, after the first training step, the training step further includes:

[0071] The sample title feature information is regularized to obtain the target sample title feature information as the sample title feature information. In practice, the sample title feature vector included in the sample title feature information can be input into a random deactivation function to obtain the target sample title feature information as the sample title feature information. Thus, for attribute information that is less correlated with the item image, the multimodal feature fusion model can pay more attention to the item title information, thereby improving the robustness of the multimodal feature fusion model.

[0072] The above-mentioned various embodiments of the present disclosure have the following beneficial effects: the accuracy of the key information represented by the image-text fusion feature vector can be improved through the image-text fusion feature information generation method of some embodiments of the present disclosure. Specifically, the reason for the low accuracy of the key information represented by the fused feature vector in the text information is that the different texts in the text information have different degrees of association with the object image, and the feature vector of the image is directly fused with the feature vector of the text, resulting in the low accuracy of the key information represented by the fused feature vector in the text information. Based on this, the image-text fusion feature information generation method of some embodiments of the present disclosure firstly performs feature extraction processing on the object image, the object title information and the object attribute setting information of the target object to obtain image feature information, title feature information and first attribute feature information. Thus, the image feature vector, the title feature vector and the feature vector of the set object attribute of the target object can be obtained, so as to generate the image-text fusion feature vector. Secondly, the above-mentioned image feature information and the above-mentioned title feature information are input into the first image-text feature fusion model to obtain the first image-text fusion feature information. Wherein, the above-mentioned first image-text feature fusion model is included in the pre-trained multimodal feature fusion model. The first image-text feature fusion model is an image-text feature fusion model based on the first attention mechanism. Thus, the features of the item image and the features of the item title information can be fused based on the attention mechanism, so as to obtain the key feature information of the fused image and the title. Finally, the first image-text fusion feature information and the first attribute feature information are input into the second image-text feature fusion model to obtain the second image-text fusion feature information. Among them, the second image-text feature fusion model is included in the multimodal feature fusion model. The second image-text feature fusion model is an image-text feature fusion model based on the second attention mechanism. Thus, the features of the set item attributes can be fused with the first image-text fusion feature information, so as to obtain feature information focusing on more attributes. Also, when the item image and the text information representing the item attributes of the same item are fused with multimodal features, the features of the item image, the item title information and the item attribute setting information are fused based on the attention mechanism, so as to obtain feature information focusing on more attributes, so as to improve the accuracy of the key information represented by the image-text fusion feature vector.

[0073] Further references Figure 4 , which shows a process 400 of another embodiment of a method for generating feature information of image-text fusion. The process 400 of the method for generating feature information of image-text fusion includes the following steps:

[0074] Step 401, performing feature extraction processing on the object image, object title information and object attribute setting information of the target object to obtain image feature information, title feature information and first attribute feature information.

[0075] Step 402: input the image feature information and the title feature information into a first image-text feature fusion model to obtain first image-text fusion feature information.

[0076] In some embodiments, the specific implementation of steps 401-402 and the technical effects brought about can be referred to Figure 2 The corresponding steps 201-202 in the embodiments are not described in detail here.

[0077] Step 403 , using the feature weight generation sub-model included in the second image-text feature fusion model, a feature weight information set corresponding to the first image-text fusion feature information is generated according to the first image-text fusion feature information and the first attribute feature information.

[0078] In some embodiments, the above execution entities (e.g. Figure 1 The computing device 101 shown in the figure can utilize the feature weight generation submodel included in the second image-text feature fusion model to generate a feature weight information set corresponding to the first image-text fusion feature information according to the first image-text fusion feature information and the first attribute feature information. The feature weight generation submodel can be used to generate the respective weights of the vectors of the respective dimensions corresponding to the first image-text fusion feature information. The feature weight information in the feature weight information set can be the weights of the vectors of the corresponding dimensions. For example, the feature weight generation submodel can be a bilinear model.

[0079] In practice, the execution subject may generate a sub-model using the feature weights included in the second image-text feature fusion model through the following steps, and generate a feature weight information set corresponding to the first image-text fusion feature information according to the first image-text fusion feature information and the first attribute feature information:

[0080] In the first step, the feature weight generation sub-model included in the second image-text feature fusion model is used to initialize the first preset matrix to obtain a first initialization matrix. The first preset matrix can be a pre-set matrix. In practice, the execution subject can randomly initialize the first preset matrix to obtain the first initialization matrix.

[0081] In the second step, a sub-model is generated using the feature weights included in the second image-text feature fusion model, and the number of dimensions of the first image-text fusion feature information is determined as the number of features.

[0082] In the third step, a sub-model is generated using the feature weights included in the second image-text feature fusion model, and the product of the first initialization matrix and the above-mentioned number of features is determined as the first target matrix.

[0083] The fourth step is to use the feature weight generation sub-model included in the second image-text feature fusion model to initialize the second preset matrix to obtain a second initialization matrix. The second preset matrix can be a pre-set matrix. In practice, the execution subject can randomly initialize the second preset matrix to obtain a second initialization matrix.

[0084] The fifth step is to generate a sub-model using the feature weights included in the second image-text feature fusion model, and perform the following sub-steps for the vector of each dimension included in the first image-text fusion feature information:

[0085] The first sub-step is to perform column concatenation on the first attribute feature vector included in the first attribute feature information and the vector corresponding to the dimension to obtain a concatenated feature vector.

[0086] The second sub-step is to determine the product of the second initialization matrix and the concatenated feature vector as the second target matrix.

[0087] In the third sub-step, the convolution of the first target matrix and the second target matrix is ​​determined as feature weight information.

[0088] In the sixth step, a sub-model is generated by using the feature weights included in the second image-text feature fusion model, and each determined feature weight information is determined as a feature weight information set.

[0089] Step 404 , using the weighted sum sub-model included in the second image-text feature fusion model, and according to the feature weight information set, performing weighted sum processing on the first image-text fusion feature information to obtain second image-text fusion feature information.

[0090] In some embodiments, the execution subject may utilize the weighted summation submodel included in the second image-text feature fusion model to perform weighted summation processing on the first image-text fusion feature information according to the feature weight information set to obtain the second image-text fusion feature information. The weighted summation submodel may be used to perform weighted summation on the vectors of each dimension of the first image-text fusion feature information according to the corresponding feature weight information. For example, the weighted summation submodel may be a weighted summation function. The second image-text fusion feature information may be a feature vector that integrates the features of the item image, item title information, and item attribute setting information.

[0091] In practice, the execution subject may use the weighted summation sub-model included in the second image-text feature fusion model to perform weighted summation processing on the first image-text fusion feature information according to the feature weight information set to obtain the second image-text fusion feature information through the following steps:

[0092] In the first step, the weighted summation sub-model included in the second image-text feature fusion model is used to determine the sum of the feature weights of each feature in the feature weight information set as the feature weight sum.

[0093] In the second step, using the weighted summation sub-model included in the second image-text feature fusion model, for the vector of each dimension included in the first image-text fusion feature information, perform the following sub-steps:

[0094] In the first sub-step, the feature weight information of the vector corresponding to the above dimension is determined as the target feature weight information.

[0095] In the second sub-step, the ratio of the target feature weight information to the sum of the feature weights is determined as the feature weight coefficient.

[0096] In the third sub-step, the product of the feature weight coefficient and the vector corresponding to the dimension is determined as the target vector.

[0097] In the third step, the weighted summation sub-model included in the second image-text feature fusion model is used to determine the sum of the determined target vectors as the second image-text fusion feature information.

[0098] from Figure 4 It can be seen that Figure 2 Compared with the description of some corresponding embodiments, Figure 4 The process 400 of the method for generating the image-text fusion feature information in some corresponding embodiments embodies the step of expanding the feature fusion of the first image-text fusion feature information and the first attribute feature information. Therefore, the schemes described in these embodiments can automatically change the attention through different input attributes, so as to obtain image-text fusion feature information with a high correlation with the attribute information.

[0099] Further references Figure 5 As an implementation of the methods shown in the above figures, the present disclosure provides some embodiments of a device for generating image-text fusion feature information. These device embodiments are similar to Figure 2 Corresponding to the method embodiments shown, the device can be specifically applied to various electronic devices.

[0100] like Figure 5As shown, in some embodiments, the apparatus 500 for generating feature information of image-text fusion includes: a feature extraction unit 501, a first input unit 502, and a second input unit 503. The feature extraction unit 501 is configured to perform feature extraction processing on the object image, the object title information, and the object attribute setting information of the target object to obtain image feature information, title feature information, and first attribute feature information; the first input unit 502 is configured to input the image feature information and the title feature information into a first image-text feature fusion model to obtain first image-text fusion feature information, wherein the first image-text feature fusion model is included in a pre-trained multimodal feature fusion model, and the first image-text feature fusion model is an image-text feature fusion model based on a first attention mechanism; the second input unit 503 is configured to input the first image-text fusion feature information and the first attribute feature information into a second image-text feature fusion model to obtain second image-text fusion feature information, wherein the second image-text feature fusion model is included in the multimodal feature fusion model, and the second image-text feature fusion model is an image-text feature fusion model based on a second attention mechanism.

[0101] Optionally, the image-text fusion feature information generating device 500 may further include a second feature extraction unit and a proofreading unit (not shown in the figure). The second feature extraction unit is configured to perform feature extraction processing on the item attribute filled-in information to obtain second attribute feature information. The proofreading unit is configured to perform proofreading processing on the second image-text fusion feature information and the second attribute feature information to obtain item proofreading result information.

[0102] Optionally, the feature extraction unit 501 can be further configured to: determine the preset category information and item attribute setting information corresponding to the above-mentioned target item as the target item attribute information; input the above-mentioned target item attribute information into a pre-trained attribute feature extraction model to obtain first attribute feature information.

[0103] Optionally, the second input unit 503 can be further configured to: utilize the feature weight generation sub-model included in the above-mentioned second image-text feature fusion model to generate a feature weight information set corresponding to the above-mentioned first image-text fusion feature information according to the above-mentioned first image-text fusion feature information and the above-mentioned first attribute feature information; utilize the weighted summation sub-model included in the above-mentioned second image-text feature fusion model to perform weighted summation processing on the above-mentioned first image-text fusion feature information according to the above-mentioned feature weight information set to obtain the second image-text fusion feature information.

[0104] Optionally, the multimodal feature fusion model is pre-trained through the following steps: obtaining a sample, wherein the sample includes a sample item image, sample item title information, and sample item attribute filling information; performing the following training steps on the sample: performing feature extraction processing on the sample item image, sample item title information, and item attribute setting information included in the sample to obtain sample image feature information, sample title feature information, and first attribute feature information; inputting the sample image feature information and the sample title feature information into an initial first image-text feature fusion model to obtain sample first image-text fusion feature information, wherein the initial first image-text feature fusion model is included in the initial multimodal feature fusion model; inputting the sample first image-text fusion feature information and the first attribute feature information into the initial A second image-text feature fusion model is used to obtain the sample's second image-text fusion feature information, wherein the above-mentioned initial second image-text feature fusion model is included in the above-mentioned initial multimodal feature fusion model; feature extraction processing is performed on the sample item attribute filling information to obtain the sample's second attribute feature information; the sample's second image-text fusion feature information is compared with the sample's second attribute feature information; according to the comparison result, it is determined whether the initial multimodal feature fusion model has achieved a preset optimization goal; in response to determining that the initial multimodal feature fusion model has not achieved the above-mentioned optimization goal, the network parameters of the initial multimodal feature fusion model are adjusted, and the adjusted initial multimodal feature fusion model is used as the initial multimodal feature fusion model, and the item sample is reacquired as a sample, and the above-mentioned training steps are performed again.

[0105] Optionally, the training step further includes: in response to determining that the initial multimodal feature fusion model achieves the optimization target, determining the initial multimodal feature fusion model as a trained multimodal feature fusion model.

[0106] Optionally, after performing feature extraction processing on the sample item image, sample item title information and item attribute setting information included in the sample to obtain sample image feature information, sample title feature information and first attribute feature information, the above-mentioned training step also includes: performing regularization processing on the sample title feature information to obtain target sample title feature information as the sample title feature information.

[0107] It can be understood that the units recorded in the image-text fusion feature information generating device 500 are similar to those in the reference Figure 2 Therefore, the operations, features and beneficial effects described above for the method are also applicable to the device 500 and the units included therein, and will not be described in detail here.

[0108] Reference below Figure 6 , which shows an electronic device 600 (eg, Figure 1The electronic devices in some embodiments of the present disclosure may include but are not limited to mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 6 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0109] like Figure 6 As shown, the electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device 600 are also stored. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0110] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 6 The electronic device 600 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 6 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0111] In particular, according to some embodiments of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from the network through the communication device 609, or installed from the storage device 608, or installed from the ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of some embodiments of the present disclosure are executed.

[0112] It should be noted that the computer-readable medium recorded in some embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In some embodiments of the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0113] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (Hyper Text Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0114] The computer-readable medium may be included in the electronic device; or it may exist independently without being installed in the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: performs feature extraction processing on the object image, the object title information and the object attribute setting information of the target object to obtain image feature information, title feature information and first attribute feature information; inputs the image feature information and the title feature information into a first image-text feature fusion model to obtain first image-text fusion feature information, wherein the first image-text feature fusion model is included in a pre-trained multimodal feature fusion model, and the first image-text feature fusion model is an image-text feature fusion model based on a first attention mechanism; inputs the first image-text fusion feature information and the first attribute feature information into a second image-text feature fusion model to obtain second image-text fusion feature information, wherein the second image-text feature fusion model is included in the multimodal feature fusion model, and the second image-text feature fusion model is an image-text feature fusion model based on a second attention mechanism.

[0115] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0116] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0117] The units described in some embodiments of the present disclosure may be implemented by software or hardware. The described units may also be set in a processor, for example, it may be described as: a processor includes a feature extraction unit, a first input unit, and a second input unit. The names of these units do not constitute limitations on the units themselves in certain cases. For example, the feature extraction unit may also be described as "a unit that performs feature extraction processing on the object image, the object title information, and the object attribute setting information of the target object to obtain image feature information, title feature information, and first attribute feature information."

[0118] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0119] Some embodiments of the present disclosure further provide a computer program product, including a computer program, which implements any of the above-mentioned methods for generating image-text fusion feature information when executed by a processor.

[0120] The above descriptions are only some preferred embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with the technical features with similar functions disclosed in the embodiments of the present disclosure (but not limited to) and the technical solutions formed.

Claims

1. A method for generating image-text fusion feature information, comprising: Performing feature extraction processing on the object image, the object title information and the object attribute setting information of the target object to obtain image feature information, title feature information and first attribute feature information; Inputting the image feature information and the title feature information into a first image-text feature fusion model to obtain first image-text fusion feature information, wherein the first image-text feature fusion model is included in a pre-trained multimodal feature fusion model, and the first image-text feature fusion model is an image-text feature fusion model based on a first attention mechanism; The first image-text fusion feature information and the first attribute feature information are input into a second image-text feature fusion model to obtain second image-text fusion feature information, wherein the second image-text feature fusion model is included in the multimodal feature fusion model, and the second image-text feature fusion model is an image-text feature fusion model based on a second attention mechanism.

2. The method according to claim 1, wherein: The method further comprises: Perform feature extraction processing on the item attribute filled-in information to obtain second attribute feature information; The second image-text fusion feature information and the second attribute feature information are proofread to obtain item proofreading result information.

3. The method according to claim 1, wherein: The feature extraction process is performed on the object image, the object title information and the object attribute setting information of the target object to obtain the image feature information, the title feature information and the first attribute feature information, including: Determining the preset category information and item attribute setting information corresponding to the target item as the target item attribute information; The target item attribute information is input into a pre-trained attribute feature extraction model to obtain first attribute feature information.

4. The method according to claim 1, wherein: The step of inputting the first image-text fusion feature information and the first attribute feature information into a second image-text feature fusion model to obtain second image-text fusion feature information includes: Using the feature weight generation submodel included in the second image-text feature fusion model, a feature weight information set corresponding to the first image-text fusion feature information is generated according to the first image-text fusion feature information and the first attribute feature information; The weighted sum sub-model included in the second image-text feature fusion model is used to perform weighted sum processing on the first image-text fusion feature information according to the feature weight information set to obtain second image-text fusion feature information.

5. The method according to claim 1, wherein: The multimodal feature fusion model is pre-trained through the following steps: Acquire a sample, wherein the sample includes a sample item image, sample item title information, and sample item attribute filled-in information; Perform the following training steps on the sample: Performing feature extraction processing on the sample item image, the sample item title information and the item attribute setting information included in the sample to obtain sample image feature information, sample title feature information and first attribute feature information; Inputting sample image feature information and sample title feature information into an initial first image-text feature fusion model to obtain sample first image-text fusion feature information, wherein the initial first image-text feature fusion model is included in the initial multimodal feature fusion model; Inputting the sample first image-text fusion feature information and the first attribute feature information into the initial second image-text feature fusion model to obtain the sample second image-text fusion feature information, wherein the initial second image-text feature fusion model is included in the initial multimodal feature fusion model; Perform feature extraction on the filled-in information of the sample item attributes to obtain feature information of the second attribute of the sample; Compare the second image-text fusion feature information of the sample with the second attribute feature information of the sample; Determine whether the initial multimodal feature fusion model reaches the preset optimization goal according to the comparison result; In response to determining that the initial multimodal feature fusion model does not achieve the optimization goal, adjusting the network parameters of the initial multimodal feature fusion model, using the adjusted initial multimodal feature fusion model as the initial multimodal feature fusion model, reacquiring object samples as samples, and performing the training step again.

6. The method according to claim 5, wherein: The training step further comprises: In response to determining that the initial multimodal feature fusion model achieves the optimization goal, the initial multimodal feature fusion model is determined as a trained multimodal feature fusion model.

7. The method according to claim 5, wherein: After performing feature extraction processing on the sample item image, sample item title information and item attribute setting information included in the sample to obtain sample image feature information, sample title feature information and first attribute feature information, the training step further includes: The sample title feature information is regularized to obtain the target sample title feature information as the sample title feature information.

8. A device for generating feature information of image-text fusion, comprising: a feature extraction unit configured to perform feature extraction processing on an object image, object title information, and object attribute setting information of a target object to obtain image feature information, title feature information, and first attribute feature information; A first input unit is configured to input the image feature information and the title feature information into a first image-text feature fusion model to obtain first image-text fusion feature information, wherein the first image-text feature fusion model is included in a pre-trained multimodal feature fusion model, and the first image-text feature fusion model is an image-text feature fusion model based on a first attention mechanism; The second input unit is configured to input the first image-text fusion feature information and the first attribute feature information into a second image-text feature fusion model to obtain second image-text fusion feature information, wherein the second image-text feature fusion model is included in the multimodal feature fusion model, and the second image-text feature fusion model is an image-text feature fusion model based on a second attention mechanism.

9. An electronic device, comprising: one or more processors; a storage device having one or more programs stored thereon, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A computer readable medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

11. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 7.

Images