Model training method, and multi-modal feature information generation method and device

Abstract

The embodiment of the invention discloses a model training method, a multi-modal feature information generation method and a multi-modal feature information generation device. A specific embodiment of the method comprises the following steps: selecting image-text data from an image-text data set, and executing the following training steps: inputting the image data into an initial image feature extraction model to generate at least one piece of image block feature information, and inputting text data into an initial text feature extraction model to generate at least one piece of image block feature information; generating at least one piece of text character feature information; generating at least one piece of modal common feature information and at least one piece of modal proprietary feature information; multi-modal feature information is generated; generating attribute difference loss information; determining whether to train the initial multi-modal feature fusion model or not; and determining the initial multi-modal feature fusion model as a multi-modal feature fusion model in response to determining that training is not performed. The implementation mode is related to artificial intelligence, the multi-modal feature fusion model can be utilized, and the multi-modal feature information for the image-text data can be accurately generated.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of computer technology, and in particular to a model training method, a multimodal feature information generating method and a device. Background Art

[0002] At present, the generation of cross-modal representation information based on image and text pairs has made rapid progress. For the feature representation of image and text data, the usual sampling method is to generate the corresponding feature representation information of image and text data through a single-stream model or a two-stream model.

[0003] However, the inventors have found that when the above method is used to generate feature representation information corresponding to graphic data, the following technical problems often occur:

[0004] There is a problem of inconsistent information between image and text data, that is, the attribute information of the item corresponding to the image cannot be fully reflected by the text, resulting in the lack of some item attribute information in the text. Single-stream models or dual-stream models can often only perform feature representation on the common item attribute information between the image and the text, resulting in the obtained feature representation information being inaccurate.

[0005] The above information disclosed in this Background section is only for enhancement of understanding of the background of the inventive concept and therefore it may contain information that does not form the prior art that is already known in this country to a person of ordinary skill in the art. Summary of the invention

[0006] The content of this disclosure is used to introduce concepts in a brief form, which will be described in detail in the detailed implementation section below. The content of this disclosure is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to limit the scope of the technical solution claimed for protection.

[0007] Some embodiments of the present disclosure propose model training methods, multimodal feature information generation methods, devices, electronic devices, computer-readable media and program products to solve the technical problems mentioned in the above background technology section.

[0008] In a first aspect, some embodiments of the present disclosure provide a model training method, comprising: obtaining a graphic data set and an item attribute information set corresponding to each graphic data; selecting graphic data from the graphic data set as first target graphic data, and performing the following training steps: inputting the image data included in the first target graphic data into an initial image feature extraction model included in an initial multimodal feature fusion model to generate at least one image block feature information, and inputting the text data included in the first target graphic data into an initial text feature extraction model included in the initial multimodal feature fusion model to generate at least one text character feature information; according to the at least one image block feature information and the above-mentioned at least one text character feature information, generate at least one modal common feature information and at least one modal specific feature information; generate multimodal feature information according to the feature fusion model included in the above-mentioned at least one modal common feature information, the above-mentioned at least one modal specific feature information and the above-mentioned initial multimodal feature fusion model; generate attribute difference loss information for the above-mentioned multimodal feature information according to the item attribute information set corresponding to the above-mentioned first target graphic data; determine whether to train the above-mentioned initial multimodal feature fusion model according to the above-mentioned attribute difference loss information; in response to determining not to train, determine the above-mentioned initial multimodal feature fusion model as the multimodal feature fusion model.

[0009] Optionally, the above method also includes: in response to determining the training, updating the above initial multimodal feature fusion model according to the above attribute difference loss information, obtaining an updated multimodal feature fusion model as the initial multimodal feature fusion model, and re-selecting image and text data from the above image and text data set as the first target image and text data to continue executing the above training steps.

[0010] Optionally, the generating of at least one modality common feature information and at least one modality specific feature information based on the at least one image block feature information and the at least one text character feature information comprises: for each image block feature information in the at least one image block feature information, performing the following generating steps: determining text character feature information corresponding to the image block feature information as target text character feature information; determining semantic feature similarity between the image block feature information and the target text character feature information; in response to determining that the semantic feature similarity is greater than a predetermined value, generating modality common feature information based on the image block feature information and the target text character feature information.

[0011] Optionally, after generating modal common feature information according to the image block feature information and the target text character feature information in response to determining that the semantic feature similarity is greater than a predetermined value, the method further includes: in response to determining that the semantic feature similarity is less than or equal to the predetermined value, respectively determining the image block feature information and the target text character feature information as modal-specific feature information.

[0012] Optionally, the multimodal feature information is generated based on the feature fusion model included in the at least one modal common feature information, the at least one modal specific feature information and the initial multimodal feature fusion model, including: splicing feature information of the at least one modal common feature information and the at least one modal specific feature information to obtain spliced ​​feature information; and inputting the spliced ​​feature information into the feature fusion model to generate the multimodal feature information.

[0013] Optionally, the above-mentioned initial image feature extraction model is initialized based on a pre-trained image semantic feature extraction model, the above-mentioned initial text feature extraction model and the above-mentioned feature fusion model are initialized based on a pre-trained language model, and the above-mentioned language model and the above-mentioned image semantic feature extraction model are pre-semantically aligned models.

[0014] In a second aspect, some embodiments of the present disclosure provide a model training device, comprising: a first acquisition unit, configured to acquire a graphic data set and an item attribute information set corresponding to each graphic data set; a training unit, configured to select graphic data from the above-mentioned graphic data set as first target graphic data, and perform the following training steps: input the image data included in the above-mentioned first target graphic data into an initial image feature extraction model included in an initial multimodal feature fusion model to generate at least one image block feature information, and input the text data included in the above-mentioned first target graphic data into an initial text feature extraction model included in the above-mentioned initial multimodal feature fusion model to generate at least one text character feature information; based on At least one modality common feature information and at least one modality specific feature information are generated based on the at least one image block feature information and the at least one text character feature information; multimodal feature information is generated based on the at least one modality common feature information, the at least one modality specific feature information and the feature fusion model included in the initial multimodal feature fusion model; attribute difference loss information for the multimodal feature information is generated based on the item attribute information set corresponding to the first target graphic data; whether to train the initial multimodal feature fusion model is determined based on the attribute difference loss information; in response to determining not to train, the initial multimodal feature fusion model is determined as a multimodal feature fusion model.

[0015] Optionally, the above-mentioned device also includes: in response to determining the training, updating the above-mentioned initial multimodal feature fusion model according to the above-mentioned attribute difference loss information, obtaining an updated multimodal feature fusion model as the initial multimodal feature fusion model, and re-selecting graphic data from the above-mentioned graphic data set as the first target graphic data to continue executing the above-mentioned training steps.

[0016] Optionally, the training unit can be configured to: for each image block feature information of the at least one image block feature information mentioned above, perform the following generation steps: determine the text character feature information corresponding to the above image block feature information as the target text character feature information; determine the semantic feature similarity between the above image block feature information and the above target text character feature information; in response to determining that the above semantic feature similarity is greater than a predetermined value, generate modal common feature information based on the above image block feature information and the above target text character feature information.

[0017] Optionally, the training unit may be configured to: in response to determining that the semantic feature similarity is less than or equal to the predetermined value, respectively determine the image block feature information and the target text character feature information as modality-specific feature information.

[0018] Optionally, the training unit can be configured to: splice feature information of the at least one modality common feature information and the at least one modality specific feature information to obtain spliced ​​feature information; input the spliced ​​feature information into the feature fusion model to generate the multimodal feature information.

[0019] Optionally, the above-mentioned initial image feature extraction model is initialized based on a pre-trained image semantic feature extraction model, the above-mentioned initial text feature extraction model and the above-mentioned feature fusion model are initialized based on a pre-trained language model, and the above-mentioned language model and the above-mentioned image semantic feature extraction model are pre-semantically aligned models.

[0020] In a third aspect, some embodiments of the present disclosure provide a method for generating multimodal feature information, comprising: obtaining second target image and text data; inputting the second target image and text data into a pre-trained multimodal feature fusion model to generate multimodal feature information for the second target image and text data, wherein the multimodal feature fusion model is generated based on the model training method disclosed in the first method.

[0021] Optionally, the method further includes: inputting the multimodal feature information for the second target graphic data into a pre-trained item attribute recognition model to generate an item attribute information set for the second target graphic data.

[0022] In a fourth aspect, some embodiments of the present disclosure provide a multimodal feature information generating device, comprising: a second acquisition unit, configured to acquire second target image and text data; a generation unit, configured to input the above-mentioned second target image and text data into a pre-trained multimodal feature fusion model to generate multimodal feature information for the above-mentioned second target image and text data, wherein the above-mentioned multimodal feature fusion model is generated based on the model training method disclosed in the first method.

[0023] Optionally, the device further includes: inputting the multimodal feature information for the second target graphic data into a pre-trained item attribute recognition model to generate an item attribute information set for the second target graphic data.

[0024] In a fifth aspect, some embodiments of the present disclosure provide an electronic device comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any one of the implementation methods in the first and third aspects.

[0025] In a sixth aspect, some embodiments of the present disclosure provide a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the method described in any one of the implementation modes of the first aspect and the third aspect is implemented.

[0026] In a seventh aspect, some embodiments of the present disclosure provide a computer program product, including a computer program, which, when executed by a processor, implements the method described in any one of the implementation modes of the first and third aspects above.

[0027] The above-mentioned various embodiments of the present disclosure have the following beneficial effects: through the model training method of some embodiments of the present disclosure, a multimodal feature fusion model can be used to accurately generate multimodal feature information for image and text data. Specifically, the reason for the inaccurate generation of relevant multimodal feature information is that there is a problem of inconsistency between image and text data, that is, the item attribute information corresponding to the image cannot be fully reflected by the text, resulting in the lack of some item attribute information in the text. However, a single-stream model or a dual-stream model can often only perform feature characterization on the common item attribute information between the image and the text, resulting in the obtained feature characterization information being not accurate enough. Based on this, the model training method of some embodiments of the present disclosure, first, obtains an image and text data set and an item attribute information set corresponding to each image and text data. Here, the image and text data set is obtained as a training data set to perform model training on the subsequent initial multimodal feature fusion model. The obtained item attribute information set can be used as a training label to perform model training on the subsequent initial multimodal feature fusion model. Then, the image data is selected from the above-mentioned image data set as the first target image data, and the following training steps are performed: the first step is to input the image data included in the above-mentioned first target image data into the initial image feature extraction model included in the initial multimodal feature fusion model to generate at least one image block feature information, and the text data included in the above-mentioned first target image data is input into the initial text feature extraction model included in the above-mentioned initial multimodal feature fusion model to generate at least one text character feature information. Here, the image block feature extraction of the image data included in the first target image data is realized by the initial image feature extraction model, so as to facilitate the subsequent generation of multimodal feature information. The text character feature extraction of the text data included in the first target image data is realized by the initial text feature extraction model, so as to facilitate the subsequent generation of multimodal feature information. The second step is to accurately generate at least one modality common feature information and at least one modality specific feature information based on the above-mentioned at least one image block feature information and the above-mentioned at least one text character feature information. Here, by generating at least one modality common feature information, the common feature information between image data and text data can be effectively distinguished. By generating at least one modality professional feature information, feature information with large differences between image data and text data can be effectively distinguished. In the third step, multimodal feature information is generated based on the feature fusion model included in the at least one modality common feature information, the at least one modality-specific feature information, and the initial multimodal feature fusion model. Here, by considering at least one modality common feature information and at least one modality-specific feature information to subsequently generate multimodal feature information, it can be ensured that multimodal feature information with richer and more comprehensive feature information is generated under the premise that there is no redundancy in the feature information. In addition, the feature fusion model can also ensure the accuracy of the generation of multimodal feature information.The fourth step is to generate attribute difference loss information for the multimodal feature information according to the item attribute information set corresponding to the first target graphic data, so as to be used for the subsequent determination of whether the initial multimodal feature fusion model has been trained. The fifth step is to accurately determine whether to train the initial multimodal feature fusion model according to the attribute difference loss information. The sixth step is to determine the initial multimodal feature fusion model as a multimodal feature fusion model in response to the determination not to train. In summary, by generating at least one modality common feature information and at least one modality-specific feature information, not only can the feature difference between image data and text data be effectively distinguished, but also multimodal feature information including richer and more comprehensive feature information can be generated. Therefore, by training the initial multimodal feature fusion model through the attribute difference loss information corresponding to the multimodal feature information, a multimodal feature fusion model that generates multimodal feature information more accurately can be obtained. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0029] Figure 1 is a schematic diagram of an application scenario of a model training method according to some embodiments of the present disclosure;

[0030] Figure 2 is a flowchart of some embodiments of the model training method according to the present disclosure;

[0031] Figure 3 is a flowchart of other embodiments of the model training method according to the present disclosure;

[0032] Figure 4 is a flowchart of some embodiments of the multimodal feature information generation method according to the present disclosure;

[0033] Figure 5 is a schematic diagram of the structure of some embodiments of the model training device according to the present disclosure;

[0034] Figure 6 is a schematic diagram of the structure of some embodiments of the multimodal feature information generating device according to the present disclosure;

[0035] Figure 7 It is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0036] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0037] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure can be combined with each other.

[0038] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0039] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0040] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0041] With regard to the collection, storage, and use of user personal information (such as graphic and text data) involved in this disclosure, before performing the corresponding operations, the relevant organizations or individuals shall fulfill their obligations, including conducting personal information security impact assessments, fulfilling the obligation to inform the personal information subject, and obtaining the authorization and consent of the personal information subject in advance.

[0042] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0043] Figure 1 It is a schematic diagram of an application scenario of the model training method according to some embodiments of the present disclosure.

[0044] exist Figure 1In the application scenario, first, the electronic device 101 can obtain a graphic data set and an item attribute information set corresponding to each graphic data. Then, graphic data is selected from the above-mentioned graphic data set as the first target graphic data 102, and the following training steps are performed: In the first step, the electronic device 101 can input the image data 1021 included in the above-mentioned first target graphic data 102 into the initial image feature extraction model 1031 included in the initial multimodal feature fusion model 103 to generate at least one image block feature information 104, and input the text data 1022 included in the above-mentioned first target graphic data 102 into the initial text feature extraction model 1032 included in the above-mentioned initial multimodal feature fusion model 103 to generate at least one text character feature information 105. In the second step, the electronic device 101 can generate at least one modality common feature information 106 and at least one modality specific feature information 107 based on the above-mentioned at least one image block feature information 104 and the above-mentioned at least one text character feature information 105. In the third step, the electronic device 101 may generate multimodal feature information 108 based on the at least one modal common feature information 106, the at least one modal specific feature information 107 and the feature fusion model 1033 included in the initial multimodal feature fusion model 103. In the fourth step, the electronic device 101 may generate attribute difference loss information 110 for the multimodal feature information 108 based on the item attribute information set 109 corresponding to the first target graphic data 102. In the fifth step, the electronic device 101 may determine whether to train the initial multimodal feature fusion model 103 based on the attribute difference loss information 110. In the sixth step, in response to determining not to train, the electronic device 101 may determine the initial multimodal feature fusion model 103 as a multimodal feature fusion model.

[0045] It should be noted that the electronic device 101 can be hardware or software. When the electronic device is hardware, it can be implemented as a distributed cluster consisting of multiple servers or terminal devices, or it can be implemented as a single server or a single terminal device. When the electronic device is embodied as software, it can be installed in the hardware devices listed above. It can be implemented as multiple software or software modules for providing distributed services, for example, or it can be implemented as a single software or software module. No specific limitation is made here.

[0046] It should be understood that Figure 1 The number of electronic devices in the embodiment is only for illustration. Any number of electronic devices may be provided according to implementation requirements.

[0047] Continue to refer Figure 2 , shows a process 200 of some embodiments of the model training method according to the present disclosure. The model training method comprises the following steps:

[0048] Step 201: Obtain a set of image and text data and an item attribute information set corresponding to each image and text data.

[0049] In some embodiments, the execution entity of the above model training method (for example Figure 1 The electronic device 101 shown) can obtain a graphic data set and an item attribute information set corresponding to each graphic data through a wired connection or a wireless connection. The graphic data in the graphic data set can be a data pair consisting of image data and text data. Specifically, for e-commerce scenarios, the graphic data can be a data pair consisting of an item image and an item title. The item attribute information in the item attribute information set can be the attribute content of the item attribute. Specifically, for e-commerce scenarios, the item attribute can be a commodity attribute. For example, the item attribute can be one of the following: item price, item sales volume, item discount, item color.

[0050] It should be noted that each graphic data has a corresponding, pre-labeled item attribute information set.

[0051] Step 202, selecting image and text data from the above image and text data set as the first target image and text data, and performing the following training steps:

[0052] Step 2021: input the image data included in the above-mentioned first target image and text data into the initial image feature extraction model included in the initial multimodal feature fusion model to generate at least one image block feature information, and input the text data included in the above-mentioned first target image and text data into the initial text feature extraction model included in the above-mentioned initial multimodal feature fusion model to generate at least one text character feature information.

[0053] In some embodiments, the execution subject may input the image data included in the first target image-text data into the initial image feature extraction model included in the initial multimodal feature fusion model to generate at least one image block feature information, and input the text data included in the first target image-text data into the initial text feature extraction model included in the initial multimodal feature fusion model to generate at least one text character feature information. The initial multimodal feature fusion model may be a multimodal feature fusion model whose model has not yet been trained. The multimodal feature fusion model may be a model that fuses information of modal feature information of multiple modalities. In practice, the multiple modalities may include: image modality and text modality. The multimodal feature fusion model may be a semantically aligned visual and text pre-training model. In practice, the multimodal feature fusion model may be a Visual BERT model. The initial image feature extraction model may be an image feature extraction model whose model has not yet been trained. The image feature extraction model may be a model that extracts image feature information. In practice, the image feature extraction model may be a multi-layer serially connected convolutional neural network. The image block feature information in at least one image block feature information may represent feature information of an image block in the image data. At least one image block feature information may represent the overall image feature information of the image data. The initial text feature extraction model may be a text feature extraction model whose training has not yet been completed. The text feature extraction model may be a model for extracting text feature information from text data. In practice, the text feature extraction model may be a predetermined layer model in the BERT model. The text character feature information in at least one text character feature information may represent the feature information of the characters in the text. At least one text character feature information may represent the feature information of the text as a whole. There is a semantic correspondence between the image block in at least one image block feature information and the text character in at least one text character feature information.

[0054] Step 2022: Generate at least one modality common feature information and at least one modality specific feature information based on the at least one image block feature information and the at least one text character feature information.

[0055] In some embodiments, the execution subject may generate at least one modality-common feature information and at least one modality-specific feature information based on the at least one image block feature information and the at least one text character feature information. The modality-common feature information may be feature information with similar semantic features between image data and text data. The modality-common feature information may be feature information with similar semantic features between image data and text data. The modality-specific feature information may be feature information with large semantic feature differences between image data and text data. At least one modality-specific feature information may be all feature information with large semantic feature differences between image data and text data. In practice, the modality-common feature information may be attribute feature information with a small semantic gap between image data and text data. The modality-specific feature information may be attribute feature information with a large semantic gap between image data and text data.

[0056] As an example, first, the execution subject may input at least one image block feature information and at least one text character feature information into a similar feature fusion model to generate at least one similar fused feature information as at least one modality common feature information. The similar feature fusion model may be a model that fuses the image block feature information and the text character feature information. In practice, the similar feature fusion model may be a convolutional neural network with multiple layers connected in series. Then, the execution subject may remove the image block feature information subset and the text character feature information subset associated with the at least one modality common feature information from the at least one image block feature information and the at least one text character feature information to obtain the removed image block feature information subset and the removed text character feature information subset. Finally, the removed image block feature information subset and the removed text character feature information subset are subjected to feature information fusion to obtain a fused feature information set as at least one modality-specific feature information.

[0057] In some optional implementations of some embodiments, the above-mentioned initial image feature extraction model is initialized based on a pre-trained image semantic feature extraction model. The above-mentioned initial text feature extraction model and the above-mentioned feature fusion model are initialized based on a pre-trained language model. The above-mentioned language model and the above-mentioned image semantic feature extraction model are pre-semantically aligned models. Among them, the pre-trained image semantic feature extraction model may be an image semantic feature extraction model whose model parameters have been pre-updated. The image semantic feature extraction model may be a model for extracting semantic feature information from an input image. In practice, the image semantic feature extraction model may be a residual neural network with multiple layers of serial connections. The pre-trained language model may be a language model whose model parameters have been pre-updated. In practice, the language model may be a 12-layer BERT model.

[0058] As an example, the execution subject may use the model parameter set of the image semantic feature extraction model as the initial parameter set of the initial image feature extraction model to achieve model initialization of the initial image feature extraction model. Similarly, the execution subject may use the model parameter set of the first 6 layers of the BERT model of the language model as the initial parameter set of the initial text feature extraction model to achieve model initialization of the initial text feature extraction model. The execution subject may use the model parameter set of the last 6 layers of the BERT model of the language model as the initial parameter set of the feature fusion model to achieve model initialization of the feature fusion model.

[0059] Step 2023: Generate multimodal feature information based on the at least one modality-common feature information, the at least one modality-specific feature information, and the feature fusion model included in the initial multimodal feature fusion model.

[0060] In some embodiments, the execution subject may generate multimodal feature information based on the feature information common to the at least one modality, the feature information specific to the at least one modality, and the feature fusion model included in the initial multimodal feature fusion model. The feature fusion model may be a model for performing attention fusion on feature information. Specifically, the feature fusion model may be a multi-head attention mechanism model. The multimodal feature information may characterize the overall feature information of the first target graphic data.

[0061] As an example, the execution entity may directly input the at least one modality common feature information and the at least one modality specific feature information into a feature fusion model to generate multimodal feature information.

[0062] In some optional implementations of some embodiments, the step of generating multimodal feature information according to the at least one modality common feature information, the at least one modality-specific feature information, and the feature fusion model included in the initial multimodal feature fusion model may include the following steps:

[0063] In the first step, the execution entity may concatenate the at least one modality-common feature information and the at least one modality-specific feature information to obtain concatenated feature information.

[0064] In a second step, the execution entity may input the spliced ​​feature information into the feature fusion model to generate the multimodal feature information.

[0065] Step 2024: Generate attribute difference loss information for the multimodal feature information based on the item attribute information set corresponding to the first target graphic data.

[0066] In some embodiments, the execution subject may generate attribute difference loss information for the multimodal feature information based on the item attribute information set corresponding to the first target graphic data. The attribute difference loss information may represent the attribute information difference between the item attribute prediction information set corresponding to the multimodal feature information and the item attribute information set corresponding to the first target graphic data.

[0067] As an example, first, the execution subject may input the multimodal feature information into the attribute recognition model to generate an item attribute prediction information set corresponding to the multimodal feature information. Then, based on the item attribute prediction information set and the item attribute information set, the attribute difference loss information for the multimodal feature information is generated using a preset loss function.

[0068] Step 2025: Determine whether to train the initial multimodal feature fusion model based on the attribute difference loss information.

[0069] In some embodiments, the execution entity may determine whether to train the initial multimodal feature fusion model based on the attribute difference loss information.

[0070] As an example, in response to determining that the attribute difference loss information is less than a predetermined value, it is determined not to train the initial multimodal feature fusion model. In response to determining that the attribute difference loss information is greater than or equal to a predetermined value, it is determined to train the initial multimodal feature fusion model.

[0071] Step 2026, in response to determining not to train, determining the above-mentioned initial multimodal feature fusion model as the multimodal feature fusion model.

[0072] In some embodiments, in response to determining not to train, the execution entity may determine the initial multimodal feature fusion model as the multimodal feature fusion model.

[0073] In some optional implementations of some embodiments, after step 202, the steps further include:

[0074] In response to determining the training, the above-mentioned execution entity can update the above-mentioned initial multimodal feature fusion model according to the above-mentioned attribute difference loss information, obtain the updated multimodal feature fusion model as the initial multimodal feature fusion model, and re-select image and text data from the above-mentioned image and text data set as the first target image and text data to continue to execute the above-mentioned training steps.

[0075] It should be noted that the reselection of graphic data from the graphic data set may be random selection of graphic data that has never been selected in the past.

[0076] The above-mentioned various embodiments of the present disclosure have the following beneficial effects: through the model training method of some embodiments of the present disclosure, a multimodal feature fusion model can be used to accurately generate multimodal feature information for image and text data. Specifically, the reason for the inaccurate generation of relevant multimodal feature information is that there is a problem of inconsistency between image and text data, that is, the item attribute information corresponding to the image cannot be fully reflected by the text, resulting in the lack of some item attribute information in the text. However, a single-stream model or a dual-stream model can often only perform feature characterization on the common item attribute information between the image and the text, resulting in the obtained feature characterization information being not accurate enough. Based on this, the model training method of some embodiments of the present disclosure, first, obtains an image and text data set and an item attribute information set corresponding to each image and text data. Here, the image and text data set is obtained as a training data set to perform model training on the subsequent initial multimodal feature fusion model. The obtained item attribute information set can be used as a training label to perform model training on the subsequent initial multimodal feature fusion model. Then, the image data is selected from the above-mentioned image data set as the first target image data, and the following training steps are performed: the first step is to input the image data included in the above-mentioned first target image data into the initial image feature extraction model included in the initial multimodal feature fusion model to generate at least one image block feature information, and the text data included in the above-mentioned first target image data is input into the initial text feature extraction model included in the above-mentioned initial multimodal feature fusion model to generate at least one text character feature information. Here, the image block feature extraction of the image data included in the first target image data is realized by the initial image feature extraction model, so as to facilitate the subsequent generation of multimodal feature information. The text character feature extraction of the text data included in the first target image data is realized by the initial text feature extraction model, so as to facilitate the subsequent generation of multimodal feature information. The second step is to accurately generate at least one modality common feature information and at least one modality specific feature information based on the above-mentioned at least one image block feature information and the above-mentioned at least one text character feature information. Here, by generating at least one modality common feature information, the common feature information between image data and text data can be effectively distinguished. By generating at least one modality professional feature information, feature information with large differences between image data and text data can be effectively distinguished. In the third step, multimodal feature information is generated based on the feature fusion model included in the at least one modality common feature information, the at least one modality-specific feature information, and the initial multimodal feature fusion model. Here, by considering at least one modality common feature information and at least one modality-specific feature information to subsequently generate multimodal feature information, it can be ensured that multimodal feature information with richer and more comprehensive feature information is generated under the premise that there is no redundancy in the feature information. In addition, the feature fusion model can also ensure the accuracy of the generation of multimodal feature information.The fourth step is to generate attribute difference loss information for the multimodal feature information according to the item attribute information set corresponding to the first target graphic data, so as to be used for the subsequent determination of whether the initial multimodal feature fusion model has been trained. The fifth step is to accurately determine whether to train the initial multimodal feature fusion model according to the attribute difference loss information. The sixth step is to determine the initial multimodal feature fusion model as a multimodal feature fusion model in response to the determination not to train. In summary, by generating at least one modality common feature information and at least one modality-specific feature information, not only can the feature difference between image data and text data be effectively distinguished, but also multimodal feature information including richer and more comprehensive feature information can be generated. Therefore, by training the initial multimodal feature fusion model through the attribute difference loss information corresponding to the multimodal feature information, a multimodal feature fusion model that generates multimodal feature information more accurately can be obtained.

[0077] Further references Figure 3 , shows a process 300 of other embodiments of the model training method according to the present disclosure. The model training method comprises the following steps:

[0078] Step 301: Obtain a set of image and text data and an item attribute information set corresponding to each image and text data.

[0079] Step 302, selecting image and text data from the above image and text data set as the first target image and text data, and performing the following training steps:

[0080] Step 3021, input the image data included in the above-mentioned first target image and text data into the initial image feature extraction model included in the initial multimodal feature fusion model to generate at least one image block feature information, and input the text data included in the above-mentioned first target image and text data into the initial text feature extraction model included in the above-mentioned initial multimodal feature fusion model to generate at least one text character feature information.

[0081] Step 3022: for each piece of image block feature information in the at least one piece of image block feature information, perform the following generation steps:

[0082] Step 30221, determining text character feature information corresponding to the above-mentioned image block feature information as target text character feature information.

[0083] In some embodiments, an execution entity (e.g., Figure 1 The electronic device 101 shown can determine the text character feature information corresponding to the above-mentioned image block feature information as the target text character feature information.

[0084] Step 30222, determining the semantic feature similarity between the above-mentioned image block feature information and the above-mentioned target text character feature information.

[0085] In some embodiments, the execution entity may determine the semantic feature similarity between the image block feature information and the target text character feature information. The semantic feature similarity may represent the feature semantic similarity between the image block feature information and the target text character feature information. The semantic feature similarity may be a value between 0 and 1. The greater the semantic feature similarity, the higher the feature semantic similarity between the image block feature information and the target text character feature information.

[0086] As an example, the execution subject may determine the semantic feature similarity between the image block feature information and the target text character feature information through a similarity measurement function.

[0087] Step 30223, in response to determining that the semantic feature similarity is greater than a predetermined value, modal common feature information is generated based on the image block feature information and the target text character feature information.

[0088] In some embodiments, in response to determining that the semantic feature similarity is greater than a predetermined value, the execution subject may generate modality common feature information based on the image block feature information and the target text character feature information. For example, the predetermined value may be 0.5.

[0089] As an example, the execution subject may randomly select a piece of feature information from the image block feature information and the target text character feature information as the modality common feature information.

[0090] As another example, the execution entity may perform weighted sum processing on the image block feature information and the target text character feature information to generate weighted sum feature information as modality common feature information.

[0091] In some optional implementations of some embodiments, after step 30223, the steps further include:

[0092] In response to determining that the semantic feature similarity is less than or equal to the predetermined value, the execution entity may determine the image block feature information and the target text character feature information as modality-specific feature information, respectively.

[0093] Step 3023: Generate multimodal feature information based on the at least one modality-common feature information, the at least one modality-specific feature information, and the feature fusion model included in the initial multimodal feature fusion model.

[0094] Step 3024: Generate attribute difference loss information for the multimodal feature information according to the item attribute information set corresponding to the first target graphic data.

[0095] Step 3025: Determine whether to train the initial multimodal feature fusion model based on the attribute difference loss information.

[0096] Step 3026, in response to determining not to train, determining the above-mentioned initial multimodal feature fusion model as the multimodal feature fusion model.

[0097] In some embodiments, the specific implementation of steps 301, 3021, 3023-3026 and the technical effects thereof can be referred to in Figure 2 Steps 201, 2021, 2023-2026 in the corresponding embodiments are not repeated here.

[0098] from Figure 3 It can be seen that Figure 2 Compared with the description of some corresponding embodiments, Figure 3 In the corresponding process 300 of the model training method in some embodiments, the common feature information of the modalities is determined in the form of semantic feature similarity, so as to obtain more accurate at least one common feature information of the modalities and at least one modality-specific feature information, thereby making the subsequent training of the initial multimodal feature fusion model more accurate and efficient.

[0099] Continue to refer Figure 4 , shows a process 400 of some embodiments of the multimodal feature information generation method according to the present disclosure. The multimodal feature information generation method comprises the following steps:

[0100] Step 401, obtaining second target graphic data.

[0101] In some embodiments, the execution subject of the above multimodal feature information generation method (for example Figure 1 The electronic device 101) shown can obtain the second target graphic data in a wired manner or a wireless manner.

[0102] Step 402: input the second target image-text data into a pre-trained multimodal feature fusion model to generate multimodal feature information for the second target image-text data.

[0103] In some embodiments, the execution subject may input the second target image and text data into a pre-trained multimodal feature fusion model to generate multimodal feature information for the second target image and text data. The multimodal feature fusion model is generated based on the model training method disclosed in the present invention. The multimodal feature information may characterize the overall feature information of the second target image and text data.

[0104] Optionally, the second target graphic data may be data that is different from the first target graphic data and includes image modality and text modality.

[0105] In some optional implementations of some embodiments, after step 402, the steps further include:

[0106] The execution subject may input the multimodal feature information for the second target graphic data into a pre-trained item attribute recognition model to generate an item attribute information set for the second target graphic data. The item attribute recognition model may be a model for generating item attribute information. In practice, the item attribute recognition model may be a fully connected network with multiple layers connected in series. The item attribute information set may be an attribute information set for the target item involved in the second target graphic data.

[0107] The above-mentioned various embodiments of the present disclosure have the following beneficial effects: through the multimodal feature information generation method of some embodiments of the present disclosure, multimodal feature information for the above-mentioned second target graphic data can be accurately generated.

[0108] Further references Figure 5 As an implementation of the methods shown in the above figures, the present disclosure provides some embodiments of a model training device, and these device embodiments are Figure 2 Corresponding to the method embodiments shown, the model training device can be specifically applied to various electronic devices.

[0109] like Figure 5As shown, a model training device 500 includes: a first acquisition unit 501 and a training unit 502. The first acquisition unit 501 is configured to acquire a picture and text data set and an item attribute information set corresponding to each picture and text data; the training unit 502 is configured to select picture and text data from the above picture and text data set as the first target picture and text data, and perform the following training steps: input the image data included in the above first target picture and text data to the initial image feature extraction model included in the initial multimodal feature fusion model to generate at least one image block feature information, and input the text data included in the above first target picture and text data to the initial text feature extraction model included in the initial multimodal feature fusion model to generate at least one text character feature information; according to the above at least one image block feature information and the above-mentioned at least one text character feature information, generate at least one modal common feature information and at least one modal specific feature information; generate multimodal feature information according to the feature fusion model included in the above-mentioned at least one modal common feature information, the above-mentioned at least one modal specific feature information and the above-mentioned initial multimodal feature fusion model; generate attribute difference loss information for the above-mentioned multimodal feature information according to the item attribute information set corresponding to the above-mentioned first target graphic data; determine whether to train the above-mentioned initial multimodal feature fusion model according to the above-mentioned attribute difference loss information; in response to determining not to train, determine the above-mentioned initial multimodal feature fusion model as the multimodal feature fusion model.

[0110] In some optional implementations of some embodiments, the device 500 further includes: an updating unit (not shown in the figure). The updating unit may be configured to: in response to determining the training, update the initial multimodal feature fusion model according to the attribute difference loss information, obtain an updated multimodal feature fusion model as the initial multimodal feature fusion model, and reselect graphic data from the graphic data set as the first target graphic data to continue to perform the training step.

[0111] In some optional implementations of some embodiments, the training unit 502 can be configured to: for each image block feature information of the at least one image block feature information mentioned above, perform the following generation steps: determine the text character feature information corresponding to the image block feature information as the target text character feature information; determine the semantic feature similarity between the image block feature information and the target text character feature information; in response to determining that the semantic feature similarity is greater than a predetermined value, generate modal common feature information based on the image block feature information and the target text character feature information.

[0112] In some optional implementations of some embodiments, the training unit 502 may be configured to: in response to determining that the semantic feature similarity is less than or equal to the predetermined value, respectively determine the image block feature information and the target text character feature information as modality-specific feature information.

[0113] In some optional implementations of some embodiments, the training unit 502 can be configured to: splice feature information of the at least one modality common feature information and the at least one modality-specific feature information to obtain spliced ​​feature information; input the spliced ​​feature information into the feature fusion model to generate the multimodal feature information.

[0114] It is understandable that the units recorded in the model training device 500 are similar to the reference Figure 2 Therefore, the operations, features and beneficial effects described above for the method are also applicable to the model training device 500 and the units contained therein, and will not be repeated here.

[0115] Further references Figure 6 As an implementation of the methods shown in the above figures, the present disclosure provides some embodiments of a multimodal feature information generating device. These device embodiments are similar to Figure 4 Corresponding to the method embodiments shown, the multimodal feature information generating device can be specifically applied to various electronic devices.

[0116] like Figure 6 As shown, a multimodal feature information generating device 600 includes: a second acquisition unit 601 and a generation unit 602. The second acquisition unit 601 is configured to acquire second target image and text data; the generation unit 602 is configured to input the second target image and text data into a pre-trained multimodal feature fusion model to generate multimodal feature information for the second target image and text data, wherein the multimodal feature fusion model is generated based on the model training method disclosed in the present invention.

[0117] It can be understood that the units recorded in the multimodal feature information generating device 600 are similar to those in the reference Figure 4 Therefore, the operations, features and beneficial effects described above for the method are also applicable to the multimodal feature information generating device 600 and the units included therein, and will not be described in detail here.

[0118] Reference below Figure 7 , which shows an electronic device (eg, Figure 1 Schematic diagram of the structure of the electronic device 101)700. Figure 7The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0119] like Figure 7 As shown, the electronic device 700 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory 702 or a program loaded from a storage device 708 into a random access memory 703. Various programs and data required for the operation of the electronic device 700 are also stored in the random access memory 703. The processing device 701, the read-only memory 702, and the random access memory 703 are connected to each other via a bus 704. An input / output interface 705 is also connected to the bus 704.

[0120] Typically, the following devices may be connected to the input / output interface 705: an input device 706 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 707 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 708 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 709. The communication device 709 may allow the electronic device 700 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 7 The electronic device 700 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 7 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0121] In particular, according to some embodiments of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from a network through a communication device 709, or installed from a storage device 708, or installed from a read-only memory 702. When the computer program is executed by the processing device 701, the above-mentioned functions defined in the method of some embodiments of the present disclosure are executed.

[0122] It should be noted that the computer-readable medium in some embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present disclosure, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In some embodiments of the present disclosure, the computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0123] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0124] The above-mentioned computer-readable medium may be included in the above-mentioned electronic device; or it may exist independently without being assembled into the electronic device. The above-mentioned computer-readable medium carries one or more programs. When the above-mentioned one or more programs are executed by the electronic device, the electronic device: obtains a graphic data set and an item attribute information set corresponding to each graphic data; selects graphic data from the above-mentioned graphic data set as the first target graphic data, and performs the following training steps: inputs the image data included in the above-mentioned first target graphic data into the initial image feature extraction model included in the initial multimodal feature fusion model to generate at least one image block feature information, and inputs the text data included in the above-mentioned first target graphic data into the initial text feature extraction model included in the above-mentioned initial multimodal feature fusion model to generate at least one text character feature information; based on According to the at least one image block feature information and the at least one text character feature information, at least one modality common feature information and at least one modality specific feature information are generated; according to the at least one modality common feature information, the at least one modality specific feature information and the feature fusion model included in the initial multimodal feature fusion model, multimodal feature information is generated; according to the item attribute information set corresponding to the first target image and text data, attribute difference loss information for the multimodal feature information is generated; according to the attribute difference loss information, whether to train the initial multimodal feature fusion model is determined; in response to determining not to train, the initial multimodal feature fusion model is determined as a multimodal feature fusion model. The second target image and text data is obtained; the second target image and text data is input into the pre-trained multimodal feature fusion model to generate multimodal feature information for the second target image and text data, wherein the multimodal feature fusion model is generated based on the model training method disclosed in the present invention.

[0125] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0126] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0127] The units described in some embodiments of the present disclosure may be implemented by software or hardware. The described units may also be provided in a processor, for example, may be described as: a processor including a first acquisition unit and a training unit. The names of these units do not constitute limitations on the units themselves in certain circumstances, for example, the first acquisition unit may also be described as "a unit for acquiring a graphic data set and an item attribute information set corresponding to each graphic data set".

[0128] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0129] Some embodiments of the present disclosure also provide a computer program product, including a computer program, which implements any of the above-mentioned model training methods or multimodal feature information generation methods when executed by a processor.

[0130] The above descriptions are only some preferred embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with the technical features with similar functions disclosed in the embodiments of the present disclosure (but not limited to) and the technical solutions formed.

Claims

1. A model training method, comprising: Obtain a data set of images and texts and a set of item attribute information corresponding to each image and text data; Selecting image and text data from the image and text data set as the first target image and text data, and performing the following training steps: Inputting the image data included in the first target image-text data into an initial image feature extraction model included in an initial multimodal feature fusion model to generate at least one image block feature information, and inputting the text data included in the first target image-text data into an initial text feature extraction model included in the initial multimodal feature fusion model to generate at least one text character feature information; generating at least one modality common feature information and at least one modality specific feature information according to the at least one image block feature information and the at least one text character feature information; Generate multimodal feature information according to the at least one modality common feature information, the at least one modality specific feature information and a feature fusion model included in the initial multimodal feature fusion model; generating attribute difference loss information for the multimodal feature information according to the item attribute information set corresponding to the first target graphic data; Determining whether to train the initial multimodal feature fusion model according to the attribute difference loss information; In response to determining not to train, the initial multimodal feature fusion model is determined as a multimodal feature fusion model.

2. The method according to claim 1, wherein: The method further comprises: In response to determining the training, the initial multimodal feature fusion model is updated according to the attribute difference loss information to obtain an updated multimodal feature fusion model as the initial multimodal feature fusion model, and graphic data is reselected from the graphic data set as the first target graphic data to continue executing the training step.

3. The method according to claim 1, wherein: The step of generating at least one modality common feature information and at least one modality specific feature information according to the at least one image block feature information and the at least one text character feature information comprises: For each piece of image block feature information in the at least one piece of image block feature information, the following generating steps are performed: Determining text character feature information corresponding to the image block feature information as target text character feature information; Determining the semantic feature similarity between the image block feature information and the target text character feature information; In response to determining that the semantic feature similarity is greater than a predetermined value, modality common feature information is generated according to the image block feature information and the target text character feature information.

4. The method according to claim 3, wherein: After generating modality common feature information according to the image block feature information and the target text character feature information in response to determining that the semantic feature similarity is greater than a predetermined value, the method further includes: In response to determining that the semantic feature similarity is less than or equal to the predetermined value, the image block feature information and the target text character feature information are respectively determined as modality-specific feature information.

5. The method according to claim 1, wherein: The generating multimodal feature information according to the at least one modality common feature information, the at least one modality specific feature information and the feature fusion model included in the initial multimodal feature fusion model comprises: Performing feature information splicing on the at least one modality common feature information and the at least one modality specific feature information to obtain spliced ​​feature information; The spliced ​​feature information is input into the feature fusion model to generate the multimodal feature information.

6. The method according to claim 1, wherein: The initial image feature extraction model is initialized based on a pre-trained image semantic feature extraction model, the initial text feature extraction model and the feature fusion model are initialized based on a pre-trained language model, and the language model and the image semantic feature extraction model are pre-semantically aligned models.

7. A method for generating multimodal feature information, comprising: Acquire second target graphic data; The second target image-text data is input into a pre-trained multimodal feature fusion model to generate multimodal feature information for the second target image-text data, wherein the multimodal feature fusion model is generated based on the method described in one of claims 1-6.

8. The method according to claim 7, wherein: The method further comprises: The multimodal feature information for the second target graphic data is input into a pre-trained item attribute recognition model to generate an item attribute information set for the second target graphic data.

9. A model training device, comprising: A first acquisition unit is configured to acquire a set of image and text data and an item attribute information set corresponding to each image and text data; A training unit is configured to select graphic data from the graphic data set as first target graphic data, and perform the following training steps: input the image data included in the first target graphic data to the initial image feature extraction model included in the initial multimodal feature fusion model to generate at least one image block feature information, and input the text data included in the first target graphic data to the initial text feature extraction model included in the initial multimodal feature fusion model to generate at least one text character feature information; generate at least one modality common feature information and at least one modality specific feature information according to the at least one image block feature information and the at least one text character feature information; generate multimodal feature information according to the at least one modality common feature information, the at least one modality specific feature information and the feature fusion model included in the initial multimodal feature fusion model; generate attribute difference loss information for the multimodal feature information according to the item attribute information set corresponding to the first target graphic data; Determining whether to train the initial multimodal feature fusion model according to the attribute difference loss information; In response to determining not to train, the initial multimodal feature fusion model is determined as a multimodal feature fusion model.

10. A multimodal feature information generating device, comprising: A second acquisition unit is configured to acquire second target graphic data; A generating unit is configured to input the second target image and text data into a pre-trained multimodal feature fusion model to generate multimodal feature information for the second target image and text data, wherein the multimodal feature fusion model is generated based on the method described in one of claims 1-6.

11. An electronic device, comprising: one or more processors; a storage device having one or more programs stored thereon, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 8.

12. A computer readable medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

13. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 8.

Images