Model training method and device, information generation method and device, equipment and medium

By adding the model structure of the modal information to be added to the multimodal data feature extraction model and training it, a multimodal feature extraction model that supports modal union information is generated, and the poor modal expansion ability in the existing technology is solved. The problem has improved the application scalability and efficiency of the model.

CN119004091APending Publication Date: 2024-11-22BEIJING WODONG TIANJUN INFORMATION TECH CO LTD +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310552706.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-16
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

When the existing multimodal data feature extraction model is modal expansion, it needs to be reconstructed and retrained, resulting in high training costs, long application cycles and poor expansion.

Method used

By obtaining the pre-trained data feature extraction model and the data modal information to be added, combining the training data set, adding the model structure of the data modal information to be added, generating a model that supports multimodal feature extraction, and using the training data set for model training , generate a multimodal feature extraction model, supporting the input of modal union information.

Benefits of technology

It improves the scalability of the model input mode, reduces the training cost and cycle during modal expansion, and enables the multimodal feature extraction model to be applied to more scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119004091A_ABST
    Figure CN119004091A_ABST
Patent Text Reader

Abstract

The embodiment of the invention discloses a model training method and device, an information generation method and device, equipment and a medium. A specific embodiment of the method comprises the steps of obtaining a pre-trained data feature extraction model, to-be-added data modal information and a training data set for the to-be-added data modal information; adding a model structure corresponding to the to-be-added data modal information to the data feature extraction model to obtain an added feature extraction model; according to the training data set, model training is conducted on the feature extraction model after adding, a multi-modal feature extraction model is obtained, input data modal information corresponding to the multi-modal feature extraction model is modal union set information, and the modal union set information is union set information of the target input data modal information and the data modal information to be added. The implementation mode is related to artificial intelligence, the multi-modal feature extraction model supporting data information input of the expansion mode can be generated, and the expansibility of the model input mode is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of computer technology, and in particular to a model training method, an information generation method, an apparatus, a device, and a medium. Background Art

[0002] With the rapid development of Internet content platforms, Internet content has shown increasingly rich forms of expression, and the corresponding data modalities are also increasing. For example, data modalities may include: title modality, text modality, image modality, video modality, audio modality, etc. For the extraction of multimodal data features, the commonly used method is to use a dual-stream model to achieve feature extraction of multimodal data.

[0003] However, the inventors have found that when the above method is used to extract features, the following technical problems often occur:

[0004] The model has a fixed input mode and poor scalability. If the mode is expanded, the model needs to be rebuilt and retrained, resulting in high training costs and a long application cycle.

[0005] The above information disclosed in this Background section is only for enhancement of understanding of the background of the inventive concept and therefore it may contain information that does not form the prior art that is already known in this country to a person of ordinary skill in the art. Summary of the invention

[0006] The content of this disclosure is used to introduce concepts in a brief form, which will be described in detail in the detailed implementation section below. The content of this disclosure is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to limit the scope of the technical solution claimed for protection.

[0007] Some embodiments of the present disclosure propose model training methods, devices, equipment and media to solve the technical problems mentioned in the above background technology section.

[0008] In a first aspect, some embodiments of the present disclosure provide a model training method, comprising: obtaining a pre-trained data feature extraction model, data modal information to be added, and a training data set for the data modal information to be added; adding a model structure corresponding to the data modal information to be added to the data feature extraction model to obtain a post-addition feature extraction model; performing model training on the post-addition feature extraction model according to the training data set to obtain a multimodal feature extraction model, wherein the input data modal information corresponding to the multimodal feature extraction model is modal union information, the modal union information is the union information of the target input data modal information and the data modal information to be added, and the target input data modal information is the modal information of the input data corresponding to the data feature extraction model.

[0009] Optionally, the above-mentioned post-addition feature extraction model is trained according to the above-mentioned training data set to obtain a multimodal feature extraction model, including: determining a first initial task execution model for training the above-mentioned post-addition feature extraction model; for each training data in the above-mentioned training data set, performing the following training steps: inputting the above-mentioned training data into the above-mentioned post-addition feature extraction model to output first feature information; inputting the above-mentioned first feature information into the above-mentioned first initial task execution model to output a task execution result; determining loss information for the above-mentioned task execution result and the target data label, wherein the above-mentioned target data label is a label associated with the above-mentioned training data; and updating model parameters of the model structure corresponding to the data modality information to be added in the above-mentioned post-addition feature extraction model according to the above-mentioned loss information.

[0010] Optionally, the above-mentioned training data includes: a training sub-data set for the above-mentioned data modal information to be added and the above-mentioned target input data modal information; and the above-mentioned input of the above-mentioned training data into the above-mentioned post-addition feature extraction model to output the first feature information, including: for each training sub-data in the above-mentioned training sub-data set, executing the following data processing steps: performing data encoding on the above-mentioned training sub-data to obtain an encoding vector; performing vector normalization on the above-mentioned encoding vector to obtain a standardized vector; performing modal feature fusion on the obtained standardized vector set to generate a modal fusion feature vector for the above-mentioned training sub-data set as the above-mentioned first feature information.

[0011] Optionally, the above-mentioned encoding vector is vector normalized to obtain a standardized vector, including: generating a key vector, a value vector and a query vector for the above-mentioned encoding vector; inputting the key vector, the value vector and the query vector into a cross-attention model to generate a standardized vector.

[0012] Optionally, the above-mentioned feature extraction model after adding includes: at least one data coding model, and the data coding model in at least one data coding model has corresponding data modal sub-information; and the above-mentioned data encoding of the above-mentioned training sub-data to obtain the encoding vector includes: determining the data modal sub-information corresponding to the above-mentioned training sub-data as the target data modal sub-information; using the data coding model in the above-mentioned at least one data coding model corresponding to the above-mentioned target data modal sub-information, data encoding the above-mentioned training sub-data to obtain the encoding vector.

[0013] Optionally, the data feature extraction model is trained through the following steps: determining a second initial task execution model trained for the initial data feature extraction model; obtaining sample data for the target input data modal information, wherein the sample data includes: target data and target label, the target data includes: a data subset, each data in the data subset has corresponding data modal sub-information in the target input data modal information; selecting at least one data from the data subset; training the initial data feature extraction model and the second initial task execution model according to the at least one data and the target label to obtain a data feature extraction model and a second task execution model.

[0014] In a second aspect, some embodiments of the present disclosure provide a model training device, comprising: a first acquisition unit, configured to acquire a pre-trained data feature extraction model, data modal information to be added, and a training data set for the data modal information to be added; an adding unit, configured to add the model structure corresponding to the data modal information to be added to the data feature extraction model to obtain a post-addition feature extraction model; a training unit, configured to perform model training on the post-addition feature extraction model according to the training data set to obtain a multimodal feature extraction model, wherein the input data modal information corresponding to the multimodal feature extraction model is modal union information, the modal union information is the union information of the target input data modal information and the data modal information to be added, and the target input data modal information is the modal information of the input data corresponding to the data feature extraction model.

[0015] Optionally, the training unit can be configured to: determine a first initial task execution model for training the above-mentioned post-addition feature extraction model; for each training data in the above-mentioned training data set, perform the following training steps: input the above-mentioned training data into the above-mentioned post-addition feature extraction model to output first feature information; input the above-mentioned first feature information into the above-mentioned first initial task execution model to output a task execution result; determine loss information for the above-mentioned task execution result and the target data label, wherein the above-mentioned target data label is a label associated with the above-mentioned training data; and according to the above-mentioned loss information, update the model parameters of the model structure corresponding to the data modal information to be added in the above-mentioned post-addition feature extraction model.

[0016] Optionally, the above-mentioned training data includes: a training sub-data set for the above-mentioned data modal information to be added and the above-mentioned target input data modal information; and the training unit can be configured to: for each training sub-data in the above-mentioned training sub-data set, perform the following data processing steps: perform data encoding on the above-mentioned training sub-data to obtain a coding vector; perform vector normalization on the above-mentioned coding vector to obtain a standardized vector; perform modal feature fusion on the obtained standardized vector set to generate a modal fusion feature vector for the above-mentioned training sub-data set as the above-mentioned first feature information.

[0017] Optionally, the training unit may be configured to: generate a key vector, a value vector and a query vector for the encoding vector; and input the key vector, the value vector and the query vector into a cross-attention model to generate a standardized vector.

[0018] Optionally, the above-mentioned feature extraction model after adding includes: at least one data coding model, and the data coding model in at least one data coding model has corresponding data modal sub-information; and the training unit can be configured to: determine the data modal sub-information corresponding to the above-mentioned training sub-data as the target data modal sub-information; use the data coding model in the above-mentioned at least one data coding model corresponding to the above-mentioned target data modal sub-information to data encode the above-mentioned training sub-data to obtain a coding vector.

[0019] Optionally, the data feature extraction model is trained through the following steps: determining a second initial task execution model trained for the initial data feature extraction model; obtaining sample data for the target input data modal information, wherein the sample data includes: target data and target label, the target data includes: a data subset, each data in the data subset has corresponding data modal sub-information in the target input data modal information; selecting at least one data from the data subset; training the initial data feature extraction model and the second initial task execution model according to the at least one data and the target label to obtain a data feature extraction model and a second task execution model.

[0020] In a third aspect, some embodiments of the present disclosure provide an information generation method, comprising: obtaining data to be feature extracted, wherein the data modal information to be added includes at least one first data modal sub-information, and the target input data modal information includes at least one second data modal sub-information; the data modal information corresponding to the data to be feature extracted is at least one of the at least one first data modal sub-information and the at least one second data modal sub-information; using a multimodal feature extraction model, generating second feature information for the data to be feature extracted, wherein the multimodal feature extraction model is generated based on the model training method of the first aspect.

[0021] In a fourth aspect, some embodiments of the present disclosure provide an information generating device, comprising: a second acquiring unit, configured to acquire data to be feature extracted, wherein the data modal information to be added includes at least one first data modal sub-information, and the target input data modal information includes at least one second data modal sub-information; the data modal information corresponding to the data to be feature extracted is at least one of the at least one first data modal sub-information and the at least one second data modal sub-information; a generating unit, configured to generate second feature information for the data to be feature extracted using a multimodal feature extraction model, wherein the multimodal feature extraction model is generated based on the model training method of the first aspect.

[0022] In a fifth aspect, some embodiments of the present disclosure provide an electronic device comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any one of the implementation methods in the first and third aspects.

[0023] In a sixth aspect, some embodiments of the present disclosure provide a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the method described in any one of the implementation modes of the first aspect and the third aspect is implemented.

[0024] In a seventh aspect, some embodiments of the present disclosure provide a computer program product, including a computer program, which, when executed by a processor, implements the method described in any one of the implementation modes of the first and third aspects above.

[0025] The above-mentioned embodiments of the present disclosure have the following beneficial effects: through the model training method of some embodiments of the present disclosure, a multimodal feature extraction model that supports the input of data information of the extended modality can be generated, thereby improving the extensibility of the model input modality. Specifically, the reason for the poor extensibility of the relevant model input modality is that the input modality of the model is fixed and the extensibility is poor. If the modality is expanded, the model needs to be rebuilt and the reconstructed model is retrained, resulting in high training costs and a long application cycle. Based on this, the model training method of some embodiments of the present disclosure first obtains a pre-trained data feature extraction model, the modal information of the data to be added, and a training data set for the above-mentioned modal information of the data to be added. Here, the obtained data feature extraction model is used as the basic model structure for the generation of the subsequent multimodal feature extraction model. Here, with the data feature extraction model as the basic model structure, the multimodal feature extraction model obtained later can support the input of modal data corresponding to the target input data modal information. By using the acquired modal information of the data to be added as the extended modality, the modal extensibility of the model is improved. The obtained training data set is used for the training of the feature extraction model after addition. Next, the model structure corresponding to the above-mentioned data modal information to be added is added to the above-mentioned data feature extraction model to obtain the feature extraction model after addition. Here, through the corresponding addition of the model structure, a model that supports the input of modal data corresponding to the data modal information to be added is generated without large-scale reconstruction of the model structure of the data feature extraction model, that is, the feature extraction model after addition. Finally, according to the above-mentioned training data set, the above-mentioned feature extraction model after addition is trained to obtain a multimodal feature extraction model. Among them, the input data modal information corresponding to the above-mentioned multimodal feature extraction model is modal union information. The above-mentioned modal union information is the union information of the target input data modal information and the above-mentioned data modal information to be added. The above-mentioned target input data modal information is the modal information of the input data corresponding to the above-mentioned data feature extraction model. Here, the multimodal feature extraction model is a pre-trained model that supports the input of modal data corresponding to the data modal information to be added. In summary, without large-scale adjustment of the model structure and high training cost, by determining the modal information of the data to be added, acquiring the training data set and making targeted model structure adjustments based on the data feature extraction model, the multimodal feature extraction model can not only support the input of modal data corresponding to the modal information of the target input data, but also support the input of modal data corresponding to the modal information of the data to be added, which expands the modal input form of feature extraction and improves the scalability of the model input modality, so that the multimodal feature extraction model can be applied to more application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0027] Figure 1 is a schematic diagram of an application scenario of a model training method according to some embodiments of the present disclosure;

[0028] Figure 2 is a flowchart of some embodiments of the model training method according to the present disclosure;

[0029] Figure 3 is a flowchart of other embodiments of the model training method according to the present disclosure;

[0030] Figure 4 are flow charts of some embodiments of the information generation method according to the present disclosure;

[0031] Figure 5 is a schematic diagram of the structure of some embodiments of the model training device according to the present disclosure;

[0032] Figure 6 is a schematic diagram of the structure of some embodiments of the information generating device according to the present disclosure;

[0033] Figure 7 It is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0034] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0035] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure can be combined with each other.

[0036] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0037] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0038] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0039] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0040] Figure 1 It is a schematic diagram of an application scenario of the model training method according to some embodiments of the present disclosure.

[0041] exist Figure 1 In the application scenario, first, the electronic device 101 can obtain a pre-trained data feature extraction model 102, the data modal information to be added 103, and a training data set 104 for the data modal information to be added 103. In this application scenario, the data modal information to be added 103 can be "audio modality". Then, the electronic device 101 can add the model structure 105 corresponding to the data modal information to be added 103 to the data feature extraction model 102 to obtain the added feature extraction model 106. Finally, the electronic device 101 can perform model training on the added feature extraction model 106 according to the training data set 104 to obtain a multimodal feature extraction model 107. Among them, the input data modal information corresponding to the multimodal feature extraction model 107 is the modal union information 109. The modal union information 109 is the union information of the target input data modal information 108 and the data modal information to be added 103. The target input data modal information 108 is the modal information of the input data corresponding to the data feature extraction model 102. In this application scenario, the modality union information 109 may be: "text modality", "title modality", "image modality", "audio modality". The target input data modality information 108 may be: "text modality", "title modality", "image modality".

[0042] It should be noted that the electronic device 101 can be hardware or software. When the electronic device is hardware, it can be implemented as a distributed cluster consisting of multiple servers or terminal devices, or it can be implemented as a single server or a single terminal device. When the electronic device is embodied as software, it can be installed in the hardware devices listed above. It can be implemented as multiple software or software modules for providing distributed services, for example, or it can be implemented as a single software or software module. No specific limitation is made here.

[0043] It should be understood that Figure 1The number of electronic devices in the embodiment is only for illustration. Any number of electronic devices may be provided according to implementation requirements.

[0044] Continue to refer Figure 2 , shows a process 200 of some embodiments of the model training method according to the present disclosure. The model training method comprises the following steps:

[0045] Step 201: obtain a pre-trained data feature extraction model, data modal information to be added, and a training data set for the data modal information to be added.

[0046] In some embodiments, the execution entity of the above model training method (for example Figure 1 The electronic device 101 shown) can obtain a pre-trained data feature extraction model, data modality information to be added, and a training data set for the above-mentioned data modality information to be added through a wired connection or a wireless connection. The data feature extraction model can be a network model for extracting feature information from data. The data feature extraction model can support the input of multimodal data. Multimodal data can be data in multimodal form. In practice, the modality of the data can include but is not limited to at least one of the following: image modality, text modality, title modality, and video modality. The image modality corresponds to the image. The text modality corresponds to the text content. The title modality corresponds to the title content. The video modality corresponds to the video. For example, the multimodal data includes: image, text content, and title content. The data feature extraction model can be a multi-layer convolutional neural network (CNN) + multi-layer recurrent neural network (RNN) that supports multimodal data input. The above-mentioned target input data modality information is the modality information of the input data corresponding to the above-mentioned data feature extraction model. The target input data modality information includes at least one second data modality sub-information. The data modal information to be added may be the modal information of at least one data modal sub-information to be supported and expanded by the data feature extraction model. The data modal information to be added includes at least one first data modal sub-information. Different data modal sub-information is determined to exist in at least one first data modal sub-information and at least one second data modal sub-information, and the same data modal sub-information may exist. For example, at least one second data modal sub-information may include: audio modal information, image modal information, and title modal information. At least one first data modal sub-information may include: image modal information, title modal information, and text content information. The training data in the training data set may include: at least one modal data corresponding to at least one first data modal sub-information. That is, the training data is multimodal data.

[0047] In some optional implementations of some embodiments, the data feature extraction model is trained by the following steps:

[0048] The first step is to determine the second initial task execution model for training the initial data feature extraction model. The second initial task execution model may be a downstream task execution model whose model has not yet been trained. For example, the downstream task execution model may be a target classification model. The downstream task execution model may also be a target segmentation model. The corresponding network structures may be different for different execution tasks corresponding to the downstream task execution model. For example, the downstream task execution model is a target segmentation model, and the corresponding network structure is a Mask R-CNN network structure. The second initial task execution model may be used for model training of subsequent initial data feature extraction models. The initial data feature extraction model may be a data feature extraction model whose model has not yet been trained.

[0049] The second step is to obtain sample data for the above target input data modality information. The above sample data includes: target data and target label. The above target data includes: data subset, each data in the above data subset has corresponding data modality sub-information in the target input data modality information.

[0050] For example, the target input data modality information corresponds to at least one second data modality sub-information which may be audio modality information, image modality information, or title modality information. The corresponding target data may be audio, image, or title content. The target tag may be an identification tag. For example, the target tag may be "chair".

[0051] The third step is to select at least one data from the above data subset.

[0052] As an example, the execution entity may randomly select at least one data from the data subset.

[0053] In the fourth step, the initial data feature extraction model and the second initial task execution model are trained according to the at least one data and the target label to obtain a data feature extraction model and a second task execution model.

[0054] As an example, first, the execution subject may input at least one data into the initial data feature extraction model to obtain initial feature information. Then, the initial feature information is input into the second initial task execution model to obtain the initial task execution result. Next, the loss information between the initial task execution result and the target label is generated. Finally, according to the loss information, the initial data feature extraction model and the second initial task execution model are trained to obtain the data feature extraction model and the second task execution model.

[0055] Step 202, adding the model structure corresponding to the above-mentioned data modal information to be added to the above-mentioned data feature extraction model to obtain a feature extraction model after addition.

[0056] In some embodiments, the execution entity may add the model structure corresponding to the data modal information to be added to the data feature extraction model to obtain a feature extraction model after addition.

[0057] The data modality information to be added includes: at least one first data modality sub-information. Each first data modality sub-information may have a corresponding sub-model structure. Each second data modality sub-information may have a corresponding sub-model structure.

[0058] As an example, first, the execution subject may determine modal sub-information having the same modal information as at least one first data modal sub-information and at least one second data modal sub-information as target modal sub-information, and obtain at least one target modal sub-information. Then, at least one sub-model structure corresponding to at least one target modal sub-information is combined to obtain a model structure. Finally, the model structure is added to the data feature extraction model to obtain a feature extraction model after addition.

[0059] Step 203: Based on the training data set, the added feature extraction model is trained to obtain a multimodal feature extraction model.

[0060] In some embodiments, the execution subject may perform model training on the feature extraction model after addition according to the training data set to obtain a multimodal feature extraction model. The input data modal information corresponding to the multimodal feature extraction model is modal union information. The modal union information is the union information of the target input data modal information and the modal information of the data to be added. The target input data modal information is the modal information of the input data corresponding to the data feature extraction model. The multimodal feature extraction model supports modal data input corresponding to at least one of the multiple data modal sub-information corresponding to the modal union information. The multimodal feature extraction model may be a model after the model training is completed. The input data modal information corresponding to the multimodal feature extraction model is a data modal sub-information set after the union of at least one first data modal sub-information and at least one second data modal sub-information.

[0061] In some optional implementations of some embodiments, the above-mentioned model training of the added feature extraction model according to the above-mentioned training data set to obtain the multimodal feature extraction model may include the following steps:

[0062] The first step is to determine the first initial task execution model trained for the above-mentioned added feature extraction model.

[0063] In the second step, for each training data in the above training data set, perform the following training steps:

[0064] Sub-step 1: input the training data into the added feature extraction model to output first feature information, wherein the first feature information may represent feature information of the training data.

[0065] Sub-step 2: inputting the first feature information into the first initial task execution model to output the task execution result.

[0066] Sub-step 3, determining the loss information for the task execution result and the target data label. The target data label is a label associated with the training data. The target data label can be a label of the actual task execution result corresponding to the training data.

[0067] As an example, the above execution entity can input the task execution result and the target data label into the cross entropy loss function to output loss information.

[0068] Sub-step 4, based on the above loss information, update the model parameters of the model structure corresponding to the data modal information to be added in the above added feature extraction model.

[0069] It should be noted that the parameters corresponding to the data feature extraction model in the feature extraction model after the addition will not be updated. That is, only the parameters corresponding to the model structure corresponding to the data modal information to be added are updated.

[0070] As an example, in response to determining that the loss information is less than a predetermined loss value, the model parameters of the model structure corresponding to the data modal information to be added in the above-mentioned feature extraction model after addition are not updated. In response to determining that the loss information is greater than or equal to the predetermined loss value, the model parameters of the model structure corresponding to the data modal information to be added in the above-mentioned feature extraction model after addition are updated.

[0071] The above-mentioned embodiments of the present disclosure have the following beneficial effects: through the model training method of some embodiments of the present disclosure, a multimodal feature extraction model that supports the input of data information of the extended modality can be generated, thereby improving the extensibility of the model input modality. Specifically, the reason for the poor extensibility of the relevant model input modality is that the input modality of the model is fixed and the extensibility is poor. If the modality is expanded, the model needs to be rebuilt and the reconstructed model is retrained, resulting in high training costs and a long application cycle. Based on this, the model training method of some embodiments of the present disclosure first obtains a pre-trained data feature extraction model, the modal information of the data to be added, and a training data set for the above-mentioned modal information of the data to be added. Here, the obtained data feature extraction model is used as the basic model structure for the generation of the subsequent multimodal feature extraction model. Here, with the data feature extraction model as the basic model structure, the multimodal feature extraction model obtained later can support the input of modal data corresponding to the target input data modal information. By using the acquired modal information of the data to be added as the extended modality, the modal extensibility of the model is improved. The obtained training data set is used for the training of the feature extraction model after addition. Next, the model structure corresponding to the above-mentioned data modal information to be added is added to the above-mentioned data feature extraction model to obtain the feature extraction model after addition. Here, through the corresponding addition of the model structure, a model that supports the input of modal data corresponding to the data modal information to be added is generated without large-scale reconstruction of the model structure of the data feature extraction model, that is, the feature extraction model after addition. Finally, according to the above-mentioned training data set, the above-mentioned feature extraction model after addition is trained to obtain a multimodal feature extraction model. Among them, the input data modal information corresponding to the above-mentioned multimodal feature extraction model is modal union information. The above-mentioned modal union information is the union information of the target input data modal information and the above-mentioned data modal information to be added. The above-mentioned target input data modal information is the modal information of the input data corresponding to the above-mentioned data feature extraction model. Here, the multimodal feature extraction model is a pre-trained model that supports the input of modal data corresponding to the data modal information to be added. In summary, without large-scale adjustment of the model structure and high training cost, by determining the modal information of the data to be added, acquiring the training data set and making targeted model structure adjustments based on the data feature extraction model, the multimodal feature extraction model can not only support the input of modal data corresponding to the modal information of the target input data, but also support the input of modal data corresponding to the modal information of the data to be added, which expands the modal input form of feature extraction and improves the scalability of the model input modality, so that the multimodal feature extraction model can be applied to more application scenarios.

[0072] Further references Figure 3 , shows a process 300 of other embodiments of the model training method according to the present disclosure. The model training method comprises the following steps:

[0073] Step 301, obtaining a pre-trained data feature extraction model, data modal information to be added, and a training data set for the data modal information to be added.

[0074] Step 302: Add the model structure corresponding to the data modal information to be added to the data feature extraction model to obtain a feature extraction model after addition.

[0075] Step 303: determine a first initial task execution model trained for the above-added feature extraction model.

[0076] In some embodiments, the execution subject (for example, the electronic device 101) may determine the first initial task execution model trained for the above-mentioned added feature extraction model. Among them, the first initial task execution model may be a downstream task execution model whose model has not yet been trained. For example, the downstream task execution model may be a target classification model. The downstream task execution model may also be a target segmentation model. The corresponding network structure is different for different execution tasks corresponding to the downstream task execution model. For example, the downstream task execution model is a target segmentation model, and the corresponding network structure is a Mask R-CNN network structure.

[0077] Step 304: for each training data in the training data set, perform the following training steps:

[0078] Step 3041, input the above training data into the above added feature extraction model to output the first feature information.

[0079] In some embodiments, the execution entity may input the training data into the post-addition feature extraction model to output first feature information, wherein the first feature information may represent feature information of the training data.

[0080] Step 3042: input the first feature information into the first initial task execution model to output the task execution result.

[0081] In some embodiments, the execution entity may input the first feature information into the first initial task execution model to output a task execution result.

[0082] Step 3043, determining the loss information of the above task execution results and the target data label.

[0083] In some embodiments, the execution subject may determine loss information for the task execution result and the target data label, wherein the target data label is a label associated with the training data.

[0084] As an example, the above execution entity can input the task execution result and the target data label into the cross entropy loss function to output loss information.

[0085] Step 3044: Update the model parameters of the model structure corresponding to the data modal information to be added in the above-mentioned feature extraction model after addition according to the above-mentioned loss information.

[0086] In some embodiments, the execution subject may update the model parameters of the model structure corresponding to the data modal information to be added in the feature extraction model after addition according to the loss information. The parameters corresponding to the data feature extraction model in the feature extraction model after addition will not be updated. That is, only the parameters corresponding to the model structure corresponding to the data modal information to be added are updated.

[0087] As an example, in response to determining that the loss information is less than a predetermined loss value, the model parameters of the model structure corresponding to the data modal information to be added in the above-mentioned feature extraction model after addition are not updated. In response to determining that the loss information is greater than or equal to the predetermined loss value, the model parameters of the model structure corresponding to the data modal information to be added in the above-mentioned feature extraction model after addition are updated.

[0088] Optionally, the training data includes: training sub-data sets for the data modality information to be added and the target input data modality information. Each training sub-data has corresponding data modality sub-information.

[0089] Optionally, the step of inputting the training data into the post-addition feature extraction model to output the first feature information may include the following steps:

[0090] In the first step, for each training sub-data in the above training sub-datasets, perform the following data processing steps:

[0091] Sub-step 1: encode the training sub-data to obtain an encoding vector.

[0092] As an example, the execution subject may input the training sub-data into the encoding layer in the feature extraction model after adding to obtain the encoding vector. The encoding layer may be a network layer for encoding the data. In practice, the encoding layer may be a Bert encoding layer.

[0093] Sub-step 2: normalize the above encoding vector to obtain a normalized vector.

[0094] As an example, the execution subject may input the encoding vector into a normalization layer in the added feature extraction model to output a normalized vector, wherein the normalization layer may be a network layer that converts the vector dimension corresponding to the encoding vector into the target dimension.

[0095] In the second step, modal feature fusion is performed on the obtained standardized vector set to generate a modal fusion feature vector for the above training sub-data set as the above first feature information.

[0096] As an example, the autonomous subject may input the standardized vector set into a modal feature fusion layer in the post-addition feature extraction model to generate the first feature information. The modal feature fusion layer may be a network layer that performs feature fusion on the data features of each modality. For example, the modal feature fusion layer may perform vector concatenation or vector addition on the standardized vector set to perform feature fusion on the data features of each modality.

[0097] Optionally, the step of performing vector normalization on the encoding vector to obtain a normalized vector may include the following steps:

[0098] The first step is to generate the key vector, value vector and query vector for the above encoding vector.

[0099] The key vector may be the K (Key) vector in the self-attention mechanism. The value vector may be the V (Value) vector in the self-attention mechanism. The query vector may be the Q (Query) vector in the self-attention mechanism.

[0100] As an example, the execution subject may multiply the encoding vector with the key mapping matrix to generate a key vector. The execution subject may multiply the encoding vector with the value mapping matrix vector to generate a value vector. The parameters in the key mapping matrix and the value mapping matrix may be pre-trained. The Q vector may be a Prompt matrix. The parameters in the Q vector may be pre-trained.

[0101] In the second step, the key vector, the value vector and the query vector are input into a cross attention model to generate a normalized vector. In practice, the cross attention model may be a Cross Attention model.

[0102] As an example, the above execution entity can generate a normalized vector by the following formula:

[0103]

[0104] Where S can be a normalized vector. Q can be a key vector. K can be a value vector. E can be the number of column vectors of the normalized vector. V can be a query vector. Softmax can be a normalized exponential function.

[0105] from Figure 3 It can be seen that Figure 2 Compared with the description of some corresponding embodiments, Figure 3The process 300 of the model training method in some corresponding embodiments further highlights the model training process of the post-addition feature extraction model. By only performing model training on the model structure corresponding to the data modal information to be added in the post-addition feature extraction model, not only can the model training speed of the post-addition feature extraction model be accelerated, but also the model accuracy of the subsequent multimodal feature extraction model can be guaranteed.

[0106] Continue to refer Figure 4 , shows a process 400 of some embodiments of the information generation method according to the present disclosure. The information generation method comprises the following steps:

[0107] Step 401, obtaining data to be feature extracted.

[0108] In some embodiments, the execution subject (for example, an electronic device) of the above-mentioned information generation method can obtain the data to be feature extracted by wired or wireless means. Among them, the data modality information to be added includes at least one first data modality sub-information. The target input data modality information includes at least one second data modality sub-information. The data modality information corresponding to the above-mentioned data to be feature extracted is at least one of the above-mentioned at least one first data modality sub-information and the above-mentioned at least one second data modality sub-information. It is determined that there are different data modality sub-information in at least one first data modality sub-information and at least one second data modality sub-information, and the same data modality sub-information may exist. For example, at least one second data modality sub-information may include: audio modality information, image modality information, and title modality information. At least one first data modality sub-information may include: image modality information, title modality information, and text content information. The data modality information corresponding to the data to be feature extracted may be image modality information and title modality information, or may be audio modality information and image modality information, or may be image modality information and title modality information.

[0109] Step 402: Generate second feature information for the data to be feature extracted using a multimodal feature extraction model.

[0110] In some embodiments, the execution subject may generate second feature information for the data to be feature extracted using a multimodal feature extraction model. The second feature information may represent feature information of the data to be feature extracted. The multimodal feature extraction model may be a model for extracting data features from multimodal data, wherein the multimodal feature extraction model is generated based on the model training method disclosed herein.

[0111] As an example, the above-mentioned execution entity may input the data to be feature extracted into the multimodal feature extraction model to output the second feature information.

[0112] In some optional implementations of some embodiments, the data to be extracted with features includes: a sub-data set to be extracted with features, wherein each sub-data to be extracted with features has corresponding data modality sub-information.

[0113] Optionally, the above-mentioned use of the multimodal feature extraction model to generate the second feature information for the above-mentioned data to be feature extracted may include the following steps:

[0114] The first step is to determine whether there is data to be extracted with missing information in the above sub-dataset to be extracted.

[0115] The missing information mentioned above may be the lack of data of a certain modality.

[0116] In the second step, in response to the determination of existence, the data to be feature extracted with missing information is removed from the above-mentioned sub-dataset to be feature extracted to obtain a data set after removal.

[0117] The third step is to input the above-mentioned removed data set into the above-mentioned multimodal feature extraction model to output the above-mentioned second feature information.

[0118] The above-mentioned various embodiments of the present disclosure have the following beneficial effects: through the information generation method of some embodiments of the present disclosure, a multimodal feature extraction model can be used to accurately extract feature information of the data to be feature extracted.

[0119] Further references Figure 5 As an implementation of the methods shown in the above figures, the present disclosure provides some embodiments of a model training device, and these device embodiments are Figure 2 Corresponding to the method embodiments shown, the model training device can be specifically applied to various electronic devices.

[0120] like Figure 5 As shown, a model training device 500 includes: a first acquisition unit 501, an adding unit 502 and a training unit 503. The first acquisition unit 501 is configured to acquire a pre-trained data feature extraction model, data modal information to be added and a training data set for the data modal information to be added; the adding unit 502 is configured to add the model structure corresponding to the data modal information to be added to the data feature extraction model to obtain a feature extraction model after addition; the training unit 503 is configured to perform model training on the feature extraction model after addition according to the training data set to obtain a multimodal feature extraction model, wherein the input data modal information corresponding to the multimodal feature extraction model is modal union information, the modal union information is the union information of the target input data modal information and the data modal information to be added, and the target input data modal information is the modal information of the input data corresponding to the data feature extraction model.

[0121] In some optional implementations of some embodiments, the training unit 503 can be further configured to: determine a first initial task execution model for training the above-mentioned post-addition feature extraction model; for each training data in the above-mentioned training data set, perform the following training steps: input the above-mentioned training data into the above-mentioned post-addition feature extraction model to output the first feature information; input the above-mentioned first feature information into the above-mentioned first initial task execution model to output the task execution result; determine the loss information for the above-mentioned task execution result and the target data label, wherein the above-mentioned target data label is a label associated with the above-mentioned training data; and according to the above-mentioned loss information, update the model parameters of the model structure corresponding to the data modal information to be added in the above-mentioned post-addition feature extraction model.

[0122] In some optional implementations of some embodiments, the above-mentioned training data includes: a training sub-data set for the above-mentioned data modal information to be added and the above-mentioned target input data modal information; and the training unit 503 can be further configured to: for each training sub-data in the above-mentioned training sub-data set, perform the following data processing steps: perform data encoding on the above-mentioned training sub-data to obtain a coding vector; perform vector normalization on the above-mentioned coding vector to obtain a standardized vector; perform modal feature fusion on the obtained standardized vector set to generate a modal fusion feature vector for the above-mentioned training sub-data set as the above-mentioned first feature information.

[0123] In some optional implementations of some embodiments, the training unit 503 may be further configured to: generate a key vector, a value vector and a query vector for the above-mentioned encoding vector; and input the above-mentioned key vector, the above-mentioned value vector and the above-mentioned query vector into a cross-attention model to generate a standardized vector.

[0124] In some optional implementations of some embodiments, the above-mentioned feature extraction model after adding includes: at least one data coding model, and the data coding model in at least one data coding model has corresponding data modal sub-information; and the training unit 503 can be further configured to: determine the data modal sub-information corresponding to the above-mentioned training sub-data as the target data modal sub-information; use the data coding model in the above-mentioned at least one data coding model corresponding to the above-mentioned target data modal sub-information to data encode the above-mentioned training sub-data to obtain a coding vector.

[0125] In some optional implementations of some embodiments, the data feature extraction model is trained through the following steps: determining a second initial task execution model trained for the initial data feature extraction model; obtaining sample data for the target input data modal information, wherein the sample data includes: target data and target label, and the target data includes: a data subset, and each data in the data subset has corresponding data modal sub-information in the target input data modal information; selecting at least one data from the data subset; training the initial data feature extraction model and the second initial task execution model according to the at least one data and the target label to obtain a data feature extraction model and a second task execution model.

[0126] It is understandable that the units recorded in the model training device 500 are similar to the reference Figure 2 Therefore, the operations, features and beneficial effects described above for the method are also applicable to the model training device 500 and the units contained therein, and will not be repeated here.

[0127] Further references Figure 6 As an implementation of the methods shown in the above figures, the present disclosure provides some embodiments of an information generating device, and these device embodiments are Figure 4 Corresponding to the method embodiments shown, the information generating device can be specifically applied to various electronic devices.

[0128] like Figure 6 As shown, an information generating device 600 includes: a second acquiring unit 601 and a generating unit 602. The second acquiring unit 601 is configured to acquire data to be feature extracted, wherein the data modality information to be added includes at least one first data modality sub-information, and the target input data modality information includes at least one second data modality sub-information; the data modality information corresponding to the data to be feature extracted is at least one of the at least one first data modality sub-information and the at least one second data modality sub-information; the generating unit 602 is configured to generate second feature information for the data to be feature extracted using a multimodal feature extraction model, wherein the multimodal feature extraction model is generated based on the model training method disclosed in the present invention.

[0129] In some optional implementations of some embodiments, the above-mentioned data to be feature extracted includes: a sub-data set to be feature extracted; and the generation unit 602 can be further configured to: determine whether there is data to be feature extracted with missing information in the above-mentioned sub-data set to be feature extracted; in response to determining that there is, remove the data to be feature extracted with missing information from the above-mentioned sub-data set to be feature extracted to obtain a data set after removal; input the above-mentioned data set after removal into the above-mentioned multimodal feature extraction model to output the above-mentioned second feature information.

[0130] It is understandable that the units recorded in the information generating device 600 are similar to those in the reference Figure 4 Therefore, the operations, features and beneficial effects described above for the method are also applicable to the information generating device 600 and the units included therein, and will not be described in detail here.

[0131] Reference below Figure 7 , which shows an electronic device (eg, Figure 1 Schematic diagram of the structure of the electronic device 101)700. Figure 7 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0132] like Figure 7 As shown, the electronic device 700 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory 702 or a program loaded from a storage device 708 into a random access memory 703. Various programs and data required for the operation of the electronic device 700 are also stored in the random access memory 703. The processing device 701, the read-only memory 702, and the random access memory 703 are connected to each other via a bus 704. An input / output interface 705 is also connected to the bus 704.

[0133] Typically, the following devices may be connected to the input / output interface 705: an input device 706 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 707 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 708 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 709. The communication device 709 may allow the electronic device 700 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 7 The electronic device 700 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 7Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0134] In particular, according to some embodiments of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from a network through a communication device 709, or installed from a storage device 708, or installed from a read-only memory 702. When the computer program is executed by the processing device 701, the above-mentioned functions defined in the method of some embodiments of the present disclosure are executed.

[0135] It should be noted that the computer-readable medium in some embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present disclosure, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In some embodiments of the present disclosure, the computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0136] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0137] The computer-readable medium may be included in the electronic device; or it may exist independently without being installed in the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: obtains a pre-trained data feature extraction model, data modal information to be added, and a training data set for the data modal information to be added; adds the model structure corresponding to the data modal information to be added to the data feature extraction model to obtain a feature extraction model after addition; performs model training on the feature extraction model after addition according to the training data set to obtain a multimodal feature extraction model, wherein the input data modal information corresponding to the multimodal feature extraction model is modal union information, the modal union information is the union information of the target input data modal information and the data modal information to be added, and the target input data modal information is the modal information of the input data corresponding to the data feature extraction model. Acquire data to be feature extracted, wherein the data modal information to be added includes at least one first data modal sub-information, and the target input data modal information includes at least one second data modal sub-information; the data modal information corresponding to the data to be feature extracted is at least one of the at least one first data modal sub-information and the at least one second data modal sub-information; utilize a multimodal feature extraction model to generate second feature information for the data to be feature extracted, wherein the multimodal feature extraction model is generated based on the model training method disclosed in the present invention.

[0138] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0139] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0140] The units described in some embodiments of the present disclosure may be implemented by software or by hardware. The described units may also be provided in a processor, for example, may be described as: a processor comprising a first acquisition unit, an adding unit and a training unit. Among them, the names of these units do not constitute a limitation on the unit itself in certain cases, for example, the first acquisition unit may also be described as "a unit for acquiring a pre-trained data feature extraction model, data modal information to be added, and a training data set for the above-mentioned data modal information to be added".

[0141] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0142] Some embodiments of the present disclosure also provide a computer program product, including a computer program, which implements any of the above-mentioned model training methods and information generation methods when executed by a processor.

[0143] The above descriptions are only some preferred embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with the technical features with similar functions disclosed in the embodiments of the present disclosure (but not limited to) and the technical solutions formed.

Claims

1. A model training method, comprising: Obtaining a pre-trained data feature extraction model, data modality information to be added, and a training data set for the data modality information to be added; Adding the model structure corresponding to the data modal information to be added to the data feature extraction model to obtain a feature extraction model after addition; According to the training data set, the added feature extraction model is trained to obtain a multimodal feature extraction model, wherein the input data modal information corresponding to the multimodal feature extraction model is modal union information, the modal union information is the union information of the target input data modal information and the modal information of the data to be added, and the target input data modal information is the modal information of the input data corresponding to the data feature extraction model.

2. The method according to claim 1, wherein: The step of performing model training on the added feature extraction model according to the training data set to obtain a multimodal feature extraction model comprises: Determining a first initial task execution model trained for the added feature extraction model; For each training data in the training data set, the following training steps are performed: Inputting the training data into the post-addition feature extraction model to output first feature information; Inputting the first feature information into the first initial task execution model to output a task execution result; Determining loss information for the task execution result and a target data label, wherein the target data label is a label associated with the training data; According to the loss information, model parameters of the model structure corresponding to the data modal information to be added in the added feature extraction model are updated.

3. The method according to claim 2, wherein: The training data includes: a training sub-data set for the to-be-added data modality information and the target input data modality information; and The step of inputting the training data into the post-addition feature extraction model to output first feature information comprises: For each training sub-data in the training sub-data set, the following data processing steps are performed: Performing data encoding on the training sub-data to obtain an encoding vector; Performing vector normalization on the encoding vector to obtain a normalized vector; Modal feature fusion is performed on the obtained standardized vector set to generate a modal fusion feature vector for the training sub-data set as the first feature information.

4. The method according to claim 3, wherein: The step of performing vector normalization on the encoding vector to obtain a normalized vector includes: generating a key vector, a value vector, and a query vector for the encoding vector; The key vector, the value vector, and the query vector are input into a criss-cross attention model to generate a normalized vector.

5. The method according to claim 3, wherein: The added feature extraction model includes: at least one data coding model, and the data coding model in the at least one data coding model has corresponding data modality sub-information; and The step of performing data encoding on the training sub-data to obtain an encoding vector includes: Determining data modality sub-information corresponding to the training sub-data as target data modality sub-information; The training sub-data is data encoded by utilizing a data encoding model in the at least one data encoding model corresponding to the target data modality sub-information to obtain an encoding vector.

6. The method according to claim 1, wherein: The data feature extraction model is trained by the following steps: Determining a second initial task execution model trained for the initial data feature extraction model; Acquire sample data for the target input data modality information, wherein the sample data includes: target data and a target label, and the target data includes: a data subset, and each data in the data subset has corresponding data modality sub-information in the target input data modality information; Selecting at least one data from the data subset; The initial data feature extraction model and the second initial task execution model are trained according to the at least one data and the target label to obtain a data feature extraction model and a second task execution model.

7. A method for generating information, comprising: Acquire data to be feature extracted, wherein the data modality information to be added includes at least one first data modality sub-information, and the target input data modality information includes at least one second data modality sub-information; and the data modality information corresponding to the data to be feature extracted is at least one of the at least one first data modality sub-information and the at least one second data modality sub-information; A multimodal feature extraction model is used to generate second feature information for the data to be feature extracted, wherein the multimodal feature extraction model is generated based on the method described in one of claims 1-6.

8. The method according to claim 7, wherein: The data to be feature extracted includes: a sub-data set to be feature extracted; and The step of using a multimodal feature extraction model to generate second feature information for the data to be feature extracted includes: Determine whether there is data to be extracted with missing information in the sub-data set to be extracted with features; In response to determining that there is, removing the data to be feature extracted with missing information from the sub-data set to be feature extracted to obtain a data set after removal; The removed data set is input into the multimodal feature extraction model to output the second feature information.

9. A model training device, comprising: A first acquisition unit is configured to acquire a pre-trained data feature extraction model, data modality information to be added, and a training data set for the data modality information to be added; An adding unit, configured to add a model structure corresponding to the data modal information to be added to the data feature extraction model to obtain a feature extraction model after addition; The training unit is configured to perform model training on the added feature extraction model according to the training data set to obtain a multimodal feature extraction model, wherein the input data modal information corresponding to the multimodal feature extraction model is modal union information, the modal union information is the union information of the target input data modal information and the modal information of the data to be added, and the target input data modal information is the modal information of the input data corresponding to the data feature extraction model.

10. An information generating device, comprising: A second acquisition unit is configured to acquire data to be feature extracted, wherein the data modality information to be added includes at least one first data modality sub-information, and the target input data modality information includes at least one second data modality sub-information; and the data modality information corresponding to the data to be feature extracted is at least one of the at least one first data modality sub-information and the at least one second data modality sub-information; A generating unit is configured to generate second feature information for the data to be feature extracted using a multimodal feature extraction model, wherein the multimodal feature extraction model is generated based on the method described in one of claims 1-6.

11. An electronic device, comprising: one or more processors; a storage device having one or more programs stored thereon, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 8.

12. A computer readable medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

13. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 8.