Multimodal data processing and pre-training method of pre-training model, and electronic device

Abstract

Embodiments of the present application provide a multi-modal data processing method and a pre-training method of a pre-training model, and an electronic device, wherein the multi-modal data processing method comprises: obtaining modal data to be processed, and inputting the modal data into a preset modal data processing model, wherein the modal data processing model is obtained based on task migration of a pre-training model after training is completed, the modal data processing model comprises a plurality of feature fusion parts, the plurality of feature fusion parts are based on text features of text data in the modal data, and are assisted by image features of image data in the modal data, the image features are added and spliced to the text features to obtain spliced features; at least the spliced features obtained through the feature fusion parts are used to perform task processing on the modal data, and task data of a preset mode is obtained according to a processing result. Through the embodiments of the present application, the accuracy of multi-modal data processing can be improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a multimodal data processing, a pre-training method for a pre-trained model, and an electronic device. Background Technology

[0002] With the development of artificial intelligence in natural language processing technology, pre-trained language models, which can be effectively transferred and applied to various downstream natural language processing scenarios, have been widely used. Due to the powerful learning capabilities of pre-trained language models, they have gradually transitioned from being applied to unimodal text scenarios to being applicable to multimodal scenarios.

[0003] However, in multimodal scenarios involving both text and image data, text data is typically short and highly abstract, while image data, on the other hand, can form long visual sequences rich in information. Consequently, when processing multimodal data using a pre-trained model, the smaller volume of text data compared to image data leads to distorted processing results. For example, in... Figure 1 The illustrated method for processing multimodal data uses a visual encoder to obtain image features and a text encoder to obtain text features. Then, the image and text features are simply concatenated as input to achieve feature fusion. Next, a self-attention mechanism is used to align the modalities from the bottom up. However, this approach treats both image and text data equally. Due to the asymmetry of the modal data (text features have a much smaller data volume than image features), especially when there are significant differences in data density or sequence length between different modalities, the model may produce inaccurate results.

[0004] Therefore, how to reduce the asymmetry in the fusion of text features and image features in this scenario in order to improve the accuracy of multimodal data processing has become an urgent problem to be solved. Summary of the Invention

[0005] In view of this, embodiments of this application provide a pre-training scheme for multimodal data processing and pre-trained models to at least partially solve the above-mentioned problems.

[0006] According to a first aspect of the embodiments of this application, a multimodal data processing method is provided, comprising: acquiring modal data to be processed, and inputting the modal data into a preset modal data processing model, wherein the modal data processing model is obtained by task transfer based on a pre-trained model that has been trained, the modal data processing model includes multiple feature fusion parts, the multiple feature fusion parts using text features of text data in the modal data as a basis and image features of image data in the modal data as an auxiliary, summing and concatenating the image features into the text features to obtain concatenated features; performing task processing on the modal data using at least the concatenated features obtained by the feature fusion parts, and obtaining task data of a preset modality based on the processing result.

[0007] According to a second aspect of the embodiments of this application, a pre-training method for a pre-trained model is provided, wherein the pre-trained model includes a visual encoder, a text encoder, multiple feature fusion parts, and a decoder; the method includes:

[0008] Training samples for pre-training the pre-trained model are obtained, including image sample data and corresponding text sample data. The image sample data is encoded by the visual encoder to obtain image sample features, and the text sample features are encoded by the text encoder to obtain text sample features. The image sample features and the text sample features are input into the multiple feature fusion part, which sums and concatenates the image sample features into the text sample features based on the text sample features and with the image sample features as an auxiliary, to obtain concatenated features. The concatenated features are input into the decoder to predict and generate modal sample data of a preset modality. The multimodal data processing model is pre-trained based on the modal sample data and a preset loss function.

[0009] According to a third aspect of the embodiments of this application, another multimodal data processing method is provided, comprising: acquiring image data to be processed; obtaining descriptive text data corresponding to the image data through a preset modal data processing model, wherein the descriptive text data is used to describe the content of the image data, and the modal data processing model is a model obtained after task transfer based on a pre-trained model that has been trained, wherein the pre-trained model is trained by the method described in the second aspect.

[0010] According to a fourth aspect of the embodiments of this application, another multimodal data processing method is provided, comprising: acquiring image data to be processed and text retrieval requirement data for retrieving content in the image data; retrieving retrieval results corresponding to the text retrieval requirement data from the image data through a preset modal data processing model, wherein the retrieval results are used to retrieve image content results obtained by retrieving the image data through text descriptions, wherein the modal data processing model is a model obtained after task transfer based on a pre-trained model that has been trained, and the pre-trained model is trained by the method described in the second aspect.

[0011] According to a fifth aspect of the embodiments of this application, another multimodal data processing method is provided, comprising: acquiring image data to be processed and text question data corresponding to the image data; obtaining text answer data based on the image data through a preset modal data processing model, wherein the modal data processing model is a model obtained by task transfer based on a pre-trained model that has been trained, and the pre-trained model is trained by the method described in the second aspect.

[0012] According to a sixth aspect of the embodiments of this application, another multimodal data processing method is provided, comprising: acquiring image data to be processed and text location request data for locating a target in the image data; obtaining location information of the target based on the image data through a preset modal data processing model, wherein the location information is used to describe the position of the target in the image data in text, and the modal data processing model is a model obtained after task transfer based on a pre-trained model that has been trained, wherein the pre-trained model is trained by the method described in the second aspect.

[0013] According to a seventh aspect of the embodiments of this application, a multimodal data processing method is provided, comprising: acquiring modal data to be processed; receiving information on a modal data processing model for processing the modal data input through an interactive interface; invoking the modal data processing model indicated by the information to process the modal data, and obtaining task data of a preset modality based on the processing result, wherein the modal data processing model is a model obtained after task transfer based on a pre-trained model that has been trained, and the pre-trained model is trained by the method described in the second aspect.

[0014] According to an eighth aspect of the present application, an electronic device is provided, comprising: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other via the communication bus; the memory is used to store at least one executable instruction, wherein the executable instruction causes the processor to perform an operation corresponding to any of the methods described in the first to seventh aspects.

[0015] According to a ninth aspect of the embodiments of this application, a computer storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method described in any of the first to seventh aspects.

[0016] According to the solution provided in the embodiments of this application, a preset modal data processing model is used in multimodal data processing scenarios, especially in the processing of text and image data. Because the multiple feature fusion components in this model can use text features as the basis and image features as an aid, the text features are summed and concatenated with image features. This ensures that when using text features as the basis and image features as an aid, the image features are not obscured or significantly affected, while allowing the text features to carry rich image information, or in other words, image information can supplement the text features. Therefore, although the amount of text data is still less than that of image data, the participation of image features effectively avoids this data asymmetry, thereby obtaining text features that more accurately express image information and improving the accuracy of multimodal data processing. Furthermore, it also achieves more precise and efficient task processing. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings.

[0018] Figure 1 This is a schematic diagram of a model for multimodal data processing in the prior art;

[0019] Figure 2 A schematic diagram of an exemplary system to which the embodiments of this application are applicable;

[0020] Figure 3A This is a flowchart illustrating the steps of a pre-training method for a pre-trained model according to Embodiment 1 of this application;

[0021] Figure 3B for Figure 3A A schematic diagram of the structure of a pre-trained model in the embodiment shown;

[0022] Figure 4A This is a flowchart illustrating the steps of a multimodal data processing method according to Embodiment 2 of this application;

[0023] Figure 4B for Figure 4A A schematic diagram of the first scenario example in the illustrated embodiment;

[0024] Figure 4C for Figure 4A A schematic diagram of the second scenario example in the illustrated embodiment;

[0025] Figure 4D for Figure 4A A schematic diagram of the third scenario example in the illustrated embodiment;

[0026] Figure 4E for Figure 4A A schematic diagram of the fourth scenario example in the illustrated embodiment;

[0027] Figure 5 This is a flowchart illustrating the steps of a multimodal data processing method according to Embodiment 3 of this application;

[0028] Figure 6 This is a schematic diagram of the structure of an electronic device according to Embodiment 4 of this application. Detailed Implementation

[0029] To enable those skilled in the art to better understand the technical solutions in the embodiments of this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art should fall within the protection scope of the embodiments of this application.

[0030] The specific implementation of the embodiments of this application will be further described below with reference to the accompanying drawings.

[0031] Figure 2 An exemplary system applicable to embodiments of this application is shown. For example... Figure 2 As shown, the system 100 may include a cloud server 102, a communication network 104, and / or one or more user devices 106. Figure 2 The example in the text shows multiple user devices.

[0032] The cloud server 102 can be any suitable device for storing information, data, programs, and / or any other suitable type of content, including but not limited to distributed storage system devices, server clusters, computing cloud server clusters, etc. In some embodiments, the cloud server 102 can perform any suitable function. For example, in some embodiments, a modal data processing model is obtained by task transfer based on a pre-trained model that has been trained. This modal data processing model includes multiple feature fusion parts for adding and concatenating image features of image data in the modal data based on text features of text data in the modal data to obtain fused text features. Here, different data existence forms or data sources can all be referred to as a modality. Modal data refers to data existing in a certain modality, such as image data or text data. Multimodal data refers to data formed by two or more modalities. In some embodiments, the cloud server 102 obtains the text features corresponding to the modal data to be processed through the modal data processing model, and then processes the corresponding task based on the text features to obtain the final task data. As an optional example, in some embodiments, the cloud server 102 can also be used to pre-train the pre-trained model. As another example, in some embodiments, the cloud server 102 can also be used to transfer the pre-trained model to downstream tasks. As another example, in some embodiments, the cloud server 102 can also receive multimodal data processing requests from the user device 106 and return the processed data to the user device 106.

[0033] In some embodiments, communication network 104 may be any suitable combination of one or more wired and / or wireless networks. For example, communication network 104 may include any one or more of the following: the Internet, intranet, wide area network (WAN), local area network (LAN), wireless network, digital subscriber line (DSL) network, frame relay network, asynchronous transfer mode (ATM) network, virtual private network (VPN), and / or any other suitable communication network. User equipment 106 may be connected to communication network 104 via one or more communication links (e.g., communication link 112), and communication network 104 may be linked to cloud server 102 via one or more communication links (e.g., communication link 114). Communication links may be any communication link suitable for transmitting data between user equipment 106 and cloud server 102, such as network links, dial-up links, wireless links, hardwired links, any other suitable communication links, or any suitable combination of such links.

[0034] User equipment 106 may include any one or more user devices suitable for interacting with a user. In some embodiments, user equipment 106 may send a multimodal data processing request and information about the requested multimodal data to cloud server 102, so that cloud server 102 performs multimodal data processing based on the request. In some embodiments, user equipment 106 may also receive multimodal data processing results returned by cloud server 102. In some embodiments, user equipment 106 may include any suitable type of device. For example, in some embodiments, user equipment 106 may include mobile devices, tablet computers, laptop computers, desktop computers, wearable computers, game consoles, media players, vehicle entertainment systems, and / or any other suitable type of user device.

[0035] Based on the above system, the solution of this application will be described below through several embodiments.

[0036] To facilitate understanding of the solutions in the embodiments of this application, the following is combined with... Figure 3A and Figure 3B First, the pre-training process of the pre-trained model used in the embodiments of this application will be described.

[0037] Example 1

[0038] First, such as Figure 3B As shown, the pre-trained model in this embodiment includes a visual encoder, a text encoder, multiple feature fusion parts, and a decoder. Figure 3B The multiple feature fusion components are simply illustrated as two, but those skilled in the art should understand that in practical applications, they can set more feature fusion components according to actual needs. Furthermore, in the embodiments of this application, unless otherwise specified, "multiple," "various types," "multi-layers," and other quantities related to "multiple" all refer to two or more.

[0039] Depend on Figure 3BAs can be seen, in at least some feature fusion parts (such as 1, 2, 3... 9, or 10 of 10 feature fusion parts), each feature fusion part can include multiple multimodal feature summing layers and one multimodal feature concatenation layer. After text features are input into each multimodal feature summing layer, the multimodal feature summing layer adds the image features into the text features before extracting the text features. After the extracted text features are input into the last multimodal feature summing layer, the text features are then added to the image features initially input into the current feature fusion part by the multimodal feature concatenation layer to form the concatenated features. Between two adjacent feature fusion parts, the output of the previous feature fusion part, i.e., the concatenated features, serves as the input to the next feature fusion part. For multimodal feature summing layers, the computation speed is relatively fast, but some visual information, i.e., image features, may be lost. For multimodal feature concatenation layers, visual information is effectively preserved, but the speed is slower. By combining a multimodal feature summation layer and a multimodal feature splicing layer, an effective balance can be achieved between speed and information, resulting in better feature extraction results.

[0040] Based on this, the pre-training method of the pre-trained model in this embodiment is as follows: Figure 3A As shown, it includes:

[0041] Step S202: Obtain training samples for pre-training the pre-trained model.

[0042] The training samples include image sample data and corresponding text sample data. The text sample data corresponding to the image sample data is related to the content of the image. For example, if there is a puppy in the image, the corresponding text sample data could be "A puppy is sitting in front of a door," or "Is there a puppy in the picture?", or "Please mark the location of the puppy in the picture," and so on. This application embodiment does not limit the specific implementation of the image sample data and its corresponding text sample data. However, it is not limited to this; the training samples will also include some negative samples. For example, there may be no puppy in the image, but the text sample data could be "A puppy is sitting in front of a door," etc. By combining positive and negative training samples, the trained model can be made more robust and powerful.

[0043] Step S204: Encode the image sample data using a visual encoder to obtain image sample features, and encode the text sample features using a text encoder to obtain text sample features.

[0044] The specific implementations of visual encoders and text encoders can be achieved by those skilled in the art using appropriate encoder structures according to actual needs, including but not limited to encoders based on the Transformer structure. The Transformer is an encoder-decoder structure based on an attention mechanism. By introducing the attention mechanism, the Transformer can extract target features relevant to the model task more efficiently.

[0045] The visual encoder and text encoder are connected to the first feature fusion part, specifically the first multimodal summation layer of the first feature fusion part. The visual encoder takes as input the vector corresponding to the image sample data and outputs an image encoded vector, i.e., image sample features; the text encoder takes as input the vector corresponding to the text sample data and outputs a text encoded vector, i.e., text sample features.

[0046] Step S206: Input image sample features and text sample features into multiple feature fusion parts, so that the image sample features are summed and concatenated into the text sample features based on the text sample features and with the image sample features as an auxiliary feature, in order to obtain concatenated features.

[0047] The multimodal feature summation layer is used to sum the text sample features using image sample features, and then extract text features based on the summed features. The multimodal feature concatenation layer is used to concatenate the text feature extraction results and the image sample features. For example, the feature summation layer can directly sum the features or use a co-attention mechanism to sum the text sample features and image sample features.

[0048] by Figure 3B The model structure shown is an example. In the first feature fusion section, the text sample features output by the text encoder are input to the first multimodal feature summing layer. In this layer, the image sample features V1 output by the image encoder are summed with the text sample features T11. Then, this multimodal feature summing layer extracts features from the text sample features T11 after summing with the image sample features V1, resulting in the text sample features T12 output by the first multimodal feature summing layer. The text sample features T12 output by the first multimodal feature summing layer are input to the second multimodal feature summing layer. In this layer, the text sample features T12 are again summed with the image sample features V1 output by the image encoder. Then, this multimodal feature summing layer extracts features from the text sample features T12 after summing with the image sample features V1 again, resulting in the text sample features T13 output by the second multimodal feature summing layer. This process continues until the last multimodal feature summing layer of the first feature fusion section.

[0049] Assuming there are two multimodal feature summation layers in the first feature fusion part, as mentioned earlier, the second multimodal feature summation layer will output text sample feature T13. Next, this text sample feature T13 will be input together with the image sample feature V1 into the multimodal feature concatenation layer for feature concatenation. After concatenation, the text sample feature incorporates information from the image sample feature, and similarly, the image sample feature also incorporates information from the text sample feature.

[0050] The spliced ​​features are still divided into two parts: new image sample features with the same dimension as the original image sample features, simply denoted as V2, and new text sample features with the same dimension as the original text sample features, simply denoted as T21.

[0051] Next, in the second feature fusion section, in its first multimodal feature summing layer, image sample feature V2 is summed with text sample feature T21. Then, this multimodal feature summing layer extracts features from the text sample feature T21 after summing with image sample feature V2, obtaining text sample feature T22 output by the first multimodal feature summing layer. The text sample feature T22 output by the first multimodal feature summing layer is input to the second multimodal feature summing layer, where the text sample feature T22 is summed again with image sample feature V2 output by the image encoder. Then, this multimodal feature summing layer extracts features from the text sample feature T22 after summing with image sample feature V2 again, obtaining text sample feature T23 output by the second multimodal feature summing layer.

[0052] Next, the text sample feature T23, along with the image sample feature V2, is input into the multimodal feature concatenation layer of the second feature fusion part for feature concatenation. After concatenation, the text sample feature incorporates information from the image sample feature, and similarly, the image sample feature incorporates information from the text sample feature. If the current feature fusion part is the last feature fusion part of the model, the concatenated features will not be further segmented and will be input into the subsequent decoder as complete concatenated features.

[0053] Step S208: Input the spliced ​​features into the decoder to predict and generate modal sample data of the preset modality.

[0054] Because this splicing feature effectively integrates textual and image information, becoming cross-modal feature data, it can be effectively used to learn text generation after being input into a text decoder, in order to predict and obtain modal sample data of the corresponding modality. In this embodiment, the preset modality is mainly the text modality.

[0055] This decoder can predict and generate text sample data such as image description text, search result text sample data such as image text retrieval, answer text sample data such as visual question answering, or text sample data for visual localization results. The final predicted result generated by the decoder is related to the pre-training task. In practical applications, those skilled in the art can use training samples and loss functions adapted to the pre-training task, as well as matching decoder task settings, according to actual needs.

[0056] In this embodiment, the specific implementation structure of the decoder is not limited. Optionally, a decoder based on the Transformer structure may be used.

[0057] Step S210: Based on the modal sample data and the preset loss function, pre-train the multimodal data processing model.

[0058] As mentioned above, the loss function can be set by those skilled in the art according to the actual task. In this embodiment, the specific implementation of the loss function is not limited. In one feasible approach, the loss function can be a self-supervised loss function.

[0059] After obtaining the modal sample data predicted by the decoder, the corresponding loss value can be obtained based on the preset loss function. Then, the pre-trained model can be trained based on this loss value until the training termination condition is met, such as reaching a preset number of training iterations, or the loss value meeting a preset threshold.

[0060] In this embodiment, during the training process of the pre-trained model, two modal fusion methods are combined in each feature fusion section. First, a multimodal feature summing layer fuses image sample features and text sample features to significantly improve the training speed of the model. Then, a multimodal feature concatenation layer connects and fuses the two modalities. Finally, the output cross-modal concatenated features are input into the text decoder for text generation learning. Thus, the pre-trained model obtained after training possesses both understanding and generation capabilities, and can generate more accurate result data.

[0061] After training, the pre-trained model will be used in downstream tasks. Therefore, task transfer can be achieved by fine-tuning the pre-trained model to suit the characteristics of the downstream tasks, thereby improving the model training efficiency for the downstream tasks. The following examples are based on the pre-trained model after task transfer. For ease of distinction, the pre-trained model used for the downstream tasks will be referred to as the modal data processing model.

[0062] The following describes the method for multimodal data processing using a modal data processing model.

[0063] Example 2

[0064] Reference Figure 4A The flowchart illustrates the steps of a multimodal data processing method according to Embodiment 2 of this application.

[0065] The multimodal data processing method in this embodiment includes the following steps:

[0066] Step S302: Obtain the modal data to be processed and input the modal data into the preset modal data processing model.

[0067] Different data formats or sources can be referred to as modalities. Modal data refers to data existing in a certain modality, such as image data or text data. Multimodal data refers to data formed by two or more modalities. In this embodiment, the modal data to be processed is usually multimodal data, such as image data and its corresponding text data. However, it is not limited to this. The modal data processing model in this embodiment can also process single-modal data, i.e., image data, which will be explained in detail below.

[0068] As mentioned earlier, the modal data processing model in this embodiment is a model obtained by task transfer from the pre-trained model trained in Embodiment 1. It also includes multiple feature fusion parts. These parts use the text features of the text data in the modal data as a foundation and the image features of the image data in the modal data as an aid, summing and concatenating the image features into the text features to obtain concatenated features. Specifically, the feature fusion part includes a multimodal feature summing layer and a multimodal feature concatenation layer. The multimodal feature summing layer uses the image features corresponding to the image data to sum the text features corresponding to the text data, and extracts text features based on the summed features. The multimodal feature concatenation layer concatenates the extracted text features and the image features to obtain concatenated features.

[0069] Step S304: At least through the spliced ​​features obtained from the feature fusion part, perform task processing on the modal data, and obtain the task data of the preset modality based on the processing results.

[0070] Similar to the pre-trained model in Example 1, the modal data processing model in this example also includes a visual encoder, a text encoder, and a decoder. The visual encoder encodes the image data to be processed to generate initial image features for inputting the first feature fusion part; the text encoder encodes the text data to be processed to generate initial text features for inputting the first feature fusion part; and the decoder decodes and generates task data for the preset modality based on the concatenated features output by multiple feature fusion parts.

[0071] Based on this, when the modal data to be processed includes image data to be processed and text data corresponding to the image data to be processed, the above-mentioned spliced ​​features obtained at least through the feature fusion part can be used to perform task processing on the modal data as follows: For a certain feature fusion part among multiple feature fusion parts, image features and text features are received as input to the feature fusion part, wherein the image features received by the first feature fusion part are the image features corresponding to the image data to be processed, and the text features received are the text features corresponding to the text data to be processed; through the multimodal feature summing layer of the feature fusion part, the image features are used to sum the text features, and text features are extracted based on the summed features to obtain the summed text features; through the multimodal feature splicing layer of the feature fusion part, the summed text features and image features are spliced, and the result of the feature splicing is used as the input of the next feature fusion part, until the spliced ​​features output by the last feature fusion part are obtained; based on the spliced ​​features output by the last feature fusion part, the image data and text data are processed.

[0072] For example, consider a modal data processing model that includes two feature fusion parts, each of which includes two multimodal feature summation layers and one multimodal feature splicing layer.

[0073] In the first feature fusion section, the text features output by the text encoder are input to the first multimodal feature summing layer. In this layer, the image features V'1 output by the image encoder are summed with the text features T'11. Then, this multimodal feature summing layer extracts features from the text features T'11 after summing with the image features V'1, resulting in the text features T'12 output by the first multimodal feature summing layer. The text features T'12 output by the first multimodal feature summing layer are input to the second multimodal feature summing layer. In this layer, the text features T'12 are again summed with the image features V'1 output by the image encoder. Then, this multimodal feature summing layer extracts features from the text features T'12 after summing with the image features V'1 again, resulting in the text features T'13 output by the second multimodal feature summing layer. This process continues until the last multimodal feature summing layer of the first feature fusion section.

[0074] Next, the text feature T'13 and the image feature V'1 are input into the multimodal feature concatenation layer for feature concatenation. After concatenation, the text feature incorporates the information of the image feature, and similarly, the image feature incorporates the information of the text feature.

[0075] The concatenated features are still divided into two parts: new image features with the same dimension as the original image features, simply denoted as V'2, and new text features with the same dimension as the original text features, simply denoted as T'21.

[0076] Next, in the second feature fusion section, in its first multimodal feature summing layer, image feature V'2 is summed with text feature T'21. Then, this multimodal feature summing layer extracts features from the text feature T'21 after summing with image feature V'2, obtaining text feature T'22 output by the first multimodal feature summing layer. The text feature T'22 output by the first multimodal feature summing layer is input to the second multimodal feature summing layer, where text feature T'22 is again summed with image feature V'2 output by the image encoder. Then, this multimodal feature summing layer extracts features from the text feature T'22 after summing with image feature V'2 again, obtaining text feature T'23 output by the second multimodal feature summing layer.

[0077] Next, the text feature T'23 and the image feature V'2 are input together into the multimodal feature splicing layer of the second feature fusion part for feature splicing. After splicing, the text feature is fused with the information of the image feature, and similarly, the image feature is also fused with the information of the text feature.

[0078] Furthermore, the complete spliced ​​features output by the second feature fusion part will be input into the decoder for corresponding task processing, including but not limited to image text retrieval tasks, visual question answering tasks, visual localization tasks, etc.

[0079] When the modal data to be processed is only the image data to be processed, the above-mentioned stitched features obtained through at least the feature fusion part can be used to process the modal data as follows: For a certain feature fusion part among multiple feature fusion parts, the image features input to the feature fusion part are received, wherein the image features received by the first feature fusion part are the image features corresponding to the image data to be processed; the image features are processed through the multimodal feature stitching layer of the feature fusion part, and the processing result is used as the input of the next feature fusion part, until the stitched features output by the last feature fusion part are obtained; based on the stitched features output by the last feature fusion part, the image data is processed.

[0080] Still with Figure 3BThe model structure shown is an example. When the input data to the model is only image data, since there is no text data, the model part for processing text data, including the text decoder and the multimodal feature summing layer mainly for extracting text features, will no longer be effective. After the image data is encoded by the visual encoder to obtain image features, it will be processed by the multimodal feature concatenation layer. At this time, although there is no concatenation of text features and image features, due to the parameter adjustment of the model during the training phase, the multimodal feature concatenation layer will extract richer information from the image features. The image features output by the first multimodal feature concatenation layer will be input into the next multimodal feature concatenation layer for further processing to extract even richer image features. For ease of explanation, in this embodiment, the features output by the multimodal feature concatenation layer are uniformly referred to as concatenated features. However, in this scenario, the concatenated features are mainly image features. The concatenated features output by the last multimodal feature concatenation layer are input into the decoder for corresponding task processing, such as generating image description text.

[0081] Regardless of whether the modal data to be processed includes image data and text data, or whether the modal data to be processed is image data, the task data of the preset modality obtained based on the processing result of the modal data can be at least one of the following: text data generated based on the image data to be processed (such as image description task, image text retrieval task, visual question answering task, etc.) and / or image localization data (such as visual localization task, etc.).

[0082] As can be seen, this embodiment utilizes a pre-defined modal data processing model in multimodal data processing scenarios, particularly those involving text and image data. Because the multiple feature fusion components in this model can use text features as a foundation and image features as an aid, it adds and concatenates image features to text features. This ensures that when using text features as a foundation and image features as an aid, image features are neither obscured nor significantly impacted, while allowing text features to carry rich image information, or in other words, image information can supplement text features. Therefore, although the amount of text data is still less than that of image data, the participation of image features effectively avoids this data asymmetry, resulting in text features that more accurately express image information and improving the accuracy of multimodal data processing. Consequently, more precise and efficient task processing is achieved.

[0083] The following examples illustrate the above process using multiple scenarios.

[0084] First scenario - Image description task scenario:

[0085] In this scenario, the process of implementing the image description task based on the aforementioned modal data processing model may include: acquiring image data to be processed; obtaining descriptive text data corresponding to the image data through a preset modal data processing model, wherein the descriptive text data is used to describe the content of the image data, and the modal data processing model is a model obtained after task transfer based on a pre-trained model that has been trained, such as the model trained by the method described in Example 1.

[0086] For example, such as Figure 4B As shown, the image depicts a person sitting on a sofa reading a book, with a smart speaker playing music on the table in front of them. After the image is input into the modal data processing model, it is processed sequentially through a visual encoder, a multi-feature fusion part (a multi-modal feature splicing layer of the multi-feature fusion part), and a decoder. The output is corresponding descriptive text data for the image, such as "A lady is sitting on a sofa reading a book and listening to music played by a smart speaker on the table in front of her."

[0087] The second scenario - image-text retrieval task scenario:

[0088] In this scenario, the process of implementing the image description task based on the aforementioned modal data processing model may include: acquiring image data to be processed and text retrieval requirement data for retrieving content from the image data; retrieving retrieval results corresponding to the text retrieval requirement data from the image data through a preset modal data processing model; the retrieval results are used to retrieve image content results from the image data through text description; the modal data processing model is a model obtained after task transfer based on a pre-trained model that has been trained, such as the model trained by the method described in Example 1.

[0089] For example, such as Figure 4C As shown, the image depicts a person sitting on a sofa reading a book, with a smart speaker playing music on the table in front of them. The corresponding text search query is "Is there a person in the image?". After the image and text search query are input into the modal data processing model, they are processed sequentially through (visual encoder + text encoder) - multiple feature fusion parts - decoder, which outputs the search result in text form for the text search query data of the image, such as "Someone is here".

[0090] The third scenario - visual question answering task scenario:

[0091] In this scenario, the process of implementing the image description task based on the aforementioned modal data processing model may include: acquiring the image data to be processed and the text question data corresponding to the image data; obtaining the text answer data based on the image data through a preset modal data processing model, wherein the modal data processing model is a model obtained after task transfer based on a pre-trained model that has been trained, such as the model trained by the method described in Example 1.

[0092] For example, such as Figure 4D As shown, the image depicts a person sitting on a sofa reading a book, with a smart speaker playing music on the table in front of them. The corresponding text question is "What is the person in the image doing?". After the image and text question data are input into the modal data processing model, they are processed sequentially through (visual encoder + text encoder) - multiple feature fusion parts - decoder, which outputs the answer text for the text question data of the image, such as "The person in the image is reading a book."

[0093] Fourth scenario - Visual positioning task scenario:

[0094] In this scenario, the process of implementing the image description task based on the aforementioned modal data processing model may include: acquiring image data to be processed and text location request data for locating targets in the image data; obtaining the location information of the target based on the image data through a preset modal data processing model, wherein the location information is used to describe the position of the target in the image data through text; the modal data processing model is a model obtained after task transfer based on a pre-trained model that has been trained, such as the model trained by the method described in Example 1.

[0095] For example, such as Figure 4E As shown, the image depicts a person sitting on a sofa reading a book, with a smart speaker playing music on the table in front of them. The corresponding text location request data is "Where is the smart speaker in the image?". After the image and text location request data are input into the modal data processing model, they are processed sequentially through (visual encoder + text encoder) - multiple feature fusion parts - decoder, which outputs the location information determined by the text location request data for the image, such as "The smart speaker in the image is on the table in front of the sofa, image coordinates XXXXXX".

[0096] It is evident that the pre-trained model after task transfer can be effectively applied to various downstream task scenarios.

[0097] It should be noted that the above scenarios are merely illustrative examples, and the solutions in this application are not limited to the above scenarios. Furthermore, in the above multimodal data processing method and the above scenario examples, some steps are described in a relatively simple manner. Those skilled in the art can implement these steps with reference to the relevant descriptions in Embodiment 1, and will not be repeated here.

[0098] Example 3

[0099] Reference Figure 5 The flowchart illustrates the steps of a multimodal data processing method according to Embodiment 3 of this application.

[0100] In some scenarios, there may also be staff members on the back-end server. Therefore, the solution in this application embodiment can also be implemented by relying on the back-end staff to accurately determine the required modal data processing model when there are multiple modal data processing models.

[0101] Based on this, the multimodal data processing method of this embodiment includes the following steps:

[0102] Step S301: Obtain the modal data to be processed.

[0103] In this embodiment, the modal data to be processed can be multimodal data, such as image data and its corresponding text data. However, it is not limited to this; the solution of this embodiment can also be applied to single-modal data such as image data.

[0104] Step S303: Receive information about the modal data processing model, which is input through the interactive interface, for processing the modal data.

[0105] In this case, the system will provide an interactive interface that displays the modal data to be processed, as well as information on at least one modal data processing model to be used. Users can select the desired modal data processing model by checking boxes or clicking on buttons.

[0106] Step S305: Invoke the modal data processing model indicated by the information to process the modal data and obtain the preset modal task data based on the processing result.

[0107] The modal data processing model is a model obtained by transferring a pre-trained model after training, as described in the aforementioned embodiment one. This modal data processing model includes multiple feature fusion components. These components use text features from the text data in the modal data as a base and image features from the image data in the modal data as an auxiliary, summing and concatenating the image features into the text features to obtain concatenated features. Then, at least using the concatenated features obtained from the feature fusion components, the modal data is processed, and task data for a preset modality is obtained based on the processing results.

[0108] Optionally, the task data of the preset modality may include at least one of the following: text data generated based on the image data to be processed (such as image description task, image text retrieval task, visual question answering task, etc.) and / or image localization data (such as visual localization task, etc.).

[0109] This embodiment utilizes a modal data processing model to process modal data. Because the model's multiple feature fusion components can use text features as a foundation and image features as an aid, it adds and concatenates image features to text features. This ensures that when using text features as a foundation and image features as an aid, image features are neither obscured nor significantly impacted, while allowing text features to carry rich image information, or in other words, image information can supplement text features. Therefore, it effectively avoids the asymmetry of multimodal data, improves the accuracy of multimodal data processing, and achieves more precise and efficient task processing. Furthermore, the interactive interface allows backend staff to filter and control the modal data processing model, increasing the flexibility of the solution.

[0110] The implementation of some steps in this embodiment is relatively simple. For specific implementation, please refer to the description of the relevant parts in the foregoing embodiments.

[0111] Example 4

[0112] Reference Figure 6 The diagram shows a structural schematic of an electronic device according to Embodiment 4 of this application. The specific embodiments of this application do not limit the specific implementation of the electronic device.

[0113] like Figure 6 As shown, the electronic device may include: a processor 402, a communications interface 404, a memory 406, and a communications bus 408.

[0114] in:

[0115] The processor 402, communication interface 404, and memory 406 communicate with each other via communication bus 408.

[0116] Communication interface 404 is used to communicate with other electronic devices or servers.

[0117] The processor 402 is used to execute program 410, specifically to perform the relevant steps in any of the above method embodiments.

[0118] Specifically, program 410 may include program code that includes computer operation instructions.

[0119] Processor 402 may be a CPU, an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application. The smart device includes one or more processors, which may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0120] Memory 406 is used to store program 410. Memory 406 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0121] Program 410 has executable instructions that, when executed, cause processor 402 to perform the operations corresponding to the methods described in any of the foregoing method embodiments.

[0122] The specific implementation of each step in procedure 410 can be found in the corresponding descriptions of the steps and units in the above method embodiments, and has corresponding beneficial effects, which will not be repeated here. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the devices and modules described above can be referred to the corresponding process descriptions in the foregoing method embodiments, and will not be repeated here.

[0123] This application also provides a computer storage medium storing a computer program thereon, which, when executed by a processor, implements any of the methods described in the above-described plurality of method embodiments.

[0124] This application also provides a computer program product, including computer instructions that instruct a computing device to perform an operation corresponding to any of the methods in the above-described multiple method embodiments.

[0125] Furthermore, it should be noted that the user-related information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to sample data used for training the model, data used for analysis, stored data, displayed data, etc.) involved in the embodiments of this application are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0126] It should be noted that, depending on the implementation needs, the various components / steps described in the embodiments of this application can be broken down into more components / steps, or two or more components / steps or parts of the operation of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of this application.

[0127] The methods described in the embodiments of this application can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium (such as a CD-ROM, RAM, floppy disk, hard disk, or magneto-optical disk), or as computer code downloaded over a network that is originally stored in a remote recording medium or a non-transitory machine-readable medium and will be stored in a local recording medium. Thus, the methods described herein can be processed by software stored on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware (such as an ASIC or FPGA). It is understood that the computer, processor, microprocessor controller, or programmable hardware includes storage components (e.g., RAM, ROM, flash memory, etc.) capable of storing or receiving software or computer code that, when accessed and executed by the computer, processor, or hardware, implements the methods described herein. Furthermore, when a general-purpose computer accesses code used to implement the methods shown herein, the execution of the code transforms the general-purpose computer into a dedicated computer for executing the methods shown herein.

[0128] Those skilled in the art will recognize that the units and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this application.

[0129] The above embodiments are only used to illustrate the embodiments of this application, and are not intended to limit the embodiments of this application. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the embodiments of this application. Therefore, all equivalent technical solutions also fall within the scope of the embodiments of this application, and the patent protection scope of the embodiments of this application should be defined by the claims.

Claims

1. A multimodal data processing method, comprising: The modal data to be processed is acquired and input into a preset modal data processing model. The modal data processing model is obtained by task transfer based on a pre-trained model that has been trained. The modal data processing model includes multiple feature fusion parts. The multiple feature fusion parts use the text features of the text data in the modal data as the basis and the image features of the image data in the modal data as the auxiliary, and add and concatenate the image features to the text features to obtain concatenated features. At least through the splicing features obtained by the feature fusion part, the modal data is processed, and task data of the preset modality is obtained based on the processing results; The spliced ​​features obtained at least through the feature fusion portion are used to process the modal data, including: For the features corresponding to the modal data input to the feature fusion part, feature processing is performed, and the processing result is used as the input to the next feature fusion part until the spliced ​​features output by the last feature fusion part are obtained; Based on the spliced ​​features output from the last feature fusion part, the modal data is processed.

2. The method according to claim 1, wherein, The feature fusion component includes a multimodal feature summation layer and a multimodal feature splicing layer; The multimodal feature summing layer is used to sum the image features corresponding to the image data and the text features corresponding to the text data, and to extract text features based on the summed features; The multimodal feature splicing layer is used to splice the text feature extraction results and the image features to obtain spliced ​​features.

3. The method according to claim 2, wherein, When the modal data to be processed includes image data to be processed and text data corresponding to the image data to be processed, feature processing is performed on the features corresponding to the modal data input to the feature fusion part, and the processing result is used as the input of the next feature fusion part until the concatenated features output by the last feature fusion part are obtained, including: For a certain feature fusion part among the multiple feature fusion parts, the image features and text features input to the feature fusion part are received, wherein the image features received by the first feature fusion part are the image features corresponding to the image data to be processed, and the text features received are the text features corresponding to the text data to be processed; Through the multimodal feature summing layer of this feature fusion part, the image features are used to sum the text features, and text features are extracted based on the summed features to obtain the summed text features; The multimodal feature splicing layer of the feature fusion part splices the summed text features and the image features, and uses the splicing result as the input of the next feature fusion part until the spliced ​​features output by the last feature fusion part are obtained.

4. The method according to claim 2, wherein, When the modal data to be processed is image data to be processed, feature processing is performed on the features corresponding to the modal data input to the feature fusion part, and the processing result is used as the input of the next feature fusion part until the spliced ​​features output by the last feature fusion part are obtained, including: For a specific feature fusion part among the multiple feature fusion parts, image features input to that feature fusion part are received, wherein the image features received by the first feature fusion part are the image features corresponding to the image data to be processed; The image features are processed by the multimodal feature stitching layer of the feature fusion part, and the processing result is used as the input of the next feature fusion part until the stitched features output by the last feature fusion part are obtained.

5. The method according to claim 3, wherein, The modal data processing model further includes: a visual encoder and a text encoder connected before the multiple feature fusion components; The visual encoder is used to encode the image data to be processed to generate initial image features for inputting the first feature fusion part; The text encoder is used to encode the text data to be processed to generate initial text features for inputting the first feature fusion part.

6. The method according to claim 5, wherein, The modal data processing model further includes a decoder connected after the multiple feature fusion components; The decoder is used to decode and generate task data of a preset modality based on the spliced ​​features output by the multiple feature fusion parts.

7. The method according to any one of claims 1-4, wherein, When the modal data to be processed includes at least the image data to be processed, the task data of the preset modality includes at least one of the following: text data and / or image positioning data generated based on the image data to be processed.

8. A pre-training method for a pre-trained model, wherein, The pre-trained model includes a visual encoder, a text encoder, multiple feature fusion components, and a decoder; the method includes: Obtain training samples for pre-training the pre-trained model, the training samples including image sample data and text sample data corresponding to the image sample data; The image sample data is encoded by the visual encoder to obtain image sample features, and the text sample features are encoded by the text encoder to obtain text sample features. The image sample features and the text sample features are input into the multiple feature fusion part, so that the multiple feature fusion part, based on the text sample features and with the image sample features as an aid, adds and concatenates the image sample features into the text sample features to obtain concatenated features; The spliced ​​features are input into the decoder to predict and generate modal sample data of a preset modality; Based on the modal sample data and a preset loss function, a multimodal data processing model is pre-trained. The multimodal data processing model includes multiple feature fusion parts. These feature fusion parts are used to perform feature processing on the features corresponding to the modal data input to the feature fusion parts, and use the processing results as the input to the next feature fusion part, until the concatenated features output by the last feature fusion part are obtained. Based on the concatenated features output by the last feature fusion part, the modal data is processed.

9. A multimodal data processing method, comprising: Acquire the modal data to be processed; Receive information about a modal data processing model, input through an interactive interface, for processing the modal data; The modal data processing model indicated by the information is invoked to process the modal data, and task data of a preset modality is obtained based on the processing result. The modal data processing model is a model obtained after task transfer based on a pre-trained model that has been trained. The pre-trained model is trained by the method described in claim 8. The multimodal data processing model includes multiple feature fusion parts. The multiple feature fusion parts are used to perform feature processing on the features corresponding to the modal data input to the feature fusion parts, and use the processing result as the input of the next feature fusion part until the spliced ​​feature output of the last feature fusion part is obtained. Based on the spliced ​​feature output of the last feature fusion part, the modal data is processed.

10. A multimodal data processing method, comprising: Acquire the image data to be processed; The image data is described by obtaining descriptive text data corresponding to the image data through a preset modal data processing model. The descriptive text data is used to describe the content of the image data. The modal data processing model is a model obtained by task transfer based on a pre-trained model that has been trained. The pre-trained model is trained by the method described in claim 8. The multimodal data processing model includes multiple feature fusion parts. The multiple feature fusion parts are used to perform feature processing on the features corresponding to the modal data input to the feature fusion parts, and use the processing result as the input of the next feature fusion part, until the spliced ​​feature output of the last feature fusion part is obtained. Based on the spliced ​​feature output of the last feature fusion part, the modal data is processed.

11. A multimodal data processing method, comprising: Acquire the image data to be processed and the text retrieval requirements data for searching the content in the image data; The multimodal data processing model retrieves the search results corresponding to the text search requirements from the image data using a pre-defined modal data processing model. The search results are used to retrieve image content results from the image data using text descriptions. The modal data processing model is a model obtained after task transfer based on a pre-trained model that has been trained. The pre-trained model is trained using the method described in claim 8. The multimodal data processing model includes multiple feature fusion parts. The multiple feature fusion parts are used to perform feature processing on the features corresponding to the modal data input to the feature fusion parts, and use the processing results as the input to the next feature fusion part, until the spliced ​​features output by the last feature fusion part are obtained. Based on the spliced ​​features output by the last feature fusion part, the modal data is processed.

12. A multimodal data processing method, comprising: Obtain the image data to be processed and the corresponding text question data of the image data; Text answer data based on the image data is obtained through a preset modal data processing model. The modal data processing model is a model obtained after task transfer based on a pre-trained model that has been trained. The pre-trained model is trained by the method described in claim 8. The multimodal data processing model includes multiple feature fusion parts. The multiple feature fusion parts are used to perform feature processing on the features corresponding to the modal data input to the feature fusion parts, and use the processing result as the input of the next feature fusion part, until the spliced ​​feature output of the last feature fusion part is obtained. Based on the spliced ​​feature output of the last feature fusion part, the modal data is processed.

13. A multimodal data processing method, comprising: Acquire the image data to be processed and the text location request data for locating the target in the image data; The target's location information is obtained based on the image data using a pre-defined modal data processing model. This location information is used to describe the target's position in the image data via text. The modal data processing model is a model obtained after task transfer based on a pre-trained model that has been trained. The pre-trained model is trained using the method described in claim 8. The multimodal data processing model includes multiple feature fusion parts. These multiple feature fusion parts are used to perform feature processing on the features corresponding to the modal data input to the feature fusion parts, and use the processing result as the input to the next feature fusion part, until the spliced ​​features output by the last feature fusion part are obtained. Based on the spliced ​​features output by the last feature fusion part, the modal data is processed.

14. An electronic device comprising: The processor, memory, communication interface, and communication bus are provided, wherein the processor, memory, and communication interface communicate with each other via the communication bus. The memory is used to store at least one executable instruction that causes the processor to perform the operation corresponding to the method as described in any one of claims 1-13.

Images