Method, apparatus, medium and program product for processing multi-modal data

Abstract

The invention provides a method and device for processing multi-modal data, electronic equipment, a computer readable medium and a computer program product. The method comprises the steps of performing word representation processing on text content of to-be-processed data to obtain a text feature vector; if the to-be-processed data contains a predetermined number of pictures, extracting picture features of each picture and mapping the picture features to a text feature space to obtain a picture feature vector, and if the number of the pictures in the to-be-processed data is smaller than the predetermined number, mapping the picture features to the text feature space to obtain a picture feature vector, and if the number of the pictures in the to-be-processed data is smaller than the predetermined number, mapping the picture features to the text feature space; performing zero filling on a mapping result according to a preset input sequence length; carrying out feature fusion on the text feature vector and the picture feature vector to obtain a multi-modal fusion feature, and taking the multi-modal fusion feature as the input of a multi-modal model; and training the multi-modal model to output a prediction result of a predetermined prediction task in the multi-modal model based on the input multi-modal fusion features. According to the method and the device, plain text data and data containing texts and pictures are simultaneously processed by using one model, so that computing resources are saved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device, electronic device, computer-readable medium, and computer program product for processing multimodal data. Background Art

[0002] In existing social networking platforms, dynamic sharing has become an important tool for users to exchange information in real time. Users can publish dynamic content containing text and pictures, and enjoy interactive services such as forwarding, commenting, and liking. Dynamic content is mainly divided into two categories: dynamic content with pictures and dynamic content with pure text. Dynamic content with pictures not only contains text information, but also can be attached with a predetermined number of pictures, while dynamic content with pure text consists only of text and does not contain any pictures.

[0003] Given the differences in content features between dynamics containing images and pure text, existing technologies usually use different models to process data with different features when conducting machine review of dynamics and other similar data. For example, in the scenario of reviewing dynamics for violations, two different risk classification models are used to identify illegal content. One is a multimodal risk classification model, which is specially designed to identify illegal information in dynamics containing images; the other is a text risk classification model, which is used to detect whether pure text dynamics contain illegal content.

[0004] However, this separate model training method has some obvious shortcomings. Because different types of dynamic content need to be processed separately, two models with similar structure need to be trained and deployed, resulting in duplicate investment in machine review resources and double consumption of human resources and time costs in subsequent model iteration deployment. Summary of the invention

[0005] Multiple aspects of the present application provide a method, an apparatus, an electronic device, a computer-readable medium, and a computer program product for processing multimodal data.

[0006] In one aspect of the present application, a method for processing multimodal data is provided, wherein the method comprises:

[0007] The text content of the data to be processed is processed by word representation to obtain a text feature vector;

[0008] If the data to be processed contains a predetermined number of pictures, the picture feature vector is obtained by extracting the picture features of each picture and mapping the picture features to the text feature space; if the number of pictures in the data to be processed is less than the predetermined number, after mapping the picture features to the text feature space, the mapping result is zero-filled according to the preset input sequence length;

[0009] The multimodal fusion feature obtained by performing feature fusion on the text feature vector and the image feature vector is used as the input of the multimodal model;

[0010] The multimodal model is trained to output a prediction result of a predetermined prediction task based on the input multimodal fusion features in the multimodal model.

[0011] In one aspect of the present application, a device for processing multimodal data is provided, wherein the device comprises:

[0012] A device for obtaining a text feature vector by performing word representation processing on the text content of the data to be processed;

[0013] A device for obtaining a picture feature vector by extracting picture features of each picture and mapping the picture features to a text feature space if the data to be processed contains a predetermined number of pictures, and for performing zero padding on the mapping result according to a preset input sequence length after mapping the picture features to the text feature space if the number of pictures in the data to be processed is less than the predetermined number;

[0014] A device for using the multimodal fusion feature obtained by performing feature fusion of the text feature vector and the image feature vector as an input of the multimodal model;

[0015] A device for training the multimodal model to output a prediction result of a predetermined prediction task based on the input multimodal fusion features in the multimodal model.

[0016] In another aspect of the present application, a computer-readable storage medium is provided, on which computer program instructions are stored. The computer program instructions can be executed by a processor to implement the method of the embodiment of the present application.

[0017] Another aspect of the present application provides a computer program product, including a computer program, which implements the method of the embodiment of the present application when executed by a processor.

[0018] In the solution provided in the embodiment of the present application, the text features and image features corresponding to the data to be processed are used as two input tokens of a multimodal model based on BERT, and the text features and image features are concatenated and fused in the multimodal model, and then the prediction results of the predetermined prediction task are output based on the fused features. This achieves the use of one model to simultaneously process pure text data and data containing text and images, without the need to train different models separately to process the two types of data as in the traditional way, thereby saving computing resources and the manpower cost of subsequent iterative optimization of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0020] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0021] Figure 1 A schematic diagram of a process for processing multimodal data provided by an embodiment of the present application is shown;

[0022] Figure 2 A schematic diagram of an exemplary model structure according to an embodiment of the present application is shown;

[0023] Figure 3 A schematic diagram of the structure of a device for processing multimodal data provided by an embodiment of the present application is shown;

[0024] Figure 4 A schematic diagram of the structure of a device suitable for implementing the solution in the embodiment of the present application is shown.

[0025] The same or similar reference numerals in the drawings represent the same or similar components. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0027] In a typical configuration of the present application, the terminal and the equipment of the service network each include one or more processors (CPU), input / output interface, network interface and memory.

[0028] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0029] Computer readable media include permanent and non-permanent, removable and non-removable media, and can be implemented by any method or technology to store information. Information can be computer program instructions, data structures, modules of programs or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0030] Figure 1 A schematic flow chart of a method for processing multimodal data provided in an embodiment of the present application is shown. The method at least includes step S101, step S102, step S103 and step S104.

[0031] In actual scenarios, the execution subject of the method can be a network device, or an application running on a network device, wherein the network device includes but is not limited to a network host, a single network server, a plurality of network server sets, or a collection of computers based on cloud computing, and can be used to implement some processing functions when setting an alarm. Here, the cloud is composed of a large number of hosts or network servers based on cloud computing, wherein cloud computing is a type of distributed computing, a virtual computer composed of a group of loosely coupled computers.

[0032] The concepts involved in the embodiments of the present application are explained below.

[0033] BERT (Bidirectional Encoder Representations from Transformers) model: BERT is a bidirectional encoding representation method based on the Transformer unit, and the Transformer unit is an encoding unit based on the self-attention mechanism.

[0034] Token: The smallest unit in the text. In the BERT model, the text is first divided into tokens. These tokens can be words, phrases, or characters, depending on how the model is trained. The BERT model has a fixed vocabulary that contains all the tokens that the model can recognize. Each token is mapped to a vector space of fixed dimension, a process called word embedding. The BERT model uses word embedding to convert text information into a numerical form that the model can understand. Each token has a unique index in the vocabulary. The BERT model uses some special tokens to mark the beginning and end of sentences, such as [CLS] and [SEP]. [CLS] is usually used for aggregate representation of classification tasks, while [SEP] is used to separate sentences or paragraphs.

[0035] The multimodal model according to the embodiment of the present application is a BERT model. In some embodiments, the BERT model according to the embodiment of the present application learns the deep features of the language through pre-training tasks (such as Masked Language Model, MLM, and Next Sentence Prediction, NSP). In the MLM task, the model needs to predict tokens that are randomly masked in the sentence. The pre-trained BERT model can be used for various downstream tasks, usually by fine-tuning on a specific task.

[0036] In some embodiments, the multimodal model according to this embodiment is a risk sub-model, which is used to predict risk violations of input data to be processed. The model can simultaneously identify illegal data to be processed containing images and texts and illegal plain text data to be processed, without the need to train a multimodal risk sub-model and a text risk sub-model to identify illegal image-containing data and illegal plain text data respectively as in the traditional way.

[0037] Reference Figure 1 In step S101, the text content of the data to be processed is processed by word representation to obtain a text feature vector.

[0038] The data to be processed includes various data that may contain text and pictures at the same time.

[0039] Optionally, the data to be processed is dynamic. Dynamic can be a social tool for video websites to share users' real-time information through a follow mechanism, providing services such as publishing dynamic content, displaying dynamics of followed users, forwarding dynamics, commenting on dynamics, and liking. Dynamics are mainly divided into two categories: dynamics with pictures and dynamics with pure text. Dynamics with pictures refer to dynamic types that are accompanied by a predetermined number of pictures in addition to text content; and dynamics with pure text refer to dynamic types that only have text content and no pictures.

[0040] Specifically, the text content is processed by the embedding layer of the BERT model to obtain the corresponding text feature vector. Among them, through token embedding, each word or token in the vocabulary is mapped to a vector of a fixed size.

[0041] In step S102, if the data to be processed contains a predetermined number of pictures, a picture feature vector is obtained by extracting picture features of each picture and mapping the picture features to a text feature space; if the number of pictures in the data to be processed is less than a predetermined number, after mapping the picture features to the text feature space, the mapping result is zero-filled according to a preset input sequence length.

[0042] Specifically, the step S102 includes step S1021, step S1022 and step S1023.

[0043] In step S1021, it is determined whether the data to be processed includes a predetermined number of pictures.

[0044] Among them, the methods of determining whether the data to be processed contains images include but are not limited to content inspection, file format analysis, media type detection, encoding analysis or visual recognition.

[0045] For example, for data to be processed in HTML format, by checking whether the HTML code contains Tags or image links Tag to determine whether to include images.

[0046] In step S1022, if the data to be processed contains a predetermined number of pictures, the image features of each picture are extracted through a pre-trained convolutional neural network model, and mapped to the text feature space through a fully connected layer to obtain the image feature vector of the multimodal model.

[0047] Optionally, if the data to be processed contains a predetermined number of pictures, the image features of each picture are extracted through a pre-trained ResNet50 model, and mapped to the text feature space through a fully connected layer (MLP) to obtain the image feature vector of the multimodal model.

[0048] Among them, after mapping to the semantic space of the text, the feature vector of the image is converted into an image feature vector with the same dimension and semantic alignment as the text feature.

[0049] Among them, those skilled in the art should be familiar with that there are many other ways to extract image features, for example, using one or more convolutional layers, recurrent neural networks, Transformer and other pre-trained neural network models to extract image features, etc. Those skilled in the art can choose a suitable image feature extraction method based on actual needs, which will not be repeated here.

[0050] In step S1023, if the number of pictures in the data to be processed is less than the predetermined number, the picture features of each picture are extracted through the pre-trained convolutional neural network model, and mapped to the text feature space through the fully connected layer, and the mapping result is filled based on the zero filling technology. By filling the zero vector, a vector with the same length as the preset input sequence length is obtained as the picture feature vector.

[0051] The padding process ensures that the lengths of the feature sequences input to the model are the same by adding vectors.

[0052] The zero padding is a technique for processing sequences of different lengths. When processing data, the BERT model needs to unify the length of all input sequences to a fixed maximum length. This is done to allow the model to use fixed-size inputs for batch processing. For sequences whose length is less than the maximum length, the BERT model will add a zero vector (i.e., a vector of all zeros) to the end until the sequence length reaches the set maximum length. The padded zero vectors will not affect the hidden state of the model because they do not contain any useful information. The BERT model ignores these padded zero vectors through a mask mechanism. During the BERT model training process, the zero padding part is marked with a mask to avoid affecting the model training effect.

[0053] In step S103, the multimodal fusion feature obtained by performing feature fusion on the text feature vector and the image feature vector is used as the input of the multimodal model.

[0054] Specifically, the text feature vector obtained in step S101 and the image feature vector obtained in step S102 are concatenated and fused to obtain corresponding multimodal fusion features.

[0055] The multimodal fusion features are used as input embedding of the BERT model.

[0056] The splicing and fusion includes a splicing operation and a feature fusion operation. The splicing operation splices the text feature vector and the image feature vector together along a specific dimension.

[0057] Optionally, after the splicing and fusion, segment representation (Segment Embedding) processing and position representation (Position Embedding) processing are performed on the text feature vector and the image feature vector respectively.

[0058] The segment representation process is used to distinguish different parts in the text. The position representation process is used to help the model understand the order information of words in the text.

[0059] In step S104, the multimodal model is trained to output a prediction result of a predetermined prediction task based on the input multimodal fusion features in the multimodal model.

[0060] According to one embodiment, the multimodal model is a risk sub-model, and in step S104, a risk prediction result is output based on the input multimodal fusion feature, and the risk prediction result includes a violation determination result of the data to be processed.

[0061] Optionally, the risk prediction result may also include other information, such as a risk score for assessing the possibility of a violation.

[0062] According to an embodiment of the present application, the obtained text feature vector is used as the first input sequence, the obtained image feature vector is used as the second input sequence, and the first input sequence and the second input sequence are used as two inputs of the next sentence prediction task (Next Sentence Prediction, NSP) in the BERT-based multimodal model to train the BERT-based multimodal model.

[0063] For example, the text feature vector obtained in step S101 is used as token a, the image feature vector obtained in step S102 is used as token b, and token a and token b are used as two inputs of the NSP task in the BERT model to train a BERT-based multimodal model.

[0064] Typically, in the NSP task, the model needs to determine whether the given second sentence is the direct continuation of the first sentence. This task helps the model capture the coherence of the text, thereby better understanding the relationship between sentences in downstream tasks such as question answering systems and natural language inference. In practical applications, the NSP task takes two sentences as input and separates them with specific tokens, such as [CLS] to mark the beginning of a sentence and [SEP] to mark the end of a sentence. The model learns how to determine whether they are coherent based on the representation of these sentences.

[0065] In the scenario where the multimodal model is a risk classification model, the purpose of the NSP task according to the embodiment of the present application is to allow the multimodal model to learn the correlation between images and texts based on the first input sequence and the second input sequence.

[0066] According to the method of the embodiment of the present application, by using the text features and image features corresponding to the data to be processed as two input tokens of a multimodal model based on BERT, and concatenating and fusion the text features and image features in the multimodal model, and then outputting the prediction results of the predetermined prediction task based on the fused features, it is possible to use one model to simultaneously process pure text data and data containing text and images, without the need to train different models separately to process the two types of data as in the traditional way, thereby saving computing resources and saving the manpower cost of subsequent iterative optimization of the model.

[0067] Refer to the following Figure 2 The exemplary model structure shown is used to illustrate the method of the embodiment of the present application.

[0068] Reference Figure 2 The model in this example is a multimodal risk classification model based on the BERT model. This model is designed for complex situations in which the dynamic machine review business includes both plain text dynamic machine review and dynamic machine review containing images. It implements the function of using one model to simultaneously identify illegal dynamics containing images and illegal plain text dynamics. The data to be processed in this example is the dynamics for publishing on the video website, which includes two types: plain text dynamics and dynamics containing images. The dynamics in this example can contain up to 18 images.

[0069] Reference Figure 2 The BERT model in this example mainly includes three parts: feature input, model backbone, and risk score output. These three parts are introduced below.

[0070] 1) Feature input: The input features of a dynamic response containing 18 images are the dynamic content and the image features of each image. The input features of a pure text dynamic response or a dynamic response containing less than 18 images are the dynamic content and the image vector supplemented by zero padding.

[0071] Specifically, this example draws on the idea of ​​Next Sentence Prediction of the BERT model. For dynamics containing pictures, the ResNet50 model is used to extract the image features corresponding to the pictures in the dynamics and map them to the text feature space through the fully connected layer. The dynamic text content is used as the input token A of the BERT model, and the extracted image features are used as the next sentence, that is, the input token B of the BERT model. For dynamics with less than 18 pictures (including pure text dynamics without pictures), after the ResNet50 model is used to extract the image features corresponding to the pictures in the dynamics and map them to the text feature space through the fully connected layer, zero padding is used to fill the image vector of the same length as the input token B, and mask is used as the position marker of zero padding. Among them, the input form of token A is expressed as [CLS]+dynamic content+[SEP], and the input form of token B is expressed as image vector+[SEP].

[0072] 2) Model backbone:

[0073] For the input features, the image features are mapped to the feature vector of the text semantic space through the multi-layer perceptron and then concatenated with the word representation of the text content by the embedding layer of the Bert model. After that, the concatenated features are respectively represented by segments and positions to obtain the input embedding vector of the BERT model after the fusion of the two modal features of image and text.

[0074] The BERT model structure of this example also includes: multi-head attention, residual connection & LayerNorm, feedforward network, residual connection (Add) & LayerNorm, and classification head. The input embedding vector will be input into these structures for processing.

[0075] Among them, the Multi-Head Attention mechanism uses the self-attention mechanism to make each token take into account the information of the entire sequence. Multi-Head Attention allows the model to learn information in parallel in different representation subspaces. The Residual Connection adds the output of each sublayer to its input to form a residual connection to alleviate the gradient vanishing problem in deep networks. LayerNorm is used for layer normalization. Layer normalization is performed after the residual connection to reduce the difference in data distribution between layers and accelerate the training process. The Feed-Forward Neural Network is used after the attention mechanism, and the model sends data to a feed-forward neural network for further nonlinear transformation. The Classification Head is used to obtain prediction results for specific tasks for specific tasks.

[0076] 3) Risk score output;

[0077] Through the classification head structure, the final binary classification prediction result is output. In this example, the task is to determine whether the dynamics violates the rules, and the prediction result is the determination result of whether the dynamics violates the rules or not.

[0078] The method in this example trains a Bert-based multimodal risk score model and adopts the ideas of zero padding and masking to achieve the function of a model to simultaneously identify illegal dynamics containing images and illegal dynamics containing only text, thus avoiding the waste of machine review resources and saving the manpower cost of subsequent iterative optimization of the model.

[0079] In addition, the present application also provides a device for processing multimodal data, the structure of which is as follows: Figure 3 As shown. The device includes: a device for obtaining a text feature vector by performing word representation processing on the text content of the data to be processed (hereinafter referred to as "text feature acquisition device 101"), a device for obtaining a picture feature vector by extracting picture features of each picture and mapping the picture features to the text feature space if the data to be processed contains a predetermined number of pictures, and a device for zero-filling the mapping result according to a preset input sequence length after mapping the picture features to the text feature space if the number of pictures in the data to be processed is less than the predetermined number (hereinafter referred to as "picture feature acquisition device 102"), a device for fusing the text feature vector and the picture feature vector to obtain a multimodal fusion feature as an input of a multimodal model (hereinafter referred to as "feature fusion device 103"), and a device for training the multimodal model to output a prediction result of a predetermined prediction task in the multimodal model based on the input multimodal fusion feature (hereinafter referred to as "model training device 104").

[0080] Reference Figure 3 The text feature acquisition device 101 obtains a text feature vector by performing word representation processing on the text content of the data to be processed.

[0081] The data to be processed includes various data that may contain text and pictures at the same time.

[0082] Optionally, the data to be processed is dynamic. Dynamic can be a social tool for video websites to share users' real-time information through a follow mechanism, providing services such as publishing dynamic content, displaying dynamics of followed users, forwarding dynamics, commenting on dynamics, and liking. Dynamics are mainly divided into two categories: dynamics with pictures and dynamics with pure text. Dynamics with pictures refer to dynamic types that are accompanied by a predetermined number of pictures in addition to text content; and dynamics with pure text refer to dynamic types that only have text content and no pictures.

[0083] Specifically, the text content is processed by the embedding layer of the BERT model to obtain the corresponding text feature vector. Among them, through token embedding, each word or token in the vocabulary is mapped to a vector of a fixed size.

[0084] If the data to be processed contains a predetermined number of pictures, the picture feature acquisition device 102 obtains a picture feature vector by extracting the picture features of each picture and mapping the picture features to the text feature space. If the number of pictures in the data to be processed is less than the predetermined number, the picture feature acquisition device 102 maps the picture features to the text feature space and fills the mapping result with zeros according to the preset input sequence length.

[0085] Specifically, the picture feature acquisition device 102 determines whether the data to be processed contains a predetermined number of pictures.

[0086] Among them, the methods of determining whether the data to be processed contains images include but are not limited to content inspection, file format analysis, media type detection, encoding analysis or visual recognition.

[0087] For example, for data to be processed in HTML format, by checking whether the HTML code contains Tags or image links Tag to determine whether to include images.

[0088] If the data to be processed contains a predetermined number of pictures, the picture feature acquisition device 102 extracts the picture features of each picture through a pre-trained convolutional neural network model, and maps them to the text feature space through a fully connected layer to obtain the picture feature vector of the multimodal model.

[0089] Optionally, if the data to be processed contains a predetermined number of pictures, the picture feature acquisition device 102 extracts the picture features of each picture through a pre-trained ResNet50 model, and maps them to the text feature space through a fully connected layer (MLP) to obtain the picture feature vector of the multimodal model.

[0090] Among them, after mapping to the semantic space of the text, the feature vector of the image is converted into an image feature vector with the same dimension and semantic alignment as the text feature.

[0091] Among them, those skilled in the art should be familiar with that there are many other ways to extract image features, for example, using one or more convolutional layers, recurrent neural networks, Transformer and other pre-trained neural network models to extract image features, etc. Those skilled in the art can choose a suitable image feature extraction method based on actual needs, which will not be repeated here.

[0092] If the number of pictures in the data to be processed is less than a predetermined number, the picture feature acquisition device 102 extracts the picture features of each picture through a pre-trained convolutional neural network model, and maps them to the text feature space through a fully connected layer, and fills the mapping result based on the zero filling technology, and obtains a vector with the same length as the preset input sequence length by filling the zero vector, as the picture feature vector.

[0093] The padding process ensures that the lengths of the feature sequences input to the model are the same by adding vectors.

[0094] The feature fusion device 103 uses the multimodal fusion feature obtained by feature fusion of the text feature vector and the image feature vector as the input of the multimodal model.

[0095] Specifically, the feature fusion device 103 concatenates and fuses the text feature vector obtained by the text feature acquisition device 101 and the picture feature vector obtained by the picture feature acquisition device 102 to obtain corresponding multimodal fusion features.

[0096] The multimodal fusion features are used as input embedding of the BERT model.

[0097] The splicing and fusion includes a splicing operation and a feature fusion operation. The splicing operation splices the text feature vector and the image feature vector together along a specific dimension.

[0098] Optionally, after the splicing and fusion, segment representation (Segment Embedding) processing and position representation (Position Embedding) processing are performed on the text feature vector and the image feature vector respectively.

[0099] The segment representation process is used to distinguish different parts in the text. The position representation process is used to help the model understand the order information of words in the text.

[0100] The model training device 104 trains the multimodal model to output a prediction result of a predetermined prediction task based on the input multimodal fusion features in the multimodal model.

[0101] According to one embodiment, the multimodal model is a risk sub-model, and the model training device 104 outputs a risk prediction result based on the input multimodal fusion features, and the risk prediction result includes a violation determination result of the data to be processed.

[0102] Optionally, the risk prediction result may also include other information, such as a risk score for assessing the possibility of a violation.

[0103] The obtained text feature vector is used as the first input sequence, the obtained image feature vector is used as the second input sequence, and the first input sequence and the second input sequence are used as two inputs of the next sentence prediction task (NSP) in the BERT-based multimodal model to train the BERT-based multimodal model.

[0104] For example, the text feature vector obtained by the text feature acquisition device 101 is used as token a, the image feature vector obtained by the image feature acquisition device 102 is used as token b, and token a and token b are used as two inputs of the NSP task in the BERT model to train a BERT-based multimodal model.

[0105] Typically, in the NSP task, the model needs to determine whether the given second sentence is the direct continuation of the first sentence. This task helps the model capture the coherence of the text, thereby better understanding the relationship between sentences in downstream tasks such as question answering systems and natural language inference. In practical applications, the NSP task takes two sentences as input and separates them with specific tokens, such as [CLS] to mark the beginning of a sentence and [SEP] to mark the end of a sentence. The model learns how to determine whether they are coherent based on the representation of these sentences.

[0106] In the scenario where the multimodal model is a risk classification model, the purpose of the NSP task according to the embodiment of the present application is to allow the multimodal model to learn the correlation between images and texts based on the first input sequence and the second input sequence.

[0107] According to the device of the embodiment of the present application, by using the text features and image features corresponding to the data to be processed as two input tokens of a multimodal model based on BERT, and concatenating and fusion the text features and image features in the multimodal model, and then outputting the prediction results of the predetermined prediction task based on the fused features, it is possible to use one model to simultaneously process pure text data and data containing text and images, without the need to train different models separately to process the two types of data as in the traditional way, thereby saving computing resources and saving the manpower cost of subsequent iterative optimization of the model.

[0108] Refer to the following Figure 2 The exemplary model structure shown is used to illustrate the device of the embodiment of the present application.

[0109] Reference Figure 2The model in this example is a multimodal risk classification model based on the BERT model. This model is designed for complex situations in which the dynamic machine review business includes both plain text dynamic machine review and dynamic machine review containing images. It implements the function of using one model to simultaneously identify illegal dynamics containing images and illegal plain text dynamics. The data to be processed in this example is the dynamics for publishing on the video website, which includes two types: plain text dynamics and dynamics containing images. The dynamics in this example can contain up to 18 images.

[0110] Reference Figure 2 The BERT model in this example mainly includes three parts: feature input, model backbone, and risk score output. These three parts are introduced below.

[0111] 1) Feature input: The input features of a dynamic response containing 18 images are the dynamic content and the image features of each image. The input features of a pure text dynamic response or a dynamic response containing less than 18 images are the dynamic content and the image vector supplemented by zero padding.

[0112] Specifically, this example draws on the idea of ​​Next Sentence Prediction of the BERT model. For dynamic images that contain pictures, the image feature acquisition device 102 uses the ResNet50 model to accurately extract the image features corresponding to the pictures contained in the dynamic image. The text feature acquisition device 101 uses the dynamic text content as the input token A of the BERT model, and uses the extracted image features as the next sentence, that is, the input token B of the BERT model. For dynamic images with less than 18 pictures (including pure text dynamic images that do not contain pictures), after using the ResNet50 model to extract the image features corresponding to the pictures contained in the dynamic image and mapping them to the text feature space through the fully connected layer, zero padding is used to supplement the image vector of the same length as the input token B, and mask is used as the position marker of zero padding. Among them, the input form of token A is expressed as [CLS]+dynamic content+[SEP], and the input form of token B is expressed as image vector+[SEP]

[0113] 2) Model backbone:

[0114] For the input features, the feature fusion device 103 maps the image features to the feature vector of the text semantic space through the multi-layer perceptron and concatenates and fuses the result of the word representation of the text content by the embedding layer of the Bert model. After that, the concatenated features are respectively represented by segments and positions to obtain the input embedding vector of the BERT model after the fusion of the two modal features of image and text.

[0115] The BERT model structure of this example also includes: multi-head attention, residual connection & LayerNorm, feedforward network, residual connection (Add) & LayerNorm, and classification head. The input embedding vector will be input into these structures for processing.

[0116] Among them, the Multi-Head Attention mechanism uses the self-attention mechanism to make each token take into account the information of the entire sequence. Multi-Head Attention allows the model to learn information in parallel in different representation subspaces. The Residual Connection adds the output of each sublayer to its input to form a residual connection to alleviate the gradient vanishing problem in deep networks. LayerNorm is used for layer normalization. Layer normalization is performed after the residual connection to reduce the difference in data distribution between layers and accelerate the training process. The Feed-Forward Neural Network is used after the attention mechanism, and the model sends data to a feed-forward neural network for further nonlinear transformation. The Classification Head is used to obtain prediction results for specific tasks for specific tasks.

[0117] 3) Risk score output;

[0118] Through the classification head structure, the final binary classification prediction result is output. In this example, the task is to determine whether the dynamics violates the rules, and the prediction result is the determination result of whether the dynamics violates the rules or not.

[0119] The device in this example trains a Bert-based multimodal risk classification model and adopts the ideas of zero padding and masking to achieve the function of a model that can simultaneously identify illegal dynamic images and illegal pure text dynamics, thereby avoiding the waste of machine review resources and saving the manpower cost of subsequent iterative optimization of the model.

[0120] Based on the same inventive concept, an electronic device is also provided in an embodiment of the present application, and the method corresponding to the electronic device may be the method for processing multimodal data in the aforementioned embodiment, and its principle of solving the problem is similar to that of the method. The electronic device provided in an embodiment of the present application includes: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the methods and / or technical solutions of the aforementioned multiple embodiments of the present application.

[0121] The electronic device may be a user device, or a device formed by integrating a user device and a network device through a network, or may be an application running on the above device. The user device includes but is not limited to various terminal devices such as computers, mobile phones, tablet computers, smart watches, and bracelets. The network device includes but is not limited to network hosts, single network servers, multiple network server sets, or cloud computing-based computer sets, which can be used to implement some processing functions when setting an alarm. Here, the cloud is composed of a large number of hosts or network servers based on cloud computing, where cloud computing is a type of distributed computing, a virtual computer composed of a group of loosely coupled computer sets.

[0122] Figure 4 The structure of a device suitable for implementing the method and / or technical solution in the embodiment of the present application is shown, and the device 1200 includes a central processing unit (CPU, Central Processing Unit) 1201, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM, Read Only Memory) 1202 or the program loaded from the storage part 1208 to the random access memory (RAM, Random Access Memory) 1203. In RAM1203, various programs and data required for system operation are also stored. CPU 1201, ROM 1202 and RAM 1203 are connected to each other through bus 1204. Input / output (I / O, Input / Output) interface 1205 is also connected to bus 1204.

[0123] The following components are connected to the I / O interface 1205: an input section 1206 including a keyboard, a mouse, a touch screen, a microphone, an infrared sensor, etc.; an output section 1207 including a cathode ray tube (CRT), a liquid crystal display (LCD), an LED display, an OLED display, etc., and a speaker, etc.; a storage section 1208 including one or more computer-readable media such as a hard disk, an optical disk, a magnetic disk, a semiconductor memory, etc.; and a communication section 1209 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 1209 performs communication processing via a network such as the Internet.

[0124] In particular, the methods and / or embodiments in the embodiments of the present application may be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes program code for executing the method shown in the flowchart. When the computer program is executed by the central processing unit (CPU) 1201, the above functions defined in the method of the present application are executed.

[0125] Another embodiment of the present application further provides a computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions can be executed by a processor to implement the methods and / or technical solutions of any one or more embodiments of the present application described above.

[0126] Specifically, the present embodiment may adopt any combination of one or more computer-readable media. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device, or device.

[0127] Computer readable signal media may include a data signal propagated in baseband or as part of a carrier wave, which carries a computer readable program code. Such propagated data signals may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than a computer readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0128] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0129] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0130] The flow chart or block diagram in the accompanying drawings shows the possible architecture, function and operation of the equipment, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated system for hardware that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0131] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0132] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or page components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0133] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0134] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0135] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform some steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program codes.

[0136] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

[0137] In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in a device claim can also be implemented by one unit or device through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any particular order.

Claims

1. A method for processing multimodal data, wherein: The method comprises: The text content of the data to be processed is processed by word representation to obtain a text feature vector; If the data to be processed contains a predetermined number of pictures, the picture feature vector is obtained by extracting the picture features of each picture and mapping the picture features to the text feature space; if the number of pictures in the data to be processed is less than the predetermined number, after mapping the picture features to the text feature space, the mapping result is zero-filled according to the preset input sequence length; The multimodal fusion feature obtained by performing feature fusion on the text feature vector and the image feature vector is used as the input of the multimodal model; The multimodal model is trained to output a prediction result of a predetermined prediction task based on the input multimodal fusion features in the multimodal model.

2. The method according to claim 1, wherein: The training of the multimodal model comprises: The obtained text feature vector is used as the first input sequence, the obtained image feature vector is used as the second input sequence, and the first input sequence and the second input sequence are used as two inputs of the next sentence prediction task in the BERT-based multimodal model to train the BERT-based multimodal model.

3. The method according to claim 2, wherein: In the scenario where the multimodal model is a risk classification model, the purpose of the next sentence prediction task is to allow the multimodal model to learn the correlation between the image and the text based on the first input sequence and the second input sequence.

4. The method according to claim 1 or 2, wherein: If the data to be processed includes a predetermined number of pictures, the picture feature vector is obtained by extracting the picture features of each picture and mapping the picture features to the text feature space, including: If the data to be processed contains a predetermined number of pictures, the image features of each picture are extracted through a pre-trained convolutional neural network model, and mapped to the text feature space through a fully connected layer to obtain the image feature vector of the multimodal model.

5. The method according to claim 1 or 2, wherein: If the number of images in the data to be processed is less than a predetermined number, after mapping the image features to the text feature space, zero padding the mapping results according to a preset input sequence length includes: The image features of each image are extracted through the pre-trained convolutional neural network model and mapped to the text feature space through the fully connected layer; The mapping result is padded based on the zero padding technology, and a vector with the same length as the preset input sequence length is obtained by padding the zero vector as the image feature vector.

6. The method according to any one of claims 1 to 3, wherein: The method further comprises: After splicing and fusion, the text feature vector and image feature vector are processed for segment representation and position representation respectively.

7. The method according to claim 1 or 2, wherein: The multimodal model is a risk classification model, and the prediction result of the predetermined prediction task outputted based on the input multimodal fusion features in the multimodal model includes: A risk prediction result corresponding to the data to be processed is output based on the multimodal fusion feature, and the risk prediction result includes a violation determination result of the data to be processed.

8. A device for processing multimodal data, wherein: The device comprises: A device for obtaining a text feature vector of a multimodal model by performing word representation processing on the text content of the data to be processed; A device for obtaining a picture feature vector by extracting picture features of each picture and mapping the picture features to a text feature space if the data to be processed contains a predetermined number of pictures, and for performing zero padding on the mapping result according to a preset input sequence length after mapping the picture features to the text feature space if the number of pictures in the data to be processed is less than the predetermined number; A device for performing feature fusion by splicing the text feature vector and the image feature vector of the multimodal model together, so as to use the obtained multimodal fusion feature as the input of the multimodal model; A device for training the multimodal model to output a prediction result of a predetermined prediction task based on the input multimodal fusion features in the multimodal model.

9. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 6.

10. A computer readable medium having computer program instructions stored thereon, wherein the computer program instructions can be executed by a processor to implement the method according to any one of claims 1 to 7.

11. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images