Cross-modal data processing method and device, equipment, medium and program product

Abstract

The invention provides a cross-modal data processing method and device, equipment, a storage medium and a program product. The method comprises the following steps: acquiring to-be-processed first modal data; performing feature extraction based on the first modal data to obtain first modal data features; obtaining second modal data based on the first modal data features and a cross-modal processing model, wherein the first modal data and the second modal data have different modals; wherein the cross-modal processing model needs to be pre-trained based on series training samples, and the series training samples comprise series image samples and corresponding series text samples.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a cross-modal data processing method, device, equipment, medium and program product. Background Art

[0002] Existing visual text models have some limitations in processing the temporal semantic representation and correlation between images and videos. Visual text models often cannot learn the ability to understand temporal order in the pre-training stage. Even if fine-tuned and trained later, the final performance of the model will be average due to the limited amount of data. Even if pre-training is performed by jointly using image-text and video-text data, the number of samples in the image-text corpus is much larger than that in the video-text corpus, and the video-text compression samples are easily ignored. In addition, there are problems such as high visual redundancy in the video-text corpus and single scenes and descriptions. As a result, the accuracy of the cross-modal data processing model based on visual text is not high, and the task processing performance cannot be improved. Summary of the invention

[0003] The present disclosure proposes a cross-modal data processing method, apparatus, device, storage medium and program product to solve the technical problem of low accuracy in cross-modal data processing to a certain extent.

[0004] In a first aspect, the present disclosure provides a cross-modal data processing method, comprising:

[0005] Acquire first modal data to be processed;

[0006] Performing feature extraction based on the first modal data to obtain first modal data features;

[0007] Obtaining second modality data based on the first modality data features and a cross-modality processing model, wherein the first modality data and the second modality data have different modalities;

[0008] The cross-modal processing model needs to be pre-trained based on serial training samples, and the serial training samples include serial image samples and corresponding serial text samples.

[0009] In a second aspect of the present disclosure, a cross-modal data processing device is provided, comprising:

[0010] An acquisition module, used for acquiring first modal data to be processed;

[0011] A model module, configured to extract features based on the first modal data to obtain first modal data features; and to obtain second modal data based on the first modal data features and a cross-modal processing model, wherein the first modal data and the second modal data have different modalities;

[0012] The cross-modal processing model needs to be pre-trained based on serial training samples, and the serial training samples include serial image samples and corresponding serial text samples.

[0013] According to a third aspect of the present disclosure, an electronic device is provided, characterized in that it includes one or more processors, a memory; and one or more programs, wherein the one or more programs are stored in the memory and executed by the one or more processors, and the program includes instructions for executing the method described in the first aspect.

[0014] According to a fourth aspect of the present disclosure, a non-volatile computer-readable storage medium containing a computer program is provided. When the computer program is executed by one or more processors, the processors execute the method described in the first aspect.

[0015] In a fifth aspect of the present disclosure, a computer program product is provided, comprising computer program instructions, which, when executed on a computer, cause the computer to execute the method described in the first aspect.

[0016] From the above, it can be seen that the present disclosure provides a cross-modal data processing method, apparatus, device, medium and program product, which, by using serial image samples and text samples to train the cross-modal processing model in the pre-training stage, converts image and text sample pairs into serial image-serial text sample pairs, maintains the time sequence correspondence, and provides rich scene transformation and description information, so that the cross-modal processing model can learn explicit scene-level time alignment, improves the ability to learn static and temporal information, and thus improves the accuracy and efficiency of cross-modal data processing tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present disclosure or related technologies, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. Obviously, the drawings described below are only embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0018] Figure 1 A schematic diagram of a cross-modal data processing architecture according to an embodiment of the present disclosure.

[0019] Figure 2 The figure is a schematic diagram of the hardware structure of an exemplary electronic device according to an embodiment of the present disclosure.

[0020] Figure 3 Schematic diagram of the model architecture of the cross-modal data processing model of the present disclosure embodiment

[0021] Figure 4 It is a schematic diagram of the training samples of the embodiment of the present disclosure.

[0022] Figure 5 It is a schematic flowchart of the cross-modal data processing method according to an embodiment of the present disclosure.

[0023] Figure 6 Schematic diagram of a cross-modal data processing device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0024] In order to make the objectives, technical solutions and advantages of the present disclosure more clearly understood, the present disclosure is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0025] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present disclosure should be understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in the embodiments of the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. "Including" or "comprising" and similar words mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0026] Figure 1 A schematic diagram of a cross-modal data processing architecture of an embodiment of the present disclosure is shown. Figure 1 , the cross-modal data processing architecture 100 may include a server 110, a terminal 120, and a network 130 providing a communication link. The server 110 and the terminal 120 may be connected via a wired or wireless network 130. The server 110 may be an independent physical server, or a server cluster or distributed system consisting of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, security services, and CDN.

[0027] The terminal 120 may be implemented in hardware or software. For example, when the terminal 120 is implemented in hardware, it may be various electronic devices having a display screen and supporting page display, including but not limited to smart phones, tablet computers, e-book readers, laptop portable computers, and desktop computers, etc. When the terminal 120 device is implemented in software, it may be installed in the electronic devices listed above; it may be implemented as multiple software or software modules (such as software or software modules used to provide distributed services), or it may be implemented as a single software or software module, which is not specifically limited here.

[0028] It should be noted that the cross-modal data processing method provided in the embodiment of the present application can be executed by the terminal 120 or by the server 110. It should be understood that Figure 1 The number of terminals, networks and servers in the embodiment is only for illustration and is not intended to limit the number of terminals, networks and servers. Any number of terminals, networks and servers may be provided as required.

[0029] Figure 2 FIG. 2 shows a schematic diagram of the hardware structure of an exemplary electronic device 200 provided in an embodiment of the present disclosure. Figure 2 As shown, the electronic device 200 may include: a processor 202, a memory 204, a network module 206, a peripheral interface 208 and a bus 210. The processor 202, the memory 204, the network module 206 and the peripheral interface 208 are connected to each other in communication within the electronic device 200 through the bus 210.

[0030] Processor 202 may be a central processing unit (CPU), an image processor, a neural network processor (NPU), a microcontroller (MCU), a programmable logic device, a digital signal processor (DSP), an application specific integrated circuit (ASIC), or one or more integrated circuits. Processor 202 may be used to perform functions related to the technology described in this disclosure. In some embodiments, processor 202 may also include multiple processors integrated into a single logical component. For example, Figure 2 As shown, processor 202 may include a plurality of processors 202a, 202b, and 202c.

[0031] The memory 204 may be configured to store data (eg, instructions, computer code, etc.). Figure 2As shown, the data stored in the memory 204 may include program instructions (for example, program instructions for implementing the cross-modal data processing method of the embodiment of the present disclosure) and data to be processed (for example, the memory may store configuration files of other modules, etc.). The processor 202 may also access the program instructions and data stored in the memory 204, and execute the program instructions to operate on the data to be processed. The memory 204 may include a volatile storage device or a non-volatile storage device. In some embodiments, the memory 204 may include a random access memory (RAM), a read-only memory (ROM), an optical disk, a magnetic disk, a hard disk, a solid-state drive (SSD), a flash memory, a memory stick, etc.

[0032] The network module 206 can be configured to provide communication with other external devices to the electronic device 200 via a network. The network can be any wired or wireless network capable of transmitting and receiving data. For example, the network can be a wired network, a local wireless network (e.g., Bluetooth, WiFi, near field communication (NFC) etc.), a cellular network, the Internet or a combination thereof. It is understood that the type of network is not limited to the above specific examples. In some embodiments, the network module 306 can include any number of network interface controllers (NICs), radio frequency modules, transceivers, modems, routers, gateways, adapters, cellular network chips, etc., in any combination.

[0033] The peripheral interface 208 can be configured to connect the electronic device 200 to one or more peripheral devices to achieve information input and output. For example, the peripheral devices can include input devices such as a keyboard, a mouse, a touch pad, a touch screen, a microphone, and various sensors, and output devices such as a display, a speaker, a vibrator, and an indicator light.

[0034] The bus 210 can be configured to transmit information between various components of the electronic device 200 (e.g., the processor 202, the memory 204, the network module 206, and the peripheral interface 208), such as an internal bus (e.g., a processor-memory bus), an external bus (USB port, PCI-E bus), etc.

[0035] It should be noted that, although the architecture of the electronic device 200 only shows the processor 202, the memory 204, the network module 206, the peripheral interface 208 and the bus 210, in the specific implementation process, the architecture of the electronic device 200 may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the architecture of the electronic device 200 may also only include the components necessary for implementing the embodiments of the present disclosure, and does not necessarily include all the components shown in the figure.

[0036] Image-text and video-text pre-trained models have powerful cross-modal data processing capabilities between vision and language domains, such as various vision-language tasks such as cross-modal retrieval, visual description generation, and visual question answering. However, in traditional technologies, image-text and video-text models are trained independently using different cross-modal corpora, network architectures, and training objectives. Considering that videos can be viewed as a combination of multiple frames and images can be viewed as static videos, the relationship between text and vision (images, videos) can be uniformly modeled to train a general cross-modal base model.

[0037] Existing visual text models have some limitations in processing the temporal semantic representation and correlation between images and videos. Usually, the image-text basic model pre-trained on a large amount of image and text data is fine-tuned on the downstream video task. In this case, the trained model does not learn the ability to understand the video temporal sequence in the pre-training stage, and the fine-tuning stage will result in mediocre final performance of the model due to the limited amount of data. Although in the related art, image-text and video-text data can be used together for pre-training, the scale of the video-text corpus is two orders of magnitude smaller than that of the image-text corpus, so the video-text corpus is easily overwhelmed by the image-text corpus. In addition, the existing video-text corpus has problems such as high visual redundancy and single scenes and descriptions, which is not conducive to the model's learning of action-level timing and event-level timing. Therefore, how to improve the performance of the cross-modal data processing model in the pre-training stage, thereby improving the accuracy and efficiency of cross-modal data processing tasks, has become a technical problem that needs to be solved urgently.

[0038] In view of this, the embodiments of the present disclosure provide a cross-modal data processing method, apparatus, device, medium, and program product. By using serial image samples and text samples to train the cross-modal processing model in the pre-training stage, the image and text sample pairs are converted into serial image-serial text sample pairs, the temporal correspondence is maintained, and rich scene transformation and description information are provided, so that the cross-modal processing model can learn explicit scene-level time alignment, and the learning ability of static and temporal information is improved, thereby improving the accuracy and efficiency of cross-modal data processing tasks.

[0039] See also Figure 3 , Figure 3 A schematic diagram of a multimodal processing model architecture according to an embodiment of the present disclosure is shown. Figure 3In the cross-modal data processing model 300, a vision encoder 310 and a text encoder 320 may be included. The input of the vision encoder 310 may be image data (e.g., a single image or an image sequence), and the output may be a corresponding image feature (or an image feature sequence). In some embodiments, the vision encoder 310 may be a unimodal encoder. The input of the text encoder 320 may be text information, and the output may be a text feature corresponding to the text information. In some embodiments, the text encoder 320 may be a bidirectional encoder BERT (Bidirectional Encoder Representations from Transformers). Further, in some embodiments, the text encoder 320 may include a self-attention layer, a cross-attention layer, and a feedforward layer. Specifically, the self-attention layer performs self-attention calculation on the input text of the text encoder 320 to obtain a text self-attention feature, and the cross-attention layer performs cross-attention calculation on the text self-attention feature and the image feature output by the vision encoder 310 to obtain a text-image cross-attention feature, i.e., a multimodal feature; the feedforward layer outputs target data based on the multimodal feature. In this way, the text encoder 320 can be used as a unimodal encoder to encode text data, as a cross-modal encoder to achieve the fusion of visual features and text features, and as a decoder to output the final target data, thereby achieving cross-modal interaction.

[0040] In some embodiments, the multimodal processing model needs to pre-train the initial model based on the serialized training samples, specifically including:

[0041] Acquire the serialized training samples, wherein the training samples include serialized image samples and corresponding serialized text samples;

[0042] Performing feature extraction based on the serial image samples to obtain serial image features with time information and performing feature extraction based on the text samples to obtain serial text features with a time sequence relationship;

[0043] Obtaining a multimodal feature based on fusing the serial image feature and the serial text feature;

[0044] The initial model is pre-trained based on the series of image samples, the series of text samples and the multimodal features.

[0045] Among them, the images and texts in the serial training samples correspond one to one, and after feature extraction, the time information of the serial image features and the temporal relationship in the serial text features are also correlated. For example, the matching image-text pairs <image 1, text 1>, <image 2, text 2>, <image 3, text 3>, <image 4, text 4> are serially connected to obtain serial image samples <image 1-image 2-image 3-image 4> and serial text samples <text 1-text 2-text 3-text 4>. It can be seen that there is a corresponding relationship between the serial image samples and the serial text samples, and the serial order of the two is related. Then, after feature extraction of the serial image samples <image 1-image 2-image 3-image 4>, serial image features with time information are obtained, and after feature extraction of the serial text samples <text 1-text 2-text 3-text 4>, serial text features with temporal relationship are obtained. The time information in the serial image features and the temporal relationship in the serial text features are also correspondingly correlated.

[0046] In some embodiments, obtaining a training sample may further include:

[0047] Obtain at least one batch of multiple training image-text pairs;

[0048] For each training image-text pair in each batch, a preset number of training image-text pairs in the same batch are selected for concatenation to obtain concatenated image samples and corresponding concatenated text samples.

[0049] The training image-text pair may include an image sample and a corresponding text sample, for example, the training image-text pair<I,T> It includes an image sample I and a corresponding text sample T. The text sample T can be used to describe the scene of the image sample I. Images with pre-labeled or matched corresponding texts can be used as image-text pairs. During the pre-training process, the initial model can be trained in batches, and each batch can include multiple training image-text pairs. For each training image-text pair in each batch<I,T> i , i is a positive integer, and a certain number of other training image-text pairs can be randomly selected<I,T> j , j≠i and j is a positive integer, and each training image text pair<I,T> i Specifically, the image-text database can be converted into a concatenated image-concatenated text database by online sample concatenation. For each image-text sample in each batch, a certain number of other image-text samples can be randomly selected from the same batch to be concatenated with it. Figure 4 , Figure 4 A schematic diagram of a training sample according to an embodiment of the present disclosure is shown. Figure 4In the example, for the training image-text pair 401, a preset number (e.g., 3) of training image-text pairs 403, 405, and 406 can be selected from the same batch of training image-text pairs 402-407 for concatenation. The concatenation order can be that the training image-text pairs 401, 405, 403, and 406 are concatenated in sequence, that is, the image samples of the training image-text pairs 401, 405, 403, and 406 are concatenated in sequence to form a concatenated image sample 410, and the text samples are spliced ​​into a text paragraph, namely, a concatenated text sample 420. The concatenated image sample 410 and the concatenated text sample 420 form a concatenated training sample. It can be seen that the concatenation of images and texts ensures a clear correspondence between the events represented by the images and the sentences of the texts.

[0050] In some embodiments, the concatenated image samples include a plurality of image samples concatenated;

[0051] Extracting features based on the serial image samples to obtain serial image features with time information includes:

[0052] Extracting features from the image samples in the series of image samples in sequence according to the series order to obtain image features;

[0053] The corresponding time information is embedded into the image features based on the series connection sequence, and the image features embedded with the time information are connected in series to obtain the series connection image features.

[0054] In some embodiments, the concatenated text sample comprises a plurality of concatenated text samples, the text samples corresponding to the image samples;

[0055] The feature extraction based on the text sample is performed to obtain a series of text features with a temporal relationship, including:

[0056] splicing the corresponding text samples into text segments based on the serial order of the image samples;

[0057] Feature extraction is performed on the text segment, and the position information of the text sample in the text segment is embedded to obtain the serial text features with a time sequence relationship.

[0058] Specifically, for the image sample part, the image samples in the concatenated image samples are sequentially input into the visual encoder, the temporal information of the image samples is embedded into the output features of the concatenated image samples, and they are concatenated along the temporal dimension to obtain the concatenated image features. For the text sample part, the corresponding text samples can be directly concatenated into a long paragraph according to the concatenation order of the image samples, and the temporal relationship of the text samples can be encoded by the position embedding layer of the text encoder. In this way, the image is regarded as a snapshot of multiple fragments, which constitute a pseudo-long form of video, each of which captures a different scene and its corresponding text description. By concatenating samples and position embedding, the model can be trained to learn clear scene-level temporal alignment.

[0059] In the prior art, due to the high cost of video uploading, storage and downloading, there is less available open source video-text data, which limits the performance of video-based visual-language models in temporal semantic representation and correlation modeling, and the pre-training of the model is limited by the scale and quality of the video-text training corpus. Compared with the prior art, according to the serial training samples of the embodiment of the present disclosure, the image and text samples are converted into serial image-serial text samples by online sample splicing and serialization, and the temporal clues of the visual content and event level are modeled at the same time. The serialized images and texts maintain the corresponding relationship of the time sequence, provide rich scene changes and description information, and reduce visual redundancy through sampling randomness. At the same time, through sample splicing and position embedding coding, the model can learn explicit scene-level time alignment, improving the model's learning ability for static and temporal information.

[0060] Traditional large-scale visual-language models are usually trained only on image-text corpora, ignoring the joint modeling between images and videos. In the pre-training process of the cross-modal data processing model according to the embodiment of the present disclosure, by concatenating samples of images and texts, the association between static and temporal information can be captured in the pre-training stage, thereby improving the ability of video-text reasoning. At the same time, based on the joint modeling of images and texts, the scope of video semantic modeling is effectively expanded.

[0061] In some embodiments, the multimodal feature is obtained by fusing the serial image feature and the serial text feature, including:

[0062] Performing self-attention calculation based on the concatenated text features to obtain text self-attention features;

[0063] A cross-attention calculation is performed based on the text self-attention feature and the serial image feature to obtain the multimodal feature.

[0064] To enhance the model’s cross-modal alignment, understanding, and generation capabilities, at least one of the following training objectives can be adopted: image-text comparison (ITC), image-text matching (ITM), masked language modeling (MLM), and generative modeling (GM).

[0065] In some embodiments, pre-training the initial model based on the serial image samples, the serial text samples and the multimodal features includes:

[0066] Using the tandem image features of the tandem image samples and the corresponding tandem text features in the tandem text samples as a positive sample pair, and using the tandem image features and the non-corresponding tandem text features as a negative sample pair;

[0067] Calculating a first similarity of the positive sample pair and a second similarity of the negative sample pair;

[0068] The first model parameters of the initial model are adjusted so that the first similarity is minimized and the second similarity is maximized.

[0069] Among them, for the training objective of concatenated image-concatenated text contrast (CITC), the global representation of the concatenated image can be obtained through the visual encoder, and the representation of the concatenated text can be obtained through the category features (such as [CLS] features) of the text encoder. The paired concatenated samples are clustered together using the bidirectional contrast loss, and the unpaired samples are pushed apart. For example, m (m is a positive integer) frames of images are input into the visual encoder respectively, and the average value of the output category features (such as [CLS] features) is used as the global feature of the concatenated image sample, i.e., the concatenated image feature. The m text samples corresponding to the m frames of images are concatenated to obtain the concatenated text sample; the concatenated text sample is input into the text encoder, and the output category features (such as [CLS] features) are used as the global text feature of the concatenated text sample, i.e., the concatenated text feature. It can be seen that the concatenated image features match the corresponding concatenated text features, and contrastive learning can be used to shorten the distance between the matched concatenated image features and the concatenated text features in the metric space, and to lengthen the distance between the unmatched concatenated image features and the concatenated text features.

[0070] In some embodiments, pre-training the initial model based on the serial image samples, the serial text samples and the multimodal features includes:

[0071] Calculating matching confidence for the series of image samples and the series of text samples;

[0072] Calculate the sum of the binary classification losses of all the matching confidences based on the category features to obtain a first loss function;

[0073] A second model parameter of the initial model is adjusted based on the first loss function to minimize the first loss function.

[0074] Among them, the training objective of serial image-serial text matching (CITM) can be used to train the model to determine whether long videos and paragraphs correspond to each other. The hard negative sample mining method can be adopted, and the binary classification loss function is calculated by using the multi-layer perceptron (MLP) through the category features of the text encoder (such as [CLS] features). Specifically, one of the corresponding serial image samples A_image and serial text samples A_text is replaced with other unmatched samples in the same batch. For example, the serial image sample A_image is replaced with other serial image samples B_image in the same batch. Then, based on the category features of the text encoder, it is determined whether the serial image sample B_image and the serial text sample A_text match, and then the binary classification loss function is obtained.

[0075] In some embodiments, pre-training the initial model based on the serial image samples, the serial text samples and the multimodal features includes:

[0076] Generating a first concatenated text mask sample based on the concatenated text sample;

[0077] Performing self-attention calculation based on the first series of text mask samples to obtain a first series of text mask features;

[0078] Performing a cross attention calculation based on the series image feature and the first series text mask feature to obtain a first image-text mask feature;

[0079] Calculating a second loss function between the first image-text mask cross-attention feature and the multimodal feature;

[0080] A third model parameter of the initial model is adjusted based on the second loss function to minimize the second loss function.

[0081] Among them, for the training goal of serial masked language modeling (CMLM), a certain proportion (for example, 15%) of the tokens in the serial text can be randomly masked, and the prediction layer of the text encoder can be used to reconstruct the masked tokens in the context of the serial image samples. Then the second loss function between the two is calculated with the real sample (for example, multimodal features). Text mask samples can be obtained based on the samples in this article and the preset text masking strategy. Furthermore, the text mask samples are obtained based on the samples in this article and the preset text masking strategy, including: randomly selecting a preset proportion of words in the text sample for masking to generate the text mask sample. It should be understood that the number of words in the preset proportion may not be an integer, and an integer (for example, rounding) can be taken as needed for masking, which is not limited here.

[0082] In some embodiments, pre-training the initial model based on the tandem image feature, the tandem text feature, and the multimodal feature includes:

[0083] Generating a second concatenated text mask sample based on the concatenated text sample;

[0084] Performing causal self-attention calculation based on the second series text mask sample to obtain a second series text mask feature;

[0085] Performing cross attention calculation based on the series image feature and the second series text mask feature to obtain a second image-text mask feature;

[0086] Calculating a second loss function between the second image-text mask cross-attention feature and the multimodal feature;

[0087] A fourth model parameter of the initial model is adjusted based on the third loss function to minimize the third loss function.

[0088] Among them, for the training objective of concatenated generative modeling (CGM), a certain proportion (e.g. 60%) of the tokens in the paragraph can be randomly masked, and the masked tokens are reconstructed in the context of concatenated image samples using the same prediction layer as CMLM. CGM introduces causal attention masks in the self-attention layer of the text encoder to prevent information leakage and enhance the model's text generation ability.

[0089] In some embodiments, pre-training the initial model based on the tandem image feature, the tandem text feature, and the multimodal feature includes:

[0090] The image sample and the corresponding text sample are taken as a single positive sample pair, and the image sample and a non-corresponding text sample in the training sample are taken as a single negative sample pair;

[0091] Calculating a third similarity of the single positive sample pair and a fourth similarity of the single negative sample pair;

[0092] A fifth model parameter of the initial model is adjusted so that the third similarity is minimized and the fourth similarity is maximized.

[0093] Among them, image-text contrastive learning (ITC) can also be used as a training objective. Specifically, an image sample and its corresponding text sample can be regarded as a single positive sample pair, and all other samples in the same batch can be regarded as a single negative sample pair. Then, the loss function is calculated by calculating the cosine similarity distance between them. Comparing the distance between a single positive sample pair and a single negative sample pair can make the distance between positive sample pairs closer and the distance between negative sample pairs farther. .

[0094] In some embodiments, pre-training the initial model based on the tandem image feature, the tandem text feature, and the multimodal feature includes:

[0095] Calculating image-sample matching confidence for the image sample and the text sample;

[0096] Calculating the sum of the binary classification losses of all the image-sample matching confidences based on the category labels to obtain a fourth loss function;

[0097] A fifth model parameter of the initial model is adjusted based on the fourth loss function to minimize the fourth loss function.

[0098] Specifically, image-text comparison matching (ITM) can also be used as a training target, which is the same as the training target of image-text comparison, and can enhance the model's ability to process a single sample.

[0099] After the above pre-training stage, the pre-trained model is obtained, which can use the rich information of images and texts to better capture the correlation between vision and language and the accurate event-description correspondence, greatly enhancing the performance of the model. On this basis, the pre-trained model can also be trained specifically for different downstream tasks to obtain a multimodal data processing model for different downstream tasks (for example, long / short video-text tasks and image-text tasks such as retrieval, subtitle generation, and question-answering).

[0100] In some embodiments, the method may further include: training the pre-trained model based on task training samples to obtain the cross-modal data processing model.

[0101] Furthermore, in some embodiments, training the pre-trained model based on the task training sample to obtain the cross-modal data processing model may further include:

[0102] Acquire the task training samples, wherein the task training samples include at least one video-text task training pair (or image-text task training pair, including an image task training sample and a corresponding text task training sample), and each video-text task training pair includes a video task training sample and a corresponding text task training sample;

[0103] The pre-trained model is trained based on the task training samples until the target training requirements are met to obtain the cross-modal data processing model.

[0104] Among them, for different cross-modal data processing tasks, the content contained in the task training samples may be different. For example, for a visual information generation task, the cross-modal data processing model can generate text information (such as a summary, title, introduction, etc.) associated with the visual data based on visual data (such as video data or image data). Then, the task training sample corresponding to the video information generation task may include at least one video-text information pair, each video-text information pair including a video training sample and corresponding text information such as a summary, title, introduction, etc.

[0105] For the text-visual generation task, the cross-modal data processing model can generate visual data (such as images, videos, etc.) corresponding to the text data based on the text data. Then, the task training sample corresponding to the text-visual generation task can include at least one text-visual information pair, each of which includes text information such as abstract, title, introduction, etc. and corresponding visual data.

[0106] It can be seen that the pre-trained model obtained according to the cross-modal data processing method of the embodiment of the present disclosure can ensure high efficiency and high performance of task processing between different modal data.

[0107] See also Figure 5 , Figure 5 FIG. 1 is a schematic flow chart of a cross-modal data processing method according to an embodiment of the present disclosure. Figure 5 As shown, the cross-modal data processing method 500 can be based on Figure 3 The cross-modal data processing model implementation shown further includes the following steps.

[0108] Step S510, obtaining first modal data to be processed.

[0109] The first modality data may refer to visual data, including image data or video data. The first modality data may also refer to text data.

[0110] Step S520: extracting features based on the first modal data to obtain first modal data features.

[0111] Specifically, the first modality data features may be obtained by performing feature extraction on the first modality data based on the image encoder 310 or the text encoder 320 .

[0112] Step S530, obtaining second modal data based on the first modal data features and a cross-modal processing model, wherein the first modal data and the second modal data have different modalities; wherein the cross-modal processing model needs to be pre-trained based on serial training samples, and the serial training samples include serial image samples and corresponding serial text samples.

[0113] The second modal data is used for a different modality than the first modal data. For example, when the first modal data is visual data, the second modal data may be text data. When the first modal data is text data, the second modal data may be visual data.

[0114] Specifically, for the video data to be processed, the user hopes to generate corresponding summary information for the video data. The cross-modal data processing model can perform feature extraction on the video data to obtain a first modal data feature, namely a video feature, and the first modal data feature can be a feature vector. The cross-modal data processing model searches and matches in a text feature set based on the first modal data feature, and the text feature set can be a set of text features obtained after feature extraction of a preset text. After searching and matching, one or more target text features that match the temporal image features can be obtained, and based on the target preset text corresponding to the one or more target text features, a target text about the video data can be formed as summary information. According to the cross-modal data processing method of an embodiment of the present disclosure, a cross-modal data processing model is used to generate relevant text information based on a video, which can improve the accuracy of the text information.

[0115] For the text data to be processed, the user hopes to generate corresponding video data for the text data. The cross-modal data processing model can perform feature extraction on the text data to obtain a first modal data feature, namely a text feature, and the first modal data feature can be a feature vector. The cross-modal data processing model searches and matches in a video feature set based on the first modal data feature, and the video feature set can be a set of video features obtained after feature extraction of a preset video. After searching and matching, one or more target video features that match the text features can be obtained, and based on the target preset video corresponding to the one or more target video features, target video data about the text data can be formed. According to the cross-modal data processing method of the embodiment of the present disclosure, the accuracy of video data generation can be improved by using a cross-modal data processing model to generate related video information based on text.

[0116] It should be noted that the method of the embodiment of the present disclosure can be performed by a single device, such as a computer or a server. The method of the present embodiment can also be applied in a distributed scenario and completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only perform one or more steps in the method of the embodiment of the present disclosure, and the multiple devices will interact with each other to complete the described method.

[0117] It should be noted that the above describes some embodiments of the present disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the above embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0118] Based on the same technical concept, corresponding to any of the above-mentioned embodiments and methods, the present disclosure also provides a cross-modal data processing device, see Figure 6 , the cross-modal data processing device comprises:

[0119] An acquisition module, used for acquiring first modal data to be processed;

[0120] A model module, configured to extract features based on the first modal data to obtain first modal data features; and to obtain second modal data based on the first modal data features and a cross-modal processing model, wherein the first modal data and the second modal data have different modalities;

[0121] The cross-modal processing model needs to be pre-trained based on serial training samples, and the serial training samples include serial image samples and corresponding serial text samples.

[0122] For the convenience of description, the above device is described by dividing it into various modules according to its functions. Of course, when implementing the present disclosure, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0123] The device of the above embodiment is used to implement the corresponding cross-modal data processing method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0124] Based on the same technical concept, corresponding to any of the above-mentioned embodiment methods, the present disclosure also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the cross-modal data processing method described in any of the above embodiments.

[0125] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0126] The computer instructions stored in the storage medium of the above embodiments are used to enable the computer to execute the cross-modal data processing method described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0127] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present disclosure (including the claims) is limited to these examples. Based on the concept of the present disclosure, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present disclosure as described above, which are not provided in detail for the sake of simplicity.

[0128] In addition, to simplify the description and discussion, and in order not to make the embodiments of the present disclosure difficult to understand, the known power / ground connections to the integrated circuit (IC) chips and other components may or may not be shown in the provided figures. In addition, the device can be shown in the form of a block diagram to avoid making the embodiments of the present disclosure difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform on which the embodiments of the present disclosure will be implemented (that is, these details should be fully within the scope of understanding of those skilled in the art). Where specific details (e.g., circuits) are set forth to describe exemplary embodiments of the present disclosure, it is apparent to those skilled in the art that the embodiments of the present disclosure can be implemented without these specific details or with changes in these specific details. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0129] Although the present disclosure has been described in conjunction with specific embodiments of the present disclosure, many replacements, modifications and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.

[0130] The embodiments of the present disclosure are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present disclosure should be included in the scope of protection of the present disclosure.

Claims

1. A cross-modal data processing method, It is characterized in that include: Acquire first modal data to be processed; Performing feature extraction based on the first modal data to obtain first modal data features; Obtaining second modality data based on the first modality data features and a cross-modality processing model, wherein the first modality data and the second modality data have different modalities; The cross-modal processing model needs to be pre-trained based on serial training samples, and the serial training samples include serial image samples and corresponding serial text samples.

2. The method according to claim 1, It is characterized in that The cross-modal processing model needs to be pre-trained based on serial training samples, including: Acquire the serialized training samples, wherein the training samples include serialized image samples and corresponding serialized text samples; Performing feature extraction based on the serial image samples to obtain serial image features with time information and performing feature extraction based on the text samples to obtain serial text features with a time sequence relationship; Obtaining a multimodal feature based on fusing the serial image feature and the serial text feature; The initial model is pre-trained based on the series of image samples, the series of text samples and the multimodal features.

3. The method according to claim 2, It is characterized in that Acquiring the series training sample includes: Obtain at least one batch of multiple training image-text pairs; For each training image-text pair of each batch, a preset number of training image-text pairs in the same batch are selected for concatenation to obtain the concatenated image samples and the corresponding concatenated text samples.

4. The method according to claim 3, It is characterized in that The serially connected image samples include a plurality of serially connected image samples; Extracting features based on the serial image samples to obtain serial image features with time information includes: Extracting features from the image samples in the series of image samples in sequence according to the series order to obtain image features; The corresponding time information is embedded into the image features based on the series connection sequence, and the image features embedded with the time information are connected in series to obtain the series connection image features.

5. The method according to claim 3, It is characterized in that The serial text sample includes a plurality of serially connected text samples, and the text sample corresponds to the image sample; The feature extraction based on the text sample is performed to obtain a series of text features with a temporal relationship, including: splicing the corresponding text samples into text segments based on the serial order of the image samples; Feature extraction is performed on the text segment, and the position information of the text sample in the text segment is embedded to obtain the serial text features with a time sequence relationship.

6. The method according to claim 2, It is characterized in that The multimodal features are obtained by fusing the serial image features and the serial text features, including: Performing self-attention calculation based on the concatenated text features to obtain text self-attention features; A cross-attention calculation is performed based on the text self-attention feature and the serial image feature to obtain the multimodal feature.

7. The method according to claim 2, It is characterized in that Pre-training the initial model based on the serial image samples, the serial text samples and the multimodal features includes: Using the tandem image features of the tandem image samples and the corresponding tandem text features in the tandem text samples as a positive sample pair, and using the tandem image features and the non-corresponding tandem text features as a negative sample pair; Calculating a first similarity of the positive sample pair and a second similarity of the negative sample pair; The first model parameters of the initial model are adjusted so that the first similarity is minimized and the second similarity is maximized.

8. The method according to claim 2, It is characterized in that Pre-training the initial model based on the serial image samples, the serial text samples and the multimodal features includes: Calculating matching confidence for the serial image samples and the serial text samples; Calculate the sum of the binary classification losses of all the matching confidences based on the category labels to obtain a first loss function; A second model parameter of the initial model is adjusted based on the first loss function to minimize the first loss function.

9. The method according to claim 2, It is characterized in that Pre-training the initial model based on the serial image samples, the serial text samples and the multimodal features includes: Generating a first concatenated text mask sample based on the concatenated text sample; Performing self-attention calculation based on the first series of text mask samples to obtain a first series of text mask features; Performing a cross attention calculation based on the series image feature and the first series text mask feature to obtain a first image-text mask feature; Calculating a second loss function between the first image-text mask cross-attention feature and the multimodal feature; A third model parameter of the initial model is adjusted based on the second loss function to minimize the second loss function.

10. The method according to claim 2, It is characterized in that Pre-training the initial model based on the serial image feature, the serial text feature and the multimodal feature includes: Generating a second concatenated text mask sample based on the concatenated text sample; Performing causal self-attention calculation based on the second series text mask sample to obtain a second series text mask feature; Performing cross attention calculation based on the series image feature and the second series text mask feature to obtain a second image-text mask feature; Calculating a second loss function between the second image-text mask cross-attention feature and the multimodal feature; A fourth model parameter of the initial model is adjusted based on the third loss function to minimize the third loss function.

11. A cross-modal data processing device, It is characterized in that include: An acquisition module, used for acquiring first modal data to be processed; A model module, configured to extract features based on the first modal data to obtain first modal data features; and obtaining second modal data based on the first modal data features and the cross-modal processing model, wherein the first modal data and the second modal data have different modalities; The cross-modal processing model needs to be pre-trained based on serial training samples, and the serial training samples include serial image samples and corresponding serial text samples.

12. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 10 when executing the program.

13. A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute the method according to any one of claims 1 to 10.

14. A computer program product, comprising computer program instructions, which, when executed on a computer, enable the computer to execute the method according to any one of claims 1 to 10.

Images