Multimodal data processing method, device, electronic device and readable storage medium

Abstract

The present disclosure provides a multimodal data processing method, device, electronic device and readable storage medium, which relate to the field of artificial intelligence technology, especially to the technical fields of knowledge graphs, data processing, etc., and can be applied to scenarios such as video understanding. The specific implementation scheme is: obtaining multimodal data of the target video; determining the semantic information of at least two unimodal data in the multimodal data; and determining the redundant data in the multimodal data based on the similarity of the semantic information. In this scheme, the redundant data existing in the multimodal data can be effectively determined based on the similarity of the semantic information of each unimodal data, which provides a basis for the subsequent targeted processing of the redundant data, thereby ensuring the understanding effect of the video content.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to the fields of knowledge graphs and data processing, and can be applied to scenarios such as video understanding. Specifically, the present disclosure relates to a multimodal data processing method, device, electronic device and readable storage medium. Background Art

[0002] In recent years, with the rapid development of software and hardware technologies, video has become one of the main ways of information transmission. In order to quickly distribute and recommend videos to users, a deep understanding of video content is required.

[0003] Videos generally contain multimodal data, such as image data, audio data, and text data. If only single-modal data in the multimodal data is used to understand the video content, it may lead to a one-sided understanding of the video content. Therefore, it is necessary to use multimodal data to understand the video content.

[0004] There may be a lot of redundant data in multimodal information, which affects the understanding of video content. Therefore, how to effectively determine the redundant data in multimodal information has become an important technical issue. Summary of the invention

[0005] In order to solve at least one of the above-mentioned defects, the present disclosure provides a multimodal data processing method, device, electronic device and readable storage medium.

[0006] According to a first aspect of the present disclosure, a multimodal data processing method is provided, the method comprising:

[0007] Obtain multimodal data of the target video;

[0008] Determining semantic information of at least two unimodal data in the multimodal data;

[0009] Determining redundant data in multimodal data based on the similarity of semantic information.

[0010] According to a second aspect of the present disclosure, a multimodal data processing device is provided, the device comprising:

[0011] A multimodal data acquisition module, used to acquire multimodal data of a target video;

[0012] A semantic information determination module, used to determine semantic information of at least two types of unimodal data in the multimodal data;

[0013] The redundant data determination module is used to determine the redundant data in the multimodal data based on the similarity of semantic information.

[0014] According to a third aspect of the present disclosure, an electronic device is provided, the electronic device comprising:

[0015] at least one processor; and

[0016] A memory in communication with the at least one processor; wherein,

[0017] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the multimodal data processing method.

[0018] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the above-mentioned multimodal data processing method.

[0019] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising a computer program, which implements the above-mentioned multimodal data processing method when executed by a processor.

[0020] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0022] Figure 1 is a flowchart of a multimodal data processing method provided by an embodiment of the present disclosure;

[0023] Figure 2 is a flowchart of another multimodal data processing method provided by an embodiment of the present disclosure;

[0024] Figure 3 is a structural schematic diagram of a multimodal data processing device provided by an embodiment of the present disclosure;

[0025] Figure 4 is a structural schematic diagram of another multimodal data processing device provided by an embodiment of the present disclosure;

[0026] Figure 5 It is a block diagram of an electronic device used to implement the multimodal data processing method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0027] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0028] When using multimodal data to understand video content, it is generally necessary to perform multimodal fusion on the multimodal data. In related technologies, when performing multimodal fusion on multimodal data, the multimodal data are generally treated equally, and there may be a large amount of redundant data in the multimodal information, which reduces the weight of the effective information in the multimodal information, affects the multimodal fusion effect, and further affects the understanding of the video content.

[0029] If the redundant data in the multimodal information can be effectively determined, the redundant data in the multimodal information can be processed in a targeted manner to avoid affecting the understanding of the video content.

[0030] The multimodal data processing method, device, electronic device and readable storage medium provided by the embodiments of the present disclosure are intended to solve at least one of the above technical problems in the prior art.

[0031] Figure 1 A schematic diagram of a multimodal data processing method provided by an embodiment of the present disclosure is shown. Figure 1 As shown in , the method can mainly include:

[0032] Step S110: Acquire multimodal data of the target video;

[0033] Step S120: respectively determining semantic information of at least two types of unimodal data in the multimodal data;

[0034] Step S130: Determine redundant data in the multimodal data based on the similarity of semantic information.

[0035] The target video may be a video whose video content needs to be understood, and the multimodal data of the target video may include single-modal data such as images, audio, and text.

[0036] Each unimodal data of the target video will express some semantic information. When the semantic information expressed by each unimodal data in the target video has certain similarities, redundant data may exist in the multimodal data.

[0037] For example, the target video contains the text "A woman made a bowl of noodles, and the noodles were so delicious." Some images in the target video can express the semantic information of "woman" and "noodles." It can be seen that the semantic information expressed by some images of the target video is similar to the semantic information expressed by the text of the target video, which may cause some redundant data in the multimodal data of the target video.

[0038] Based on the similarity of the semantic information of each single-modal data, the redundant data in the multimodal data can be determined, which provides a basis for targeted processing of the redundant data in the subsequent multimodal fusion to ensure the effect of multimodal fusion.

[0039] The method provided by the embodiment of the present disclosure obtains multimodal data of the target video, determines the semantic information of at least two types of unimodal data in the multimodal data, and determines the redundant data in the multimodal data based on the similarity of the semantic information. In this solution, the redundant data in the multimodal data can be effectively determined based on the similarity of the semantic information of each unimodal data, which provides a basis for subsequent targeted processing of the redundant data, thereby ensuring the understanding effect of the video content.

[0040] In actual use, when the multimodal data includes two unimodal data, the semantic information of the two unimodal data can be determined respectively, and then the redundant data existing in the multimodal data can be determined based on the similarity of the semantic information of the two unimodal data. When the multimodal data includes more than two unimodal data, the semantic information of these unimodal data can be determined respectively, and then any two unimodal data are taken as a group, and the redundant data existing in the group of unimodal data is determined based on the similarity of the semantic information of the two unimodal data in the group, and then the redundant data existing in each group of modal data is collectively used as the redundant data of the multimodal data as a whole. For example, when the multimodal data includes three types of unimodal data, namely, image, audio and text, the image and audio, image and text, and audio and text can be respectively used as three groups of unimodal data, and the redundant data existing in the three groups of unimodal data can be determined based on the similarity of the semantic information in the three groups of unimodal data, and then the redundant data existing in the three groups of unimodal data can be collectively used as the final redundant data.

[0041] In an optional manner of the present disclosure, the at least two unimodal data include target image data and target text data, and determining semantic information of the at least two unimodal data in the multimodal data includes:

[0042] Determining semantic information of target image data and semantic information of target text data;

[0043] Determine redundant data in multimodal data based on the similarity of semantic information, including:

[0044] Based on the similarity between the semantic information of the target image data and the semantic information of the target text data, redundant data in the target image data and / or redundant data in the target text data are determined.

[0045] In the embodiment of the present disclosure, the multimodal data of the target video may include two types of unimodal data: target image data and target text data.

[0046] As an example, the target image data may be a key frame image of a target video, and the target text data may be a title text of the target video.

[0047] In the disclosed embodiment, the semantic information of the target image data and the semantic information of the target text can be determined respectively, and then based on the similarity between the semantic information of the target image data and the semantic information of the target text data, the redundant data in the target image data and / or the redundant data in the target text data can be determined. Specifically, when the possibility that the redundant data exists in the target image data is relatively high, the redundant data can be determined from the target image data; when the possibility that the redundant data exists in the target text data is relatively high, the redundant data can be determined from the target text data; when the redundant data exists in both the target image data and the target text data, the redundant data can be determined from the target image data and the target text data respectively.

[0048] As an example, the target image data is a plurality of key frame images, and the target text data is the title of the target video. In this case, the target image data has a large amount of data and is more likely to contain redundant data. Therefore, the semantic information of the target image data and the semantic information of the target text data can be obtained respectively, and then the redundant data in the target image data can be determined based on the similarity between the semantic information of the target image data and the semantic information of the target text data, that is, the key frame images that may be redundant data can be determined from the plurality of key frame images.

[0049] In an optional manner of the present disclosure, determining the semantic information of the target image data and the semantic information of the target text data includes:

[0050] The first semantic feature of the target image data in the semantic space is determined as the semantic information of the target image data, and the second semantic feature of the target text data in the semantic space is determined as the semantic information of the target text data.

[0051] In the disclosed embodiment, the target image data and the target text data can be mapped to the same semantic space respectively, and the obtained first semantic feature can be used to describe the semantics of the target image data, and the second semantic feature can be used to describe the semantics of the target text data. The first semantic feature can be determined as the semantic information of the target image data, and the second semantic feature can be determined as the semantic information of the target text data.

[0052] In actual use, the contrastive language-image pre-training (CLIP) model can be used to map the target image data and the target text data to the same semantic space respectively.

[0053] In an optional manner of the present disclosure, determining the semantic information of the target image data and the semantic information of the target text data includes:

[0054] The first entity name identified in the target image data is determined as the semantic information of the target image data, and the second entity name identified in the target text data is determined as the semantic information of the target text data.

[0055] The first entity name is the name of the entity identified from the target image data, and the second entity name is the name of the entity identified from the target text data.

[0056] In the disclosed embodiment, some entities can generally be identified in the target image data and the target text data, and the entity names of these entities can represent semantic information. Therefore, the first entity name identified from the target image data can be determined as the semantic information of the target image data, and the second entity name identified in the target text data can be determined as the semantic information of the target text data.

[0057] As an example, a target detection model, such as a Faster Region Convolutional Neural Network (FasterR-CNN) model, can be used to identify entities in target image data to obtain a first entity name. A Named Entity Recognition (NER) model can be used to identify entities in target text data to obtain a second entity name.

[0058] In an optional manner of the present disclosure, determining redundant data in the target image data and / or redundant data in the target text data based on similarity between semantic information of the target image data and semantic information of the target text data includes:

[0059] The target image data and / or the target text data whose similarity between the semantic information of the target image data and the semantic information of the target text data is greater than a preset value is determined as redundant data.

[0060] In the disclosed embodiment, the similarity of semantic information can be measured by similarity, and a preset value of the similarity can be set. Target image data and / or target text data whose similarity between the semantic information of the target image data and the semantic information contained in the target text is higher than the preset value can be considered as redundant data, and target image data and / or target text data whose similarity between the semantic information of the target image data and the semantic information contained in the target text is not higher than the preset value are not redundant data.

[0061] As an example, when the semantic information of the target image data is a first semantic feature and the semantic information of the target text data is a second semantic feature, the similarity may be a semantic similarity calculated based on the first semantic feature and the second semantic feature.

[0062] Specifically, the CLIP model is defined as M v-t , the target image data can be the key frame image of the video, and the key frame image sequence of the video can be recorded as [img1, img2, ...., img n ], the target text data is the title text of the target video, and the title text is denoted as text.

[0063] The semantic similarity between the i-th key frame image and the title text can be expressed by the following formula 1.

[0064] Sim i =M v-t (img i ,text) (Formula 1)

[0065] Among them, Sim i represents the semantic similarity between the key frame image and the title text of the i-th frame, M v-t Represents the CLIP model, img i Represents the i-th key frame image.

[0066] Whether the i-th key frame image is redundant data can be defined by the following formula 2.

[0067]

[0068] Among them, r i Indicates whether the i-th key frame image is redundant data, Sim i represents the semantic similarity between the key frame image of the i-th frame and the title text, and v represents the preset value of the similarity.

[0069] As an example, when the semantic information of the target image data is a first entity name and the semantic information of the target text data is a second entity name, the similarity can be determined based on the number of identical entity names contained in the first entity name and the second entity name. Specifically, the identical entity names existing in the first entity name and the second entity name can be defined as target entity names, and the ratio of the number of target entity names to the number of first entity names can be determined as the similarity.

[0070] For example, the first entity names extracted from the target image data are "noodles", "woman", "dog", and "house", and the second entity names extracted from the target text data are "woman" and "noodles". It can be seen that if the number of identical entity names in the first entity name and the second entity name is 2, that is, the number of target entity names is 2, and the ratio of the number of target entity names to the number of first entity names is 0.5, that is, the similarity is 0.5.

[0071] In actual use, when the first entity name and the second entity name have the same entity name, it is considered that the similarity is higher than a preset value, and the corresponding target image data is determined as redundant data. i Indicates whether the i-th picture is redundant data, which can be defined by the following formula 3.

[0072]

[0073] Among them, r i Indicates whether the i-th key frame image is redundant data, o i is the first entity name identified in the i-th key frame image, and E represents the sequence of the second entity names identified in the title text, denoted as E = [e1, e2, ..., e n ], o i in E means that the first entity name and the second entity name have the same entity name. i not in E indicates that the first entity name and the second entity name do not have the same entity name.

[0074] In an optional manner of the present disclosure, the target text data includes at least one of the following:

[0075] The video title of the target video;

[0076] Text information extracted from target image data;

[0077] The text information is recognized from the target audio data of the target video.

[0078] In the disclosed embodiment, the text contained in the video title of the target video may be used as the target text data.

[0079] In the disclosed embodiment, the target image data may contain some text information, such as subtitles contained in the frame image of the target video, etc. Specifically, the text information is extracted from the frame image of the target video by optical character recognition (OCR).

[0080] In the disclosed embodiment, the target audio data of the target video may contain voice audio data, and some text information can be extracted by performing voice recognition on the voice.

[0081] In an optional manner of the present disclosure, if the target text data includes text information identified from the target audio data of the target video, the above method further includes:

[0082] Performing voice detection on the target audio data to determine whether the target audio data contains voice data;

[0083] If included, the text information contained in the voice data is recognized.

[0084] In the embodiment of the present disclosure, when extracting text from target audio data, voice detection can be performed on the target audio data, and when it is determined that the target audio data contains voice audio data, the text information contained in the voice audio data is identified.

[0085] Specifically, whether the target audio data contains speech audio data can be identified based on a pre-trained speech detection model.

[0086] In an optional manner of the present disclosure, the above method further includes:

[0087] De-redundancy processing is performed on the multimodal data based on the redundant data to determine valid multimodal data in the multimodal data;

[0088] Perform multimodal fusion on effective multimodal data to obtain multimodal fusion features.

[0089] In the embodiment of the present disclosure, after the redundant data included in the multimodal data is determined, the multimodal data can be de-redundantly processed based on the redundant data. After the de-redundantly processing, the redundant data in the multimodal data can be greatly reduced or completely eliminated.

[0090] The effective multimodal data may be the data remaining after de-redundancy processing in the multimodal data. Since the redundant data is removed, the multimodal fusion features obtained by multimodal fusion based on the effective multimodal data are more accurate and effective. Video understanding based on the multimodal fusion features can improve the understanding effect of the video content.

[0091] By extracting multimodal features, the information contained is more comprehensive than that of unimodal features, which helps to understand video resources from multiple dimensions and improves the understanding of video content.

[0092] In an optional manner of the present disclosure, de-redundancy processing is performed on multimodal data based on redundant data, including:

[0093] Multimodal data is de-redundanted by reducing the self-attention weights corresponding to redundant data.

[0094] In the disclosed embodiment, a self-attention mechanism may be used to perform multimodal fusion on at least two types of unimodal data in the multimodal data.

[0095] To deduplicate multimodal data, the self-attention weight corresponding to the redundant data can be reduced, so that less attention can be paid to the redundant data during multimodal fusion, thereby eliminating the influence of redundant data in the multimodal fusion process.

[0096] In actual use, reducing the self-attention weight corresponding to redundant data can be achieved by setting a smaller value for the self-attention weight corresponding to the redundant data.

[0097] In an optional manner of the present disclosure, the above method further includes:

[0098] The target video is classified based on multimodal fusion features.

[0099] In the disclosed embodiment, since the multimodal fusion feature is obtained by multimodal fusion of effective multimodal data obtained by de-redundancy processing, it is more accurate and effective than the multimodal fusion feature obtained by directly performing multimodal fusion on the multimodal data without de-redundancy processing. It can make the video classification based on the multimodal fusion feature more accurate, thereby improving the accuracy of target video classification.

[0100] By extracting multimodal features, it helps to understand video resources from multiple dimensions and improve the understanding of video content. Multimodal data can be de-redundanted based on redundant data to eliminate the impact of redundant data, making the determined multimodal fusion features more accurate and effective, which can improve the video classification effect of the target video.

[0101] As an example, the specific process of performing multi-mode fusion on the target image data and the target text data to obtain the multi-mode fusion features is as follows:

[0102] Extract image features: Extract all key frame images of the target video, and use image expert models such as Resnet or Extreme Inception (Xception) to extract image features of key frame images. The image features of the i-th key frame image in the key frame image sequence can be obtained by express.

[0103] Extract text features: Use language models such as text expert models such as ERNIE or Bidirectional Encoder Representation from Transformers (BERT) to extract text features of target text data. The target text data is a video title. The characters in the video title form a text sequence according to their order in the video title. The text features of the i-th character can be obtained by express.

[0104] In addition, the definition is the identification vector of the single-modal data of the target image data, and defines is the identification vector of the target text data, which is the unimodal data. It is initialized by taking the maximum pooling of the feature vector sequence of each unimodal data. Definition is the identification vector of the expert model used to extract image features, and defines is the identification vector of the expert model used to extract text features. Define P i is the position feature vector, i represents the i-th position in the key frame image or text sequence.

[0105] Taking image features as an example, the feature vector sequence of image features can be expressed by the following formula 4.

[0106]

[0107] Among them, F represents the feature vector sequence of image features, is the identification vector of the target image data, The feature vector representing the image features of the first keyframe image, The feature vector representing the image features of the lth key frame image, l represents the length of the feature vector sequence of the image features.

[0108] The feature vector sequence of the expert model used to extract image features can be expressed by the following formula 5.

[0109]

[0110] Where M represents the feature vector sequence of the expert model used to extract image features. is the identification vector of the expert model used to extract image features, The feature vector representing the expert model used to extract the image features of the first key frame image, The feature vector of the expert model representing the image features of the lth key frame image extracted.

[0111] The position feature vector sequence can be expressed by the following formula 6.

[0112] P=[P agg , P1, ..., P l ] (Formula 6)

[0113] Among them, P represents the position feature vector sequence, P agg is the identification vector of the position feature, P1 represents the position feature vector of the first key frame image, P l Represents the position feature vector of the lth key frame image.

[0114] The final output feature vector can be expressed by the following formula 7.

[0115] H(i)=F(i)+M(i)+P(i) (Formula 7)

[0116] Among them, H represents the feature vector sequence of the final output, F represents the feature vector sequence of image features, M represents the feature vector sequence of the expert model used to extract image features, P represents the position feature vector sequence, and i represents the i-th position in the above feature vector sequences.

[0117] As an example, the multi-mode fusion used in the embodiments of the present disclosure can be implemented based on the Transformer model. The Transformer model interactively models image data and text data through a multi-head self-attention mechanism. The value of the attention matrix can be used to make the Transformer model pay less attention to redundant data, thereby eliminating the influence of redundant data in the multi-mode fusion modeling process.

[0118] The specific process of de-redundancy processing of multimodal data based on redundant data is as follows:

[0119] Define the image feature sequence as F img , the text feature sequence is F text , then the input sequence of the Transformer model is the concatenation of the image feature sequence and the image feature sequence. The input sequence can be expressed by the following formula eight.

[0120] F all =[F img ; F text] (Formula 8)

[0121] Among them, F all Represents the input sequence of the Transformer model, by transforming the image feature sequence into F img With the text feature sequence F text Get it by splicing.

[0122] In the self-attention module of the Transformer model, the conventional self-attention calculation formula is generally shown in Formula 9.

[0123]

[0124] Among them, Attention(Q, K, V) represents the value of self-attention, Q represents the matrix of query vector, K represents the matrix of key vector, V represents the matrix of value vector, and Q, K, V can be determined based on the input sequence. Represents the softmax function as the activation function.

[0125] In order to eliminate the influence of redundant data, the conventional self-attention formula can be modified as follows:

[0126] Assume q i is the i-th element in the Q sequence, then its similarity with the j-th element in the K sequence can be expressed by the following formula 10.

[0127] e ij =sim(q i , k j ) (Formula 10)

[0128] Among them, q i represents the i-th element in the Q sequence, k j represents the jth element in the K sequence, sim(q i , k j ) indicates q i , k j The similarity of ij .

[0129] q i The similarity value sequence with the sequence K can be expressed by the following formula 11.

[0130] E i =[e i1 , e i2 , ..., e in ] (Formula 11)

[0131] Among them, E i Indicates q i The similarity value sequence with sequence K, ei1 Indicates q i The similarity value with the first element in sequence K, e i2 Indicates q i The similarity value with the second element in sequence K, e in Indicates q i The similarity value with the nth element in sequence K, where n is the total number of elements in sequence K.

[0132] Obtain the redundant sequence R from the redundant data judgment module and use the redundant sequence R to i The re-assignment can be expressed by the following formula twelve.

[0133] E i ′=E i -R*1000 (Formula 12)

[0134] Among them, E i ′ represents the updated E i ,E i -R*1000 indicates the similarity value e of the corresponding position through the redundant sequence ij Subtract a larger value so that after the softmax function operation, the self-attention value of the corresponding position is approximately equal to 0. At this time, the Transformer model will pay very little attention to the redundant data, thereby eliminating the influence of redundant data in the multimodal fusion process and realizing the de-redundancy processing of multimodal data.

[0135] At this time, the modified self-attention calculation formula is shown in Formula 13.

[0136]

[0137] Among them, Attention(Q, K, V)′ represents the modified self-attention value, Q represents the matrix of query vectors, K represents the matrix of key vectors, V represents the matrix of value vectors, and Q, K, V can be determined based on the input sequence. represents the softmax function as the activation function. E represents the E i ’ is a sequence composed of .

[0138] As an example, the process of applying multi-modal fusion features to video classification is as follows:

[0139] remember is the input sequence F of the Transformer model all , the output sequence after the Transformer model encoding, takes the identification vectors of each single-mode output and concatenates them as the final output vector of the video, denoted as H f , H fIt can be expressed by the following formula 14.

[0140]

[0141] in, Represents the identity vector of single-modal data such as images, An identifier vector representing unimodal data such as text. It represents the concatenation operation of the identification vectors of the two unimodal data, image and text.

[0142] Through the fully connected layer of the video classification model, H f Mapping is performed and the predicted value is output according to the following formula 15

[0143]

[0144] in, represents the predicted value output by the video classification model, softmax(·) represents the softmax function, W represents the weight, and b represents the bias term.

[0145] The loss function of the video classification model can adopt the cross entropy loss function.

[0146] Figure 2 A schematic diagram of a process flow of another multimodal data processing method provided by an embodiment of the present disclosure is shown. Figure 2 As shown in , the method can mainly include:

[0147] Step S210: Acquire multimodal data of the target video;

[0148] Step S220: determining semantic information of at least two types of unimodal data in the multimodal data;

[0149] Step S230: determining redundant data in the multimodal data based on the similarity of semantic information;

[0150] Step S240: performing redundancy removal processing on the multimodal data based on the redundant data to determine valid multimodal data in the multimodal data;

[0151] Step S250: Perform multimodal fusion on the effective multimodal data to obtain multimodal fusion features.

[0152] The target video may be a video whose video content needs to be understood, and the multimodal data of the target video may include single-modal data such as images, audio, and text.

[0153] Each unimodal data of the target video will express some semantic information. When the semantic information expressed by each unimodal data in the target video has certain similarities, redundant data may exist in the multimodal data.

[0154] For example, the target video contains the text "A woman made a bowl of noodles, and the noodles were so delicious." Some images in the target video can express the semantic information of "woman" and "noodles." It can be seen that the semantic information expressed by some images of the target video is similar to the semantic information expressed by the text of the target video, which may cause some redundant data in the multimodal data of the target video.

[0155] Based on the similarity of the semantic information of each single-modal data, the redundant data in the multimodal data can be determined, which provides a basis for targeted processing of the redundant data in the subsequent multimodal fusion to ensure the effect of multimodal fusion.

[0156] The method provided by the embodiment of the present disclosure obtains multimodal data of the target video, determines the semantic information of at least two types of unimodal data in the multimodal data, and determines the redundant data in the multimodal data based on the similarity of the semantic information. In this solution, the redundant data in the multimodal data can be effectively determined based on the similarity of the semantic information of each unimodal data, which provides a basis for subsequent targeted processing of the redundant data, thereby ensuring the understanding effect of the video content.

[0157] In the embodiment of the present disclosure, after the redundant data included in the multimodal data is determined, the multimodal data can be de-redundantly processed based on the redundant data. After the de-redundantly processing, the redundant data in the multimodal data can be greatly reduced or completely eliminated.

[0158] The effective multimodal data may be the data remaining after de-redundancy processing in the multimodal data. Since the redundant data is removed, the multimodal fusion features obtained by multimodal fusion based on the effective multimodal data are more accurate and effective. Video understanding based on the multimodal fusion features can improve the understanding effect of the video content.

[0159] Based on Figure 1 The same principle as shown in the method, Figure 3 A schematic diagram of the structure of a multimodal data processing device provided by an embodiment of the present disclosure is shown, Figure 3 As shown, the multimodal data processing device 30 may include:

[0160] A multimodal data acquisition module 310, used to acquire multimodal data of a target video;

[0161] A semantic information determination module 320, used to determine semantic information of at least two types of unimodal data in the multimodal data;

[0162] The redundant data determination module 330 is used to determine redundant data in the multimodal data based on the similarity of semantic information.

[0163] The device provided by the embodiment of the present disclosure obtains multimodal data of the target video, determines the semantic information of at least two types of unimodal data in the multimodal data, and determines the redundant data in the multimodal data based on the similarity of the semantic information. In this solution, the redundant data in the multimodal data can be effectively determined based on the similarity of the semantic information of each unimodal data, which provides a basis for subsequent targeted processing of the redundant data, thereby ensuring the understanding effect of the video content.

[0164] Optionally, the at least two unimodal data include target image data and target text data.

[0165] The semantic information determination module is specifically used for:

[0166] Determining semantic information of target image data and semantic information of target text data;

[0167] The redundant data determination module is specifically used for:

[0168] Based on the similarity between the semantic information of the target image data and the semantic information of the target text data, redundant data in the target image data and / or redundant data in the target text data are determined.

[0169] Optionally, when determining the semantic information of the target image data and the semantic information of the target text data, the semantic information determination module is specifically used to:

[0170] The first semantic feature of the target image data in the semantic space is determined as the semantic information of the target image data, and the second semantic feature of the target text data in the semantic space is determined as the semantic information of the target text data.

[0171] Optionally, when determining the semantic information of the target image data and the semantic information of the target text data, the semantic information determination module is specifically used to:

[0172] The first entity name identified in the target image data is determined as the semantic information of the target image data, and the second entity name identified in the target text data is determined as the semantic information of the target text data.

[0173] Optionally, when the redundant data determination module determines the redundant data in the target image data and / or the redundant data in the target text data based on the similarity between the semantic information of the target image data and the semantic information of the target text data, it is specifically used to:

[0174] The target image data and / or the target text data whose similarity between the semantic information of the target image data and the semantic information of the target text data is greater than a preset value is determined as redundant data.

[0175] Optionally, the target text data includes at least one of the following:

[0176] The video title of the target video;

[0177] Text information extracted from target image data;

[0178] The text information is recognized from the target audio data of the target video.

[0179] Optionally, if the target text data includes text information recognized from target audio data of the target video, the above device further includes a speech recognition module, and the speech recognition module is used to:

[0180] Performing voice detection on the target audio data to determine whether the target audio data contains voice audio data;

[0181] If included, the text information contained in the voice audio data is identified.

[0182] Optionally, the above device further includes:

[0183] A redundancy removal module, used for performing redundancy removal processing on the multimodal data based on the redundant data, and determining valid multimodal data in the multimodal data;

[0184] The multimodal fusion module is used to perform multimodal fusion on effective multimodal data to obtain multimodal fusion features.

[0185] Optionally, when the redundancy removal module performs redundancy removal processing on the multimodal data based on the redundant data, it is specifically used to:

[0186] Multimodal data is de-redundanted by reducing the self-attention weights corresponding to redundant data.

[0187] Optionally, the above device further includes:

[0188] The video classification module is used to classify the target video based on multi-mode fusion features.

[0189] It can be understood that the above modules of the multimodal data processing device in the embodiment of the present disclosure have the function of realizing Figure 1The functions of the corresponding steps of the multimodal data processing method in the embodiment shown in . The function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions. The above modules can be software and / or hardware, and the above modules can be implemented separately or integrated with multiple modules. For the functional description of each module of the above multimodal data processing device, please refer to Figure 1 The corresponding description of the multimodal data processing method in the embodiment shown in will not be repeated here.

[0190] Based on Figure 2 The same principle as shown in the method, Figure 4 FIG. 4 shows a schematic diagram of the structure of another multimodal data processing device provided by an embodiment of the present disclosure, such as Figure 4 As shown, the multimodal data processing device 40 may include:

[0191] A multimodal data acquisition module 410, used to acquire multimodal data of a target video;

[0192] A semantic information determination module 420, configured to determine semantic information of at least two types of unimodal data in the multimodal data;

[0193] A redundant data determination module 430, configured to determine redundant data in the multimodal data based on similarity of semantic information;

[0194] A de-redundancy module 440 is used to perform de-redundancy processing on the multimodal data based on the redundant data to determine valid multimodal data in the multimodal data;

[0195] The multi-modal fusion module 450 is used to perform multi-modal fusion on the effective multi-modal data to obtain multi-modal fusion features.

[0196] The device provided by the embodiment of the present disclosure obtains multimodal data of the target video, determines the semantic information of at least two types of unimodal data in the multimodal data, and determines the redundant data in the multimodal data based on the similarity of the semantic information. In this solution, the redundant data in the multimodal data can be effectively determined based on the similarity of the semantic information of each unimodal data, which provides a basis for subsequent targeted processing of the redundant data, thereby ensuring the understanding effect of the video content.

[0197] Optionally, when the redundancy removal module performs redundancy removal processing on the multimodal data based on the redundant data, it is specifically used to:

[0198] Multimodal data is de-redundanted by reducing the self-attention weights corresponding to redundant data.

[0199] It can be understood that the above modules of the multimodal data processing device in the embodiment of the present disclosure have the function of realizing Figure 2 The functions of the corresponding steps of the multimodal data processing method in the embodiment shown in . The function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions. The above modules can be software and / or hardware, and the above modules can be implemented separately or integrated with multiple modules. For the functional description of each module of the above multimodal data processing device, please refer to Figure 2 The corresponding description of the multimodal data processing method in the embodiment shown in will not be repeated here.

[0200] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0201] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0202] The electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the multimodal data processing method provided in the embodiment of the present disclosure.

[0203] Compared with the prior art, the electronic device obtains the multimodal data of the target video, determines the semantic information of at least two types of unimodal data in the multimodal data, and determines the redundant data in the multimodal data based on the similarity of the semantic information. In this solution, the redundant data in the multimodal data can be effectively determined based on the similarity of the semantic information of each unimodal data, which provides a basis for subsequent targeted processing of the redundant data, thereby ensuring the understanding effect of the video content.

[0204] The readable storage medium is a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the multimodal data processing method provided in the embodiment of the present disclosure.

[0205] Compared with the prior art, the readable storage medium obtains the multimodal data of the target video, determines the semantic information of at least two types of unimodal data in the multimodal data, and determines the redundant data in the multimodal data based on the similarity of the semantic information. In this solution, the redundant data in the multimodal data can be effectively determined based on the similarity of the semantic information of each unimodal data, which provides a basis for subsequent targeted processing of the redundant data, thereby ensuring the understanding effect of the video content.

[0206] The computer program product includes a computer program, and when the computer program is executed by a processor, the computer program implements the multimodal data processing method provided in the embodiment of the present disclosure.

[0207] Compared with the prior art, this computer program product obtains multimodal data of the target video, determines semantic information of at least two types of unimodal data in the multimodal data, and determines redundant data in the multimodal data based on the similarity of the semantic information. In this solution, the redundant data in the multimodal data can be effectively determined based on the similarity of the semantic information of each unimodal data, which provides a basis for subsequent targeted processing of the redundant data, thereby ensuring the understanding effect of the video content.

[0208] Figure 5 A schematic block diagram of an example electronic device 50 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0209] like Figure 5 As shown, the electronic device 50 includes a computing unit 510, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 520 or a computer program loaded from a storage unit 580 into a random access memory (RAM) 530. In the RAM 530, various programs and data required for the operation of the device 50 can also be stored. The computing unit 510, the ROM 520, and the RAM 530 are connected to each other via a bus 540. An input / output (I / O) interface 550 is also connected to the bus 540.

[0210] A number of components in the device 50 are connected to the I / O interface 550, including: an input unit 560, such as a keyboard, a mouse, etc.; an output unit 570, such as various types of displays, speakers, etc.; a storage unit 580, such as a disk, an optical disk, etc.; and a communication unit 590, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 590 allows the device 50 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0211] The computing unit 510 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 510 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 510 performs the multimodal data processing method provided in the embodiments of the present disclosure. For example, in some embodiments, the execution of the multimodal data processing method provided in the embodiments of the present disclosure may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 580. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 50 via the ROM 520 and / or the communication unit 590. When the computer program is loaded into the RAM 530 and executed by the computing unit 510, one or more steps of the multimodal data processing method provided in the embodiments of the present disclosure may be performed. Alternatively, in other embodiments, the computing unit 510 may be configured in any other appropriate manner (for example, by means of firmware) to execute the multimodal data processing method provided in the embodiments of the present disclosure.

[0212] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: being implemented in one or more computer programs, which may be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general programmable processor, which may receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0213] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0214] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0215] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0216] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communications network). Examples of communications networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0217] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0218] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0219] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A multimodal data processing method, comprising: Acquire multimodal data of a target video, wherein the multimodal data includes at least target image data and target text data; Determine a first semantic feature of the target image data in a semantic space as semantic information of the target image data, and determine a second semantic feature of the target text data in the semantic space as semantic information of the target text data; Determine the target image data and / or target text data whose similarity between the semantic information of the target image data and the semantic information of the target text data is greater than a preset value as redundant data; By reducing the self-attention weight corresponding to the redundant data, the multimodal data is de-redundanted to obtain valid multimodal data in the multimodal data.

2. The method according to claim 1, wherein: The method further comprises: The first entity name identified in the target image data is determined as the semantic information of the target image data, and the second entity name identified in the target text data is determined as the semantic information of the target text data.

3. The method according to claim 1, wherein: The target text data includes at least one of the following: The video title of the target video; Text information extracted from the target image data; Text information recognized from target audio data of the target video.

4. The method according to claim 3, wherein: If the target text data includes text information identified from the target audio data of the target video, the method further includes: Performing voice detection on the target audio data to determine whether the target audio data contains voice data; If included, the text information contained in the voice data is identified.

5. The method according to any one of claims 1 to 4, further comprising: Multimodal fusion is performed on the effective multimodal data to obtain multimodal fusion features.

6. The method according to claim 5, further comprising: The target video is classified based on the multi-modal fusion features.

7. A multimodal data processing device, comprising: A multimodal data acquisition module, used to acquire multimodal data of a target video, wherein the multimodal data includes at least target image data and target text data; A semantic information determination module, used to determine a first semantic feature of the target image data in a semantic space as the semantic information of the target image data, and to determine a second semantic feature of the target text data in the semantic space as the semantic information of the target text data; A redundant data determination module, configured to determine target image data and / or target text data whose similarity between the semantic information of the target image data and the semantic information of the target text data is greater than a preset value as redundant data; A de-redundancy module is used to perform de-redundancy processing on the multimodal data by reducing the self-attention weight corresponding to the redundant data to obtain valid multimodal data in the multimodal data.

8. The device according to claim 7, wherein: The semantic information determination module is also used for: The first entity name identified in the target image data is determined as the semantic information of the target image data, and the second entity name identified in the target text data is determined as the semantic information of the target text data.

9. The device according to claim 7, wherein: The target text data includes at least one of the following: The video title of the target video; Text information extracted from the target image data; Text information recognized from target audio data of the target video.

10. The device according to claim 9, if the target text data includes text information recognized from the target audio data of the target video, the device further includes a speech recognition module, wherein the speech recognition module is used to: Performing voice detection on the target audio data to determine whether the target audio data contains voice audio data; If included, identify the text information contained in the voice audio data.

11. The device according to any one of claims 7 to 10, further comprising: The multimodal fusion module is used to perform multimodal fusion on the effective multimodal data to obtain multimodal fusion features.

12. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 6.

13. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-6.

14. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 6.

Images