Feature extraction method of multimedia data, multimedia data retrieval method and device

Abstract

The application discloses a feature extraction method of multimedia data, a multimedia data retrieval method and device. The feature extraction method of multimedia data comprises the following steps: performing image feature extraction on target multimedia data to obtain an image feature sequence of the target multimedia data; performing speech recognition on audio data in the target multimedia data, and performing text feature extraction on a speech recognition result to obtain a text feature sequence of the target multimedia data; fusing the image feature sequence and the text feature sequence to obtain a fused feature sequence; performing self-attention calculation on the fused feature sequence to obtain a target feature sequence; and determining a target feature of the target multimedia data according to the target feature sequence. The method can improve the accuracy of feature extraction of multimedia data.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multimedia, in particular to a multimedia data feature extraction method, a multimedia data retrieval method and device. BACKGROUND

[0002] In recent years, the Internet technology develops rapidly, and people's life has been inseparable from the Internet. With the maturation of Internet technology and communication technology, the multimedia industry has also ushered in a vigorous development, and various multimedia content providing applications such as videos and short videos have emerged in an endless stream.

[0003] In some cases, such as multimedia data deduplication or copyright protection, multimedia data needs to be retrieved.

[0004] The retrieval method for multimedia data in the related art mainly uses feature extraction and comparison methods for retrieval, but the accuracy of the multimedia data features extracted by the current multimedia data feature extraction method is poor, which leads to poor accuracy of the retrieval results of the current multimedia data retrieval method. SUMMARY

[0005] The embodiments of the present application provide a multimedia data feature extraction method, a multimedia data retrieval method and device, which can effectively improve the accuracy of multimedia data feature extraction, and further improve the accuracy of multimedia data retrieval results.

[0006] The first aspect of the present application provides a multimedia data feature extraction method, which comprises:

[0007] performing image feature extraction on the target multimedia data to obtain an image feature sequence of the target multimedia data;

[0008] performing speech recognition on the audio data in the target multimedia data, and performing text feature extraction on the speech recognition result to obtain a text feature sequence of the target multimedia data;

[0009] fusing the image feature sequence and the text feature sequence to obtain a fused feature sequence;

[0010] performing self-attention calculation on the fused feature sequence to obtain a target feature sequence;

[0011] determining a target feature of the target multimedia data according to the target feature sequence.

[0012] Correspondingly, the second aspect of the present application provides a multimedia data retrieval device, which comprises:

[0013] The first extraction unit is configured to perform image feature extraction on the target multimedia data to obtain an image feature sequence of the target multimedia data.

[0014] The second extraction unit is configured to perform speech recognition on audio data in the target multimedia data and perform text feature extraction on a speech recognition result to obtain a text feature sequence of the target multimedia data.

[0015] The fusion unit is configured to fuse the image feature sequence and the text feature sequence to obtain a fused feature sequence.

[0016] The first calculation unit is configured to perform self-attention calculation on the fused feature sequence to obtain a target feature sequence.

[0017] The first determination unit is configured to determine a target feature of the target multimedia data according to the target feature sequence.

[0018] In some embodiments, the fusion unit comprises:

[0019] The first processing subunit is configured to perform local pooling processing on the image feature sequence to obtain a target image feature sequence.

[0020] The second processing subunit is configured to perform local pooling processing on the text feature sequence to obtain a target text feature sequence.

[0021] The fusion subunit is configured to fuse the target image feature sequence and the target text feature sequence to obtain a fused feature sequence.

[0022] In some embodiments, the fusion subunit comprises:

[0023] The first acquisition module is configured to acquire a preset category token feature.

[0024] The splicing module is configured to splice the category token feature, the target image feature sequence, and the target text feature sequence to obtain a spliced feature sequence.

[0025] The second acquisition module is configured to acquire a position encoding sequence and a dimension encoding sequence corresponding to the spliced feature sequence.

[0026] The fusion module is configured to fuse the spliced feature sequence, the position encoding sequence, and the dimension encoding sequence to obtain a fused feature sequence.

[0027] In some embodiments, the first calculation unit comprises:

[0028] The first calculation subunit is configured to map the fusion feature sequence into a space of query features, key features, and value features to obtain a query feature sequence, a key feature sequence, and a value feature sequence, and calculate a correlation matrix of the query feature sequence and the key feature sequence.

[0029] The determination subunit is configured to determine a category fusion feature corresponding to the category token feature in the fusion feature sequence, an image fusion feature sequence corresponding to the target image feature sequence, and a text fusion feature sequence corresponding to the target text feature sequence.

[0030] The second calculation subunit is configured to calculate a self-attention weight matrix based on an interaction result of the category fusion feature and each feature in the fusion feature sequence, an interaction result between each feature in the image fusion feature sequence and each feature in the text fusion feature sequence, and the correlation matrix.

[0031] The third processing subunit is configured to perform weighted processing on the value feature sequence based on the self-attention weight matrix to obtain a target feature sequence.

[0032] In some embodiments, the first processing subunit includes:

[0033] The first division module is configured to divide image features in the image feature sequence into a plurality of sub-image feature sequences according to an image sampling sequence.

[0034] The first processing module is configured to perform average pooling processing on each sub-image feature sequence to obtain a plurality of first average pooling features, and the plurality of first average pooling features constitute a target image feature sequence.

[0035] In some embodiments, the second processing subunit includes:

[0036] The third acquisition module is configured to acquire target punctuation information in the speech recognition result.

[0037] The second division module is configured to divide the text feature sequence into a plurality of sub-text feature sequences according to the target punctuation information.

[0038] The second processing module is configured to perform average pooling processing on each sub-text feature sequence to obtain a plurality of second average pooling features, and the plurality of second average pooling features constitute a target text feature sequence.

[0039] In some embodiments, the first determination unit includes:

[0040] The extraction subunit is configured to extract a target category token feature corresponding to the category token feature from the target feature sequence.

[0041] The fourth processing subunit is configured to perform first residual connection and normalization processing on the target category token feature to obtain a first transition feature;

[0042] The fifth processing subunit is configured to perform processing on the first transition feature by using a preset feedforward neural network to obtain a second transition feature.

[0043] The sixth processing subunit is configured to perform second residual connection and normalization processing on the second transition feature to obtain a target feature of the target multimedia data.

[0044] In some embodiments, the first computing unit comprises:

[0045] The third computing subunit is configured to perform at least one self-attention computation on the fusion feature sequence to obtain at least one sub-target feature sequence.

[0046] The splicing subunit is configured to splice the at least one sub-target feature sequence to obtain a target feature sequence.

[0047] In some embodiments, the multimedia feature extraction device can be loaded with a multimedia feature extraction model, and the method is applied to the multimedia feature extraction model. The multimedia feature extraction model comprises an image feature extraction module, a text feature extraction module, and a multi-dimensional feature fusion module. The first extraction subunit is further configured to:

[0048] The image feature extraction module of the multimedia feature extraction model is used to perform feature extraction on the target multimedia data to obtain an image feature sequence of the target multimedia data.

[0049] The text feature extraction module of the multimedia feature extraction model is used to perform feature extraction on the target multimedia data to obtain a text feature sequence of the target multimedia data.

[0050] The fusion unit, the first computing unit, and the first determination unit are further configured to:

[0051] The multi-dimensional feature fusion module of the multimedia feature extraction model is used to fuse the image feature sequence and the text feature sequence to obtain a target feature of the target multimedia data.

[0052] In some embodiments, a training device for training the multimedia feature extraction model comprises:

[0053] The acquisition unit is configured to acquire a training sample set, and perform sample augmentation on a training sample in the training sample set to obtain an augmented training sample set.

[0054] a fourth extraction unit, configured to perform feature extraction on the training sample set by using the first neural network model to be trained to obtain a first multimedia feature sequence;

[0055] a fifth extraction unit, configured to perform feature extraction on the augmented training sample set by using the second neural network model to be trained to obtain a second multimedia feature sequence, the model structures of the first neural network model and the second neural network model being the same as that of the multimedia feature extraction model, and initial model parameters of the first neural network model and the second neural network model being the same;

[0056] a sampling unit, configured to update a preset augmented feature sequence based on the second multimedia feature sequence, and sample a preset number of multimedia features from the augmented feature sequence to obtain a third multimedia feature sequence;

[0057] a third calculation unit, configured to calculate a contrastive loss according to the first multimedia feature sequence, the second multimedia feature sequence, and the third multimedia feature sequence;

[0058] a training unit, configured to perform iterative training on the first neural network model and the second neural network model based on the contrastive loss to obtain a trained first neural network model, the trained first neural network model being a trained multimedia feature extraction model.

[0059] In some embodiments, the training unit comprises:

[0060] a first updating sub-unit, configured to update model parameters of the first neural network model based on back propagation of the contrastive loss;

[0061] a second updating sub-unit, configured to update parameters of the second neural network model by using a momentum updating method according to the updated first neural network model;

[0062] an execution sub-unit, configured to return to performing the steps of performing feature extraction on the training sample set by using the first neural network model and performing feature extraction on the augmented sample set by using the second neural network model until model parameters of the first neural network model converge.

[0063] In some embodiments, the obtaining unit comprises:

[0064] an obtaining sub-unit, configured to obtain a preset multimedia data set, the multimedia data set comprising a plurality of multimedia data;

[0065] a dividing sub-unit, configured to divide the multimedia data set into a first sub-set and a second sub-set based on a number of effective features in a text feature sequence of each multimedia data;

[0066] a sampling subunit configured to sample a preset number of multimedia data from the first subset and the second subset respectively to obtain a training sample set;

[0067] an augmenting subunit configured to perform sample augmentation on first samples in the training sample set by using a first sample augmentation method and perform sample augmentation on second samples in the training sample set by using a second sample augmentation method, the first samples being samples sampled from the first subset, and the second samples being samples sampled from the second subset.

[0068] The third aspect of the present application provides a multimedia data retrieval method, and the method comprises:

[0069] extracting first features of to-be-retrieved multimedia data and second features of each multimedia data in a candidate multimedia database by using any one of the multimedia data feature extraction methods provided in the first aspect;

[0070] calculating cosine similarity of the first features and each second feature;

[0071] determining target multimedia data similar to the to-be-retrieved multimedia data based on the cosine similarity.

[0072] Correspondingly, the fourth aspect of the present application provides a multimedia data retrieval device, and the device comprises:

[0073] a third extracting unit configured to extract first features of to-be-retrieved multimedia data and second features of each multimedia data in a candidate multimedia database by using any one of the multimedia data feature extraction methods provided in the first aspect;

[0074] a second calculating unit configured to calculate cosine similarity of the first features and each second feature;

[0075] a second determining unit configured to determine target multimedia data similar to the to-be-retrieved multimedia data based on the cosine similarity.

[0076] The fifth aspect of the present application further provides a computer readable storage medium, and the computer readable storage medium stores a plurality of instructions, the instructions being suitable for being loaded by a processor to execute steps in the multimedia data feature extraction method or the multimedia data retrieval method provided in the first aspect of the present application.

[0077] The sixth aspect of the present application provides a computer device, and the computer device comprises a memory, a processor and a computer program stored in the memory and capable of running on the processor, and the processor implements steps in the multimedia data feature extraction method or the multimedia data retrieval method provided in the first aspect of the present application when executing the computer program.

[0078] The seventh aspect of the present application provides a computer program product comprising computer programs / instructions which, when executed by a processor, implement the steps of the multimedia data feature extraction method or the multimedia data retrieval method provided in the first aspect of the present application.

[0079] The multimedia data feature extraction method provided by the embodiments of the present application comprises the following steps: image feature extraction is performed on target multimedia data to obtain an image feature sequence of the target multimedia data; speech recognition is performed on audio data in the target multimedia data, and text feature extraction is performed on a speech recognition result to obtain a text feature sequence of the target multimedia data; the image feature sequence and the text feature sequence are fused to obtain a fused feature sequence; self-attention calculation is performed on the fused feature sequence to obtain a target feature sequence; and a target feature of the target multimedia data is determined according to the target feature sequence.

[0080] Therefore, the multimedia data feature extraction method provided by the present application extracts multi-dimensional features from the image features of the multimedia data and the text features corresponding to the speech recognition result of the audio data in the multimedia data; then the multi-dimensional features are fused and further self-attention processing is performed on the fused features, and the target feature for describing the multimedia data is determined based on the self-attention processing result. Since the target feature which fuses the image features of the multimedia data and the text features corresponding to the audio data is used to describe the multimedia data, the description of the target feature to the multimedia data is more accurate, thereby improving the accuracy of feature extraction of the multimedia data. BRIEF DESCRIPTION OF DRAWINGS

[0081] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0082] Figure 1a is a scene diagram of multimedia data feature extraction in the present application;

[0083] Figure 1b is another scene diagram of multimedia data feature extraction in the present application;

[0084] Figure 2 is a flow diagram of the multimedia data feature extraction method provided by the present application;

[0085] Figure 3 is a model structure diagram of the multimedia feature extraction model provided by the present application;

[0086] Figure 4 is a structural schematic diagram of a multi-dimensional feature fusion module of a multimedia feature extraction model provided by the present application;

[0087] Figure 5 is another flow schematic diagram of a multimedia data feature extraction method provided by the present application;

[0088] Figure 6 is a flow schematic diagram of a multimedia data retrieval method provided by the present application;

[0089] Figure 7 is a structural schematic diagram of a multimedia data retrieval device provided by the present application;

[0090] Figure 8 is a structural schematic diagram of a computer device provided by the present application. DETAILED DESCRIPTION

[0091] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person skilled in the art without creative labor fall within the scope of protection of the present application.

[0092] Please refer to Figure 1a is a scene schematic diagram of a multimedia data feature extraction method provided by the present application. As shown in the figure, image and text features are extracted from multimedia data to obtain image feature sequences and text feature sequences. Then, the image feature sequences and the text feature sequences are fused to obtain fused feature sequences. Then, self-attention calculation is performed on the fused feature sequences to obtain target feature sequences, and target features of the multimedia data are determined based on the target feature sequences, thereby completing the feature extraction of the multimedia data.

[0093] Please refer to Figure 1b is another scene schematic diagram of a multimedia data feature extraction method provided by the present application. In this scene, the multimedia data can be video data. As shown in the figure, terminal B sends target video data that needs to perform video feature extraction to server A, and server A receives the target video data sent by terminal B, extracts image features of the target video data to obtain image feature sequences of the target video data.

[0094] Further, server A performs speech recognition on audio data in the target video data to obtain a speech recognition result, and performs text feature extraction on the speech recognition result to obtain text feature sequences of the target video data.

[0095] Then, the server A continues to fuse the extracted image feature sequence and the text feature sequence to obtain a fused feature sequence. Then, the server A performs self-attention calculation on the fused feature sequence to obtain a target feature sequence. After obtaining the target feature sequence of the target video, the server A further determines the target feature of the target video data according to the target feature sequence.

[0096] After extracting the target feature of the target video data, the server A can send the target feature to the terminal B. The terminal B can further perform further retrieval operation on the target video data according to the target feature.

[0097] It should be noted that, Figure 1a and Figure 1b The scene diagrams of the multimedia data feature extraction shown in FIGS. 1 to 3 are only two examples, and the scene of the multimedia data feature extraction described in the embodiments of the present application is to more clearly illustrate the technical solutions of the present application, and does not constitute a limitation on the technical solutions provided by the present application. Those skilled in the art can know that, with the evolution of the scene of multimedia data feature extraction and the appearance of new business scenes, the technical solutions provided by the present application are also applicable to similar technical problems.

[0098] Based on the above implementation scenarios, the following will be described in detail.

[0099] In the related art, in the scene of searching, comparing, querying and classifying multimedia data, it is necessary to extract features of the multimedia data. At present, when extracting features of the multimedia data, only image features of the multimedia data are extracted, and other dimensional features of the multimedia data are not considered. Therefore, due to the single dimension of the multimedia data feature extraction, the extracted multimedia data features have insufficient description ability for the multimedia data, that is, the accuracy of the extracted multimedia data features is not high. Specifically, for example, when the multimedia data is video data, and the content of the video data is online teaching content, when the same teacher teaches different content, it is difficult to distinguish different videos only by extracting image features from the video data.

[0100] Therefore, the present application provides a multimedia data feature extraction method, which can extract features of multimedia data from two dimensions of image features and text features, thereby improving the description ability of the extracted multimedia data features for the multimedia data, that is, improving the accuracy of the multimedia data feature extraction.

[0101] Embodiments of the present application will be described from the perspective of a multimedia data feature extraction device, which can be integrated in a computer device. The computer device can be a terminal or a server. The terminal can be a mobile phone, a tablet computer, a notebook computer, a smart television, a wearable smart device, a personal computer (PC), a vehicle terminal, etc. The server can be a standalone physical server, a server cluster or a distributed system composed of multiple physical servers, a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery network (CDN), and big data and artificial intelligence platforms, etc. basic cloud computing services. As shown in Figure 2 The flowchart of the multimedia data feature extraction method provided by the present application is shown in FIG. 1. The method comprises the following steps:

[0102] In step 101, image features of target multimedia data are extracted to obtain an image feature sequence of the target multimedia data.

[0103] The target multimedia data is multimedia data that needs to be extracted. The multimedia data can be video data, including but not limited to long video data and short video data. The multimedia data can also be other multimedia data containing images and audio, such as multimedia data generated by editing multiple frames of the same image and audio data.

[0104] The image features of the target multimedia data can be extracted by first sampling the image frames of the multimedia data to obtain multiple images. The image frames of the target multimedia data can be sampled according to a preset sampling frequency, wherein the preset sampling frequency can be set by the user, for example, one frame per second. Alternatively, the image frames can be sampled without a preset sampling frequency, for example, the sampling frequency can be adjusted by the multimedia data feature extraction device according to the similarity between the image frames.

[0105] Whether the image frames of the target multimedia data are sampled according to a preset sampling frequency or not, the multimedia data can be sampled to obtain multiple images, where the multiple images can be at least one image. Then, the features of the image dimension of the target multimedia data can be extracted based on the multiple images.

[0106] The image feature extraction on the multiple frames of images can be the extraction of the gray scale feature of each frame of image, the extraction of the texture feature of each frame of image, the extraction of the shape feature of each frame of image, or the extraction of the invariant moment feature of each frame of image. Of course, the image feature can also be other features that can distinguish between images. The image feature extraction on different frames of images can be performed by using the same image feature extraction method or multiple image feature extraction methods.

[0107] When the gray scale feature extraction is performed on the multiple frames of images, the gray scale average value or the gray scale variance method can be used for the extraction. When the texture feature extraction is performed on the multiple frames of images, the gray scale difference statistics method, the gray scale co-occurrence matrix, or the gray scale-gradient co-occurrence matrix method can be used for the extraction. When the shape feature extraction is performed on the multiple frames of images, the corresponding invariant moment feature extraction algorithm can be used for the extraction.

[0108] The image features of the multiple frames of images are extracted by using the above method, and the image features corresponding to each frame of image are obtained. The image features constitute the image feature sequence of the multimedia data. The extracted image frame features can be a feature vector or a feature matrix.

[0109] In the embodiments of the present application, a single image feature extraction model can be used for the image feature extraction on the multimedia data, or a multimedia feature extraction model including an image feature extraction module can be used for the image feature extraction on the multimedia data.

[0110] In step 102, the speech recognition is performed on the audio data in the target multimedia data, and the text feature extraction is performed on the speech recognition result, so as to obtain the text feature sequence of the target multimedia data.

[0111] In the embodiments of the present application, the feature extraction can be performed not only on the image dimension of the multimedia data, but also on the audio dimension of the multimedia data.

[0112] Specifically, the audio data of the multimedia data can be acquired from the audio stream of the multimedia data. Then, the automatic speech recognition (ASR) is performed on the audio data, so as to obtain the speech recognition result. The speech recognition result is output in the form of text, and the text data corresponding to the audio data is obtained.

[0113] Then, the text data can be segmented according to a preset segmentation method, so as to obtain multiple words. The preset segmentation method can be the segmentation according to the text semantics or the segmentation according to the character unit.

[0114] The word feature extraction is performed on each word, the word embedding method can be used to map each word to a vector space to obtain the word feature of each word. The identification data corresponding to each word can also be obtained from the preset mapping relationship table, and then the word feature extraction is performed based on the identification data to obtain the word feature of each word. The word features of these words constitute the text feature sequence of the target multimedia data.

[0115] Similarly, in the embodiments of the present application, a separate text feature extraction model can be used to extract the text feature sequence of the multimedia data, or a multimedia data feature extraction model containing a text feature extraction module can be used to perform text feature extraction on the multimedia data to obtain the text feature sequence of the multimedia data.

[0116] In step 103, the image feature sequence and the text feature sequence are fused to obtain a fused feature sequence.

[0117] The image feature sequence corresponding to the image feature of the multimedia data and the text feature sequence corresponding to the text feature are fused, which can be a certain operation or a certain splicing operation on the two feature sequences to obtain a new feature sequence, and then the target feature of the target multimedia data is determined according to the new feature sequence. The target feature can describe the multimedia data from two dimensions of image and audio, and the representation of the multimedia data is more accurate.

[0118] In some embodiments, the image feature sequence and the text feature sequence are fused to obtain a fused feature sequence, including:

[0119] 1. The image feature sequence is locally pooled to obtain a target image feature sequence;

[0120] 2. The text feature sequence is locally pooled to obtain a target text feature sequence;

[0121] 3. The target image feature sequence and the target text feature sequence are fused to obtain a fused feature sequence.

[0122] In the embodiments of the present application, the image feature sequence and the text feature sequence of the multimedia data can be locally pooled respectively, which can reduce the number of features in the image feature sequence and the text feature sequence, reduce the amount of calculation, and thus improve the feature extraction efficiency of the multimedia data.

[0123] Specifically, the image feature sequence is locally pooled to obtain a target image feature sequence; and the text feature sequence is locally pooled to obtain a target text feature sequence. Then, the target image feature sequence and the target text feature sequence can be fused to obtain a fused feature sequence. This method greatly reduces the number of features in the fused feature sequence, reduces the operation efficiency, and improves the multimedia data feature extraction efficiency.

[0124] The pooling processing can be weighted pooling, average pooling, etc.

[0125] In some embodiments, the image feature sequence is locally pooled to obtain a target image feature sequence, including:

[0126] 1.1, dividing the image features in the image feature sequence into a plurality of sub-image feature sequences according to the image sampling order;

[0127] 1.2, performing average pooling processing on each sub-image feature sequence to obtain a plurality of first average pooling features, and the plurality of first average pooling features constitute the target image feature sequence.

[0128] In the embodiments of the present application, the average pooling method can be used for local pooling of the image feature sequence. Specifically, the image feature sequence can be first divided into a plurality of sub-image feature sequences, and each sub-image feature sequence can contain the same number of image features. Then, average pooling is performed on each sub-image feature sequence to obtain the average pooling feature corresponding to each sub-image feature sequence. For distinction from other average pooling features in the present application, this can be referred to as a first average pooling feature. The plurality of first average pooling features reconstitute a feature sequence to obtain the target image feature sequence.

[0129] For example, the image feature sequence contains 100 image features, and the 100 image features are sorted in the image feature sequence according to the sampling order of the images corresponding to the image features. In this way, the 100 image features can be sequentially divided into 10 sub-image feature sequences, and each sub-image feature sequence contains 10 sequentially continuous image features. The 10 image features in each sub-image feature sequence are subjected to average pooling to obtain a first average pooling feature corresponding to each sub-image feature sequence. The plurality of first average pooling features constitute a new image feature sequence, i.e., the target image feature sequence.

[0130] In some embodiments, the text feature sequence is locally pooled to obtain a target text feature sequence, including:

[0131] 2.1, obtaining target punctuation information in the speech recognition result;

[0132] 2.2, dividing the text feature sequence into a plurality of sub-text feature sequences according to the target punctuation information;

[0133] 2.3, average pooling processing is performed on each sub-text feature sequence to obtain a plurality of second average pooling features, and the plurality of second average pooling features constitute the target text feature sequence.

[0134] In the embodiment of the present application, the average pooling is performed on the text feature sequence, which is no longer average pooling according to the equal amount of features, but according to the punctuation in the text to determine the features for average pooling. Specifically, since in the text, different sentences are generally divided by periods. Therefore, the period in the text data obtained by speech recognition can be searched first to obtain the position of the period in the text. Then, the text feature sequence is divided into a plurality of sub-text feature sequences according to the positions of the periods, and the word features corresponding to different words in a sentence belong to the same sub-text feature sequence.

[0135] Then, the plurality of word features in each sub-text feature sequence are further subjected to average pooling processing to obtain a second average pooling feature corresponding to each sub-text feature sequence, and a plurality of second average pooling features constitute the target text feature sequence.

[0136] In some embodiments, the target image feature sequence and the target text feature sequence are fused to obtain a fusion feature sequence, including:

[0137] 3.1, obtaining a preset category token feature;

[0138] 3.2, splicing the category token feature, the target image feature sequence and the target text feature sequence to obtain a spliced feature sequence;

[0139] 3.3, obtaining a position encoding sequence and a dimension encoding sequence corresponding to the spliced feature sequence;

[0140] 3.4, fusing the spliced feature sequence, the position encoding sequence and the dimension encoding sequence to obtain a fusion feature sequence.

[0141] The preset category token feature can be a feature that assists in feature fusion, and the feature can be a randomly generated and trained parameter.

[0142] Splicing the category token feature, the target image feature sequence and the target text feature sequence can combine the token feature, the target image feature sequence and the target text feature sequence into a sequence containing all the features. For example, when the target image feature sequence contains m features and the target text feature sequence contains n features, the spliced feature sequence obtained by splicing can include m+n+1 features. These features can be arranged in a certain order in the spliced feature sequence, or can be arranged in any order.

[0143] Further, the position encoding sequence corresponding to the spliced feature sequence and the dimension encoding sequence can be obtained. The number of encodings in the position encoding sequence is the same as the number of features in the spliced feature sequence, and the position encoding is encoding data for describing the position of the feature in the spliced feature sequence. The number of encodings in the dimension encoding sequence is also the same as the number of features in the spliced feature sequence, and the dimension encoding is encoding data for describing the dimension of the feature in the spliced feature sequence. It can be understood that, since the position of each feature in the spliced feature sequence is different, each position encoding data in the position encoding sequence is also different. The spliced feature sequence contains features of three dimensions, so the dimension encoding sequence contains three kinds of dimension encoding data. Specifically, the dimension encoding data corresponding to the category token feature can be set to 0, the dimension encoding data corresponding to the image feature can be set to 1, and the dimension encoding data corresponding to the text feature can be set to 2. Here, only an example is given, and the dimension encoding data can also be a dimension encoding vector or other forms of expression.

[0144] The spliced feature sequence, the position encoding sequence, and the dimension encoding sequence are fused to perform weighted processing on the feature dimensions of the three sequences to obtain a fused feature sequence.

[0145] In the embodiments of the present application, the process of fusing the image feature sequence and the text feature sequence can be implemented by an independent feature fusion model, or the image feature sequence and the text feature sequence can be fused by a multimedia feature extraction model containing a feature fusion module to obtain a fused feature sequence.

[0146] In the embodiments of the present application, the process of fusing the image feature sequence and the text feature sequence can be implemented by an independent feature fusion model, or the image feature sequence and the text feature sequence can be fused by a multimedia feature extraction model containing a feature fusion module to obtain a fused feature sequence.

[0147] In the embodiments of the present application, the process of fusing the image feature sequence and the text feature sequence can be implemented by an independent feature fusion model, or the image feature sequence and the text feature sequence can be fused by a multimedia feature extraction model containing a feature fusion module to obtain a fused feature sequence.

[0148] In some embodiments, the self-attention calculation is performed on the fused feature sequence to obtain a target feature sequence, including:

[0149] 1. At least one self-attention calculation is performed on the fused feature sequence to obtain at least one sub-target feature sequence;

[0150] 2. The at least one sub-target feature sequence is spliced to obtain a target feature sequence.

[0151] In the embodiments of the present application, the fusion feature sequence can be processed by using a multi-head self-attention structure, that is, the fusion feature sequence is subjected to multiple self-attention calculations to obtain multiple fusion feature sequences after self-attention calculation, which can be referred to as sub-target feature sequences.

[0152] Further, the multiple sub-target feature sequences obtained by multiple self-attention calculations can be combined to obtain a target feature sequence. The combination of the multiple sub-target feature sequences can be splicing of the multiple sub-target feature sequences.

[0153] In some embodiments, the fusion feature sequence is subjected to self-attention calculation to obtain a target feature sequence, including:

[0154] A, mapping the fusion feature sequence into a query feature, a key feature and a value feature space to obtain a query feature sequence, a key feature sequence and a value feature sequence, and calculating a correlation matrix of the query feature sequence and the key feature sequence;

[0155] B, determining a category fusion feature corresponding to a category token feature in the fusion feature sequence, an image fusion feature sequence corresponding to a target image feature sequence, and a text fusion feature sequence corresponding to a target text feature sequence;

[0156] C, calculating a self-attention weight matrix based on an interaction result of the category fusion feature and each feature in the fusion feature sequence, an interaction result between each feature in the image fusion feature sequence and each feature in the text fusion feature sequence, and the correlation matrix;

[0157] D, performing weighted processing on the value feature sequence based on the self-attention weight matrix to obtain a target feature sequence.

[0158] In the embodiments of the present application, the fusion feature sequence is subjected to self-attention calculation, which can first map each feature in the fusion feature sequence into a feature space corresponding to a query feature, a key feature and a value feature to obtain a query feature sequence, a key feature sequence and a value feature sequence. Then, a correlation matrix of the query feature sequence and the key feature sequence is calculated. Specifically, a first matrix corresponding to the query feature sequence and a second matrix corresponding to the key feature sequence can be determined, and then a matrix product between a transpose matrix of the first matrix and the second matrix is calculated to obtain the correlation matrix.

[0159] As described above, the fused feature sequence can be a feature sequence obtained by fusing the spliced feature sequence with the position encoding sequence and the dimension encoding sequence, and the spliced feature sequence is obtained by splicing the class token feature with the target image feature and the target text feature. Therefore, the fused feature sequence contains the class fusion feature corresponding to the class token feature, the image fusion feature sequence corresponding to the target image feature sequence, and the text fusion feature sequence corresponding to the target text feature sequence.

[0160] Then, the self-attention weight matrix is calculated based on the interaction result of the class fusion feature and each feature in the fused feature sequence, the interaction result between each feature in the image fusion feature sequence and each feature in the text fusion feature sequence, and the aforementioned correlation matrix.

[0161] Specifically, the self-attention mask matrix can be calculated based on the interaction result of the class fusion feature and each feature in the fused feature sequence, the interaction result between each feature in the image fusion feature sequence and each feature in the text fusion feature sequence, and then the self-attention weight matrix is calculated based on the self-attention mask matrix and the correlation matrix.

[0162] Further, the value feature matrix can be weighted using the self-attention weight matrix to obtain the target feature sequence.

[0163] Step 105, determining the target feature of the target multimedia data according to the target feature sequence.

[0164] Wherein, after the self-attention calculation on the fused feature sequence to obtain the target feature sequence, the target feature of the target multimedia data can be further calculated based on the target feature sequence.

[0165] Specifically, in some embodiments, determining the target feature of the target multimedia data according to the target feature sequence includes:

[0166] 1. Extracting the target class token feature sequence corresponding to the class token feature from the target feature sequence;

[0167] 2. Performing first residual connection and normalization processing on the target class token feature sequence to obtain a first transition feature;

[0168] 3. Processing the first transition feature using a pre-set feedforward neural network to obtain a second transition feature;

[0169] 4. Performing second residual connection and normalization processing on the second transition feature to obtain the target feature of the target multimedia data.

[0170] Wherein, the target category token feature corresponding to the category token feature is extracted from the target feature sequence, that is, the category token feature sequence after the cross-latitude information interaction is calculated. Then, the category token feature sequence is first added to the fusion feature sequence through a residual link, then the added result is normalized, then the normalized result is processed through a feedforward neural network, and then residual link and normalization processing are performed again to obtain the target feature of the target multimedia data.

[0171] In some embodiments, the feature extraction method of the multimedia data provided by the present application can be applied to a multimedia feature extraction model, which includes an image feature extraction module, a text feature extraction module, and a multi-dimensional feature fusion module. Then, the image feature extraction of the target multimedia data is performed to obtain the image feature sequence of the target multimedia data, which can be:

[0172] The image feature extraction module of the multimedia feature extraction model extracts the features of the target multimedia data to obtain the image feature sequence of the target multimedia data;

[0173] The speech recognition of the audio data in the target multimedia data is performed, and the text feature extraction of the speech recognition result is performed to obtain the text feature sequence of the target multimedia data, which can be:

[0174] The text feature extraction module of the multimedia feature extraction model extracts the features of the target multimedia data to obtain the text feature sequence of the target multimedia data;

[0175] The image feature sequence and the text feature sequence are fused to obtain a fusion feature sequence, and the target feature sequence is obtained by performing self-attention calculation on the fusion feature sequence. The target feature of the target multimedia data can be determined according to the target feature sequence, which can be:

[0176] The multi-dimensional feature fusion module of the multimedia feature extraction model fuses the image feature sequence and the text feature sequence to obtain the target feature of the target multimedia data.

[0177] As shown in Figure 3 , it is a model structure diagram of the multimedia feature extraction model provided by the present application. The image feature sequence is extracted from the multimedia data by the image feature extraction module in the multimedia feature extraction model, the text feature sequence is extracted by the text feature extraction module, and then the text feature sequence and the image feature sequence are input into the multi-dimensional feature fusion module of the multimedia feature extraction model for fusion, and the multimedia feature is output. This multimedia feature is the target feature of the multimedia data.

[0178] As shown in Figure 4As shown, it is a structural schematic diagram of a multi-dimensional feature fusion module of a multimedia feature extraction model provided in the present application. The multi-dimensional feature fusion module includes a multi-dimensional transformation encoder, which includes a multi-head self-attention layer, two residual connections and a normalization layer, and a feedforward neural network layer. When the image feature sequence and the text feature sequence are input into the multi-dimensional feature fusion module, the multi-dimensional feature fusion module first fuses the image feature sequence, the text feature sequence and the preset category token feature into a fused feature sequence, and generates a corresponding position encoding sequence and a dimension encoding sequence for the fused feature sequence, and then inputs the fused feature sequence, the position encoding sequence and the dimension encoding sequence into the multi-dimensional transformation encoder for further feature fusion to obtain the multimedia feature, i.e., the target feature of the target multimedia data.

[0179] In order to enable the above multimedia feature extraction model to extract more accurate multimedia features, i.e., to enable the multimedia feature extraction model to have better model effect, model training needs to be performed on the multimedia feature extraction model.

[0180] In the embodiments of the present application, the training of the above multimedia feature extraction model can be performed using a momentum contrast learning method. The momentum contrast learning (MoCo) is a self-supervised learning framework. The model training method includes the following steps:

[0181] A, obtain a training sample set, and perform sample augmentation on the training samples in the training sample set to obtain an augmented training sample set;

[0182] B, using a first neural network model to be trained to extract features of the training sample set to obtain a first multimedia feature sequence;

[0183] C, using a second neural network model to be trained to extract features of the augmented training sample set to obtain a second multimedia feature sequence, the model structure of the first neural network model and the second neural network model being the same as that of the multimedia feature extraction model, and the initial model parameters of the first neural network model and the second neural network model being the same;

[0184] D, updating the preset augmented feature sequence based on the second multimedia feature sequence, and sampling a preset number of multimedia features from the augmented feature sequence to obtain a third multimedia feature sequence;

[0185] E, calculating a contrastive loss according to the first multimedia feature sequence, the second multimedia feature sequence and the third multimedia feature sequence;

[0186] F, iteratively training the first neural network model and the second neural network model based on a contrastive loss, to obtain a trained first neural network model, the trained first neural network model being a trained multimedia feature extraction model.

[0187] In the embodiments of the present application, a self-supervised model training method based on the MoCo framework is used to train the multimedia feature extraction model. Specifically, a training sample set is first obtained, and the training sample set includes a plurality of multimedia data. Then, the training sample set is augmented to obtain an augmented training sample set. Then, the multimedia feature extraction model is trained using the training sample set and the augmented training sample set.

[0188] In some embodiments, the training sample set is obtained, and the training samples in the training sample set are augmented to obtain an augmented training sample set, including:

[0189] A1, obtaining a preset multimedia data set, the multimedia data set including a plurality of multimedia data;

[0190] A2, dividing the multimedia data set into a first sub-set and a second sub-set based on the number of effective features in the text feature sequence of each multimedia data;

[0191] A3, sampling a preset number of multimedia data from the first sub-set and the second sub-set respectively to obtain a training sample set;

[0192] A4, using a first sample augmentation method to augment the first samples in the training sample set, and using a second sample augmentation method to augment the second samples in the training sample set, the first samples being samples sampled from the first sub-set, and the second samples being samples sampled from the second sub-set.

[0193] In the embodiments of the present application, the training sample data can be sampled from the preset multimedia data set. Specifically, the text feature sequence of each multimedia data in the multimedia data set can be extracted first, which can be extracted by the text feature extraction module of the multimedia feature extraction model provided in the present application, or can be extracted by other methods.

[0194] Then, the number of effective features in the text feature sequence corresponding to each multimedia data is obtained. When the number of effective features is greater than a preset first number, the corresponding multimedia data is determined to form a first sub-set; when the number of effective features is less than a preset second number, the corresponding multimedia data is determined to form a second sub-set.

[0195] Then, a third preset number of multimedia data are sampled in the first and second subsets respectively to form a training sample set. The multimedia data sampled from the first subset are first samples, and the multimedia data sampled from the second subset are second samples. That is, the training sample set contains a plurality of first samples and the same number of second samples.

[0196] Further, the training sample set can be augmented, which can be separately augmented for the first samples and the second samples. In the embodiments of the present application, two different data augmentation methods can be used to separately augment the first samples and the second samples. Specifically, augmenting the samples can include two parts of image augmentation and text augmentation. For the first samples, random color transformation, affine transformation, and local erasing processing of image frames can be used for image augmentation, and low proportion text discarding and random noise adding can be used for text augmentation. For the second samples, random color transformation, affine transformation, and local erasing processing of image frames can also be used for image augmentation, but when augmenting the text, random exchange of text data of all samples in the text can be used to achieve the augmentation.

[0197] The number of augmented samples is the same as the number of samples in the training sample set, and the samples in the training sample set and the samples in the augmented sample set form a positive sample pair one by one.

[0198] Then, two multimedia feature extraction models with the same initial model parameters but different model parameters are used to extract multimedia features from the samples in the training sample set and the augmented sample set respectively. The two multimedia feature extraction models are neural network models with the structure of Figure 3 , which are denoted as a first neural network model and a second neural network model, or a first multimedia feature extraction model and a second multimedia feature extraction model.

[0199] Specifically, the first multimedia feature extraction model is used to extract features from the training sample set to obtain a first multimedia feature sequence, and the second multimedia feature extraction model is used to extract features from the augmented sample set to obtain a second multimedia feature sequence.

[0200] Then, the augmented feature sequence is initialized, where the augmented feature sequence is a sequence of features used to extract negative feature pairs with the first multimedia feature sequence. The augmented feature sequence initially does not have features, and the second multimedia feature sequence can be used to update it, that is, the second multimedia feature sequence is added to the augmented feature sequence. Then, in the process of cyclic training of the model, the second multimedia feature sequence generated in the current cycle is added to the augmented feature sequence each time to continuously update the augmented feature sequence.

[0201] After the first multimedia feature sequence of the training sample and the second multimedia feature sequence of the augmented sample are extracted respectively, an augmented feature sequence can be sampled from the augmented sample feature sequence, which can be referred to as a third multimedia feature sequence. The features in the third multimedia feature sequence and the features in the first multimedia feature sequence form a negative feature pair.

[0202] Further, a contrastive loss can be calculated according to the first multimedia feature sequence, the second multimedia feature sequence, and the third multimedia feature sequence. Then, the first multimedia feature extraction model and the second multimedia feature extraction model are iteratively trained based on the contrastive loss to obtain the trained first multimedia feature extraction model and the trained second multimedia feature extraction model. The first multimedia feature extraction model is the trained multimedia feature extraction model.

[0203] In some embodiments, iteratively training the first neural network model and the second neural network model based on the contrastive loss comprises:

[0204] F1, updating the model parameters of the first neural network model by backpropagation based on the contrastive loss;

[0205] F2, updating the parameters of the second neural network model using a momentum update method according to the updated first neural network model;

[0206] F3, returning to the step of performing feature extraction on the training sample set using the first neural network model and performing feature extraction on the augmented sample set using the second neural network model until the model parameters of the first neural network model converge.

[0207] Wherein, the obtained contrastive loss can be solved by gradient descent, and the model parameters in the first multimedia feature extraction model are updated by backpropagation according to the result. At this time, the model parameters in the second multimedia feature extraction model are not updated synchronously with the model parameters of the first multimedia feature extraction model, but the parameters of the second neural network model are updated using a momentum update method. Then, return to the step of performing feature extraction on the training sample set and the augmented sample set, and iterate until the first multimedia feature extraction model and the second multimedia feature extraction model converge.

[0208] According to the above description, the method for extracting features of multimedia data provided in the embodiments of the present application can obtain an image feature sequence of the target multimedia data by performing image feature extraction on the target multimedia data; perform speech recognition on the audio data in the target multimedia data, and perform text feature extraction on the speech recognition result to obtain a text feature sequence of the target multimedia data; fuse the image feature sequence and the text feature sequence to obtain a fused feature sequence; perform self-attention calculation on the fused feature sequence to obtain a target feature sequence; and determine a target feature of the target multimedia data according to the target feature sequence.

[0209] Therefore, the method for extracting features of multimedia data provided in the present application extracts multi-dimensional features from the image features of the multimedia data and the text features corresponding to the speech recognition result of the audio data in the multimedia data; then fuses the multi-dimensional features and further performs self-attention processing on the fused features, and determines a target feature for describing the multimedia data based on the self-attention processing result. Since the target feature that fuses the image features of the multimedia data and the text features corresponding to the audio data is used to describe the multimedia data, the description of the target feature to the multimedia data is more accurate, thereby improving the accuracy of feature extraction of the multimedia data.

[0210] Moreover, the multimedia feature extraction model provided in the present application can add the image feature extraction module and the text feature extraction module to the model for training, which can further improve the accuracy of image feature extraction and text feature extraction, thereby further improving the model performance of the multimedia feature extraction model, and further improving the accuracy of feature extraction of the multimedia data.

[0211] In addition, the training method of the multimedia feature extraction model provided in the present application can obtain better model training effect in the case of lacking labeled data, and since the training samples are augmented and more training data are introduced in the training process, the robustness of the multimedia feature extraction model can be improved.

[0212] The present application also provides a method for extracting features of multimedia data, which can be used in a computer device, which can be a terminal or a server. Figure 5 As shown in FIG. 5, another flowchart of the method for extracting features of multimedia data provided in the present application is shown, in which the multimedia data can be video data, and the method specifically includes the following steps.

[0213] In step 201, the computer device obtains a training sample set.

[0214] The training sample set can include a plurality of video data. The plurality of video data can be randomly obtained video data. Each video data includes image stream data and audio stream data.

[0215] At step 202, the computer device performs text feature extraction on each video data.

[0216] In the embodiments of the present application, the video feature extraction model in the present application can be used to perform text feature extraction on each video data, or a separate text feature extractor can be used to extract the text features of each video data. The process of extracting the text features of the video data includes performing speech recognition on the audio stream data contained in the video data to obtain text data, and then performing text feature extraction on the text data to obtain the text features corresponding to each video data. The text features corresponding to each video data can be multiple, and the multiple text features constitute a text feature sequence.

[0217] The video feature extraction model provided in the present application can be used to extract the text features of the video, which can generate a corresponding word id sequence (w1,…,w m ) and an attention mask for each ASR text of the video. The id sequence is used to initialize the word vector for the text feature extraction module in the video feature extraction model, and the attention mask is used to identify the padding word in the id sequence.

[0218] At step 203, the computer device divides the training sample set into a first sample set and a second sample set based on the number of valid information in the text features.

[0219] The video feature extraction model provided in the present application can control the length of the extracted text sequence to be consistent when extracting the text sequence from each video. That is, the number of text features contained in the text feature sequence corresponding to each video is consistent for subsequent training. However, the number of valid word features that can be extracted from the ASR text corresponding to each video data is not necessarily the same, and the attention mask padding method can be used to make the length of the text feature sequence consistent. The word features extracted from the ASR text of the video data are valid features, or effective information, and the attention mask is non-valid text features, or ineffective information.

[0220] At this time, the number of valid text features in the extracted text feature sequence corresponding to each video data can be obtained. And the training sample set is classified based on the number of valid text features or valid information. Specifically, the video with the number of valid text features greater than 32 can be set to constitute a first sample set, denoted as S1; and the video with the number of valid text features less than 16 can be set to constitute a second sample set, denoted as S2.

[0221] In step 204, the computer device samples a first number of video data from the first sample set and the second sample set respectively to obtain a combined training sample set.

[0222] Among them, n / 2 training samples are sampled from S1 and S2 respectively to constitute the combined training sample set. Wherein, n is an integer multiple of 2. The training sample sampling from S1 and S2 can be random sampling or sampling according to a preset rule. The combined training sample set can be denoted as

[0223] In step 205, the computer device performs data augmentation on the combined training sample set to obtain an augmented sample set.

[0224] Among them, two types of samples d 1 and d 2 in the combined training sample set can be augmented by different data augmentation methods respectively. Among them, since both types of samples are video data, and video data contains visual features (i.e. image features) and auditory features (i.e. audio features), and in this application, the text features corresponding to the ASR of the audio are used to represent the audio features. That is, for each video data, data augmentation can be performed from two aspects of image and text.

[0225] In the embodiment of the present application, the data augmentation of the video data in d 1 can be performed by random color transformation, radiation transformation, and local erasing processing of video frames for image processing, using low proportion of text discarding and randomly adding noise to process the ASR text, to obtain the corresponding augmented video data. The data augmentation of the video data in d 2 can be performed by random color transformation, radiation transformation, and local erasing processing of video frames for image processing, and using a method of randomly exchanging the ASR text of the video data in d 2 to obtain a plurality of augmented video data.

[0226] Among them, in the embodiment of the present application, the number of augmented video data in the augmented sample set can be controlled to be consistent with the number of video data in the combined training sample set, at this time, the video data in each combined training sample and an augmented video data can constitute a positive sample pair.

[0227] Step 206, the computer device adopts the first feature extraction model to perform feature extraction on the video data in the combined training sample set, to obtain a first video feature sequence.

[0228] The first feature extraction model is a video feature extraction model provided by the present application, which has a model structure as shown in the following formula (1). Figure 3 The model is a model that needs to be trained, and the parameters of the model can be initialized before performing feature extraction on the video data in the combined training sample set.

[0229] The process of performing feature extraction on each video data in the combined training sample set by using the first feature extraction model can include three major steps, which are: extracting an image feature sequence of the video data by using an image feature extraction module V(f i ) of the first feature extraction model; extracting a text feature sequence of the video data by using a text feature extraction module T(w i ) of the first feature extraction model; and performing feature fusion on the image feature sequence and the text feature sequence of the video data by using a multi-dimensional feature fusion module F(c i ) of the first feature extraction model, to obtain a video feature.

[0230] Here, the above three steps can be described in detail as follows:

[0231] 1. Video frame sampling: a group of sparse image sequences (f1,…,f n ) are collected from the original video frame sequence at a frequency of 1 frame per second, which are used as the input of the image feature extraction module V(f i ).

[0232] 2. ASR word segmentation: the audio stream in the video data is recognized by ASR to obtain ASR text, and then the ASR text is segmented to obtain a word id sequence. This process has been described in detail in step 202, and will not be repeated here.

[0233] 3. Image feature extraction: the image feature extraction module V(f i ) is used to encode each frame of image in the image sequence (f1,…,f n ), to output an image feature sequence (v1,…,v n ) with the same length as the sequence.

[0234] 4. Word-level feature extraction of ASR text: the text feature extraction module T(w i ) is used to extract word features from the text sequence (w1,…,w m ), to output a text feature sequence (t1,…,t n ) with the same length as the sequence.

[0235] 5、aggregate the image feature sequence and the text feature sequence. The extracted full image feature sequence and the text feature sequence are aggregated by using a local pooling method.

[0236] When aggregating the image feature sequence, first, the image feature sequence (v1, …, vn) is divided into several equal-length subsequences by using a fixed-length division strategy, and then the subsequences are averaged-pooled according to formula (1). n

[0237]

[0238] wherein, is the jth image feature in the aggregated image feature sequence, l is the length of the subsequence, and c ∈ R d v ∈ R d It is indicated that the image features before and after aggregation are both feature vectors with a dimension of d.

[0239] When aggregating the text feature sequence, a plurality of text features in the text feature sequence can be divided into a plurality of subsequences by using a special identifier division strategy. The number of features in these subsequences can be the same or different. In the ASR text, a period is often used to distinguish between different sentences, so the special identifier can be a period. That is, when aggregating the text feature sequence, the period can be used as the identifier to perform the identifier indefinite-length division strategy. The specific division and feature aggregation method can be as shown in formula (2):

[0240]

[0241] wherein, is the jth text feature in the aggregated text feature sequence, p j is the coordinate of the jth period in the text feature sequence before aggregation.

[0242] 6、fuse the aggregated image feature sequence and the text feature sequence into a new fused feature sequence. In the embodiments of the present application, a class token feature (cls token) can be used to fuse the aggregated image feature sequence and the aggregated text feature sequence to obtain a fused feature sequence. The fused feature sequence can be represented as wherein, l is the number of image features in the fused image feature sequence, and w is the number of text features in the fused text feature sequence.

[0243] 7、encode the fused feature sequence. Further, for the fused feature sequence, its corresponding position encoding feature sequence and dimension encoding feature sequence are obtained. The position encoding sequence uses different position encoding vectors p i ​The positional information of each feature in the fused feature sequence is represented; dimensionality encoding uses a dimensionality encoding vector m that is independent for each class of dimensions. * The dimensional information of each feature in the fused feature sequence is represented. This application includes three dimensional encoded features: m cls m v and m t , where m cls It is the dimension-encoded feature corresponding to the cls token, m v It is the dimension-encoded feature corresponding to each feature in the aggregated image feature sequence, m t It refers to the dimensionality encoding feature corresponding to each feature in the aggregated text feature sequence. Specifically, the dimensionality encoding features corresponding to different image features in the aggregated image feature sequence are the same, and the dimensionality encoding features corresponding to different text features in the aggregated text feature sequence are also the same.

[0244] Then, the fused feature sequence is encoded according to the position-encoded feature sequence and the dimension-encoded feature sequence, specifically according to formula (3):

[0245]

[0246] Where, x i For the i-th feature in the encoded feature sequence, It is the i-th feature in the fused feature sequence, p i It is the positional feature corresponding to the i-th feature in the fused feature sequence, m * It is the dimensional feature corresponding to the i-th feature in the fused feature sequence. *∈{cls,v,t} means that * is an element in the set {cls,v,t}, and its specific value can be determined according to the specific value of i.

[0247] 8. Input the encoded feature sequence into a multi-head self-attention layer for cross-dimensional self-attention processing. In this layer, the feature corresponding to the cls token interacts with each other feature, while the feature corresponding to the image feature only interacts with other features besides the image feature. The specific process is as follows:

[0248] 8.1 Determine the input to the self-attention layer, including: the encoded feature sequence. The feature sequence has a length of n and a feature dimension of d1; the mask vector v mask ∈R n ; and dimension identifier vector v modal ∈R n .

[0249] 8.2 Determine the spatial mapping parameters (W) for the dimensions of query feature Q, key feature K, and value feature V.Q ,b Q W K ,b K W V ,b V ),W * ∈R d1×d2 ,b*∈R d2 Among them, W Q Let b be the weight matrix in Q-space. Q The offset parameters are in Q-space. The weight matrix is ​​a d1×d2 dimensional matrix, and the offset parameters are d2 dimensional vectors.

[0250] 8.3. Map the feature sequence X to three feature spaces: Q, K, and V, to obtain the feature sequence corresponding to each space: X * =XW * +b*,X * ∈R n×d2 Where *∈{Q,K,V}.

[0251] 8.4 Calculate the correlation matrix between the feature sequences corresponding to feature space Q and feature space K. Specifically, it can be calculated according to formula (4):

[0252] S = X Q X KT ,S∈R n×n Formula (4)

[0253] Among them, X KT Let be the transpose of the feature sequence corresponding to the feature space K.

[0254] 8.5 Calculate the cross-dimensional self-attention mask matrix. The calculation process is as follows:

[0255] M o =Onehot(v modal ),M o ∈R n×3 Formula (5)

[0256]

[0257] Among them, Onehot(v modal ) is a process of effectively encoding the dimensional feature vector. M i It is the i-th row vector of the self-attention mask matrix M, and diag(n) represents constructing a diagonal matrix with n. This indicates the calculation of the tensor product between eigenvectors.

[0258] 8.6 Calculate the self-attention weight matrix based on the self-attention mask matrix. The specific calculation process is as follows:

[0259]

[0260] wherein A i is the i-th row vector of the self-attention weight matrix.

[0261] 8.7, calculating the output token sequence according to the self-attention weight matrix, the specific calculation process is as follows:

[0262]

[0263] wherein X O is the output token sequence, wherein V is the mapped feature sequence of the input feature sequence X in the feature space V. Then output the token sequence X O after cross-dimension information interaction.

[0264] 8.8, determining the final output result according to the output result of the multi-head self-attention layer. Wherein, the above X O is the output result of a self-attention layer, and the application adopts a multi-head self-attention layer, which can output multiple X O , the multiple output results are spliced in the feature dimension, and the final output result is obtained, and the output feature sequence after self-attention processing is obtained.

[0265] 9, processing the output feature sequence of the multi-head self-attention layer to obtain the first video feature Q.

[0266] Specifically, the feature sequence X O output by the multi-head self-attention layer can be added to the original feature sequence X through a residual connection, and then processed through a feedforward neural network layer composed of a full connection layer, and then repeated once again residual connection and normalization processing. The specific processing can be carried out according to the following formula:

[0267] X′=(X O +X) Formula (9)

[0268] X″=LayerNorm(X ′ +FFN(X ′ ))

[0269] Formula (10)

[0270] Wherein, X ′ is the feature sequence after one residual processing; X ′′ is the feature sequence after two residual processing. FFN(X ′ ) represents that the feedforward neural network composed of the full connection layer is used to process the feature sequence after one residual processing, and LayerNorm represents the normalization processing of the feature sequence.

[0271] In step 207, the computer device extracts features of the video data in the augmented sample set by using the second feature extraction model to obtain a second video feature sequence.

[0272] The second feature extraction model and the first feature extraction model are neural network models with the same structure and the same initial parameters. However, the model parameters of the first feature extraction model and the second feature extraction model are not shared, that is, during the training process, when the model parameters of the first feature extraction model are updated, the model parameters of the second feature extraction model are not updated synchronously. That is, the first feature extraction model and the second feature extraction model are two models that are the same but independent of each other.

[0273] The second feature extraction model extracts features of the video data in the augmented sample set to obtain a second video feature K + The specific feature extraction process is the same as that of the first feature extraction model extracting features of the video data in the combined training sample set, which will not be repeated here. Wherein, Q, K + ∈R n×d . Since the video data in the augmented sample set and the video data in the combined sample set form a positive sample pair, the second video feature K + is also called a positive sample feature. Q and K + form a positive feature pair.

[0274] In step 208, the computer device samples a second number of features from the augmented feature sequence to obtain a third video feature sequence.

[0275] The augmented feature sequence queueK is a feature sequence updated based on the second video feature K + extracted from the augmented sample set. When initialized, the augmented feature sequence does not exist, that is, 0. In each update cycle, the newly generated K + is added to queueK to update it. The number of queueK can be set to an upper limit, which is an integer multiple of the number of video data in the augmented sample set.

[0276] Wherein, the sampling is performed from the augmented feature sequence, and the sampled feature is taken as a negative sample feature K - , wherein K - may be referred to as a third video feature sequence. The sampling number can be f times the number of video data in the augmented sample, then K - ∈R fn ×d , Q and K - form a negative feature pair.

[0277] Step 209: The computer device calculates the contrast loss based on the first video feature sequence, the second video feature sequence, and the third video feature sequence.

[0278] The contrastive loss (i.e., Info NCE loss) can be calculated based on positive and negative feature pairs, and the specific calculation formula is as follows:

[0279]

[0280] Among them, L q Let q be any video feature in the feature sequence Q, and k be the contrast loss. + For the second characteristic sequence K + The feature corresponding to feature q, k - The third characteristic sequence K - The features corresponding to feature q. Where k - There are a total of f. Let i be the i-th one among them.

[0281] Step 210: The computer device updates the model parameters of the first feature extraction model according to the contrast loss, and updates the second feature extraction model based on the updated first feature extraction model.

[0282] Specifically, based on the contrastive loss function, gradient descent processing is applied to the loss function, and gradient backpropagation is performed to update the parameters of the first feature extraction model.

[0283] Furthermore, the parameters of the second feature extraction model can be updated using a momentum update method. The formula for the momentum update method is as follows:

[0284] θ K =gθ K +(1-g)θ Q

[0285] Formula (12)

[0286] Where g is the momentum update coefficient, θ K θ is the model parameter for the second feature extraction model. Q These are the model parameters for the first feature extraction model.

[0287] Step 211: The computer device updates the augmented feature sequence based on the second video feature sequence.

[0288] In this step, the augmented feature sequence can be updated based on the second video feature sequence, that is, the second video feature sequence can be added to the augmented feature sequence.

[0289] In some embodiments, the number of the augmented feature sequence can also be controlled, that is, the number of the sequence is detected, when the number does not reach the preset number, the second video feature sequence is used for updating; if the number has reached the preset number, the augmented feature sequence is not updated.

[0290] In step 212, the computer device determines whether the first feature extraction model converges.

[0291] The determination of whether the first feature extraction model converges can be specifically determining whether the number of iterations reaches a preset number, or determining whether the value of the contrast loss is less than a preset value, etc.

[0292] When it is determined that the first feature extraction model converges, step 213 is entered, and if it is determined that the first feature extraction model has not converged, step 204 is returned to perform loop iteration until the first feature extraction model converges, and the training of the first feature extraction model is completed.

[0293] In step 213, the computer device extracts features of a target video by using the trained first feature extraction model.

[0294] After the training of the first feature extraction model is completed, the trained first feature extraction model can be used to extract features of a target video to be extracted, and target features of the target video are obtained.

[0295] According to the above description, the multimedia data feature extraction method provided by the application extracts multi-dimensional features of image features of multimedia data and text features corresponding to speech recognition results of audio data in the multimedia data. Then, the multi-dimensional features are fused, and the fused features are further processed by self-attention, and the target features for describing the multimedia data are determined based on the self-attention processing result. Since the target features of two dimensions of the image features of the multimedia data and the text features corresponding to the audio data are fused to describe the multimedia data, the target features can more accurately describe the multimedia data, thereby improving the accuracy of feature extraction of the multimedia data.

[0296] Moreover, the multimedia feature extraction model provided by the application can add the image feature extraction module and the text feature extraction module to the model for training, which can further improve the accuracy of image feature extraction and text feature extraction, thereby further improving the model performance of the multimedia feature extraction model, and further improving the accuracy of feature extraction of the multimedia data.

[0297] In addition, the training method of the multimedia feature extraction model provided in the application can obtain better model training effect in the case of lacking labeled data, and because the training samples are augmented and more training data are introduced in the training process, the robustness of the multimedia feature extraction model can be improved.

[0298] The application further provides a multimedia data retrieval method, which can be applied to a multimedia data retrieval device that can be integrated in a terminal or a server. The computer device can be a terminal or a server. The terminal can be a mobile phone, a tablet computer, a notebook computer, a smart television, a wearable smart device, a personal computer (PC), a vehicle-mounted terminal, or the like. The server can be a standalone physical server, a server cluster composed of multiple physical servers, or a distributed system, and can also be a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery network (CDN) services, and basic cloud computing services such as big data and artificial intelligence platforms. As shown in the following figure, the multimedia data retrieval method provided in the application includes the following steps. Figure 6

[0299] In step 301, feature extraction is performed on the multimedia data to be retrieved to obtain first features.

[0300] In the embodiments of the application, the method of performing feature extraction on the multimedia data to be retrieved can be any one of the feature extraction methods of the multimedia data provided in the foregoing embodiments of the application. Specifically, the feature extraction method provided in the embodiments can be used to perform feature extraction on the multimedia data to be retrieved. Figure 2 The feature extraction method provided in the embodiments is used to perform feature extraction on the multimedia data to be retrieved, and the feature data of the multimedia data that can represent the multimedia data from multiple dimensions is extracted. That is, the first features of the multimedia data to be retrieved are extracted.

[0301] In step 302, feature extraction is performed on each multimedia data in the candidate multimedia database to obtain second features.

[0302] The method of performing feature extraction on each multimedia data in the multimedia database is the same as the extraction method of performing feature extraction on the multimedia data to be retrieved in step 301. The second features corresponding to each multimedia data in the multimedia database are extracted.

[0303] In step 303, the cosine similarity between the first features and each second feature is calculated.

[0304] ​The first feature and the second feature can be represented in the form of a feature vector. In this application, feature extraction is performed on the multimedia data to obtain a feature vector corresponding to the multimedia data, i.e., the first feature and the second feature are both feature vectors, and a plurality of second features can form a feature matrix.

[0305] Further, the cosine similarity between the first feature and the vector corresponding to each second feature can be calculated to obtain a plurality of cosine similarity values.

[0306] Step 304: determining target multimedia data similar to the multimedia data to be searched based on the cosine similarity.

[0307] A preset threshold can be set. When the cosine similarity value is higher than the preset threshold, the video data corresponding to the second feature is determined as target multimedia data similar to the multimedia data to be searched. When the cosine similarity is lower than the preset threshold, the video data corresponding to the second feature is determined as target multimedia data not similar to the multimedia data to be searched.

[0308] According to the above description, the multimedia data retrieval method provided by the present application extracts the first feature of the multimedia data to be searched and the second feature of each multimedia data in the candidate multimedia database by using the feature extraction method of the multimedia data provided by the present application. The cosine similarity between the first feature and each second feature is calculated. The target multimedia data similar to the multimedia data to be searched is determined based on the cosine similarity. Therefore, the features of the multimedia data are extracted from the image dimension and the text dimension, and the extracted features are fused to obtain feature data of the multimedia data which can represent the multimedia data from multiple dimensions. Based on the feature data, the multimedia data is searched to obtain more accurate search results.

[0309] In order to better implement the above method, the embodiment of the present application further provides a feature extraction device of multimedia data, which can be integrated in a terminal or a server.

[0310] For example, as shown in FIG. 4, a structure schematic diagram of the feature extraction device of multimedia data provided by the embodiment of the present application can include a first extraction unit 401, a second extraction unit 402, a fusion unit 403, a first calculation unit 404, and a first determination unit 405, as follows: Figure 6 The first extraction unit 401 is configured to perform image feature extraction on the target multimedia data to obtain an image feature sequence of the target multimedia data.

[0311]

[0312] ​The second extraction unit 402 is configured to perform speech recognition on the audio data in the target multimedia data, and perform text feature extraction on a speech recognition result to obtain a text feature sequence of the target multimedia data.

[0313] The fusion unit 403 is configured to fuse the image feature sequence and the text feature sequence to obtain a fusion feature sequence.

[0314] The first calculation unit 404 is configured to perform self-attention calculation on the fusion feature sequence to obtain a target feature sequence.

[0315] The first determination unit 405 is configured to determine a target feature of the target multimedia data according to the target feature sequence.

[0316] In some embodiments, the fusion unit comprises:

[0317] The first processing subunit is configured to perform local pooling processing on the image feature sequence to obtain a target image feature sequence.

[0318] The second processing subunit is configured to perform local pooling processing on the text feature sequence to obtain a target text feature sequence.

[0319] The fusion subunit is configured to fuse the target image feature sequence and the target text feature sequence to obtain the fusion feature sequence.

[0320] In some embodiments, the fusion subunit comprises:

[0321] The first acquisition module is configured to acquire a preset class token feature.

[0322] The splicing module is configured to splice the class token feature, the target image feature sequence, and the target text feature sequence to obtain a spliced feature sequence.

[0323] The second acquisition module is configured to acquire a position encoding sequence and a dimension encoding sequence corresponding to the spliced feature sequence.

[0324] The fusion module is configured to fuse the spliced feature sequence, the position encoding sequence, and the dimension encoding sequence to obtain the fusion feature sequence.

[0325] In some embodiments, the first calculation unit comprises:

[0326] The first calculation subunit is configured to map the fusion feature sequence into a space of a query feature, a key feature, and a value feature to obtain a query feature sequence, a key feature sequence, and a value feature sequence, and calculate a relevance matrix of the query feature sequence and the key feature sequence.

[0327] The determining subunit is configured to determine a category fusion feature corresponding to the category token feature in the fusion feature sequence, an image fusion feature sequence corresponding to the target image feature sequence, and a text fusion feature sequence corresponding to the target text feature sequence;

[0328] The second calculating subunit is configured to calculate a self-attention weight matrix based on an interaction result of the category fusion feature and each feature in the fusion feature sequence, an interaction result between each feature in the image fusion feature sequence and each feature in the text fusion feature sequence, and the correlation matrix;

[0329] The third processing subunit is configured to perform weighted processing on the value feature sequence based on the self-attention weight matrix to obtain the target feature sequence.

[0330] In some embodiments, the first processing subunit includes:

[0331] The first dividing module is configured to divide the image features in the image feature sequence into a plurality of sub-image feature sequences according to an image sampling sequence.

[0332] The first processing module is configured to perform average pooling processing on each sub-image feature sequence to obtain a plurality of first average pooling features, and the plurality of first average pooling features constitute the target image feature sequence.

[0333] In some embodiments, the second processing subunit includes:

[0334] The third obtaining module is configured to obtain target punctuation information in the speech recognition result.

[0335] The second dividing module is configured to divide the text feature sequence into a plurality of sub-text feature sequences according to the target punctuation information.

[0336] The second processing module is configured to perform average pooling processing on each sub-text feature sequence to obtain a plurality of second average pooling features, and the plurality of second average pooling features constitute the target text feature sequence.

[0337] In some embodiments, the first determining unit includes:

[0338] The extracting subunit is configured to extract a target category token feature corresponding to the category token feature from the target feature sequence.

[0339] The fourth processing subunit is configured to perform first residual connection and normalization processing on the target category token feature to obtain a first transition feature.

[0340] The fifth processing subunit is configured to process the first transition feature by using a preset feedforward neural network to obtain a second transition feature.

[0341] The sixth processing subunit is configured to perform second residual connection and normalization processing on the second transition feature to obtain a target feature of the target multimedia data.

[0342] In some embodiments, the first computing unit comprises:

[0343] The third computing subunit is configured to perform at least one self-attention computation on the fused feature sequence to obtain at least one sub-target feature sequence.

[0344] The splicing subunit is configured to splice the at least one sub-target feature sequence to obtain the target feature sequence.

[0345] In some embodiments, the multimedia feature extraction device can be equipped with a multimedia feature extraction model, and the method is applied to the multimedia feature extraction model. The multimedia feature extraction model comprises an image feature extraction module, a text feature extraction module, and a multi-dimensional feature fusion module. The first extraction subunit is further configured to:

[0346] The image feature extraction module based on the multimedia feature extraction model performs feature extraction on the target multimedia data to obtain an image feature sequence of the target multimedia data.

[0347] The second extraction subunit is further configured to perform feature extraction on the target multimedia data based on the text feature extraction module of the multimedia feature extraction model to obtain a text feature sequence of the target multimedia data.

[0348] The fusion unit, the first computing unit, and the first determination unit are further configured to:

[0349] The multi-dimensional feature fusion module based on the multimedia feature extraction model fuses the image feature sequence and the text feature sequence to obtain a target feature of the target multimedia data.

[0350] In some embodiments, the training device for training the multimedia feature extraction model comprises:

[0351] The acquisition unit is configured to acquire a training sample set, and perform sample augmentation on training samples in the training sample set to obtain an augmented training sample set.

[0352] The fourth extraction unit is configured to perform feature extraction on the training sample set by using a first neural network model to be trained to obtain a first multimedia feature sequence.

[0353] The fifth extraction unit is configured to perform feature extraction on the augmented training sample set by using a second neural network model to be trained to obtain a second multimedia feature sequence. The model structures of the first neural network model and the second neural network model are the same as the model structure of the multimedia feature extraction model, and the initial model parameters of the first neural network model and the second neural network model are the same.

[0354] a sampling unit, configured to update a preset augmented feature sequence based on the second multimedia feature sequence, and sample a preset number of multimedia features from the augmented feature sequence to obtain a third multimedia feature sequence;

[0355] a third calculating unit, configured to calculate a contrastive loss based on the first multimedia feature sequence, the second multimedia feature sequence, and the third multimedia feature sequence;

[0356] a training unit, configured to iteratively train the first neural network model and the second neural network model based on the contrastive loss to obtain a trained first neural network model, the trained first neural network model being a trained multimedia feature extraction model.

[0357] In some embodiments, the training unit comprises:

[0358] a first updating subunit, configured to update model parameters of the first neural network model based on backpropagation of the contrastive loss;

[0359] a second updating subunit, configured to update parameters of the second neural network model using a momentum updating method based on the updated first neural network model;

[0360] an executing subunit, configured to return to executing the steps of extracting features from the training sample set using the first neural network model and extracting features from the augmented sample set using the second neural network model until the model parameters of the first neural network model converge.

[0361] In some embodiments, the obtaining unit comprises:

[0362] an obtaining subunit, configured to obtain a preset multimedia data set, the multimedia data set comprising a plurality of multimedia data;

[0363] a dividing subunit, configured to divide the multimedia data set into a first sub-set and a second sub-set based on a number of valid features in a text feature sequence of each multimedia data;

[0364] a sampling subunit, configured to sample a preset number of multimedia data from the first sub-set and the second sub-set respectively to obtain a training sample set;

[0365] an augmenting subunit, configured to perform sample augmentation on first samples in the training sample set using a first sample augmentation method and perform sample augmentation on second samples in the training sample set using a second sample augmentation method, the first samples being samples sampled from the first sub-set, and the second samples being samples sampled from the second sub-set.

[0366] In practice, each of the above units can be implemented as an independent entity or can be arbitrarily combined to be implemented as the same or several entities. For the specific implementation of each of the above units, please refer to the previous method embodiments, which will not be repeated here.

[0367] As described above, the feature extraction device for multimedia data provided in this application embodiment extracts image features from the target multimedia data through a first extraction unit 401 to obtain an image feature sequence of the target multimedia data; a second extraction unit 402 performs speech recognition on the audio data in the target multimedia data and extracts text features from the speech recognition results to obtain a text feature sequence of the target multimedia data; a fusion unit 403 fuses the image feature sequence and the text feature sequence to obtain a fused feature sequence; a first calculation unit 404 performs self-attention calculation on the fused feature sequence to obtain a target feature sequence; and a first determination unit 405 determines the target features of the target multimedia data based on the target feature sequence.

[0368] Therefore, the multimedia data feature extraction device provided in this application extracts multi-dimensional features from the image features of the multimedia data and the text features corresponding to the speech recognition results of the audio data in the multimedia data; then, it fuses the multi-dimensional features and performs further self-attention processing on the fused features, and determines the target features for describing the multimedia data based on the self-attention processing results. Because the target features, which fuse the image features of the multimedia data and the text features corresponding to the audio data, are used to describe the multimedia data, the description of the multimedia data by the target features is more accurate, thereby improving the accuracy of feature extraction from the multimedia data.

[0369] To better implement the above methods, this application also provides a multimedia data retrieval device, which can be integrated into a terminal or server.

[0370] For example, such as Figure 7 The diagram shown is a structural schematic of the multimedia data retrieval device provided in this application. The multimedia data retrieval device may include: a third extraction unit 501, a fourth extraction unit 502, a second calculation unit 503, and a second determination unit 504. Details are as follows:

[0371] The third extraction unit 501 is used to extract features from the multimedia data to be retrieved to obtain the first feature.

[0372] The fourth extraction unit 502 is used to extract features from each multimedia data in the candidate multimedia database to obtain the second feature.

[0373] The second calculation unit 503 is used to calculate the cosine similarity between the first feature and each second feature;

[0374] The second determining unit 504 is used to determine target multimedia data that is similar to the multimedia data to be retrieved based on cosine similarity.

[0375] In practice, each of the above units can be implemented as an independent entity or can be arbitrarily combined to be implemented as the same or several entities. For the specific implementation of each of the above units, please refer to the previous method embodiments, which will not be repeated here.

[0376] As described above, the multimedia data retrieval device provided in this application extracts the first feature of the multimedia data to be retrieved by the third extraction unit 501 using the feature extraction method of multimedia data provided in this application; the fourth extraction unit 502 extracts the second feature of each multimedia data in the candidate multimedia database using the feature extraction method of multimedia data provided in this application; the second calculation unit 503 calculates the cosine similarity between the first feature and each second feature; and the second determination unit 504 determines the target multimedia data similar to the multimedia data to be retrieved based on the cosine similarity.

[0377] Therefore, the multimedia data retrieval device provided in this application extracts features from multimedia data from both image and text dimensions, and then fuses the extracted features to obtain feature data that can represent multimedia data from multiple dimensions. Retrieving multimedia data based on this feature data yields more accurate retrieval results.

[0378] This application also provides a computer device, which can be a terminal or a server, such as... Figure 8 The diagram shown is a structural schematic of the computer device provided in this application. Specifically:

[0379] The computer device may include components such as a processing unit 601 with one or more processing cores, a storage unit 602 with one or more storage media, a power module 603, and an input module 604. Those skilled in the art will understand that... Figure 7 The computer device structure shown does not constitute a limitation on the computer device and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein:

[0380] The processing unit 601 is the control center of the computer device, connects each part of the computer device through various interfaces and lines, executes various functions of the computer device and processes data by running or executing software programs and / or modules stored in the storage unit 602, and calling data stored in the storage unit 602, so as to perform overall detection on the computer device. Optionally, the processing unit 601 can include one or more processing cores; preferably, the processing unit 601 can integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, the user interface and the application program, and the modem processor mainly processes wireless communication. It can be understood that the above-mentioned modem processor can also not be integrated into the processing unit 601.

[0381] The storage unit 602 can be used to store software programs and modules, and the processing unit 601 executes various functions and data processing by running the software programs and modules stored in the storage unit 602. The storage unit 602 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application program required by a function (such as a sound playing function, an image playing function and a web page access, etc.), and the like; the data storage area can store data created according to the use of the computer device, etc. In addition, the storage unit 602 can include a high-speed random access memory, and can also include a non-volatile memory, for example, at least one magnetic disk storage device, a flash memory device, or other volatile solid-state memory device. Accordingly, the storage unit 602 can also include a memory controller to provide access of the processing unit 601 to the storage unit 602.

[0382] The computer device further includes a power module 603 for supplying power to each component, and preferably, the power module 603 can be logically connected with the processing unit 601 through a power management system, so as to realize functions such as management of charging, discharging and power consumption management through the power management system. The power module 603 can also include one or more than one direct current or alternating current power supply, a recharging system, a power failure detection circuit, a power converter or inverter, a power state indicator and any other components.

[0383] The computer device can also include an input module 604, which can be used to receive input digital or character information, and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0384] Although not shown, the computer device can also include a display unit and the like, which will not be described here. In particular in the present embodiment, the processing unit 601 in the computer device will load one or more executable files corresponding to processes of one or more application programs into the storage unit 602 according to the following instructions, and run the application programs stored in the storage unit 602 by the processing unit 601, thereby realizing various functions, as follows:

[0385] image feature extraction is performed on the target multimedia data to obtain an image feature sequence of the target multimedia data; speech recognition is performed on audio data in the target multimedia data, and text feature extraction is performed on a speech recognition result to obtain a text feature sequence of the target multimedia data; the image feature sequence and the text feature sequence are fused to obtain a fused feature sequence; self-attention calculation is performed on the fused feature sequence to obtain a target feature sequence; and a target feature of the target multimedia data is determined according to the target feature sequence.

[0386] Alternatively, a first feature of the to-be-searched multimedia data and a second feature of each multimedia data in the candidate multimedia database are extracted by using the multimedia data feature extraction method provided in the present application; cosine similarity of the first feature and each second feature is calculated; and target multimedia data similar to the to-be-searched multimedia data is determined based on the cosine similarity.

[0387] It should be noted that the computer device provided in the embodiments of the present application and the method in the above embodiments belong to the same concept, and the specific implementation of each operation can be referred to the previous embodiments, which will not be described here.

[0388] Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructions, or by related hardware controlled by the instructions, which can be stored in a computer readable storage medium and loaded and executed by a processor.

[0389] To this end, the embodiments of the present application provide a computer readable storage medium, which stores a plurality of instructions capable of being loaded by a processor to execute the steps in any one of the methods provided by the embodiments of the present application. For example, the instructions can execute the following steps:

[0390] image feature extraction is performed on the target multimedia data to obtain an image feature sequence of the target multimedia data; speech recognition is performed on audio data in the target multimedia data, and text feature extraction is performed on a speech recognition result to obtain a text feature sequence of the target multimedia data; the image feature sequence and the text feature sequence are fused to obtain a fused feature sequence; self-attention calculation is performed on the fused feature sequence to obtain a target feature sequence; and a target feature of the target multimedia data is determined according to the target feature sequence.

[0391] Alternatively, the first feature of the multimedia data to be searched and the second feature of each multimedia data in the candidate multimedia database are extracted by using the feature extraction method of the multimedia data provided in the present application; the cosine similarity of the first feature and each second feature is calculated; and the target multimedia data similar to the multimedia data to be searched is determined based on the cosine similarity.

[0392] The specific implementation of each operation above can refer to the foregoing embodiments, which will not be repeated here.

[0393] The computer readable storage medium can include a read only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0394] The steps in any of the methods provided by the embodiments of the present application can be executed due to the instructions stored in the computer readable storage medium, and thus the beneficial effects of any of the methods provided by the embodiments of the present application can be achieved. Details are shown in the foregoing embodiments, which will not be repeated here.

[0395] According to an aspect of the present application, a computer program product or computer program is provided, which includes computer instructions stored in a storage medium. The processor of the computer device reads the computer instructions from the storage medium, and the processor executes the computer instructions to make the computer device execute the method provided in any of the optional implementation manners of the feature extraction method of the multimedia data or the multimedia data searching method.

[0396] The feature extraction method of the multimedia data, the multimedia data searching method and the device provided by the embodiments of the present application are described in detail above, and the principle and implementation manner of the present application are described by applying specific examples in the present application. The foregoing embodiment is only used to help understand the method of the present application and its core idea; meanwhile, for those skilled in the art, the specific implementation manner and application range will be changed according to the idea of the present application, and the content of the specification should not be understood as the limitation of the present application.

Claims

1. A feature extraction method of multimedia data, characterized by, The method comprises: image feature extraction is performed on target multimedia data to obtain an image feature sequence of the target multimedia data, wherein the image feature extraction comprises: sampling image frames of the target multimedia data to obtain a plurality of image frames; and image feature extraction is performed on the plurality of image frames to obtain an image feature corresponding to each image frame, and a plurality of the image features constitute the image feature sequence, the target multimedia data being video data; speech recognition is performed on audio data in the target multimedia data, and text feature extraction is performed on a speech recognition result to obtain a text feature sequence of the target multimedia data; the image feature sequence and the text feature sequence are fused to obtain a fused feature sequence, wherein the fusion comprises: local pooling processing is performed on the image feature sequence to obtain a target image feature sequence; local pooling processing is performed on the text feature sequence to obtain a target text feature sequence; and the target image feature sequence and the target text feature sequence are fused to obtain the fused feature sequence; the local pooling processing of the image feature sequence to obtain the target image feature sequence comprises: the plurality of image features included in the image feature sequence are sorted according to a sampling order of the images corresponding to the image features to obtain a sorted image feature sequence; the sorted image feature sequence is divided into a plurality of sub-image feature sequences; and average pooling is performed on each of the sub-image feature sequences to obtain a corresponding first average-pooled feature, and a plurality of the first average-pooled features constitute the target image feature sequence; the fusion of the target image feature sequence and the target text feature sequence to obtain the fused feature sequence comprises: a preset category token feature is obtained; the category token feature, the target image feature sequence, and the target text feature sequence are spliced to obtain a spliced feature sequence; a position encoding sequence and a dimension encoding sequence corresponding to the spliced feature sequence are obtained; and the spliced feature sequence, the position encoding sequence, and the dimension encoding sequence are fused to obtain the fused feature sequence, the position encoding being encoding data describing positions of features in the spliced feature sequence; self-attention calculation is performed on the fused feature sequence to obtain a target feature sequence; a target feature of the target multimedia data is determined according to the target feature sequence.

2. The method of claim 1, wherein, the self-attention calculation of the fused feature sequence to obtain the target feature sequence comprises: the fused feature sequence is mapped into a space of a query feature, a key feature, and a value feature to obtain a query feature sequence, a key feature sequence, and a value feature sequence, and a correlation matrix of the query feature sequence and the key feature sequence is calculated; a category fusion feature corresponding to the category token feature, an image fusion feature sequence corresponding to the target image feature sequence, and a text fusion feature sequence corresponding to the target text feature sequence in the fused feature sequence are determined; and the target feature sequence is obtained by performing self-attention calculation on the fused feature sequence, and the target feature of the target multimedia data is determined according to the target feature sequence. The self-attention weight matrix is calculated based on the interaction result of each feature in the category fusion feature and the fusion feature sequence, the interaction result between each feature in the image fusion feature sequence and each feature in the text fusion feature sequence, and the correlation matrix; The value feature sequence is weighted based on the self-attention weight matrix to obtain a target feature sequence.

3. The method of claim 1, wherein, The local pooling processing of the image feature sequence to obtain a target image feature sequence comprises: The image features in the image feature sequence are divided into a plurality of sub-image feature sequences according to an image sampling sequence; Each sub-image feature sequence is subjected to average pooling processing to obtain a plurality of first average pooling features, and the plurality of first average pooling features constitute the target image feature sequence.

4. The method of claim 1, wherein, The local pooling processing of the text feature sequence to obtain a target text feature sequence comprises: Target punctuation information in the speech recognition result is obtained; The text feature sequence is divided into a plurality of sub-text feature sequences according to the target punctuation information; Each sub-text feature sequence is subjected to average pooling processing to obtain a plurality of second average pooling features, and the plurality of second average pooling features constitute the target text feature sequence.

5. The method of claim 1, wherein, The target feature of the target multimedia data is determined according to the target feature sequence, which comprises: A target category token feature sequence corresponding to the category token feature is extracted from the target feature sequence; The target category token feature sequence is subjected to first residual connection and normalization processing to obtain a first transition feature; The first transition feature is processed by a preset feedforward neural network to obtain a second transition feature; The second transition feature is subjected to second residual connection and normalization processing to obtain the target feature of the target multimedia data.

6. The method of claim 1, wherein, The self-attention calculation of the fusion feature sequence to obtain a target feature sequence comprises: The fusion feature sequence is subjected to at least one self-attention calculation to obtain at least one sub-target feature sequence; The at least one sub-target feature sequence is spliced to obtain the target feature sequence.

7. The method of claim 1, wherein, The method is applied to a multimedia feature extraction model, the multimedia feature extraction model comprises an image feature extraction module, a text feature extraction module, and a multi-dimensional feature fusion module, and the image feature extraction of the target multimedia data to obtain an image feature sequence of the target multimedia data comprises: The image feature extraction of the target multimedia data is performed based on the image feature extraction module of the multimedia feature extraction model to obtain the image feature sequence of the target multimedia data; The speech recognition of the audio data in the target multimedia data and the text feature extraction of the speech recognition result to obtain a text feature sequence of the target multimedia data comprise: The feature extraction of the target multimedia data is performed based on the text feature extraction module of the multimedia feature extraction model to obtain the text feature sequence of the target multimedia data; The image feature sequence and the text feature sequence are fused to obtain a fusion feature sequence, and self-attention calculation is performed on the fusion feature sequence to obtain a target feature sequence, and a target feature of the target multimedia data is determined according to the target feature sequence. The image feature sequence and the text feature sequence are fused by a multi-dimensional feature fusion module of the multimedia feature extraction model to obtain the target feature of the target multimedia data.

8. The method of claim 7, wherein, The training process of the multimedia feature extraction model includes the following steps: A training sample set is obtained, and sample augmentation is performed on the training samples in the training sample set to obtain an augmented training sample set; A first neural network model to be trained is used to extract features from the training sample set to obtain a first multimedia feature sequence; A second neural network model to be trained is used to extract features from the augmented training sample set to obtain a second multimedia feature sequence, the model structure of the first neural network model and the second neural network model is the same as that of the multimedia feature extraction model, and the initial model parameters of the first neural network model and the second neural network model are the same; The augmented feature sequence is updated based on the second multimedia feature sequence, and a preset number of multimedia features are sampled from the augmented feature sequence to obtain a third multimedia feature sequence; A contrast loss is calculated according to the first multimedia feature sequence, the second multimedia feature sequence and the third multimedia feature sequence; The first neural network model and the second neural network model are iteratively trained based on the contrast loss to obtain a trained first neural network model, and the trained first neural network model is a trained multimedia feature extraction model.

9. The method of claim 8, wherein, The first neural network model and the second neural network model are iteratively trained based on the contrast loss, including: The model parameters of the first neural network model are updated based on the backpropagation of the contrast loss; The parameters of the second neural network model are updated using a momentum update method according to the updated first neural network model; The steps of extracting features from the training sample set using the first neural network model and extracting features from the augmented sample set using the second neural network model are returned until the model parameters of the first neural network model converge.

10. The method of claim 8, wherein, The training sample set is obtained, and sample augmentation is performed on the training samples in the training sample set to obtain an augmented training sample set, including: A preset multimedia data set is obtained, and the multimedia data set includes a plurality of multimedia data; The multimedia data set is divided into a first sub-set and a second sub-set based on the number of effective features in the text feature sequence of each multimedia data; A preset number of multimedia data are sampled from the first sub-set and the second sub-set respectively to obtain a training sample set; The first sample in the training sample set is augmented by using a first sample augmentation method, and the second sample in the training sample set is augmented by using a second sample augmentation method, the first sample being a sample sampled from the first subset, and the second sample being a sample sampled from the second subset.

11. A multimedia data retrieval method, characterized by, The method comprises: The method comprises: The method comprises: The method comprises:

12. An apparatus for feature extraction of multimedia data, characterized by comprising: The device comprises: The first extraction unit is configured to perform image feature extraction on the target multimedia data to obtain an image feature sequence of the target multimedia data, including: performing image frame sampling on the target multimedia data to obtain a plurality of image frames; and performing image feature extraction on the plurality of image frames to obtain an image feature corresponding to each image frame, the plurality of image features constituting the image feature sequence, the target multimedia data being video data; The second extraction unit is configured to perform speech recognition on audio data in the target multimedia data and perform text feature extraction on a speech recognition result to obtain a text feature sequence of the target multimedia data; The fusion unit is configured to fuse the image feature sequence and the text feature sequence to obtain a fused feature sequence, including: performing local pooling processing on the image feature sequence to obtain a target image feature sequence; performing local pooling processing on the text feature sequence to obtain a target text feature sequence; and fusing the target image feature sequence and the target text feature sequence to obtain the fused feature sequence; The local pooling processing on the image feature sequence to obtain the target image feature sequence comprises: The image feature sequence is sorted according to the sampling order of the corresponding images to obtain a sorted image feature sequence; the sorted image feature sequence is divided into a plurality of sub-image feature sequences; and each sub-image feature sequence is subjected to average pooling to obtain a corresponding first average pooling feature, the plurality of first average pooling features constituting the target image feature sequence; The fusion of the target image feature sequence and the target text feature sequence to obtain the fused feature sequence comprises: A preset category token feature is obtained; the category token feature, the target image feature sequence, and the target text feature sequence are spliced to obtain a spliced feature sequence; a position encoding sequence and a dimension encoding sequence corresponding to the spliced feature sequence are obtained; and the spliced feature sequence, the position encoding sequence, and the dimension encoding sequence are fused to obtain the fused feature sequence, the position encoding being encoding data describing the position of a feature in the spliced feature sequence; The first calculation unit is configured to perform self-attention calculation on the fused feature sequence to obtain a target feature sequence. The first determining unit is configured to determine a target feature of the target multimedia data according to the target feature sequence.

13. A multimedia data retrieval apparatus, characterized by comprising: The device comprises: The third extracting unit is configured to extract a first feature of the multimedia data to be searched and a second feature of each multimedia data in the candidate multimedia database by using the feature extraction method of any one of claims 1-10; The second calculating unit is configured to calculate a cosine similarity between the first feature and each second feature; The second determining unit is configured to determine a target multimedia data similar to the multimedia data to be searched based on the cosine similarity.

14. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a plurality of instructions adapted to be loaded by a processor to execute the steps of the feature extraction method of any one of claims 1-10 or the multimedia data searching method of claim 11.

15. A computer device, comprising: A computer program product comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the feature extraction method of any one of claims 1-10 or the multimedia data searching method of claim 11 when executing the computer program. A computer program product comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the feature extraction method of any one of claims 1-10 or the multimedia data searching method of claim 11 when executing the computer program.

Images