Video data processing method, device, program product, computer equipment and medium

Abstract

The present application discloses a video data processing method, device, program product, computer equipment and medium, the method comprising: obtaining sample pairs and sample association pairs; the sample pairs are divided into positive sample pairs and negative sample pairs according to the content difference between the video clips and texts contained therein; the sample association pairs contain texts and the video clips to which the texts belong in the sample video data; the sample video data has a video description label; calling a generative network to perform comparative learning on the sample pairs to obtain a learning deviation; calling a generative network to predict sample video description information of the sample video data based on the sample association pairs, and obtaining a prediction deviation based on the video description label and the sample video description information; training the generative network according to the learning deviation and the prediction deviation to obtain a trained generative network. By adopting the present application, the accuracy of the generative network obtained by training can be improved, and the video description information of the video data can also be accurately generated by adopting the trained generative network.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a video data processing method, apparatus, program product, computer equipment and medium. Background Art

[0002] In video retrieval applications, users can usually enter search information to retrieve related videos. The background can match the search information entered by the user with the title information of the videos in the video library. If the match is successful (such as the text similarity between the search information and the title information of the video is high), the video whose title information matches the search information entered by the user can be pushed to the user.

[0003] However, if the video content of the video does not match the content described in the video title information, the user cannot retrieve the accurate video by searching the information. Therefore, how to accurately generate relevant description information of the video data (for matching with the search information entered by the user) becomes a problem that needs to be solved urgently. Summary of the invention

[0004] The present application provides a video data processing method, apparatus, program product, computer equipment and medium, which can improve the accuracy of a trained generation network, and then use the trained generation network to accurately generate video description information of video data.

[0005] On the one hand, the present application provides a video data processing method, the method comprising:

[0006] Obtain sample pairs and sample association pairs; a sample pair includes a first video clip and a first text, and the sample pair is divided into a positive sample pair and a negative sample pair according to the content difference between the first video clip and the first text; a sample association pair includes a second text and a second video clip to which the second text belongs in the sample video data; the sample video data has a video description tag;

[0007] Calling the generative network to perform feature contrast learning on the sample pair to obtain a learning bias; the learning bias is used to reduce the feature difference between the first video clip and the first text in the positive sample pair, and is used to increase the feature difference between the first video clip and the first text in the negative sample pair;

[0008] Calling the generative network to perform prediction processing on the sample association pairs to obtain sample video description information of the sample video data, and obtaining the prediction deviation of the generative network for the sample video description information based on the video description label;

[0009] The network parameters of the generated network are corrected according to the learning deviation and the prediction deviation to obtain a trained generated network; the trained generated network is used to generate video description information of the video data.

[0010] On one hand, the present application provides a video data processing device, the device comprising:

[0011] The acquisition module is used to acquire sample pairs and sample association pairs; a sample pair includes a first video clip and a first text, and the sample pair is divided into a positive sample pair and a negative sample pair according to the content difference between the first video clip and the first text; a sample association pair includes a second text and a second video clip to which the second text belongs in the sample video data; the sample video data has a video description tag;

[0012] A learning module, used for calling the generation network to perform feature comparison learning on the sample pair to obtain a learning bias; the learning bias is used to reduce the feature difference between the first video clip and the first text in the positive sample pair, and to increase the feature difference between the first video clip and the first text in the negative sample pair;

[0013] A prediction module is used to call the generation network to perform prediction processing on the sample association pair, obtain sample video description information of the sample video data, and obtain the prediction deviation of the generation network for the sample video description information based on the video description label;

[0014] The training module is used to correct the network parameters of the generated network according to the learning deviation and the prediction deviation to obtain a trained generated network; the trained generated network is used to generate video description information of the video data.

[0015] Optionally, the video content of the first video clip included in the positive sample pair is the same as the content of the first text description included in the positive sample pair; the video content of the first video clip included in the negative sample pair is different from the content of the first text description included in the negative sample pair.

[0016] Optionally, the above device is also used for:

[0017] Acquire sample audio data of the sample video data, and perform text conversion processing on the sample audio data to obtain sample text data;

[0018] The sample text data is subjected to text segmentation processing to obtain at least one second text contained in the sample text data.

[0019] Optionally, any one of the at least one second text is represented as a target text; and the apparatus is further used for:

[0020] Obtain the audio segment corresponding to the target text in the sample audio data;

[0021] Obtain the start and end time of the audio clip in the sample video data;

[0022] Cutting the sample video data to obtain a second video segment to which the target text belongs according to the start playback time and the end playback time;

[0023] A sample association pair is constructed according to the target text and the second video segment to which the target text belongs.

[0024] Optionally, the learning module calls the generation network to perform feature comparison learning on the sample pairs to obtain the learning deviation, including:

[0025] Calling the generative network to generate a first video feature of a first video clip in the positive example pair and a first text feature of a first text in the positive example pair;

[0026] Calling the generative network to generate a second video feature of the first video clip in the negative sample pair and a second text feature of the first text in the negative sample pair;

[0027] Based on the first video feature, the first text feature, the second video feature, and the second text feature, a learning bias of the generation network for the sample pair is obtained.

[0028] Optionally, the prediction module calls the generation network to perform prediction processing on the sample association pairs to obtain sample video description information of the sample video data, including:

[0029] Calling the generation network to generate a third text feature of the second text in the sample association pair and a third video feature of the second video clip in the sample association pair;

[0030] Obtain at least one word segment included in the second text in the sample association pair, and obtain a word embedding feature of each word segment;

[0031] The call generation network predicts the sample video description information based on the third text feature, the third video feature and the word embedding feature of each word segment.

[0032] Optionally, the prediction module calls the generation network to generate a third text feature of the sub-text in the sample association pair and a third video feature of the associated video segment of the sub-text in the sample association pair, including:

[0033] Obtaining a text embedding feature of a second text in the sample association pair and a video embedding feature of a second video clip in the sample association pair;

[0034] The generation network is called to generate a third text feature based on the text embedding feature, and the generation network is called to generate a third video feature based on the video embedding feature.

[0035] Optionally, the prediction module calls the generation network to predict the sample video description information based on the third text feature, the third video feature, and the word embedding feature of each word segment, including:

[0036] Calling the generation network to generate multimodal features of the sample video data based on the third text feature, the third video feature, and the word embedding feature of each word segment;

[0037] Predicting sample video description information of sample video data based on multimodal features.

[0038] Optionally, the generation network includes a video encoder and a text decoder, and the multimodal features are generated based on the video encoder;

[0039] The method in which the prediction module predicts sample video description information of sample video data based on the multimodal features includes:

[0040] Call the text decoder to predict the sample video description information based on multimodal features.

[0041] Optionally, the prediction module calls the text decoder to predict the sample video description information based on the multimodal features, including:

[0042] Calling the text decoder to predict the nth description character of the sample video data based on the multimodal features and the first n-1 description characters of the sample video data predicted by the text decoder; n is a positive integer;

[0043] If the nth description character does not carry a prediction end mark, calling a text decoder to predict the n+1th description character of the sample video data based on the multimodal features, the first n-1 description characters, and the nth description character;

[0044] If the n+1th description character carries a prediction end flag, the sample video description information is determined according to the first n-1 description characters, the nth description character and the n+1th description character.

[0045] Optionally, the training module corrects network parameters of the generated network according to the learning deviation and the prediction deviation to obtain a trained generated network, including:

[0046] Obtaining a first bias weight for a learning bias and a second bias weight for a prediction bias;

[0047] Based on the first bias weight and the second bias weight, a weighted sum of the learning bias and the prediction bias is performed to obtain a comprehensive training bias of the generating network for the sample pair and the sample association pair;

[0048] The network parameters of the generated network are corrected based on the comprehensive training deviation to obtain the trained generated network.

[0049] Optionally, the above device is also used for:

[0050] Acquire target video data and target text data extracted from the target video data;

[0051] Constructing target association pairs according to each subtext of the target text data and the video segments to which each subtext belongs in the target video data;

[0052] The trained generative network is called to perform prediction processing on the target association pairs to obtain the target video description information of the target video data.

[0053] Optionally, the above device is also used for:

[0054] Get the video retrieval text sent by the video client;

[0055] If the text similarity between the video retrieval text and the target video description information is greater than or equal to the similarity threshold, the target video data is returned to the video client, so that the video client outputs the target video data.

[0056] In one aspect, the present application provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes a method in one aspect of the present application.

[0057] In one aspect, the present application provides a computer-readable storage medium storing a computer program. The computer program includes program instructions. When the program instructions are executed by a processor, the processor executes the method in the above aspect.

[0058] According to one aspect of the present application, a computer program product or a computer program is provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method provided in various optional modes such as the above-mentioned one aspect.

[0059] The present application can first obtain sample pairs and sample association pairs; a sample pair includes a first video clip and a first text, and the sample pair is divided into a positive sample pair and a negative sample pair according to the content difference between the first video clip and the first text; a sample association pair includes a second text and a second video clip to which the second text belongs in the sample video data; the sample video data has a video description label; then, the generation network can be called to perform feature comparison learning on the sample pair to obtain a learning deviation; the learning deviation is used to reduce the feature difference between the first video clip and the first text in the positive sample pair, and to increase the feature difference between the first video clip and the first text in the negative sample pair; further, the generation network can be called to perform prediction processing on the sample association pair to obtain sample video description information of the sample video data, and obtain the prediction deviation of the generation network for the sample video description information based on the video description label; finally, the network parameters of the generation network can be corrected according to the learning deviation and the prediction deviation to obtain a trained generation network; the trained generation network can be used to generate video description information for the video data. It can be seen that the method proposed in the present application can combine video data and related text data of the video data (such as the second text) to train the generation network together. At the same time, comparative learning of sample pairs is also performed to assist the generation network in learning the deviation between the video data and the related text data of the video data. This doubly improves the accuracy of training the generation network, and then the trained generation network can also accurately generate video description information of the video data. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the technical solutions in the present application or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0061] Figure 1 It is a structural diagram of a network architecture provided by this application;

[0062] Figure 2 It is a schematic diagram of a network training scenario provided by this application;

[0063] Figure 3 It is a flowchart of a video data processing method provided by the present application;

[0064] Figure 4 It is a schematic diagram of a scenario for constructing a sample association pair provided by this application;

[0065] Figure 5 is a schematic diagram of a scenario for constructing a sample pair provided by this application;

[0066] Figure 6 It is a schematic diagram of a scenario for constructing negative sample pairs provided by this application;

[0067] Figure 7 It is a page schematic diagram of a client page provided by this application;

[0068] Figure 8 It is a flowchart of an information prediction method provided by this application;

[0069] Fig. 9 It is a schematic diagram of a feature addition scenario provided by this application;

[0070] Fig.10 This is a schematic diagram of a scenario of information prediction provided by this application;

[0071] Fig.11 This is a schematic diagram of a process for generating video description information provided by this application;

[0072] Fig.12 It is a structural schematic diagram of a video data processing device provided by the present application;

[0073] Fig.13 It is a structural schematic diagram of a computer device provided by this application. DETAILED DESCRIPTION

[0074] The following will be combined with the drawings in this application to clearly and completely describe the technical solutions in this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0075] This application involves artificial intelligence related technologies. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0076] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0077] This application mainly involves machine learning in artificial intelligence. Among them, machine learning (ML) is a multi-field interdisciplinary subject, involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines, specializing in how computers simulate or implement human learning behavior to acquire new knowledge or skills, and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all fields of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learning by teaching.

[0078] The machine learning involved in this application mainly refers to how to train a generative network to generate accurate video description information for video data through the trained generative network. For details, please refer to the following Figure 3 Description in the corresponding embodiment.

[0079] This application involves cloud technology, which refers to a hosting technology that unifies hardware, software, network and other resources within a wide area network or a local area network to achieve data computing, storage, processing and sharing.

[0080] Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, application technology, etc. based on the cloud computing business model. It can form a resource pool, which can be used on demand and is flexible and convenient. Cloud computing technology will become an important support. The backend services of the technical network system require a large amount of computing and storage resources, such as video websites, image websites and more portal websites. With the rapid development and application of the Internet industry, in the future, each item may have its own identification mark, which needs to be transmitted to the backend system for logical processing. Data of different levels will be processed separately. All kinds of industry data need strong system backing support, which can only be achieved through cloud computing.

[0081] The cloud technology involved in this application means that the background can push videos to the video client through the "cloud".

[0082] First of all, it should be explained that this application can display a prompt interface or pop-up window before collecting relevant user data (such as user data such as video retrieval information input by the user) and during the process of collecting relevant user data. The prompt interface or pop-up window is used to prompt the user that its relevant data is currently being collected, so that this application only starts to execute the relevant steps of obtaining user-related data after obtaining the user's confirmation operation on the prompt interface or pop-up window, otherwise (that is, when the user's confirmation operation on the prompt interface or pop-up window is not obtained), the relevant steps of obtaining user-related data are terminated, that is, the user's relevant data is not obtained. In other words, all user data collected by this application are collected with the user's consent and authorization, and the collection, use and processing of relevant user data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0083] See also Figure 1 , Figure 1 This is a schematic diagram of a network architecture provided by this application. Figure 1 As shown, the network architecture may include a server 200 and a terminal device cluster, and the terminal device cluster may include one or more terminal devices, and the number of terminal devices is not limited here. Figure 1 As shown, the multiple terminal devices may specifically include a terminal device 100a, a terminal device 101a, a terminal device 102a, ..., a terminal device 103a; Figure 1 As shown, the terminal device 100a, the terminal device 101a, the terminal device 102a, ..., the terminal device 103a can all be connected to the server 200 through a network, so that each terminal device can exchange data with the server 200 through the network connection.

[0084] like Figure 1 The server 200 shown can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal device can be a smart terminal such as a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart TV, etc. The following takes the communication between the terminal device 100a and the server 200 as an example to describe the specific embodiments of the present application.

[0085] Please also see Figure 2 , Figure 2 This is a schematic diagram of a network training scenario provided by this application. Figure 2As shown, the present application can be trained by the above-mentioned server 200 to generate a network, and the server 200 can obtain sample video data for training the generation network. Furthermore, the server 200 can construct a number of sample association pairs through the sample video data, and any sample association pair can include a sentence (i.e., text) and an associated video clip of the sentence. The sentence can be obtained by text conversion of the audio data of the sample video data, and the associated video clip of the sentence is a video clip in the sample video data that is played synchronously with the audio clip corresponding to the sentence, and the audio clip corresponding to the sentence can refer to the audio clip of the sample video data that can be converted into text to obtain the audio clip of the sentence. Therefore, it can be understood that the content expressed by the sentence in any sample association pair is consistent with the video content of the video clip in the sample association pair.

[0086] The server 200 may also obtain several sample pairs, which may include several positive sample pairs and several negative sample pairs. Any sample pair may also include a sentence and a video clip, but the content described by the sentence included in any positive sample pair is consistent with the video content of the video clip included in the positive sample pair, but the content described by the sentence included in any negative sample pair is inconsistent with the video content of the video clip included in the negative sample pair.

[0087] Therefore, the server 200 can train the generation network through the above-mentioned sample association pairs and the positive and negative sample pairs, so that the generation network can generate more accurate video description information of the sample video data by fusing the information of the video clips of the sample video data and the information of the sentences obtained by converting the audio of the sample video data into text through the sample association pairs.

[0088] At the same time, the server 200 also performs feature comparison learning on a number of positive and negative sample pairs to align the contextual semantics between sentences and video clips, so that the generation network can more accurately generate video description information of sample video data through sample association pairs.

[0089] By training the generative network with the above sample association pairs and sample pairs, a very accurate generative network can be trained. The generative network can combine the audio-converted text of the video data to more fully understand and learn the video data. The generative network obtained through training can generate accurate video description information for the video data. The specific process of training the generative network can be found in the following Figure 3 Corresponding to the relevant description in the embodiment.

[0090] Furthermore, the trained generation network can be used to generate video description information for each video data in the video library. Furthermore, if the server 200 obtains the video retrieval text sent by the terminal device 100a for retrieving videos, the server 200 can compare the text similarity between the video retrieval text and the video description information of each video data in the video library, and push the video data with a high text similarity between the video description information and the video retrieval text to the terminal device 200, and the terminal device 200 can display the video data pushed by the server 200 in the terminal interface, so as to achieve the purpose of accurate video retrieval by the terminal device 200 through the video retrieval text.

[0091] See also Figure 3 , Figure 3 It is a flowchart of a video data processing method provided by the present application. The execution subject in the embodiment of the present application can be a computer device or a computer device cluster composed of multiple computer devices. The computer device can be a server or a terminal device. Therefore, the execution subject in the embodiment of the present application can be a server, a terminal device, or a combination of a server and a terminal device. In the following, the execution subject in the embodiment of the present application is collectively referred to as a computer device for illustration. Figure 3 As shown, the method may include:

[0092] Step S101, obtaining sample pairs and sample association pairs; a sample pair includes a first video clip and a first text, and the sample pair is divided into a positive sample pair and a negative sample pair according to the content difference between the first video clip and the first text; a sample association pair includes a second text and a second video clip to which the second text belongs in the sample video data; the sample video data has a video description tag.

[0093] Optionally, the computer device can obtain several sample pairs and several sample association pairs. Among them, any sample pair can include a video clip and a text, the video clip included in the sample pair can be called a first video clip, and the text included in the sample pair can be called a first text. Therefore, it can be understood that a sample pair includes a first video clip and a first text. The number of sample pairs is determined according to the actual application scenario and is not limited to this. The sample pairs are mainly used for feature comparison learning of the generative network. For details, please refer to the description in the following step S102.

[0094] The sample pairs can be divided into positive sample pairs and negative sample pairs according to the content difference between the first video clip and the first text. The positive sample pair refers to a sample pair in which the video content of the first video clip is the same as the content described by the first text. In other words, the positive sample pair refers to a sample pair in which the content described by the first text is consistent with the video content of the first video clip, and the content described by the first text in the positive sample pair is the content of the first video clip in the positive sample pair.

[0095] On the contrary, a negative sample pair may refer to a sample pair in which the video content of the included first video clip is different from the content described by the included first text. In other words, a negative sample pair refers to a sample pair in which the content described by the included first text is inconsistent with the video content of the included first video clip, and the content described by the first text in the negative sample pair is not the content that occurred in the first video clip in the negative sample pair.

[0096] For a positive example pair, for example, the first text in a positive example pair may be "Now let's cut the watermelon, try to cut it as small as possible", then the first video clip in the positive example pair may be a clip of people cutting watermelon; for another example, the first text in a positive example pair may be "Now let's put salt in the soup", then the first video clip in the positive example pair may be a video clip of putting salt in the soup.

[0097] For negative sample pairs, for example, the first text in a negative sample pair may be "Next we add some ginger slices to the soup to make it taste better", but the first video clip in the negative sample pair is not a video clip of adding ginger slices to the soup; for another example, the first text in a negative sample pair is "We have just cut the vegetables", but the first video clip in the negative sample pair is not a video clip of cutting vegetables.

[0098] Any sample association pair may include a text and the video clip to which the text belongs in the corresponding video data. The text in the sample association pair may be referred to as the second text, and the video clip in the sample association pair may be referred to as the second video clip. The second text and the second video clip may be derived from the sample video data. For details, please refer to the following content description.

[0099] Optionally, the method of obtaining the above sample pairs and sample association pairs may include:

[0100] The computer device can obtain a number of sample video data for training the generation network. The number of the sample video data is determined according to the actual application scenario and is not limited. The following is an example of processing a sample audio data. Please refer to the following content description.

[0101] The computer device can obtain the audio data contained in the sample video data (which can be referred to as sample audio data). The sample audio data can be the audio data that comes with the sample video data and is played synchronously when the sample video data is played. The computer device can perform text conversion on the sample audio data, that is, convert the sample audio data into text data to obtain sample text data, and can obtain multiple sentences contained in the sample text data. A sentence can be understood as a subtext of the sample text data, and a subtext in the sample text data can be referred to as a second text, that is, through the above, several second texts contained in the sample text data can be obtained. One sample audio data can have one sample text data, and one sample text data can contain several second texts (such as at least one).

[0102] Optionally, the computer device may use ASR (Automatic Speech Recognition) to convert the sample audio data into text to obtain sample text data. The sample text data may contain separators (such as punctuation marks, such as commas, periods, etc.) obtained by text conversion. Furthermore, the computer device may perform text segmentation processing on the sample text data according to the separators in the sample text data (such as segmenting according to periods), and obtain several sentences in the sample text data. The several sentences may be several second texts contained in the sample text data, and the several second texts may be texts converted from the sound of people speaking in the sample video data.

[0103] Therefore, the process of constructing a sample association pair through a sample video data is taken as an example to illustrate: any one of the several second texts contained in the sample text data can be represented as the target text, and the audio segment corresponding to the target text in the sample audio data can be obtained. The audio segment corresponding to the target text can be the audio segment that can obtain the target text by performing text conversion. For example, the target text can be obtained by performing text conversion on audio segment 1 in the sample audio data, and then the audio segment 1 can be the audio segment corresponding to the target text in the sample audio data.

[0104] Furthermore, the computer device can obtain the start playback time and the end playback time of the audio segment corresponding to the target text in the sample video data. The start playback time is the time when the audio segment corresponding to the target text starts to be played when the sample video data is played (which can be understood as the start playback time of the audio segment corresponding to the target text), and the end playback time is the time when the audio segment corresponding to the target text ends to be played when the sample video data is played (which can be understood as the last playback time of the audio segment corresponding to the target text).

[0105] Therefore, the computer device can cut the second video segment to which the target text belongs in the sample video data according to the start playback time and the end playback time from the sample video data, and the cut second video segment is the video segment from the start playback time to the end playback time in the sample video data. In other words, the second video segment to which the target text belongs can also be the video segment in the sample video data that is played synchronously with the audio segment corresponding to the target text.

[0106] Therefore, the computer device can construct a sample association pair through the target text and the second video segment to which the target text belongs in the sample video data. The second video segment to which the target text belongs in the sample video data can be called the associated video segment of the target text.

[0107] The computer device can construct sample association pairs to which each second text in the sample video data belongs in the same manner as constructing sample association pairs to which the target text belongs, and obtain a plurality of sample association pairs, wherein one second text corresponds to one sample association pair. The sample association pairs can be pre-constructed before training the generative network, or can be constructed in real time when training the generative network, which is determined according to the actual application scenario and is not limited thereto.

[0108] See also Figure 4 , Figure 4 This is a schematic diagram of a scenario for constructing sample association pairs provided by this application. Figure 4 As shown, the sample video data has sample audio data, and the sample video data and the sample audio data are played synchronously. Sample text data can be obtained by converting the sample audio data into text, and the sample text data can include a plurality of sentences, and the plurality of sentences can be a plurality of subtexts included in the sample text data, and the plurality of subtexts can be a plurality of second texts included in the sample text data, and the plurality of second texts can include text 1, text 2, text 3 and text 4.

[0109] Among them, text 1 can be obtained by converting audio segment 1 in the sample audio data into text, text 2 can be obtained by converting audio segment 2 in the sample audio data into text, text 3 can be obtained by converting audio segment 3 in the sample audio data into text, and text 4 can be obtained by converting audio segment 4 in the sample audio data into text. It can be understood that only audio segments 1 to 4 in the sample audio data have speaking sounds and can be converted into corresponding texts.

[0110] The video clip played synchronously with audio clip 1 in the sample video data may be video clip 1, that is, the playback start time and playback end time of audio clip 1 and video clip 1 are the same; the video clip played synchronously with audio clip 2 in the sample video data may be video clip 2, that is, the playback start time and playback end time of audio clip 2 and video clip 2 are the same; the video clip played synchronously with audio clip 3 in the sample video data may be video clip 3, that is, the playback start time and playback end time of audio clip 3 and video clip 3 are the same; the video clip played synchronously with audio clip 4 in the sample video data may be video clip 4, that is, the playback start time and playback end time of audio clip 4 and video clip 4 are the same.

[0111] Therefore, if Figure 4 As shown, a sample association pair (such as sample association pair 1) can be constructed through the above-mentioned video clip 1 and text 1, a sample association pair (such as sample association pair 2) can be constructed through the above-mentioned video clip 2 and text 2, a sample association pair (such as sample association pair 3) can be constructed through the above-mentioned video clip 3 and text 3, and a sample association pair (such as sample association pair 4) can be constructed through the above-mentioned video clip 4 and text 4.

[0112] Furthermore, the computer device may obtain the above-mentioned positive sample pairs in the following manner: Generally, it can be considered that the content of a speech at a certain time in a certain video data is the same or similar (i.e., consistent) to the video content of the video clips before and after the speech at the time in the video data. Therefore, the computer device may generate positive sample pairs through each second text of the sample video data and the video clips before and after each second text in the sample video data. That is, the first text in the positive sample pair may be derived from the second text of the sample video data, and the first video clip in the positive sample pair may also be derived from the second video clip of the sample video data.

[0113] Among them, since the present application combines sample video data and sample text data obtained by converting sample audio data of the sample video data to collaboratively train and generate a network, the present application focuses on video clips with speaking sounds in the sample video data, that is, the present application focuses on video clips in the sample video data whose playing audio can be converted into the corresponding second text.

[0114] It can be understood that the second texts contained in the sample text data converted from the sample audio data of the sample video data are sorted according to the time sequence in which the corresponding audio clips are played, that is, the second texts contained in the sample text data are in order. Therefore, in the present application, a positive sample pair can be constructed based on any second text and other second texts adjacent to the second text (such as 5 second texts adjacent to and before the second text) in the sample video data in the associated video clip, that is, a positive sample pair can include any second text and other second texts adjacent to the second text in the sample video data; or, the present application can include constructing a positive sample pair based on any second text and the associated video clip of the second text in the sample video data, that is, a positive sample pair can include any second text and the associated video clip of the second text in the sample video data.

[0115] Furthermore, the computer device may obtain the above-mentioned negative sample pairs in the following ways: generally, it can be considered that the content of speech at a certain time in a certain video data is different from the video content of a video clip in the video data at a long time before and after the speech at that time (that is, it is inconsistent), and it can also be considered that the content of speech in a certain video data is different from the content of a video clip in another video data.

[0116] Therefore, the present application can construct negative sample pairs through the second texts of the same sample video data and the associated video clips of the second texts in the sample video data that are far away from the second texts, that is, the second text and video clips contained in the negative sample pair can be from the same sample video data, but the associated video clip of the second text needs to be separated from the video clip by a long time interval in the sample video data (such as more than 5 associated video clips of the second text); or, the present application can also construct negative sample pairs through the second text of a sample video data and the associated video clip of the second text in another sample video data, that is, the second text and video clip contained in the negative sample pair can come from different sample video data. That is, the first text in the negative sample pair can be the second text derived from the sample video data, and the first video clip in the negative sample pair can also be the second video clip derived from the sample video data.

[0117] For example, if the sample text data 1 of the sample video data 1 can sequentially include the second text 1, the second text 2, the second text 3, the second text 4, the second text 5, the second text 6, and the second text 7, the positive and negative sample pairs can be constructed by dividing them into two video clips (or other values, which are set according to the actual application scenario). The positive sample pairs can be constructed within two video clips, and the negative sample pairs can be constructed with a distance of more than two video clips.

[0118] Then, a positive sample pair can be constructed by the second text 4 and the associated video clip of the second text 4 in the sample video data 1, a positive sample pair can be constructed by the second text 4 and the associated video clip of the second text 2 in the sample video data 1, a positive sample pair can be constructed by the second text 4 and the associated video clip of the second text 3 in the sample video data 1, a positive sample pair can be constructed by the second text 4 and the associated video clip of the second text 5 in the sample video data 1, and a positive sample pair can be constructed by the second text 4 and the associated video clip of the second text 6 in the sample video data 1; moreover, a negative sample pair can be constructed by the second text 4 and the associated video clip of the second text 1 in the sample video data 1, and a negative sample pair can be constructed by the second text 4 and the associated video clip of the second text 7 in the sample video data 1. Alternatively, the second text 4 can also be used to construct a corresponding negative sample pair with an associated video clip of a second text in the sample video data 2, and the sample video data 2 and the sample video data 1 are not the same sample video data.

[0119] See also Figure 5 , Figure 5 is a schematic diagram of a scenario for constructing a sample pair provided by this application. Figure 5 As shown, the associated video segments corresponding to the second texts cut from the sample video data according to the audio-to-text conversion of the sample video data may include video segment 1, video segment 2, ..., video segment a, video segment a+1, ... video segment a+c. Wherein a and c are both positive integers, and the specific values ​​can be determined according to the actual application scenario.

[0120] Optionally, video segments within 2 video segments adjacent to a certain video segment can be considered as video segments adjacent to the video segment, and a positive sample pair can be constructed by a video segment adjacent to any video segment and a second text corresponding to the video segment (the associated video segment of the second text can be the video segment), and a negative sample pair can be constructed by a video segment that is not adjacent to any video segment (such as the interval is greater than 2 video segments) and the second text corresponding to the video segment.

[0121] Here, the second text corresponding to the video segment a+3 may be text a+3, and the associated video segment of text a+3 is video segment a+3, that is, the audio segment corresponding to text a+3 is played synchronously with video segment a+3.

[0122] like Figure 5As shown, the video segments adjacent to the video segment a+3 may include video segment a+1, video segment a+2, video segment a+4 and video segment a+5. Therefore, a negative sample pair can be constructed here by video segment 1 and text a+3, ..., a negative sample pair can be constructed by video segment a and text a+3, a positive sample pair can be constructed by video segment a+1 and text a+3, a positive sample pair can be constructed by video segment a+2 and text a+3, a positive sample pair can be constructed by video segment a+3 and text a+3, a positive sample pair can be constructed by video segment a+4 and text a+3, a positive sample pair can be constructed by video segment a+5 and text a+3, a negative sample pair can be constructed by video segment a+6 and text a+3, ..., a negative sample pair can be constructed by video segment a+c and text a+3. The principle of constructing positive and negative sample pairs by the second text corresponding to other video segments in the sample video data is the same.

[0123] See also Figure 6 , Figure 6 is a schematic diagram of a scenario for constructing negative sample pairs provided by this application. Figure 6 As shown, the several second texts of the sample video data obtained by text conversion of the audio data of the sample video data 1 may include text 1, text 2, text 3, .... The several video segments (i.e., the associated video segments of each second text) obtained by cutting the several second texts of the sample video data 2 from the sample video data may include video segment 1, video segment 2, video segment 3, ...

[0124] Therefore, the computer device can construct a negative sample pair by combining several second texts of sample video data 1 (text 1, text 2, text 3, ...) and several video clips of sample video data 2 (video clip 1, video clip 2, video clip 3, ...).

[0125] It is understandable that when the generation network is trained by the above-mentioned sample video data, the generation network needs to predict the video description information (which can be called sample video description information) of each sample video data, and the prediction deviation of the generation network for the video description information can also be obtained. The prediction deviation can be used to correct the network parameters of the generation network. For details, please refer to the content description of each step below. Therefore, in this application, the above-mentioned sample pairs are constructed by the above-mentioned sample video data, and the generation network is trained together with the sample pairs constructed based on the above-mentioned sample video data and the sample video data. The source of the samples can be unified, that is, the source of the sample pairs and the video data that need to predict the video description information are all sample video data, which can improve the training effect of the generation network, and then improve the accuracy of the generated network obtained by training.

[0126] Or, optionally, the sample pair may not be constructed by the above-mentioned sample video data, and the first text and the first video clip in the sample pair may come from any video data, as long as the content described by the first text in the positive sample pair is the same as the video content of the first video clip in the positive sample pair, and the content described by the first text in the negative sample pair is different from the video content of the first video clip in the negative sample pair. That is, the first text in the sample pair may not be derived from the second text in the sample video data, and the first video clip in the sample pair may not be derived from the second video clip in the sample video data. It is only by training the generative network together with the sample pairs obtained in this way and the above-mentioned sample video data (i.e., non-uniform samples) for which the video description information needs to be predicted, that the performance of the generative network obtained by training at this time will be slightly weaker than the performance of the generative network obtained by training when the above-mentioned samples are unified.

[0127] Among them, constructing the above-mentioned sample pairs and sample association pairs can be achieved by a machine. For example, the generative network can also have the ability to construct the above-mentioned sample pairs and sample association pairs. The above-mentioned sample pairs and sample association pairs can be constructed in advance before training the generative network, and can be directly obtained when training the generative network.

[0128] Step S102, calling the generation network to perform feature contrast learning on the sample pair to obtain a learning bias; the learning bias is used to reduce the feature difference between the first video clip and the first text in the positive sample pair, and to increase the feature difference between the first video clip and the first text in the negative sample pair.

[0129] Optionally, the computer device may call the generation network to perform feature comparison learning on the above-mentioned sample pairs to obtain the learning deviation of the generation network for the sample pairs, as described below.

[0130] The computer device may call the generation network to generate the video feature of the first video clip in the positive sample pair (which may be referred to as the first video feature) and the text feature of the first text in the positive sample pair (which may be referred to as the first text feature), and the computer device may also call the generation network to generate the video feature of the first video clip in the negative sample pair (which may be referred to as the second video feature) and the text feature of the first text in the negative sample pair (which may be referred to as the second text feature):

[0131] The computer device can obtain the video embedding feature of the first video clip in the positive sample pair, and the video embedding feature can be obtained by inputting the first video clip in the positive sample pair into a trained S3D network (separable 3DCNN, a video understanding network) and extracting it through the S3D network.

[0132] The computer device can also obtain the text embedding feature of the first text in the positive sample pair, and the text embedding feature can be obtained by inputting the first text in the positive sample pair into a trained BERT model (a word vector model) and extracting it through the BERT model.

[0133] Similarly, the computer device can also obtain the video embedding feature of the first video clip in the negative sample pair. The video embedding feature can also be obtained by inputting the first video clip in the negative sample pair into the above-mentioned trained S3D network and extracting it through the S3D network.

[0134] The computer device can also obtain the text embedding feature of the first text in the negative sample pair. The text embedding feature can also be obtained by inputting the first text in the negative sample pair into the above-mentioned trained BERT model and extracting it through the BERT model.

[0135] Furthermore, the computer device can call the generation network to perform feature projection on the text embedding feature of the first text in the positive example pair, and project it into a semantic space to obtain the first text feature of the first text in the positive example pair. It can be understood that the process of calling the generation network to perform feature projection on the text embedding feature of the first text in the positive example pair is the process of generating the first text feature through the text embedding feature of the first text in the positive example pair.

[0136] The computer device may also call the generation network to perform feature projection on the video embedding feature of the first video clip in the positive example pair, and project it into the semantic space to obtain the first video feature of the first video clip in the positive example pair. It can be understood that the process of calling the generation network to perform feature projection on the video embedding feature of the first video clip in the positive example pair is the process of generating the first video feature through the video embedding feature of the first video clip in the positive example pair.

[0137] Similarly, the computer device can call the generation network to perform feature projection on the text embedding feature of the first text in the above negative example pair, and project it into the above semantic space to obtain the second text feature of the first text in the above negative example pair. It can be understood that the process of calling the generation network to perform feature projection on the text embedding feature of the first text in the negative example pair is the process of generating the second text feature through the text embedding feature of the first text in the negative example pair.

[0138] The computer device can also call the generation network to perform feature projection on the video embedding feature of the first video clip in the negative sample pair, and project it into the semantic space to obtain the second video feature of the first video clip in the negative sample pair. It can be understood that the process of calling the generation network to perform feature projection on the video embedding feature of the first video clip in the negative sample pair is the process of generating the second video feature through the video embedding feature of the first video clip in the negative sample pair.

[0139] Optionally, the text embedding feature of any first text in the positive and negative sample pairs can be recorded as s, and the generative network can perform feature projection on the text embedding feature s of the first text through the feature projection function g to obtain the text feature x of the first text s , as shown in the following formula:

[0140] x s =g(s) (1)

[0141] For example, the feature projection function g can be w1x1+b1, where w1 is the network parameter that needs to be trained and updated in the generative network, x1 represents the input of the function g (such as s), and b1 represents the bias, and the value of b1 can be set in advance. The text features of the first text in the positive and negative sample pairs can be generated by calling the generative network through the principle of formula (1), except that the text features of the first text in the positive sample pair can be called the first text features, and the text features of the first text in the negative sample pair can be called the second text features.

[0142] Optionally, the video embedding feature of any first video clip in the positive and negative sample pairs can be recorded as v, and the generating network can perform feature projection on the video embedding feature v of the first video clip through the feature projection function f to obtain the video feature x of the first video clip. v , as shown in the following formula:

[0143] x v =f(v) (2)

[0144] For example, the feature projection function f can be w2x2+b2, where w2 is a network parameter that needs to be trained and updated in the generative network, x2 represents the input of the function f (such as v), and b2 represents the bias, and the value of b2 can be set in advance. The video features of the first video clip in the positive and negative sample pairs can be generated by calling the generative network through the principle of formula (2), except that the video features of the first video clip in the positive sample pair can be called the first video features, and the video features of the first video clip in the negative sample pair can be called the second video features.

[0145] The first video feature, the first text feature, the second video feature and the second text feature are all features in the same semantic space.

[0146] Furthermore, the computer network can obtain the learning deviation L of the generated network for the sample pair through the first video feature, the first text feature, the second video feature and the second text feature generated above. CL , as shown in the following formula:

[0147]

[0148] Among them, x s x v represents the inner product between the text feature of the first text s and the video feature of the first video clip v. P represents a set of several positive sample pairs. A positive sample pair in the set P can correspond to an x s x v (i.e., the inner product between the text feature of the first text in the positive sample pair and the video feature of the first video clip in the positive sample pair), that is, a positive sample pair in the set P can correspond to a (index), Indicates the corresponding positive sample pairs in the set P Similarly, N represents the set of negative sample pairs. A negative sample pair in set N can correspond to an x s X v (i.e., the inner product between the text feature of the first text in the negative sample pair and the video feature of the first video clip in the negative sample pair), that is, a negative sample pair in the set N can correspond to a (index), Indicates the corresponding negative sample pairs in the set P sum.

[0149] It should be noted that during the training process, the learning deviation L CL Approaching 0, making the learning deviation approach 0 is to make the latter part of the log Approaching 1, that is, making the inner product x corresponding to each positive sample pair in the set P s x v Larger, and makes the inner product x corresponding to each pair of negative samples in the set N s x v Smaller.

[0150] Among them, it can be understood that the inner product x corresponding to each positive sample pair in the set P is s x vThe effect achieved is that the text features of the first text in each positive example pair are closer to the video features of the first video clip in each positive example pair, that is, the feature difference between the first text and the first video clip in the same positive example pair is smaller, thereby realizing cross-modal feature alignment of texts and video clips with similar or identical content (which can be understood as context alignment).

[0151] Similarly, the inner product x corresponding to each pair of negative samples in the set N is s x v The effect of being smaller is that the text features of the first text in each negative sample pair (generated by the generative network) are less similar to the video features of the first video clip in each negative sample pair (generated by the generative network), that is, the feature difference between the first text and the first video clip in the same negative sample pair is greater.

[0152] In order to perform effective contrastive learning on the generative network, each positive sample pair in the above-mentioned set P and each negative sample pair in the set N can contain the same first text, and each positive sample pair in the set P and each negative sample pair in the set N can contain different first video clips. It can be understood that during contrastive learning, it is necessary to find an anchor point (such as the same first text in each positive sample pair in the set P and each negative sample pair in the set N), and perform contrastive learning based on the anchor point, such as aligning the features of the video clip with the same content as the first text with the features of the text (that is, the feature difference is smaller), and making the features of the video clip with different content from the first text different from the features of the text more different.

[0153] Therefore, it can be understood that, for different second texts of the sample video data, for any second text, a positive sample pair containing the second text (which may contain the associated video clips of the second text, or may contain the associated video clips of other second texts in the sample video data that are adjacent to the second text) and a negative sample pair (which may contain the associated video clips of other second texts in the sample video data that are not adjacent to the second text, or may contain video clips in the sample video data other than the sample video data to which the second text belongs) can be constructed, and several positive sample pairs containing the second text can constitute the above-mentioned set P corresponding to the second text, and several negative sample pairs containing the second text can constitute the above-mentioned set N corresponding to the second text.

[0154] Therefore, for each second text of the sample video data, a learning deviation L as shown in formula (3) can be generated for each set P and set N corresponding to the second text. CL , by learning the deviation L corresponding to each second text CLThe final learning bias of the generative network for the sample pair can be obtained by summing them up. During training, the learning bias is used to reduce the feature difference between the first video clip and the first text in the positive sample pair, and to increase the feature difference between the first video clip and the first text in the negative sample pair.

[0155] The above-mentioned method of comparative learning of sample pairs can be called a method of comparative spatial constraints on sample pairs. Optionally, when performing comparative learning on sample pairs, in addition to the comparative spatial constraint method, an attention mechanism or Euclidean distance constraint can also be used to perform comparative learning on sample pairs. The principle of learning is also to shorten the feature distance between the first text and the first video clip in the positive sample pair (i.e., reduce the feature difference), and to increase the feature distance between the first text and the first video clip in the negative sample pair (i.e., increase the feature difference). The specific comparative learning method used for the sample pairs can be determined according to the actual application scenario, and there is no limitation on this.

[0156] Step S103, calling the generation network to perform prediction processing on the sample association pairs, obtaining sample video description information of the sample video data, and obtaining the prediction deviation of the generation network for the sample video description information based on the video description label.

[0157] Optionally, the computer device can call the generation network to perform prediction processing on the above-mentioned sample association pairs to obtain video description information of the sample video data. The video description information of the sample video data predicted by the generation network can be referred to as the sample video description information. In other words, the computer device can call the generation network to predict the video description information of the sample video data based on the above-mentioned sample association pairs. Here, the sample video description information of the sample video data is predicted by a sample association pair of a sample video data. If multiple sample video data are input, the sample video description information of each sample video data is predicted separately by the sample association pair of each sample video data. Specifically, how to call the generation network to predict the sample video description information of the sample video data through the sample association pairs can be found in the following. Figure 8 The content description in the corresponding embodiment.

[0158] Furthermore, the sample video data may also have a video description tag, which marks the actual video description information of the sample video data. One sample video data may have one video description tag, and different sample video data may have different video description tags.

[0159] Therefore, the computer device can obtain the prediction deviation L of the sample video description information of the sample video data generated by the generation network through the video description label of the sample video data. ce , as shown in the following formula:

[0160] L ce =crossentropy(Y ′ ,Y) (4)

[0161] Among them, crossentropy represents cross entropy, and prediction deviation L ce It can be Y ′ The cross entropy loss between Y and ′ represents the sample video description information (which can be text) of the sample video data predicted by calling the generation network, and Y represents the real video description information (which can also be text) of the sample video data marked by the video description label of the sample video data.

[0162] If multiple sample video data are input to the generative network at the same time, the generative network can obtain the prediction deviation L as shown in formula (4) by predicting each sample video data. ce , one sample video data corresponds to one prediction deviation L ce , by taking the prediction deviation L corresponding to each sample video data ce By summing up, we can get the final prediction deviation of the generated network for the sample video data. The prediction deviation represents the difference between the sample video description information of the sample video data predicted by the generated network and the actual video description information of the sample video data.

[0163] Step S104, correcting the network parameters of the generated network according to the learning deviation and the prediction deviation to obtain a trained generated network; the trained generated network is used to generate video description information of the video data.

[0164] Optionally, the computer device can correct the network parameters of the generation network according to the final learning deviation and prediction deviation of the generation network obtained above to obtain a trained generation network. The trained generation network can be used to accurately generate video description information for any video data.

[0165] The video description information of the video data generated by the trained generation network can be used to support the user's retrieval of the video data in the future. For example, the user can input text information (which can be called video retrieval text) for video retrieval in the video client, and the text information can be given to the computer device, and the computer device can compare the text information with the video description information (which can also be text) of each video data generated by the trained generation network, and push the video data whose text similarity ranking of the compared corresponding video description information and the text information is in the top T (positive integer, the specific value can be set according to the actual application scenario) to the user (i.e., to the video client), or push the video data whose text similarity of the compared corresponding video description information and the text information is greater than or equal to the similarity threshold (which can be set according to the actual application scenario) to the user, so as to achieve the purpose of accurately pushing videos to the user.

[0166] Optionally, the process of the computer device correcting the network parameters of the generated network according to the above-mentioned final learning deviation and prediction deviation of the generated network may include:

[0167] The computer device can obtain a comprehensive training bias for the generated network based on the learning bias and the prediction bias: the computer device can obtain a first bias weight for the learning bias and a second bias weight for the prediction bias. The first bias weight and the second bias weight can be set according to the importance of the two tasks of comparing learning and describing information prediction in the actual application scenario. For example, the first bias weight can be λ and the second bias weight can be 1.

[0168] Furthermore, the computer performs a weighted summation of the learning bias and the prediction bias through the first bias weight and the second bias weight, and obtains the final comprehensive training bias of the generated network for the sample pair and the sample association pair. As shown in the following formula, the comprehensive training bias L of the generated network can be:

[0169] L=L ce +λL CL (5)

[0170] Furthermore, the computer device can correct the network parameters of the generated network through the above-mentioned comprehensive training deviation. The goal of the correction can be to make the comprehensive training deviation reach the minimum value (such as approaching 0). The computer network can continuously use a number of sample video data to iteratively train the generated network through the above process. When the number of training times for the generated network reaches a threshold number or the generated network is trained to a convergence state, the generated network at this time can be used as a trained generated network.

[0171] Optionally, after obtaining the trained generative network, the computer device can also use the sample video description information generated by the generative network to test the quality of the generative network, saving the workload of manually re-labeling test data.

[0172] After training the above-mentioned trained generation network, the computer device can also obtain target video data, and the target video data can be any video data for which video description information needs to be generated.

[0173] The computer device can extract the target text data according to the target video data. The process of extracting the target text data according to the target video data is the same as the process of extracting the sample text data according to the sample video data. For example, the computer device can first obtain the audio data of the target video data, and then perform text conversion on the audio data to obtain the target text data. Then, the target text data is also subjected to text segmentation processing to obtain several subtexts contained in the target text data. Here, the meanings of the several subtexts contained in the target text data are equivalent to the meanings of the several second texts contained in the sample text data.

[0174] Therefore, the computer device can construct the target association pair according to each subtext of the target text data and the video clips to which each subtext belongs in the target video data (such as the associated video clips of each subtext in the target video data). The process of determining the associated video clips of each subtext of the target text data in the target video data is the same as the process of determining the associated video clips of the target text in the sample video data, and the process of constructing the target association pair according to each subtext and the associated video clips of each subtext is the same as the process of constructing the sample association pair according to each second text and the associated video clips of each second text.

[0175] Then, the computer device can call the trained generation network to perform prediction processing on the target association pair, so as to generate the video description information of the target video data. The generated video description information of the target video data can be referred to as the target video description information. The process of the computer device calling the trained generation network to perform prediction processing on the target association pair to generate the target video description information is the same as the process of calling the generation network to perform prediction processing on the sample association pair to generate the sample video description information. For details, please refer to the following Figure 8 Corresponding to the description in the embodiment.

[0176] The generated target video description information of the target video data may be used to support the user's retrieval of the target video data.

[0177] Subsequently, if the computer device obtains a video retrieval text (which may be a text entered by a user of the video client for retrieving related videos) sent by a video client (which may be any client capable of performing video retrieval), the computer device may compare the text similarity between the video retrieval text and the target video description information of the target video data. If the text similarity is greater than or equal to a similarity threshold, the target video data may be returned to the video client so that the video client may input the target video data. For example, the video client may output and display the target video data on a video retrieval result page, thereby achieving the purpose of pushing the target video data to the user.

[0178] See also Figure 7 , Figure 7 This is a schematic diagram of a client page provided by this application. Figure 7 As shown, the video client can be a communication client, and the client page 1 includes several function bars of the video client, including a function bar for the circle of friends, a function bar for scanning, a function bar for nearby, a function bar for shaking, and a function bar for searching.

[0179] The user can click the search function bar in the client page 1. The video client can be displayed from the client page 1 to the client page 2. The client page 2 includes an input box for data retrieval. The user can enter the relevant search text (which can be called video search text) of the video to be searched in the input box. Here, the search text is "how to plant flowers", and the video client can be displayed from the client page 2 to the client page 3.

[0180] The client page 3 can display the related videos retrieved by the above search text "how to plant flowers", and the related videos can include video data whose text similarity between the video description information in the video library (generated by the above trained generation network) and the above search text "how to plant flowers" is greater than or equal to the threshold.

[0181] In the present application, accurate video description information for each video data in the video library can be generated through a trained generation network. Even if there are some video data in the video library whose video content does not match the corresponding video title, the video title of the video data can be ignored. Instead, the accurate video description information of each video data generated can be used to achieve accurate retrieval of related video data (such as video data whose video content is the same as that described by the retrieval text) through the retrieval text.

[0182] The present application can utilize the information of the ASR audio transcription text that comes with the video (such as the sample text data of the sample video data mentioned above) to fully understand the content of the video, and generate accurate descriptive sentences for the video (such as video description information) through aligning the contextual semantics of the video and the ASR audio transcription text (such as the semantics between the second text and the associated video clips of the second text in the sample video data). The accurate descriptive sentences of the video can also improve the accuracy and experience of the video search service.

[0183] In addition, in the present application, there may be a situation where there is a content difference between the content described in the second text in the sample association pair and the video content of the associated video clip corresponding to the second text. Therefore, the present application can solve the problem of content differences between the second text and the corresponding video clip in the sample association pair by performing self-supervised comparative learning on positive and negative sample pairs, thereby achieving accurate alignment training of the contextual semantics between the text and the corresponding video clip, and further improving the training effect of the generative network.

[0184] In summary, this application can reduce cross-modal semantic noise by contextually aligning text with corresponding video clips (such as projecting the text features of the text and the video features of the video clips into the same semantic space), and can fully learn the representation of contextual video clips and sentences (i.e., text), making the obtained representation more sound. In addition, this application can optimize the learning of the generative network under the self-supervised paradigm without manually marking the contextual correspondence between videos (i.e., video clips) and sentences (i.e., texts), and strengthen the understanding of local semantic features (such as the features of local video clips and text semantics) to make the final generated video description information more accurate.

[0185] The present application can first obtain sample pairs and sample association pairs; a sample pair includes a first video clip and a first text, and the sample pair is divided into a positive sample pair and a negative sample pair according to the content difference between the first video clip and the first text; a sample association pair includes a second text and a second video clip to which the second text belongs in the sample video data; the sample video data has a video description label; then, the generation network can be called to perform feature comparison learning on the sample pair to obtain a learning deviation; the learning deviation is used to reduce the feature difference between the first video clip and the first text in the positive sample pair, and to increase the feature difference between the first video clip and the first text in the negative sample pair; further, the generation network can be called to perform prediction processing on the sample association pair to obtain sample video description information of the sample video data, and obtain the prediction deviation of the generation network for the sample video description information based on the video description label; finally, the network parameters of the generation network can be corrected according to the learning deviation and the prediction deviation to obtain a trained generation network; the trained generation network can be used to generate video description information for the video data. It can be seen that the method proposed in the present application can combine video data and related text data of the video data (such as the second text) to train the generation network together. At the same time, comparative learning of sample pairs is also performed to assist the generation network in learning the deviation between the video data and the related text data of the video data. This doubly improves the accuracy of training the generation network, and then the trained generation network can also accurately generate video description information of the video data.

[0186] See also Figure 8 , Figure 8 is a flow chart of an information prediction method provided by the present application. The execution subject in the embodiment of the present application may also be the above-mentioned computer device, such as Figure 8 As shown, the method may include:

[0187] Step S201, calling a generation network to generate a third text feature of a second text in a sample association pair and a third video feature of a second video clip in the sample association pair.

[0188] Optionally, the computer device may call the generation network to generate the text feature of the second text in the sample association pair, and the text feature of the second text in the sample association pair generated by calling the generation network may be referred to as the third text feature.

[0189] The computer device may also call the generation network to generate the video feature of the second video clip in the sample association pair, and the video feature of the second video clip in the sample association pair generated by calling the generation network may be referred to as the third video feature.

[0190] The principle of the computer device calling the generation network to generate the text features of the second text in the sample association pair is the same as the principle of the computer device calling the generation network to generate the text features of the first text in the sample pair (such as the principle of the above formula (1)). For example, the computer device can obtain the text embedding features of the second text in the sample association pair, and can call the generation network to perform feature projection on the text embedding features of the second text in the sample association pair, so as to obtain the above third text features of the second text in the sample association pair.

[0191] Similarly, the principle of the computer device calling the generation network to generate the video features of the second video clip in the sample association pair is the same as the principle of the computer device calling the generation network to generate the video features of the first video clip in the sample pair (such as the principle of the above formula (2)). For example, the computer device can obtain the video embedding features of the second video clip in the sample association pair, and can call the generation network to perform feature projection on the video embedding features of the second video clip in the sample association pair, so as to obtain the video features of the second video clip in the sample association pair.

[0192] Step S202: Obtain at least one word segment included in the second text in the sample association pair, and obtain word embedding features of each word segment.

[0193] Optionally, the computer device may also obtain at least one word segmentation included in the second text in the sample association pair. For example, the computer device may perform word segmentation processing on the second text in the sample association pair to obtain several word segments included in the second text in the sample association pair.

[0194] The computer device can also obtain the word embedding features of each word segment contained in the second text in the sample association pair, such as inputting each word segment into a trained BERT model, and generating the word embedding features (which can be feature vectors) of each word segment through the BERT model.

[0195] Step S203, calling the generation network to predict sample video description information based on the third text feature, the third video feature and the word embedding feature of each word segment.

[0196] Optionally, the computer device can call the generation network to generate a multimodal feature of the sample video data based on the above-mentioned third text feature, the third video feature and the word embedding feature of each word segment. The multimodal feature can be understood as a feature that integrates the video clips of the sample video data itself and the semantic alignment between the second texts of the sample video data. The multimodal feature is the final global representation of the sample video data.

[0197] Furthermore, the computer device can call the generation network to predict the sample video description information of the sample video data based on the above multimodal features. The process can be described as follows.

[0198] Optionally, the generative network may include a video encoder and a text decoder, both of which may be based on a Transformer network (a graph neural network, mainly a network based on a self-attention mechanism), which is mainly composed of modules such as a multi-head self-attention mechanism, skip connections, layer normalization, and a feed-forward neural network.

[0199] Among them, the video encoder can be used to combine the contextual semantic information between the second video clip and the second text in the sample association pair to obtain a better video representation of the sample video data (such as a multimodal feature). Therefore, the computer device can input the third text feature of the second text in the above-mentioned sample association pair, the third video feature of the second video clip in the sample association pair, and the word embedding features of each word segment contained in the second text in the sample association pair into the above-mentioned video encoder. The feature dimensions of the third text feature, the third video feature, and the word embedding feature of any word segment can be the same, such as 512 dimensions. The video encoder can add the third text feature, the third video feature, and the word embedding feature of any word segment belonging to the same sample association pair in the sample association pair (such as adding the feature values ​​at the corresponding positions of the three features, a total of 512 positions), and obtain the features after the addition, and the features after the addition are also 512 dimensions.

[0200] From the above, we can know that since the video encoder can sum the third text features, the third video features and the word embedding features of any word segmentation belonging to the same sample association pair, a word segmentation contained in the second text of a sample association pair can correspond to a summed feature. In this way, if there are multiple sample association pairs, the second text in each sample association pair can also contain multiple word segmentations, then several summed features can be obtained, and one word segmentation corresponds to one summed feature.

[0201] See also Fig. 9 , Fig. 9 is a schematic diagram of a feature addition scenario provided by this application. Fig. 9 As shown, the third text feature of the second text in the sample association pair can be [A1, A2, A3, A4, A5], the third video feature of the second video clip in the sample association pair can be [B1, B2, B3, B4, B5], and the word embedding feature of a word segment of the second text in the sample association pair can be [C1, C2, C3, C4, C5].

[0202] Therefore, the third text feature [A1, A2, A3, A4, A5], the third video feature [B1, B2, B3, B4, B5] and the word embedding feature [C1, C2, C3, C4, C5] are added together to obtain a summed feature [D1, D2, D3, D4, D5]. Among them, D1 is equal to A1+B1+C1, D2 is equal to A2+B2+C2, D3 is equal to A3+B3+C3, D4 is equal to A4+B4+C4, and D5 is equal to A5+B5+C5.

[0203] Furthermore, the video encoder can use a multi-head self-attention mechanism to cross-learn the summed features (the summed features belonging to the same sample video data), which can be understood as feature transfer (i.e., information transfer) between the summed features, so as to learn and generate the multimodal features of the sample video data. The multimodal features finally generated can also be of the same dimension as the third text feature, the third video feature, and the word embedding feature of the word segmentation. For example, the multimodal features can also be 512-dimensional.

[0204] It is understandable that if multiple sample video data are input, multimodal features of each sample video data can be generated according to a number of added features corresponding to each sample video data.

[0205] The multimodal features generated above are the global features of the sample video data. The global features are high-quality video features obtained by fully integrating the global information of the sample video data (such as the features of the entire video clip and the second text) and the local information (such as the features of each word in the second text). As shown in the following formula, the multimodal features x of the sample video data are D It can be:

[0206] x D = TransformerEncoder(w, x sg , x vg ) (6)

[0207] Wherein, TransformerEncoder represents the above video encoder, w represents each word contained in the second text in the sample association pair, and x sg represents the third text feature of the second text in the sample association pair, x vg The third video feature represents the second video segment in the sample association pair.

[0208] Furthermore, the computer device can input the multimodal features of the sample video data generated above into the text decoder to call the text decoder to predict the sample video description information of the sample video data based on the multimodal features. As shown in the following formula, the sample video description information Y′ of the sample video data is:

[0209] Y′=TransformerDecoder(x D ) (7)

[0210] Among them, TransformerDecoder represents the text decoder, x D is the multimodal feature of the above sample video data.

[0211] Optionally, the process of a computer device calling a text decoder to generate sample video description information of the sample video data based on the multimodal features of the sample video data may include: the computer device may call a text decoder to perform word-by-word prediction based on the multimodal features of the sample video data to obtain the sample video description information of the sample video data, and when predicting the subsequent characters (which may be referred to as description characters), the previously predicted characters (which may be referred to as description characters) may be used together with the multimodal features for prediction, as described below.

[0212] The computer device may call the text decoder to predict the nth description character of the sample video data based on the multimodal features and the first n-1 description characters of the sample video data predicted by the text decoder, where n is a positive integer. For example, the text decoder may predict the probability of each character in the character table being the nth character based on the multimodal features and the predicted first n-1 description characters, and use the character with the highest predicted probability in the character table as the predicted nth description character.

[0213] It is understandable that if n is equal to 1, the first n-1 description characters do not exist, and the nth description character is the first description character that needs to be predicted. At this time, the first description character can be predicted by directly inputting the multimodal features into the text decoder.

[0214] If n is equal to 2, the first n-1 description characters include the predicted first description character, and the nth description character is the second description character to be predicted. At this time, the multimodal features and the first description character are input into the text decoder to predict the second description character.

[0215] If n is equal to 3, the first n-1 description characters include the predicted first and second description characters, and the nth description character is the third description character to be predicted. At this time, the multimodal features, the first and second description characters are input into the text decoder to predict the third description character. This is analogous. If a description character is predicted and the description character also has a prediction end character, it indicates that the description character is the last description character predicted, and the description character is the last description character of the predicted sample video description information.

[0216] In summary, if the nth description character does not carry a prediction end mark, the text decoder can be called to predict the n+1th description character of the sample video data based on the above multimodal features, the first n description characters (including the first n-1 description characters and the nth description character). If the n+1th description character carries a prediction end mark, the sample video description information can be obtained based on the predicted n+1 description characters (including the above-mentioned first n-1 description characters, the nth description character and the n+1th description character), and the sample video description information includes the predicted n+1 description characters in sequence (according to the prediction order of the n+1 description characters).

[0217] See also Fig.10 , Fig.10 This is a schematic diagram of a scenario of information prediction provided by the present application. The computer device can input the above-obtained third text features, the third video features, and the word embedding features of each word in the second text into a video encoder, generate multimodal features of sample video data through the video encoder, and then input the multimodal features into a text decoder, so that the sample video description information of the sample video data can be predicted by the text decoder.

[0218] Through the above process, the sample video description information of the sample video data can be predicted by the video encoder and the text decoder. Optionally, in addition to using the above-mentioned transformer encoder and transformer decoder to predict the sample video description information of the sample video data, the present application can also replace the Transformer with other networks, such as LSTM (long short-term memory neural network) or ConvLSTM (convolutional long short-term memory neural network) and other networks that can learn video clips / sentence sequences to generate video representations (such as multimodal features), and then predict the sample video description information of the sample video data through the video representation.

[0219] The present application can achieve a more comprehensive understanding of the video through text by performing contextual semantic alignment of the video clip and the corresponding second text (for example, the content semantics of a video clip can be considered to be the same as the content semantics of the second text converted from the audio clip played synchronously with the video clip), thereby generating accurate video description information of the video data, and the video description information can subsequently be used to achieve accurate retrieval (i.e., search) of the video data by retrieving the text.

[0220] See also Fig.11 , Fig.11 This is a schematic diagram of a process for generating video description information provided by this application. Fig.11 As shown, S1, the computer device may first obtain the video data to be processed (such as the sample video data mentioned above). S2, the computer device may also obtain the audio transcription text of the video data to be processed (such as the sample text data mentioned above).

[0221] Furthermore, in S3, the computer device can segment the above-mentioned video data to be processed according to the sentences of the text (such as several second texts included in the sample video data) and time (such as the playback time of the corresponding audio segment) to obtain the associated video segments of each sentence (the same principle as obtaining the associated video segments of the target text). The above-mentioned sample association pairs can be constructed through each sentence and the associated video segments of each sentence.

[0222] Then, S4, the computer device can use the sample pairs to perform comparative learning on the generative network, so that the generative network can complete the semantic alignment of the context between the video clip and the corresponding text (such as the above can be a video clip, and the following can be the text corresponding to the video clip) in a self-supervised manner. S5, the computer device can use an encoder (such as the above video encoder) to fuse global information (such as the information of the video clip and the text corresponding to the video clip included in the sample association pair) and local information (such as the information of several word segments included in the second text in the sample association pair), and the video feature representation of the video data to be processed (such as the above multimodal feature) can be obtained.

[0223] Finally, S6, the computer device can input the video feature representation into a decoder (such as the above-mentioned text decoder), and generate video description information of the sample video data to be processed (such as the above-mentioned sample video description information) in the decoder. The principle of calling the generation network to generate the video description information of the target video data is the same as the principle of calling the generation network to generate the video description information of the sample video data.

[0224] This application uses ASR text (such as sample text data) to supplement video data, which can supplement a lot of details or conceptual information of video data, thereby generating video description information with better quality of video data. In addition, using the context of locally aligned video clips and sentences (such as second text) can handle the difficulty of inconsistent semantic description of audio and video of video data, improve the multimodal video representation capability, and thus improve the accuracy of video description information generated by the generation network for video data.

[0225] See also Fig.12 , Fig.12 : is a schematic diagram of the structure of a video data processing device provided by the present application. The video data processing device may be a computer program (including program code) running in a computer device. For example, the video data processing device is an application software. The video data processing device may be used to execute the corresponding steps of the method provided in the embodiment of the present application. Fig.12 As shown, the video data processing device 1 may include: an acquisition module 11 , a learning module 12 , a prediction module 13 and a training module 14 .

[0226] The acquisition module 11 is used to acquire sample pairs and sample association pairs; a sample pair includes a first video clip and a first text, and the sample pair is divided into a positive sample pair and a negative sample pair according to the content difference between the first video clip and the first text; a sample association pair includes a second text and a second video clip to which the second text belongs in the sample video data; the sample video data has a video description tag;

[0227] The learning module 12 is used to call the generation network to perform feature comparison learning on the sample pair to obtain a learning deviation; the learning deviation is used to reduce the feature difference between the first video clip and the first text in the positive sample pair, and to increase the feature difference between the first video clip and the first text in the negative sample pair;

[0228] The prediction module 13 is used to call the generation network to perform prediction processing on the sample association pair, obtain sample video description information of the sample video data, and obtain the prediction deviation of the generation network for the sample video description information based on the video description label;

[0229] The training module 14 is used to correct the network parameters of the generated network according to the learning deviation and the prediction deviation to obtain a trained generated network; the trained generated network is used to generate video description information of the video data.

[0230] Optionally, the video content of the first video clip included in the positive sample pair is the same as the content of the first text description included in the positive sample pair; the video content of the first video clip included in the negative sample pair is different from the content of the first text description included in the negative sample pair.

[0231] Optionally, the device 1 is also used for:

[0232] Acquire sample audio data of the sample video data, and perform text conversion processing on the sample audio data to obtain sample text data;

[0233] The sample text data is subjected to text segmentation processing to obtain at least one second text contained in the sample text data.

[0234] Optionally, any one of the at least one second text is represented as a target text; and the apparatus 1 is further used for:

[0235] Obtain the audio segment corresponding to the target text in the sample audio data;

[0236] Obtain the start and end time of the audio clip in the sample video data;

[0237] Cutting the sample video data to obtain a second video segment to which the target text belongs according to the start playback time and the end playback time;

[0238] A sample association pair is constructed according to the target text and the second video segment to which the target text belongs.

[0239] Optionally, the learning module 12 calls the generation network to perform feature comparison learning on the sample pairs to obtain a learning deviation, including:

[0240] Calling the generative network to generate a first video feature of a first video clip in the positive example pair and a first text feature of a first text in the positive example pair;

[0241] Calling the generative network to generate a second video feature of the first video clip in the negative sample pair and a second text feature of the first text in the negative sample pair;

[0242] Based on the first video feature, the first text feature, the second video feature, and the second text feature, a learning bias of the generation network for the sample pair is obtained.

[0243] Optionally, the prediction module 13 calls the generation network to perform prediction processing on the sample association pairs to obtain the sample video description information of the sample video data, including:

[0244] Calling the generation network to generate a third text feature of the second text in the sample association pair and a third video feature of the second video clip in the sample association pair;

[0245] Obtain at least one word segment included in the second text in the sample association pair, and obtain a word embedding feature of each word segment;

[0246] The call generation network predicts the sample video description information based on the third text feature, the third video feature and the word embedding feature of each word segment.

[0247] Optionally, the prediction module 13 calls the generation network to generate the third text feature of the sub-text in the sample association pair and the third video feature of the associated video segment of the sub-text in the sample association pair, including:

[0248] Obtaining a text embedding feature of a second text in the sample association pair and a video embedding feature of a second video clip in the sample association pair;

[0249] The generation network is called to generate a third text feature based on the text embedding feature, and the generation network is called to generate a third video feature based on the video embedding feature.

[0250] Optionally, the prediction module 13 calls the generation network to predict the sample video description information based on the third text feature, the third video feature and the word embedding feature of each word segment, including:

[0251] Calling the generation network to generate multimodal features of the sample video data based on the third text feature, the third video feature, and the word embedding feature of each word segment;

[0252] Predicting sample video description information of sample video data based on multimodal features.

[0253] Optionally, the generation network includes a video encoder and a text decoder, and the multimodal features are generated based on the video encoder;

[0254] The prediction module 13 predicts the sample video description information of the sample video data based on the multimodal features, including:

[0255] Call the text decoder to predict the sample video description information based on multimodal features.

[0256] Optionally, the prediction module 13 calls the text decoder to predict the sample video description information based on the multimodal features, including:

[0257] Calling the text decoder to predict the nth description character of the sample video data based on the multimodal features and the first n-1 description characters of the sample video data predicted by the text decoder; n is a positive integer;

[0258] If the nth description character does not carry a prediction end mark, calling a text decoder to predict the n+1th description character of the sample video data based on the multimodal features, the first n-1 description characters, and the nth description character;

[0259] If the n+1th description character carries a prediction end flag, the sample video description information is determined according to the first n-1 description characters, the nth description character and the n+1th description character.

[0260] Optionally, the training module 14 corrects the network parameters of the generated network according to the learning deviation and the prediction deviation to obtain a trained generated network, including:

[0261] Obtaining a first bias weight for a learning bias and a second bias weight for a prediction bias;

[0262] Based on the first bias weight and the second bias weight, a weighted sum of the learning bias and the prediction bias is performed to obtain a comprehensive training bias of the generating network for the sample pair and the sample association pair;

[0263] The network parameters of the generated network are corrected based on the comprehensive training deviation to obtain the trained generated network.

[0264] Optionally, the device 1 is also used for:

[0265] Acquire target video data and target text data extracted from the target video data;

[0266] Constructing target association pairs according to each subtext of the target text data and the video segments to which each subtext belongs in the target video data;

[0267] The trained generative network is called to perform prediction processing on the target association pairs to obtain the target video description information of the target video data.

[0268] Optionally, the device 1 is also used for:

[0269] Get the video retrieval text sent by the video client;

[0270] If the text similarity between the video retrieval text and the target video description information is greater than or equal to the similarity threshold, the target video data is returned to the video client, so that the video client outputs the target video data.

[0271] According to one embodiment of the present application, Figure 3 The steps involved in the video data processing method shown can be represented by Fig.12 The various modules in the video data processing device 1 shown in the figure are executed. For example, Figure 3 The step S101 shown in FIG. 1 can be performed by Fig.12 The acquisition module 11 in is executed, Figure 3 The step S102 shown in FIG. 1 can be performed by Fig.12 The learning module 12 is used to execute; Figure 3 The step S103 shown in FIG. 1 can be performed by Fig.12 The prediction module 13 in the Figure 3 The step S104 shown in FIG. 1 can be performed by Fig.12 The training module 14 is used to execute.

[0272] The present application can first obtain sample pairs and sample association pairs; a sample pair includes a first video clip and a first text, and the sample pair is divided into a positive sample pair and a negative sample pair according to the content difference between the first video clip and the first text; a sample association pair includes a second text and a second video clip to which the second text belongs in the sample video data; the sample video data has a video description label; then, the generation network can be called to perform feature comparison learning on the sample pair to obtain a learning deviation; the learning deviation is used to reduce the feature difference between the first video clip and the first text in the positive sample pair, and to increase the feature difference between the first video clip and the first text in the negative sample pair; further, the generation network can be called to perform prediction processing on the sample association pair to obtain sample video description information of the sample video data, and obtain the prediction deviation of the generation network for the sample video description information based on the video description label; finally, the network parameters of the generation network can be corrected according to the learning deviation and the prediction deviation to obtain a trained generation network; the trained generation network can be used to generate video description information for the video data. It can be seen that the device proposed in the present application can combine the video data and the relevant text data of the video data (such as the second text) to train the generation network together. At the same time, it will also conduct comparative learning on the sample pairs to assist the generation network in learning the deviation between the video data and the relevant text data of the video data. This doubly improves the accuracy of training the generation network, and then the trained generation network can also accurately generate video description information of the video data.

[0273] According to one embodiment of the present application, Fig.12 The various modules in the video data processing device 1 shown can be separately or all combined into one or several units to constitute, or one (some) of the units can be further divided into multiple functionally smaller sub-units, which can achieve the same operation without affecting the realization of the technical effects of the embodiments of the present application. The above modules are divided based on logical functions. In practical applications, the functions of one module can also be implemented by multiple units, or the functions of multiple modules can be implemented by one unit. In other embodiments of the present application, the video data processing device 1 may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented by the collaboration of multiple units.

[0274] According to one embodiment of the present application, the program can be executed by running on a general-purpose computer device such as a computer including a central processing unit (CPU), a random access memory medium (RAM), a read-only memory medium (ROM), and other processing elements and storage elements. Figure 3 A computer program (including program code) for each step involved in the corresponding method shown in Fig.12The video data processing device 1 shown in the figure is used to implement the video data processing method of the embodiment of the present application. The above computer program can be recorded on a computer-readable recording medium, for example, and loaded into the above computing device through the computer-readable recording medium and run therein.

[0275] See also Fig.13 , Fig.13 Schematic diagram of the structure of a computer device provided by this application. Fig.13 As shown, the computer device 1000 may include: a processor 1001, a network interface 1004 and a memory 1005. In addition, the computer device 1000 may also include: a user interface 1003, and at least one communication bus 1002. The communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), a keyboard (Keyboard), and the user interface 1003 may optionally include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory, or a non-volatile memory (non-volatile memory), such as at least one disk storage. The memory 1005 may optionally be at least one storage device located away from the aforementioned processor 1001. As Fig.13 As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module, and a device control application program.

[0276] exist Fig.13 In the computer device 1000 shown, the network interface 1004 can provide a network communication function; the user interface 1003 is mainly used to provide an input interface for the user; and the processor 1001 can be used to call the device control application stored in the memory 1005 to achieve:

[0277] Obtain sample pairs and sample association pairs; a sample pair includes a first video clip and a first text, and the sample pair is divided into a positive sample pair and a negative sample pair according to the content difference between the first video clip and the first text; a sample association pair includes a second text and a second video clip to which the second text belongs in the sample video data; the sample video data has a video description tag;

[0278] Calling the generative network to perform feature contrast learning on the sample pair to obtain a learning bias; the learning bias is used to reduce the feature difference between the first video clip and the first text in the positive sample pair, and is used to increase the feature difference between the first video clip and the first text in the negative sample pair;

[0279] Calling the generative network to perform prediction processing on the sample association pairs to obtain sample video description information of the sample video data, and obtaining the prediction deviation of the generative network for the sample video description information based on the video description label;

[0280] The network parameters of the generated network are corrected according to the learning deviation and the prediction deviation to obtain a trained generated network; the trained generated network is used to generate video description information of the video data.

[0281] It should be understood that the computer device 1000 described in the embodiment of the present application can execute the above Figure 3 The description of the video data processing method in the corresponding embodiment can also be performed as described above. Fig.12 The description of the video data processing device 1 in the corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of the same method will not be repeated here either.

[0282] In addition, it should be pointed out that: the present application also provides a computer-readable storage medium, and the computer-readable storage medium stores a computer program executed by the video data processing device 1 mentioned above, and the computer program includes program instructions. When the processor executes the program instructions, it can execute the above-mentioned Figure 3 The description of the video data processing method in the corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of the same method will not be repeated. For technical details not disclosed in the computer storage medium embodiment involved in this application, please refer to the description of the method embodiment of this application.

[0283] As an example, the above program instructions may be deployed on a computer device for execution, or deployed on multiple computer devices located at one location for execution, or executed on multiple computer devices distributed at multiple locations and interconnected by a communication network. Multiple computer devices distributed at multiple locations and interconnected by a communication network may constitute a blockchain network.

[0284] The computer-readable storage medium may be the video data processing device provided in any of the aforementioned embodiments or the internal storage unit of the computer device, such as a hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the computer device. Further, the computer-readable storage medium may also include both the internal storage unit of the computer device and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by the computer device. The computer-readable storage medium may also be used to temporarily store data that has been output or is to be output.

[0285] The present application provides a computer program product or a computer program, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the above Figure 3 The description of the above-mentioned video data processing method in the corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of adopting the same method will not be repeated. For technical details not disclosed in the computer-readable storage medium embodiment involved in this application, please refer to the description of the method embodiment of this application.

[0286] The terms "first", "second", etc. in the description, claims, and drawings of the embodiments of the present application are used to distinguish different objects, rather than to describe a specific order. In addition, the term "comprising" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, device, product, or equipment that includes a series of steps or units is not limited to the listed steps or modules, but optionally includes steps or modules that are not listed, or optionally includes other step units inherent to these processes, methods, devices, products, or equipment.

[0287] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0288] The method and related apparatus provided by the embodiment of the present application are described with reference to the method flow chart and / or structural diagram provided by the embodiment of the present application. Specifically, each process and / or box in the method flow chart and / or structural diagram, as well as the combination of the processes and / or boxes in the flow chart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the process in the process. Figure 1 A process or multiple processes and / or structures Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 A process or multiple processes and / or structures Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A flow or multiple flows and / or structures illustrate the steps of the functions specified in one block or multiple blocks.

[0289] The above disclosure is only the preferred embodiment of the present application, which certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. A video data processing method, characterized in that: The method comprises: Obtain sample pairs and sample association pairs; a sample pair includes a first video clip and a first text, and the sample pair is divided into a positive sample pair and a negative sample pair according to the content difference between the first video clip and the first text; a sample association pair includes a second text and a second video clip to which the second text belongs in sample video data; the sample video data has a video description tag; Calling the generative network to perform feature contrast learning on the sample pair to obtain a learning bias; the learning bias is used to reduce the feature difference between the first video clip and the first text in the positive sample pair, and is used to increase the feature difference between the first video clip and the first text in the negative sample pair; Calling the generation network to perform prediction processing on the sample association pair to obtain sample video description information of the sample video data, and obtaining a prediction deviation of the generation network for the sample video description information based on the video description label; According to the learning deviation and the prediction deviation, the network parameters of the generating network are corrected to obtain a trained generating network; the trained generating network is used to generate video description information of the video data; Among them, the calling of the generation network to perform feature contrast learning on the sample pair to obtain a learning bias includes: calling the generation network to generate a first video feature of the first video clip in the positive sample pair and a first text feature of the first text in the positive sample pair, calling the generation network to generate a second video feature of the first video clip in the negative sample pair and a second text feature of the first text in the negative sample pair; based on the first video feature, the first text feature, the second video feature and the second text feature, obtaining the learning bias of the generation network for the sample pair.

2. The method according to claim 1, characterized in that The video content of the first video clip included in the positive sample pair is the same as the content of the first text description included in the positive sample pair; the video content of the first video clip included in the negative sample pair is different from the content of the first text description included in the negative sample pair.

3. The method according to claim 1, characterized in that The method further comprises: Acquire sample audio data of the sample video data, and perform text conversion processing on the sample audio data to obtain sample text data; The sample text data is subjected to text segmentation processing to obtain at least one second text contained in the sample text data.

4. The method according to claim 3, characterized in that Any one of the at least one second text is represented as a target text; the method further includes: Obtain an audio segment corresponding to the target text in the sample audio data; Obtaining the start playing time and the end playing time of the audio clip in the sample video data; Cutting the sample video data to obtain a second video segment to which the target text belongs according to the start playback time and the end playback time; The sample association pair is constructed according to the target text and the second video segment to which the target text belongs.

5. The method according to claim 1, characterized in that The calling of the generation network to perform prediction processing on the sample association pairs to obtain sample video description information of the sample video data includes: Calling the generation network to generate a third text feature of the second text in the sample association pair and a third video feature of the second video clip in the sample association pair; Obtain at least one word segment included in the second text in the sample association pair, and obtain a word embedding feature of each word segment; The generation network is called to predict the sample video description information based on the third text feature, the third video feature and the word embedding feature of each word segment.

6. The method according to claim 5, characterized in that The calling of the generation network to generate the third text feature of the sub-text in the sample association pair and the third video feature of the associated video segment of the sub-text in the sample association pair includes: Acquire a text embedding feature of a second text in the sample association pair and a video embedding feature of a second video clip in the sample association pair; The generation network is called to generate the third text feature based on the text embedding feature, and the generation network is called to generate the third video feature based on the video embedding feature.

7. The method according to claim 5, characterized in that The calling the generation network to predict the sample video description information based on the third text feature, the third video feature, and the word embedding feature of each word segmentation includes: Calling the generation network to generate multimodal features of the sample video data based on the third text feature, the third video feature and the word embedding feature of each word segment; The sample video description information of the sample video data is predicted based on the multimodal features.

8. The method according to claim 7, characterized in that The generation network includes a video encoder and a text decoder, and the multimodal features are generated based on the video encoder; The predicting the sample video description information of the sample video data based on the multimodal features includes: The text decoder is called to predict the sample video description information based on the multimodal features.

9. The method according to claim 8, characterized in that The calling the text decoder to obtain the sample video description information based on the multimodal feature prediction includes: Calling the text decoder to predict the nth description character of the sample video data based on the multimodal features and the first n-1 description characters of the sample video data predicted by the text decoder; n is a positive integer; If the nth description character does not carry a prediction end mark, calling the text decoder to predict the n+1th description character of the sample video data based on the multimodal features, the first n-1 description characters, and the nth description character; If the n+1th description character carries the prediction end identifier, the sample video description information is determined according to the first n-1 description characters, the nth description character and the n+1th description character.

10. The method according to claim 1, characterized in that The step of correcting the network parameters of the generated network according to the learning deviation and the prediction deviation to obtain a trained generated network includes: Obtaining a first bias weight for the learning bias and a second bias weight for the prediction bias; Performing a weighted summation of the learning bias and the prediction bias based on the first bias weight and the second bias weight to obtain a comprehensive training bias of the generation network for the sample pair and the sample association pair; The network parameters of the generated network are corrected based on the comprehensive training deviation to obtain the trained generated network.

11. The method according to claim 1, characterized in that The method further comprises: Acquire target video data and target text data extracted from the target video data; Constructing a target association pair according to each subtext of the target text data and the video segments to which each subtext belongs in the target video data; The trained generation network is called to perform prediction processing on the target association pair to obtain target video description information of the target video data.

12. The method according to claim 11, characterized in that The method further comprises: Get the video retrieval text sent by the video client; If the text similarity between the video retrieval text and the target video description information is greater than or equal to a similarity threshold, the target video data is returned to the video client so that the video client outputs the target video data.

13. A video data processing device, characterized in that: The device comprises: An acquisition module is used to acquire sample pairs and sample association pairs; a sample pair includes a first video clip and a first text, and the sample pair is divided into a positive sample pair and a negative sample pair according to the content difference between the first video clip and the first text; a sample association pair includes a second text and a second video clip to which the second text belongs in sample video data; the sample video data has a video description tag; A learning module, used for calling a generation network to perform feature contrast learning on the sample pair to obtain a learning deviation; the learning deviation is used to reduce the feature difference between the first video clip and the first text in the positive sample pair, and to increase the feature difference between the first video clip and the first text in the negative sample pair; A prediction module, configured to call the generation network to perform prediction processing on the sample association pair to obtain sample video description information of the sample video data, and obtain a prediction deviation of the generation network for the sample video description information based on the video description tag; A training module, used for correcting the network parameters of the generating network according to the learning deviation and the prediction deviation to obtain a trained generating network; the trained generating network is used to generate video description information of the video data; Among them, the learning module calls the generation network to perform feature comparison learning on the sample pair to obtain a learning bias, including: calling the generation network to generate a first video feature of the first video clip in the positive sample pair and a first text feature of the first text in the positive sample pair, calling the generation network to generate a second video feature of the first video clip in the negative sample pair and a second text feature of the first text in the negative sample pair; based on the first video feature, the first text feature, the second video feature and the second text feature, obtaining the learning bias of the generation network for the sample pair.

14. A computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 12.

15. A computer device, characterized in that: The invention comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the method according to any one of claims 1 to 12.

16. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor and executing the method according to any one of claims 1 to 12.

Images