A video summary generation method and related device

Abstract

The present application discloses a video summary generation method and related devices, which relate to the field of artificial intelligence technology. After obtaining a multimodal video representation of each video clip in a target video, the process of extracting an optimal video clip combination from multiple video clips in the target video is modeled as a Markov decision process, taking into account the local importance and global coherence of the video clips, and using a reinforcement learning algorithm to solve the optimal strategy of the Markov decision process, extracting a more concise and coherent optimal video clip combination composed of important video clips, thereby generating an accurate video summary based on the optimal video clip combination, thereby improving the quality of the video summary.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and more specifically, to a video summary generation method and related devices. Background Art

[0002] Faced with massive amounts of videos, users need to quickly understand the main content of the video to decide whether to watch it, and video summarization technology has emerged. Video summarization technology extracts key clips from the video and generates a refined video preview to improve video browsing efficiency.

[0003] However, the video summaries generated by current video summary generation methods have low accuracy and there is much room for improvement. Summary of the invention

[0004] In view of the above problems, this application provides a video summary generation method and related devices to achieve the purpose of improving the accuracy of video summaries. The specific scheme is as follows:

[0005] The first aspect of the present application provides a method for generating a video summary, comprising:

[0006] Obtain a multimodal video representation of each video clip in the target video;

[0007] Constructing a Markov decision process according to the multimodal video representation of each of the video clips, and solving the optimal video clip combination that maximizes the return of the Markov decision process using a reinforcement learning algorithm, wherein the state space in the Markov decision process includes multiple video clip combinations, and the action space includes selecting or not selecting the video clip;

[0008] A video summary of the target video is generated according to the optimal video segment combination.

[0009] In a possible implementation, obtaining a multimodal video representation of each video segment in the target video includes:

[0010] Obtaining multimodal features of each of the video clips;

[0011] For each of the video clips, using a multi-head attention mechanism to calculate the attention weights between different modal features;

[0012] The multimodal features are fused according to the attention weights between the features of different modalities to obtain the fused multimodal features;

[0013] Splicing the fused multimodal features to obtain splicing features;

[0014] The concatenated features are mapped to a common semantic space to obtain a multimodal video representation of the video clip.

[0015] In a possible implementation, constructing a Markov decision process according to the multimodal video representation of each of the video clips, and using a reinforcement learning algorithm to solve the optimal video clip combination that maximizes the reward of the Markov decision process includes:

[0016] Inputting the multimodal video representation of each of the video clips into a pointer network, and predicting the probability of each of the video clips being the starting position of a summary and the probability of each of the video clips being the ending position of a summary;

[0017] Determine the video segment with the highest probability of being the starting position of the summary as the starting video segment, and determine the video segment with the highest probability of being the ending position of the summary as the ending video segment, to obtain the plurality of video segment combinations;

[0018] A Markov decision process is constructed according to the plurality of video clip combinations, and a reinforcement learning algorithm is used to solve an optimal video clip combination that maximizes the reward of the Markov decision process.

[0019] In a possible implementation, inputting the multimodal video representation of each video segment into a pointer network to predict the probability of each video segment being a summary start position and a summary end position includes:

[0020] Inputting the multimodal video representation of each of the video clips into a bidirectional long short-term memory network in the pointer network to obtain a contextual representation of each of the video clips;

[0021] Inputting the context representation of each video clip into the first multi-layer perceptron in the pointer network to obtain the probability of each video clip being the starting position of the summary;

[0022] The context representation of each video segment is input into the second multi-layer perceptron in the pointer network to obtain the probability of each video segment being the end position of the summary.

[0023] In a possible implementation, generating a video summary of the target video according to the optimal video segment combination includes:

[0024] Constructing a graph-based integer programming model according to the optimal video segment combination, wherein a global objective function of the graph-based integer programming model represents the sum of importance of the selected video segments and the sum of similarities between the selected video segments;

[0025] Solving the global objective function by taking maximization of the global objective function as an optimization goal, and extracting a plurality of target video segments from the optimal video segment combination;

[0026] The multiple target video clips are sorted in chronological order to obtain a video summary of the target video.

[0027] In a possible implementation, after generating the video summary of the target video according to the optimal video segment combination, the method further includes:

[0028] extracting visual features and audio features of the video summary;

[0029] Performing feature fusion on the visual features and audio features of the video summary to obtain fusion features of the video summary;

[0030] The fused features of the video summary are mapped into a natural language description using a pre-trained summary text generation model, wherein the summary text generation model includes an encoder and a decoder, and is trained using a video description dataset.

[0031] A second aspect of the present application provides a video summary generation device, including:

[0032] A multimodal video representation acquisition unit, used to acquire a multimodal video representation of each video segment in a target video;

[0033] A summary segment extraction unit is used to construct a Markov decision process according to the multimodal video representation of each video segment, and use a reinforcement learning algorithm to solve the optimal video segment combination that maximizes the reward of the Markov decision process, wherein the state space in the Markov decision process includes multiple video segment combinations, and the action space includes selecting or not selecting the video segment;

[0034] A summary segment organization unit is used to generate a video summary of the target video according to the optimal video segment combination.

[0035] A third aspect of the present application provides a computer program product, including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements the video summary generation method according to the first aspect or any implementation manner of the first aspect.

[0036] A fourth aspect of the present application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:

[0037] The memory is used to store computer programs;

[0038] The processor is configured to execute the computer program so that the electronic device can implement the video summary generation method of the first aspect or any implementation manner of the first aspect.

[0039] According to a fifth aspect of the present application, a computer storage medium is provided, wherein the storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the video summary generation method according to the first aspect or any implementation of the first aspect.

[0040] By means of the above technical scheme, the present application provides a video summary generation method and related devices. After obtaining the multimodal video representation of each video clip in the target video, the process of extracting the optimal video clip combination from multiple video clips in the target video is modeled as a Markov decision process, taking into account the local importance and global coherence of the video clips, and using a reinforcement learning algorithm to solve the optimal strategy of the Markov decision process, extracting a simpler and more coherent optimal video clip combination composed of important video clips, thereby generating an accurate video summary based on the optimal video clip combination, thereby improving the quality of the video summary. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0042] Figure 1 A schematic diagram of a system architecture provided for this application;

[0043] Figure 2 A schematic diagram of the structure of a terminal 100 provided in this application;

[0044] Figure 3 A schematic diagram of the structure of a server 200 provided in this application;

[0045] Figure 4 A schematic diagram of a process for generating a video summary provided in an embodiment of the present application;

[0046] Figure 5 A schematic diagram of the structure of a video summary generation device provided in an embodiment of the present application;

[0047] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0048] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0049] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0050] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0051] See also Figure 1 , Figure 1 A schematic diagram of a system architecture is shown. The system may include a terminal 100 and a server 200. The server 200 may include one or more servers ( Figure 1 In the example, a server is included, and the server 200 can provide the method provided in the embodiment of the present application for one or more terminals.

[0052] Among them, an application can be installed on the terminal 100, and the above application can provide an interface. The terminal 100 can receive relevant parameters entered by the user on the interface, such as the target video identifier, and send the above parameters to the server 200. The server 200 can obtain processing results based on the received parameters and return the processing results to the terminal 100, such as receiving the target video identifier, obtaining the video summary corresponding to the target video identifier, and returning the video summary to the terminal 100.

[0053] It should be understood that in some optional implementations, the terminal 100 can also complete the action of obtaining the processing result based on the received parameters by itself without the cooperation of the server, and the embodiments of the present application are not limited to this.

[0054] Next describe Figure 1 The product form of the mid-terminal 100;

[0055] The terminal 100 in the embodiment of the present application can be a mobile phone, a tablet computer, a wearable device, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), etc., and the embodiment of the present application does not impose any limitation on this.

[0056] Figure 2 An optional hardware structure diagram of the terminal 100 is shown.

[0057] refer to Figure 2 As shown, the terminal 100 may include a radio frequency unit 110, a memory 120, an input unit 130, a display unit 140, a camera 150 (optional), an audio circuit 160 (optional), a speaker 161 (optional), a microphone 162 (optional), an earphone jack 163 (optional), a processor 170, an external interface 180, a power supply 190 and other components. Those skilled in the art will appreciate that Figure 2 This is merely an example of a terminal and does not constitute a limitation on the terminal. The terminal may include more or fewer components than shown in the figure, or a combination of certain components, or different components.

[0058] The input unit 130 can be used to receive input digital or character information, and generate key signal input related to the user settings and function control of the terminal 100. Specifically, the input unit 130 may include a touch screen 131 (optional) and / or other input devices 132. The touch screen 131 can collect the user's touch operations on or near it (such as the user's operation on or near the touch screen using any suitable object such as fingers, joints, stylus, etc.), and drive the corresponding connection device according to a pre-set program. The touch screen can detect the user's touch action on the touch screen, convert the touch action into a touch signal and send it to the processor 170, and can receive and execute the command sent by the processor 170; the touch signal at least includes the touch point coordinate information. The touch screen 131 can provide an input interface and an output interface between the terminal 100 and the user. In addition, the touch screen can be implemented using multiple types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch screen 131, the input unit 130 can also include other input devices. Specifically, other input devices 132 may include, but are not limited to, one or more of a physical keyboard, function keys (such as a volume control key, a switch key, etc.), a trackball, a mouse, a joystick, and the like.

[0059] Among them, the input device 132 can receive input data and the like.

[0060] The display unit 140 may be used to display information input by the user or provided to the user, various menus of the terminal 100, interactive interfaces, file display, and / or playback of any multimedia file. In the embodiment of the present application, the display unit 140 may be used to display an interface for selecting a target video, a processing result of a video summary of a target video, etc.

[0061] The memory 120 can be used to store instructions and data. The memory 120 can mainly include an instruction storage area and a data storage area. The data storage area can store various data, such as multimedia files, texts, etc.; the instruction storage area can store software units such as operating systems, applications, instructions required for at least one function, or their subsets and extensions. It can also include a non-volatile random access memory; provide the processor 170 with hardware, software and data resources including management of computing and processing equipment, and support control software and applications. It is also used for the storage of multimedia files, and the storage of running programs and applications.

[0062] The processor 170 is the control center of the terminal 100. It uses various interfaces and lines to connect various parts of the entire terminal 100. By running or executing instructions stored in the memory 120 and calling data stored in the memory 120, it executes various functions of the terminal 100 and processes data, thereby controlling the terminal device as a whole. Optionally, the processor 170 may include one or more processing units; preferably, the processor 170 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface and application program, and the modem processor mainly processes wireless communication. It is understandable that the above-mentioned modem processor may not be integrated into the processor 170. In some embodiments, the processor and the memory may be implemented on a single chip, and in some embodiments, they may also be implemented separately on separate chips. The processor 170 may also be used to generate corresponding operation control signals, send them to corresponding components of the computing and processing device, read and process data in the software, especially read and process data and programs in the memory 120, so that each functional module therein performs corresponding functions, thereby controlling the corresponding components to act according to the requirements of the instructions.

[0063] The memory 120 may be used to store software codes related to the video summary generation method, the processor 170 may execute the steps of the video summary generation method, and may also schedule other units (such as the above-mentioned input unit 130 and display unit 140) to implement corresponding functions.

[0064] The radio frequency unit 110 (optional) can be used for receiving and sending information or receiving and sending signals during a call, for example, after receiving the downlink information of the base station, it is sent to the processor 170 for processing; in addition, the designed uplink data is sent to the base station. Generally, the RF circuit includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (Low Noise Amplifier, LNA), a duplexer, etc. In addition, the radio frequency unit 110 can also communicate with network devices and other devices through wireless communication. The wireless communication can use any communication standard or protocol, including but not limited to Global System of Mobile communication (Global System of Mobile communication, GSM), General Packet Radio Service (General Packet Radio Service, GPRS), Code Division Multiple Access (Code Division Multiple Access, CDMA), Wideband Code Division Multiple Access (Wideband Code Division Multiple Access, WCDMA), Long Term Evolution (Long Term Evolution, LTE), email, Short Messaging Service (SMS), etc.

[0065] In this embodiment of the present application, the RF unit 110 can send data to the server 200 and receive processing results sent by the server 200.

[0066] It should be understood that the radio frequency unit 110 is optional and can be replaced by other communication interfaces, such as a network port.

[0067] The terminal 100 also includes a power supply 190 (such as a battery) for supplying power to various components. Preferably, the power supply can be logically connected to the processor 170 through a power management system, so that the power management system can manage functions such as charging, discharging, and power consumption.

[0068] The terminal 100 further includes an external interface 180 , which may be a standard Micro USB interface or a multi-pin connector, and may be used to connect the terminal 100 to communicate with other devices, or to connect a charger to charge the terminal 100 .

[0069] Although not shown, the terminal 100 may also include a flashlight, a wireless fidelity (WiFi) module, a Bluetooth module, sensors with different functions, etc., which are not described in detail here. Some or all of the methods described below may be applied in the following embodiments. Figure 2 In the terminal 100 shown.

[0070] Next describe Figure 1 The product form of the server 200;

[0071] Figure 3 A structural diagram of a server 200 is provided, such as Figure 3 As shown, the server 200 includes a bus 201, a processor 202, a communication interface 203, and a memory 204. The processor 202, the memory 204, and the communication interface 203 communicate with each other via the bus 201.

[0072] The bus 201 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0073] The processor 202 may be any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0074] The memory 204 may include a volatile memory, such as a random access memory (RAM). The memory 204 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard drive (HDD), or a solid state drive (SSD).

[0075] The memory 204 may be used to store software codes related to the video summary generation method, and the processor 202 may execute the steps of the video summary generation method of the chip, and may also schedule other units to implement corresponding functions.

[0076] It should be understood that the above-mentioned terminal 100 and server 200 can be centralized or distributed devices, and the processors in the above-mentioned terminal 100 and server 200 (such as processor 170 and processor 202) can be hardware circuits (such as application specific integrated circuit (ASIC), field-programmable gate array (FPGA), general-purpose processor, digital signal processor (DSP), microprocessor or microcontroller, etc.), or a combination of these hardware circuits. For example, the processor can be a hardware system with an instruction execution function, such as a CPU, DSP, etc., or a hardware system without an instruction execution function, such as an ASIC, FPGA, etc., or a combination of the above-mentioned hardware systems without an instruction execution function and hardware systems with an instruction execution function.

[0077] Video summarization technology extracts key clips from videos and generates refined video previews to improve video browsing efficiency. Traditional video summarization methods lack the ability to represent video features and are unable to accurately describe video semantics, resulting in low accuracy of video summaries based on video feature extraction, lack of global optimization of summary segment extraction, redundant or incoherent segments, and ultimately poor quality of the generated video summary, which affects user experience.

[0078] In order to solve the above problems, the embodiment of the present application provides a method for generating a video summary. The method for generating a video summary in the embodiment of the present application is described in detail below with reference to the accompanying drawings.

[0079] Reference Figure 4 , Figure 4 A schematic diagram of a video summary generation method provided in an embodiment of the present application is shown in FIG. Figure 4 As shown, a video summary generation method provided in an embodiment of the present application may include steps 401 to 404, and these steps are described in detail below.

[0080] 401: Obtain a multimodal video representation of each video clip in the target video;

[0081] Taking a 5-minute video as an example, the video can be divided into 30 10-second video clips.

[0082] First, the multimodal features of each video clip are obtained. For example, the multimodal features of the video clip include: image features, audio features and text features. Then, the multimodal features of each video clip are fused using a multi-head attention mechanism to obtain a multimodal video representation of each video clip. The multimodal video representation represents the video clip by fusing the multimodal features of the video clip.

[0083] In this embodiment, a multi-head attention mechanism is used to learn the semantic associations and importance weights between different modal features from different angles through multiple attention heads, obtain the attention weights between image features and text features, image features and audio features, and text features and audio features, and weightedly fuse different modal features to obtain a more comprehensive and accurate multimodal video representation.

[0084] 402: constructing a Markov decision process based on the multimodal video representation of each video clip, and using a reinforcement learning algorithm to solve the optimal video clip combination that maximizes the reward of the Markov decision process;

[0085] The purpose of extracting the optimal video clip combination is to determine the initial scope of the video summary. Taking the target video length of 5 minutes as an example, the optimal video clip combination can be composed of a video length of 1 minute, including 6 video clips of 10 seconds in length.

[0086] The problem of extracting the optimal combination of video clips from multiple video clips of the target video is modeled as a Markov decision process and solved using a reinforcement learning algorithm, taking into account global importance and global coherence to extract a simpler and more coherent optimal combination of video clips.

[0087] 403: Generate a video summary of the target video according to the optimal video segment combination.

[0088] Exemplarily, the video segments in the optimal video segment combination are sorted in chronological order to obtain a video summary of the target video.

[0089] For example, the video clips in the optimal video clip combination are rearranged and organized, such as using a graph-based integer programming model to rearrange the video clips to obtain a semantically similar and naturally connected summary clip sequence. This method can improve the readability and fluency of the generated summary, and is more in line with human viewing habits.

[0090] A video summary generation method disclosed in this embodiment, after obtaining a multimodal video representation of each video clip in a target video, models the process of extracting an optimal video clip combination from multiple video clips in the target video as a Markov decision process, taking into account the local importance and global coherence of the video clips, and uses a reinforcement learning algorithm to solve the optimal strategy of the Markov decision process, extracting a more concise and coherent optimal video clip combination composed of important video clips, thereby generating an accurate video summary based on the optimal video clip combination, thereby improving the quality of the video summary.

[0091] This embodiment introduces advanced artificial intelligence technologies such as deep learning, reinforcement learning, and integer programming in key technical links such as multimodal feature extraction, multimodal feature fusion, summary segment extraction, and summary segment organization, which significantly improves the accuracy, coherence, and intelligence level of video summary generation, and makes up for the shortcomings of existing methods. These improvements help to better understand the semantic content of the video and automatically generate high-quality video summaries, thereby improving video processing efficiency and user experience.

[0092] In a possible implementation, in the above embodiment, 401: obtaining a multimodal video representation of each video segment in the target video specifically includes the following steps 4011-4015:

[0093] 4011: Obtain multimodal features of each video clip;

[0094] Take the multimodal features of video clips including image features, audio features and text features as an example:

[0095] A video clip includes multiple image frames, and multiple image frames form an image frame sequence. One possible implementation method for extracting image features is to use a pre-trained convolutional neural network (CNN) to extract image features from the image frame sequence corresponding to the video clip. Specifically, use a CNN model such as ResNet and Inception that has been pre-trained with a large-scale data set, input each frame of the video clip into the CNN model, extract its high-level semantic features, and the output of the last fully connected layer of the CNN model is the feature vector representation of the image, that is, the image feature. For ease of description, the image feature corresponding to the i-th frame of the video clip is , the image feature sequence corresponding to the video clip is [ , ,…, ], where n represents the total number of frames in the video clip.

[0096] Pre-training CNN models usually takes the following steps: (1) Prepare large-scale image datasets, such as ImageNet and COCO, which contain millions of images and corresponding category labels. (2) Select a CNN architecture, such as ResNet or Inception, and design hyperparameters such as the number of network layers, convolution kernel size, and pooling operations. (3) Randomly initialize the weight parameters of the CNN model. (4) Divide the image dataset into a training set and a validation set, and perform data preprocessing, such as scaling, normalization, and data augmentation. (5) Use the training set data to train the CNN model, calculate the predicted output through forward propagation, use the cross-entropy loss function to compare the predicted output with the true label, and update the model weights through backpropagation and gradient descent algorithms. Iterate continuously until the model converges or reaches the preset number of iterations. (6) During the training process, use the validation set data to evaluate the model performance and adjust the hyperparameters to optimize the model. (7) After training, use the test set data to evaluate the generalization performance of the model. (8) Save the trained CNN model weights for subsequent transfer learning tasks. Pre-trained CNN models can learn general features of images, such as edges, textures, shapes, etc. These features can be effectively transferred to new tasks to improve the performance and generalization ability of the model.

[0097] A possible implementation method for extracting audio features is: first, an acoustic feature extraction method is used to obtain a low-level feature representation of the audio signal corresponding to the video clip, and then the audio signal is divided into frames of fixed length. For the low-level feature representation of each frame, acoustic features such as Mel frequency cepstral coefficients (MFCC) and Mel spectrogram are extracted to obtain an audio feature sequence, and then the audio feature sequence is input into a pre-trained long short-term memory network LSTM. LSTM processes the audio feature sequence frame by frame and outputs a hidden state at each time step. The hidden state of the last time step is recorded as , as audio features of video clips.

[0098] The following steps are usually used to pre-train LSTM models: (1) Prepare audio feature sequence data and pre-process the audio, such as framing and extracting Mel Spectral Coefficients (MFCC). (2) Design the architecture of the LSTM network, such as the number of hidden layers and the number of hidden units. (3) Randomly initialize the weight parameters of the LSTM, including the weight matrix and bias vector of the input gate, forget gate, and output gate. (4) Divide the audio feature sequence into a training set and a validation set. (5) Use the training set data to train the LSTM, calculate the hidden state and output through forward propagation, use loss functions such as mean square error (MSE) or cross entropy to compare the predicted output with the true label, and update the LSTM weight parameters through back propagation and gradient descent algorithms (such as BPTT), and iterate until the model converges or reaches the preset number of iterations. (6) During the training process, use the validation set data to evaluate the model performance and adjust the hyperparameters to optimize the model. (7) After training, use the test set data to evaluate the generalization performance of the model. (8) Save the trained LSTM model weights for subsequent feature extraction tasks. LSTM can effectively model the long-term dependencies of audio feature sequences and extract high-level semantic features through gating mechanisms and memory units. The trained LSTM model can be used for audio feature extraction, and the hidden state of the last time step is used as the feature representation of the entire audio.

[0099] One possible way to extract text features is to first perform preprocessing such as word segmentation and part-of-speech tagging on the subtitle text of the video clip, and then use large-scale pre-trained language models such as BERT and GPT to encode each word. Specifically, the word is converted into a word embedding vector, and the word embedding vector is input into the pre-trained language model. The output of the language model is pooled (such as average pooling, maximum pooling, etc.) to obtain a feature vector representation, that is, a text feature, denoted as t. Among them, Word2Vec, GloVe, etc. can be used to convert words into word embedding vectors.

[0100] This embodiment uses a multimodal feature extraction mechanism to perform feature encoding for different modal data such as video image frames, audio, and subtitle text using corresponding pre-trained deep learning models. Among them, image frames use convolutional neural networks (CNN) to extract semantic features, audio uses long short-term memory networks (LSTM) to extract timing information, and text uses pre-trained language models such as BERT to learn semantic representation. Multimodal feature extraction makes full use of the semantic information contained in different modal data, laying a good foundation for subsequent video content understanding.

[0101] 4012: For each video clip, the multi-head attention mechanism is used to calculate the attention weights between different modal features;

[0102] The multi-head attention mechanism calculates the attention weight matrices between different modal features respectively, and learns the relevance and importance of different modal features.

[0103] Taking the attention calculation of image features and audio features as an example, firstly, the image features and audio features are mapped to the same dimension through linear transformation:

[0104] ,

[0105] in:

[0106] Represents the image features of the i-th frame image The linear transformation result is mapped to the same dimension as the audio feature, which can also be expressed as ;

[0107] The linear transformation matrix representing the image features is from Dimensions are mapped to dimension;

[0108] A linear transformation bias term representing image features;

[0109] Representing audio features The linear transformation result is mapped to the same dimension as the image feature;

[0110] The linear transformation matrix representing the audio features is from Dimensions are mapped to dimension;

[0111] A linear transformation bias term that represents audio features.

[0112] The purpose of the above linear transformation is to map different modal features to the same dimension so that the attention weights between them can be calculated later. Mapping all features to the same dimension is called the "query-key-value" dimension of attention. The linear transformation matrix , and the bias term , It is a learnable parameter that is trained and updated through optimization algorithms such as gradient descent.

[0113] Then calculate the attention weights between image features and audio features:

[0114]

[0115] in, Represents the image features of the i-th frame image With audio features The attention weight between

[0116] Represents the image features of the i-th frame image The linear transformation result of ;

[0117] Representing audio features The linear transformation result of ;

[0118] It means to perform dot product on the linearly transformed image features and audio features to obtain their similarity scores. The larger the dot product, the more similar the two features are.

[0119] represents the scaling factor, Used to normalize the dot product results to prevent the gradient vanishing problem caused by too high a dimension;

[0120] exp() represents an exponential function, which maps the similarity score to the non-negative real number domain and amplifies the difference;

[0121] Represents the exponential sum of the similarity scores of all image features and audio features, which is used to normalize the attention weights so that their sum is 1.

[0122] Similarly, the attention weights between image features and text features, and the attention weights between audio features and text features can be calculated.

[0123] The multi-head attention mechanism calculates the attention weights between each pair of features separately to characterize the correlation between them. The multi-head attention mechanism can model various interaction modes between features by calculating multiple sets of attention weights in parallel.

[0124] 4013: Perform feature fusion on multimodal features according to the attention weights between different modal features to obtain fused multimodal features;

[0125] After calculating the attention weights between different modal features, the multimodal features are fused using weighted summation. Taking image features as an example, the fused image features are expressed as:

[0126]

[0127] in, represents the image features of the fused i-th frame image, represents the attention weight between the image features of the i-th frame image and the image features of the j-th frame image, realizing the self-attention mechanism within the image features. represents the image features of the j-th frame image, represents the attention weight between the image features and audio features of the i-th frame image, represents the attention weight between the image features and text features of the i-th frame image. This formula fuses the attention information within the image features and between the image-audio and image-text into the image features of the i-th frame image by weighted summation to obtain the fused feature representation. Among them, the weights of different features are determined by their The attention weight between them determines that the more relevant the feature is, the greater its contribution will be. This fusion method combines the complementary information of different modal features, making It can more comprehensively characterize the semantic content of the i-th frame image while taking into account its association with audio and text.

[0128] Through attention weighting, information from different modalities can be adaptively fused to highlight the audio and text information related to the current image frame.

[0129] The fused audio features are expressed as follows:

[0130]

[0131] in, represents the fused audio features, represents the attention weight between the image features and audio features of the i-th frame image, represents the image features of the i-th frame image, represents the attention weight between the original audio features and the fused audio features, Represents the attention weight between the text feature and the fused audio feature. This formula fuses the image feature and text feature into the audio feature to obtain a fused audio feature representation.

[0132] The fused text features are expressed as follows:

[0133]

[0134] in, represents the fused text features, represents the attention weight between the image features and text features of the i-th frame image, represents the image features of the i-th frame image, represents the attention weight between the audio features and the fused text features, Represents the attention weight between the original text features and the fused text features. This formula fuses the image features and audio features into the text features to obtain a fused text feature representation.

[0135] 4014: splicing the fused multi-modal features to obtain splicing features;

[0136] For example, the splicing feature is [ , ,…, , , ].

[0137] This splicing method sequentially splices the fused image features, audio features, and text features into a higher-dimensional feature vector, retaining the information of each modal feature. The spliced ​​feature vector z contains the global multimodal information of the video.

[0138] 4015: Map the concatenated features into a common semantic space to obtain a multimodal video representation of the video clips.

[0139] Exemplarily, the concatenated features are mapped to a common semantic space through a fully connected layer, and finally a multimodal video representation z of the video clip is obtained:

[0140] z = tanh(W z [ , ,…, , , ]+b z );

[0141] Among them, W z and b z Represent the weight matrix and bias term of the fully connected layer, W z and b z The subscript z in denotes the dimension of the multimodal video representation.

[0142] The fully connected layer is part of the entire multimodal fusion model and is located at the last layer of the model. The function of this fully connected layer is to further integrate the fusion features of different modalities and extract higher-level semantic representations. Through end-to-end training, the fully connected layer can learn the optimal combination of features of different modalities and obtain a compact and discriminative video representation z. This representation can be used for tasks such as video classification, retrieval, and summarization.

[0143] The parameter W of the fully connected layer z and b zLearning and updating are performed through optimization algorithms such as back-propagation and gradient descent, and are trained end-to-end with other parts of the multimodal fusion model.

[0144] This embodiment utilizes a multi-head attention mechanism to learn the semantic associations and importance weights between different modal features from different angles through multiple attention heads, and weightedly fuses different modal features to obtain a more comprehensive and accurate representation of video content. The fusion mechanism can adaptively capture the importance of different modal features and improve the effect of video semantic understanding.

[0145] In a possible implementation, in the above embodiment 402: constructing a Markov decision process according to the multimodal video representation of each video clip, and using a reinforcement learning algorithm to solve the optimal video clip combination that maximizes the reward of the Markov decision process, specifically includes the following steps 4021-4023:

[0146] 4021: Input the multimodal video representation of each video clip into the pointer network to predict the probability of each video clip being the starting position of the summary and the probability of being the ending position of the summary;

[0147] Exemplarily, the pointer network includes a bidirectional long short-term memory network, a first multi-layer perceptron (MLP) and a second multi-layer perceptron (MLP). In the pointer network, a bidirectional long short-term memory network is used to contextually encode the multimodal video representation of the video clip, capture the long-distance dependencies between the video clips, and use a multi-layer perceptron (MLP) to calculate the pointer vector. MLP is a feedforward neural network, which consists of an input layer, a hidden layer and an output layer, and can realize nonlinear mapping. In this embodiment, the first multi-layer perceptron is used to calculate the probability of each video clip as the starting position of the summary, and the second multi-layer perceptron is used to calculate the probability of each video clip as the ending position of the summary.

[0148] Specifically, the multimodal video representation of each video clip is first input into the bidirectional long short-term memory network in the pointer network to obtain the contextual representation of each video clip;

[0149] The bidirectional long short-term memory network consists of two LSTMs, forward and backward, which process the forward and reverse feature sequences respectively and output the hidden state at each time step:

[0150]

[0151] in, and Represent the forward and backward hidden states of the i-th video clip respectively. Concatenate them to get the contextual representation of the i-th clip:

[0152]

[0153] Among them, 2d h Represents the dimension of the hidden layer of the bidirectional long short-term memory network.

[0154] Then, the context representation of each video clip is input into the first multi-layer perceptron in the pointer network to obtain the probability of each video clip being the starting position of the summary; the context representation of each video clip is input into the second multi-layer perceptron in the pointer network to obtain the probability of each video clip being the ending position of the summary.

[0155] Define two pointer vectors and , respectively represent the probability of the video clip being the starting position of the summary and the probability of being the ending position of the summary.

[0156]

[0157]

[0158] Each MLP network contains two fully connected layers. The first layer uses the ReLU activation function to introduce nonlinearity, and the second layer uses the softmax function to output probability distribution. By adjusting the number of MLP layers, the number of neurons, and the activation function, the mapping relationship from context representation to pointer vector can be flexibly modeled.

[0159] In the architecture of the pointer network, the bidirectional long short-term memory network acts as an encoder to encode the multimodal video representation of the video clip into a context representation, and then the MLP acts as a decoder to calculate the pointer vector based on the context representation to achieve the mapping from the context information to the summary clip position.

[0160] As the decoder of the pointer network, the parameters of MLP can be optimized through end-to-end training, and together with other parts of the pointer network, the loss function of the summary segment prediction is minimized. The design of MLP (such as the number of layers, activation function, etc.) can be adjusted according to the requirements of the task to improve the performance of the pointer network. MLP is a key component of the pointer network, responsible for calculating the pointer vector based on the context representation to predict the location of the summary segment. Through end-to-end training, MLP works together with other parts of the pointer network to generate high-quality video summaries.

[0161] The pointer network is a neural network model that needs to be trained. The training process of the pointer network is as follows:

[0162] (1) Prepare training data: Represent each video clip context as a sequence [ , ,…, ] is used as the input of the pointer network, and the start and end positions of the manually annotated summary fragments are used as supervision signals for training.

[0163] (2) Design the architecture of the pointer network: The pointer network consists of an encoder (bidirectional LSTM) and two decoders (MLP). The encoder maps the context representation into a fixed-length vector, and the decoder calculates the pointer vector based on the encoder output.

[0164] (3) Forward propagation: Input the context representation sequence into the pointer network and calculate the predicted starting position pointer vector and ending position pointer vector.

[0165] (4) Calculate the loss function: Use the cross entropy loss function to measure the difference between the predicted pointer vector and the true position.

[0166] (5) Back propagation: Based on the gradient of the loss function, the back propagation algorithm is used to calculate the gradient of each parameter of the pointer network, and the optimization algorithm (such as Adam) is used to update the parameters.

[0167] (6) Repeat steps (3)-(5) until the model converges or the preset number of iterations is reached.

[0168] Through the above training process, the pointer network can learn the mapping relationship from the context representation of the video clip to the position of the summary clip, and realize the prediction of the summary clip.

[0169] 4022: Determine the video segment with the highest probability of being the starting position of the summary as the starting video segment, and determine the video segment with the highest probability of being the ending position of the summary as the ending video segment, to obtain a combination of multiple video segments;

[0170] The number of video segments contained in the video segment combination is positively correlated with the length of the target video. The corresponding relationship between the target video length and the length of the video composed of the video segment combination can be pre-configured. Taking the target video length of 5 minutes as an example, the length of the video composed of the video segment combination can be 1 minute, including 6 video segments of 10 seconds in length.

[0171] By determining the video segment with the highest probability of being the starting position of the summary as the starting video segment and the video segment with the highest probability of being the ending position of the summary as the ending video segment, the number of video segment combinations is reduced and the amount of subsequent data processing is reduced.

[0172] 4023: Construct a Markov decision process based on multiple video clip combinations, and use a reinforcement learning algorithm to solve the optimal video clip combination that maximizes the return of the Markov decision process.

[0173] The process of extracting the optimal video clip combination from multiple video clips of the target video is modeled as a Markov decision process. Markov decision process (MDP) is a mathematical framework for modeling decision problems, which provides a theoretical basis for reinforcement learning algorithms. MDP defines the basic elements of reinforcement learning algorithms, including state space, action space, transition probability and reward function, which constitute the environment model required by reinforcement learning algorithms. In this embodiment, the model for solving the optimal video clip combination that maximizes the return of the Markov decision process using the reinforcement learning algorithm is defined as a summary extraction model. The state space includes multiple video clip combinations, each video clip in the video clip combination corresponds to a state, the action space includes selecting or not selecting a video clip, and the reward function evaluates the quality of the video clip combination, while taking into account the importance and global coherence of the video clip. At each decision time step, the agent (summary extraction model) selects an action according to the current state, that is, selects or does not select the current video clip, and obtains an immediate reward and the next state. The goal is to maximize the cumulative reward, that is, maximize the return. The reward function is as follows:

[0174] <m> max < / m> E τ ~ πθ [ ∑ t = 1 T r ( s t , a t ) ]

[0175] in, Represents the parameters of the summary extraction model, such as the weights and biases of the neural network. The goal is to adjust To optimize the summary extraction strategy.

[0176] represents the summary extraction strategy, which is a parameterized function that selects the probability distribution of actions based on the current state. Here, the strategy Indicates in status Next select action The probability of decided.

[0177] Represents a summary extraction trajectory, which is a sequence of alternating states and actions (s1, a1, s2, a2, ...), representing the process of interaction between the agent and the environment. πθ [ ] According to the strategy The process of sampling trajectories. In this task, each state Indicates the selected video clip, each action Indicates whether to select or not select the current video clip. Sampling trajectories, based on the current policy interacting with the environment, generates a video clip extraction trajectory. Different trajectories may lead to different cumulative rewards.

[0178] For all possible trajectories, according to the strategy Sampling and calculating the expected value. The expected value here represents the average cumulative reward we expect to obtain under the current strategy.

[0179] πθ [ ∑ t = 1 T r ( s t , a t ) ] Represents the cumulative reward of a trajectory. It is composed of the immediate reward at each time step In this task, the immediate reward evaluates the Next select action Finally, a summary of the quality of the fragment combination is generated.

[0180] T represents the length of the trajectory, i.e., the total number of decision time steps. In this task, T depends on the number of video clips in the video clip combination.

[0181] represents the optimization objective, i.e., maximizing the expected cumulative reward. We adjust the model parameters , so that in the strategy The average cumulative reward obtained under maximization. This optimization process can be achieved through reinforcement learning algorithms (such as PolicyGradient).

[0182] In summary, the reward function represents the goal of optimizing the summary extraction strategy by maximizing the expected cumulative reward under the Markov decision process. The model parameters are optimized by the reinforcement learning algorithm to maximize the average reward obtained on the generated summary extraction trajectory, thereby obtaining the optimal summary extraction strategy.

[0183] The goal of the reinforcement learning algorithm is to learn an optimal strategy by interacting with the environment under the MDP framework to maximize the long-term cumulative reward. The reinforcement learning algorithm continuously updates the strategy through trial and error and experience accumulation, and gradually approaches the optimal solution of the MDP. This embodiment can use reinforcement learning algorithms, such as Q-learning, SARSA, Policy Gradient, etc., which are all developed on the basis of MDP. These algorithms use the properties of MDP (such as Markov properties, reward assumptions, etc.) to design learning strategies and update rules.

[0184] Exemplarily, this embodiment adopts the REINFORCE algorithm optimization strategy to update the model parameters according to the gradient of the reward function:

[0185] ∇ θ J ( θ ) = E τ ~ πθ [ ∑ t = 1 T ∇ θ log πθ ( a t | s t ) ( ∑ t ′ = t T r ( s t ′ , a t ′ ) − b ( s t ) ) ]

[0186] in, represents the policy gradient and the expected return of the strategy For model parameters The gradient of is updated by gradient ascent , in order to maximize the expected return.

[0187] Represents a parameterized policy function, which means that in the state Next select action The probability of Determine the probability distribution.

[0188] represents the log-likelihood function, which means that in the state Next select action In the optimization process, the log-likelihood function is maximized to increase the probability of excellent actions being selected.

[0189] Represents part of the policy gradient, representing the log-likelihood function for the model parameters The gradient is used to adjust To increase the probability of selecting excellent actions.

[0190] represents the cumulative reward from time step t to the end of the trajectory, and represents the cumulative reward from time step t to the end of the trajectory. Next select action The total reward generated by the subsequent state-action pair after the epoch. The larger this value is, the more important the current state-action pair is for obtaining a high reward.

[0191] represents the baseline function, which means that in state s t It is used to reduce the variance of gradient estimation and improve the stability of training.

[0192] represents the advantage function, which means that in the state Next select action After the advantage function is calculated, it is the difference between the actual cumulative reward and the expected reward. If the advantage function is positive, it means that the current state-action comparison is better than the average level; if it is negative, it means that it is worse than the average level.

[0193] represents the sum over all time steps in the trajectory. Here we calculate the gradient contribution of each time step and add them together to get the gradient of the entire trajectory.

[0194] For all possible trajectories Find the expected value, where According to the current strategy This expected value can be obtained from Sample multiple trajectories and then calculate the empirical average to estimate.

[0195] In summary, the above formula describes the calculation method of the policy gradient in the REINFORCE algorithm. By increasing the probability of selecting good actions and reducing the probability of selecting bad actions, the expected return of the strategy can be improved. The model can learn an optimal summary segment extraction strategy and generate high-quality video summaries.

[0196] In a possible implementation, in the above embodiment, 403: generating a video summary of a target video according to an optimal video segment combination specifically includes the following steps 4031-4033:

[0197] 4031: Constructing a graph-based integer programming model based on the optimal video clip combination;

[0198] Exemplarily, a segment similarity graph is constructed according to the optimal video segment combination, each node in the segment similarity graph represents a video segment in the optimal video segment combination, and the weight of the edge between two nodes represents the semantic similarity of the two video segments corresponding to the two nodes, wherein the semantic similarity between the video segments can be calculated by cosine similarity or dot product:

[0199]

[0200] Among them, a pre-trained Transformer Encoder can be used to calculate the semantic representation of the video clips, thereby calculating the semantic similarity of the video clips. Indicates a video clip The semantic representation of Indicates a video clip The semantic representation of Indicates a video clip With video clips The semantic similarity between .

[0201] After obtaining the similarity of the video clips, an integer programming model is established to globally optimize the optimal video clip combination and define the binary decision variables Indicates whether to select the i-th video clip, Indicates whether to select the jth video clip, When it is 0, it means not selecting the i-th video clip. When it is 1, it means selecting the i-th video segment and defining represents the importance score of the i-th video clip, which can be calculated by the local features of the clip (such as frame-level saliency, semantic information, etc.). The global objective function is as follows:

[0202]

[0203] The first term in the above global objective function represents the sum of the importance of the selected video clips, and the second term represents the sum of the similarities between the selected video clips.

[0204] Furthermore, constraints can be added to the above global objective function, such as the sum of the lengths of the selected video clips does not exceed the length limit of the video summary, at least one video clip is selected, the time order constraints of adjacent video clips, the coverage constraints of key objects, etc., to further improve the fluency and information coverage of the video summary.

[0205] Among them, the importance score of the video clip can be calculated based on the frame-level saliency, semantic information, topic relevance, speaker importance, and comprehensive importance score. The calculation method of frame-level saliency, semantic information, topic relevance, speaker importance, and comprehensive importance score is as follows:

[0206] (1) Frame-level saliency:

[0207] Extract saliency maps for each image frame of a video clip to highlight visually important areas. Commonly used methods include spectral residual method, graph convolutional network, etc.

[0208] To calculate the saliency score of each frame, you can use indicators such as the mean, maximum, or weighted sum of the saliency map.

[0209] For each video clip, the average of the saliency scores of all its frames is taken as the saliency score of the video clip.

[0210] (2) Semantic information:

[0211] Use a pre-trained language model (such as BERT) to encode the text description of each summary clip to obtain the semantic representation vector of the video clip.

[0212] The cosine similarity or dot product between the semantic representation vector of each video segment and the semantic representation vector of the entire video (the average or weighted sum of the semantic representation vectors of all segments can be used) is calculated to obtain the semantic relevance score of the segment.

[0213] The higher the semantic relevance score of a video clip, the more relevant the semantic information it contains is to the entire video, and therefore the greater the amount of semantic information.

[0214] (3) Topic relevance:

[0215] Use topic models (such as LDA) to perform topic analysis on the text description of the video clips to obtain the main topic distribution of the video clips.

[0216] Perform topic inference on the text description of each video clip to obtain the topic distribution of the video clip. Calculate the KL divergence or JS divergence between the topic distribution of each video clip and the topic distribution of the entire video to obtain the topic relevance score of the video clip.

[0217] The lower the topic relevance score of a video clip, the more relevant it is to the main topic of the video, and therefore the higher the topic relevance.

[0218] (4) Importance of the speaker:

[0219] Use a speaker recognition algorithm (such as x-vector) to perform speaker recognition on the audio of each video clip and obtain the speaker label of the video clip.

[0220] The frequency and duration of each speaker in the entire video are counted to obtain the importance score of the speaker.

[0221] For each video clip, according to its speaker label, the importance score of the corresponding speaker is used as the speaker importance score of the video clip.

[0222] (5) Comprehensive importance score:

[0223] The scores of the above aspects are normalized to range between [0, 1].

[0224] Set weight coefficients for different aspects and adjust the importance of each aspect according to the requirements of the task.

[0225] The weighted scores of each aspect are added together to obtain the comprehensive importance score of each summary fragment.

[0226] Through the above steps, we can calculate the importance score of each summary fragment ,It comprehensively considers various local features of video clips, such as visual saliency, semantic information, topic relevance, and speaker importance.,These scores can be used as an optimization target in the integer programming model to help select important video clips and generate the globally optimal video summary.

[0227] 4032: solving the global objective function by taking the maximization of the global objective function as the optimization goal, and extracting multiple target video segments from the optimal video segment combination;

[0228] Exemplarily, the integer programming solver Gurobi is used to solve the global objective function.

[0229] 4033: Sort multiple target video clips in chronological order to obtain a video summary of the target video.

[0230] Specifically, multiple target video clips are sorted in chronological order from early to late to obtain a video summary of the target video.

[0231] For example, if the optimal video clips include 10 video clips, the selection scheme is {1, 0, 0, 1, 0, 1, 0,0, 0, 1}, which means that the 1st, 4th, 6th, and 10th video clips are selected. Then the final video summary is [video clip 1, video clip 4, video clip 6, video clip 10].

[0232] In a possible implementation, in the above embodiment, after 403: generating a video summary of the target video according to the optimal video segment combination, the following steps are further included:

[0233] 404: Extract visual features and audio features of video summary;

[0234] Extract visual features in video summarization, such as key frames, key objects, scene information, etc.

[0235] Among them, the method of extracting key frames, key objects, and scene information is as follows:

[0236] Key frame extraction: You can use the inter-frame difference method to calculate the difference between adjacent frames in the video summary, and use the frames with large differences as key frames. Alternatively, you can use a convolutional neural network (such as ResNet) to extract features from each frame in the video summary, and then use a clustering algorithm (such as K-means) to cluster similar frames into one category, and select one frame from each category as a key frame. The key frame can be represented by the feature vector corresponding to the frame, such as the high-level semantic features extracted by CNN, or the key frame can be scaled to a fixed size and the pixel values ​​can be flattened into a one-dimensional vector.

[0237] Key object detection: You can use object detection algorithms such as Faster R-CNN, YOLO, etc. to pre-train the detection model on a large-scale dataset. Input the video frames in the video summary into the detection model to identify important objects in the picture (such as people, objects, etc.). Objects that appear frequently or are related to the theme of the video can be used as key objects. Key objects can be represented by their category labels (one-hot encoding) and position coordinates (bounding boxes). For example, [0,1,0,…,x1,y1,x2,y2] means that the object belongs to the second category and the position coordinates are (x1,y1,x2,y2). For multiple objects of the same class, you can take the union or weighted average of their bounding boxes.

[0238] Scene information extraction: Image classification algorithms, such as CNN (VGGNet, ResNet, etc.), can be used to perform scene recognition on the video frames in the video summary to determine what scene category each frame belongs to (such as indoor, outdoor, street, seaside, etc.). Semantic segmentation algorithms (such as FCN, DeepLab) can also be used to extract and identify scene information at the pixel level. Scene information can be represented by the category label (one-hot encoding) of image classification, such as [0,0,1,0,…], which means it belongs to the third category scene. If it is a pixel-level semantic segmentation result, the category label of each pixel can be mapped to a vector of fixed length, and then all pixels are averaged to obtain the scene representation vector of the entire frame.

[0239] Extract audio features from video summaries, such as using speech recognition technology to convert audio in video summaries into text, and then extract keywords from the text. Convert keywords into word embedding vectors, and input the word embedding vectors into a pre-trained language model. The output of the language model is pooled (such as average pooling, maximum pooling, etc.) to obtain a feature vector representation as an audio feature.

[0240] 405: Fusing the visual features and the audio features of the video summary to obtain fused features of the video summary;

[0241] Exemplarily, a multi-head attention mechanism is used to fuse the visual features and audio features of the video summary to obtain fused features of the video summary. The principle of feature fusion is consistent with the principle of feature fusion in the above embodiment 401, and will not be repeated here.

[0242] 406: Using the pre-trained summary text generation model, the fused features of the video summary are mapped into a natural language description.

[0243] For example, a large-scale video description dataset is used to train a sequence-to-sequence summary text generation model, such as an encoder-decoder architecture based on an attention mechanism. The encoder part receives the fused features of the video summary and encodes the fused features using a recurrent neural network (such as LSTM, GRU) or a Transformer encoder. The decoder part uses another recurrent neural network or a Transformer decoder to generate a natural language description based on the output of the encoder.

[0244] The summary text is generated using the trained summary text generation model. Specifically, the fusion features are input into the summary text generation model. The summary text generation model automatically generates the corresponding natural language description based on the input fusion features. The generated natural language description can capture the main content of the video summary and express it in a coherent and fluent language.

[0245] The summary text generation model can be integrated into the above-mentioned video summary generation framework, that is, the summary text generation module is integrated into the entire video summary generation framework. While generating the video summary, the corresponding natural language description is automatically generated. Finally, the generated natural language description can be presented to the user together with the video summary, thereby improving the readability and semantic expression ability of the video summary.

[0246] A video summary generation method provided in an embodiment of the present application is introduced above. A device for executing the above video summary generation method will be introduced below.

[0247] See also Figure 5 , Figure 5 This is a schematic diagram of the structure of a video summary generation device provided in an embodiment of the present application. Figure 5 As shown, the video summary generating device comprises:

[0248] A multimodal video representation acquisition unit 501, used to acquire a multimodal video representation of each video segment in a target video;

[0249] A summary segment extraction unit 502 is used to construct a Markov decision process according to the multimodal video representation of each video segment, and use a reinforcement learning algorithm to solve the optimal video segment combination that maximizes the reward of the Markov decision process, wherein the state space in the Markov decision process includes multiple video segment combinations, and the action space includes selecting or not selecting the video segment;

[0250] The summary segment organizing unit 503 is configured to generate a video summary of the target video according to the optimal video segment combination.

[0251] In a possible implementation, the multimodal video representation acquisition unit 501 is specifically used to obtain the multimodal features of each of the video clips, and for each of the video clips, use a multi-head attention mechanism to calculate the attention weights between different modal features; perform feature fusion on the multimodal features according to the attention weights between different modal features to obtain fused multimodal features; splice the fused multimodal features to obtain spliced ​​features; and map the spliced ​​features to a common semantic space to obtain a multimodal video representation of the video clip.

[0252] In a possible implementation, the summary segment extraction unit 502 includes:

[0253] A probability prediction subunit, used for inputting the multimodal video representation of each video segment into a pointer network, and predicting the probability of each video segment being a starting position of a summary and a probability of being an ending position of a summary;

[0254] a video segment combination determination subunit, configured to determine the video segment with the highest probability of being the start position of the summary as the start video segment, and determine the video segment with the highest probability of being the end position of the summary as the end video segment, to obtain the plurality of video segment combinations;

[0255] The optimal video clip combination solving subunit is used to construct a Markov decision process according to the plurality of video clip combinations, and use a reinforcement learning algorithm to solve the optimal video clip combination that maximizes the reward of the Markov decision process.

[0256] In a possible implementation, the probability prediction subunit is specifically used to input the multimodal video representation of each of the video clips into a bidirectional long short-term memory network in a pointer network to obtain a contextual representation of each of the video clips; input the contextual representation of each of the video clips into a first multi-layer perceptron in the pointer network to obtain a probability of each of the video clips being the starting position of a summary; and input the contextual representation of each of the video clips into a second multi-layer perceptron in the pointer network to obtain a probability of each of the video clips being the ending position of a summary.

[0257] In a possible implementation, the summary segment organizing unit 503 is specifically used to construct a graph-based integer programming model according to the optimal video segment combination, wherein the global objective function of the graph-based integer programming model represents the sum of the importance of the selected video segments and the sum of the similarities between the selected video segments; solve the global objective function with maximization of the global objective function as the optimization goal, extract multiple target video segments from the optimal video segment combination; sort the multiple target video segments in chronological order to obtain the video summary of the target video.

[0258] In a possible implementation, the video summary generating device further includes:

[0259] The summary text generation unit is used to extract the visual features and audio features of the video summary; perform feature fusion on the visual features and audio features of the video summary to obtain the fused features of the video summary; and map the fused features of the video summary into a natural language description using a pre-trained summary text generation model, wherein the summary text generation model includes an encoder and a decoder, and is trained using a video description dataset.

[0260] The present embodiment discloses a video summary generation device, which, after acquiring the multimodal features of each video clip in the target video, uses a multi-head attention mechanism to perform feature fusion on the multimodal features of each video clip respectively, so as to obtain a more comprehensive and accurate multimodal video representation of each video clip. The process of extracting the optimal video clip combination from multiple video clips in the target video is modeled as a Markov decision process, taking into account the local importance and global coherence of the video clips, and using a reinforcement learning algorithm to solve the optimal strategy of the Markov decision process, extracting a more concise and coherent optimal video clip combination composed of important video clips, thereby generating an accurate video summary based on the optimal video clip combination, and improving the quality of the video summary.

[0261] The present application also provides an electronic device in an embodiment. Figure 6 As shown, it shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiment of the present application. The electronic device in the embodiment of the present application may include but is not limited to fixed terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 6 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0262] like Figure 6 As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 to a random access memory (RAM) 603. When the electronic device is powered on, various programs and data required for the operation of the electronic device are also stored in the RAM 603. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0263] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a memory card, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Figure 6 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0264] An embodiment of the present application also provides a computer program product, including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any one of the video summary generation methods provided in the embodiments of the present application.

[0265] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any video summary generation method provided in the embodiment of the present application.

[0266] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0267] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0268] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0269] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

Claims

1. A video summary generation method, characterized in that: include: Obtain a multimodal video representation of each video clip in the target video; Constructing a Markov decision process according to the multimodal video representation of each of the video clips, and solving the optimal video clip combination that maximizes the return of the Markov decision process using a reinforcement learning algorithm, wherein the state space in the Markov decision process includes multiple video clip combinations, and the action space includes selecting or not selecting the video clip; Generating a video summary of the target video according to the optimal video segment combination; The step of obtaining a multimodal video representation of each video clip in the target video includes: Acquire multimodal features of each of the video clips, the multimodal features comprising: image features, audio features, and text features; For each of the video clips, using a multi-head attention mechanism to calculate the attention weights between different modal features; The multimodal features are fused according to the attention weights between the features of different modalities to obtain the fused multimodal features; Splicing the fused multimodal features to obtain splicing features; Mapping the concatenated features into a common semantic space to obtain a multimodal video representation of the video clip; The step of generating a video summary of the target video according to the optimal video segment combination includes: A graph-based integer programming model is constructed according to the optimal video segment combination, wherein a global objective function of the graph-based integer programming model represents the sum of the importance of the selected video segments and the sum of the similarities between the selected video segments, wherein the importance of the video segments is calculated according to the saliency at the frame level, the amount of semantic information, the topic relevance, the importance of the speaker, and the comprehensive importance score; Solving the global objective function by taking maximization of the global objective function as an optimization goal, and extracting a plurality of target video segments from the optimal video segment combination; The multiple target video clips are sorted in chronological order to obtain a video summary of the target video.

2. The video summary generation method according to claim 1, characterized in that: The step of constructing a Markov decision process according to the multimodal video representation of each video clip, and using a reinforcement learning algorithm to solve an optimal video clip combination that maximizes the return of the Markov decision process, includes: Inputting the multimodal video representation of each of the video clips into a pointer network, and predicting the probability of each of the video clips being the starting position of a summary and the probability of each of the video clips being the ending position of a summary; Determine the video segment with the highest probability of being the starting position of the summary as the starting video segment, and determine the video segment with the highest probability of being the ending position of the summary as the ending video segment, to obtain the plurality of video segment combinations; A Markov decision process is constructed according to the plurality of video clip combinations, and a reinforcement learning algorithm is used to solve an optimal video clip combination that maximizes the reward of the Markov decision process.

3. The video summary generation method according to claim 2, characterized in that: Inputting the multimodal video representation of each video clip into a pointer network, and predicting the probability of each video clip being a summary starting position and a summary ending position, comprises: Inputting the multimodal video representation of each of the video clips into a bidirectional long short-term memory network in the pointer network to obtain a contextual representation of each of the video clips; Inputting the context representation of each video clip into the first multi-layer perceptron in the pointer network to obtain the probability of each video clip being the starting position of the summary; The context representation of each video segment is input into the second multi-layer perceptron in the pointer network to obtain the probability of each video segment being the end position of the summary.

4. The video summary generation method according to claim 1, characterized in that: After generating the video summary of the target video according to the optimal video segment combination, the method further includes: extracting visual features and audio features of the video summary; Performing feature fusion on the visual features and audio features of the video summary to obtain fusion features of the video summary; The fused features of the video summary are mapped into a natural language description using a pre-trained summary text generation model, wherein the summary text generation model includes an encoder and a decoder, and is trained using a video description dataset.

5. A video summary generation device, characterized in that: include: A multimodal video representation acquisition unit, used to acquire a multimodal video representation of each video segment in a target video; A summary segment extraction unit is used to construct a Markov decision process according to the multimodal video representation of each video segment, and use a reinforcement learning algorithm to solve the optimal video segment combination that maximizes the reward of the Markov decision process, wherein the state space in the Markov decision process includes multiple video segment combinations, and the action space includes selecting or not selecting the video segment; A summary segment organization unit, used for generating a video summary of the target video according to the optimal video segment combination; The multimodal video representation acquisition unit is specifically used for: Acquire multimodal features of each of the video clips, the multimodal features comprising: image features, audio features, and text features; For each of the video clips, using a multi-head attention mechanism to calculate the attention weights between different modal features; The multimodal features are fused according to the attention weights between the features of different modalities to obtain the fused multimodal features; Splicing the fused multimodal features to obtain splicing features; Mapping the concatenated features into a common semantic space to obtain a multimodal video representation of the video clip; The summary fragment organization unit is specifically used for: A graph-based integer programming model is constructed according to the optimal video segment combination, wherein a global objective function of the graph-based integer programming model represents the sum of the importance of the selected video segments and the sum of the similarities between the selected video segments, wherein the importance of the video segments is calculated according to the saliency at the frame level, the amount of semantic information, the topic relevance, the importance of the speaker, and the comprehensive importance score; Solving the global objective function by taking maximization of the global objective function as an optimization goal, and extracting a plurality of target video segments from the optimal video segment combination; The multiple target video clips are sorted in chronological order to obtain a video summary of the target video.

6. A computer program product, characterized in that The method comprises computer-readable instructions, and when the computer-readable instructions are executed on an electronic device, the electronic device implements the video summary generation method according to any one of claims 1 to 4.

7. An electronic device, characterized in that: The method comprises at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the electronic device can implement the video summary generation method according to any one of claims 1 to 4.

8. A computer storage medium, characterized in that: The storage medium carries one or more computer programs, and when the one or more computer programs are executed by an electronic device, the electronic device can implement the video summary generation method according to any one of claims 1 to 4.

Images