Sports event commentary video clipping method and system based on multi-modal fusion model

Abstract

The application discloses a sports event commentary video clipping method and system based on a multi-modal fusion model, relates to the technical field of video clipping, and comprises the following steps: receiving a sports event commentary video and a clipping slice, integrating the sports event commentary video and the clipping slice to generate a sports event commentary video clipping dataset; inputting the sports event commentary video clipping dataset into a pre-established multi-modal fusion model, outputting inference results, calculating the inference results and data in the sports event commentary video clipping dataset, and obtaining a cross-entropy loss function; setting a function threshold value; if the cross-entropy loss function is less than the threshold value, updating and saving multi-modal fusion model parameters to obtain trained multi-modal fusion model parameters; if the cross-entropy loss function is greater than or equal to the threshold value, re-calculating until the cross-entropy loss function is less than the threshold value; inputting the sports event commentary video into the trained multi-modal fusion model parameters to obtain optimized inference results, and clipping the sports event commentary video.

Description

Technical Field

[0001] This invention relates to the field of video editing technology, specifically to a method and system for editing sports event commentary videos based on a multimodal fusion model. Background Technology

[0002] With societal progress and technological advancements, people's entertainment options are becoming increasingly diverse. Watching sporting events is a particularly popular choice. However, traditional sports commentary videos often run for hours, which is inconvenient for viewers. Therefore, to meet audience demands, sports broadcasters need to quickly edit commentary videos into engaging short clips and upload them to short video platforms to attract more viewers.

[0003] In the field of sports commentary video editing, traditional methods mainly rely on the subjective opinions and experience of observers to determine the editing content. However, this method has a certain degree of subjectivity and one-sidedness, and cannot fully and accurately capture the key moments and important information of the game. In addition, traditional video editing methods often require manual operation, which is inefficient and prone to errors. Summary of the Invention

[0004] To address the shortcomings mentioned in the background section, the present invention aims to provide a method and system for editing sports event commentary videos based on a multimodal fusion model, which can fully utilize the multimodal information of sports event commentary videos to quickly and accurately edit out exciting sports event commentary video clips.

[0005] Firstly, the objective of this invention can be achieved through the following technical solution: a method for editing sports event commentary videos based on a multimodal fusion model, the method comprising the following steps:

[0006] Receive sports event commentary videos and clips, and integrate the sports event commentary videos and clips to generate a sports event commentary video clip dataset;

[0007] The sports event commentary video clip dataset is input into a pre-established multimodal fusion model, and the inference result is output. The inference result is then used to calculate the cross-entropy loss function with the data in the sports event commentary video clip dataset.

[0008] Set a threshold for the function. If the cross-entropy loss function is less than the threshold, update and save the parameters of the multimodal fusion model to obtain the trained multimodal fusion model parameters. If the cross-entropy loss function is greater than or equal to the threshold, recalculate the cross-entropy loss function until the cross-entropy loss function is less than the threshold.

[0009] The sports event commentary videos are input into the parameters of the trained multimodal fusion model to obtain optimized inference results, and the sports event commentary videos are edited based on the optimized inference results.

[0010] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: the process of integrating sports event commentary videos and clips to generate a sports event commentary video clip dataset includes the following steps:

[0011] Initialize the dataset D, where D is an empty set;

[0012] Search for sports commentary videos on video websites and label them as V. Find one high-view count clip from each of the M videos on different platforms and label it as CV. i , of which CV i ={CV1,CV2,…,CV M}, where M is the number of heads in the multi-head self-attention mechanism of the multimodal fusion model;

[0013] Get each CV i Start time CS within V i and end time CE i ;

[0014] All CVs in V i Start time CS i and end time CE i Combined, this generates dataset entries DE = {V, CS1, CS2, ..., CS} M ,CE1,CE2,…,CE M};

[0015] Input dataset entries DE into dataset D, repeat the above steps to continue obtaining dataset entries and inputting them into dataset D, until dataset D reaches the set number requirement, and output the sports event commentary video clip dataset D.

[0016] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: the process of inputting the sports event commentary video clip dataset into a pre-established multimodal fusion model and outputting the inference result includes the following steps:

[0017] The original sports commentary videos V in the sports commentary video clip dataset D are converted from video to audio A; the audio A is then converted from speech to text to obtain the text of each sentence in the audio {T1,T2,…,T…}. n}, get the start time {S1, S2, ..., S} of the text of each sentence in audio A. n}, and the end time {E1, E2, ..., E} of the text of each sentence in audio A. n};

[0018] Extract the text modal information and visual modal information of each sentence; for each sentence's text T i Extract the text modality vector (TF) of this sentence. i According to S i and E i In the original sports commentary video V, the set of frames C is located, and the frame modal information vector CF of the set of frames C is extracted. i ;Transform the text modality information vector TF i and image modal information vector CF i By concatenating these features to form an augmented matrix, we obtain the multimodal feature fusion vector (MF) of this sentence. i =[TF i ,CF i ];

[0019] Using the obtained MF i ={MF1,MF2,…,MF n Using a multi-head self-attention mechanism with M heads, a multimodal clipping site recommendation tensor of size M*n is obtained. AFM ij Indicates in MF j The output of the i-th attention head is a vector with dimension MF. i Same; M is the number of heads in the multi-head self-attention mechanism of the multimodal fusion model;

[0020] Each vector in the Multimodal Clipping Site Recommendation Tensor (MCT) is AFM. ij All of them are transformed into a two-dimensional clipping point recommendation value vector (MCV) through a fully connected layer. ij ; the recommended value vector (MCV) of the clipping site ij Normalization is performed using a softmax layer to obtain the clip position recommendation probability vector (MCPV). ij The first element of the clipping location recommendation probability vector represents the probability that text segment j will be clipped into the i-th clipping result. This probability value is denoted as MCP. ij =MCPV ij [0]; Let MCPG be the set of recommended probabilities for the editing position of the i-th head. i ={MCP i1 MCP i2 ,…,MCP in}, as a result of reasoning.

[0021] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: the text modality information vector TF i and image modal information vector CF i The acquisition process:

[0022] Text T i Inputting the pre-trained BERT model yields n vectors of dimension 768, denoted as {TV1, TV2, ..., TV...} n};

[0023] The resulting n vectors {TV1,TV2,…,TV} n Temporal feature extraction is performed using LSTM, and the last hidden state of the LSTM is taken as the text modality vector (TF) of the text. i ;

[0024] Uniformly sample t = 2*(E) from the image set C. i -S i +1) images, forming the extracted image set. For each image in the IM, a CNN is used to extract image features, resulting in an image feature vector.

[0025] The obtained t image feature vectors are used to perform temporal feature extraction using LSTM, and the last hidden state of the LSTM is taken as the image modality information vector (CF) of the image set. i .

[0026] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: using a CNN to extract image features to obtain an image feature vector. The process is as follows:

[0027] For the input image IMG, a 3*3 convolution kernel is used to perform a convolution operation on the IMG; then, a 2*2 max pooling layer is used to max pool the convolution result; and the max pooling result is transformed into a one-dimensional vector.

[0028] A linear transformation is performed on the one-dimensional vector, followed by a ReLU activation function, to obtain the vector after feature extraction from the input image using a CNN, which serves as the image feature vector.

[0029] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: the process of calculating the cross-entropy loss function using the obtained inference results and data within the sports event commentary video clip dataset includes the following steps:

[0030] The method to form the fact vector of the output of the i-th head is: find the CS in the dataset entry corresponding to video V. i and CE i Initialize the fact vector G i = {0,0,…,0}, with a length of n, where n is the number of segments in the resulting text; for the j-th segment, if the interval formed by the start and end times of the text is CS i and CE i If the interval formed is a subinterval, then G i [j] = 0, otherwise G i [j] = 1;

[0031] Use G i With MCPG i Calculate the cross-entropy loss function and sum all elements in the result to obtain the loss function value L for the i-th head. i

[0032] Perform the above operation on all M heads, sum all the loss function values, and obtain the final loss function value L, where L = L1 + L2 + ... + L M .

[0033] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: the process of inputting the sports event commentary video into the parameters of the trained multimodal fusion model to obtain the optimized inference result, and editing the sports event commentary video based on the optimized inference result, includes the following steps:

[0034] The reasoning result MCPG i The set R = {MCPG1, MCPG2, ..., MCPG} is composed of... M};

[0035] For MCPG i Remove all elements less than 0.6 to obtain the filtered set of recommended probabilities for editing positions. Use dynamic programming to find the second index k. m A contiguous subarray with the largest average is denoted as . Where the subscript satisfies t i+1 =t i +1; TMCPG i The ordered list formed by the second subscript is the recommended cut point subscript TI of the i-th cut result. i =t1,t2,…,t j Based on the index and the start and end times of each sentence in audio A, locate the position TP of this edited result in the original video. i =[S t1 Etj ]; Take the original video from S t1 To E tj Edit the portion to obtain the i-th edit result CR. i For all M heads, perform the above operations to obtain M edit results CR1, CR2, ..., CR M ;

[0036] The number of edited results must meet the requirements; otherwise, update FMCPG. i =FMCPG i -TMCPG i In FMCPG i Remove TMCPG i All elements in the file, up to the number of edited results, meet the requirements.

[0037] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: deriving the TMCPG using a dynamic programming algorithm. i The process includes the following steps:

[0038] FMCPG (Probability Set of Editing Sites) i Find the first index k m A contiguous subarray, denoted as {x1, x2, ..., x...} n Let start be the index of the first element of the current subarray in the original array; initialize the global optimal solution p. max =0,s max =0,e max =0, representing the average value when the average value of the subarray is maximized, the starting index of the subarray in the original array, and the ending index of the subarray in the original array, respectively;

[0039] Let the array dp[i] represent the array considering x. i If it is definitely in a subarray, only consider x. i The maximum average of the subarray when considering x and its preceding elements; let array s[i] represent the maximum average of the subarray when considering x and its preceding elements. i Given that the subarray must be within the subarray, find the length of the subarray whose average value is maximized.

[0040] Use the following state transition equation to iterate through i from 2 to n.

[0041] dp[1]=x1

[0042] s[1]=1

[0043]

[0044]

[0045] Iterate through the dp array from 1 to n, find the index that makes the dp array the largest, and denote it as r.

[0046] r = argmax{dp[i]}

[0047] If dp[r] is greater than p max So update p max =dp[r],s max =start+rs[r],e max =start+r-1

[0048] Find the next index k m A contiguous subarray, denoted as {x1, x2, ..., x...} n Let start be the index of the first element of the current subarray in the original array; repeat the operation until the end of the array.

[0049] Results The algorithm has ended.

[0050] Secondly, in order to achieve the above objectives, this invention discloses a sports event commentary video editing system based on a multimodal fusion model, comprising:

[0051] The video processing module is used to receive sports event commentary videos and clips, and integrate the sports event commentary videos and clips to generate a sports event commentary video clip dataset;

[0052] The loss calculation module is used to input the sports event commentary video clip dataset into a pre-established multimodal fusion model, output the inference result, and use the obtained inference result and the data in the sports event commentary video clip dataset to calculate the cross-entropy loss function.

[0053] The model training module is used to set a function threshold. If the cross-entropy loss function is less than the threshold, the multimodal fusion model parameters are updated and saved to obtain the trained multimodal fusion model parameters. If the cross-entropy loss function is greater than or equal to the threshold, the cross-entropy loss function is recalculated until the cross-entropy loss function is less than the threshold.

[0054] The video editing module is used to input sports event commentary videos into the parameters of the trained multimodal fusion model to obtain optimized inference results, and then edit the sports event commentary videos based on the optimized inference results.

[0055] In another aspect of the present invention, in order to achieve the above-mentioned objective, a device is disclosed, characterized in that it comprises:

[0056] One or more processors;

[0057] Memory, used to store one or more programs;

[0058] When one or more of the programs are executed by one or more of the processors, the one or more processors implement the sports event commentary video editing method based on the multimodal fusion model as described above.

[0059] The beneficial effects of this invention are:

[0060] This invention makes full use of multiple modal information from sports commentary videos to quickly and accurately edit out exciting sports commentary video clips. Attached Figure Description

[0061] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0062] Figure 1 This is a schematic diagram of the method flow of the present invention;

[0063] Figure 2 This is a schematic diagram of the inference process of the multimodal fusion model of the present invention;

[0064] Figure 3 This is a schematic diagram of the training process of the multimodal fusion model of the present invention;

[0065] Figure 4 This is a schematic diagram of the system structure of the present invention;

[0066] Figure 5 This is a graph showing the change in the loss function in a specific embodiment. Detailed Implementation

[0067] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0068] Example 1:

[0069] The following is a description of the relevant terms used in the embodiments of this application:

[0070] Video editing: Video editing is the non-linear editing of video sources using software. It involves adding images, background music, special effects, scenes, and other materials to remix the video, cutting and merging the video source, and generating new videos with different expressive power through secondary encoding.

[0071] Multimodality: The concept of multimodality originated from research on information representation methods in the field of computer-human interaction. The term "modality" was defined as the way information is represented and exchanged on a specific physical medium. Research found that using media designations such as language, video, and audio to describe information representation methods was too broad and granular; insufficient to distinguish the actual representation methods used. Therefore, the more granular concept of "modality" was introduced. Multimedia media can be decomposed into multiple monomodals. For example, video, as a multimedia medium, can be decomposed into multiple monomodals such as dynamic images, dynamic speech, and dynamic text. For the scientific and practical definition of the modality concept, the classification of monomodals must meet the requirements of completeness, orthogonality, relevance, and intuitiveness. The phenomenon of multiple types of monomodal information coexisting or co-occurring on the same thing is very common. Spoken language and written text coexist when people converse; images and their corresponding explanatory text co-occur on internet web pages, and so on. The collective term for multiple coexisting or co-occurring monomodal information is what is known as multimodal information. Multimodal information processing, which integrates multiple unimodal information processing methods, involves the acquisition, organization, analysis, retrieval, understanding, and creation of multimodal information.

[0072] like Figure 1 As shown, the method for editing sports event commentary videos based on a multimodal fusion model is characterized by the following steps:

[0073] Receive sports event commentary videos and clips, and integrate the sports event commentary videos and clips to generate a sports event commentary video clip dataset;

[0074] The process of integrating sports event commentary videos and clips to generate a sports event commentary video clip dataset includes the following steps:

[0075] Initialize the dataset D, where D is an empty set;

[0076] Search for sports commentary videos on video websites and label them as V. Find one high-view count clip from each of the M videos on different platforms and label it as CV. i , of which CV i ={CV1,CV2,…,CV M}, where M is the number of heads in the multi-head self-attention mechanism of the multimodal fusion model;

[0077] Get each CV i Start time CS within V i and end time CE i ;

[0078] All CVs in V i Start time CS i and end time CE i Combined, this generates dataset entries DE = {V, CS1, CS2, ..., CS} M ,CE1,CE2,…,CE M};

[0079] Input dataset entries DE into dataset D, repeat the above steps to continue obtaining dataset entries and inputting them into dataset D, until dataset D reaches the set number requirement, and output the sports event commentary video clip dataset D.

[0080] The sports event commentary video clip dataset is input into a pre-established multimodal fusion model, and the inference result is output. The inference result is then used to calculate the cross-entropy loss function with the data in the sports event commentary video clip dataset.

[0081] In this embodiment, the process of inputting a dataset of sports event commentary video clips into a pre-established multimodal fusion model and outputting inference results includes the following steps:

[0082] (3.1) Convert the original sports commentary video V to audio A; use a speech recognition model to convert audio A to text, obtaining the text {T1,T2,…,T…} for each sentence in the audio. n}, the start time of each sentence in audio A is {S1, S2, ..., S}. n}, and the end time {E1, E2, ..., E} of each sentence in audio A. n};

[0083] (3.2) Extract the text modal information and visual modal information of each sentence; for each sentence T i Extract the text modality vector (TF) of this sentence. i According to S i and E i Locate the set of frames C in the original video and extract the frame modal information vector CF of this set of frames. i ;Transform the text modality information vector TF i and image modal information vector CF i By concatenating these components, an augmented matrix is ​​formed, resulting in the multimodal feature fusion vector (MF) of this sentence. i =[TF i,CF i ];

[0084] (3.3) For the {MF1,MF2,…,MF} obtained in step (2) n Using a multi-head self-attention mechanism with M heads, a multimodal clipping site recommendation tensor of size M*n is obtained. AFM ij Indicates in MF j The output of the i-th attention head is a vector with dimension MF. i Same; M is the number of heads, which means the number of video platforms the model user wants to publish the clips on, because the characteristics of a video with high play counts are different on different video platforms, so different heads are needed to capture these characteristics;

[0085] (3.4) AFM is generated from each vector in the multimodal clipping site recommendation tensor MCT. ij All of them are transformed into a two-dimensional clipping point recommendation value vector (MCV) through a fully connected layer. ij ; the recommended value vector (MCV) of the clipping site ij Normalization is performed using a softmax layer to obtain the clip position recommendation probability vector (MCPV). ij The first element of the clipping location recommendation probability vector represents the probability that text segment j will be clipped into the i-th clipping result. This probability value is denoted as MCP. ij =MCPV ij [0]; Let MCPG be the set of recommended probabilities for the editing position of the i-th head. i ={MCP i1 MCP i2 ,…,MCP in};

[0086] (3.5) The inference result of the multimodal fusion model is the set of recommendation probabilities for all M clip positions (MCPG). i The set is composed of MCPG1, MCPG2, ..., MCPG, resulting in the inference R = {MCPG1, MCPG2, ..., MCPG}. M}

[0087] Among them, the text modality information vector TF i and image modal information vector CF i The acquisition process:

[0088] (3.2.1) Transfer text T i Inputting the pre-trained BERT model yields n vectors of dimension 768, denoted as {TV1, TV2, ..., TV...} n}

[0089] (3.2.2) Use LSTM to extract temporal features from the n vectors obtained in (3.2.1), and take the last hidden state of LSTM as the text modality information vector TF of the text. i

[0090] (3.2.3) Uniformly sample t = 2*(E) from the image set C. i -S i +1) images, forming the extracted image set. For each image in the IM, a CNN is used to extract image features, resulting in an image feature vector.

[0091] (3.2.4) Use LSTM to extract temporal features from the t image feature vectors obtained in (3.2.3), and take the last hidden state of LSTM as the image modality information vector CF of the image set. i .

[0092] In this process, CNN is used for image feature extraction to obtain image feature vectors. The process is as follows:

[0093] For the input image IMG, a 3*3 convolution kernel is used to perform a convolution operation on the IMG; then, a 2*2 max pooling layer is used to max pool the convolution result; and the max pooling result is transformed into a one-dimensional vector.

[0094] A linear transformation is performed on the one-dimensional vector, followed by a ReLU activation function, to obtain the vector after feature extraction from the input image using a CNN, which serves as the image feature vector.

[0095] The process of deriving the cross-entropy loss function by using the obtained inference results and data from the sports event commentary video clip dataset includes the following steps:

[0096] (4.1) The method to form the fact vector of the output of the i-th head is: find the CS in the dataset entry corresponding to video V. i and CE i Initialize the fact vector G i = {0,0,…,0}, with a length of n, where n is the number of text segments obtained in step (3.1); for the j-th text segment, if the interval formed by the start and end times of the text segment is CS i and CE i If the interval formed is a subinterval, then G i [j] = 0, otherwise G i [j] = 1;

[0097] (4.2) Using G i The MCPG calculated in step (3.4) i Calculate the cross-entropy loss function and sum all elements in the result to obtain the loss function value L for the i-th head. i

[0098] (4.3) Perform steps (4.1) and (4.2) on all M heads, sum all the loss function values, and obtain the final loss function value L = L1 + L2 + ... + L M

[0099] Set a threshold for the function. If the cross-entropy loss function is less than the threshold, update and save the parameters of the multimodal fusion model to obtain the trained multimodal fusion model parameters. If the cross-entropy loss function is greater than or equal to the threshold, recalculate the cross-entropy loss function until the cross-entropy loss function is less than the threshold.

[0100] The sports event commentary videos are input into the parameters of the trained multimodal fusion model to obtain optimized inference results, and the sports event commentary videos are edited based on the optimized inference results.

[0101] Specifically, the present invention will be further illustrated below through embodiments, the process of which is as follows:

[0102] The set of recommended probabilities for all M head clipping points MCPG i The set R = {MCPG1, MCPG2, ..., MCPG} is composed of {MCPG1, MCPG2, ..., MCPG}. M}

[0103] (6.2) For the MCPG set of recommended editing points for the i-th head i =MCP i1 MCP i2 ,…,MCP in Remove all elements with a probability less than 0.6 to obtain the filtered set of recommended editing positions. Use dynamic programming to find the second index k. m A contiguous subarray with the largest average is denoted as . Where the subscript satisfies t i+1 =t i +1; TMCPG i The ordered list formed by the second subscript is the recommended cut point subscript TI of the i-th cut result. i =t1,t2,…,t j Based on the subscripts and the start and end times of each sentence in audio A, the position TP of this edited result in the original video can be located. i =[S t1E tj ]; Take the original video from S t1 To E tj The portion is edited out to obtain the i-th edit result CR. i ;

[0104] (6.3) For all M heads, perform the operation in (6.2) to obtain M clipping results CR1, CR2, ..., CR M ;

[0105] (6.4) If the number of edited results meets the requirements, proceed to the next step; otherwise, update FMCPG. i =FMCPG i -TMCPG i That is, in FMCPG i Remove TMCPG i All elements in the file are processed until the number of edited results meets the requirements.

[0106] Furthermore, the TMCPG was derived using a dynamic programming algorithm. i The method is as follows:

[0107] (6.2.1) Recommendation probability set FMCPG at the editing site i Find the first index k m A contiguous subarray, denoted as {x1, x2, ..., x...} n Let start be the index of the first element of the current subarray in the original array; initialize the global optimal solution p. max =0,s max =0,e max =0, representing the average value when the average value of the subarray is maximized, the starting index of the subarray in the original array, and the ending index of the subarray in the original array, respectively;

[0108] (6.2.2) Let the array dp[i] represent the array when considering x i If it is definitely in a subarray, only consider x. i The maximum average of the subarray when considering x and its preceding elements; let array s[i] represent the maximum average of the subarray when considering x and its preceding elements. i Given that the subarray must be within the subarray, find the length of the subarray whose average value is maximized.

[0109] (6.2.3) Use the following state transition equation to traverse i from 2 to n.

[0110] dp[1]=x1

[0111] s[1]=1

[0112]

[0113]

[0114] (6.2.4) Traverse the dp array from 1 to n, find the index that makes the dp array the largest, and denote it as r.

[0115] r = argmax{dp[i]}

[0116] (6.2.5) If dp[r] is greater than p max So update p max =dp[r],s max =start+rs[r],e max =start+r-1

[0117] (6.2.6) Find the next index k m A contiguous subarray, denoted as {x1, x2, ..., x...} n Let start be the index of the first element of the current subarray in the original array; repeat step (6.2.2) until the end of the array;

[0118] (6.2.7) The final result is obtained Algorithm ends;

[0119] Example 2: Second aspect, such as Figure 4 As shown, this invention discloses a sports event commentary video editing system based on a multimodal fusion model, characterized by comprising:

[0120] The video processing module is used to receive sports event commentary videos and clips, and integrate the sports event commentary videos and clips to generate a sports event commentary video clip dataset;

[0121] The loss calculation module is used to input the sports event commentary video clip dataset into a pre-established multimodal fusion model, output the inference result, and use the obtained inference result and the data in the sports event commentary video clip dataset to calculate the cross-entropy loss function.

[0122] The model training module is used to set a function threshold. If the cross-entropy loss function is less than the threshold, the multimodal fusion model parameters are updated and saved to obtain the trained multimodal fusion model parameters. If the cross-entropy loss function is greater than or equal to the threshold, the cross-entropy loss function is recalculated until the cross-entropy loss function is less than the threshold.

[0123] The video editing module is used to input sports event commentary videos into the parameters of the trained multimodal fusion model to obtain optimized inference results, and then edit the sports event commentary videos based on the optimized inference results.

[0124] Example 3:

[0125] This embodiment will use a racing commentary video from a TV station as an example to demonstrate the workflow of this system. The process includes the following steps:

[0126] (1) Create a sports event commentary video clip dataset D; in this example, the number of heads in the multi-head attention mechanism is set to M=2; find the official account of a TV station on two video platforms and download the racing commentary videos from that account. The specific number of videos is not limited in this embodiment, but should be no less than 200 to ensure the effect; for each downloaded video, find a video clip with a high number of views from the two video platforms mentioned above. The specific number of views is not limited in this embodiment, but video clips with more than 20,000 views are selected in this embodiment; create the sports event commentary video clip dataset D according to the method mentioned in Example 1; some entries in the dataset are shown in the table below (the time unit in the table is seconds).

[0127]

[0128] (2) For each video in the sports commentary video clip dataset D, perform inference and calculate the cross-entropy loss function; the inference and cross-entropy loss function calculation use the method mentioned in Example 1; the dimension of the LSTM hidden state is not limited in this example, and the hidden state dimension size is taken as 2048 in this example.

[0129] (3) After calculating the cross-entropy loss function, the model parameters are updated using the backpropagation algorithm, and inference and cross-entropy loss function values ​​are calculated again until the loss function value is lower than the set threshold; in this embodiment, the set loss function threshold is 0.01, and the change in the loss function value during the training process is as follows: Figure 5 As shown; save the model parameters after training.

[0130] (4) Input a racing commentary video not in the dataset into the model. In this embodiment, the qualifying commentary video of the first race of a racing event in 2023 is selected. Use the model to perform inference and obtain the inference results MCPG1={0.0852,0.0745,…,0.0677},MCPG2{0.0140,0.0399,…,0.2676}, where MCPG1 and MCPG2 are both n-dimensional real vectors, and n is the number of texts contained in the video. In this embodiment, n is 5019. Remove the elements with values ​​less than 0.6. Use the dynamic programming algorithm mentioned in Embodiment 1 to obtain the corresponding editing point TMCPG1={MCP 1,1524 MCP 1,1525 ,…,MCP1,1687},TMCPG2={MCP 2,2231 MCP 1,2232 ,…,MCP 2,2266};

[0131] (5) The video segment corresponding to the editing point is then cut out to obtain the video slice; specifically in this embodiment, since TMCPG1 is from MCP 1,1524 To MCP 1,1687 Therefore, the time position of the first slice in the original video is S. 1524 To E 1687 The second slice is similar; its time position in the original video is S. 2231 To E 2266 You can use video editing software to cut out the video at the corresponding time position from the original video to get the edited slices; you can also manually enhance the video slices to improve their appearance.

[0132] (6) By comparing the overlap between the edited video slice C1 and the existing video slice C2 on the video platform, the editing effect of this method can be evaluated. Specifically, in this embodiment, a video slice C2 with 15,000 views can be found on video platform 1. The starting time of C2 in the original video is from 710 seconds to 785 seconds, and the starting time of the edited video slice C1 is from 716 seconds to 778 seconds in the original video. Through calculation, the overlap can be obtained as follows:

[0133]

[0134] This verifies the reliability of the method.

[0135] Based on the same inventive concept, this invention also provides a computer device, comprising: one or more processors, and a memory for storing one or more computer programs; the programs include program instructions, and the processor executes the program instructions stored in the memory. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, used to implement one or more instructions, specifically for loading and executing one or more instructions stored in a computer storage medium to implement the above-described method.

[0136] It should be further explained that, based on the same inventive concept, the present invention also provides a computer storage medium storing a computer program, which, when executed by a processor, performs the above-described method. This storage medium can be any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In the present invention, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0137] In the description of this specification, references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0138] The foregoing has shown and described the basic principles, main features, and advantages of this disclosure. Those skilled in the art should understand that this disclosure is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of this disclosure. Various changes and modifications can be made to this disclosure without departing from its spirit and scope, and all such changes and modifications fall within the scope of this disclosure as claimed.

Claims

1. A method for editing sports event commentary videos based on a multimodal fusion model, characterized in that, The method includes the following steps: Receive sports event commentary videos and clips, and integrate the sports event commentary videos and clips to generate a sports event commentary video clip dataset; The process of integrating sports event commentary videos and clips to generate a sports event commentary video clip dataset includes the following steps: Initialize the dataset D, where D is an empty set; Search for sports commentary videos on video websites and mark them as V. Find one high-viewed clip from each of the M videos on different platforms and mark it as... ,in , where M is the number of heads in the multi-head self-attention mechanism of the multimodal fusion model; Get each Start time within V and end time ; All of V start time and end time Combine them to generate dataset entries. ; Input dataset entries DE into dataset D, repeat the above steps to continue obtaining dataset entries and inputting them into dataset D, until dataset D reaches the set number requirement, and output the sports event commentary video clip dataset D; The sports event commentary video clip dataset is input into a pre-established multimodal fusion model, and the inference result is output. The inference result is then used to calculate the cross-entropy loss function with the data in the sports event commentary video clip dataset. The process of deriving the cross-entropy loss function by using the obtained inference results and data from the sports event commentary video clip dataset includes the following steps: Forming the first The method for outputting the fact vector of the head is: find the data entry corresponding to video V in the dataset. and Initialize the fact vector The length is n, where n is the number of segments of the resulting text; for the th segment... A text segment, if the interval formed by the start and end times of the text is... and The subintervals of the formed interval, then ,otherwise ; use With the result of reasoning Calculate the cross-entropy loss function and sum all elements in the result to obtain the first... Loss function value of size Perform the above operation on all M heads, sum all the loss function values, and obtain the final loss function value. ; Set a threshold for the function. If the cross-entropy loss function is less than the threshold, update and save the parameters of the multimodal fusion model to obtain the trained multimodal fusion model parameters. If the cross-entropy loss function is greater than or equal to the threshold, recalculate the cross-entropy loss function until the cross-entropy loss function is less than the threshold. The sports event commentary videos are input into the parameters of the trained multimodal fusion model to obtain optimized inference results, and the sports event commentary videos are edited based on the optimized inference results; The process of inputting sports event commentary videos into the parameters of a trained multimodal fusion model to obtain optimized inference results, and then editing the sports event commentary videos based on these optimized inference results, includes the following steps: The reasoning results Form a set ; for Remove all elements less than 0.6 to obtain the filtered set of recommended probabilities for editing positions. Use dynamic programming to find the second index. A contiguous subarray with the largest average is denoted as . The subscript satisfies ; The ordered list formed by the second subscript is the first... Recommended edit point index for each edit result Based on the index and the start and end times of each sentence in audio A, locate the position of this edited result in the original video. ; Take the original video from arrive Edit the portion to obtain the first... Editing results For all Perform the above operations to obtain M edited results. ; Update the process until the number of edited results meets the requirement; otherwise, update. ,exist Remove from middle All elements in the file, up to the number of edited results, meet the requirements.

2. The method for editing sports event commentary videos based on a multimodal fusion model according to claim 1, characterized in that, The process of inputting a dataset of sports event commentary video clips into a pre-established multimodal fusion model and outputting inference results includes the following steps: The original sports commentary videos V in the sports commentary video clip dataset D are converted from video to audio A; the audio A is then converted from speech to text to obtain the text of each sentence in the audio. Get the start time of the text of each sentence in audio A. And the end time of the text of each sentence in audio A. ; Extract the text modal information and visual modal information of each sentence; for each sentence's text... Extract the text modality information vector of this sentence. ;according to and In the original sports commentary video V, the set of images C is located, and the image modal information vector of the set of images C is extracted. ;Transform text modal information vectors and image modal information vector By concatenating these features to form an augmented matrix, we obtain the multimodal feature fusion vector of this sentence. ; Utilize Using a multi-head self-attention mechanism with M heads, we obtain a size of Multimodal clipping site recommendation tensor ,in Indicates in In the The output of each attention head is a vector, with dimension ... Same; M is the number of heads in the multi-head self-attention mechanism of the multimodal fusion model; Recommend tensors for multimodal clipping sites Each vector in All of them are transformed into a two-dimensional clipping point recommendation value vector through a fully connected layer. ; The recommended value vector of the clipping site Normalization is performed using a softmax layer to obtain the clip location recommendation probability vector. The first element of the clipping site recommendation probability vector represents the first... The edited result is a Chinese text fragment. The probability of being edited in is denoted as . ; Record the first The set of recommended probabilities for editing positions of different sizes. As a result of reasoning.

3. The method for editing sports event commentary videos based on a multimodal fusion model according to claim 2, characterized in that, The text modality information vector and image modal information vector The acquisition process: Text Inputting the pre-trained BERT model yields n vectors of dimension 768, denoted as... ; The resulting n vectors Temporal feature extraction is performed using LSTM, and the last hidden state of the LSTM is taken as the text modality information vector of the text. ; Uniformly sample t=2 from the image set C. ( ( ) images, forming an extracted image set. For each image in the IM, a CNN is used to extract image features, resulting in an image feature vector. ; The obtained t image feature vectors are used to extract temporal features using LSTM, and the last hidden state of the LSTM is taken as the image modality information vector of the image set. .

4. The method for editing sports event commentary videos based on a multimodal fusion model according to claim 3, characterized in that, The image feature vector is obtained by using CNN for image feature extraction. The process is as follows: For the input image IMG, use a size of 3 A convolution operation is performed on the IMG image using a kernel of size 3; then the convolution result is processed using a kernel of size 2. Max pooling is performed in layer 2; the result of max pooling is then transformed into a one-dimensional vector. A linear transformation is performed on the one-dimensional vector, followed by a ReLU activation function, to obtain the vector after feature extraction from the input image using a CNN, which serves as the image feature vector. .

5. The method for editing sports event commentary videos based on a multimodal fusion model according to claim 1, characterized in that, The result is obtained using dynamic programming algorithm. The process includes the following steps: Recommendation probability set at editing points Find the first index A contiguous subarray, denoted as Let the index of the first element of the current subarray in the original array be . Initialize the global optimal solution. , respectively represent the average value when the average value of the subarray is maximized, the starting index of the subarray in the original array, and the ending index of the subarray in the original array; Let array Indicates under consideration If it is definitely in a subarray, only consider Given the elements preceding and following it, find the maximum average value of the subarray; let the array... Indicates under consideration Given that the subarray must be within the subarray, find the length of the subarray whose average value is maximized. Use the following state transition equation to iterate through i from 2 to n. Iterate through the dp array from 1 to n, and find the index that makes the dp array the largest, denoted as r. if Greater than So, update Find the next index A contiguous subarray, denoted as Let the index of the first element of the current subarray in the original array be . Repeat the operation until the end of the array; Results The algorithm ends.

6. A sports event commentary video editing system based on a multimodal fusion model, characterized in that, include: The video processing module is used to receive sports event commentary videos and clips, and integrate the sports event commentary videos and clips to generate a sports event commentary video clip dataset; The process of integrating sports event commentary videos and clips to generate a sports event commentary video clip dataset includes the following steps: Initialize the dataset D, where D is an empty set; Search for sports commentary videos on video websites and mark them as V. Find one high-viewed clip from each of the M videos on different platforms and mark it as... ,in , where M is the number of heads in the multi-head self-attention mechanism of the multimodal fusion model; Get each Start time within V and end time ; All of V start time and end time Combine them to generate dataset entries. ; Input dataset entries DE into dataset D, repeat the above steps to continue obtaining dataset entries and inputting them into dataset D, until dataset D reaches the set number requirement, and output the sports event commentary video clip dataset D; The sports event commentary video clip dataset is input into a pre-established multimodal fusion model, and the inference result is output. The inference result is then used to calculate the cross-entropy loss function with the data in the sports event commentary video clip dataset. The process of deriving the cross-entropy loss function by using the obtained inference results and data from the sports event commentary video clip dataset includes the following steps: Forming the first The method for outputting the fact vector of the head is: find the data entry corresponding to video V in the dataset. and Initialize the fact vector The length is n, where n is the number of segments of the resulting text; for the th segment... A text segment, if the interval formed by the start and end times of the text is... and The subintervals of the formed interval, then ,otherwise ; use With the result of reasoning Calculate the cross-entropy loss function and sum all elements in the result to obtain the first... Loss function value of size Perform the above operation on all M heads, sum all the loss function values, and obtain the final loss function value. ; Set a threshold for the function. If the cross-entropy loss function is less than the threshold, update and save the parameters of the multimodal fusion model to obtain the trained multimodal fusion model parameters. If the cross-entropy loss function is greater than or equal to the threshold, recalculate the cross-entropy loss function until the cross-entropy loss function is less than the threshold. The video editing module is used to input sports event commentary videos into the parameters of the trained multimodal fusion model to obtain optimized inference results, and then edit the sports event commentary videos based on the optimized inference results; The process of inputting sports event commentary videos into the parameters of a trained multimodal fusion model to obtain optimized inference results, and then editing the sports event commentary videos based on these optimized inference results, includes the following steps: The reasoning results Form a set ; for Remove all elements less than 0.6 to obtain the filtered set of recommended probabilities for editing positions. Use dynamic programming to find the second index. A contiguous subarray with the largest average is denoted as . The subscript satisfies ; The ordered list formed by the second subscript is the first... Recommended editing point index for each editing result Based on the index and the start and end times of each sentence in audio A, locate the position of this edited result in the original video. ; from the original video arrive Edit the portion to obtain the first... Editing results For all Perform the above operations to obtain M edited results. ; Update the process until the number of edited results meets the requirement; otherwise, update. ,exist Remove from middle All elements in the file, up to the number of edited results, meet the requirements.

7. A device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When one or more of the programs are executed by one or more of the processors, the one or more of the processors implement the sports event commentary video editing method based on a multimodal fusion model as described in any one of claims 1-5.

Images