News video dynamic summary extraction method, device, medium and system

Abstract

The application discloses a news video dynamic summary extraction method, device, medium and system, belongs to the field of news video intelligent editing and news video summary extraction, and comprises the following steps: collecting news video materials and performing shot segment division based on news video visual features and audio features; designing a multi-modal semantic hidden feature representation space covering text, audio and images, and constructing a news video shot segment scoring model based on multi-modal semantic features; acquiring multi-modal hidden features of the news video and the news video shot segments, and training the news video shot segment scoring model based on multi-modal semantic features; and constructing a three-stage news video dynamic summary extraction system by using the trained news video shot segment scoring model based on multi-modal semantic features, so as to perform video summary extraction on the input news video. The application improves the implementability of the news video summary technology, and increases the richness of the video summary content on the premise of retaining core content.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of news video intelligent editing and news video summary extraction, and more particularly to a news video dynamic summary extraction method, device, medium and system. BACKGROUND

[0002] In recent years, with the vigorous development of the self-media industry, the explosive growth of short videos has provided a new path for the dissemination of news content. There are a large number of news videos on radio, television or the Internet, but these videos often have a long duration, which does not meet the requirements of efficiency and simplicity of short video dissemination in contemporary society. Naturally, it is necessary to edit and condense these original videos and extract more refined video summaries containing the core content of the original videos to meet the basic needs of short video dissemination. It is of great significance to use artificial intelligence technology to utilize the information of the three modalities of text, audio and video images possessed by news videos to improve the efficiency and quality of news short video production. This is an important practice for the transition of multimedia creation from traditional manual editing to total creation relying on intelligent tools.

[0003] Video is an important multimedia form that integrates text, audio and image multi-modal information. Existing video assisted editing techniques that rely on rules or single modal deep learning methods cannot comprehensively evaluate the representativeness and difference of video content segments, thereby affecting the auxiliary role of intelligent technology in video editing. In addition, news videos themselves contain coherent factual logic and core narrative subjects. Traditional video summary methods based on a single modality cannot accurately capture the context relationship in news videos, which easily leads to the loss of core content or the redundancy of content information. SUMMARY

[0004] The purpose of the present application is to overcome the shortcomings of the prior art and provide a news video dynamic summary extraction method, device, medium and system, which improves the implementability of news video summary technology and increases the richness of video summary content under the premise of preserving core content.

[0005] The purpose of the present application is achieved by the following scheme:

[0006] A news video dynamic summary extraction method comprises the following steps:

[0007] A. Collect news video materials and divide them into shot segments based on visual features and audio features of news videos;

[0008] B. Design a multi-modal semantic feature representation space covering text, audio and image and construct a news video shot segment scoring model based on multi-modal semantic features;

[0009] C. Obtain multimodal latent features of news videos and news video clips, and train a news video clip scoring model based on multimodal semantic features;

[0010] D. Construct a three-stage dynamic news video summarization system using a pre-trained news video clip scoring model based on multimodal semantic features to extract video summaries from input news videos.

[0011] Further, step A includes the following sub-steps:

[0012] A1. Obtain news video footage with a single video length greater than or equal to a set duration through internet news channels or radio / television news channels, and record the number of news video footage obtained as follows: ;

[0013] A2, for the nth news video clip, obtain the images corresponding to all frames in the video, denoted as... ,in This represents the total number of frames in the nth news video clip.

[0014] A3, extract the HSV color parameters from all frames of the nth news video clip, denoted as... ;

[0015] A4, calculate the HSV color parameter difference value between adjacent frames of the nth news video clip, denoted as . ;

[0016] A5, Set the color difference threshold If the difference in HSV color parameters between adjacent frames of the nth news video clip is greater than 1, then the difference in HSV color parameters between adjacent frames is greater than 1. Then it is taken as the visual candidate segmentation point, and the visual candidate segmentation point of the nth news video material is denoted as . ;

[0017] A6, calculate the audio energy at all visual candidate segmentation points for the nth news video clip, denoted as... ;

[0018] A7, Set Audio Energy Threshold If the nth news video material is in the nth... Audio energy at each visual candidate segmentation point Less than or equal to Then As the segmentation point for the nth news video clip, all segmentation points for the nth news video clip are denoted as . ;

[0019] A8, segment the nth news video clip according to the clip segmentation points, and finally obtain... A single shot, denoted as ;

[0020] A9, for all Each news video clip is processed sequentially to create images in sections A2-A8, resulting in a collection of all video clips' shots, denoted as _____. .

[0021] Furthermore, step B includes the following sub-steps:

[0022] B1, constructing a text semantic latent feature representation space;

[0023] B2, constructing an audio semantic latent feature representation space;

[0024] B3, constructing an image semantic latent feature representation space;

[0025] B4. Construct a news video clip scoring model based on multimodal semantic features.

[0026] Furthermore, step B1 involves constructing a text semantic latent feature representation space, including the following sub-steps:

[0027] B1-1, Construct the text semantic feature mapping matrix ;in, Represents the real number field. Given a positive integer, represent the number of all possible text characters. Given a positive integer, represents the dimension of the latent space of text semantic features;

[0028] B1-2, for any given text input ,in Given the number of characters in the text, Indicates the first Each character is determined by searching the text semantic feature mapping matrix. The mapping is represented as a vector, denoted as ,in for exist The corresponding dimension is ;

[0029] B1-3, Constructing a bidirectional long short-term memory network model ,by Vectorization representation As input, perform contextual feature fusion and output the fused result. Character hidden features ,in It is a dimension of ;

[0030] B1-4, Construct an attention evaluation network based on a fully connected feedforward network. ,by As input, their respective attention weights are calculated and denoted as follows: ,in [0,1] represents the first... Weights of latent features of each character;

[0031] B1-5, Calculate the given text semantic latent feature vectors The calculation formula is as follows:

[0032]

[0033] Furthermore, step B2 involves constructing the audio semantic latent feature representation space, including the following sub-steps:

[0034] B2-1, Set the audio feature extraction window dimension to... Set the number of Mel filters to ,in, Given a positive integer;

[0035] B2-2, for any given audio input, The total duration is denoted as Divide into equal parts A set of non-overlapping audio segments, denoted as The duration of the segment is recorded as ,satisfy ;

[0036] B2-3, based on a preset Mel filter, extracts log-Mel spectral features from P audio segments, denoted as... ,in Indicates the first Log-Mel spectrum characteristics of an audio segment;

[0037] B2-4, Constructing a two-layer convolutional neural network model ,by As input, the given audio input is calculated. Hidden features of fragments ,in Indicates the first The latent feature vectors of each audio segment;

[0038] B2-5, construct an attention evaluation network based on a fully connected feedforward network, to... As input, their respective attention weights are calculated and denoted as... ,in [0,1] represents the first... weights of the audio segment hidden features;

[0039] B2-6, calculating the semantic hidden feature vector of the given audio , the calculation formula is as follows:

[0040]

[0041] Further, the image semantic hidden feature representation space is constructed in step B3, including the following sub-steps:

[0042] B3-1, for any given video input , the number of image frames contained in the video is denoted as , all frames corresponding to the image are obtained, denoted as ;

[0043] B3-2, extract the RGB three-channel numerical value of , denoted as ;

[0044] B3-3, construct a basic image feature extraction convolutional neural network model , taking as input, calculate the image hidden feature, denoted as , wherein represents the hidden feature vector of the frame image;

[0045] B3-4, calculate the first-order, second-order, and fourth-order difference features , , , of the frame image respectively, the calculation formula is as follows:

[0046]

[0047]

[0048]

[0049] B3-5, for all frame images of the given video input , execute step B3-4 to obtain the difference feature sequence , wherein ; if ( ) exceeds the upper limit of the number of images , the cyclic mapping mechanism is adopted as follows:

[0050]

[0051] B3-6, Construct an attention evaluation network based on a fully connected feedforward network. ,by As input, calculate the first... 1st, 2nd, and 4th order differential attention weights for frame images ,in ;

[0052] B3-7, calculate the... Frame Image Differential Fusion Latent Features ;

[0053] B3-8, for the given video input All For each frame image, steps B3-6 and B3-7 are performed to obtain the differential fusion latent feature sequence of all frames. ;

[0054] B3-9, Constructing a bidirectional long short-term memory network model Differential fusion of latent feature sequences The context fusion latent feature sequence and differential fusion latent feature sequence of all frames are calculated. ,in Indicates the first Frame image context fusion latent feature vectors;

[0055] B3-10, Calculate the given video Image semantic latent feature vector The calculation formula is as follows:

[0056]

[0057] Furthermore, step B4 involves constructing a news video clip scoring model based on multimodal semantic features, including the following sub-steps:

[0058] B4-1, for the first Obtain the corresponding audio text from each news video clip. Audio ,video Perform steps B1, B2, and B3 respectively to obtain the text semantic latent feature vector. Audio semantic latent feature vector Image semantic latent feature vector ;

[0059] B4-2, for the first News video footage A scene clip Obtain the speech text corresponding to each segment. Audio ,video Then, steps B1, B2, and B3 are executed to obtain the text semantic latent feature vector, audio semantic latent feature vector, and image semantic latent feature vector for each segment, denoted as... ,in, , Indicates the first The latent semantic feature vectors of each text segment Indicates the first The audio semantic latent feature vector of each segment Indicates the first Image semantic latent feature vectors of each segment;

[0060] B4-3, set the... Binary decision vector of a news video clip ,in, It only has two values, 0 and 1. This indicates that the l-th segment was not selected as a summary. This indicates that the l-th segment was selected as the abstract;

[0061] B4-4, Calculate in the decision vector Below, the importance score of all selected footage segments from the nth video for summary purposes. as follows:

[0062]

[0063]

[0064]

[0065]

[0066] B4-5, Calculation in the decision vector Below, the novelty score of all selected footage segments from the nth video that were used as summaries. as follows:

[0067]

[0068]

[0069]

[0070]

[0071] in, ;

[0072] B4-6, Setting Importance Weights Set novelty weights Calculate in the decision vector Below, the video dynamic summary score of the nth video. ;

[0073] B4-7, Perform steps B4-1 to B4-6 on the collected N news videos to obtain the video dynamic summary score sequence for all N news videos under their respective decision vectors. .

[0074] Furthermore, step C includes the following sub-steps:

[0075] C1, the video dynamic summarization score sequence calculated based on step B4. ,by To optimize the objective function, the maximum likelihood method is used to calculate the optimization of the objective function with respect to learnable parameters. and decision vector sequence The gradient is denoted as ,in Models that are probability expectation functions and contain learnable parameters include: , , , , , , , ;

[0076] C2, Set the learning rate The gradient calculated based on C1 Update the learnable parameters and decision vector sequence as follows:

[0077]

[0078]

[0079] Where the function ;

[0080] C3, based on the updated learnable parameters and decision vector sequence A new video dynamic summarization score sequence is calculated based on step B4. ;

[0081] C4 sets the maximum number of iterations. Repeat steps C2 and C3 for all N collected news videos until the video dynamic summary score sequence no longer changes or the preset maximum number of iterations is reached, then stop training and obtain the trained model: , , , , 、 、 、 .

[0082] Further, step D comprises the following sub-steps:

[0083] D1, constructing and executing the news video shot segment splitting stage of the news video dynamic summary system;

[0084] D2, constructing and executing the multi-modal semantic hidden feature extraction and score calculation stage of the news video dynamic summary system;

[0085] D3, constructing and executing the video dynamic summary generation stage of the news video dynamic summary system.

[0086] Further, the news video shot segment splitting stage of the news video dynamic summary system constructed and executed in step D1 comprises the following sub-steps:

[0087] D1-1, given the input news video , sequentially execute steps A2 to A5 to obtain the visual candidate cut points, denoted as

[0088] D1-2, based on the candidate cut points sequentially execute steps A6 to A9 to obtain all shot segments of the news video , denoted as .

[0089] Further, the multi-modal semantic hidden feature extraction and score calculation stage of the news video dynamic summary system constructed and executed in step D2 comprises the following sub-steps:

[0090] D2-1, obtain the corresponding speech text , audio , video of the news video ;

[0091] D2-2, obtain the speech text, audio, video corresponding to all shot segments, denoted as ;

[0092] D2-3, for the speech text of , based on the trained text semantic feature mapping matrix , model , model , update the corresponding model parameters in B1-2 to B1-5 and execute sequentially to obtain the text semantic hidden feature vector of ;​

[0093] D2-4, for The audio, based on the trained model ,Model Update the corresponding model parameters in B2-1 to B2-5 and execute them sequentially to obtain the results. Audio semantic latent feature vector ;

[0094] D2-5, for The video is based on the trained model. ,Model ,Model Update the corresponding model parameters in B3-1 to B3-10 and execute them sequentially to obtain the results. Image semantic latent feature vector ;

[0095] D2-6, traverse sequentially all The audio, text, and video corresponding to each shot segment; the audio text for each segment is based on the trained text semantic feature mapping matrix. ,Model ,Model Update the corresponding model parameters in B1-2 to B1-5 and execute them sequentially to obtain the semantic latent feature vectors of the text; for the audio of each segment, use the trained model. ,Model Update the corresponding model parameters in B2-1 to B2-5 and execute them sequentially to obtain the audio semantic latent feature vector; for each video segment, based on the trained model... ,Model ,Model Update the corresponding model parameters in B3-1 to B3-10 and execute them sequentially to obtain the semantic latent feature vector of the image; the semantic latent features of all obtained segments are denoted as... ,in, Indicates the first The textual semantic latent feature vector of each shot segment, Indicates the first The audio semantic latent feature vector of each shot segment, Indicates the first The video semantic latent feature vector of each shot segment;

[0096] D2-7, Given a set of news videos Decision vector News videos acquired from D2-3 to D2-6 and news videos Based on the semantic latent features of all shot segments, steps B4-4 to B4-6 are performed to calculate the video dynamic summary score under the current decision vector. ;

[0097] D2-8, Traverse news videos Of all possible values ​​for the decision vector, the decision vector corresponding to the video dynamic summary score is selected as the optimal decision vector, denoted as . ;

[0098] D2-9, Calculating News Videos Ratings for all footage clips ,in, The calculation formula is as follows:

[0099]

[0100]

[0101]

[0102] Furthermore, step D3, which involves building and executing the news video dynamic summarization system, includes the following sub-steps:

[0103] D3-1, Set up news video Video summary reference duration Video summary segment sequence number set Total duration of video summary clips ;

[0104] D3-2, the videos are ranked from highest to lowest based on the score of each shot segment. The shot clips are sorted, and the sorted shot clip numbers are denoted as follows: The duration of each segment is recorded as follows: ;

[0105] D3-3, iterate through the sorted shot sequence numbers in sequence, if and Then the first Each shot clip was selected as a summary shot clip, and the update was performed. , Obtain the final set of selected abstract shot fragment numbers. M represents a total of One segment was selected;

[0106] D3-4, for Sort the clips to obtain a sorted set of clip numbers. ;

[0107] D3-5, selecting the video segment corresponding to the ordered lens segment number set , merging in order to obtain the final news video dynamic summary.

[0108] A news video dynamic summary extraction device, the computer device comprising a processor and a memory, the memory having stored therein a computer program, when the computer program is loaded and executed by the processor, the method of any one of the above.

[0109] A readable storage medium, in which a computer program is stored, the computer program is loaded and executed by the processor, the method of any one of the above.

[0110] A news video dynamic summary extraction system comprising a news video dynamic summary extraction device as described above.

[0111] The beneficial effects of the present application include:

[0112] The technical scheme of the embodiment of the present application converts text, audio, and video images into latent semantic features, and selects video summary segments by fusing multi-modal features. In the construction of image latent semantic features, a downward feature sharing mechanism using the difference idea is used, and an unsupervised method is used for model training, which improves the implementability of the news video summary technology and increases the richness of the video summary content on the premise of retaining the core content. BRIEF DESCRIPTION OF DRAWINGS

[0113] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiment or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0114] Figure 1 The flowchart of the news video dynamic summary extraction method based on multi-modal semantic features of the embodiment of the present application.

[0115] Figure 2 The principle diagram of the news video dynamic summary extraction method based on multi-modal semantic features of the embodiment of the present application and the three-stage news video dynamic summary extraction system. DETAILED DESCRIPTION

[0116] All features disclosed in all embodiments in the present specification, or all steps in the methods or processes impliedly disclosed, can be combined and / or extended, replaced, in any manner, except for mutually exclusive features and / or steps.

[0117] AsFigure 1 As shown, the embodiment proposes a news video dynamic summary extraction method based on multi-modal semantic features, including the following steps:

[0118] A, collect news video materials and cut the shot segments based on the visual features and audio features of the news video;

[0119] B, design a multi-modal semantic hidden feature representation space covering text, audio, and image, and construct a news video shot segment scoring model based on multi-modal semantic features;

[0120] C, obtain the multi-modal hidden features of the news video and the news video shot segments, and train the news video shot segment scoring model based on multi-modal semantic features;

[0121] D, use the trained news video shot segment scoring model based on multi-modal semantic features to construct a three-stage news video dynamic summary extraction system to extract video summaries from input news videos.

[0122] In some embodiments, step A includes the following sub-steps:

[0123] A1, obtain news video materials with a single video duration greater than or equal to 10 minutes through Internet news programs or radio and television news programs, and record the number of obtained video materials as ;

[0124] A2, obtain the images corresponding to all frames in the nth news video material, denoted as , where is the total number of frames of the nth news video material;

[0125] A3, extract the HSV color parameters of all frame images of the nth news video material, denoted as ;

[0126] A4, calculate the HSV color parameter difference value of the adjacent frame images of the nth news video material, denoted as , and the formula is as follows: ;

[0127] A5, set the color difference threshold ; if the HSV color parameter difference value of the adjacent frame images of the nth news video material is greater than , it is taken as a visual candidate cut point, and the visual candidate cut point of the nth news video material is denoted as ;

[0128] A6, calculate the audio energy (decibels) at all visual candidate cut points of the nth news video material, denoted as ;

[0129] A7, Set Audio Energy Threshold [8, 32]; If the nth news video material is in the nth... Audio energy at each visual candidate segmentation point Less than or equal to Then As the segmentation point for the nth news video clip, all segmentation points for the nth news video clip are denoted as . ;

[0130] A8, segment the nth news video clip according to the clip segmentation points, and finally obtain... Each shot clip is recorded as ;

[0131] A9, for all Each news video clip is processed sequentially to create images in sections A2-A8, resulting in a collection of all video clips' shots, denoted as _____. ;

[0132] In some embodiments, step B includes the following sub-steps:

[0133] B1, Constructing the text semantic latent feature representation space:

[0134] B1-1, Construct the text semantic feature mapping matrix ;in, Represents the real number field. Given a positive integer, represent the number of all possible text characters. Let a given positive integer represent the dimension of the latent space of the text semantic features;

[0135] B1-2, for any given text input ,in Given the number of characters in the text, Indicates the first Each character is determined by searching the text semantic feature mapping matrix. The mapping is represented as a vector, denoted as ,in for exist The corresponding dimension is ;

[0136] B1-3, Constructing a bidirectional long short-term memory network model ,by Vectorization representation As input, perform contextual feature fusion and output the fused result. Character hidden features ,in is a vector with dimension ;

[0137] B1-4, constructing an attention evaluation network based on a fully connected feedforward network , taking as input, respectively calculating the respective attention weight , where [0, 1] represents the weight of the th character hidden feature;

[0138] B1-5, calculating the semantic hidden feature vector of a given text , the calculation formula is as follows:

[0139]

[0140] B2, constructing an audio semantic hidden feature representation space:

[0141] B2-1, setting the audio feature extraction window dimension to , and setting the number of mel filters to , where is a given positive integer;

[0142] B2-2, for any given audio input , , the total duration is denoted as , which is evenly divided into non-overlapping audio segments, denoted as , and the segment duration is denoted as , satisfying ;

[0143] B2-3, based on the preset mel filter, the log mel spectrum features of the P audio segments are extracted, denoted as , where represents the log mel spectrum feature of the th audio segment;

[0144] B2-3, constructing a double-layer convolutional neural network model , taking as input, calculating the segment hidden feature of the given audio input , where represents the hidden feature vector of the th audio segment;

[0145] B2-4, constructing an attention evaluation network based on a fully connected feedforward network , taking ​As input, the respective attention weight is calculated as where [0,1] represents the weight of the th audio segment hidden feature;

[0146] B2-5, the semantic hidden feature vector of the given audio is calculated, and the calculation formula is as follows:

[0147]

[0148] B3, the image semantic hidden feature representation space is constructed:

[0149] B3-1, for any given video input , the number of image frames contained in the video is denoted as , and the images corresponding to all frames are obtained, denoted as ;

[0150] B3-2, the RGB three-channel numerical value of is extracted, denoted as ;

[0151] B3-3, a basic image feature extraction convolutional neural network model is constructed , taking as input, the image hidden feature is calculated, denoted as , where represents the hidden feature vector of the th frame image;

[0152] B3-4, the first-order, second-order, and fourth-order difference features of the th frame image are calculated respectively , , , and the calculation formula is as follows:

[0153]

[0154]

[0155]

[0156] B3-5, for all frame images of the given video input , step B3-4 is performed to obtain the difference feature sequence , where ; if ( ) exceeds the upper limit of the number of images , a cyclic mapping mechanism is adopted as follows: ​

[0157]

[0158] B3-6, constructing an attention evaluation network based on a fully connected feedforward network , taking as input, to calculate the 1st, 2nd, and 4th order difference attention weights of the i-th frame image , wherein ; ;

[0159] B3-7, calculating the difference fusion hidden feature of the i-th frame image ; ;

[0160] B3-8, for all the frame images of a given video input , performing steps B3-6 and B3-7 to obtain the difference fusion hidden feature sequence of all the frames ; ;

[0161] B3-9, constructing a bidirectional long short-term memory network model , taking the difference fusion hidden feature sequence , to calculate the context fusion hidden feature sequence of all the frames , wherein represents the context fusion hidden feature vector of the i-th frame image ;

[0162] B3-10, calculating the image semantic hidden feature vector of a given video , the calculation formula is as follows:

[0163]

[0164] B4, constructing a news video shot segment scoring model based on multi-modal semantic features:

[0165] B4-1, for the i-th news video material, obtaining its corresponding speech text , audio , and video ; performing steps B1, B2, and B3 respectively to obtain the text semantic hidden feature vector , the audio semantic hidden feature vector , and the image semantic hidden feature vector ;

[0166] B4-2, for the j-th shot segment of the i-th news video material , obtaining the speech text corresponding to each segment ; ​​​ Audio ,video Then, steps B1, B2, and B3 are executed to obtain the text semantic latent feature vector, audio semantic latent feature vector, and image semantic latent feature vector for each segment, denoted as... ,in, , This represents the latent semantic feature vector of the l-th segment. The audio segment representing the l-th segment Semantic latent feature vectors This represents the semantic latent feature vector of the l-th segment of the image;

[0167] B4-3, set the... Binary decision vector of a news video clip ,in, It only has two values, 0 and 1. This indicates that the l-th segment was not selected as a summary. This indicates that the l-th segment was selected as the abstract;

[0168] B4-4, Calculate in the decision vector Below, the importance score of all selected footage segments from the nth video for summary purposes. as follows:

[0169]

[0170]

[0171]

[0172]

[0173] B4-5, Calculation in the decision vector Below, the novelty score of all selected footage segments from the nth video that were used as summaries. as follows:

[0174]

[0175]

[0176]

[0177]

[0178] in, ;

[0179] B4-6, Setting Importance Weights Set novelty weights , the video dynamic summary score of the nth video under the decision vector Next, the video dynamic summary score of the nth video under the decision vector ;

[0180] B4-7, performing steps B4-1 to B4-6 on the collected N news videos to obtain the video dynamic summary score sequence of all N news videos under the respective decision vector ;

[0181] In some embodiments, step C comprises the following sub-steps:

[0182] C1, based on the video dynamic summary score sequence calculated in step B4 , taking the maximum likelihood method to calculate the gradient of the optimization objective function with respect to the learnable parameters and the decision vector sequence , denoted as , where is the probability expectation function, and the model containing the learnable parameters is: , , , , , , , , ;

[0183] C2, setting the learning rate , based on the gradient calculated in C1 , updating the learnable parameters and the decision vector sequence as follows:

[0184]

[0185]

[0186] where the function ;

[0187] C3, based on the updated learnable parameters and the decision vector sequence , calculating a new video dynamic summary score sequence based on step B4 ;

[0188] C4, setting the maximum number of iterations , repeating steps C2 and C3 for all collected N news videos until the video dynamic summary score sequence no longer changes or the maximum number of iterations is reached, then stopping training and obtaining the trained model: , , , 、 、 、 、 ;

[0189] In some embodiments, as shown in FIG. 8, step D comprises the following sub-steps: Figure 2

[0190] D1, constructing and executing the news video shot segment splitting stage of the news video dynamic summarization system:

[0191] D1-1, given the input news video , sequentially execute steps A2 to A5 to obtain the visual candidate cut points, denoted as

[0192] D1-2, based on the candidate cut points sequentially execute steps A6 to A9 to obtain all shot segments of the news video , denoted as ;

[0193] D2, constructing and executing the multi-modal semantic hidden feature extraction and score calculation stage of the news video dynamic summarization system:

[0194] D2-1, obtaining the corresponding speech text , audio , video of the news video ;

[0195] D2-2, obtaining the speech text, audio, and video corresponding to all shot segments, denoted as ;

[0196] D2-3, for the speech text of , based on the trained text semantic feature mapping matrix , model , model , update the corresponding model parameters in B1-2 to B1-5 and sequentially execute to obtain the text semantic hidden feature vector of ;

[0197] D2-4, for the audio of , based on the trained model , model , update the corresponding model parameters in B2-1 to B2-5 and sequentially execute to obtain the audio semantic hidden feature vector of ;

[0198] ​​D2-5, for the video, based on the trained model , model , model , model , update the corresponding model parameters in B3-1 to B3-10 and execute in turn, obtain the image semantic hidden feature vector ;

[0199] D2-6, sequentially traverse all the speech text, audio, video corresponding to each shot segment; for each segment of speech text based on the trained text semantic feature mapping matrix , model , model , update the corresponding model parameters in B1-2 to B1-5 and execute in turn, obtain the text semantic hidden feature vector; for each segment of audio based on the trained model , model , update the corresponding model parameters in B2-1 to B2-5 and execute in turn, obtain the audio semantic hidden feature vector; for each segment of video based on the trained model , model , model , update the corresponding model parameters in B3-1 to B3-10 and execute in turn, obtain the image semantic hidden feature vector; the semantic hidden feature of all segments obtained is recorded as , wherein represents the text semantic hidden feature vector of the th shot segment, represents the audio semantic hidden feature vector of the th shot segment, represents the video semantic hidden feature vector of the th shot segment.

[0200] D2-7, given a set of news videos determination vector , based on the semantic hidden features of all shot segments of the news video and the news video obtained by D2-3 to D2-6, execute steps B4-4 to B4-6 to calculate the video dynamic summary score under the current determination vector ;

[0201] D2-8, traverse all possible values of the determination vector of the news video , select the determination vector corresponding to the video dynamic summary score as the optimal determination vector, recorded as ;

[0202] D2-9, calculate news video Score of all shot segments Wherein, The calculation formula is as follows:

[0203]

[0204]

[0205]

[0206] D3, video dynamic summary generation phase of constructing and executing news video dynamic summary system:

[0207] D3-1, set the video summary reference duration of news video , the video summary segment serial number set (initially empty set), the video summary segment cumulative duration (initially 0);

[0208] D3-2, sort the shot segments of the video in descending order of shot segment score, and the sorted shot segment serial number is recorded as , and the segment duration is recorded as ;

[0209] D3-3, traverse the sorted shot segment serial number from left to right, if and , the first shot segment is selected as the selected summary shot segment, and , are updated; obtain the final selected summary shot segment serial number set (here it is assumed that a total of segments are selected);

[0210] D3-4, sort in ascending order of natural number to obtain the sorted shot segment serial number set ;

[0211] D3-5, select the corresponding video segments based on the sorted shot segment serial number set , and merge them in order to obtain the dynamic summary of the final news video .

[0212] ​Up to now, the steps A-D are summarized, so that the video dynamic summary containing the main content of the video can be automatically extracted from the given news video. Moreover, the text, audio and video image are converted into latent semantic features, and the multi-modal features are fused to select the video summary segment. In the construction of the image latent semantic feature, the downward feature sharing mechanism of the difference idea is used, and the unsupervised method is used for model training, so that the implementability of the news video summary technology is improved, and the richness of the video summary content is increased on the premise of retaining the core content.

[0213] It should be noted that the following embodiments can be combined and / or extended, replaced, in any logical manner within the scope of protection defined in the claims of the present application, such as disclosed technical principles, disclosed technical features or implied disclosed technical features.

[0214] Embodiment 1

[0215] A news video dynamic summary extraction method comprises the following steps:

[0216] A, collect news video materials and divide the shot segments based on the visual features and audio features of the news video;

[0217] B, design a multi-modal semantic latent feature representation space covering text, audio and image, and construct a news video shot segment scoring model based on multi-modal semantic features;

[0218] C, obtain the multi-modal latent features of the news video and the news video shot segments, and train the news video shot segment scoring model based on multi-modal semantic features;

[0219] D, use the trained news video shot segment scoring model based on multi-modal semantic features to construct a three-stage news video dynamic summary extraction system to extract video summary from the input news video.

[0220] Embodiment 2

[0221] On the basis of embodiment 1, step A comprises the following sub-steps:

[0222] A1, obtain news video materials with a single video duration greater than or equal to a set duration through an Internet news column or a radio and television news column, and record the number of obtained news video materials as ;

[0223] A2, obtain the images corresponding to all frames in the nth news video material, denoted as , wherein is the total frame number of the nth news video material;

[0224] A3, extract HSV color parameters of all frame images of the nth news video material, denoted as ;

[0225] A4, calculate the HSV color parameter difference value of adjacent frame images of the nth news video material, denoted as ;

[0226] A5, set a color difference threshold ; if the HSV color parameter difference value of adjacent frame images of the nth news video material is greater than , it is regarded as a visual candidate cut point, and the visual candidate cut point of the nth news video material is denoted as ;

[0227] A6, calculate the audio energy at all visual candidate cut points of the nth news video material, denoted as ;

[0228] A7, set an audio energy threshold ; if the audio energy of the nth news video material at the mth visual candidate cut point is less than or equal to , the mth visual candidate cut point is regarded as a shot segment cut point of the nth news video material, and all shot segment cut points of the nth news video material are denoted as ; ;

[0229] A8, according to the shot segment cut points, the nth news video material is cut, and finally shot segments are obtained, denoted as ;

[0230] A9, for all news video materials, A2-A8 are sequentially completed, and the shot segment set of all video materials is obtained, denoted as .

[0231] Embodiment 3

[0232] On the basis of embodiment 1, step B includes the following sub-steps:

[0233] B1, construct a text semantic hidden feature representation space;

[0234] B2, construct an audio semantic hidden feature representation space;

[0235] B3, construct an image semantic hidden feature representation space;

[0236] B4, construct a news video shot segment scoring model based on multi-modal semantic features.

[0237] Embodiment 4​​

[0238] On the basis of embodiment 3, the text semantic hidden feature representation space is constructed in step B1, including the following sub-steps:

[0239] B1-1, constructing a text semantic feature mapping matrix ; wherein, represents the real number field, is a given positive integer, representing the number of all possible text characters, is a given positive integer, representing the text semantic feature hidden space dimension;

[0240] B1-2, for any given text input , wherein is the number of characters of the given text, represents the th character, which is mapped to the vectorized representation by looking up the text semantic feature mapping matrix, denoted as , wherein is the th character, the corresponding dimension in is vector;

[0241] B1-3, constructing a bidirectional long short-term memory network model , taking the vectorized representation as input, performing context feature fusion and outputting the fused character hidden feature , wherein is a vector with a dimension of ;

[0242] B1-4, constructing an attention evaluation network based on a fully connected feedforward network , taking as input, respectively calculating the respective attention weights denoted as , wherein [0,1] represents the weight of the th character hidden feature;

[0243] B1-5, calculating the semantic hidden feature vector of the given text , the calculation formula is as follows:

[0244]

[0245] Embodiment 5

[0246] On the basis of embodiment 3, the audio semantic hidden feature representation space is constructed in step B2, including the following sub-steps:​​

[0247] B2-1, set the audio feature extraction window dimension as , set the number of mel filters as , wherein, is a given positive integer;

[0248] B2-2, for any given audio input , , the total duration is denoted as , which is evenly divided into non-overlapping audio segments, denoted as , and the segment duration is denoted as , satisfying ;

[0249] B2-3, based on the preset mel filter, the log mel spectrum features of the P audio segments are extracted, denoted as , wherein represents the log mel spectrum features of the th audio segment;

[0250] B2-4, a double-layer convolutional neural network model is constructed , taking as input, the segment hidden features of the given audio input are calculated, wherein represents the hidden feature vector of the th audio segment;

[0251] B2-5, an attention evaluation network based on a fully connected feedforward network is constructed , taking as input, the respective attention weights are calculated, denoted as , wherein [0, 1] represents the weight of the th audio segment hidden feature;

[0252] B2-6, the semantic hidden feature vector of the given audio is calculated, and the calculation formula is as follows:

[0253]

[0254] Embodiment 6

[0255] On the basis of embodiment 3, the image semantic hidden feature representation space is constructed in step B3, including the following sub-steps:

[0256] B3-1, for any given video input , the number of image frames contained in the video is denoted as , obtain the image corresponding to all frames, denoted as ;

[0257] B3-2, extract the RGB three-channel numerical value of , denoted as ;

[0258] B3-3, construct a basic image feature extraction convolutional neural network model , take as input, calculate the image hidden feature, denoted as , wherein represents the hidden feature vector of the frame image;

[0259] B3-4, calculate the first-order, second-order, and fourth-order difference features , , , of the frame image respectively, and the calculation formula is as follows:

[0260]

[0261]

[0262]

[0263] B3-5, for all frame images of the given video input , perform step B3-4 to obtain the difference feature sequence , wherein ; if ( ) exceeds the upper limit of the number of images , a cyclic mapping mechanism is adopted as follows:

[0264]

[0265] B3-6, construct an attention evaluation network based on a fully connected feedforward network , take as input, and calculate the first-order, second-order, and fourth-order difference attention weights of the frame image , wherein ;

[0266] B3-7, calculate the difference fusion hidden feature of the frame image ;

[0267] B3-8, for all frame images of the given video input For each frame image, steps B3-6 and B3-7 are performed to obtain the differential fusion latent feature sequence of all frames. ;

[0268] B3-9, Constructing a bidirectional long short-term memory network model Differential fusion of latent feature sequences The context fusion latent feature sequence and differential fusion latent feature sequence of all frames are calculated. ,in Indicates the first Frame image context fusion latent feature vectors;

[0269] B3-10, Calculate the given video Image semantic latent feature vector The calculation formula is as follows:

[0270]

[0271] Example 7

[0272] Based on Example 3, step B4 constructs a news video clip scoring model based on multimodal semantic features, including the following sub-steps:

[0273] B4-1, for the first Obtain the corresponding audio text from each news video clip. Audio ,video Perform steps B1, B2, and B3 respectively to obtain the text semantic latent feature vector. Audio semantic latent feature vector Image semantic latent feature vector ;

[0274] B4-2, for the first News video footage A scene clip Obtain the speech text corresponding to each segment. Audio ,video Then, steps B1, B2, and B3 are executed to obtain the text semantic latent feature vector, audio semantic latent feature vector, and image semantic latent feature vector for each segment, denoted as... ,in, , Indicates the first The latent semantic feature vectors of each text segment Indicates the first The audio semantic latent feature vector of each segment Indicates the first Image semantic latent feature vectors of each segment;

[0275] B4-3, set the... Binary decision vector of a news video clip ,in, It only has two values, 0 and 1. This indicates that the l-th segment was not selected as a summary. This indicates that the l-th segment was selected as the abstract;

[0276] B4-4, Calculate in the decision vector Below, the importance score of all selected footage segments from the nth video for summary purposes. as follows:

[0277]

[0278]

[0279]

[0280]

[0281] B4-5, Calculation in the decision vector Below, the novelty score of all selected footage segments from the nth video that were used as summaries. as follows:

[0282]

[0283]

[0284]

[0285]

[0286] in, ;

[0287] B4-6, Setting Importance Weights Set novelty weights Calculate in the decision vector Below, the video dynamic summary score of the nth video. ;

[0288] B4-7, Perform steps B4-1 to B4-6 on the collected N news videos to obtain the video dynamic summary score sequence for all N news videos under their respective decision vectors. .

[0289] Example 8

[0290] Based on Example 3, step C includes the following sub-steps:

[0291] C1, calculate the video dynamic summary score sequence based on step B4 , to the optimization objective function, using the maximum likelihood method to calculate the gradient of the optimization objective function to the learnable parameters and the decision vector sequence , denoted as , where is the probability expectation function, and the model containing the learnable parameters is: , , , , , , , ;

[0292] C2, set the learning rate , based on the gradient calculated in C1 , update the learnable parameters and the decision vector sequence as follows:

[0293]

[0294]

[0295] where the function ;

[0296] C3, based on the updated learnable parameters and the decision vector sequence , calculate a new video dynamic summary score sequence based on step B4;

[0297] C4, set the maximum number of iterations , repeat steps C2 and C3 for all collected N news videos until the video dynamic summary score sequence no longer changes or reaches the preset maximum number of iterations, then stop training and obtain the trained model: , , , , , , , .

[0298] Embodiment 9

[0299] Based on embodiment 1, step D includes the following sub-steps:

[0300] D1, constructing and executing the news video shot segment splitting stage of the news video dynamic summary system;

[0301] D2, constructing and executing the multi-modal semantic hidden feature extraction and score calculation stage of the news video dynamic summary system;

[0302] D3, constructing and executing the video dynamic summary generation stage of the news video dynamic summary system.

[0303] Embodiment 10

[0304] On the basis of Embodiment 9, the news video shot segment splitting stage of the news video dynamic summary system constructed and executed in step D1 includes the following sub-steps:

[0305] D1-1, given the input news video , sequentially execute steps A2 to A5 to obtain the visual candidate split points, denoted as

[0306] D1-2, based on the candidate split points sequentially execute steps A6 to A9 to obtain all shot segments of the news video , denoted as .

[0307] Embodiment 11

[0308] On the basis of Embodiment 9, the multi-modal semantic hidden feature extraction and score calculation stage of the news video dynamic summary system constructed and executed in step D2 includes the following sub-steps:

[0309] D2-1, obtain the news video corresponding speech text , audio , video ;

[0310] D2-2, obtain all shot segments corresponding speech text, audio, video, denoted as ;

[0311] D2-3, for the speech text of , based on the trained text semantic feature mapping matrix , model , model , update the corresponding model parameters in B1-2 to B1-5 and execute sequentially to obtain the text semantic hidden feature vector of ;

[0312] D2-4, for the audio of The audio, based on the trained model ,Model Update the corresponding model parameters in B2-1 to B2-5 and execute them sequentially to obtain the results. Audio semantic latent feature vector ;

[0313] D2-5, for The video is based on the trained model. ,Model ,Model Update the corresponding model parameters in B3-1 to B3-10 and execute them sequentially to obtain the results. Image semantic latent feature vector ;

[0314] D2-6, traverse sequentially all The audio, text, and video corresponding to each shot segment; the audio text for each segment is based on the trained text semantic feature mapping matrix. ,Model ,Model Update the corresponding model parameters in B1-2 to B1-5 and execute them sequentially to obtain the semantic latent feature vectors of the text; for the audio of each segment, use the trained model. ,Model Update the corresponding model parameters in B2-1 to B2-5 and execute them sequentially to obtain the audio semantic latent feature vector; for each video segment, based on the trained model... ,Model ,Model Update the corresponding model parameters in B3-1 to B3-10 and execute them sequentially to obtain the semantic latent feature vector of the image; the semantic latent features of all obtained segments are denoted as... ,in, Indicates the first The textual semantic latent feature vector of each shot segment, Indicates the first The audio semantic latent feature vector of each shot segment, Indicates the first The video semantic latent feature vector of each shot segment;

[0315] D2-7, Given a set of news videos Decision vector News videos acquired from D2-3 to D2-6 and news videos Based on the semantic latent features of all shot segments, steps B4-4 to B4-6 are performed to calculate the video dynamic summary score under the current decision vector. ;

[0316] D2-8, traversing the news video All possible value of the decision vector, select the corresponding video dynamic summary score of the decision vector as the optimal decision vector, denoted as ;

[0317] D2-9, calculating the news video The score of all shot segments , wherein, The calculation formula is as follows:

[0318]

[0319]

[0320]

[0321] Embodiment 12

[0322] On the basis of embodiment 9, the video dynamic summary generation phase of the news video dynamic summary system constructed and executed in step D3 includes the following sub-steps:

[0323] D3-1, setting the video summary reference duration of the news video , the video summary segment serial number set , the video summary segment cumulative duration ;

[0324] D3-2, sorting the shot segments of the video in the order of high to low according to the shot segment score, the sorted shot segment serial number is denoted as , and the segment duration is denoted as ;

[0325] D3-3, traversing the sorted shot segment serial number in sequence, if and , the first shot segment is selected as the selected summary shot segment, and , are updated; the final selected summary shot segment serial number set is obtained, and M represents that a total of segments are selected;

[0326] D3-4, sorting to obtain the sorted shot segment serial number set ;

[0327] D3-5, selecting the corresponding video segment based on the sorted shot segment serial number set ​, the merging is performed in order to obtain a final news video a dynamic summary of the news video.

[0328] Embodiment 13

[0329] A news video dynamic summary extraction device, comprising a processor and a memory, the memory storing a computer program, when the computer program is loaded and executed by the processor, the method of any one of the above is executed.

[0330] Embodiment 14

[0331] A readable storage medium, a computer program is stored in the readable storage medium, the computer program is loaded and executed by the processor, the method of any one of the embodiments 1~embodiment 12 is executed.

[0332] Embodiment 15

[0333] A news video dynamic summary extraction system, comprising the news video dynamic summary extraction device of embodiment 13.

[0334] The embodiment of the application utilizes a neural network model to deeply fuse semantic information of three dimensions of text, audio and image, scores the importance and novelty of news video shot segments based on an unsupervised learning method, and constructs a video dynamic summary extraction system in combination with short video production field knowledge, so as to realize a high-quality news video dynamic summary extraction technology for new media field.

[0335] The units described in the embodiments of the application can be implemented in the form of software, or can be implemented in the form of hardware, and the described units can also be arranged in a processor. In some cases, the names of these units do not constitute a limitation on the units themselves.

[0336] According to an aspect of the embodiment of the application, a computer program product or computer program is provided, which includes computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method provided in the various optional implementation manners.

[0337] As another aspect, the embodiment of the application further provides a computer readable medium, which can be included in the electronic device described in the above embodiments; or can exist independently without being assembled into the electronic device. The above computer readable medium carries one or more programs, when the one or more programs are executed by the electronic device, the electronic device implements the method described in the above embodiments.

[0338] The part of the present application not involved is the same as the prior art or can be realized by using the prior art.

[0339] The technical solution described above is only one embodiment of the present application. Based on the application method and principle disclosed by the present application, various types of improvements or modifications can be easily made by those skilled in the art without being limited to the method described in the above specific embodiment of the present application. Therefore, the above-described method is only preferred and does not have a limiting meaning.

[0340] In addition to the above examples, those skilled in the art can obtain other embodiments by making changes based on the above disclosure or using knowledge or technology in related fields. The features of each embodiment can be interchanged or replaced. Changes and variations made by those skilled in the art do not deviate from the spirit and scope of the present application, and should be within the protection scope of the claims of the present application.

Claims

1. A method for dynamic summarization of news video, characterized in that, The method comprises the following steps: A. Collecting news video materials and performing shot segmentation based on visual and audio features of the news video; B. Designing a multi-modal semantic hidden feature representation space covering text, audio, and image, and constructing a news video shot scoring model based on multi-modal semantic features; step B comprises the following sub-steps: B1, constructing a text semantic hidden feature representation space; B2, constructing an audio semantic hidden feature representation space; B3, constructing an image semantic hidden feature representation space; B4, constructing a news video shot scoring model based on multi-modal semantic features; In step B4, the news video shot scoring model based on multi-modal semantic features is constructed, comprising the following sub-steps: B4-1, for the first news video material to obtain its corresponding voice text , audio , video ; respectively execute steps B1, B2, B3, obtain text semantic hidden feature vector , audio semantic hidden feature vector , image semantic hidden feature vector ; B4-2, for the first news video material lens segment , respectively, obtain the voice text corresponding to each segment , audio , video , and execute steps B1, B2, B3 to obtain the text semantic hidden feature vector, audio semantic hidden feature vector and image semantic hidden feature vector of each segment, denoted as , wherein , The text semantic hidden feature vector of the first segment is represented as The audio semantic hidden feature vector of the first segment is represented as The image semantic hidden feature vector of the first segment is represented as B4-3, set the second binary determination vector of the news video material wherein, only two values of 0 and 1, the lth segment is not selected as a summary, the lth segment is selected as a summary; B4-4, compute the importance score of all selected shot segments for the summary Next, the nth video all selected as the importance score of the summary of the shot segment As follows: B4-5, compute novelty score for all selected shots in the decision vector Next, the novelty score of the nth video for all selected shot segments as the summary As follows: wherein ; B4-6, set importance weight , set novelty weight , calculate video dynamic summary score of nth video under decision vector ;​ B4-7. performing steps B4-1 to B4-6 on the collected N news videos to obtain a sequence of video dynamic summary scores of all N news videos under respective decision vectors ; C. Acquiring multi-modal hidden features of the news video and the news video shot, and training the news video shot scoring model based on multi-modal semantic features; D. Using the trained news video shot scoring model based on multi-modal semantic features to construct a three-stage news video dynamic summary extraction system to extract a video summary from an input news video.

2. The news video dynamic summary extraction method according to claim 1, wherein step A comprises the following sub-steps:

3. The news video dynamic summary extraction method according to claim 1, wherein step B1 comprises the following sub-steps: A1, obtaining news video materials with a single video duration greater than or equal to a set duration through an Internet news column or a broadcast television news column, recording the number of the obtained news video materials as ; A2, obtaining images corresponding to all frames in the video of the nth news video material, denoted as wherein is the total number of frames of the nth news video material; A3, extract HSV color parameters of all frame images of the nth news video material, denoted as ; A4, calculate the HSV color parameter difference value of the adjacent frame image of the nth news video material, denoted as ; A5, setting color difference threshold If the HSV color parameter difference value of the adjacent frame image of the nth news video material is greater than , it is regarded as a visual candidate cut point, and the visual candidate cut point of the nth news video material is recorded as ; A6, calculate the audio energy at all visual candidate cut points for the nth news video material, denoted as ; A7, Set Audio Energy Threshold If the nth news video material is in the nth... Audio energy at each visual candidate segmentation point Less than or equal to Then As the segmentation point for the nth news video clip, all segmentation points for the nth news video clip are denoted as . ; A8, according to the lens segment cut point, the n-th news video material is cut, and finally the lens segment is obtained, recorded as ; A9, for all news video materials, sequentially complete A2-A8, get all video material shot fragment set, recorded as .

4. The news video dynamic summary extraction method according to claim 1, wherein step B2 comprises the following sub-steps:

5. The news video dynamic summary extraction method according to claim 1, wherein step B3 comprises the following sub-steps: B1-1, constructing a text semantic feature mapping matrix ; wherein, denotes the real number field, is a given positive integer, denotes the number of all possible text characters, is a given positive integer, denotes the text semantic feature latent space dimension; B1-2, for any given text input wherein is the number of characters of the given text, represents the character, which is mapped to a vectorized representation by looking up the text semantic feature mapping matrix, denoted as wherein is the vectorized representation of the thcharacter in the text, with the corresponding dimension in being the vector; B1-3, construct a bidirectional long short-term memory network model , in a vectorized representation as input, perform context feature fusion and output the fused character hidden features wherein is a vector with a dimension of ​ B1-4, construct an attention evaluation network based on a fully connected feedforward network , and As input, the respective attention weights are calculated , where [0,1] represents the weight of the i-th character hidden feature ​ B1-5, a semantic hidden feature vector of a given text is calculated as follows: is calculated as follows: 。 Step C comprises the following sub-steps: Step D comprises the following sub-steps: B2-1, setting the audio feature extraction window dimension to , setting the number of mel filters to wherein is a given positive integer; B2-2, for any given audio input , total duration, denoted by , will be evenly divided into non-overlapping audio segments, denoted by , with segment duration denoted by , satisfying ; B2-3, based on a preset Mel filter, extracts log-Mel spectral features from P audio segments, denoted as... ,in Indicates the first Log-Mel spectrum characteristics of an audio segment; B2-3, constructing a double-layer convolutional neural network model , as input, to compute the segmental hidden features of a given audio input wherein represents the hidden feature vector of the i-th audio segment; and represents the hidden feature vector of the i-th audio segment; and​​ B2-4, Construct an attention evaluation network based on a fully connected feedforward network. ,by As input, their respective attention weights are calculated and denoted as... ,in [0,1] represents the first... Weights of latent features in each audio segment; B2-5, computing a semantic feature vector for a given audio The formula for computing is as follows:​ 。 D1, constructing and executing a news video shot segmentation stage of the news video dynamic summary system; D2, constructing and executing a multi-modal semantic hidden feature extraction and scoring calculation stage of the news video dynamic summary system; B3-1. For any given video input , the number of image frames contained in the video is denoted as , the images corresponding to all the frames are obtained and denoted as ; B3-2, extraction the RGB three-channel values, denoted as ; B3-3, constructing a base image feature extraction convolutional neural network model , as input, calculate the image hidden feature, denoted as , wherein represents the hidden feature vector of the first frame image;​ B3-4, on the first Frame image respectively calculate 1 order, 2 order, 4 order difference characteristics 、 、 , the calculation formula is as follows: B3-5, for a given video input of all frame images, perform step B3-4 to obtain a sequence of difference features wherein, ; if ( ) exceeds an upper limit on the number of images , then a circular mapping mechanism is employed as follows: B3-6, construct an attention evaluation network based on a fully connected feedforward network , as input, calculate the 1st, 2nd, 4th order difference attention weights of the 1st, 2nd, 4th frame images , wherein ;​​ B3-7, compute the first frame image difference fusion hidden features ; B3-8, for a given video input all frame images, performing steps B3-6 and B3-7 yields a sequence of differential fused hidden features for all frames ; B3-9, construct a bidirectional long short-term memory network model , calculate the context fusion hidden feature sequence difference of all frames , calculate the context fusion hidden feature sequence difference of all frames , wherein represents the context fusion hidden feature vector of the i-th frame image context fusion hidden feature vector B3-10, computing image semantic hidden feature vectors for a given video The formula is as follows:​ 。 6. The method of claim 1, wherein, D3, constructing and executing a video dynamic summary generation stage of the news video dynamic summary system. C1, the video dynamic summarization score sequence calculated based on step B4. ,by To optimize the objective function, the maximum likelihood method is used to calculate the optimization of the objective function with respect to learnable parameters. and decision vector sequence The gradient is denoted as ,in Models that are probability expectation functions and contain learnable parameters include: , , , , , , , ; C2, set the learning rate based on the gradient calculated by C1 update the learnable parameters and the sequence of decision vectors as follows: where the function ; C3, based on the updated learnable parameters and the sequence of decision vectors , based on the new video dynamic summary score sequence computed in step B4 ; C4, set maximum iteration rounds Steps C2 and C3 are repeated for all collected N news videos until the video dynamic summary score sequence no longer changes or reaches a preset maximum iteration round, after which the training stops and a trained model is obtained: , , , , , , , .

7. The method of claim 1, wherein, In step D1, the news video shot segmentation stage of the news video dynamic summary system is constructed and executed, comprising the following sub-steps:

9. The news video dynamic summary extraction method according to claim 7, wherein step D2 comprises the following sub-steps:

10. The news video dynamic summary extraction method according to claim 7, wherein step D3 comprises the following sub-steps: The computer device comprises a processor and a memory, and the memory stores a computer program, which is loaded and executed by the processor to perform the method of any one of claims 1-10.

8. The method of claim 7, wherein, A computer program is stored in a readable storage medium, which is loaded and executed by a processor to perform the method of any one of claims 1-10. D1-1, news video with given input , sequentially performing steps A2 to A5 to obtain visual candidate cut points, denoted as D1-2, based on candidate cut points Steps A6 to A9 are sequentially performed to obtain all shot segments of the news video , denoted as . ​ ​ D2-1, acquire news video corresponding speech text , audio , video ; D2-2, obtaining All The speech text, audio, and video corresponding to each shot segment are denoted as ; D2-3, the voice text, based on the trained text semantic feature mapping matrix , the model , the model , the model , update the corresponding model parameters in B1-2 to B1-5 and execute in turn, obtain the text semantic hidden feature vector ; D2-4, pair of audio, based on the trained model , model , update the corresponding model parameters in B2-1 to B2-5 and execute in turn, obtain audio semantic hidden feature vector ; D2-5, pairs of video based on the trained model , model , model , update the corresponding model parameters in B3-1 to B3-10 and execute in turn, obtain image semantic hidden feature vector ; D2-6, traverse sequentially all The audio, text, and video corresponding to each shot segment; the audio text for each segment is based on the trained text semantic feature mapping matrix. ,Model ,Model Update the corresponding model parameters in B1-2 to B1-5 and execute them sequentially to obtain the semantic latent feature vectors of the text; for the audio of each segment, use the trained model. ,Model Update the corresponding model parameters in B2-1 to B2-5 and execute them sequentially to obtain the audio semantic latent feature vector; for each video segment, based on the trained model... ,Model ,Model Update the corresponding model parameters in B3-1 to B3-10 and execute them sequentially to obtain the semantic latent feature vector of the image; the semantic latent features of all obtained segments are denoted as... ,in, Indicates the first The textual semantic latent feature vector of each shot segment, Indicates the first The audio semantic latent feature vector of each shot segment, Indicates the first The video semantic latent feature vector of each shot segment; D2-7, given a set of news videos the decision vector , based on the news videos obtained in D2-3 to D2-6 and the news videos the semantic latent features of all shot segments, the video dynamic summary score under the current decision vector is calculated by performing steps B4-4 to B4-6 ; D2-8, traversing news videos All possible values of the decision vector, select the corresponding video dynamic summary score decision vector as the optimal decision vector, denoted as ; D2-9, calculating news video score of all shot segments wherein, The calculation formula is as follows: ； ； 。 ​ ​ D3-1, setting a video summary reference duration of a news video , a video summary segment number set , a video summary segment cumulative duration ; D3-2, the video is sorted according to the lens segment scores from high to low, the sorted lens segment sequence number is recorded as , and the time length of each segment is recorded as . ; D3-3, sequentially traversing the sorted shot segment serial numbers, if and , the first shot segment is taken as the selected abstract shot segment, and the , is updated; the final selected abstract shot segment serial number set is obtained, and M represents that a total of shot segments are selected. D3-4, to sorting to obtain a sorted sequence of shot segment numbers ; D3-5, selecting the video clip corresponding to the ordered set of clip serial numbers , merging in order to obtain the final news video dynamic summary.

11. A news video dynamic summary extraction device, characterized by, ​ 12. A readable storage medium, characterized by, ​ 13. A dynamic summary extraction system for news video, characterized by, A news video dynamic summary extraction device as claimed in claim 11 is included.

Images