Multimodal video highlight analysis method, apparatus, device, and medium
By converting video audio into text, and utilizing pre-trained language models and image encoding models in conjunction with a single-layer linear network, the system automatically analyzes video content, solving the problems of high manual costs and incomplete data representation in multimodal video highlight analysis, and achieving unified and accurate video content outline generation.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- GUANGZHOU SHIYUAN ELECTRONICS CO LTD
- Filing Date
- 2023-03-07
- Publication Date
- 2026-05-19
AI Technical Summary
Existing multimodal video highlight analysis methods are labor-intensive, generate inconsistent video content outlines, and provide incomplete representations of input videos using single-modal data.
By converting speech in input videos into text, and using pre-trained language models and image encoding models combined with a single-layer linear network, the system automatically analyzes video content, generates a unified video content outline, and aggregates multi-source data information.
It saves on labor costs, generates unified and accurate video content outlines, and provides more complete representation of multi-source data.
Smart Images
Figure CN118632090B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of video analytics technology, and for example to a multimodal video highlights analysis method, apparatus, device, and medium. Background Technology
[0002] The number of videos on the internet is exploding, and people's demand for quickly browsing video content is increasing. To help users quickly understand video content and improve user experience, video highlight extraction is necessary. Video highlight extraction includes extracting the start time of video highlights and a structured summary of the highlight segment. There are two existing methods for multimodal video highlight analysis: The first is manual analysis, where the video uploader observes and slices the video frame by frame to determine the location of highlight segments, extracts the start time of the segment, and summarizes the segment to obtain a highlight summary. Manual analysis is labor-intensive, and different people have different understandings of video highlights, resulting in inconsistent video content outlines. The second method generates video content outlines using single-modal data such as text or image information. Single-modal data is not a complete representation of the input video.
[0003] In summary, existing multimodal video highlight analysis methods suffer from high labor costs, inconsistent video content outlines, and / or incomplete representation of the input video by single-modal data. Summary of the Invention
[0004] The purpose of this application is to provide a multimodal video highlight analysis method, apparatus, device, and medium, which can solve the problems of existing multimodal video highlight analysis methods, such as high labor costs, inconsistent video content outlines, and / or incomplete representation of the input video by single-modal data. To achieve the above objective, this application provides a multimodal video highlight analysis method, including:
[0005] Acquire the input video, convert the speech in the input video into text, and obtain the text content;
[0006] The timestamp corresponding to the first character of each text statement in the text content is obtained according to the ending punctuation mark. The text statements are then concatenated according to the timestamps to obtain the concatenated text. The length of the concatenated text is a first preset length.
[0007] The last character of the i-th concatenated statement is slid forward by a second preset length to obtain the starting character; starting from the starting character, a third preset length of characters are truncated, and the video title and marker are added before the starting character to obtain the (i+1)-th concatenated statement; where i ≥ 1;
[0008] Each of the concatenated sentences is input into a pre-trained language model, and the concatenated sentences are represented by vectors to obtain concatenated vectors;
[0009] Predict the start and end positions of the highlights based on the splicing vector, and extract the highlights segments and segment start times based on the start and end positions of the highlights.
[0010] Extract the key takeaways from the highlighted segments;
[0011] Obtain the frame image corresponding to the start time of the segment, and encode the frame image using an image encoding model to obtain image information;
[0012] The summary of highlights and the image information are concatenated and fused to obtain fused information; a single-layer linear network is used to predict each character in the fused information to obtain a video content outline.
[0013] Preferably, encoding the frame image using an image encoding model to obtain image information includes:
[0014] The image information is obtained by encoding the frame image using the Taiyi model.
[0015] Preferably, predicting the start and end positions of the viewpoints based on the splicing vector includes:
[0016] The concatenated vector is input into the softmax layer of a single-layer pointer network for mapping to obtain a mapped vector.
[0017] The mapping vector is split to obtain a start position vector and an end position vector;
[0018] The starting position of the highlight is determined based on the starting position vector, and the ending position of the highlight is determined based on the ending position vector. The time corresponding to the starting position of the highlight is taken as the starting time of the segment.
[0019] Preferably, the step of using a single-layer linear network to predict each character in the fused information to obtain a video content outline includes:
[0020] The vector corresponding to each character in the fused information is input into the sigmoid layer of the single-layer linear network to classify each character in the fused information, thereby obtaining a first classification label and a second classification label.
[0021] Delete the characters corresponding to the second category label to obtain the video content outline.
[0022] Preferably, obtaining the timestamp corresponding to the first character of each text statement in the text content according to the ending punctuation mark includes:
[0023] Count the ending punctuation marks in the text content;
[0024] The timestamp corresponding to the first character of each text statement is obtained according to the ending punctuation mark.
[0025] Preferably, the step of converting the speech in the input video into text to obtain text content includes:
[0026] Obtain the audio from the input video;
[0027] The ASR speech recognition tool is used to convert the speech in the input video into text to obtain the text content.
[0028] Preferably, the extraction of the highlight summary from the highlight segment includes:
[0029] The key information segment is input into the pre-trained language model, and the key information segment is represented by a vector to obtain the key information summary.
[0030] This application provides a multimodal video highlight analysis device, including:
[0031] The speech-to-text conversion module is used to acquire input video, convert the speech in the input video into text, and obtain text content;
[0032] The text statement concatenation module is used to obtain the timestamp corresponding to the first character of each text statement in the text content according to the end punctuation mark, and concatenate each text statement according to the timestamp to obtain the concatenated text; the length of the concatenated text is a first preset length;
[0033] The concatenated statement sliding module is used to slide the last character of the i-th concatenated statement forward by a second preset length to obtain the starting character; and to extract a third preset length of characters from the starting character, and to add the video title and marker symbols before the starting character to obtain the (i+1)-th concatenated statement; where i≥1;
[0034] The first vector representation module is used to input each of the concatenated sentences into a pre-trained language model, perform vector representation on the concatenated sentences, and obtain a concatenated vector.
[0035] The highlight segment extraction module is used to predict the start position and end position of the highlight based on the splicing vector, and extract the highlight segment and segment start time based on the start position and end position of the highlight;
[0036] The highlight summary extraction module is used to extract a highlight summary of the highlight segment;
[0037] The frame image encoding module is used to obtain the frame image corresponding to the start time of the segment, and encode the frame image using an image encoding model to obtain image information;
[0038] The character prediction module is used to concatenate and fuse the highlight summary and the image information to obtain fused information; and to use a single-layer linear network to predict each character in the fused information to obtain a video content outline.
[0039] This application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the multimodal video highlight analysis method described in any one of the above claims and / or the multimodal video highlight analysis method described in any one of the above claims.
[0040] This application also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the multimodal video highlight analysis method described in any of the preceding claims and / or the multimodal video highlight analysis method described in any of the preceding claims.
[0041] This application discloses a multimodal video highlight analysis method, which includes acquiring an input video, converting the speech in the input video into text to obtain text content. The method involves acquiring the timestamp corresponding to the first character of each text statement in the text content, concatenating the text statements according to the timestamps to obtain concatenated text; the length of the concatenated text is a first preset length; sequentially sliding the last character of the i-th concatenated statement forward by a second preset length to obtain a starting character; truncating a third preset length of characters from the starting character, adding the video title and marker symbols before the starting character to obtain the (i+1)-th concatenated statement; where i ≥ 1, and the first preset length is equal to the sum of the second and third preset lengths; inputting each concatenated statement into a pre-trained language model to represent the concatenated statement as a vector to obtain a concatenated vector. The method employs a fixed-step sliding method to segment the concatenated text, allowing the concatenated text to be encoded by the pre-trained language model, preventing the segmentation of text exceeding the length limit, thereby preventing the loss of a large amount of text information. The starting and ending positions of key points are predicted based on the concatenated vectors. Key point segments and their start times are then extracted from these positions. A key point summary is extracted from each segment, and the frame image corresponding to the segment's start time is obtained. This frame image is encoded using an image encoding model to obtain image information. The key point summary and image information are then concatenated and fused to obtain fused information. A single-layer linear network is used to predict each character in the fused information to obtain a video content outline. Automated calculation saves manpower and provides a unified analysis standard, ensuring consistent video content outlines. Since key point summaries are textual information, concatenating and fusing key point summaries and image information aggregates information from multiple sources, resulting in a more complete representation of the input video and a more accurate generated video content outline. Attached Figure Description
[0042] Figure 1 This is a flowchart illustrating a multimodal video highlight analysis method according to one embodiment.
[0043] Figure 2 This is a flowchart illustrating the prediction of the starting and ending positions of a viewpoint in one embodiment.
[0044] Figure 3 This is a schematic diagram illustrating the process of predicting characters from fused information according to one embodiment;
[0045] Figure 4 This is a schematic block diagram of the structure of a multimodal video highlight analysis device according to one embodiment;
[0046] Figure 5 This is a schematic block diagram of the structure of a computer device according to one embodiment.
[0047] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation
[0048] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.
[0049] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this specification means the presence of features, integers, steps, operations, elements, modules, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, modules, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any modules and all combinations of one or more associated listed items.
[0050] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.
[0051] In one embodiment, refer to Figure 1 This is a flowchart illustrating the multimodal video viewpoint analysis method disclosed in this application, the method comprising:
[0052] S1: Obtain the input video, convert the speech in the input video into text, and obtain the text content.
[0053] Obtain the audio from the input video;
[0054] The ASR speech recognition tool is used to convert the speech in the input video into text to obtain the text content.
[0055] The input video includes image data and audio data. The video format of the input video includes one or more of the following formats: MPEG, AVI, ASF, MOV, and WMV. No specific format is specified here.
[0056] The ASR speech recognition tool can be either Tencent Cloud ASR or Baidu Cloud ASR; no specific choice is made here.
[0057] S2: Obtain the timestamp corresponding to the first character of each text statement in the text content according to the ending punctuation mark, and concatenate each text statement according to the timestamp to obtain the concatenated text; the length of the concatenated text is a first preset length.
[0058] Count the ending punctuation marks in the text content;
[0059] The timestamp corresponding to the first character of each text statement is obtained according to the ending punctuation mark.
[0060] Ending punctuation marks include periods, question marks, and exclamation marks. The first character is the character that begins each text statement, and the timestamp corresponding to the first character is calculated in seconds.
[0061] For example, the text content is (1) Hi everyone, welcome to this episode of Constellation, (2) I am the host, Teacher Gu. (3) Today we will talk about the three constellations in (4) of the 12 constellations. The timestamp corresponding to the first character of sentence (1) is the 1st second, the timestamp corresponding to the first character of sentence (2) is the 4th second, the timestamp corresponding to the first character of sentence (3) is the 6th second, and the timestamp corresponding to the first character of sentence (4) is the 8th second. By splicing the text sentences according to the timestamps, the spliced text is (1) Hi everyone, welcome to this episode of Constellation, I am the host, Teacher Gu. (2) Today we will talk about the three constellations in the 12 constellations.
[0062] The first preset length represents the maximum input length of the pre-trained language model. Since the concatenated text is relatively long, it needs to be divided. The first preset length can be 512 or other lengths.
[0063] S3: Slide the last character of the i-th concatenation statement forward by a second preset length to obtain the starting character; take the starting character as the starting point and cut off a third preset length of characters to the end, and add the video title and marker symbols before the starting character to obtain the (i+1)-th concatenation statement; where i≥1.
[0064] The second preset length is fixed to prevent the splicing of text. The video title and marker are added sequentially before the first character of the spliced text. The video title, marker, the first character of the spliced text, and the third preset length of characters following the first character of the spliced text form the first splicing statement. The last character of the (i-1)th splicing statement is slid forward by the second preset length to obtain the corresponding starting character. From this starting character, the third preset length of characters is truncated. The video title and marker are added before this starting character. The video title, marker, the starting character, and the third preset length of characters following the starting character form the i-th splicing statement. When the last character of the i-th splicing statement is the last character of the spliced text, the process of sliding the last character of the splicing statement forward stops. The sum of the length of the video title and the length of the marker is the first temporary length. The first temporary length plus 1 is the second temporary length. The first preset length minus the second temporary length is the third preset length.
[0065] Preferably, the length of the video title is set to 1, the length of the marker symbol is set to 1, the first preset length is set to 512, the second preset length is set to 127, and the third preset length is set to 509.
[0066] For example, when the concatenated text contains 7 characters, it is represented as T = t1, t2, t3, t4, t5, t6, t7. The first preset length is 5, the second preset length is 1, and the third preset length is 2. The first concatenated statement is represented as T1 = video title, SEP, t1, t2, t3, where SEP represents the marker symbol. The last character of the first concatenated statement is moved forward by the second preset length of characters to obtain the starting character t2. Starting from t2, the third preset length of characters is extracted to obtain t3 and t4. The video title and the marker symbol are added before t2 to obtain the second concatenated statement, represented as T2 = video title, SEP, t2, t3, t4.
[0067] S4: Input each of the concatenated sentences into the pre-trained language model, and represent the concatenated sentences into vectors to obtain concatenated vectors.
[0068] The pre-trained language model can be the LEBERT model or other models, which is not limited here. In this embodiment, the pre-trained language model is the LEBERT model optimized for Chinese.
[0069] The LEBERT model uses the entire vocabulary of Chinese sequence labels to enhance the BERT model. It integrates external vocabulary information directly into the BERT model through a full vocabulary adaptation layer, which facilitates deep vocabulary knowledge fusion at the lower layers of the BERT model.
[0070] The LEBERT model has two versions: a base version and a large version. The base version has 12 layers, 768 hidden layers, and 12 multi-head attention mechanisms; the large version has 24 layers, 1024 hidden layers, and 16 multi-head attention mechanisms. This application's embodiment uses the base version of the LEBERT model as an example.
[0071] The formula for calculating the concatenated vector is as follows:
[0072] E i =M(T) i );
[0073] Among them, T i This represents the i-th concatenated statement, with a first preset length of 512. M represents the base version of the LEBERT model, and E... i Let represent the i-th concatenated vector, which has a dimension of 512×768.
[0074] S5: Predict the start and end positions of the highlights based on the splicing vector, and extract the highlights segments and segment start times based on the start and end positions of the highlights.
[0075] The concatenated vector is input into the softmax layer of a single-layer pointer network for mapping to obtain a mapped vector.
[0076] The mapping vector is split to obtain a start position vector and an end position vector;
[0077] The starting position of the highlight is determined based on the starting position vector, and the ending position of the highlight is determined based on the ending position vector. The time corresponding to the starting position of the highlight is taken as the starting time of the segment.
[0078] The segment between the start and end points of a highlight is called a highlight segment.
[0079] S6: Extract the summary of the highlights of the highlighted segments.
[0080] The key information segment is input into the pre-trained language model, and the key information segment is represented by a vector to obtain the key information summary.
[0081] The formula for calculating the highlights summary is as follows:
[0082] S i =M(V) i );
[0083] Among them, S iLet M represent the base version of the LEBERT model, and V be the summary of the i-th highlight. i This represents the i-th highlight segment.
[0084] S7: Obtain the frame image corresponding to the start time of the segment, and encode the frame image using an image encoding model to obtain image information.
[0085] The image information is obtained by encoding the frame image using the Taiyi model.
[0086] The Taiyi model, pre-trained on 123 million image-text pairs, is an open-source Chinese CLIP model with strong visual and linguistic representation capabilities.
[0087] S8: The summary of highlights and the image information are spliced and fused to obtain fused information; a single-layer linear network is used to predict each character in the fused information to obtain a video content outline.
[0088] The vector corresponding to each character in the fused information is input into the sigmoid layer of the single-layer linear network to classify each character in the fused information, thereby obtaining a first classification label and a second classification label.
[0089] Delete the characters corresponding to the second category label to obtain the video content outline.
[0090] The first category label corresponds to the characters that need to be kept, and the second category label corresponds to the characters that need to be deleted.
[0091] This application provides a multimodal video highlight analysis method, which includes acquiring an input video, converting the speech in the input video into text to obtain text content. The method involves acquiring the timestamp corresponding to the first character of each text statement in the text content, concatenating the text statements according to the timestamps to obtain concatenated text; the length of the concatenated text is a first preset length; sequentially sliding the last character of the i-th concatenated statement forward by a second preset length to obtain a starting character; truncating a third preset length of characters from the starting character, adding the video title and marker symbols before the starting character to obtain the (i+1)-th concatenated statement; where i ≥ 1, and the first preset length is equal to the sum of the second and third preset lengths; inputting each concatenated statement into a pre-trained language model to represent the concatenated statement as a vector to obtain a concatenated vector. The method uses a fixed-step sliding approach to segment the concatenated text, allowing the concatenated text to be encoded by the pre-trained language model, preventing the segmentation of text exceeding the length limit, and thus preventing the loss of a large amount of text information. The starting and ending positions of key points are predicted based on the concatenated vectors. Key point segments and their start times are then extracted from these positions. A key point summary is extracted from each segment, and the frame image corresponding to the segment's start time is obtained. This frame image is encoded using an image encoding model to obtain image information. The key point summary and image information are then concatenated and fused to obtain fused information. A single-layer linear network is used to predict each character in the fused information to obtain a video content outline. Automated calculation saves manpower and provides a unified analysis standard, ensuring consistent video content outlines. Since key point summaries are textual information, concatenating and fusing key point summaries and image information aggregates information from multiple sources, resulting in a more complete representation of the input video and a more accurate generated video content outline.
[0092] In one embodiment, refer to Figure 2 The step of predicting the start and end positions of the viewpoints based on the splicing vector includes:
[0093] S51: The concatenated vector is input into the softmax layer of a single-layer pointer network for mapping to obtain a mapped vector.
[0094] When the first preset length is 512, the dimension of the concatenated vector is 512×768. The concatenated vector is mapped to the mapping vector through the softmax layer of the single-layer pointer network. The dimension of the mapping vector is 512×2.
[0095] The formula for mapping the concatenated vectors is as follows:
[0096] D i =softmax(W e E i+b e );
[0097] Among them, D i Let W be the i-th mapping vector, where softmax represents the softmax mapping function. e Let b represent the weight matrix. e E represents the bias matrix. i Let i represent the i-th concatenated vector.
[0098] S52: Split the mapping vector to obtain the starting position vector and the ending position vector.
[0099] For the i-th mapping vector D i By splitting the vector, we obtain the i-th starting position vector. and the i-th termination position vector
[0100] S53: Determine the starting position of the viewpoint based on the starting position vector, determine the ending position of the viewpoint based on the ending position vector, and take the time corresponding to the starting position of the viewpoint as the starting time of the segment.
[0101] The i-th starting position vector and the i-th termination position vector The segment between these points is the i-th highlight segment, based on the i-th starting position vector. Determine the starting position of the i-th viewpoint; the time corresponding to the starting position of the i-th viewpoint is the starting time of the i-th segment. Based on the i-th ending position vector... Determine the ending position of the i-th viewpoint.
[0102] As described above, predicting the start and end positions of viewpoints based on the spliced vectors involves inputting the spliced vectors into a softmax layer of a single-layer pointer network for mapping, resulting in a mapped vector. This mapped vector is then split to obtain a start position vector and an end position vector. The start position of the viewpoint is determined based on the start position vector, and the end position is determined based on the end position vector. The time corresponding to the start position of the viewpoint is taken as the segment start time, and the i-th start position vector... and the i-th termination position vector The segment between them is the i-th highlight segment.
[0103] In one embodiment, refer to Figure 3 The step of using a single-layer linear network to predict each character in the fused information to obtain a video content outline includes:
[0104] S81: Input the vector corresponding to each character in the fused information into the sigmoid layer of the single-layer linear network to classify each character in the fused information and obtain a first classification label and a second classification label.
[0105] For example, the i-th fused information is "Many Pinyin input software programs have an association function", where "Pinyin input software", "association", and "function" belong to the first category label, and "many", "have", and "of" belong to the second category label.
[0106] S82: Delete the characters corresponding to the second category label to obtain the video content outline.
[0107] Delete the characters corresponding to the second category label, and keep the characters corresponding to the first category label. The resulting video content outline is "Pinyin input software predictive function".
[0108] By processing all the fusion information sequentially according to the steps S81-S82 described above, a complete outline of all video content of the input video can be obtained.
[0109] As described above, a single-layer linear network is used to predict each character in the fused information to obtain a video content outline. This involves inputting the vector corresponding to each character in the fused information into the sigmoid layer of the single-layer linear network, classifying each character in the fused information to obtain a first classification label and a second classification label. The character corresponding to the second classification label is then deleted to obtain the video content outline. By classifying all the fused information sequentially and deleting the characters corresponding to the second classification labels, the complete video content outline of the input video can be obtained.
[0110] Reference Figure 4 This is a schematic block diagram of a multimodal video viewpoint analysis device disclosed in this application. The device includes:
[0111] The speech-to-text conversion module 10 is used to acquire an input video and convert the speech in the input video into text to obtain text content.
[0112] The text statement concatenation module 20 is used to obtain the timestamp corresponding to the first character of each text statement in the text content according to the end punctuation mark, and concatenate each text statement according to the timestamp to obtain the concatenated text; the length of the concatenated text is a first preset length.
[0113] The concatenated statement sliding module 30 is used to slide the last character of the i-th concatenated statement forward by a second preset length to obtain the starting character; and to extract a third preset length of characters from the starting character, and to add the video title and marker symbols before the starting character to obtain the (i+1)-th concatenated statement; where i ≥ 1;
[0114] The first vector representation module 40 is used to input each of the concatenated sentences into a pre-trained language model, perform vector representation on the concatenated sentences, and obtain a concatenated vector.
[0115] The highlight segment extraction module 50 is used to predict the start position and end position of the highlight based on the splicing vector, and extract the highlight segment and segment start time based on the start position and end position of the highlight;
[0116] The highlight summary extraction module 60 is used to extract a highlight summary of the highlight segment;
[0117] The frame image encoding module 70 is used to obtain the frame image corresponding to the start time of the segment, and encode the frame image using an image encoding model to obtain image information;
[0118] The character prediction module 80 is used to concatenate and fuse the highlight summary and the image information to obtain fused information; and to use a single-layer linear network to predict each character in the fused information to obtain a video content outline.
[0119] As described above, the multimodal video highlights analysis device can realize the multimodal video highlights analysis method.
[0120] In one embodiment, the frame image encoding module 70 further includes:
[0121] The frame image encoding unit is used to encode the frame image using the Taiyi model to obtain the image information.
[0122] In one embodiment, the highlight segment extraction module 50 further includes:
[0123] The splicing vector mapping unit is used to input the splicing vector into the softmax layer of a single-layer pointer network for mapping to obtain a mapping vector;
[0124] A mapping vector splitting unit is used to split the mapping vector to obtain a start position vector and an end position vector;
[0125] The position determination unit is used to determine the starting position of the viewpoint based on the starting position vector, determine the ending position of the viewpoint based on the ending position vector, and take the time corresponding to the starting position of the viewpoint as the starting time of the segment.
[0126] In one embodiment, the character prediction module 80 further includes:
[0127] A character classification unit is used to input the vector corresponding to each character in the fused information into the sigmoid layer of the single-layer linear network to classify each character in the fused information and obtain a first classification label and a second classification label.
[0128] The character deletion unit is used to delete the characters corresponding to the second category label to obtain the video content outline.
[0129] In one embodiment, the text statement concatenation module 20 further includes:
[0130] The end punctuation mark counting unit is used to count the end punctuation marks in the text content;
[0131] The timestamp acquisition unit is used to acquire the timestamp corresponding to the first character of each text statement according to the end punctuation mark.
[0132] In one embodiment, the speech-to-text conversion module 10 further includes:
[0133] A voice acquisition unit is used to acquire voice from the input video;
[0134] The speech-to-text unit is used to convert the speech in the input video into text using an ASR speech recognition tool to obtain the text content.
[0135] In one embodiment, the highlight summary extraction module 60 further includes:
[0136] The highlight summary extraction unit is used to input the highlight segment into the pre-trained language model, perform vector representation on the highlight segment, and obtain the highlight summary.
[0137] Reference Figure 5 This application also provides a computer device whose internal structure can be as follows: Figure 5As shown, this computer device includes a processor, memory, network interface, and database connected via a system bus. The processor is designed to provide computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores operating devices, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores splicing vectors, etc. The network interface is used for communication with external terminals via a network connection. Furthermore, the computer device may also include input devices and a display screen, etc. When executed by a processor, this computer program implements a multimodal video highlight analysis method, including the following steps: acquiring an input video; converting the speech in the input video into text to obtain text content; obtaining the timestamp corresponding to the first character of each text statement in the text content according to the end punctuation mark; concatenating each text statement according to the timestamp to obtain concatenated text; the length of the concatenated text is a first preset length; sequentially sliding the last character of the i-th concatenated statement forward by a second preset length to obtain a starting character; truncating a third preset length of characters from the starting character; adding the video title and marker symbols before the starting character to obtain the i-th... i+1 concatenated statements; where i≥1; each concatenated statement is input into a pre-trained language model, and the concatenated statement is vectorized to obtain a concatenated vector; the start and end positions of the highlights are predicted based on the concatenated vector, and highlights segments and their start times are extracted based on the start and end positions; a highlights summary of the highlights segments is extracted; the frame image corresponding to the start time of the segment is obtained, and the frame image is encoded using an image encoding model to obtain image information; the highlights summary and the image information are concatenated and fused to obtain fused information; a single-layer linear network is used to predict each character in the fused information to obtain a video content outline. Those skilled in the art will understand that... Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer equipment on which the present application is applied.
[0138] One embodiment of this application also provides a computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it implements a multimodal video highlight analysis method, including the following steps: acquiring an input video; converting the speech in the input video into text to obtain text content; obtaining the timestamp corresponding to the first character of each text statement in the text content according to the end punctuation mark; concatenating each text statement according to the timestamp to obtain concatenated text; the length of the concatenated text is a first preset length; sequentially sliding the last character of the i-th concatenated statement forward by a second preset length to obtain a starting character; truncating a third preset length of characters from the starting character to obtain the video title and marker symbols. Add the concatenated statement before the starting character to obtain the (i+1)th concatenated statement; where i≥1; input each concatenated statement into a pre-trained language model, and represent the concatenated statement as a vector to obtain a concatenated vector; predict the start and end positions of the highlights based on the concatenated vector, and extract the highlights segments and their start times based on the start and end positions; extract the highlights summary of the highlights segments; obtain the frame image corresponding to the start time of the segment, and encode the frame image using an image encoding model to obtain image information; concatenate and fuse the highlights summary and the image information to obtain fused information; use a single-layer linear network to predict each character in the fused information to obtain a video content outline.
[0139] It is understood that the computer-readable storage medium in this embodiment can be a volatile readable storage medium or a non-volatile readable storage medium.
[0140] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media provided in this application and in the embodiments may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual-speed SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
[0141] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, apparatus, article, or method that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, apparatus, article, or method. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, apparatus, article, or method that includes that element.
[0142] The above description is merely a preferred embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.
Claims
1. A multimodal video highlight analysis method, characterized in that, include: Acquire the input video, convert the speech in the input video into text, and obtain the text content; The timestamp corresponding to the first character of each text statement in the text content is obtained according to the ending punctuation mark. The text statements are then concatenated according to the timestamps to obtain the concatenated text. The length of the concatenated text is a first preset length. The last character of the i-th concatenated statement is slid forward by a second preset length to obtain the starting character; a third preset length of characters is truncated from the starting character, and the video title and marker are added before the starting character to obtain the (i+1)-th concatenated statement; where i ≥ 1; specifically, the second preset length is fixed; the video title and marker are added before the first character of the concatenated text, and the video title, marker, the first character of the concatenated text, and the third preset length of characters after the first character of the concatenated text form the first concatenated statement; the last character of the (i-1)-th concatenated statement is slid forward... The second preset length is used to obtain the corresponding starting character. Starting from this starting character, a third preset length of characters is extracted. The video title and marker are added before the starting character. The video title, marker, the starting character, and the third preset length of characters after the starting character form the i-th concatenation statement. When the last character of the i-th concatenation statement is the last character of the concatenated text, the process of sliding the last character of the concatenation statement forward stops. The sum of the length of the video title and the length of the marker is the first temporary length. The first temporary length plus 1 is the second temporary length. The first preset length minus the second temporary length is the third preset length. Each of the concatenated sentences is input into a pre-trained language model, and the concatenated sentences are represented by vectors to obtain concatenated vectors; Predict the start and end positions of the highlights based on the splicing vector, and extract the highlights segments and segment start times based on the start and end positions of the highlights. Extract the key takeaways from the highlighted segments; Obtain the frame image corresponding to the start time of the segment, and encode the frame image using an image encoding model to obtain image information; The summary of highlights and the image information are concatenated and fused to obtain fused information; a single-layer linear network is used to predict each character in the fused information to obtain a video content outline.
2. The multimodal video highlight analysis method according to claim 1, characterized in that, The process of encoding the frame image using an image encoding model to obtain image information includes: The image information is obtained by encoding the frame image using the Taiyi model.
3. The multimodal video highlight analysis method according to claim 1, characterized in that, The step of predicting the start and end positions of the viewpoints based on the splicing vector includes: The concatenated vector is input into the softmax layer of a single-layer pointer network for mapping to obtain a mapped vector. The mapping vector is split to obtain a start position vector and an end position vector; The starting position of the highlight is determined based on the starting position vector, and the ending position of the highlight is determined based on the ending position vector. The time corresponding to the starting position of the highlight is taken as the starting time of the segment.
4. The multimodal video highlight analysis method according to claim 1, characterized in that, The process of using a single-layer linear network to predict each character in the fused information to obtain a video content outline includes: The vector corresponding to each character in the fused information is input into the sigmoid layer of the single-layer linear network to classify each character in the fused information, thereby obtaining a first classification label and a second classification label. Delete the characters corresponding to the second category label to obtain the video content outline.
5. The multimodal video highlight analysis method according to claim 1, characterized in that, The step of obtaining the timestamp corresponding to the first character of each text statement in the text content according to the ending punctuation mark includes: Count the ending punctuation marks in the text content; The timestamp corresponding to the first character of each text statement is obtained according to the ending punctuation mark.
6. The multimodal video highlight analysis method according to claim 1, characterized in that, The step of converting the speech in the input video into text to obtain text content includes: Obtain the audio from the input video; The ASR speech recognition tool is used to convert the speech in the input video into text to obtain the text content.
7. The multimodal video highlight analysis method according to claim 1, characterized in that, The extraction of the highlight summary from the highlight segment includes: The key information segment is input into the pre-trained language model, and the key information segment is represented by a vector to obtain the key information summary.
8. A multimodal video highlight analysis device, used to implement the method described in any one of claims 1-7, characterized in that, include: The speech-to-text conversion module is used to acquire input video, convert the speech in the input video into text, and obtain text content; The text statement concatenation module is used to obtain the timestamp corresponding to the first character of each text statement in the text content according to the end punctuation mark, and concatenate each text statement according to the timestamp to obtain the concatenated text; the length of the concatenated text is a first preset length; The concatenated sentence sliding module is used to sequentially slide the last character of the i-th concatenated sentence forward by a second preset length to obtain the starting character; starting from the starting character, it truncates a third preset length of characters backward, and adds the video title and marker symbol before the starting character to obtain the (i+1)-th concatenated sentence; where i ≥ 1; specifically, the second preset length is fixed; the video title and marker symbol are sequentially added before the first character of the concatenated text, and the video title, marker symbol, the first character of the concatenated text, and the third preset length of characters after the first character of the concatenated text form the first concatenated sentence; the last character of the (i-1)-th concatenated sentence is then... The character slides forward by a second preset length to obtain the corresponding starting character. Starting from this starting character, a third preset length of characters is extracted. The video title and marker are added before the starting character. The video title, marker, the starting character, and the third preset length of characters after the starting character form the i-th concatenation statement. When the last character of the i-th concatenation statement is the last character of the concatenated text, the process of sliding the last character of the concatenation statement forward stops. The sum of the length of the video title and the length of the marker is the first temporary length. The first temporary length plus 1 is the second temporary length. The first preset length minus the second temporary length is the third preset length. The first vector representation module is used to input each of the concatenated sentences into a pre-trained language model, perform vector representation on the concatenated sentences, and obtain a concatenated vector. The highlight segment extraction module is used to predict the start position and end position of the highlight based on the splicing vector, and extract the highlight segment and segment start time based on the start position and end position of the highlight; The highlight summary extraction module is used to extract a highlight summary of the highlight segment; The frame image encoding module is used to obtain the frame image corresponding to the start time of the segment, and encode the frame image using an image encoding model to obtain image information; The character prediction module is used to concatenate and fuse the highlight summary and the image information to obtain fused information; and to use a single-layer linear network to predict each character in the fused information to obtain a video content outline.
9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the multimodal video highlights analysis method as described in any one of claims 1 to 7.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the multimodal video highlights analysis method as described in any one of claims 1 to 7.