Long video highlight spot detection method and device, computer equipment and storage medium

Abstract

The invention discloses a long video highlight spot detection method and device, computer equipment and a storage medium, and relates to the technical field of videos. The method comprises the following steps: performing voice recognition on a to-be-detected long video according to a voice recognition algorithm to obtain a subtitle information set; extracting a video description information set of the to-be-detected long video through the video language large model; fusing the subtitle information set and the video description information set based on a time axis to obtain a fused text information set; and detecting the fusion text information set according to the highlight prompt word through a natural language large model to obtain a highlight point time period and corresponding highlight content information. Therefore, highlight spot detection is performed by extracting the subtitle information and the video description information of the long video to be detected, so that video highlight spot information can be obtained without depending on data set training, and highlight spot contents meeting user requirements can be extracted according to the highlight prompt words.

Description

Technical Field

[0001] The present invention relates to the field of video technology, and in particular to a method, device, computer equipment and storage medium for detecting highlights in a long video. Background Art

[0002] At present, the wonderful moments based on artificial intelligence are directly predicted whether a certain moment is wonderful based on the inherent data set. First of all, the generalization of this solution is highly dependent on the data set. For long video types that do not exist in the data set, the prediction performance cannot be guaranteed; secondly, it cannot output the title of the wonderful moment and the reason for being a wonderful moment, which makes manual inspection inconvenient; and it cannot select the wonderful moments according to different needs. Summary of the invention

[0003] In view of this, the purpose of the present invention is to overcome the shortcomings of the prior art and provide a long video highlight point detection method, device, computer equipment and storage medium for understanding the content of the long video to be detected and extracting the highlight content in the long video to be detected.

[0004] The present invention provides the following technical solutions:

[0005] In a first aspect, the present invention provides a method for detecting highlight points in a long video, comprising:

[0006] Perform speech recognition on the long video to be detected according to the speech recognition algorithm to obtain a subtitle information set;

[0007] Extracting a video description information set of the long video to be detected by using a large video language model;

[0008] Merging the subtitle information set and the video description information set based on a time axis to obtain a fused text information set;

[0009] The fused text information set is detected according to the highlight prompt words through a large natural language model to obtain the highlight time period and its corresponding highlight content information.

[0010] In one implementation, performing speech recognition on the long video to be detected according to a speech recognition algorithm to obtain a subtitle information set includes:

[0011] Obtaining an audio file according to the long video to be detected;

[0012] Determine audio parameters according to the audio file, and calculate the number of audio blocks in the audio file according to the audio parameters;

[0013] Determine a speaking area according to the audio parameter and the number of audio blocks;

[0014] Speech recognition is performed on the speaking area according to the speech recognition algorithm to obtain a subtitle information set.

[0015] In one implementation, determining the speaking area according to the audio parameter and the number of audio blocks includes:

[0016] Determine an audio energy value of each audio block according to the audio parameter and the number of audio blocks;

[0017] Calculating an energy threshold according to each of the audio energy values ​​based on the percentile;

[0018] The speaking area is determined according to the energy threshold and each of the audio energy values.

[0019] In one embodiment, determining the speaking area according to the energy threshold and each of the audio energy values ​​includes:

[0020] Traversing each of the audio energy values, and determining whether the i-th audio energy value is greater than the energy threshold;

[0021] If the i-th audio energy value is less than or equal to the energy threshold, the audio block corresponding to the i-th audio energy value is in a silent state; if the i-th audio energy value is greater than the energy threshold, the audio block corresponding to the i-th audio energy value is in a non-silent state;

[0022] Determine whether there is a start time for the speaking area at present;

[0023] If the start time of the speaking area does not currently exist, and the audio block corresponding to the i-th audio energy value is in the non-silent state, the current time is used as the start time of the speaking area, and it is determined whether the i+1-th audio energy value is greater than the energy threshold;

[0024] If the start time of the speaking area does not currently exist and the audio block corresponding to the i-th audio energy value is in the silent state, determining whether the i+1-th audio energy value is greater than the energy threshold;

[0025] If there is a start time for the speaking area currently, then calculating the elapsed time between the start time and the time corresponding to the i-th audio energy value;

[0026] Determining whether the elapsed time is greater than or equal to a preset maximum area time;

[0027] If the elapsed time is greater than or equal to the preset maximum area time, the current time is used as the end time of the speaking area, and the start time of the speaking area is initialized, and it is determined whether the (i+1)th audio energy value is greater than the energy threshold;

[0028] If the elapsed time is less than the preset maximum area time, determining whether the elapsed time is greater than or equal to the preset minimum area time;

[0029] If the elapsed time is greater than or equal to the preset minimum area time, and the audio block corresponding to the i-th audio energy value is in a silent state, the current time is used as the end time of the speaking area, and the start time of the speaking area is initialized, and it is determined whether the i+1-th audio energy value is greater than the energy threshold;

[0030] If the elapsed time is greater than or equal to the preset minimum area time, and the audio block corresponding to the i-th audio energy value is in a non-silent state, determining whether the i+1-th audio energy value is greater than the energy threshold;

[0031] If the elapsed time is less than the preset minimum area time, it is determined whether the (i+1)th audio energy value is greater than the energy threshold.

[0032] In one embodiment, extracting the video description information set of the long video to be detected by using a large video language model includes:

[0033] Dividing the long video to be detected into a plurality of short videos to be detected according to the time interval corresponding to the subtitle information set;

[0034] Extracting video description information corresponding to each of the short videos to be detected by using the video language large model;

[0035] The video description information set is determined according to the video description information corresponding to each of the short videos to be detected.

[0036] In one embodiment, the subtitle information set includes subtitle information corresponding to multiple time intervals, and the video description information set includes video description information corresponding to multiple time intervals; the subtitle information set and the video description information set are merged based on the time axis to obtain a merged text information set, including:

[0037] For each time interval, the subtitle information and video description information corresponding to the same time interval are merged to obtain fused text information;

[0038] The fused text information set is determined according to the fused text information corresponding to each time interval.

[0039] In one embodiment, the fused text information set is detected according to the highlight prompt words by using the natural language big model to obtain the highlight point time period and its corresponding highlight content information, including:

[0040] Extracting features from the fused text information set using the natural language large model to obtain a feature set;

[0041] Determining a highlight content set from the feature set according to the highlight prompt word;

[0042] The highlight point time period and the highlight content information are determined according to the highlight content set.

[0043] In a second aspect, the present invention provides a long video highlight point detection device, comprising:

[0044] A recognition module, used to perform speech recognition on the long video to be detected according to a speech recognition algorithm to obtain a subtitle information set;

[0045] An extraction module, used for extracting a video description information set of the long video to be detected through a large video language model;

[0046] A fusion module, used for fusing the subtitle information set and the video description information set based on a time axis to obtain a fused text information set;

[0047] The detection module is used to detect the fused text information set according to the highlight prompt words through a large natural language model to obtain the highlight time period and its corresponding highlight content information.

[0048] In a third aspect, the present invention proposes a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the long video highlight point detection method as described in the first aspect is implemented.

[0049] In a fourth aspect, the present invention proposes a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the long video highlight point detection method as described in the first aspect.

[0050] The long video highlight point detection method, device, computer equipment and storage medium disclosed in the present invention perform speech recognition on the long video to be detected according to the speech recognition algorithm to obtain a subtitle information set; extract the video description information set of the long video to be detected through the video language large model; fuse the subtitle information set and the video description information set based on the time axis to obtain a fused text information set; detect the fused text information set according to the highlight prompt words through the natural language large model to obtain the highlight point time period and its corresponding highlight content information. In this way, by extracting the subtitle information and video description information of the long video to be detected to perform highlight point detection, the video highlight point information can be obtained without relying on data set training, and the highlight point content that meets the user's needs can be extracted according to the highlight prompt words. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solution of the present invention, the following is a brief introduction to the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and should not be regarded as limiting the scope of protection of the present invention. In each of the drawings, similar components are numbered similarly.

[0052] Figure 1 A schematic diagram of a process of a long video highlight point detection method proposed in this embodiment is shown;

[0053] Figure 2 Another schematic diagram of the process of the long video highlight point detection method proposed in this embodiment is shown;

[0054] Figure 3 A schematic diagram of a subtitle information set proposed in this embodiment is shown;

[0055] Figure 4 Another schematic diagram of the process of the long video highlight point detection method proposed in this embodiment is shown;

[0056] Figure 5 Another schematic diagram of the process of the long video highlight point detection method proposed in this embodiment is shown;

[0057] Figure 6 Another schematic diagram of the process of the long video highlight point detection method proposed in this embodiment is shown;

[0058] Figure 7 A schematic diagram of a video description information set proposed in this embodiment is shown;

[0059] Figure 8 A schematic diagram of the fused text information set proposed in this embodiment is shown;

[0060] Fig. 9 A schematic diagram showing the highlight point time period and its corresponding highlight content information proposed in this embodiment;

[0061] Fig.10 A schematic diagram of the structure of the long video highlight point detection device proposed in this embodiment is shown.

[0062] Description of the accompanying drawings:

[0063] 1000 - long video highlight point detection device; 1001 - recognition module; 1002 - extraction module; 1003 - fusion module; 1004 - detection module. DETAILED DESCRIPTION

[0064] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0065] The components of the embodiments of the present invention generally described and shown in the drawings herein may be arranged and designed in a variety of different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0066] Hereinafter, the terms "including", "having" and their cognates, which may be used in various embodiments of the present invention, are intended only to indicate specific features, numbers, steps, operations, elements, components or combinations of the foregoing items, and should not be understood as first excluding the existence of one or more other features, numbers, steps, operations, elements, components or combinations of the foregoing items or adding the possibility of one or more features, numbers, steps, operations, elements, components or combinations of the foregoing items.

[0067] Furthermore, the terms “first”, “second”, “third”, etc. are merely used for distinguishing descriptions and are not to be understood as indicating or implying relative importance.

[0068] Unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meanings as those generally understood by those skilled in the art to which the various embodiments of the present invention belong. The terms (such as those defined in generally used dictionaries) will be interpreted as having the same meanings as the contextual meanings in the relevant technical field and will not be interpreted as having idealized meanings or overly formal meanings unless clearly defined in the various embodiments of the present invention.

[0069] Example 1

[0070] The disclosed embodiments provide a method for detecting highlight points in a long video, which is used to understand the content of a long video to be detected and extract highlight content in the long video to be detected.

[0071] See also Figure 1 The long video highlight point detection method includes steps S101 to S104, and each step is described in detail below.

[0072] Step S101, performing speech recognition on the long video to be detected according to a speech recognition algorithm to obtain a subtitle information set.

[0073] In this embodiment, a speech recognition algorithm is used to perform speech recognition on the long video to be detected, so as to extract the speech and subtitle information in the long video to be detected, and the speech and subtitle information are combined into a subtitle information set according to the corresponding time sequence. The speech recognition algorithm can be a Whisper algorithm, but it is not limited here.

[0074] See also Figure 2 In a specific embodiment, step S101 includes steps S1011 to S1014, and each step is described in detail below.

[0075] Step S1011, obtaining an audio file according to the long video to be detected.

[0076] In this embodiment, the audio file of the audio part is extracted from a given long video to be detected. This can usually be done by video processing software or libraries (such as FFmpeg), which can separate the audio and video streams in the video to obtain a pure audio file.

[0077] Step S1012: determining audio parameters according to the audio file, and calculating the number of audio blocks in the audio file according to the audio parameters.

[0078] In this embodiment, the audio parameters of the audio file are determined; at the same time, the audio file is divided into smaller audio blocks, and the number of audio blocks of all audio blocks is calculated according to the audio parameters.

[0079] The size of the audio block may be based on the time length (such as 1 second per block) or based on other audio features (such as silent intervals, volume changes, etc.).

[0080] For example, by reading the audio file, the audio parameters such as the audio frame width frame_width and the number of audio frames num_frames are obtained, and the number of audio blocks

[0081] Step S1013: determining a speaking area according to the audio parameters and the number of audio blocks.

[0082] In this embodiment, the speaking area of ​​the audio file is determined according to the audio parameters and the number of audio blocks. The speaking area is the audio area in a non-silent state.

[0083] Step S1014: Perform speech recognition on the speaking area according to the speech recognition algorithm to obtain a subtitle information set.

[0084] In this embodiment, the audio signal in the speaking area is processed using a speech recognition algorithm to identify and transcribe the speaking content, thereby generating a set of all recognized subtitle information. Figure 3As shown, "00:00:09,984—>00:00:16,128" is the time interval, and "Fruit tea has lactic acid bacteria..." is the subtitle information corresponding to the time interval.

[0085] See also Figure 4 Step S1013 includes steps S401 to S403, and each step is described in detail below.

[0086] Step S401: determining an audio energy value of each audio block according to the audio parameter and the number of audio blocks.

[0087] In this embodiment, the audio parameters also include a sample width sample_width and a number of channels n_channels. For the i-th audio block (i=1, 2, 3, ..., n_chunks), the audio energy value energies[i] of the i-th audio block is rms(chunk, sample_width×n_channels). The rms function is used to calculate the root mean square (RMS) value of the audio segment, which is a way to measure the strength or volume of the audio signal.

[0088] Step S402: Calculate an energy threshold based on the percentile according to each of the audio energy values.

[0089] In this embodiment, the energy threshold threshold = percentile(energies, 0.2). Percentile is the percentile calculated for a given array (in this embodiment, each audio energy value) on a specified axis. Percentile is a statistical measure that indicates that in a set of data, a certain percentage of data is less than or equal to the value. For example, the 50th percentile is the median, that is, half of the data in the data set is less than or equal to the value, and the other half is greater than or equal to the value.

[0090] Step S403: determining the speaking area according to the energy threshold and each of the audio energy values.

[0091] In this embodiment, the energy threshold is compared with each audio energy value to screen out a plurality of speech regions containing human voices, and possible silence, background noise or other non-human voice signals are eliminated.

[0092] See also Figure 5 In a specific embodiment, step S403 includes steps S4031 to S4042, and each step is described in detail below.

[0093] Step S4031: traverse the audio energy values ​​to determine whether the i-th audio energy value is greater than the energy threshold.

[0094] In this embodiment, each audio energy value is traversed to determine whether the i-th audio energy value is greater than an energy threshold, thereby determining whether the current state is mute.

[0095] Step S4032: if the i-th audio energy value is less than or equal to the energy threshold, the audio block corresponding to the i-th audio energy value is in a silent state; if the i-th audio energy value is greater than the energy threshold, the audio block corresponding to the i-th audio energy value is in a non-silent state.

[0096] In this embodiment, if the i-th audio energy value is less than or equal to the energy threshold, it can be known that the i-th audio block is in a silent state; if the i-th audio energy value is greater than the energy threshold, the i-th audio block is in a non-silent state. By clearly distinguishing between silent segments and non-silent segments, it is helpful to more accurately identify the speaking area in subsequent processing.

[0097] Step S4033, determining whether there is a start time for the speaking area at present.

[0098] In this embodiment, it is checked whether the start time of the speech area has been currently recorded.

[0099] Step S4034: If the start time of the speaking area does not currently exist and the audio block corresponding to the i-th audio energy value is in the non-silent state, the current time is used as the start time of the speaking area, and it is determined whether the i+1-th audio energy value is greater than the energy threshold.

[0100] In this embodiment, if the start time of the speaking area is not currently recorded and the i-th audio block is in a non-silent state, the current time is taken as the start time of the speaking area, and the i+1-th audio energy value is traversed.

[0101] Step S4035: If the start time of the speaking area does not currently exist and the audio block corresponding to the i-th audio energy value is in the silent state, determine whether the i+1-th audio energy value is greater than the energy threshold.

[0102] In this embodiment, if the start time of the speaking area is not currently recorded, but the audio block corresponding to the i-th audio energy value is in a silent state, the i+1-th audio energy value is directly traversed. Since the current audio block is silent, it cannot be the beginning of the speaking area.

[0103] Step S4036: If there is a start time for the speaking area, calculate the elapsed time between the start time and the time corresponding to the i-th audio energy value.

[0104] In this embodiment, if the start time of the speaking area is currently recorded, the elapsed time elapsed_time from the start time to the corresponding time of the i-th audio block is calculated.

[0105] Step S4037, determining whether the elapsed time is greater than or equal to a preset maximum area time.

[0106] In this embodiment, judging whether the elapsed time is greater than or equal to the preset maximum area time can accurately identify the end of the speaking area and avoid the excessively long speaking segment affecting subsequent processing.

[0107] Step S4038, if the elapsed time is greater than or equal to the preset maximum area time, the current time is used as the end time of the speaking area, and the start time of the speaking area is initialized, and it is determined whether the i+1th audio energy value is greater than the energy threshold.

[0108] In this embodiment, if the elapsed time is greater than or equal to the preset maximum area time, the current time is used as the end time of the speaking area, and the start time of the speaking area is initialized, that is, the start time of the speaking area is reset to 0, and it is determined whether the i+1th audio energy value is greater than the energy threshold to prepare to start detecting the next possible speaking area.

[0109] Step S4039: If the elapsed time is less than the preset maximum area time, determine whether the elapsed time is greater than or equal to the preset minimum area time.

[0110] In this embodiment, if the elapsed time is less than the preset maximum area time, then determining whether the elapsed time is greater than or equal to the preset minimum area time helps avoid mistaking brief noises or human voices for a speaking area, and also avoids misjudging brief pauses as the end of a speaking area.

[0111] Step S4040: If the elapsed time is greater than or equal to the preset minimum area time, and the audio block corresponding to the i-th audio energy value is in a silent state, the current time is used as the end time of the speaking area, and the start time of the speaking area is initialized to determine whether the i+1-th audio energy value is greater than the energy threshold.

[0112] In this embodiment, if the elapsed time is greater than or equal to the preset minimum area time, and the audio block corresponding to the i-th audio energy value is in a silent state, the current time is used as the end time of the speaking area, and the start time of the speaking area is initialized, that is, the start time of the speaking area is reset to 0, and it is determined whether the i+1-th audio energy value is greater than the energy threshold, so as to prepare to start detecting the next possible speaking area.

[0113] Step S4041: If the elapsed time is greater than or equal to the preset minimum area time, and the audio block corresponding to the i-th audio energy value is in a non-silent state, determine whether the i+1-th audio energy value is greater than the energy threshold.

[0114] In this embodiment, if the elapsed time is greater than or equal to the preset minimum area time, if the audio blocks corresponding to the i audio energy values ​​are non-silent, then continue to check the next audio block. This helps to accurately define the boundaries of the speaking area while maintaining sensitivity to short pauses.

[0115] Step S4042: If the elapsed time is less than the preset minimum area time, determine whether the (i+1)th audio energy value is greater than the energy threshold.

[0116] In this embodiment, if the elapsed time is less than the preset minimum region time, the next audio block is checked without changing the state of the current speaking region, which helps to ignore very short silence or non-human voice parts and maintain the continuity of the speaking region.

[0117] It should be noted that for each time interval [t_start, t_end] in the set regions obtained in the speaking area detection stage, the speech recognition algorithm is used to convert it into the corresponding text content S, thereby obtaining a series of subtitle information sets S = {S_j|[t_start, t_end]∈regions} sorted in order of time intervals.

[0118] Please see again Figure 1 , step S102, extracting a set of video description information of the long video to be detected through a large video language model.

[0119] In this embodiment, the video language model is used to extract the video description information of the long video to be detected, and the video description information is combined into a video description information set according to the corresponding time. The visual language model can be MiniCPM-V, but it is not limited here.

[0120] See also Figure 6 In a specific embodiment, step S102 includes steps S1021 to S1023, and each step is described in detail below.

[0121] Step S1021: dividing the long video to be detected into a plurality of short videos to be detected according to the time interval corresponding to the subtitle information set.

[0122] In this embodiment, the long video to be detected is divided into a plurality of short videos to be detected according to the time interval corresponding to each subtitle information in the subtitle information set.

[0123] Long videos may contain a large amount of content and scenes. By dividing them into multiple short videos according to the time interval corresponding to the subtitle information, the subsequent processing steps can be simplified and each short video to be detected can be accurately aligned with the corresponding subtitle information, which facilitates subsequent information fusion.

[0124] Step S1022: extracting video description information corresponding to each of the short videos to be detected through the video language model.

[0125] In this embodiment, the video language model is used to extract the video description information corresponding to the scene in each short video to be detected. After a large amount of training, the video language model can accurately extract the key information in the video and provide an accurate data basis for subsequent highlight point detection.

[0126] Step S1023: determining the video description information set according to the video description information corresponding to each of the short videos to be detected.

[0127] In this embodiment, each video description information is combined into a video description information set in chronological order. The video description information set is exemplified as follows: Figure 7 shown.

[0128] Please see again Figure 1 , step S103, merging the subtitle information set and the video description information set based on the time axis to obtain a fused text information set.

[0129] In this embodiment, the subtitle information set and the video description information set are fused according to the time axis, that is, the subtitle information is spliced ​​with the video description information at the corresponding time to obtain fused text information in which the subtitle information and the video description information are aligned.

[0130] The dialogue between characters and the content of the screen play a decisive role in whether the content is exciting. Extracting subtitle information and video description information from audio and video to form fused text information will help large models analyze and understand what is happening in long videos.

[0131] In a specific embodiment, the subtitle information set includes subtitle information corresponding to multiple time intervals, and the video description information set includes video description information corresponding to multiple time intervals. Step S103 includes: for each time interval, fusing the subtitle information and video description information corresponding to the same time interval to obtain fused text information; and determining the fused text information set according to the fused text information corresponding to each time interval.

[0132] In this embodiment, the subtitle information and video description information corresponding to the same time interval are fused to obtain fused text information; the fused text information corresponding to each time interval is combined into a fused text information set, thereby providing a data basis of both voice information and video description information for subsequent video highlight detection, and the fused text information set based on time alignment can ensure the accuracy of highlight detection. Figure 8 shown.

[0133] Step S104, detecting the fused text information set according to the highlight prompt words through a natural language large model to obtain the highlight point time period and its corresponding highlight content information.

[0134] In this embodiment, multiple dimensions of wonderful moments are defined to obtain highlight prompt words; the highlight prompt words are used to guide the large model to find highlight information of the fused text information set, so as to obtain the highlight time period and its corresponding highlight content information that meet the user's needs. The highlight prompt words can involve multiple dimensions such as the fun, topicality, and memes of the video.

[0135] Exemplarily, the input highlight prompt words are: "This is the subtitle text and screen segment description text of a variety show video. The subtitle text and short video description are marked with time and content. You need to have a comprehensive understanding of the subtitle text and screen description. You need to output 15 highlight contents based on these contents. The highlight contents focus on {user's definition of wonderful moments}. The duration of each content must be more than 30 seconds and less than 1 minute. Please output their start and end positions, content titles, content descriptions, and reasons for judging them as highlight contents. These 15 content segments need to be sorted and output according to the degree of highlight". Examples of highlight time periods and their corresponding highlight content information output by the natural language large model are as follows: Fig. 9 shown.

[0136] In a specific embodiment, step S104 includes: performing feature extraction on the fused text information set through the natural language large model to obtain a feature set; determining a highlight content set from the feature set according to the highlight prompt words; and determining the highlight point time period and the highlight content information according to the highlight content set.

[0137] In this embodiment, the fused text information set is set to T, which is composed of the subtitle information S and the video description information D of the long video to be detected, that is, T = S + D. The highlight prompt word is set to P.

[0138] Furthermore, the fused text information set T and the highlighted word P are combined into the input I of the natural language large model, which can be expressed as I = (T, P).

[0139] Let the natural language model be M, and input I into the natural language model M for processing, that is, M(I). The natural language model M will perform multi-level feature extraction and semantic understanding on the text information in I based on its internal complex neural network structure and parameters.

[0140] For the subtitle information S and video description information D in the input T, the natural language model M extracts features such as keywords, semantic roles, emotional tendencies, etc., and the extracted feature set is set to be F.

[0141] Furthermore, based on the requirements for highlight content in the feature set F and the highlight prompt word P (especially "user's definition of wonderful moments"), the natural language model M determines which parts belong to the highlight content. Let the determined highlight content set be H, and determine the highlight time period and highlight content information based on the highlight content set H. Each highlight content h_i (i=1,2,…,n, n≥15) includes the start time t_starti, the end time t_endi, the content title Title_i, the content description Description_i, and the reason Reason_i for judging it as highlight content. The start time t_starti and the end time t_endi are combined into the highlight time period, and the content title Title_i, the content description Description_i, and the reason Reason_i for judging it as highlight content are combined into the highlight content information.

[0142] It should be noted that the natural language model uses a large amount of text data during the training process, which enables it to learn rich language expressions and semantic understanding patterns. For the subtitles and description texts related to variety shows, it can use the knowledge it has learned to accurately understand the text content.

[0143] The natural language model has a strong ability to understand semantics and can parse complex semantic relationships in text. When processing subtitles and video description texts, it can understand key information such as plot development and character relationships, and accurately determine highlight content based on the "user's definition of highlight moments."

[0144] The natural language model can extract multiple text features and conduct a comprehensive analysis of these features. For example, it can combine sentiment words and key action descriptions in subtitles with scene features in video descriptions to comprehensively evaluate whether a certain moment is a highlight moment.

[0145] The natural language model can perceive the contextual information of the text. For the subtitles and description texts of long videos, this contextual awareness ability helps to more accurately judge the importance of a certain segment in the entire video, and then determine the highlight content and sort it reasonably according to its importance.

[0146] The long video highlight point detection method proposed in this embodiment performs speech recognition on the long video to be detected according to the speech recognition algorithm to obtain a subtitle information set; extracts the video description information set of the long video to be detected through the video language large model; fuses the subtitle information set and the video description information set based on the time axis to obtain a fused text information set; detects the fused text information set according to the highlight prompt words through the natural language large model to obtain the highlight point time period and its corresponding highlight content information. In this way, by extracting the subtitle information and video description information of the long video to be detected for highlight point detection, the video highlight point information can be obtained without relying on data set training, and the highlight point content that meets user needs can be extracted according to the highlight prompt words.

[0147] Example 2

[0148] In addition, the present disclosure provides a long video highlight point detection device 1000, see Fig.10 , the device comprises:

[0149] The recognition module 1001 is used to perform speech recognition on the long video to be detected according to the speech recognition algorithm to obtain a subtitle information set;

[0150] An extraction module 1002 is used to extract a video description information set of the long video to be detected through a video language large model;

[0151] A fusion module 1003, configured to fuse the subtitle information set and the video description information set based on a time axis to obtain a fused text information set;

[0152] The detection module 1004 is used to detect the fused text information set according to the highlight prompt words through the natural language large model to obtain the highlight point time period and its corresponding highlight content information.

[0153] Optionally, the recognition module 1001 is also used to obtain an audio file based on the long video to be detected; determine audio parameters based on the audio file, and calculate the number of audio blocks in the audio file based on the audio parameters; determine a speaking area based on the audio parameters and the number of audio blocks; perform speech recognition on the speaking area according to the speech recognition algorithm to obtain a subtitle information set.

[0154] Optionally, the identification module 1001 is further used to determine the audio energy value of each audio block according to the audio parameters and the number of audio blocks; calculate an energy threshold based on each audio energy value based on percentiles; and determine the speaking area according to the energy threshold and each audio energy value.

[0155] Optionally, the identification module 1001 is further used to traverse each of the audio energy values ​​to determine whether the i-th audio energy value is greater than the energy threshold; if the i-th audio energy value is less than or equal to the energy threshold, the audio block corresponding to the i-th audio energy value is in a silent state; if the i-th audio energy value is greater than the energy threshold, the audio block corresponding to the i-th audio energy value is in a non-silent state; determine whether there is a start time of the speaking area at present; if there is no start time of the speaking area at present, and the audio block corresponding to the i-th audio energy value is in the non-silent state, take the current time as the start time of the speaking area, and determine whether the i+1-th audio energy value is greater than the energy threshold; if there is no start time of the speaking area at present, and the audio block corresponding to the i-th audio energy value is in the silent state, determine whether the i+1-th audio energy value is greater than the energy threshold; if there is a start time of the speaking area at present, calculate the elapsed time between the start time and the time corresponding to the i-th audio energy value; determine Determine whether the elapsed time is greater than or equal to a preset maximum area time; if the elapsed time is greater than or equal to the preset maximum area time, use the current time as the end time of the speaking area, initialize the start time of the speaking area, and determine whether the i+1th audio energy value is greater than the energy threshold; if the elapsed time is less than the preset maximum area time, determine whether the elapsed time is greater than or equal to the preset minimum area time; if the elapsed time is greater than or equal to the preset minimum area time, and the audio block corresponding to the i-th audio energy value is in a silent state, use the current time as the end time of the speaking area, initialize the start time of the speaking area, and determine whether the i+1th audio energy value is greater than the energy threshold; if the elapsed time is greater than or equal to the preset minimum area time, and the audio block corresponding to the i-th audio energy value is in a non-silent state, determine whether the i+1th audio energy value is greater than the energy threshold; if the elapsed time is less than the preset minimum area time, determine whether the i+1th audio energy value is greater than the energy threshold.

[0156] Optionally, the extraction module 1002 is also used to divide the long video to be detected into multiple short videos to be detected according to the time interval corresponding to the subtitle information set; extract the video description information corresponding to each of the short videos to be detected through the video language large model; and determine the video description information set according to the video description information corresponding to each of the short videos to be detected.

[0157] Optionally, the subtitle information set includes subtitle information corresponding to multiple time intervals, and the video description information set includes video description information corresponding to multiple time intervals; the fusion module 1003 is also used to fuse the subtitle information and video description information corresponding to the same time interval for each time interval to obtain fused text information; and determine the fused text information set according to the fused text information corresponding to each time interval.

[0158] Optionally, the detection module 1004 is further used to perform feature extraction on the fused text information set through the natural language large model to obtain a feature set; determine a highlight content set from the feature set according to the highlight prompt words; and determine the highlight point time period and the highlight content information according to the highlight content set.

[0159] The device provided in the embodiment of the present disclosure can execute the steps of the long video highlight point detection method provided in Example 1, which will not be described again to avoid repetition.

[0160] The long video highlight point detection device proposed in this embodiment performs speech recognition on the long video to be detected according to the speech recognition algorithm to obtain a subtitle information set; extracts the video description information set of the long video to be detected through the video language large model; fuses the subtitle information set and the video description information set based on the time axis to obtain a fused text information set; detects the fused text information set according to the highlight prompt words through the natural language large model to obtain the highlight point time period and its corresponding highlight content information. In this way, by extracting the subtitle information and video description information of the long video to be detected to perform highlight point detection, the video highlight point information can be obtained without relying on data set training, and the highlight point content that meets the user's needs can be extracted according to the highlight prompt words.

[0161] Example 3

[0162] In addition, an embodiment of the present disclosure provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the long video highlight point detection method described in Example 1 is implemented.

[0163] The device provided in the embodiment of the present disclosure can execute the steps of the long video highlight point detection method provided in Example 1, which will not be described again to avoid repetition.

[0164] Example 4

[0165] The embodiment of the present disclosure provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the long video highlight point detection method described in Embodiment 1 is implemented.

[0166] In this embodiment, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0167] The computer-readable storage medium provided in this embodiment can implement the long video highlight point detection method provided in Example 1, and will not be described again here to avoid repetition.

[0168] In all examples shown and described herein, any specific values ​​should be interpreted as merely exemplary and not as limiting, and thus other examples of the exemplary embodiments may have different values.

[0169] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0170] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the present invention. It should be pointed out that for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention.

Claims

1. A method for detecting highlights in a long video, characterized in that: include: Perform speech recognition on the long video to be detected according to the speech recognition algorithm to obtain a subtitle information set; Extracting a video description information set of the long video to be detected by using a large video language model; Merging the subtitle information set and the video description information set based on a time axis to obtain a fused text information set; The fused text information set is detected according to the highlight prompt words through a large natural language model to obtain the highlight time period and its corresponding highlight content information.

2. The method for detecting highlight points in a long video according to claim 1, characterized in that: The method performs speech recognition on the long video to be detected according to the speech recognition algorithm to obtain a subtitle information set, including: Obtaining an audio file according to the long video to be detected; Determine audio parameters according to the audio file, and calculate the number of audio blocks in the audio file according to the audio parameters; Determine a speaking area according to the audio parameter and the number of audio blocks; Speech recognition is performed on the speaking area according to the speech recognition algorithm to obtain a subtitle information set.

3. The method for detecting highlight points in a long video according to claim 2, characterized in that: The determining the speaking area according to the audio parameter and the number of audio blocks includes: Determine an audio energy value of each audio block according to the audio parameter and the number of audio blocks; Calculating an energy threshold according to each of the audio energy values ​​based on the percentile; The speaking area is determined according to the energy threshold and each of the audio energy values.

4. The method for detecting highlight points in a long video according to claim 3, characterized in that: The determining the speaking area according to the energy threshold and each of the audio energy values ​​comprises: Traversing each of the audio energy values, and determining whether the i-th audio energy value is greater than the energy threshold; If the i-th audio energy value is less than or equal to the energy threshold, the audio block corresponding to the i-th audio energy value is in a silent state; if the i-th audio energy value is greater than the energy threshold, the audio block corresponding to the i-th audio energy value is in a non-silent state; Determine whether there is a start time for the speaking area currently; If the start time of the speaking area does not currently exist, and the audio block corresponding to the i-th audio energy value is in the non-silent state, the current time is taken as the start time of the speaking area, and it is determined whether the i+1-th audio energy value is greater than the energy threshold; If the start time of the speaking area does not currently exist and the audio block corresponding to the i-th audio energy value is in the silent state, determining whether the i+1-th audio energy value is greater than the energy threshold; If there is a start time for the speaking area currently, then calculating the elapsed time between the start time and the time corresponding to the i-th audio energy value; Determining whether the elapsed time is greater than or equal to a preset maximum area time; If the elapsed time is greater than or equal to the preset maximum area time, the current time is used as the end time of the speaking area, and the start time of the speaking area is initialized, and it is determined whether the (i+1)th audio energy value is greater than the energy threshold; If the elapsed time is less than the preset maximum area time, determining whether the elapsed time is greater than or equal to the preset minimum area time; If the elapsed time is greater than or equal to the preset minimum area time, and the audio block corresponding to the i-th audio energy value is in a silent state, the current time is used as the end time of the speaking area, and the start time of the speaking area is initialized, and it is determined whether the i+1-th audio energy value is greater than the energy threshold; If the elapsed time is greater than or equal to the preset minimum area time, and the audio block corresponding to the i-th audio energy value is in a non-silent state, determining whether the i+1-th audio energy value is greater than the energy threshold; If the elapsed time is less than the preset minimum area time, it is determined whether the (i+1)th audio energy value is greater than the energy threshold.

5. The long video highlight point detection method according to claim 1, characterized in that: The step of extracting the video description information set of the long video to be detected by using the video language large model includes: Dividing the long video to be detected into a plurality of short videos to be detected according to the time interval corresponding to the subtitle information set; Extracting video description information corresponding to each of the short videos to be detected by using the video language large model; The video description information set is determined according to the video description information corresponding to each of the short videos to be detected.

6. The method for detecting highlight points in a long video according to claim 1, characterized in that: The subtitle information set includes subtitle information corresponding to a plurality of time intervals, and the video description information set includes video description information corresponding to a plurality of time intervals; the subtitle information set and the video description information set are merged based on the time axis to obtain a merged text information set, including: For each time interval, the subtitle information and video description information corresponding to the same time interval are merged to obtain fused text information; The fused text information set is determined according to the fused text information corresponding to each time interval.

7. The method for detecting highlight points in a long video according to claim 1, characterized in that: The fused text information set is detected according to the highlight prompt words by using the natural language big model to obtain the highlight point time period and its corresponding highlight content information, including: Extracting features from the fused text information set using the natural language large model to obtain a feature set; Determining a highlight content set from the feature set according to the highlight prompt word; The highlight point time period and the highlight content information are determined according to the highlight content set.

8. A long video highlight point detection device, characterized in that: include: A recognition module, used to perform speech recognition on the long video to be detected according to a speech recognition algorithm to obtain a subtitle information set; An extraction module, used for extracting a video description information set of the long video to be detected through a large video language model; A fusion module, used for fusing the subtitle information set and the video description information set based on a time axis to obtain a fused text information set; The detection module is used to detect the fused text information set according to the highlight prompt words through a large natural language model to obtain the highlight time period and its corresponding highlight content information.

9. A computer device, characterized in that: It includes a memory and a processor, the memory stores a computer program, and when the computer program is executed by the processor, the long video highlight point detection method as described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: It stores a computer program, which, when executed by a processor, implements the long video highlight point detection method as described in any one of claims 1 to 7.

Images