Video processing methods, apparatus, equipment and media

Abstract

This application relates to a video processing method, apparatus, device, and medium, belonging to the field of network media and involving artificial intelligence technology. The method includes: acquiring a video and extracting audio from the video; locating at least one background audio segment belonging to the background music category and at least one human voice audio segment belonging to the human voice category within the audio; acquiring the video's theme tags and determining a sub-category tag for each background audio segment; locating at least one target audio segment within the audio, each target audio segment including at least one complete background audio segment and at least one complete human voice audio segment, the union of the included background audio segments and human voice audio segments being continuous, and the sub-category tags of the included background audio segments matching the video's theme tags; and extracting at least one video segment from the video according to at least one target audio segment. This method avoids wasting computer hardware resources used to support video segment extraction.

Description

Technical Field

[0001] This application relates to artificial intelligence technology, and more particularly to the field of online media, and in particular to a video processing method, apparatus, device, and medium. Background Technology

[0002] With the development of online media technology, in many scenarios, it is necessary to extract video segments that meet user needs from long video clips. For example, on short video creation platforms, many online streamers use currently popular movies and TV dramas to create short video clips. They can extract exciting scenes from these shows. Another example is in the post-production stage of film and television production, where producers extract exciting scenes from each episode to create preview clips.

[0003] In traditional techniques, video segments are usually extracted directly from the video based on the background music. However, this method often results in incomplete audio recordings of human voices that do not match the video's theme, leading to low accuracy and wasting computer hardware resources used to extract the desired video segments. Summary of the Invention

[0004] Therefore, it is necessary to provide a video processing method, apparatus, device, and medium that can avoid wasting computer hardware resources used to support video segment capture, in order to address the aforementioned technical problems.

[0005] In a first aspect, this application provides a video processing method, the method comprising:

[0006] Acquire video and extract audio from video;

[0007] In the audio, at least one background audio segment belonging to the background music category and at least one human voice audio segment belonging to the human voice category are located.

[0008] Obtain the topic tags of the video and determine the subcategory tags for each background audio segment;

[0009] In the audio, at least one target audio segment is located, each target audio segment including at least one complete background audio segment and at least one complete human voice audio segment, the union of the included background audio segments and human voice audio segments is continuous, and the subcategory label of the included background audio segments matches the topic label of the video;

[0010] At least one video segment is extracted from the video according to the at least one target audio segment.

[0011] Secondly, this application provides a video processing apparatus, the apparatus comprising:

[0012] An acquisition module is used to acquire video and extract audio from the video;

[0013] The positioning module is used to locate at least one background audio segment belonging to the background music category and at least one human voice audio segment belonging to the human voice category in the audio.

[0014] The determination module is used to obtain the theme tags of the video and determine the sub-category tags for each background audio segment;

[0015] The positioning module is further configured to locate at least one target audio segment in the audio, each target audio segment including at least one complete background audio segment and at least one complete human voice audio segment, wherein the union of the included background audio segment and human voice audio segment is continuous, and the subcategory label of the included background audio segment matches the topic label of the video.

[0016] The extraction module is used to extract at least one video segment from the video according to the at least one target audio segment.

[0017] In one embodiment, the positioning module is further configured to acquire multiple temporal audio frames obtained by framing the audio; determine the audio category to which the multiple temporal audio frames belong, based on the multiple temporal audio frames, wherein the audio category is a background music category or a human voice category; and locate at least one background audio segment belonging to the background music category and at least one human voice audio segment belonging to the human voice category in the audio based on the audio category to which the multiple temporal audio frames belong.

[0018] In one embodiment, the positioning module is further configured to convert the plurality of time-domain audio frames to the frequency domain to obtain a plurality of audio frequency domain signals; extract time-domain features from the plurality of time-domain audio frames, extract frequency-domain features from the plurality of audio frequency domain signals, and fuse the time-domain features and the frequency-domain features to obtain audio fusion features; and determine the audio category to which the plurality of time-domain audio frames belong based on the audio fusion features.

[0019] In one embodiment, the positioning module is further configured to extract features from the plurality of time-domain audio frames layer by layer according to a preset plurality of levels to obtain time-domain features of each level; extract features from the plurality of audio frequency-domain signals layer by layer according to the plurality of levels to obtain frequency-domain features of each level; for each level, perform feature fusion based at least on the time-domain features and frequency-domain features of the targeted level to generate layer fusion features for the targeted level, wherein, when generating layer fusion features for levels after the first level, feature fusion is also performed based on the layer fusion features of the previous level of the targeted level; and determine audio fusion features based on the layer fusion features of the last level among the plurality of levels.

[0020] In one embodiment, the time-domain features and the frequency-domain features have different feature dimensions; the localization module is further configured to, for each layer, reconstruct the time-domain features of the targeted layer according to the feature dimensions of the frequency-domain features of the targeted layer, to obtain the reconstructed time-domain features of the targeted layer; when the targeted layer is the first layer, the frequency-domain features of the targeted layer and the reconstructed time-domain features are fused to obtain the layer-fused features of the targeted layer; when the targeted layer is a layer after the first layer, the frequency-domain features of the targeted layer, the reconstructed time-domain features of the targeted layer, and the layer-fused features of the layer preceding the targeted layer are fused to obtain the layer-fused features of the targeted layer.

[0021] In one embodiment, the layer fusion feature includes multiple feature maps; the localization module is further configured to fuse each of the feature maps in the layer fusion feature of the last layer among the multiple layers to obtain a fused feature map; and to reduce the dimensionality of the fused feature map to obtain an audio fusion feature with the same feature dimension as each of the temporal features before reconstruction.

[0022] In one embodiment, the audio category to which the plurality of time-domain audio frames belong is determined by predicting the probability that each of the plurality of time-domain audio frames belongs to each audio category through a pre-trained audio classification model; the audio classification model is obtained through a model training step, and the apparatus further includes:

[0023] A training module is used to acquire training samples, which include sample audio and the audio category to which the sample audio belongs, labeled for the sample audio. The sample audio is input into an audio classification model to be trained, so that the audio classification model predicts the category of each audio frame in the sample audio, obtaining the probability that each audio frame in the sample audio belongs to each audio category. Based on the probability that each audio frame in the sample audio belongs to each audio category, the probability that the sample audio belongs to each audio category is determined. Based on the probability that the sample audio belongs to each audio category and the audio category to which the sample audio belongs, a loss value is determined. The audio classification model to be trained is trained in the direction that reduces the loss value.

[0024] In one embodiment, the training module is further configured to, for each audio category, determine multiple candidate probabilities that the sample audio belongs to the target audio category based on the probability of each audio frame in the sample audio belonging to the target audio category using multiple preset functions; fuse the multiple candidate probabilities that the sample audio belongs to the target audio category using multiple fusion methods to obtain multiple fused probabilities that the sample audio belongs to the target audio category; and determine the probability that the sample audio belongs to the target audio category based on the multiple fused probabilities that the sample audio belongs to the target audio category.

[0025] In one embodiment, the determining module is further configured to: determine the background audio features of each background audio segment; determine the confidence level of each background audio segment belonging to a plurality of preset sub-category labels based on the background audio features; each sub-category label is used to characterize a musical mood of the background audio segment; and determine the sub-category label of the background audio segment based on the confidence level of each background audio segment belonging to a plurality of preset sub-category labels.

[0026] In one embodiment, each background audio segment included in the target audio segment intersects with at least one human voice audio segment included.

[0027] In one embodiment, for each background audio segment in the target audio segment, the proportion of time of at least one human voice audio segment that intersects with the target background audio segment in the target background audio segment exceeds a preset ratio.

[0028] In one embodiment, the video is a film or television drama video; the voice category is a voice dialogue category; the voice audio segment is a voice dialogue audio segment; the theme tag is a theme tag of the film or television drama video; and the video segment is an edited clip of the film or television drama video.

[0029] Thirdly, this application provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the various method embodiments of this application.

[0030] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in the various method embodiments of this application.

[0031] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the steps in the various method embodiments of this application.

[0032] The aforementioned video processing method, apparatus, device, medium, and computer program product acquire video and extract audio from the video; in the audio, locate at least one background audio segment belonging to the background music category and at least one human voice audio segment belonging to the human voice category; acquire the video's theme tag and determine the sub-category tag for each background audio segment; in the audio, locate at least one target audio segment, each target audio segment including at least one complete background audio segment and at least one complete human voice audio segment, the union of the included background audio segments and human voice audio segments being continuous, and the sub-category tag of the included background audio segment matching the video's theme tag; and extract at least one video segment from the video according to at least one target audio segment. Compared to traditional methods that directly extract video segments from videos based on background music, this application identifies human voice audio segments and background audio segments matching the video theme from the video's audio. By combining the identified human voice and background audio segments, a target audio segment matching the video theme, including both complete background and human voice audio segments, is located from the video's audio. Then, based on the located target audio segment, a video segment matching the video theme, including both complete background and human voice audio segments, is extracted from the video. This improves the accuracy of video segment extraction and avoids wasting computer hardware resources used to support the extraction of the required video segment. Attached Figure Description

[0033] Figure 1 This is an application environment diagram of a video processing method in one embodiment;

[0034] Figure 2 This is a flowchart illustrating a video processing method in one embodiment;

[0035] Figure 3 This is a schematic diagram of the processing procedure of an audio classification model in one embodiment;

[0036] Figure 4 This is a schematic diagram of the framework of an audio coding unit in an audio classification model in one embodiment;

[0037] Figure 5 This is a schematic diagram illustrating the process of obtaining the probability that a sample audio belongs to a specific audio category in one embodiment.

[0038] Figure 6 This is a flowchart illustrating the process of classifying background audio segments using a music emotion recognition model in one embodiment.

[0039] Figure 7 This is a schematic diagram illustrating the background audio segment filtering principle in one embodiment;

[0040] Figure 8 This is a schematic diagram illustrating the target audio segment localization principle in one embodiment;

[0041] Figure 9 This is a schematic diagram of the overall framework of a video processing method in one embodiment;

[0042] Figure 10 This is a schematic diagram of the overall framework of the video processing method in another embodiment;

[0043] Figure 11 This is a flowchart illustrating a video processing method in another embodiment;

[0044] Figure 12 This is a structural block diagram of a video processing device in one embodiment;

[0045] Figure 13 This is a structural block diagram of the video processing apparatus in another embodiment;

[0046] Figure 14 This is an internal structural diagram of a computer device in one embodiment;

[0047] Figure 15 This is a diagram of the internal structure of a computer device in another embodiment. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0049] The video processing method provided in this application can be applied to, for example... Figure 1In the application environment shown, terminal 102 communicates with server 104 via a network. The data storage system can be set up separately and can store the data that server 104 needs to process. The data storage system can be integrated into server 104 or placed in the cloud or on other servers. Terminal 102 can be, but is not limited to, various desktop computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, etc. Portable wearable devices can include smartwatches, smart bracelets, head-mounted devices, etc. Server 104 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, cloud security, host security and other network security services, CDN, and basic cloud computing services such as big data and artificial intelligence platforms. Terminal 102 and server 104 can be directly or indirectly connected via wired or wireless communication, which is not limited herein.

[0050] Server 104 can acquire video and extract audio from it. Within the audio, it locates at least one background audio segment belonging to the background music category and at least one human voice audio segment belonging to the human voice category. Server 104 can acquire the video's theme tags and determine the sub-category tags for each background audio segment. Server 104 can locate at least one target audio segment within the audio. Each target audio segment includes at least one complete background audio segment and at least one complete human voice audio segment. The union of the included background audio segments and human voice audio segments is contiguous, and the sub-category tags of the included background audio segments match the video's theme tags. Server 104 can extract at least one video segment from the video based on at least one target audio segment.

[0051] It is understood that server 104 may pre-store videos, and server 104 may directly retrieve videos from its local storage. It is also understood that terminal 102 may send videos to server 104, and server 104 may receive videos. Server 104 may store video segments extracted from the video locally, and it is understood that server 104 may also send video segments extracted from the video to terminal 102. This embodiment does not limit this; it is understood that... Figure 1 The application scenarios shown are for illustrative purposes only and are not limited to these.

[0052] It should be noted that some embodiments of this application utilize artificial intelligence (AI) technology in their video processing methods. For example, background audio segments and human voice audio segments are located from the video audio using AI technology, and the sub-category labels of the background audio segments are also predicted using AI technology. To better understand AI, the concept of AI is explained below. Specifically, AI is the theory, methods, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess perception, reasoning, and decision-making capabilities. This application, based on AI technology, extracts video segments from videos, which can further improve the accuracy of video segment extraction, thereby further avoiding the waste of computer hardware resources used to support the extraction of the required video segments.

[0053] In one embodiment, such as Figure 2 As shown, a video processing method is provided. This method can be applied to a computer device, which can be a terminal or a server. That is, the method can be executed independently by the terminal or the server, or it can be implemented through interaction between the terminal and the server. This embodiment illustrates the application of this method to a computer device as an example, including the following steps:

[0054] Step 202: Obtain the video and extract the audio from the video.

[0055] Specifically, computer devices can acquire video to be processed, and it is understood that the video carries audio. Computer devices can extract the audio from the video.

[0056] In one embodiment, the computer device can decode video. It is understood that video is typically compressed using a preset encoding format. Decoding the video converts the compressed data back to the original format, allowing the computer device to directly extract audio from the original video. It is also understood that audio is usually encapsulated in the video's audio track; after decoding the video, the computer device can extract the audio from the audio track.

[0057] Step 204: In the audio, locate at least one background audio segment belonging to the background music category and at least one human voice audio segment belonging to the human voice category.

[0058] The background music category is categorized based on the presence of background music in the audio. The vocal category is categorized based on the presence of vocals in the audio. In short, if an audio clip contains background music, it belongs to the background music category. If an audio clip contains vocals, it belongs to the vocal category. Background audio segments are audio segments belonging to the background music category. Vocal audio segments are audio segments belonging to the vocal category. In short, background audio segments are located from the audio based on the background music, and vocal audio segments are located from the audio based on the vocals.

[0059] Specifically, the audio may include background music and human voices. The computer device may locate at least one background audio segment belonging to the background music category from the audio based on the background music, and locate at least one human voice audio segment belonging to the human voice category from the audio based on the human voices.

[0060] It's understandable that background audio segments and vocal audio segments may or may not overlap in the audio. When the background audio segments and vocal audio segments overlap, it means that the background audio segment contains both background music and vocals. The vocal audio segment contains both vocals and background music. When the background audio segments and vocal audio segments do not overlap, it means that the background audio segment contains background music but no vocals. The vocal audio segment contains vocals but no background music.

[0061] In one embodiment, the computer device can extract audio features from the audio and, based on the audio features, locate at least one background audio segment belonging to the background music category in the audio, and, based on the audio features, locate at least one human voice audio segment belonging to the human voice category in the audio.

[0062] In one embodiment, a computer device may acquire a pre-trained audio classification model and input audio into the pre-trained audio classification model to locate at least one background audio segment belonging to the background music category and at least one human voice audio segment belonging to the human voice category in the audio through the pre-trained audio classification model.

[0063] Step 206: Obtain the video's theme tags and determine the sub-category tags for each background audio segment.

[0064] The video's theme tag describes the video's content theme. The background audio segment's subcategory tag describes the mood or mood of the background music.

[0065] Specifically, video producers can pre-label the video with themes, and once the computer acquires the video, it can directly retrieve these themes. Furthermore, the computer can determine the subcategory label for each background audio segment.

[0066] In one embodiment, the video's main tag and the background audio segment's sub-category tag can both include at least one of the following: sad, happy, sweet, healing, longing, cathartic, inspirational, touching, tense, epic, struggle, or funny.

[0067] In one embodiment, the computer device may extract background audio features for each background audio segment separately. For each background audio segment, the computer device may determine a sub-category label for each background audio segment based on the background audio features of the targeted background audio segment.

[0068] In one embodiment, a computer device can acquire a pre-trained music emotion recognition model and input each background audio segment into the pre-trained music emotion recognition model. The model then extracts background audio features for each background audio segment and determines a sub-category label for each segment based on these features. Each sub-category label represents a specific musical emotion within the background audio segment.

[0069] Step 208: In the audio, locate at least one target audio segment, each target audio segment including at least one complete background audio segment and at least one complete human voice audio segment, the union of the included background audio segments and human voice audio segments is continuous, and the subcategory label of the included background audio segments matches the topic label of the video.

[0070] A complete background audio segment refers to a segment of background music played without interruption. For example, a complete background audio segment could include a single section of background music or a complete piece of background music. A complete vocal audio segment refers to a segment of vocals played without interruption. For example, a complete vocal audio segment could include a single sentence or a paragraph. The union of the background audio segments and vocal audio segments included in each target audio segment is continuous, meaning the union of the time intervals of the background audio segments and the vocal audio segments included in each target audio segment is continuous. The sub-category labels of the background audio segments included in each target audio segment match the video's theme labels, meaning that the sub-category labels of the background audio segments included in each target audio segment have at least one identical or similar label to the video's theme labels.

[0071] Specifically, the computer device can locate at least one target audio segment in the audio based on at least one background audio segment and at least one human voice audio segment located from the audio. Each located target audio segment includes at least one complete background audio segment and at least one complete human voice audio segment. The union of the time periods of the included background audio segment and the time periods of the human voice audio segment is continuous, and the subcategory label of the included background audio segment has at least one identical or similar label to the video's topic label.

[0072] In one embodiment, for each target audio segment, the union of the background audio segments and the human voice audio segments included in the target audio segment is the target audio segment. This avoids the target audio segment from including redundant audio segments, thereby further improving the accuracy of video segment extraction and preventing the waste of computer hardware resources used to support the extraction of the required video segments.

[0073] In one embodiment, for each target audio segment, the union of the background audio segment and the human voice audio segment included in the target audio segment is smaller than the target audio segment.

[0074] Step 210: Extract at least one video segment from the video according to at least one target audio segment.

[0075] Specifically, the computer device can extract at least one video segment from the video based on at least one target audio segment obtained through positioning. It can be understood that the computer device can acquire the audio in-point and audio out-point of each target audio segment in the audio, and extract the video based on the time in-point and time out-point of each target audio segment in the audio, thus obtaining at least one video segment. It can be understood that the time in-point and time out-point of each video segment are consistent with the time in-point and time out-point of the corresponding target audio segment. Here, the time in-point of the target audio segment in the audio is the starting time point of the target audio segment in the audio. The time out-point of the target audio segment in the audio is the ending time point of the target audio segment in the audio. Similarly, the time in-point of the video segment in the video is the starting time point of the video segment in the video. The time out-point of the video segment in the video is the ending time point of the video segment in the video.

[0076] In the above video processing method, the following steps are taken: First, a video is acquired, and audio is extracted from the video. Then, at least one background audio segment belonging to the background music category and at least one human voice audio segment belonging to the human voice category are located within the audio. Next, the video's theme tags are acquired, and sub-category tags for each background audio segment are determined. Finally, at least one target audio segment is located within the audio, each target audio segment including at least one complete background audio segment and at least one complete human voice audio segment. The union of the included background audio segments and human voice audio segments is continuous, and the sub-category tags of the included background audio segments match the video's theme tags. Finally, at least one video segment is extracted from the video according to at least one target audio segment. Compared to traditional methods that directly extract video segments from videos based on background music, this application identifies human voice audio segments and background audio segments matching the video theme from the video's audio. By combining the identified human voice and background audio segments, a target audio segment matching the video theme, including both complete background and human voice audio segments, is located from the video's audio. Then, based on the located target audio segment, a video segment matching the video theme, including both complete background and human voice audio segments, is extracted from the video. This improves the accuracy of video segment extraction and avoids wasting computer hardware resources used to support the extraction of the required video segment.

[0077] In one embodiment, locating at least one background audio segment belonging to the background music category and at least one human voice audio segment belonging to the human voice category in the audio includes: acquiring multiple temporal audio frames obtained by framing the audio; determining the audio category to which the multiple temporal audio frames belong, wherein the audio category is either the background music category or the human voice category, based on the audio categories to which the multiple temporal audio frames belong; and locating at least one background audio segment belonging to the background music category and at least one human voice audio segment belonging to the human voice category in the audio based on the audio categories to which the multiple temporal audio frames belong.

[0078] Audio includes audio time-domain signals. A time-domain audio frame is the smallest unit of an audio time-domain signal; therefore, an audio time-domain signal can be understood to contain multiple time-domain audio frames.

[0079] Specifically, a computer device can frame audio located in the time domain, obtaining multiple time-domain audio frames. Based on these multiple time-domain audio frames, it can determine the audio category to which each time-domain audio frame belongs, where the audio category is either background music or vocals. It is understood that each time-domain audio frame can belong to at least one of the background music or vocal categories; it is also understood that each time-domain audio frame can belong only to the background music category, only to the vocal category, or simultaneously to both. In the audio, the computer device can locate at least one background audio segment belonging to the background music category and at least one vocal audio segment belonging to the vocal category based on the audio categories to which the multiple time-domain audio frames belong.

[0080] In one embodiment, the computer device can extract features from multiple time-domain audio frames to obtain audio frame features corresponding to each of the multiple time-domain audio frames. Based on the audio frame features corresponding to each of the multiple time-domain audio frames, the computer device can determine the audio category to which each of the multiple time-domain audio frames belongs, wherein the audio category is either background music or human voice.

[0081] In one embodiment, a computer device may input multiple temporal audio frames into a pre-trained audio classification model to extract features from the multiple temporal audio frames, obtain audio frame features corresponding to each of the multiple temporal audio frames, predict the probability that each of the multiple temporal audio frames belongs to each audio category based on the audio frame features corresponding to each of the multiple temporal audio frames, and determine the audio category to which each of the multiple temporal audio frames belongs based on the probability that each of the multiple temporal audio frames belongs to each audio category, wherein the audio category is a background music category or a human voice category.

[0082] In one embodiment, such as Figure 3 As shown, the pre-trained audio classification model includes an audio coding unit and a probability prediction unit. The computer device can segment audio in the temporal domain, obtaining multiple temporal audio frames. The computer device can input these multiple temporal audio frames into the audio coding unit of the pre-trained audio classification model. The audio coding unit extracts features from each of the multiple temporal audio frames, obtaining the audio frame features corresponding to each temporal audio frame, forming an audio feature sequence. The computer device can then input the audio feature sequence into the probability prediction unit of the pre-trained audio classification model. Based on the audio feature sequence, the probability prediction unit predicts the probability that each of the multiple temporal audio frames belongs to each audio category, forming a probability curve. Based on the probability curve, the audio category to which each of the multiple temporal audio frames belongs is determined, where the audio category is either background music or human voice.

[0083] In the above embodiments, by determining the audio category to which multiple temporal audio frames belong (the audio category is either background music or human voice), and by directly locating at least one background audio segment belonging to the background music category and at least one human voice audio segment belonging to the human voice category based on the audio category to which multiple temporal audio frames belong, the accuracy of locating the background audio segment and the human voice audio segment can be improved. This can further improve the accuracy of subsequent video segment extraction and further avoid the waste of computer hardware resources used to support the extraction of the required video segments.

[0084] In one embodiment, determining the audio category to which the multiple time-domain audio frames belong based on multiple time-domain audio frames includes: converting the multiple time-domain audio frames to the frequency domain to obtain multiple audio frequency domain signals; extracting time-domain features from the multiple time-domain audio frames, extracting frequency-domain features from the multiple audio frequency domain signals, and fusing the time-domain features and frequency-domain features to obtain audio fusion features; and determining the audio category to which the multiple time-domain audio frames belong based on the audio fusion features.

[0085] Among them, time-domain features are the features of the time-domain audio frame located in the time-domain space, and frequency-domain features are the features of the audio frequency-domain signal located in the frequency-domain space. Audio fusion features are the features obtained by fusing time-domain features and frequency-domain features. It can be understood that audio fusion features include both time-domain features and frequency-domain features.

[0086] Specifically, a computer device can convert multiple time-domain audio frames located in the time domain to the frequency domain, obtaining multiple audio frequency domain signals located in the frequency domain. The computer device can extract features from each of the multiple time-domain audio frames to obtain time-domain features corresponding to each frame, and extract features from each of the multiple audio frequency domain signals to obtain frequency-domain features corresponding to each signal. Furthermore, the computer device can fuse the time-domain and frequency-domain features to obtain audio fusion features. Based on these fusion features, it can predict the probability that each of the multiple time-domain audio frames belongs to each audio category, and based on this probability, determine the audio category to which each of the multiple time-domain audio frames belongs, where the audio category is either background music or vocals.

[0087] In one embodiment, the computer device can convolve multiple time-domain audio frames separately to obtain time-domain features corresponding to each time-domain audio frame. The computer device can also convolve multiple audio frequency-domain signals separately to obtain frequency-domain features corresponding to each audio frequency-domain signal. Furthermore, the computer device can fuse the time-domain features and frequency-domain features to obtain audio fusion features.

[0088] In one embodiment, a computer device can acquire a pre-trained audio classification model and input multiple time-domain audio frames and multiple audio frequency-domain signals into the pre-trained audio classification model. The model extracts features from each of the time-domain audio frames to obtain time-domain features corresponding to each frame, and extracts features from each of the audio frequency-domain signals to obtain frequency-domain features corresponding to each signal. Furthermore, the computer device can fuse the time-domain and frequency-domain features to obtain audio fusion features. Based on these fusion features, it predicts the probability that each of the multiple time-domain audio frames belongs to each audio category, and determines the audio category to which each of the multiple time-domain audio frames belongs, where the audio category is either background music or vocals.

[0089] In one embodiment, the audio frequency domain signal includes the Mel spectrum. A computer device can convert multiple time-domain audio frames located in the time domain to the frequency domain, obtaining multiple Mel spectra located in the frequency domain. The computer device can extract features from the multiple time-domain audio frames separately to obtain time-domain features corresponding to each time-domain audio frame, and extract features from the multiple Mel spectra separately to obtain frequency-domain features corresponding to each Mel spectra. Furthermore, the computer device can fuse the time-domain features and frequency-domain features to obtain audio fusion features. Based on the audio fusion features, it can predict the probability that each of the multiple time-domain audio frames belongs to each audio category, and based on the probability that each of the multiple time-domain audio frames belongs to each audio category, it can determine the audio category to which each of the multiple time-domain audio frames belongs, wherein the audio category is either background music or vocals.

[0090] In the above embodiments, by converting multiple time-domain audio frames into multiple audio frequency-domain signals, and fusing the time-domain features of the time-domain audio frames with the frequency-domain features of the audio frequency-domain signals, an audio fusion feature that includes both time-domain and frequency-domain features is obtained. Based on the audio fusion feature that includes both time-domain and frequency-domain features, the audio category to which the multiple time-domain audio frames belong can be determined. This can improve the accuracy of determining the audio category to which the time-domain audio frames belong, thereby further improving the accuracy of subsequent video segment extraction and avoiding the waste of computer hardware resources used to support the extraction of the required video segments.

[0091] In one embodiment, extracting temporal features from multiple temporal audio frames, extracting frequency features from multiple audio frequency signals, and fusing the temporal and frequency features to obtain audio fusion features includes: extracting features from multiple temporal audio frames layer by layer according to multiple preset levels to obtain temporal features for each level; extracting features from multiple audio frequency signals layer by layer according to multiple levels to obtain frequency features for each level; for each level, performing feature fusion based at least on the temporal and frequency features of the targeted level to generate layer fusion features for the targeted level, wherein when generating layer fusion features for levels after the first level, feature fusion is also performed based on the layer fusion features of the previous level of the targeted level; and determining the audio fusion features based on the layer fusion features of the last level among the multiple levels.

[0092] The layer fusion features for a specific layer are features obtained by fusing features from at least the time-domain and frequency-domain features of that specific layer. It can be understood that the layer fusion features for a specific layer include at least the time-domain and frequency-domain features of that specific layer.

[0093] Specifically, a computer device can extract features from multiple temporal audio frames layer by layer according to preset multiple levels, obtaining temporal features for each level. It can be understood that for the first level, the computer device can directly convolve the temporal audio frames to extract the temporal features of the first level. For levels after the first level, the computer device can convolve the temporal features of the previous level to obtain the temporal features of the targeted level. Similarly, a computer device can extract features from multiple audio frequency domain signals layer by layer, obtaining frequency domain features for each level. It can be understood that for the first level, the computer device can directly convolve the audio frequency domain signals to extract the frequency domain features of the first level. For levels after the first level, the computer device can convolve the frequency domain features of the previous level to obtain the frequency domain features of the targeted level. For each level, the computer device can perform feature fusion based on at least the temporal and frequency domain features of the targeted level to generate layer-fused features for the targeted level. Specifically, for the first layer, the computer device can directly perform feature fusion based on the temporal and frequency domain characteristics of the first layer to generate the layer fusion feature for the first layer. For layers subsequent to the first layer, when generating layer fusion features for these layers, the computer device can perform feature fusion based on the temporal and frequency domain characteristics of the first layer, as well as the layer fusion feature of the layer preceding the targeted layer, to generate the layer fusion feature for the targeted layer. Furthermore, the computer device can determine the audio fusion feature based on the layer fusion feature of the last layer among multiple layers.

[0094] In one embodiment, the time-domain features and frequency-domain features have the same feature dimensions. For each layer, the computer device can at least directly perform feature fusion based on the time-domain features and frequency-domain features of the targeted layer to generate the layer-fused features of the targeted layer. Specifically, when the targeted layer is the first layer, the computer device can directly fuse the frequency-domain features and time-domain features of the targeted layer to obtain the layer-fused features of the first layer. When the targeted layer is a layer after the first layer, the computer device can directly fuse the frequency-domain features of the targeted layer, the reconstructed time-domain features of the targeted layer, and the layer-fused features of the previous layer to obtain the layer-fused features of the targeted layer.

[0095] In the above embodiments, the temporal features of multiple temporal audio frames at each level and the frequency features of multiple audio frequency signals at each level are obtained. For each level, feature fusion is performed at least based on the temporal and frequency features of the target level to generate a layer fusion feature for the target level. When generating layer fusion features for levels after the first level, feature fusion is also performed based on the layer fusion features of the previous level. This allows higher-level feature fusion to perceive relevant information from lower-level feature fusion. Furthermore, based on the layer fusion feature of the last level among multiple levels, the audio fusion feature is determined, which improves the accuracy of the audio fusion feature. This, in turn, further improves the accuracy of subsequent video segment extraction and avoids wasting computer hardware resources used to support the extraction of the required video segments.

[0096] In one embodiment, the time-domain features and frequency-domain features have different feature dimensions. For each layer, feature fusion is performed at least based on the time-domain features and frequency-domain features of the targeted layer to generate layer-fusion features for the targeted layer. When generating layer-fusion features for layers after the first layer, feature fusion is also performed based on the layer-fusion features of the layer preceding the targeted layer. This includes: for each layer, feature reconstruction is performed on the time-domain features of the targeted layer according to the feature dimensions of the frequency-domain features of the targeted layer to obtain the reconstructed time-domain features of the targeted layer; when the targeted layer is the first layer, feature fusion is performed on the frequency-domain features of the targeted layer and the reconstructed time-domain features to obtain the layer-fusion features of the targeted layer; when the targeted layer is a layer after the first layer, feature fusion is performed on the frequency-domain features of the targeted layer, the reconstructed time-domain features of the targeted layer, and the layer-fusion features of the layer preceding the targeted layer to obtain the layer-fusion features of the targeted layer.

[0097] Specifically, for each layer, the computer device can reconstruct the time-domain features of the targeted layer according to the feature dimensions of the frequency-domain features of the targeted layer, thereby obtaining the reconstructed time-domain features of the targeted layer. The reconstructed time-domain features of the targeted layer have the same feature dimensions as the frequency-domain features of the targeted layer. When the targeted layer is the first layer, the computer device can fuse the frequency-domain features and the reconstructed time-domain features of the first layer to obtain the layer-fused features of the first layer. When the targeted layer is a layer after the first layer, the computer device fuses the frequency-domain features of the targeted layer, the reconstructed time-domain features of the targeted layer, and the layer-fused features of the previous layer to obtain the layer-fused features of the targeted layer. The layer-fused features of the targeted layer have the same feature dimensions as the frequency-domain features of the targeted layer.

[0098] In the above embodiments, by reconstructing the temporal features of the targeted layer according to the feature dimensions of the frequency domain features, reconstructed temporal features of the targeted layer with the same feature dimensions as the frequency domain features can be obtained. Fusing the frequency domain features with the same feature dimensions as the first layer and the reconstructed temporal features to obtain the first layer's layer-fused features can improve the accuracy of the first layer's layer-fused features. When the targeted layer is a layer after the first layer, fusing the frequency domain features of the targeted layer with the same feature dimensions, the reconstructed temporal features of the targeted layer, and the layer-fused features of the previous layer to obtain the layer-fused features of the targeted layer can improve the accuracy of the layer-fused features of layers after the first layer, thereby further improving the accuracy of subsequent video segment extraction and avoiding the waste of computer hardware resources used to support the extraction of the required video segments.

[0099] In one embodiment, the layer fusion feature includes multiple feature maps; determining the audio fusion feature based on the layer fusion feature of the last layer among multiple layers includes: fusing the feature maps of the layer fusion feature of the last layer among multiple layers to obtain a fused feature map; and reducing the dimensionality of the fused feature map to obtain an audio fusion feature with the same feature dimension as each temporal feature before reconstruction.

[0100] Specifically, the layer fusion feature comprises multiple feature maps. A computer device can fuse the feature maps from the last layer of the layer fusion feature to obtain a fused feature map. Furthermore, the computer device can reduce the dimensionality of the fused feature map to obtain audio fusion features with the same feature dimension as each temporal feature before reconstruction. In essence, the audio fusion feature can be understood as a one-dimensional feature vector.

[0101] In one embodiment, such as Figure 4As shown, the pre-trained audio classification model includes an audio coding unit and a probability prediction unit. The audio coding unit generates audio fusion features, while the probability prediction unit predicts the probability that multiple time-domain audio frames belong to each audio category based on these fusion features, and determines the audio category to which each time-domain audio frame belongs based on that probability. The audio coding unit includes a time-domain processing branch and a frequency-domain processing branch. The time-domain processing branch includes multiple one-dimensional convolutional blocks for multi-level time-domain feature extraction. The frequency-domain processing branch includes multiple two-dimensional convolutional blocks for multi-level frequency-domain feature extraction, with information exchange between the time and frequency domains at each level. Specifically, the computer device can extract features from multiple time-domain audio frames layer by layer using a pre-defined multi-level one-dimensional convolution to obtain the time-domain features at each level. The computer device can also extract features from multiple audio frequency-domain signals layer by layer using a multi-level two-dimensional convolution to obtain the frequency-domain features at each level. For each level, the computer device can reconstruct the temporal features of the target level according to the feature dimensions of the frequency domain features of the target level, obtaining the reconstructed temporal features of the target level. When the target level is the first level, the computer device can fuse the frequency domain features of the first level and the reconstructed temporal features to obtain the layer fusion features of the first level. When the target level is a level after the first level, the computer device can fuse the frequency domain features of the target level, the reconstructed temporal features of the target level, and the layer fusion features of the previous level to obtain the layer fusion features of the target level. The computer device can fuse the feature maps by finding the maximum value, averaging, and then adding the two-dimensional feature maps of the layer fusion features of the last level among multiple levels to obtain a fused feature map. Furthermore, the computer device can reduce the dimensionality of the fused feature map by using activation functions and normalization to obtain audio fusion features with the same feature dimensions as each temporal feature before reconstruction.

[0102] In the above embodiments, by fusing the feature maps of the last layer of the multi-layer fusion feature map to obtain a fused feature map, and then reducing the dimensionality of the fused feature map to obtain an audio fusion feature with the same feature dimension as each temporal feature before reconstruction, the accuracy of the audio fusion feature can be improved, thereby further improving the accuracy of subsequent video segment extraction and avoiding the waste of computer hardware resources used to support the extraction of the required video segments.

[0103] In one embodiment, the audio categories to which multiple temporal audio frames belong are determined by a pre-trained audio classification model, which predicts the probability that each temporal audio frame belongs to each audio category. The audio classification model is obtained through a model training step, which includes: acquiring training samples; the training samples include sample audio and the audio category to which the sample audio belongs, labeled for the sample audio; inputting the sample audio into the audio classification model to be trained, so that the audio classification model to be trained can predict the category of each audio frame in the sample audio and obtain the probability that each audio frame in the sample audio belongs to each audio category; determining the probability that each audio frame in the sample audio belongs to each audio category based on the probability that each audio frame in the sample audio belongs to each audio category; determining the loss value based on the probability that each audio frame in the sample audio belongs to each audio category and the audio category to which the sample audio belongs, labeled for the sample audio; and training the audio classification model to be trained in the direction that reduces the loss value.

[0104] Specifically, the computer device can acquire training samples, which include sample audio and the audio category to which the sample audio belongs, labeled accordingly. It can be understood that the labeling method for the audio to belong to the audio category is weak labeling, thus allowing for the easy acquisition of a large number of training samples. The computer device can input the sample audio into the audio classification model to be trained, so that the model can predict the category of each audio frame in the sample audio, obtaining the probability that each audio frame belongs to each audio category. Based on the probability that each audio frame belongs to each audio category, the probability that the sample audio belongs to each audio category is determined. Based on the probability that the sample audio belongs to each audio category and the audio category labeled for the sample audio, a loss value is determined, and the audio classification model to be trained is trained in the direction that reduces the loss value until the iteration stopping condition is met, resulting in a trained audio classification model.

[0105] In one embodiment, the computer device may acquire a preset function, and for each audio category, determine the probability that a sample audio belongs to a specific audio category based on the probability that each audio frame in the sample audio belongs to the specific audio category using the acquired preset function.

[0106] In one embodiment, the iteration stopping condition is specifically that the number of iterations reaches a preset number, or it can be that the loss value is less than a preset threshold.

[0107] In the above embodiments, since the training samples include sample audio and the audio category to which the sample audio belongs (i.e., the labeling is weakly labeled for the entire sample audio), the audio classification model is trained using the weakly labeled training samples. During the training process, the probability of each audio frame in the sample audio belonging to each audio category is determined based on the probability of each audio frame belonging to each audio category, so as to determine the training loss value together with the weak labeling. The weakly labeled training samples are easy to obtain, which improves the efficiency of model training and avoids the waste of computer hardware resources used to support model training.

[0108] In one embodiment, determining the probability that a sample audio belongs to each audio category based on the probability that each audio frame in the sample audio belongs to each audio category includes: for each audio category, determining multiple candidate probabilities that the sample audio belongs to the target audio category using multiple preset functions based on the probability that each audio frame in the sample audio belongs to the target audio category; fusing the multiple candidate probabilities that the sample audio belongs to the target audio category using multiple fusion methods to obtain multiple fused probabilities that the sample audio belongs to the target audio category; and determining the probability that the sample audio belongs to the target audio category based on the multiple fused probabilities that the sample audio belongs to the target audio category.

[0109] Specifically, the computer device can acquire multiple preset functions. For each audio category, the computer device can use these preset functions to determine multiple candidate probabilities that the sample audio belongs to the target audio category, based on the probability that each audio frame in the sample audio belongs to the target audio category. The computer device can then use various fusion methods to fuse these multiple candidate probabilities to obtain multiple fused probabilities that the sample audio belongs to the target audio category. It is understood that candidate probabilities and fused probabilities are intermediate probabilities; they are not used to ultimately determine the probability that the sample audio belongs to the target audio category. In essence, the computer device can determine the probability that the sample audio belongs to the target audio category based on the multiple fused probabilities.

[0110] In one embodiment, such as Figure 5As shown, the computer device can acquire five preset functions. For each audio category, the computer device can use these five preset functions to determine five candidate probabilities that the sample audio belongs to the target audio category, based on the probability that each audio frame in the sample audio belongs to the target audio category. The computer device can then use two fusion methods to fuse these five candidate probabilities to obtain two fused probabilities. Finally, the computer device can determine the probability that the sample audio belongs to the target audio category based on these two fused probabilities.

[0111] In one embodiment, the preset function may specifically be a pooling function. The computer device can use five pooling functions, along with two fusion methods—averaging and taking the maximum value—to determine the probability that a sample audio belongs to a specific audio category. For each sample audio, the probability that it belongs to the specific audio category can be calculated using the following formula:

[0112] pooling1=(Max+Avg+Linear+Exp+Attention) / 5;

[0113] pooling2=Max(Max,Avg,Linear,Exp,Attention);

[0114] pooling final = (pooling1+pooling2) / 2.

[0115]

[0116]

[0117]

[0118]

[0119]

[0120] Where Max, Avg, Linear, Exp, and Attention are pooling functions, i is each audio frame in the sample audio, n is the number of audio frames in the sample audio, and w i These are the pre-acquired weight values. pooling1 is the fusion probability obtained by averaging, pooling2 is the fusion probability obtained by taking the maximum value, and pooling... final This represents the probability that the sample audio belongs to the specified audio category.

[0121] In the above embodiments, for each audio category, multiple preset functions are used to determine multiple candidate probabilities of a sample audio belonging to the target audio category based on the probability of each audio frame in the sample audio belonging to the target audio category. This reduces the randomness of determining the sample audio belonging to the target audio category using a single preset function, thus improving the accuracy of subsequent determinations. By using multiple fusion methods to fuse the multiple candidate probabilities of a sample audio belonging to the target audio category, multiple fused probabilities are obtained. This reduces the randomness of determining the sample audio belonging to the target audio category using a single fusion method, further improving the accuracy of subsequent determinations, enhancing the model training effect, and avoiding the waste of computer hardware resources used to support model training.

[0122] In one embodiment, determining a subcategory label for each background audio segment includes: for each background audio segment, determining the background audio features of the target background audio segment; based on the background audio features, determining the confidence level of the target background audio segment belonging to a plurality of preset subcategory labels; each subcategory label is used to characterize a musical mood of the background audio segment; and based on the confidence level of the target background audio segment belonging to a plurality of preset subcategory labels, determining the subcategory label of the target background audio segment.

[0123] Specifically, for each background audio segment, the computer device can extract background audio features from the targeted background audio segment and, based on these features, determine the confidence level of whether the targeted background audio segment belongs to a set of preset sub-category labels. Each sub-category label represents a musical emotion within the background audio segment. For example, musical emotions can include at least one of the following: sadness, happiness, sweetness, healing, longing, catharsis, inspiration, touching, tension, epic, struggle, or humor. Furthermore, the computer device can determine the sub-category label of the targeted background audio segment based on the confidence level of each of the preset sub-category labels. The confidence level of a targeted background audio segment belonging to a sub-category label is positively correlated with the probability of being identified as a sub-category label.

[0124] In one embodiment, a computer device can acquire a pre-trained music emotion recognition model and input each background audio segment into the pre-trained music emotion recognition model. The model extracts background audio features from the targeted background audio segments and, based on these features, determines the confidence level of each of the targeted background audio segments belonging to a set of preset sub-category labels. Furthermore, the computer device can determine the sub-category label of the targeted background audio segment based on the confidence level of each of the preset sub-category labels.

[0125] In one embodiment, such as Figure 6 As shown, the music emotion recognition model includes a music feature extraction unit and multiple normalized exponential functions. Specifically, the computer device can acquire the background audio features of a background audio segment, i.e., an audio feature sequence, and input the audio feature sequence into the music feature extraction unit in the pre-trained music emotion recognition model. The music feature extraction unit extracts features from the audio feature sequence and determines the confidence level of the target background audio segment belonging to multiple preset sub-category labels based on the normalized exponential functions. Furthermore, the computer device can determine the sub-category label of the target background audio segment based on the confidence level of the target background audio segment belonging to multiple preset sub-category labels. The sub-category labels may include at least one of the following: sad, happy, sweet, healing, longing, catharsis, inspirational, touching, tense, epic, struggle, or funny.

[0126] In the above embodiments, by using the background audio features of the target background audio segment, the confidence level of the target background audio segment belonging to a plurality of preset sub-category labels is determined. Then, based on the confidence level of the target background audio segment belonging to a plurality of preset sub-category labels, the sub-category label of the target background audio segment is determined. This can improve the accuracy of the sub-category label of the target background audio segment, thereby further improving the accuracy of subsequent video segment extraction and avoiding the waste of computer hardware resources used to support the extraction of the required video segment.

[0127] In one embodiment, each background audio segment included in the target audio segment intersects with at least one included human voice audio segment.

[0128] Specifically, the computer device can acquire video and extract audio from it. Within the audio, the computer device can locate at least one background audio segment belonging to the background music category and at least one human voice audio segment belonging to the human voice category. The computer device can acquire the video's theme tags and determine the subcategory tag for each background audio segment. Within the audio, at least one target audio segment is located, each target audio segment comprising at least one complete background audio segment and at least one complete human voice audio segment, the union of the included background audio segments and human voice audio segments being continuous, each background audio segment included in the target audio segment intersecting with the included at least one human voice audio segment, and the subcategory tag of the included background audio segment matching the video's theme tags. The computer device can extract at least one video segment from the video based on at least one target audio segment.

[0129] For example, such as Figure 7As shown, since background audio segment 1 intersects with two human voice audio segments, background audio segment 2 intersects with three human voice audio segments, and background audio segment 3 does not intersect with any human voice audio segment, the target audio segment to be located may include background audio segment 1 or background audio segment 2, but will not include background audio segment 3.

[0130] In the above embodiments, by limiting each background audio segment included in the target audio segment to have an intersection with at least one human voice audio segment included, it can be ensured that each background audio segment included in the target audio segment contains human voice, thereby further improving the accuracy of subsequent video segment extraction and avoiding the waste of computer hardware resources used to support the extraction of the required video segments.

[0131] In one embodiment, for each background audio segment in the target audio segment, the proportion of time of at least one human voice audio segment that intersects with the target background audio segment in the target background audio segment exceeds a preset ratio.

[0132] Specifically, the computer device can acquire video and extract audio from it. Within the audio, the computer device can locate at least one background audio segment belonging to the background music category and at least one human voice audio segment belonging to the human voice category. The computer device can acquire the video's theme tags and determine the sub-category tag for each background audio segment. Within the audio, at least one target audio segment is located, each target audio segment comprising at least one complete background audio segment and at least one complete human voice audio segment. The union of the included background audio segments and human voice audio segments is continuous. Each background audio segment included in the target audio segment intersects with the included at least one human voice audio segment. For each background audio segment in the target audio segment, the proportion of time of the at least one human voice audio segment intersecting with the target background audio segment exceeds a preset ratio, and the sub-category tag of the included background audio segment matches the video's theme tag. The computer device can extract at least one video segment from the video based on at least one target audio segment.

[0133] In one embodiment, such as Figure 8As shown, for each target audio segment, the union of the background audio segments and the vocal audio segments included in the target audio segment is the target audio segment. Specifically, the computer device can acquire video and extract audio from the video. The computer device can locate at least one background audio segment belonging to the background music category and at least one vocal audio segment belonging to the vocal category in the audio. The computer device can acquire the video's topic tags and determine the subcategory tag for each background audio segment. In the audio, at least one target audio segment is located, namely target audio segment 1 and target audio segment 2. Each target audio segment includes at least one complete background audio segment and at least one complete human voice audio segment. The union of the included background audio segments and human voice audio segments is continuous. For each target audio segment, the union of the included background audio segments and human voice audio segments is the target audio segment itself. Each background audio segment included in the target audio segment intersects with at least one included human voice audio segment. For each background audio segment in the target audio segment, the proportion of time of the at least one human voice audio segment that intersects with the target background audio segment exceeds a preset ratio, and the sub-category label of the included background audio segment matches the theme label of the video. The computer device can extract at least one video segment from the video according to at least one target audio segment.

[0134] In the above embodiments, for each background audio segment in the target audio segment, by limiting the time ratio of at least one human voice audio segment that intersects with the target background audio segment to exceed a preset ratio, it can be ensured that each background audio segment included in the target audio segment has a relatively long duration of human voice, thereby further improving the accuracy of subsequent video segment extraction and avoiding the waste of computer hardware resources used to support the extraction of the required video segments.

[0135] In one embodiment, such as Figure 9As shown, the audio classification model includes an audio coding unit and a normalized exponential function. During the model training phase, the computer can acquire a massive amount of weakly labeled training samples; these samples include sample audio and the audio category to which the sample audio belongs, labeled accordingly. The computer can input the sample audio into the audio classification model to be trained, allowing the model to predict the category of each audio frame in the sample audio, obtaining the probability that each audio frame belongs to each audio category. Using at least one pooling function, based on the probability of each audio frame belonging to each audio category, the probability of the sample audio belonging to each audio category is determined. Based on the probability of the sample audio belonging to each audio category and the audio category to which the sample audio belongs, a loss value is determined, and the audio classification model is trained in a direction that minimizes the loss value, resulting in a trained audio classification model. In the model application phase, the computer device acquires the video to be processed and extracts the audio from it. The audio is then input into a trained audio classification model to determine the probability curve of each audio frame belonging to each audio category. The audio category can be either background music or human voice. The computer device can locate at least one background audio segment belonging to the background music category and at least one human voice audio segment belonging to the human voice category.

[0136] In one embodiment, such as Figure 10As shown, the audio classification model includes an audio coding unit and a normalized exponential function. During the model training phase, the computer can acquire a massive amount of weakly labeled training samples; these samples include sample audio and the audio category to which the sample audio belongs, labeled accordingly. The computer can input the sample audio into the audio classification model to be trained, allowing the model to predict the category of each audio frame in the sample audio, obtaining the probability that each audio frame belongs to each audio category. Using at least one pooling function, based on the probability of each audio frame belonging to each audio category, the probability of the sample audio belonging to each audio category is determined. Based on the probability of the sample audio belonging to each audio category and the audio category to which the sample audio belongs, a loss value is determined, and the audio classification model is trained in a direction that minimizes the loss value, resulting in a trained audio classification model. In the model application phase, the computer device acquires the video to be processed and extracts the audio from it. The audio is then input into a trained audio classification model to determine the probability curve of each audio frame belonging to each audio category. The audio category can be either background music or vocals. The computer device can locate at least one background audio segment belonging to the background music category and at least one vocal audio segment belonging to the vocal category. Furthermore, the computer device acquires the video's topic tags and determines a sub-category tag for each background audio segment. Each sub-category tag represents a musical mood within the background audio segment. A computer device can locate at least one target audio segment in audio. Each target audio segment includes at least one complete background audio segment and at least one complete human voice audio segment. The union of the included background audio segments and human voice audio segments is continuous. For each target audio segment, the union of the included background audio segments and human voice audio segments is the target audio segment itself. Each background audio segment included in the target audio segment intersects with at least one included human voice audio segment. For each background audio segment in the target audio segment, the proportion of time of the at least one human voice audio segment intersecting with the target background audio segment exceeds a preset ratio, and the sub-category label of the included background audio segment matches the theme label of the video. The computer device can extract at least one video segment from the video based on at least one target audio segment.

[0137] In one embodiment, the video is a film or television drama video; the voice category is a voice dialogue category; the voice audio segment is a voice dialogue audio segment; the topic tag is a theme tag of the film or television drama video; and the video segment is an edited clip of the film or television drama video.

[0138] Specifically, the computer device can acquire film and television videos and extract audio from them. Within the audio, the computer device can locate at least one background audio segment belonging to the background music category and at least one human voice audio segment belonging to the dialogue category. The computer device can acquire the theme tags of the film and television videos and determine the subcategory tags for each background audio segment. Within the audio, at least one target audio segment is located, each target audio segment including at least one complete background audio segment and at least one complete human voice audio segment. The union of the included background audio segments and human voice audio segments is continuous, and the subcategory tags of the included background audio segments match the theme tags of the film and television videos. The computer device can extract at least one clip from the film and television videos based on at least one target audio segment. It can be understood that the clip from the film and television videos is a highlight segment of the film and television video.

[0139] In the above embodiments, by locating the human voice audio segment and the background audio segment matching the theme of the film and television video from the audio of the film and television video, and by using the located human voice audio segment and background audio segment together to locate the target audio segment that includes the complete background audio segment and the complete human voice audio segment and matches the theme of the film and television video, and then extracting the wonderful clips that include the complete background audio segment and the complete human voice audio segment and match the theme of the film and television video from the film and television video based on the located target audio segment, the accuracy of film and television video segment extraction is improved, thereby avoiding the waste of computer hardware resources used to support the extraction of the required film and television video segments.

[0140] like Figure 11 As shown, in one embodiment, a video processing method is provided. This method can be applied to a computer device, which can be a terminal or a server. That is, the method can be executed independently by the terminal or the server, or it can be implemented through interaction between the terminal and the server. This embodiment uses the application of the method to a computer device as an example for illustration. The method specifically includes the following steps:

[0141] Step 1102: Obtain training samples; the training samples include sample audio and the audio category to which the sample audio belongs, labeled for the sample audio.

[0142] Step 1104: Input the sample audio into the audio classification model to be trained, so as to predict the category of each audio frame in the sample audio through the audio classification model to be trained, and obtain the probability that each audio frame in the sample audio belongs to each audio category.

[0143] Step 1106: For each audio category, multiple candidate probabilities of the sample audio belonging to the target audio category are determined by using various preset functions based on the probability that each audio frame in the sample audio belongs to the target audio category.

[0144] Step 1108: By using multiple fusion methods, the multiple candidate probabilities of the sample audio belonging to the target audio category are fused to obtain multiple fused probabilities of the sample audio belonging to the target audio category.

[0145] Step 1110: Determine the probability that the sample audio belongs to the target audio category based on the multiple fusion probabilities of the sample audio belonging to the target audio category.

[0146] Step 1112: Determine the loss value based on the probability of the sample audio belonging to each audio category and the audio category to which the sample audio is labeled.

[0147] Step 1114: Train the audio classification model to be trained in the direction that reduces the loss value, and obtain the trained audio classification model.

[0148] Step 1116: Acquire the video and extract the audio from it; input the audio into the trained audio classification model to obtain multiple temporal audio frames obtained by segmenting the audio.

[0149] Step 1118: Convert multiple time-domain audio frames to the frequency domain to obtain multiple audio frequency domain signals; extract features from the multiple time-domain audio frames layer by layer according to multiple preset levels to obtain the time-domain features of each level.

[0150] Step 1120: Extract features from multiple audio frequency domain signals layer by layer according to multiple levels to obtain the frequency domain features of each level.

[0151] Step 1122: For each level, according to the feature dimension of the frequency domain features of the level, perform feature reconstruction on the time domain features of the level to obtain the reconstructed time domain features of the level.

[0152] Step 1124: When the target layer is the first layer, the frequency domain features of the target layer and the reconstructed time domain features are fused to obtain the layer fusion features of the target layer.

[0153] Step 1126: When the target layer is a layer after the first layer, the frequency domain features of the target layer, the reconstructed time domain features of the target layer, and the layer fusion features of the layer preceding the target layer are fused to obtain the layer fusion features of the target layer.

[0154] Step 1128: Fuse the feature maps of the last layer of the multi-layer fusion feature map to obtain a fused feature map; reduce the dimensionality of the fused feature map to obtain audio fusion features with the same feature dimension as each temporal feature before reconstruction.

[0155] Step 1130: Based on the audio fusion features, determine the audio category to which multiple temporal audio frames belong; the audio category is either background music or human voice.

[0156] Step 1132: In the audio, based on the audio categories to which multiple time-domain audio frames belong, locate at least one background audio segment belonging to the background music category and at least one human voice audio segment belonging to the human voice category.

[0157] Step 1134: Obtain the video's theme tags and determine the background audio features for each background audio segment.

[0158] Step 1136: Based on the background audio features, determine the confidence level of the target background audio segment belonging to multiple preset sub-category labels; each sub-category label is used to characterize a musical mood of the background audio segment.

[0159] Step 1138: Determine the sub-category label of the target background audio segment based on the confidence level of the target background audio segment belonging to multiple preset sub-category labels.

[0160] Step 1140: In the audio, locate at least one target audio segment, and extract at least one video segment from the video according to the at least one target audio segment.

[0161] Each target audio segment includes at least one complete background audio segment and at least one complete human voice audio segment. The union of the included background audio segments and human voice audio segments is continuous, and the sub-category labels of the included background audio segments match the topic labels of the video. Each background audio segment included in the target audio segment intersects with at least one included human voice audio segment. For each background audio segment in the target audio segment, the proportion of time of at least one human voice audio segment that intersects with the target background audio segment exceeds a preset ratio.

[0162] This application also provides an application scenario where the aforementioned video processing method is applied. Specifically, this video processing method can be applied to scenarios where exciting video clips are extracted from film and television videos. It is understood that the video in this application refers to film and television videos; the voice category refers to voice dialogue; the voice audio segment refers to voice dialogue audio segment; the theme tag refers to the theme tag of the film and television video; and the video segment refers to an edited fragment of the film and television video. It is understood that the video processing method can be applied to specific scenarios such as: on short video creation platforms, many online streamers can extract exciting film and television clips from film and television videos to create short videos; for example, in the later stages of film and television production, the production company will extract exciting film and television clips from each episode of the main film and television series as exciting preview clips for each episode; for example, before the film and television series is released online, the production company will include a compilation album, in which the exciting film and television clips included in the compilation album are obtained using the video processing method of this application. For example, the exciting film and television clips marked on the progress bar during playback are also located using the video processing method of this application.

[0163] It's understandable that extracting exciting clips from film and television videos can be achieved through the following processing flow: Specifically, computer equipment acquires training samples; these samples include sample audio and the audio category to which the sample audio belongs, labeled accordingly. The sample audio is input into an audio classification model to be trained, which predicts the category of each audio frame in the sample audio, obtaining the probability that each audio frame belongs to each audio category. For each audio category, multiple candidate probabilities of the sample audio belonging to that category are determined using various preset functions based on the probability of each audio frame belonging to that category. These candidate probabilities are then fused using various fusion methods to obtain multiple fused probabilities. Based on these fused probabilities, the probability that the sample audio belongs to the target audio category is determined. Finally, based on the probability of the sample audio belonging to each audio category and the labeled audio category, a loss value is determined. Train the audio classification model to be trained in the direction that reduces the loss value, and you will get the trained audio classification model.

[0164] Computer equipment can acquire video footage from movies and TV shows and extract audio from it. The audio is input into a trained audio classification model to obtain multiple time-domain audio frames, which are then segmented into frames. These time-domain audio frames are converted to the frequency domain to obtain multiple audio frequency domain signals. Features are extracted from these time-domain audio frames layer by layer according to a predefined hierarchy to obtain time-domain features for each level. For each level, the time-domain features are reconstructed based on the feature dimensions of the frequency domain features, resulting in reconstructed time-domain features for that level. When the target level is the first level, the frequency domain features and the reconstructed time-domain features are fused to obtain layer-fused features for that level. When the target level is a subsequent level, the frequency domain features, the reconstructed time-domain features, and the layer-fused features from the previous level are fused to obtain layer-fused features for that level. The feature maps of the last layer in the multi-layer fusion feature set are fused to obtain a fused feature map. The fused feature map is then dimensionality-reduced to obtain audio fusion features with the same feature dimensions as each temporal feature before reconstruction. Based on the audio fusion features, the audio categories to which multiple temporal audio frames belong are determined; the audio category is either background music or spoken dialogue. Within the audio, based on the audio categories to which multiple temporal audio frames belong, at least one background audio segment belonging to the background music category and at least one spoken dialogue audio segment belonging to the spoken dialogue category are located.

[0165] The computer device can acquire the theme tags of film and television videos, and determine the background audio features of each background audio segment. Based on the background audio features, it determines the confidence level of each background audio segment belonging to multiple preset sub-category tags; each sub-category tag is used to characterize a musical mood of the background audio segment. Based on the confidence level of each background audio segment belonging to multiple preset sub-category tags, it determines the sub-category tag of the background audio segment. In the audio, at least one target audio segment is located, and at least one exciting clip is extracted from the film and television video according to the at least one target audio segment. Each target audio segment includes at least one complete background audio segment and at least one complete voice dialogue audio segment. The union of the included background audio segments and voice dialogue audio segments is continuous, and the sub-category tags of the included background audio segments match the theme tags of the film and television video; each background audio segment included in the target audio segment intersects with at least one included voice dialogue audio segment. For each background audio segment in the target audio segment, the proportion of time of at least one human voice dialogue audio segment that intersects with the target background audio segment exceeds a preset value.

[0166] It is understandable that by locating the human voice audio segment and the background audio segment matching the theme of the film or television video from the audio, and then using the located human voice and background audio segments together to locate the target audio segment that includes both complete background audio segments and complete human voice audio segments and matches the theme of the film or television video, and then using the located target audio segment to extract the best clips that include both complete background audio segments and complete human voice audio segments and match the theme of the film or television video, the accuracy of film and television video segment extraction is improved, thereby avoiding the waste of computer hardware resources used to support the extraction of the required film and television video segments.

[0167] This application also provides another application scenario where the above-described video processing method is applied. Specifically, this video processing method can be applied to scenarios where video segments are extracted from recorded videos. It is understood that the video in this application is a user-recorded video, which includes audio, including background music and human voices. By locating the human voice audio segment and the background audio segment matching the theme of the recorded video from the audio of the recorded video, and by using the located human voice and background audio segments together to locate a target audio segment that includes both complete background audio and human voice audio segments and matches the theme of the recorded video, and then extracting a recorded video segment that includes both complete background audio and human voice audio segments and matches the theme of the recorded video based on the located target audio segment, the accuracy of video segment extraction is improved, thereby avoiding the waste of computer hardware resources used to support the extraction of the required recorded video segment.

[0168] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially, these steps are not necessarily executed in that order. Unless otherwise expressly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the above embodiments may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least a portion of the sub-steps or stages of other steps.

[0169] In one embodiment, such as Figure 12 As shown, a video processing apparatus 1200 is provided, which specifically includes:

[0170] The acquisition module 1202 is used to acquire video and extract audio from video;

[0171] The positioning module 1204 is used to locate at least one background audio segment belonging to the background music category and at least one human voice audio segment belonging to the human voice category in the audio.

[0172] Module 1206 is used to obtain the theme tags of the video and determine the sub-category tags for each background audio segment;

[0173] The positioning module 1204 is also used to locate at least one target audio segment in the audio, each target audio segment including at least one complete background audio segment and at least one complete human voice audio segment, the union of the included background audio segments and human voice audio segments is continuous, and the subcategory label of the included background audio segments matches the theme label of the video.

[0174] The extraction module 1208 is used to extract at least one video segment from the video according to at least one target audio segment.

[0175] In one embodiment, the positioning module 1204 is further configured to acquire multiple temporal audio frames obtained by framing the audio; determine the audio category to which the multiple temporal audio frames belong, based on the multiple temporal audio frames, wherein the audio category is either background music or human voice; and locate at least one background audio segment belonging to the background music category and at least one human voice audio segment belonging to the human voice category in the audio, based on the audio categories to which the multiple temporal audio frames belong.

[0176] In one embodiment, the positioning module 1204 is further configured to convert multiple time-domain audio frames to the frequency domain to obtain multiple audio frequency domain signals; extract time-domain features from the multiple time-domain audio frames, extract frequency domain features from the multiple audio frequency domain signals, and fuse the time-domain features and frequency domain features to obtain audio fusion features; and determine the audio category to which the multiple time-domain audio frames belong based on the audio fusion features.

[0177] In one embodiment, the positioning module 1204 is further configured to extract features from multiple time-domain audio frames layer by layer according to a preset multiple levels to obtain time-domain features of each level; extract features from multiple audio frequency-domain signals layer by layer according to a multiple level to obtain frequency-domain features of each level; for each level, perform feature fusion at least based on the time-domain features and frequency-domain features of the targeted level to generate layer fusion features for the targeted level, wherein when generating layer fusion features for levels after the first level, feature fusion is also performed based on the layer fusion features of the previous level of the targeted level; and determine audio fusion features based on the layer fusion features of the last level among multiple levels.

[0178] In one embodiment, the feature dimensions of the time-domain features and the frequency-domain features are different; the positioning module 1204 is further configured to, for each layer, reconstruct the time-domain features of the targeted layer according to the feature dimensions of the frequency-domain features of the targeted layer, to obtain the reconstructed time-domain features of the targeted layer; when the targeted layer is the first layer, the frequency-domain features of the targeted layer and the reconstructed time-domain features are fused to obtain the layer-fused features of the targeted layer; when the targeted layer is a layer after the first layer, the frequency-domain features of the targeted layer, the reconstructed time-domain features of the targeted layer, and the layer-fused features of the layer preceding the targeted layer are fused to obtain the layer-fused features of the targeted layer.

[0179] In one embodiment, the layer fusion feature includes multiple feature maps; the localization module 1204 is further configured to fuse the feature maps of the last layer of the multiple layers to obtain a fused feature map; and to reduce the dimensionality of the fused feature map to obtain an audio fusion feature with the same feature dimension as each temporal feature before reconstruction.

[0180] In one embodiment, the audio category to which multiple temporal audio frames belong is determined by a pre-trained audio classification model, which predicts the probability that each temporal audio frame belongs to each audio category. The audio classification model is obtained through a model training process, such as... Figure 13 As shown, the video processing device 1200 also includes:

[0181] The training module 1210 is used to acquire training samples. The training samples include sample audio and the audio category to which the sample audio belongs, labeled for the sample audio. The sample audio is input into the audio classification model to be trained, so that the audio classification model to be trained can predict the category of each audio frame in the sample audio and obtain the probability of each audio frame in the sample audio belonging to each audio category. Based on the probability of each audio frame in the sample audio belonging to each audio category, the probability of each audio frame in the sample audio belonging to each audio category is determined. Based on the probability of each audio frame in the sample audio belonging to each audio category and the audio category to which the sample audio belongs, the loss value is determined. The audio classification model to be trained is trained in the direction that reduces the loss value.

[0182] In one embodiment, the training module 1210 is further configured to, for each audio category, determine multiple candidate probabilities that the sample audio belongs to the target audio category based on the probability of each audio frame in the sample audio belonging to the target audio category using multiple preset functions; fuse the multiple candidate probabilities that the sample audio belongs to the target audio category using multiple fusion methods to obtain multiple fused probabilities that the sample audio belongs to the target audio category; and determine the probability that the sample audio belongs to the target audio category based on the multiple fused probabilities that the sample audio belongs to the target audio category.

[0183] In one embodiment, the determining module 1206 is further configured to: determine the background audio features of each background audio segment; determine the confidence level of each background audio segment belonging to a plurality of preset sub-category labels based on the background audio features; each sub-category label is used to characterize a musical mood of the background audio segment; and determine the sub-category label of the background audio segment based on the confidence level of each background audio segment belonging to a plurality of preset sub-category labels.

[0184] In one embodiment, each background audio segment included in the target audio segment intersects with at least one included human voice audio segment.

[0185] In one embodiment, for each background audio segment in the target audio segment, the proportion of time of at least one human voice audio segment that intersects with the target background audio segment in the target background audio segment exceeds a preset ratio.

[0186] In one embodiment, the video is a film or television drama video; the voice category is a voice dialogue category; the voice audio segment is a voice dialogue audio segment; the topic tag is a theme tag of the film or television drama video; and the video segment is an edited clip of the film or television drama video.

[0187] The aforementioned video processing apparatus acquires video and extracts audio from the video; in the audio, it locates at least one background audio segment belonging to the background music category and at least one human voice audio segment belonging to the human voice category; it acquires the video's theme tag and determines the sub-category tag for each background audio segment; in the audio, it locates at least one target audio segment, each target audio segment including at least one complete background audio segment and at least one complete human voice audio segment, the union of the included background audio segments and human voice audio segments being continuous, and the sub-category tag of the included background audio segment matching the video's theme tag; and it extracts at least one video segment from the video according to at least one target audio segment. Compared to traditional methods that directly extract video segments from videos based on background music, this application identifies human voice audio segments and background audio segments matching the video theme from the video's audio. By combining the identified human voice and background audio segments, a target audio segment matching the video theme, including both complete background and human voice audio segments, is located from the video's audio. Then, based on the located target audio segment, a video segment matching the video theme, including both complete background and human voice audio segments, is extracted from the video. This improves the accuracy of video segment extraction and avoids wasting computer hardware resources used to support the extraction of the required video segment.

[0188] Each module in the aforementioned video processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0189] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 14 As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and databases. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media to run. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a video processing method.

[0190] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 15As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a video processing method. The display unit of the computer device is used to form a visually visible image. It can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0191] Those skilled in the art will understand that Figure 14 and Figure 15 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0192] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0193] In one embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0194] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0195] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data shall comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0196] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0197] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0198] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A video processing method, characterized in that, The method includes: Acquire video clips of movies and TV series, and extract audio from the video clips of movies and TV series; Multiple temporal audio frames obtained by segmenting the audio are acquired. Through a trained audio classification model, the probability curves of the multiple temporal audio frames belonging to each audio category are determined. Based on the probability curves of each audio frame belonging to each audio category, the audio category to which the multiple temporal audio frames belong is determined. The audio category is either background music or human voice dialogue. In the audio, based on the audio categories to which the plurality of time-domain audio frames belong, at least one background audio segment belonging to the background music category and at least one human voice audio segment belonging to the human voice dialogue category are located. Obtain the subject theme tags for the film and television videos, whereby the subject theme tags are used to describe the content theme of the film and television videos; For each background audio segment, a pre-trained music emotion recognition model is used to extract background audio features of the background audio segment. Based on the background audio features of the background audio segment, the confidence level of the background audio segment belonging to multiple preset sub-category labels is determined. Each sub-category label is used to characterize a music emotion of the background audio segment. The sub-category label of the background audio segment is determined according to the confidence level of the multiple preset sub-category labels. The sub-category label is used to describe the background music emotion of the background audio segment. In the audio, at least one target audio segment is located, and at least one exciting clip is extracted from the film / television video according to the at least one target audio segment; Each target audio segment satisfies all of the following conditions:

1. Each target audio segment includes at least one complete background audio segment and at least one complete human voice dialogue audio segment. The complete background audio segment is the background audio segment in the audio where the background music is played without interruption. The complete human voice dialogue audio segment is the human voice dialogue audio segment in the audio where the human voice is played without interruption. The union of the at least one complete background audio segment and the at least one complete human voice dialogue audio segment constitutes the target audio segment. Second, the union of the time periods of the background audio segment and the time periods of the human voice dialogue audio segment included in each target audio segment is continuous; Third, each background audio segment included in the target audio segment intersects with at least one human voice dialogue audio segment included, and the human voice dialogue audio segment intersecting with the background audio segment occupies a time proportion of the background audio segment exceeding a preset ratio, so that each background audio segment included in the target audio segment contains human voices of the corresponding duration of the time proportion. Fourth, the subcategory tags of the background audio segments included in each target audio segment have at least one tag that is the same as or similar to the theme tag of the film and television video.

2. The method according to claim 1, characterized in that, The step of determining the probability curves for each of the plurality of time-domain audio frames to belong to each audio category includes: The multiple time-domain audio frames are converted to the frequency domain to obtain multiple audio frequency domain signals; Temporal features are extracted from the multiple temporal audio frames, frequency features are extracted from the multiple audio frequency signals, and the temporal features and frequency features are fused to obtain audio fusion features; Based on the audio fusion features, the probability of each of the multiple time-domain audio frames belonging to each audio category is predicted, and based on the probability of each of the multiple time-domain audio frames belonging to each audio category, the probability curve of each of the multiple time-domain audio frames belonging to each audio category is determined.

3. The method according to claim 2, characterized in that, The step of extracting temporal features from the plurality of temporal audio frames, extracting frequency domain features from the plurality of audio frequency domain signals, and fusing the temporal features and the frequency domain features to obtain audio fusion features includes: The features of the multiple time-domain audio frames are extracted layer by layer according to multiple preset levels to obtain the time-domain features of each level. The features of the multiple audio frequency domain signals are extracted layer by layer according to the multiple levels to obtain the frequency domain features of each level; For each layer, feature fusion is performed at least based on the time-domain and frequency-domain features of the targeted layer to generate layer fusion features for the targeted layer. When generating layer fusion features for layers after the first layer, feature fusion is also performed based on the layer fusion features of the previous layer. Audio fusion features are determined based on the layer fusion features of the last layer among the multiple layers.

4. The method according to claim 3, characterized in that, The time-domain features and the frequency-domain features have different feature dimensions; for each level, feature fusion is performed at least based on the time-domain features and frequency-domain features of the targeted level to generate layer fusion features for the targeted level. Specifically, when generating layer fusion features for levels after the first level, feature fusion is also performed based on the layer fusion features of the previous level, including: For each level, based on the feature dimension of the frequency domain features of the level, the time domain features of the level are reconstructed to obtain the reconstructed time domain features of the level. When the target layer is the first layer, the frequency domain features of the target layer and the reconstructed time domain features are fused to obtain the layer fusion features of the target layer. When the target layer is a layer after the first layer, the frequency domain features of the target layer, the reconstructed time domain features of the target layer, and the layer fusion features of the layer preceding the target layer are fused to obtain the layer fusion features of the target layer.

5. The method according to claim 4, characterized in that, The layer fusion feature includes multiple feature maps; determining the audio fusion feature based on the layer fusion feature of the last layer among the multiple layers includes: The feature maps in the layer fusion feature of the last layer of the multiple layers are fused to obtain a fused feature map; The dimensionality of the fused feature map is reduced to obtain audio fusion features with the same feature dimension as each temporal feature before reconstruction.

6. The method according to claim 1, characterized in that, The audio classification model is obtained through a model training step, which includes: Obtain training samples; the training samples include sample audio and the audio category to which the sample audio belongs, labeled for the sample audio; The sample audio is input into the audio classification model to be trained, so that the audio classification model to be trained can predict the category of each audio frame in the sample audio and obtain the probability that each audio frame in the sample audio belongs to each audio category. Based on the probability that each audio frame in the sample audio belongs to each audio category, the probability that the sample audio belongs to each audio category is determined. The loss value is determined based on the probability that the sample audio belongs to each audio category, and the audio category to which the sample audio is labeled. The audio classification model to be trained is trained in the direction that reduces the loss value.

7. The method according to claim 6, characterized in that, The step of determining the probability that the sample audio belongs to each audio category based on the probability that each audio frame in the sample audio belongs to each audio category includes: For each audio category, multiple preset functions are used to determine multiple candidate probabilities that the sample audio belongs to the target audio category based on the probability that each audio frame in the sample audio belongs to the target audio category. By using multiple fusion methods, the multiple candidate probabilities of the sample audio belonging to the target audio category are fused to obtain multiple fused probabilities of the sample audio belonging to the target audio category. The probability that the sample audio belongs to the target audio category is determined based on multiple fusion probabilities of the sample audio belonging to the target audio category.

8. A video processing apparatus, characterized in that, The device includes: The acquisition module is used to acquire film and television videos and extract audio from the film and television videos; The localization module is used to acquire multiple temporal audio frames obtained by segmenting the audio, determine the probability curves of each of the multiple temporal audio frames belonging to each audio category through a trained audio classification model, and determine the audio category to which each of the multiple temporal audio frames belongs based on the probability curves of each audio frame belonging to each audio category. The audio category is either background music or human voice dialogue. In the audio, based on the audio category to which the multiple temporal audio frames belong, at least one background audio segment belonging to the background music category and at least one human voice dialogue audio segment belonging to the human voice dialogue category are located. The determination module is used to obtain the theme tags for the film and television videos, the theme tags being used to describe the content theme of the film and television videos. For each background audio segment, a pre-trained music emotion recognition model is used to extract the background audio features of the background audio segment, and based on the background audio features of the background audio segment, the confidence level of the background audio segment belonging to multiple preset sub-category tags is determined. Each sub-category tag is used to characterize a musical emotion of the background audio segment. The sub-category tag of the background audio segment is determined according to the confidence level of the multiple preset sub-category tags, and the sub-category tag is used to describe the background music emotion of the background audio segment. The positioning module is also used to locate at least one target audio segment in the audio; The extraction module is used to extract at least one exciting segment from the video of the film or television drama according to the at least one target audio segment; Each target audio segment satisfies all of the following conditions:

1. Each target audio segment includes at least one complete background audio segment and at least one complete human voice dialogue audio segment. The complete background audio segment is the background audio segment in the audio where the background music is played without interruption. The complete human voice dialogue audio segment is the human voice dialogue audio segment in the audio where the human voice is played without interruption. The union of the at least one complete background audio segment and the at least one complete human voice dialogue audio segment constitutes the target audio segment. Second, the union of the time periods of the background audio segment and the time periods of the human voice dialogue audio segment included in each target audio segment is continuous; Third, each background audio segment included in the target audio segment intersects with at least one human voice dialogue audio segment included, and the human voice dialogue audio segment intersecting with the background audio segment occupies a time proportion of the background audio segment exceeding a preset ratio, so that each background audio segment included in the target audio segment contains human voices of the corresponding duration of the time proportion. Fourth, the subcategory tags of the background audio segments included in each target audio segment have at least one tag that is the same as or similar to the theme tag of the film and television video.

9. The apparatus according to claim 8, characterized in that, The positioning module is also used to convert the plurality of time-domain audio frames to the frequency domain to obtain a plurality of audio frequency domain signals; extract time-domain features from the plurality of time-domain audio frames, extract frequency domain features from the plurality of audio frequency domain signals, and fuse the time-domain features and the frequency domain features to obtain audio fusion features; Based on the audio fusion features, the probability of each of the multiple time-domain audio frames belonging to each audio category is predicted, and based on the probability of each of the multiple time-domain audio frames belonging to each audio category, the probability curve of each of the multiple time-domain audio frames belonging to each audio category is determined.

10. The apparatus according to claim 9, characterized in that, The positioning module is also used to extract features from the multiple time-domain audio frames layer by layer according to a preset multiple levels to obtain the time-domain features of each level; and to extract features from the multiple audio frequency-domain signals layer by layer according to the multiple levels to obtain the frequency-domain features of each level. For each level, feature fusion is performed at least based on the time-domain and frequency-domain features of the target level to generate layer fusion features for the target level. When generating layer fusion features for levels after the first level, feature fusion is also performed based on the layer fusion features of the previous level. Audio fusion features are determined based on the layer fusion features of the last level among the multiple levels.

11. The apparatus according to claim 10, characterized in that, The time-domain features and the frequency-domain features have different feature dimensions; the localization module is also used to perform feature reconstruction on the time-domain features of the targeted layer according to the feature dimensions of the frequency-domain features of the targeted layer for each layer, to obtain the reconstructed time-domain features of the targeted layer; when the targeted layer is the first layer, the frequency-domain features and the reconstructed time-domain features of the targeted layer are fused to obtain the layer-fused features of the targeted layer; when the targeted layer is a layer after the first layer, the frequency-domain features of the targeted layer, the reconstructed time-domain features of the targeted layer, and the layer-fused features of the layer preceding the targeted layer are fused to obtain the layer-fused features of the targeted layer.

12. The apparatus according to claim 11, characterized in that, The layer fusion feature includes multiple feature maps; the localization module is further configured to fuse each feature map in the layer fusion feature of the last layer among the multiple layers to obtain a fused feature map; and to reduce the dimensionality of the fused feature map to obtain an audio fusion feature with the same feature dimension as each temporal feature before reconstruction.

13. The apparatus according to claim 8, characterized in that, The audio classification model is obtained through a model training step, and the device further includes: A training module is used to acquire training samples, which include sample audio and the audio category to which the sample audio belongs, labeled for the sample audio. The sample audio is input into an audio classification model to be trained, so that the audio classification model predicts the category of each audio frame in the sample audio, obtaining the probability that each audio frame in the sample audio belongs to each audio category. Based on the probability that each audio frame in the sample audio belongs to each audio category, the probability that the sample audio belongs to each audio category is determined. Based on the probability that the sample audio belongs to each audio category and the audio category to which the sample audio belongs, a loss value is determined. The audio classification model to be trained is trained in the direction that reduces the loss value.

14. The apparatus according to claim 13, characterized in that, The training module is also used to determine multiple candidate probabilities of the sample audio belonging to the target audio category for each audio category using multiple preset functions. By using multiple fusion methods, the multiple candidate probabilities of the sample audio belonging to the target audio category are fused to obtain multiple fused probabilities of the sample audio belonging to the target audio category; based on the multiple fused probabilities of the sample audio belonging to the target audio category, the probability of the sample audio belonging to the target audio category is determined.

15. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

16. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

17. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images