Target video explanation method, system and terminal based on video and audio synthesis

Abstract

The invention relates to a target video explanation method, system and terminal based on video and audio synthesis, and belongs to the technical field of video and audio processing, and the target video explanation method comprises the steps: obtaining a target video; dividing the target video into a plurality of adjustment sub-videos according to the target duration of the target video; decomposing each adjusted sub-video into a role layer, a scene layer and an interaction layer; matching a first audio according to the role layer, matching a second audio according to the scene layer, and matching a third audio according to the interaction layer; fusing the first audio, the second audio and the third audio to generate a sub-audio; associating the adjusted sub-videos with the corresponding sub-audios; and when the target video is played to a certain adjusted sub-video range, performing explanation according to the corresponding sub-audio. The method has the beneficial effect of improving the watching experience of the user.

Description

Technical Field

[0001] The present application relates to the technical field of audio - visual processing, and in particular, to a target video commentary method, system and terminal based on audio - visual synthesis. Background Art

[0002] In the current era of rapid development of digital information, video, as an important way of information dissemination and entertainment, has been widely used in various fields, such as education, entertainment, advertising, etc. With the continuous improvement of users' requirements for video content quality and viewing experience, how to provide more high - quality, accurate and attractive commentaries for videos has become a key problem to be solved urgently.

[0003] Existing video commentary methods have many limitations. On the one hand, traditional video commentaries often provide a unified and relatively single commentary for the entire video, with a low degree of fit between the commentary and the video content, making it difficult to accurately display the plot and emotions of the video. As a result, viewers are prone to understanding obstacles and have a poor viewing experience.

[0004] On the other hand, in the selection and application of audio, traditional methods are relatively simple and the audio levels are not rich enough. Usually, only a single background music or commentary sound is used, making it difficult for viewers to feel immersive. The overall quality and expressiveness of the video are limited, and the viewing experience is poor. Summary of the Invention

[0005] In order to improve the viewing experience of users, the present application provides a target video commentary method, system and terminal based on audio - visual synthesis.

[0006] In the first aspect, the present application provides a target video commentary method based on audio - visual synthesis, adopting the following technical solutions: A target video commentary method based on audio - visual synthesis, comprising: Obtaining a target video; Dividing the target video into multiple adjusted sub - videos according to the target duration of the target video; Decomposing each of the adjusted sub - videos into a character layer, a scene layer and an interaction layer; Matching a first audio according to the character layer, matching a second audio according to the scene layer, and matching a third audio according to the interaction layer; Fusing the first audio, the second audio and the third audio to generate a sub - audio; Associating the adjusted sub - video with the corresponding sub - audio; When the target video plays to a certain range of the adjusted sub - video, commentary is made according to the corresponding sub - audio.

[0007] By adopting the above technical solution, the target video is divided into multiple adjustment sub-videos, making the editing of the target video more flexible. Each adjustment sub-video is relatively independent, and can be personalized for different sub-videos, such as adding special effects, adjusting colors, etc., without affecting other parts, improving editing efficiency and accuracy. Multiple adjustment sub-videos are easy to classify and manage. The adjustment sub-videos can be sorted according to different themes, plots, etc., which is convenient for subsequent search and use; in addition, the adjustment sub-videos are decomposed into role layers, scene layers, and interactive layers, and different audios are matched respectively, which can greatly enrich the audio levels. The first audio matched by the role layer highlights the voice characteristics of the character, the second audio matched by the scene layer creates a corresponding environmental atmosphere, and the third audio matched by the interactive layer enhances the interactive effect. The fused sub-audio is more vivid and three-dimensional, which improves the overall quality of the video; matching the corresponding audio according to different levels makes the audio and video content more compatible; it can accurately select the appropriate audio according to the characteristics of different elements in the video, better show the plot and emotion of the video, and enhance the expressiveness of the video. Adjusting the close association between the sub-video and the corresponding sub-audio, and providing commentary based on the sub-audio when playing to the corresponding range, can allow the audience to obtain a more coherent and natural audio-visual experience while watching the video. For example, when watching a video with a historical theme, as the video content advances, the corresponding commentary audio sounds in a timely manner, combined with the sound that matches the characters, scenes, and interactions, so that the audience feels as if they are on the scene, and it is easier to immerse themselves in the context created by the video. Accurately matched audio and commentary can better guide the audience to understand the content of the video. The commentary can explain and illustrate the key information and plot development in the video, help the audience better grasp the main idea and details of the video, and improve the fluency and understanding of the viewing. This application can flexibly adjust the combination of audio and commentary according to different types of video content. For educational videos, the knowledge points can be explained in detail through commentary; for entertainment videos, the commentary can increase the fun and interactivity. It can meet the viewing needs of different user groups in different scenarios and improve the applicability and attractiveness of the video.

[0008] Optionally, the step of dividing the target video into a plurality of adjusted sub-videos according to the target duration of the target video includes: Divide the target video into multiple sub-videos according to the target duration; Performing correlation analysis on adjacent sub-videos to obtain analysis results; According to the analysis result, adjacent sub-videos are spliced ​​to generate an adjusted sub-video.

[0009] By adopting the above technical solution, the target video is first equally divided into multiple sub-videos according to the target duration. This preliminary division method is simple and direct, providing a basis for subsequent fine processing. It enables operations on the target video to be carried out on smaller units, reducing the complexity of editing. Editors can more conveniently view and process each sub-video individually. Conducting relevance analysis and splicing on the sub-videos to generate adjusted sub-videos further enhances the flexibility of editing. Conducting relevance analysis on adjacent sub-videos can deeply understand the internal connections between video contents. By analyzing factors such as picture content, plot development, and character relationships, accurate analysis results can be obtained. Splicing according to the analysis results ensures that the adjusted sub-videos are coherent and logical in content. After adjacent video segments are spliced, the plot transitions naturally without abrupt jumps, enabling viewers to better understand the information conveyed by the video when watching it and enhancing the narrative effect of the video. For example, in a documentary, by reasonably splicing relevant sub-videos, the theme of the entire video can be presented more clearly and orderly.

[0010] Optionally, the step of conducting relevance analysis on adjacent sub-videos to obtain analysis results includes: Conducting relevance analysis on the visual features and audio features at the same time point in adjacent sub-videos respectively to obtain multiple visual feature similarity values and audio feature similarity values; adjacent sub-videos are analyzed from the beginning and the end respectively; Screening the time points where the error values of adjacent visual feature similarity values are within the error threshold to form a first set of time points, and screening the time points where the error values of adjacent audio feature similarity values are within the error threshold to form a second set of time points; Judging whether there are consecutive adjacent time points in both the first set of time points and the second set of time points respectively; If not, the analysis result is that the current adjacent sub-videos do not need to be spliced; If so, if there are consecutive adjacent time points in the first set of time points, generating a first adjustment duration according to the consecutive adjacent time points, and determining a first adjusted video range according to the first adjustment duration; if there are consecutive adjacent time points in the second set of time points, generating a second adjustment duration according to the consecutive adjacent time points, and determining a second adjusted video range according to the second adjustment duration; Judging whether there is an overlapping adjustment range between the first adjusted video range and the second adjusted video range; If so, the analysis result is that the current adjacent sub-videos need to be split and spliced, and the split and splice range is the overlapping adjustment range; If not, the analysis result is that the current adjacent sub-videos do not need to be spliced.

[0011] By adopting the above technical solution, through the correlation analysis of the visual features and audio features of adjacent sub-videos at the same time point, and screening out the set of eligible time points based on the error threshold, it is possible to accurately find the parts with strong correlation at the visual and audio levels. For example, when processing adjacent sub-videos of different scenes in a documentary, based on the screening of visual feature similarity, it is possible to accurately identify the time intervals with relatively similar picture colors and scene elements. In terms of audio, it is also possible to find the parts where the background music and environmental sound effects are connected naturally, avoiding the problem of abrupt pictures or sounds caused by random splicing, making the content transition of the spliced video more logical and natural and smooth, and improving the overall viewing experience. At the same time, taking into account both the visual and audio, these two key feature dimensions, compared with only relying on a single feature to judge whether to splice, can more comprehensively consider the correlation between adjacent sub-videos. For example, some videos may seem to have a certain coherence in terms of pictures, but the audio is very different. If only the visual features are considered for splicing, there will be a situation of disharmony between sound and picture. And this process analyzes both of them to ensure that the spliced video can achieve a better fusion effect both visually and aurally, enhancing the integrity of the video.

[0012] Optionally, the step of fusing the first audio, the second audio, and the third audio to generate sub-audio includes: Obtain the first commentary requirement; According to the first commentary requirement, adjust the fusion order of the first audio, the second audio, and the third audio; According to the fusion order, input the first audio, the second audio, and the third audio into a pre-trained word-association model to generate sub-audio.

[0013] By adopting the above technical solution, taking the first step of the explanation requirement, the audio fusion process can be closely carried out around the specific requirements of the user or project. Different video scenes and target audiences may have different expectations for audio fusion. For example, in educational videos, they may want to highlight the explanation sound (first audio); in entertainment videos, they may emphasize the coordination of environmental sound effects (second audio) and interactive sound effects (third audio). By clarifying the explanation requirements, the audio fusion strategy can be adjusted in a targeted manner to meet diverse application scenarios and improve the applicability of video explanation. Adjusting the fusion order according to the first explanation requirement gives audio fusion great flexibility. Different fusion orders will cause the sub-audio to present different effects. For example, the sub-audio produced by first fusing the first audio and then adding the second and third audios will have obvious differences in the primary and secondary sounds and the creation of the atmosphere, etc., compared with first mixing the second and third audios and then fusing them into the first audio. This method of flexibly adjusting the fusion order according to the needs can accurately meet various explanation requirements and provide the most matching audio support for the video. The audio with the adjusted fusion order is input into the pre-trained word and sentence association model to obtain the sub-audio, which makes use of the powerful learning and processing capabilities of the model. After being trained with a large amount of data, the word association model can understand the semantic, emotional and acoustic relationships between audios, so that it can better coordinate the timbre, volume, rhythm and other elements of different audios during the fusion process, making the generated sub-audio more natural and harmonious. The model can intelligently mix and adjust the audio according to the content and characteristics of the audio, avoiding the sound conflicts or inharmonious problems that may be caused by simple superposition. For example, during the fusion process, the model can balance the emotional tendencies of different audios, so that the sub-audio can create an appropriate emotional atmosphere while conveying information, improving the quality and expressiveness of the audio.

[0014] Optionally, after generating the sub-audio, the step further includes: Obtain the second interpretation requirement; According to the second interpretation requirement, matching an interpretation template from an interpretation template library; Fill the sub-audio into the commentary template to generate the commentary audio.

[0015] By adopting the above technical solution, matching the commentary template from the commentary template library according to the second commentary requirement can make full use of a variety of pre-constructed commentary template resources. The commentary template library contains commentary templates of various styles, tones, and rhythms, covering a variety of common commentary requirement scenarios. By matching the template, a basic framework that is more suitable for the current commentary requirement can be quickly found, greatly improving the efficiency and pertinence of the commentary audio generation and meeting diverse customization requirements. By generating the commentary audio based on the commentary template, targeted creation can be carried out according to the characteristics of the sub-audio. For example, when the sub-audio has a lively rhythm and is full of vitality, the commentary template can guide the generation of a commentary audio with a slightly faster speaking speed and a lively tone that is adapted to it; when the atmosphere of the sub-audio is relatively quiet and soothing, the commentary audio can also adopt a gentle and steady style accordingly, so as to achieve a high degree of matching between the commentary audio and the sub-audio and bring a more harmonious and natural audio-visual experience to the audience.

[0016] Optionally, the commentary method further includes: Obtain the facial expression image of the target user; Input the facial expression image into a pre-trained micro-expression analysis model to obtain the expression type; Judge whether the expression type is negative; a negative expression type refers to a negative facial expression type; If so, adjust the commentary template and commentary volume corresponding to the commentary audio according to the negative level.

[0017] By adopting the above technical solution, the expression image of the target user is obtained and the expression type is obtained through the micro-expression analysis model, which can accurately understand the emotional response of the user during the video viewing process. Micro-expressions are short-term and unconscious external expressions of human inner true emotions. In this way, the subtle emotional changes of users can be captured, and their true feelings about the video content can be understood, providing a basis for subsequent adjustments. When the expression type is judged to be negative, the commentary template and commentary volume corresponding to the commentary audio are adjusted according to the negative level. This means that the system can change the commentary method in a targeted manner according to the specific emotional state of the user to better meet the needs of the user. For example, if the user shows negative emotions, lowering the commentary volume can avoid causing more user disgust; or replacing a more gentle and soothing commentary template to make the commentary style more in line with the user's current emotions, thereby improving the user's overall experience of watching the video and enhancing the interaction and resonance between the user and the video content. This mechanism of adjusting the commentary in real time according to the user's expression enables the video commentary to dynamically adapt to changes in the user's emotions. In the process of watching a video, the user's emotions may change continuously with the development of the video plot. Through continuous monitoring and adjustment, the commentary audio can always match the user's current emotional state. For example, when users are confused or dissatisfied with the video content, timely adjustment of the explanation method can better guide users to understand the video, alleviate negative emotions, and form a good interactive relationship between the explanation and the user's emotions. Reasonable adjustment of the explanation template and volume can convey the video content more effectively. When users are in a negative mood, the appropriate explanation method can attract users' attention, help them better understand the video information, and avoid missing important content due to emotional problems. At the same time, this improvement in matching can also help enhance the influence of video content on users and optimize the dissemination effect of the video.

[0018] Optionally, after adjusting the commentary template and the commentary volume corresponding to the commentary audio according to the negative level, the method further includes: Continuously and periodically analyzing the expressions of the target user; Determine whether the expression type changes from negative to positive; a positive expression type is a positive expression type; If so, adjust to the initial commentary style; If not, change the commentary language type and add commentary special effects.

[0019] By adopting the above technical solutions, when it is detected that the user's expression has not changed to a positive one, timely changing the type of commentary language and adding commentary special effects helps to re - attract the user's interest and ensure the effective transmission of information. Different types of commentary languages and special effects can arouse the user's curiosity, prompting them to continue to pay attention to the video content, thereby increasing the success rate of information transmission. Optimize user emotion management: Improving the user's emotion by adjusting the commentary method reflects the concern for the user's mental health. This ability to actively regulate the user's emotion can not only enhance the user's viewing experience but also help the user find a positive outlet when facing negative emotions, playing a certain role in promoting mental health.

[0020] In the second aspect, the present application provides a target video commentary system based on video - audio synthesis, adopting the following technical solutions: A target video commentary system based on video - audio synthesis, comprising: A video acquisition module, configured to acquire a target video; A video processing module, configured to divide the target video into a plurality of adjusted sub - videos according to the target duration of the target video, and decompose each adjusted sub - video into a character layer, a scene layer, and an interaction layer; An audio processing module, configured to match a first audio according to the character layer, match a second audio according to the scene layer, match a third audio according to the interaction layer, fuse the first audio, the second audio, and the third audio to generate a sub - audio, and associate the adjusted sub - video with the corresponding sub - audio; A commentary module, configured to provide commentary according to the corresponding sub - audio when the target video plays within the range of a certain adjusted sub - video.

[0021] Optionally, the video processing module includes: A segmentation unit, configured to equally divide the target video into a plurality of sub - videos according to the target duration; A video analysis unit, configured to perform a correlation analysis on adjacent sub - videos to obtain an analysis result; A video splicing unit, configured to splice adjacent sub - videos according to the analysis result to generate an adjusted sub - video.

[0022] In the third aspect, the present application provides a terminal, adopting the following technical solutions: A terminal, comprising: A memory, storing a target video commentary program based on video - audio synthesis; A processor, configured to execute the program stored on the memory to implement the steps of the above - mentioned target video commentary method based on video - audio synthesis.

[0023] In summary, the present application has at least the following beneficial effects: 1. According to the target duration of the target video, the target video is divided into multiple adjusted sub-videos. Then each adjusted sub-video is decomposed into a character layer, a scene layer, and an interaction layer. The first audio is matched according to the character layer, the second audio is matched according to the scene layer, and the third audio is matched according to the interaction layer. Then the first audio, the second audio, and the third audio are fused to generate a sub-audio, thereby associating the adjusted sub-video with the corresponding sub-audio. When the target video is played to a certain range of the adjusted sub-video, the purpose of the commentary according to the corresponding sub-audio is to perform personalized processing for different sub-videos. The first audio matched by the character layer highlights the voice characteristics of the characters, the second audio matched by the scene layer creates the corresponding environmental atmosphere, and the third audio matched by the interaction layer enhances the interaction effect. The fused sub-audio is more vivid and three-dimensional, improving the overall quality of the video. Matching the corresponding audio according to different levels makes the audio more compatible with the video content, and the combination of audio and commentary can be flexibly adjusted according to different types of video content. For educational videos, the knowledge points can be explained in detail through the commentary; for entertainment videos, the commentary can increase the fun and interactivity. It can meet the viewing needs of different user groups in different scenarios, improving the applicability and attractiveness of the video.

[0024] 2. Obtain the facial expression image of the target user, input the facial expression image into the micro-expression analysis model to obtain the expression type, and determine whether the expression type is negative. If so, adjust the explanation template and explanation volume corresponding to the explanation audio according to the negative level. The purpose is to accurately insight into the user's emotional reaction during the video viewing process. As a short and involuntary external manifestation of human's true inner emotions, micro-expressions can capture the user's subtle emotional changes in this way, understand their true feelings towards the video content, and provide a basis for subsequent adjustments. When it is determined that the expression type is negative, adjust the explanation template and explanation volume corresponding to the explanation audio according to the negative level. This means that the system can specifically change the explanation method according to the user's specific emotional state to better meet the user's needs. For example, if the user shows negative emotions, reducing the explanation volume can avoid causing more disgust from the user; or replacing it with a more gentle and soothing explanation template to make the explanation style more in line with the user's current emotions, thereby enhancing the overall experience of the user watching the video and strengthening the interaction and resonance between the user and the video content. This mechanism of adjusting the explanation in real time according to the user's facial expression enables the video explanation to dynamically adapt to the changes in the user's emotions. During the user's video viewing process, their emotions may continuously change as the video plot develops. Through continuous monitoring and adjustment, the explanation audio can always match the user's current emotional state. For example, when the user is confused or dissatisfied with the video content, adjusting the explanation method in a timely manner can better guide the user to understand the video, relieve negative emotions, and form a good interaction relationship between the explanation and the user's emotions. Reasonably adjusting the explanation template and volume can more effectively convey the video content. When the user is in a negative emotion, a suitable explanation method can attract the user's attention, help them better understand the video information, and avoid missing important content due to emotional problems. At the same time, the improvement of this matching degree also helps to enhance the influence of the video content on the user and optimize the dissemination effect of the video. Brief Description of the Drawings

[0025] Figure 1 is a flowchart of an implementation manner of Embodiment 1 of the method of the present application; Figure 2 is a flowchart of the specific steps of S120; Figure 3 is a flowchart of the specific steps of S122; Figure 4 is a flowchart of the specific steps of S150; Figure 5 is a flowchart of another implementation manner of Embodiment of the method of the present application; Figure 6 is a structural block diagram of the system embodiment of the present application. Detailed Embodiment

[0026] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the following will describe the technical solutions in the embodiments of the present invention in conjunction with the appended Figure 1 - appendices Figure 6 , and clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of them. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.

[0027] The first embodiment of this application discloses a target video commentary method based on video-audio synthesis. Referring to Figure 1 , as an implementation manner of the target video commentary method for this video-audio synthesis, the target video commentary method may include S110 - S170: S110, obtain a target video; S120, according to the target duration of the target video, divide the target video into multiple adjusted sub-videos; S130, decompose each adjusted sub-video into a character layer, a scene layer, and an interaction layer; S140, match a first audio according to the character layer, match a second audio according to the scene layer, and match a third audio according to the interaction layer; S150, fuse the first audio, the second audio, and the third audio to generate a sub-audio; S160, associate the adjusted sub-video with the corresponding sub-audio; S170, when the target video plays to a certain range of the adjusted sub-video, conduct commentary according to the corresponding sub-audio.

[0028] Specifically, the method for obtaining the target video is not limited. It can be manually input or downloaded through a third-party platform, etc. The target video can be a video with the sound removed or an original video. After obtaining the target video, count the target duration of the target video, that is, the playing duration of the target video, and then divide the target video into multiple adjusted sub-videos according to the target duration. Then, decompose each adjusted sub-video into a character layer, a scene layer, and an interaction layer. The character layer includes the images of people, animals, and other anthropomorphic features in the adjusted sub-video and other related features; the scene layer includes the background in the adjusted sub-video, such as location, time, weather, etc.; the interaction layer includes the actions of the characters and various interaction forms between characters, such as dialogue exchanges, eye contacts, physical contacts, conflicts, etc. A deep learning image recognition model, such as a combination of a target detection model and an image classification model, can be used to distinguish the character layer and the scene layer, and a natural language processing model, such as a combination of a sentiment analysis model and a semantic understanding model, can be used for the interaction layer and the analysis related to the characters in the character layer. Additionally, a video understanding model, such as a 3D convolutional neural network model, can also be directly used to distinguish each layer.

[0029] Taking the example of an animated short video about several small animals searching for treasure in the forest, for the character layer, first, the sub-video is imported into a tool that supports video frame extraction. After extracting the frames one by one, they are input into the YOLOv5 model for detection. The model will accurately frame the small animal characters in each frame, such as identifying different individual characters like little rabbits, little monkeys, little birds, etc., and determining their specific positions and approximate size ranges in the frame. By counting the detection results of all frames, we can clearly know the frequency of each character's appearance in the video. The little rabbit appears the most times and runs through the whole video, indicating that it may be the protagonist; while the little bird only appears occasionally and is probably a supporting role. Then, an image classification model (such as ResNet) is used to refine the classification of the identified small animal characters, determining that the appearance characteristics of the little rabbit belong to the category of white fur and long ears, the little monkey belongs to the category of brown fur and long tail, etc., to more clearly distinguish different characters from the appearance. At the same time, extract the voice dialogue content of the characters in the video (if any), and use a semantic understanding model in natural language processing (such as BERT) to analyze the semantics of the characters' speech. For example, the little rabbit always mentions the need to move forward bravely and not give up. From this, it can be inferred that the little rabbit has the character traits of bravery and perseverance, further improving the understanding of the characters.

[0030] For the scene layer, various objects in the forest scene are detected through an object detection model, such as tall trees, colorful flowers, stones hidden in the grass, etc. This object information helps us judge the richness of the scene. The image classification model further determines the general category of the entire scene, clarifying that this is an outdoor forest natural scene. Moreover, by analyzing the color tone and light and shadow effects of the picture (for example, the overall color tone of the picture is greenish, and the sunlight shines through the leaves to form mottled light and shadow, judging it as a sunny forest environment during the day), the atmosphere of the scene can be clearly described as relatively fresh and full of vitality. A video understanding model (such as C3D) is used to track the changes of the scene over time. It is found that as the small animals keep moving forward, the picture switches from an open grassland forest scene to an area with denser trees and more rugged roads. By capturing this scene change through the model, the node of the scene transition can be accurately divided, knowing that the scene transition probably occurs when the video is played to about half, and analyzing that the scene transition is naturally transitioned through the panning of the camera. This transition method conforms to the logical route of the small animals searching for treasure and maintains the coherence of the video plot.

[0031] For the interaction layer, extract the dialogue text between the small animals and input it into a sentiment analysis model (such as a sentiment analysis model based on the Transformer architecture) to analyze the sentiment tendency during their communication. For example, when the little monkey suggests taking a break but the little rabbit encourages everyone to keep going, the sentiment analysis model determines that the sentiment of the little rabbit's words is positive and firm, while the little monkey is a bit helpless but also follows the advice, which reflects the emotional state and character differences during their interaction. The semantic understanding model further analyzes the dialogue content to clarify that the purpose of this interaction is to discuss the next action. The little rabbit wants to find the treasure as soon as possible, so it encourages everyone not to stop, thereby sorting out the behavioral motivation behind the interaction. By observing the interaction between the small animals and the forest scene through a video understanding model (C3D), it can be seen that the little rabbit uses a branch to cross a small stream, and the little monkey uses a vine to swing over an obstacle. These interaction behaviors between the characters and the scene elements are completely captured, and it is analyzed that the purpose of their interaction is to overcome the difficulties on the way to find the treasure by using the objects in the scene. At the same time, it also creates an atmosphere full of fun and challenges, making the video more ornamental. Moreover, from the perspective of the overall video time period, the interaction rhythm between the small animals and between them and the scene is from slow to fast. As they get closer to the treasure, the interaction becomes more frequent and intense. The model clearly grasps this dynamic change in the interaction and its role in promoting the video plot.

[0032] Referring to Figure 2 , specifically for S120, it may include S121 - S123: S121, divide the target video into multiple sub - videos according to the target duration; S122, conduct a relevance analysis on adjacent sub - videos to obtain the analysis result; S123, splice adjacent sub - videos according to the analysis result to generate adjusted sub - videos.

[0033] Referring to Figure 3 , specifically for S122, it may include S1221 - S1228: S1221, respectively conduct a relevance analysis on the visual features and audio features at the same time point in adjacent sub - videos to obtain multiple visual feature similarity values and audio feature similarity values; adjacent sub - videos are analyzed from the beginning and the end respectively; S1222, screen the time points whose error values of adjacent visual feature similarity values are within the error threshold to form a first set of time points, and screen the time points whose error values of adjacent audio feature similarity values are within the error threshold to form a second set of time points; S1223, respectively determine whether there are consecutive adjacent time points in both the first set of time points and the second set of time points; S1224, if not, the analysis result is that the current adjacent sub - videos do not need to be spliced; S1225, if there are continuous adjacent time points in the first time point set, a first adjustment duration is generated according to the continuous adjacent time points, and a first adjustment video range is determined according to the first adjustment duration; if there are continuous adjacent time points in the second time point set, a second adjustment duration is generated according to the continuous adjacent time points, and a second adjustment video range is determined according to the second adjustment duration; S1226, determining whether the first adjustment video range and the second adjustment video range have overlapping adjustment ranges; S1227: If yes, the analysis result is that the current adjacent sub-videos need to be split and spliced, and the split and splicing range is the overlap adjustment range.

[0034] S1228: If not, the analysis result is that the current adjacent sub-videos do not need to be spliced.

[0035] For example, a video recording a campus art performance is divided into multiple sub-videos of equal length. Two adjacent sub-videos are selected. Sub-video A records the end of the students' dance performance on the stage, which is 2 minutes (120 seconds); sub-video B records the beginning of the host's appearance on stage after the dance performance and the preparation for the next program, which is also 2 minutes (120 seconds). The error threshold is set to 0.2 (the similarity value range is between 0-1 and can be flexibly adjusted according to actual conditions).

[0036] First, the feature correlation analysis of the visual and audio features of the two sub-videos is performed. In terms of visual features, the visual features of the ending part of sub-video A (the first 30 seconds can be analyzed in detail, and the time points are counted from the end, such as the 1st second corresponds to the actual 120th second, the 2nd second corresponds to the 119th second, and so on) and the beginning part of sub-video B (normally the time points are counted from the beginning) are extracted. By calculating the color histogram and comparing the color of the picture, it can be seen that the overall tone of the picture at the end of sub-video A is bright, there are lighting effects of dance performances on the stage, and the students are wearing colorful dance costumes; the beginning of sub-video B is also bright due to the stage lighting, the background is the stage layout, the host is dressed appropriately and stands on the stage, and there are props prepared for the next show next to him, etc., and the colors are also rich. Looking at the key elements again, at the end of A, the dancers do the final dance moves and pose for the curtain call, and at the beginning of B, the host walks to the center of the stage, and the staff prepares props and adjusts musical instruments in the backstage. The structural similarity algorithm SSIM is used to calculate that the visual feature similarity value corresponding to the 1st second at the end of A (actually the 120th second) and the 1st second at the beginning of B is 0.3. In this way, the visual feature similarity value of each corresponding time point is calculated in turn.

[0037] For audio features, extract the audio information at each time point of the end of sub-video A and the beginning of sub-video B, and analyze the audio spectrum, volume, and main sound types. The background music at the end of sub-video A is a lively dance music, mixed with the slight footsteps of students and the applause of the audience; the background music at the beginning of sub-video B is slow but still cheerful, and you can hear the host's speech and the sound of props being prepared backstage. Using audio feature comparison algorithms such as Mel-frequency cepstral coefficients MFCC, the audio feature similarity value corresponding to the 5th second at the end of A (actually the 116th second) and the 8th second at the beginning of B is calculated to be 0.5, and the audio feature similarity values ​​of other time points are calculated in turn.

[0038] Then filter the time point set that meets the error threshold. In terms of visual features, traverse all calculated visual feature similarity values, and select the time points whose error values ​​of adjacent similarity values ​​are within the 0.2 error threshold. For example, it is found that the time points from the 8th second at the end of A (actually the 113th second) to the 8th second at the beginning of B meet the requirements, and they are combined into the first time point set. The same screening is performed on the audio feature similarity values, and the time points from the 6th second at the end of A (actually the 115th second) to the 6th second at the beginning of B meet the conditions, forming the second time point set.

[0039] Looking at these two sets, we find that in the first time point set (visual features), the 8th second from the end of A (actually the 113th second) to the 8th second from the beginning of B are consecutive time points, and in the second time point set (audio features), the 6th second from the end of A (actually the 115th second) to the 6th second from the beginning of B are also consecutive time points, satisfying the condition that both sets have consecutive time points.

[0040] Then, the adjustment duration is generated based on the continuous adjacent time points and the adjustment video range is determined. For visual features, the first adjustment duration is generated based on the continuous adjacent time points from the 8th second at the end of A (actually the 113th second) to the 8th second at the beginning of B, which is 26 seconds. Therefore, the first adjustment video range is the video clip from the 113th second onwards in sub-video A and from the beginning to the 8th second in sub-video B. For audio features, the second adjustment duration is generated based on the continuous adjacent time points from the 6th second at the end of A (actually the 115th second) to the 6th second at the beginning of B, which is 12 seconds. The corresponding second adjustment video range is the video part from the 115th second onwards in sub-video A and from the beginning to the 6th second in sub-video B.

[0041] Comparing the two adjustment video ranges, we can find that the interval from the 115th second of sub-video A to the 6th second of sub-video B is involved in both ranges, so there is an overlapping adjustment range.

[0042] Therefore, the analysis result is that the current adjacent sub-videos A and B need to be segmented and spliced. The segmentation and splicing range is the video segment corresponding to the overlapping interval from the 115th second of sub-video A to the 6th second of sub-video B. Subsequently, the video within this range can be spliced with a fade-in and fade-out transition effect to make the transition between the two sub-videos more natural and smooth, and better display the complete scene process of the campus art performance.

[0043] Specifically for S123, the adjusted durations of adjacent sub-videos are compared respectively. The sub-video with the longer adjusted duration is used as the splicing object, and the other sub-video is used as the segmentation object; the spliced and segmented videos are the adjusted sub-videos.

[0044] For example, taking the above example, the video of the first 6 seconds of sub-video B can be spliced to the end of sub-video A; equivalently, sub-video A has 6 more seconds and sub-video B has 6 fewer seconds.

[0045] Specifically for S140: The first audio can be keywords representing the role layer, such as the role name and character traits, extracted after feature analysis of the role layer. The second audio can be keywords representing the scene layer, such as the weather and venue, extracted after feature analysis of the scene layer. The third audio can be keywords representing the interaction layer, such as the actions of the characters and the interaction between characters, extracted after feature analysis of the interaction layer.

[0046] Refer to Figure 4 , specifically for S150, S150 includes S151 - S156: S151, obtain the first commentary requirement; S152, according to the first commentary requirement, adjust the fusion order of the first audio, the second audio, and the third audio; S153, according to the fusion order, input the first audio, the second audio, and the third audio into the pre-trained word and phrase association model to generate sub-audio; S154, obtain the second commentary requirement; S155, according to the second commentary requirement, match the commentary template from the commentary template library; S156, fill the sub-audio into the commentary template to generate the commentary audio.

[0047] Specifically, for example, a video explaining the theme of a campus sports meeting, the video content includes multiple competition events and interactive scenes between different roles such as athletes and spectators. The keywords extracted to represent the role layer include "athlete", "competitive", "referee", "fair and serious", "audience", "enthusiastic cheering", etc. From the scene layer, keywords such as "track and field stadium", "sunny", "football field", "stands", "lively", and "cloudy weather" are extracted. From the interactive layer, keywords such as "chasing each other", "handing over the baton", "whistle", "cheerful", and "discussion and sharing" are extracted. The first commentary requirement is to vividly show the warm atmosphere of the campus sports meeting through audio, first let the audience feel the characteristics of different scenes, then highlight the interaction between the characters, and finally strengthen the unique performance of each character in the sports meeting, so that the entire audio and video picture are coordinated, making the audience feel as if they are at the lively sports meeting. According to the needs of commentary, the fusion order is determined as follows: first is the second audio (based on scene layer keywords), which uses audio elements corresponding to keywords such as venue and weather to create the environmental atmosphere of different competition venues, allowing the audience to quickly immerse themselves in the scene of the campus sports meeting; followed by the third audio (based on interaction layer keywords), which reflects the vibrant and passionate interaction in the sports meeting by showing the audio corresponding to various interactive behaviors between the characters; finally, the first audio (based on role layer keywords) highlights the characteristics and status of different roles such as athletes, referees, and spectators, further strengthening the theme atmosphere of the sports meeting.

[0048] The word-sentence association model is trained based on a large number of audio and text descriptions related to campus activities. It can associate a combination of audio clips that are adapted and coherent based on the input keywords at each level and the set fusion order, and generate sub-audio that fits the theme of the sports meeting.

[0049] First, input the keywords of the second audio (such as "track and field stadium", "sunny", "on the stands", etc.). The model associates the opening with a clear broadcast announcing the start of the sports meeting, followed by the noise of the crowd on the track and field stadium and the sound of the breeze blowing, accompanied by background music with a brisk rhythm and bright melody, to create an atmosphere of a lively opening of the track and field stadium under the bright sunshine. The duration is about the first 30 seconds, which serves as the starting prelude of the entire audio.

[0050] Then input the keywords of the third audio ("chasing each other", "passing the baton", "cheers", etc.), and the model associates a segment of athletes chasing each other on the track, with rapid alternation of footsteps, heavier breathing, and the crisp touch sound of the baton being handed over. At the same time, the shouts and cheers of the surrounding audiences come one after another, and the background music speeds up and becomes more passionate. It lasts about 25 seconds, naturally continuing the atmosphere of the previous scene and showing the wonderful moments of interaction in the sports meeting.

[0051] Finally, input the keywords of the first audio ("athlete", "competitive", "referee", "fair and serious", etc.). The model generates the excited shouts of the athlete after crossing the finish line, the steady voice of the referee solemnly announcing the results of the competition, and the conversations of the audience enthusiastically discussing the outcome of the competition, accompanied by a background music with a slightly slower rhythm but still full of vitality and a celebratory tone. The duration is about 20 seconds, highlighting the states of different characters in the sports meeting and forming a complete sub-audio that conforms to the fusion order.

[0052] The second commentary requirement is to use passionate and easy-to-understand language to introduce in detail the situations of various competition events, the performances of the characters in them, and the highlights of the interactive moments, and to make corresponding changes in the tone according to different scenarios and character characteristics. For example, when introducing a tense competition, the speech rate should be accelerated and the tone should be raised, and when describing a relaxed moment, the speech rate should be slowed down and the tone should be more relaxed and pleasant.

[0053] There are numerous templates in the commentary template library suitable for campus activity scenarios, covering different themes such as sports meetings, cultural performances, and campus competitions. Each template has corresponding language structures, common expressions, and suggestions for tone adjustment for different emotional atmospheres, etc. For the theme of the campus sports meeting and the second commentary requirement, a suitable commentary template is matched. The beginning of this template is generally a general description of the overall atmosphere of the sports meeting and the current scene, such as "Today, a grand sports meeting has been held on campus, and a tense and enthusiastic atmosphere pervades every competition field"; the middle part will introduce the specific competition events in detail, such as "On the track and field field, the 100-meter sprint competition is in full swing, and the athletes are all like arrows leaving the bowstring, speeding towards the finish line"; the end will summarize the highlights of the competition or make a simple outlook for the subsequent competitions, for example, "This wonderful 100-meter race is just a wonderful moment of the sports meeting. There are still more events waiting for everyone next. Let's look forward to it together!" At the same time, there are corresponding tone change markings for different character interactions and competition progress, guiding how to better convey the corresponding atmosphere through the tone. According to the structure and prompts of the commentary template, fill in the content corresponding to the sub-audio. For example, in the part corresponding to the athletes chasing each other on the track in the previous sub-audio, the commentary audio will be said at a faster speech rate and with an exciting tone: "Look! The athletes are engaged in a fierce competition on the 100-meter track. They chase each other and refuse to yield. Every step is full of strength, and every swing of the arm demonstrates the desire for victory. This is the charm of the sports meeting!" At the same time, according to the duration and rhythm changes of the sub-audio, reasonably adjust the speech rate, pauses, etc. of the commentary statements to ensure that the commentary audio is synchronized with the sub-audio in time and is in line with the emotional atmosphere.

[0054] For other parts of the entire sub-audio, such as the opening introduction of the sports meeting and the subsequent situations after the end of the game, the corresponding commentary content is generated in sequence according to the commentary template, and the text content is converted into commentary audio through speech synthesis technology (select a speech synthesis engine with clear and contagious tone), which can not only clearly convey the information of the sports meeting but also fully create a warm atmosphere.

[0055] Reference Figure 5 As another implementation of the target video interpretation method, the interpretation method may further include S210-S250: S210, obtaining an expression image of a target user; S220, inputting the expression image into a pre-trained micro-expression analysis model to obtain the expression type; S230, determining whether the expression type is negative; S240, if yes, adjusting the commentary template and the commentary volume corresponding to the commentary audio according to the negative level; S250, if not, no response.

[0056] Specifically, the negative emotions include sadness, grief, depression, etc. The negative level refers to an artificially set emotional level, for example, depression is level one, sadness is level two, and sadness is level three.

[0057] For example, when the target user is watching a video, the user's face is photographed at regular intervals (for example, every 30 seconds) by a device (such as a camera installed on a smart TV or a camera on a computer) to obtain the user's expression image. These images clearly present the user's facial features, facial muscle changes and other detailed information, providing basic data for subsequent micro-expression analysis.

[0058] The micro-expression analysis model is trained based on a large amount of facial image data with annotated expression types. It can accurately identify different types of expressions such as happiness, surprise, sadness, sadness, depression, etc. The model extracts features from the input user expression images, analyzes the subtle movements of facial muscles, changes in the morphology of facial features, etc., and then matches them with the learned expression patterns to output the corresponding expression type.

[0059] For example, a user's facial expression image obtained once shows that the user's eyebrows are slightly wrinkled, the corners of the mouth are drooping, and the eyes are slightly dull. After the image is input into the micro-expression analysis model, the model analyzes and judges, and the output expression type is "sad".

[0060] The original commentary template may be relatively plain and objective in language style, and is used to explain the video content under normal emotional state. Now, according to the situation of negative level 2, the commentary template is adjusted. For example, some descriptive sentences are made softer and more soothing, and some empathetic words are added. The expression "The protagonist in this movie encounters a difficult problem, and he needs to find a solution" is changed to "Hey, the protagonist in this movie is facing such a difficult problem at the moment, and I don't know if he can find a solution smoothly." Through this language adjustment, the commentary audio is more in line with the user's current slightly depressed emotional state. At the same time, the commentary volume is also adjusted accordingly. Generally speaking, as the negative level increases, the commentary volume will be appropriately reduced to create a relatively quiet and soothing atmosphere, and avoid excessively noisy sounds that aggravate the user's negative emotions.

[0061] For the "sad" situation with a negative level of level two, the commentary volume is reduced from the original moderate level to about 80% of the original volume, so that the commentary audio is played in a relatively soft way, making the user feel more comfortable.

[0062] If the expression type output by the micro-expression analysis model is positive (such as happiness, surprise, etc.) or neutral (such as calmness, etc.), then continue to play the commentary audio in the normal mode originally set by the system, without making additional adjustments to the commentary template and commentary volume, and maintain the original playback status.

[0063] In addition, after adjusting the commentary template and commentary volume corresponding to the commentary audio according to the negative level, the target user's expression is analyzed periodically to further determine whether the target user's expression type has changed to positive. If so, it is adjusted to the initial commentary style; if not, the commentary voice type is changed and commentary special effects are added.

[0064] For example, after adjusting the commentary audio, the system continues to obtain the target user's facial expression image every 20 seconds and inputs it into the micro-expression analysis model for analysis; after a period of observation, if the model analysis finds that the user's mouth corners are raised and his eyes are bright and energetic, and the expression type is judged to have changed to "happy" (positive expression), the system will adjust the commentary audio back to the initial commentary style.

[0065] If the analysis finds that the user's expression is still in a negative state, if the original commentary voice is standard Mandarin of a young man, it is now replaced with a gentle female voice, which is softer and more intimate, trying to use different voice characteristics to alleviate the user's negative emotions, and add some soothing background music effects to the commentary audio, such as soft piano sounds, to blend with the commentary voice. At the same time, the commentary speed is appropriately slowed down to make the commentary more stable and peaceful. Through these adjustments, users can gradually relax and improve their emotional state.

[0066] Based on the above method embodiments, a second embodiment of the present application discloses a target video commentary system based on video-audio synthesis. Referring to Figure 6 , as an implementation manner of the target video commentary system, the target video commentary system may include: A video acquisition module, configured to acquire a target video; A video processing module, configured to divide the target video into multiple adjusted sub-videos according to the target duration of the target video, and decompose each adjusted sub-video into a character layer, a scene layer, and an interaction layer; An audio processing module, configured to match a first audio according to the character layer, match a second audio according to the scene layer, match a third audio according to the interaction layer, fuse the first audio, the second audio, and the third audio to generate a sub-audio, and associate the adjusted sub-video with the corresponding sub-audio; A commentary module, configured to provide commentary according to the corresponding sub-audio when the target video is played to a certain adjusted sub-video range.

[0067] The video processing module includes: A segmentation unit, configured to equally divide the target video into multiple sub-videos according to the target duration; A video analysis unit, configured to perform correlation analysis on adjacent sub-videos to obtain an analysis result; A video splicing unit, configured to splice adjacent sub-videos according to the analysis result to generate an adjusted sub-video.

[0068] The modules of the target video commentary system based on video-audio synthesis correspond one by one to the target video commentary method based on video-audio synthesis, and will not be elaborated here.

[0069] A third embodiment of the present application provides a terminal. As an implementation manner of the terminal, the terminal may include: a memory and a processor; wherein, The memory is used to store a target video commentary program based on video-audio synthesis; The processor is configured to execute the program stored on the memory to implement the steps of the above-mentioned target video commentary method based on video-audio synthesis.

[0070] Wherein, the memory may be communicatively connected to the processor through a communication bus, and the communication bus may be an address bus, a data bus, a control bus, etc.

[0071] In addition, the memory may include a random access memory (RAM), and may also include a non-volatile memory (NVM), such as at least one disk memory.

[0072] And the processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0073] The above are all preferred embodiments of the present application, which do not limit the protection scope of the present application in sequence. Any feature disclosed in this specification (including the abstract and drawings), unless specifically described, can be replaced by other equivalent or similar-purpose alternative features. That is, unless specifically described, each feature is only an example of a series of equivalent or similar features.

Claims

1. A target video interpretation method based on video and audio synthesis, characterized in that: include: Get the target video; According to the target duration of the target video, the target video is divided into a plurality of adjusted sub-videos; Decomposing each of the adjusted sub-videos into a character layer, a scene layer, and an interaction layer; Matching a first audio according to the role layer, matching a second audio according to the scene layer, and matching a third audio according to the interaction layer; fusing the first audio, the second audio and the third audio to generate a sub-audio; Associating the adjusted sub-video with the corresponding sub-audio; When the target video is played to a certain adjusted sub-video range, a commentary is performed according to the corresponding sub-audio.

2. The target video interpretation method based on video and audio synthesis according to claim 1, characterized in that: The step of dividing the target video into a plurality of adjusted sub-videos according to the target duration of the target video comprises: Divide the target video into multiple sub-videos according to the target duration; Performing correlation analysis on adjacent sub-videos to obtain analysis results; According to the analysis result, adjacent sub-videos are spliced ​​to generate an adjusted sub-video.

3. The target video interpretation method based on video and audio synthesis according to claim 2 is characterized in that: The step of performing correlation analysis on adjacent sub-videos to obtain analysis results comprises: Relative analysis is performed on the visual features and audio features at the same time point in the adjacent sub-videos to obtain a plurality of visual feature similarity values ​​and audio feature similarity values; the adjacent sub-videos are analyzed from the beginning to the end respectively; Screening time points whose error values ​​of adjacent visual feature similarity values ​​are within an error threshold to form a first time point set, and screening time points whose error values ​​of adjacent audio feature similarity values ​​are within the error threshold to form a second time point set; Determine whether there are continuous adjacent time points in both the first time point set and the second time point set; If not, the analysis result is that the adjacent sub-videos do not need to be spliced; If yes, if there are continuous adjacent time points in the first time point set, a first adjustment duration is generated according to the continuous adjacent time points, and a first adjustment video range is determined according to the first adjustment duration; if there are continuous adjacent time points in the second time point set, a second adjustment duration is generated according to the continuous adjacent time points, and a second adjustment video range is determined according to the second adjustment duration; Determine whether the first adjustment video range and the second adjustment video range have overlapping adjustment ranges; If so, the analysis result is that the adjacent sub-videos need to be split and spliced, and the split and splicing range is the overlap adjustment range; If not, the analysis result is that the adjacent sub-videos do not need to be spliced.

4. The method for target video interpretation based on video and audio synthesis according to claim 1, characterized in that: The step of fusing the first audio, the second audio and the third audio to generate a sub-audio comprises: Get the first interpretation demand; According to the first interpretation requirement, adjusting the fusion order of the first audio, the second audio, and the third audio; According to the fusion order, the first audio, the second audio and the third audio are input into a pre-trained word-sentence association model to generate a sub-audio.

5. The target video interpretation method based on video and audio synthesis according to claim 4 is characterized in that: After the sub-audio is generated, the method includes: Obtain the second interpretation requirement; According to the second interpretation requirement, matching an interpretation template from an interpretation template library; Fill the sub-audio into the commentary template to generate the commentary audio.

6. The method for target video interpretation based on video and audio synthesis according to claim 1, characterized in that: The explanation method also includes: Get the target user's expression image; Inputting the expression image into a pre-trained micro-expression analysis model to obtain the expression type; Determine whether the expression type is negative; a negative expression type refers to a negative expression type; If so, the commentary template and the commentary volume corresponding to the commentary audio are adjusted according to the negative level.

7. The method for target video interpretation based on video and audio synthesis according to claim 6, characterized in that: After adjusting the commentary template and the commentary volume corresponding to the commentary audio according to the negative level, the method further includes: Continuously and periodically analyzing the expressions of the target user; Determine whether the expression type changes from negative to positive; a positive expression type is a positive expression type; If so, adjust to the initial commentary style; If not, change the commentary language type and add commentary special effects.

8. A target video explanation system based on video and audio synthesis, characterized in that: include: A video acquisition module is used to acquire a target video; A video processing module, configured to divide the target video into a plurality of adjusted sub-videos according to a target duration of the target video, and decompose each of the adjusted sub-videos into a character layer, a scene layer, and an interaction layer; An audio processing module, used for matching a first audio according to the character layer, matching a second audio according to the scene layer, matching a third audio according to the interaction layer, and fusing the first audio, the second audio and the third audio to generate a sub-audio, and for associating the adjusted sub-video with the corresponding sub-audio; The explanation module is used to explain the target video according to the corresponding sub-audio when the target video is played to a certain adjusted sub-video range.

9. The target video explanation system based on video and audio synthesis according to claim 8, characterized in that: The video processing module comprises: A segmentation unit, configured to equally divide the target video into a plurality of sub-videos according to the target duration; A video analysis unit, used for performing correlation analysis on adjacent sub-videos to obtain analysis results; The video splicing unit is used to splice adjacent sub-videos according to the analysis result to generate an adjusted sub-video.

10. A terminal, characterized in that: include: A memory storing a target video explanation program based on video and audio synthesis; A processor is used to execute the program stored in the memory to implement the steps of the target video interpretation method based on video and audio synthesis as described in any one of claims 1-7.

Images