Audio and video synthesis processing method based on artificial intelligence

Abstract

The invention relates to the technical field of data processing, in particular to an audio and video synthesis processing method based on artificial intelligence, and the method comprises the steps: analyzing the audio change features of video and audio data of audio and video, and obtaining an audio emotion feature tag, an audio background feature tag and an audio content feature tag; obtaining a segmentation sequence timestamp of the same audio editing material in the audio and video according to a similarity calculation result, dividing video image data through the timestamp, and analyzing and evaluating a video content tag pair audio emotion feature tag, audio background feature tag and audio content feature tag; generating a first theme label with consistent video content and audio content; calculating a background sound scene correlation degree according to the first theme label; performing background sound transition adjustment in combination with the segmentation interval time duration and the adjacent audio change features; through the method, the background sound of the preliminarily edited audio and video can be further optimized.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to an audio and video synthesis processing method based on artificial intelligence. Background Art

[0002] Audio and video is a general term for audio and video. The short videos and long videos commonly used by users are all audio and video. Audio and video provide users with multi-faceted information perception through visual and auditory data, and enhance the user's sense of immersion through a large amount of three-dimensional information.

[0003] With the advancement of audio and video editing technology, a large number of self-media industries have further processed the background sound of the original audio and video through existing editing methods, so that the edited audio and video have richer performance than the unedited audio and video in the fields of entertainment, work or education.

[0004] From a technical perspective, since the background sound processing of audio and video requires a lot of professional editing software, relevant workers need to have a lot of technical experience and work experience to present a better audio and video processing effect, which also leads to uneven results of the audio and video edited by relevant workers; in order to lower the technical threshold of relevant workers, CN110958386B proposes a video synthesis method, device, electronic device and computer-readable storage medium, which automatically generates corresponding music information candidates by identifying and selecting the target object features of video synthesis. After selecting the music, the user can merge the music with the video to simplify the video synthesis steps and reduce the operation of relevant workers; CN113329258B automatically extracts features such as song style and automatically selects corresponding scenes to better integrate the picture and the song, so as to improve the audio and video effects of MTV, reduce the editing workload of relevant workers, and improve efficiency; CN116320611B proposes an audio and video synthesis method and system, which obtains the emotional features that the video picture wants to express by extracting the features of the dubbing in the video, and optimizes the corresponding emotional features of the audio to be synthesized, so as to lower the current creation threshold of relevant workers and improve the audio and video effects.

[0005] However, in actual work, the transition of background sound effects between different screens also has a certain impact on the audio and video effects. The current technical problem is that the optimization is only carried out from the addition of background sound materials, but there is a lack of adjustment and optimization of the transition of background sound effects between different audio and video screens, and the optimization content of background sound is relatively simple.

[0006] To this end, the present invention proposes an audio and video synthesis processing method based on artificial intelligence. Summary of the invention

[0007] The purpose of the present invention is to provide an audio and video synthesis processing method based on artificial intelligence, which analyzes the audio change characteristics of video and audio data of audio and video, obtains audio emotion feature labels, audio background feature labels and audio content feature labels according to the change characteristics, obtains the segmentation sequence timestamps of the same audio editing material in the audio and video according to the similarity calculation result, divides the video image data by the timestamp and analyzes the video content label to evaluate the audio emotion feature label, audio background feature label and audio content feature label, and generates a first theme label consistent with the video content and the audio content; calculates the background sound scene relevance according to the first theme label, if it is less than the background sound scene relevance threshold; performs background sound transition adjustment according to the segmentation interval time length and the adjacent audio change characteristics; through this method, the background sound transition adjustment can be achieved for the background sound of the edited audio and video, further reducing the technical threshold of related workers, and further optimizing the entire audio and video.

[0008] To achieve the above object, the present invention provides the following technical solutions:

[0009] Acquire video image data and video audio data of the target audio and video, extract audio change features of the video audio data; acquire audio emotion feature labels, audio background feature labels, and audio content feature labels based on the audio change features;

[0010] The video image data is video image data with added image editing effects, and the image editing effects include text materials, text effect materials, and image effect materials; the video audio data is video audio data with added audio editing effects; the audio editing effects include unadjusted background music materials, unadjusted background sound effect materials, and unadjusted background sound materials;

[0011] Further, the audio change feature, the audio emotion feature label, the audio background feature label and the audio content feature label are obtained through an audio-video feature audio extraction classification model;

[0012] The audio-video feature audio extraction model includes an audio-video audio feature processing unit, an audio-video audio input unit, an audio-video audio feature extraction unit, an audio-video audio feature output unit and an audio-video audio label classification output unit;

[0013] The audio, video and audio feature processing unit separates the video and audio data from the audio editing effect to generate a plurality of audio, video and audio pre-input data; the audio, video and audio input unit divides the plurality of audio, video and audio pre-input data into equal amounts to obtain an audio, video and audio input data set;

[0014] The equal amount division is based on the maximum amplitude and the average frequency of the plurality of audio, video and audio pre-input data, and the specific division length is:

[0015]

[0016] Wherein, Length is the segmentation length, max() is the maximum value, is the amplitude of the mth data, is the frequency of the mth data;

[0017] The audio, video and audio feature extraction unit extracts the change features of the audio, video and audio input data set to generate audio, video and audio change feature data, and the audio, video and audio feature output unit classifies the audio, video and audio change feature data and outputs the audio change features;

[0018] Further, each data in the equally divided audio, video and audio input data set is divided into the same time step, and the divided audio, video and audio input data set is input into the audio, video and audio feature extraction unit;

[0019] The audio, video and audio feature extraction unit adopts convolutional neural network, Transformer encoder and dilated convolution acquisition; the specific expression is as follows:

[0020]

[0021]

[0022]

[0023] output=fea(output 1 ,output 2 );

[0024] Among them, output is output, input is input, output 1 is the output of the convolutional neural network and the Transformer encoder, Encode Transform is the Transformer encoder, Liner is the fused linear transform, CNN is the convolutional neural network, input 1 , input 2 , input n-1 , input n is the input data after division, θ n is the number of data after division, TCN 1 and TCN 2is the dilated convolution, Dropout is the Dropout layer, relu is the activation layer, Nor is the normalization layer, and fea() is the feature fusion;

[0025] The classification is to output the audio change feature through Softmax classification;

[0026] The audio-video audio label classification output unit performs further feature classification calculation on the audio-video audio change feature data, and outputs the audio emotion feature label, the audio background feature label and the audio content feature label;

[0027] The audio-video audio label classification output unit adopts feature clustering to obtain the audio emotion feature label, the audio background feature label and the audio content feature label, and the feature clustering adopts a k-means clustering method;

[0028] The audio emotion feature label, the audio background feature label and the audio content feature label are calculated based on the similarity between the audio-video audio change feature data and the audio-video audio comparison feature data in the standard library, and the highest similarity is obtained as the corresponding label data;

[0029] Segment the video and audio data according to the audio-video audio similarity of the audio emotion feature tag, the audio background feature tag, and the audio content feature tag, and generate an audio segmentation sequence and a segmentation sequence timestamp;

[0030] Furthermore, the audio emotion feature labels, the audio background feature labels and the audio content feature labels are sorted in the order of timestamps, and a fixed number N is selected. i The audio-video audio similarity is calculated based on the audio emotion feature label, the audio background feature label and the audio content feature label:

[0031]

[0032] Among them, δ N is the audio / video audio similarity, β det is the same continuity weight of the audio content feature label, β emt is the same continuity weight of the audio emotion feature label, β back is the same continuity weight of the audio background feature label, α det , α emt and α back is the audio similarity weight, and the audio similarity weight relationship is α det +α emt +α back =1;

[0033] Segmenting the video image data according to the segmentation sequence timestamp to generate a video segmentation sequence;

[0034] Extracting the video content tags of each of the video segmentation sequences; combining the audio emotion feature tags, the audio background feature tags and the audio content feature tags corresponding to each of the audio segmentation sequences to generate corresponding first topic tags; sorting the first topic tags in the order of the timestamps of the segmentation sequences;

[0035] The extraction of the video content label is to obtain the type data in the picture with the yolov5 model, and to perform statistics on the type data, and obtain the corresponding video content label according to the classification of the statistical results;

[0036] Further, the audio segmentation sequence is segmented according to the audio-video audio similarity to generate new audio emotion feature labels, audio background feature labels and audio content feature labels;

[0037] According to the time sequence of the audio segmentation sequence and the video segmentation sequence, the video content label T corresponding to the time sequence is selected. video , the new audio emotion feature label T det , the new audio background feature label T emt and the new audio content feature label T back Calculate the first topic tag to generate:

[0038]

[0039]

[0040]

[0041] Among them, δ F for the first topic label; a tag type for the first topic tag; The number of occurrences of the corresponding tag type; CL() is the tag set calculated by the audio and video topic tag clustering model, SimCount() is the number of occurrences of each tag in all tags, N n is the total number of the first topic tags;

[0042] The calculation of the audio and video topic tag clustering model is as follows:

[0043]

[0044]

[0045] Among them, CL is the label set, CLp is the pth tag, Tip() is the distance tag calculation, DIP p is the p-th cluster distance, is the coordinate of the pth cluster center, is the cluster coordinates whose distance needs to be calculated, || || is the distance calculation, DIP pc is the radius of the pth distance center, cn is the total number of all cluster labels, The audio data label coordinates of the cluster center; The video content label of the cluster center; The label coordinates of the input audio data; The video content tag is input; is the audio emotion feature label of the cluster center, is the audio background feature label of the cluster center, The audio content feature label of the cluster center;

[0046] Traversing the first topic tag, calculating the background sound scene relevance of the first topic tag of the adjacent audio segmentation sequence; if the background sound scene relevance is greater than or equal to the background sound scene relevance threshold; then no adjustment is made to the video and audio data and the traversal continues; if the background sound scene relevance is less than the background sound scene relevance threshold; then obtaining the segmentation interval time length of the adjacent audio segmentation sequence, and performing background sound transition adjustment according to the segmentation interval time length and the adjacent audio change characteristics;

[0047] Furthermore, all the first topic labels are sorted and traversed in the order of timestamps to obtain the first topic labels of the adjacent audio segmentation sequence to be calculated. and

[0048]

[0049]

[0050] Among them, δ rele is the background sound scene relevance, for The total number of tags, for The total number of tags, For the i-th The weight of the label, Tsim() is for identification and Same first topic tag calculation, For the i-th Tags, For the jth Tags, For the i-th The number of times the tag type appears, is the total number of tag types;

[0051] Further, the transition adjustment is to determine the length of the segmentation interval of the t-th background sound; if the segmentation interval is greater than or equal to the background sound adjustment time threshold, the amplitude data of the audio segmentation sequence to be adjusted is obtained, and the mean of the adjusted amplitude data is generated in combination with the segmentation interval, and the amplitude data is adjusted down to 0 according to the timestamp of the segmentation interval until it reaches 0;

[0052] If the segmentation interval time is less than the background sound adjustment time threshold, the type of the audio change feature of the t+1th background sound is obtained; the background sound material of the same type of the audio change feature is called from the material library; the background sound material is added to the background sound transition interval of the tth background sound and the t+1th background sound, and when the tth background sound reaches the background sound transition interval, the amplitude data is lowered to 0; the background sound transition interval is calculated as follows:

[0053] T start =(1-α t )·T t ;

[0054] T end =α t+1 ·T t+1 ;

[0055] Among them, T start is the starting timestamp of the background sound transition interval, T end is the end timestamp of the background sound transition interval, T t is the total number of timestamps of the tth background sound, T t+1 is the total number of timestamps of the t+1th background sound; α t is the calculation weight of the tth background sound, α t+1 is the calculation weight of the t+1th background sound.

[0056] Compared with the prior art, the present invention has the following beneficial effects:

[0057] 1. In order to obtain the connection relationship between various background sounds in audio and video, the present invention extracts the features of the original background sound data and the processed background sound data in the initially processed audio and video data through an audio and video feature audio extraction model; and generates corresponding labels for the relationship features of each audio data. Through this label, the relationship features of the background sound in the audio and video data can be automatically identified, so as to provide data support for the further transition adjustment of the subsequent background sound data, improve the accuracy of the adjustment, and realize further optimization of the initially edited audio and video background sound.

[0058] 2. In order to achieve a more natural transition during the background sound adjustment process, after extracting the similarity between the video data and the audio data in the audio and video, the picture data and audio data with high similarity are segmented to determine the theme of the current audio and video clip; since there are certain differences between the audio data labels and the video data labels, in order to determine the main content labels for the current clip theme to facilitate the subsequent background sound transition, all labels are evaluated in combination with the clustering algorithm to obtain the first theme label in order to reduce interference in the subsequent background sound transition adjustment process, ensure the accuracy of the adjustment process, and achieve further optimization process.

[0059] 3. After determining the first topic label of each segment of the audio and video, in order to determine whether the background sound of the audio and video needs to be adjusted, all the first topic labels are traversed, and the correlation between the two is calculated by comparing the correlation calculation result with the background sound correlation threshold to obtain the correlation with the background sound of the adjacent segment, so as to provide further data support for the subsequent transition adjustment and realize further background sound optimization process. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 It is a schematic diagram of the method flow of the present invention;

[0061] Figure 2 It is a schematic diagram of the text material of the present invention;

[0062] Figure 3 It is a schematic diagram of the flow of the audio and video feature audio extraction classification model of the present invention;

[0063] Figure 4 It is a schematic diagram of the yolov5 picture feature recognition of the present invention. DETAILED DESCRIPTION

[0064] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0065] As the number of people engaged in the self-media industry continues to increase, the simplification of the use of related software has also become the main direction at present. As the main presentation carrier of the self-media industry, audio and video require higher work experience of relevant staff and experience in using related software in order to produce better audio and video related works; and background sound is one of the main steps in audio and video processing. In order to reduce the processing process of background sound, a variety of methods have been used to automatically add it. However, at present, only the steps of adding background sound are optimized, and the transition effect of background sound in different pictures is not optimized. Therefore, the present invention provides an audio and video synthesis processing method based on artificial intelligence, and the technical solution is as follows:

[0066] Reference Figure 1 As shown, the video image data and video audio data of the target audio and video are obtained, and the audio change characteristics of the video audio data are extracted; the audio emotion feature label, the audio background feature label and the audio content feature label are obtained according to the audio change characteristics;

[0067] Segment the video and audio data according to the audio-video audio similarity of the audio emotion feature tag, the audio background feature tag, and the audio content feature tag, and generate an audio segmentation sequence and a segmentation sequence timestamp;

[0068] Segmenting the video image data according to the segmentation sequence timestamp to generate a video segmentation sequence;

[0069] Extracting the video content tags of each of the video segmentation sequences; combining the audio emotion feature tags, the audio background feature tags and the audio content feature tags corresponding to each of the audio segmentation sequences to generate corresponding first topic tags; sorting the first topic tags in the order of the timestamps of the segmentation sequences;

[0070] Traverse the first topic tag and calculate the background sound scene relevance of the first topic tag of the adjacent audio segmentation sequence; if the background sound scene relevance is greater than or equal to the background sound scene relevance threshold; do not adjust the video audio data and continue traversing; if the background sound scene relevance is less than the background sound scene relevance threshold; obtain the segmentation interval time length of the adjacent audio segmentation sequence, and perform background sound transition adjustment according to the segmentation interval time length and the adjacent audio change characteristics.

[0071] In order to optimize the transition of background sound between different pictures; by obtaining the initially edited audio and video, the corresponding video image data and video audio data are obtained. In order to achieve the natural transition of the background sound, the data is split and analyzed accordingly to achieve the natural transition of the subsequent processing of the background sound;

[0072] For video and audio data, in order to obtain the characteristics of audio data, the change characteristics of audio data are obtained, and classified according to audio emotion feature labels, audio background feature labels and audio content feature labels, and corresponding labels are generated. The similarity is calculated through these labels, and the time stamp intervals of audio and video with similar audio content can be obtained. The audio data can be divided according to these intervals to obtain accurate audio clips with the same theme. These clips can achieve more accurate background sound transition adjustment and optimization;

[0073] Considering that audio and video are a combination of audio and video content, in order to ensure that the background sound can ensure a more accurate timestamp transition area when transitioning, in addition to dividing the video image data according to the timestamps of the audio division and obtaining the corresponding video content tags, the audio tags and video tags in each timestamp area are evaluated to generate the first topic tag of each area, and the main topic content related to both video and audio is obtained. Through the first topic tag, a more accurate audio and video clip content expression can be obtained to provide data support for further transition optimization of the background sound;

[0074] By calculating the background sound scene relevance of the first topic tags of adjacent clips, it is possible to determine whether the background sounds of adjacent clips are similar. By comparing these data with the background sound scene relevance threshold, it is possible to determine whether adjacent areas need transition adjustments. In combination with the various data analyzed previously, corresponding adjustments can also be made to adjacent pictures that need transition adjustments. While making the judgment, it can also provide important data support for the adjustment, thereby achieving a more comprehensive background sound transition adjustment optimization process.

[0075] Embodiment 1

[0076] For the purpose of specific description, the present invention is described by taking the audio and video as humorous and funny effects as an example in combination with the following embodiments:

[0077] Obtain the video image data and video audio data of the target audio and video, extract the audio change features of the video audio data; obtain the audio emotion feature label, audio background feature label and audio content feature label based on the audio change features;

[0078] The video image data is video image data with added image editing effects, and the image editing effects include text materials, text effect materials and image effect materials;

[0079] The video and audio data is video and audio data with audio editing effects added;

[0080] Audio editing effects include unadjusted background music material, unadjusted background sound effect material, and unadjusted background sound material;

[0081] Text materials, such as post-production text added to audio and video images, refer to Figure 2 As shown; the text effect material is used to package and modify the post-production text, such as adding special effects of the sun to the happy, etc. The image effect material is mainly used to enhance the current screen content. For example, if the current environment is hot, a fire effect is added below the entire audio and video screen;

[0082] The unadjusted background music materials are mainly various music added later to reflect the current picture environment, such as the passionate music added later in the victory scene;

[0083] The unadjusted background sound effects are used to support the voices added later in the current picture content; for example, the atmosphere laughter added later in the funny scenes in the picture;

[0084] The unadjusted background sound material is the scene sound added later to reflect the environmental atmosphere, such as the bird calls added to reflect the natural environment;

[0085] Considering that audio and video are composed of video data and audio data, the transition of background sound needs to be carried out in combination with the video data content and the audio data content. To this end, the video editing data, video original data, audio editing data and audio original data are extracted separately to facilitate the feature analysis of the subsequent content and provide accurate data support for the subsequent background sound transition calculation;

[0086] Further, the audio change features, audio emotion feature labels, audio background feature labels and audio content feature labels are obtained through an audio-video feature audio extraction classification model;

[0087] Reference Figure 3 As shown, the audio and video feature audio extraction model includes an audio and video audio feature processing unit, an audio and video audio input unit, an audio and video audio feature extraction unit, an audio and video audio feature output unit and an audio and video audio label classification output unit;

[0088] The audio, video and audio feature processing unit separates the video and audio data from the audio editing effect to generate a plurality of audio, video and audio pre-input data; the audio, video and audio input unit divides the plurality of audio, video and audio pre-input data into equal amounts to obtain an audio, video and audio input data set;

[0089] The equal amount segmentation is segmentation based on the maximum amplitude and the average frequency of each of the multiple audio, video and audio pre-input data. The specific segmentation length is:

[0090]

[0091] Among them, Length is the segmentation length, max() is the maximum value, is the amplitude of the mth data, is the frequency of the mth data;

[0092] The audio, video and audio feature extraction unit extracts the change features of the audio, video and audio input data set to generate audio, video and audio change feature data, and the audio, video and audio feature output unit classifies the audio, video and audio change feature data and outputs the audio change features;

[0093] The audio change feature mainly generates corresponding labels according to its audio change, for example, the rising feature is marked as 1, the falling feature is marked as 2, the fluctuation feature is marked as 3, etc.; the specific marking content may adopt different label contents according to the actual implementation method;

[0094] Further, each data in the equally divided audio, video and audio input data set is divided into the same time step, and the divided audio, video and audio input data set is input into the audio, video and audio feature extraction unit; the same time step can be set according to the specific implementation situation, including but not limited to 5 seconds, 10 seconds, and 20 seconds; the audio, video and audio feature extraction unit adopts convolutional neural network, Transformer encoder and dilated convolution to obtain; the specific expression is as follows:

[0095]

[0096]

[0097]

[0098] output=fea(output 1 ,output 2 );

[0099] Among them, output is output, input is input, output 1 is the output of the convolutional neural network and Transformer encoder, Encode Transform is the Transformer encoder, Liner is the fused linear transform, CNN is the convolutional neural network, input 1 , input 2 , input n-1 , input nis the input data after division, θ n is the number of data after division, TCN 1 and TCN 2 is dilated convolution, Dropout is the Dropout layer, relu is the activation layer, Nor is the normalization layer, and fea() is feature fusion;

[0100] The classification is to output the audio change features through Softmax classification; the audio-video audio label classification output unit performs further feature classification calculation on the audio-video audio change feature data, and outputs the audio emotion feature label, the audio background feature label and the audio content feature label; this embodiment uses 10 groups of simulation data for label accuracy recognition, and the 10 groups of data contain different scenes under humorous effects. The label recognition accuracy results are shown in Table 1:

[0101] Table 110 groups of humorous and funny simulation data label accuracy recognition

[0102] Simulating data Tag recognition accuracy Data 1 92.61％ Data 2 93.76％ Data 3 91.48％ Data 4 92.57％ Data 5 93.22％ Data 6 94.19％ Data 7 90.26％ Data 8 92.52％ Data 9 92.74％ Data 10 93.61％

[0103] The audio and video audio label classification output unit uses feature clustering to obtain audio emotion feature labels, audio background feature labels and audio content feature labels. The feature clustering adopts the k-means clustering method. The number of k is the same as the total number of audio emotion feature labels, audio background feature labels and audio content feature labels. The k-means clustering set is obtained after training and calculation based on a large amount of audio and video data.

[0104] The audio emotion feature label is the emotional theme that the audio of the audio or video wants to express, such as sadness, happiness, etc.; the audio background feature label is the environmental theme that the audio of the audio or video wants to express, such as noisy environment, quiet environment, etc.; the audio content feature label is the content theme that the audio of the audio or video wants to express, such as positivity, etc.; taking a funny video as an example, the audio emotion feature label is mainly happy, the audio background feature label is laughter environment, and the audio content feature label is humor;

[0105] In the generation of specific label data, different data can be used; for example, there are 3 types of audio emotion feature labels, 2 types of audio background feature labels, and 4 types of audio content feature labels;

[0106] In the first implementation, labels may be made in the form of continuous integers, with audio emotion feature labels being 1, 2, and 3, audio background feature labels being 4 and 5, and audio content feature labels being 6, 7, 8, and 9;

[0107] In a second embodiment, the labels may be in the form of consecutive decimals, where the audio emotion feature labels are 1.1, 1.2, 1.3, the audio background feature labels are 1.4, 1.5, and the audio content feature labels are 1.6, 1.7, 1.8, 1.9;

[0108] In a third implementation, labels may be made in the form of non-continuous decimals, with audio emotion feature labels being 1.1, 1.2, 1.3, audio background feature labels being 2.1, 2.2, and audio content feature labels being 3.1, 3.2, 3.3, 3.4;

[0109] By extracting the features of audio data, we can obtain the corresponding change features of audio data. These features can not only ensure the analysis of the labels of the relevant topics of audio data, but also provide sufficient data support for the subsequent background sound adjustment work. By generating labels for these data through model recognition, we can also turn the abstract analysis data into more specific data, which can provide strong data support for the subsequent background sound optimization and adjustment, and improve the accuracy of audio data division, so as to facilitate the subsequent division of similar theme segments of audio and video and determine the main content interval.

[0110] The audio emotion feature label, audio background feature label and audio content feature label are calculated based on the similarity between the audio and video audio change feature data and the audio and video audio comparison feature data in the standard library, and the one with the highest similarity is taken as the corresponding label data;

[0111] Segment the video and audio data according to the audio and video audio similarity of the audio emotion feature label, the audio background feature label and the audio content feature label, and generate an audio segmentation sequence and a segmentation sequence timestamp;

[0112] Furthermore, the audio emotion feature labels, audio background feature labels, and audio content feature labels are sorted in timestamp order, and a fixed number N is selected. i The audio emotion feature labels, audio background feature labels and audio content feature labels are used to calculate the audio-video audio similarity:

[0113]

[0114] Among them, δ N is the audio-video audio similarity, β det is the same continuity weight of the audio content feature label, β emt is the same continuity weight of the audio emotion feature label, β back is the same continuity weight of the audio background feature label, α det , α emt and α back is the audio similarity weight, and the audio similarity weight relationship is αdet +α emt +α back =1;α back The default value is 0.5, α det The default value is 0.3, alpha emt is 0.2; each weight is obtained based on expert experience and can be modified according to actual conditions during the specific implementation process; N i The number is the number of label data contained in each segment of audio data after being equally divided and recognized by the model; in the calculation process, the audio data after the segmentation adjacent to both ends will be selected for similarity calculation;

[0115] In order to ensure the accuracy of audio and video features, considering that the correlation characteristics of audio and video will be stronger under continuous audio and video features, the same weight value for continuity will be higher. At the same time, considering that different audio and video types have different requirements for content, emotion and background similarity, the audio similarity weight is added in the calculation process to facilitate the calculation of audio and video similarity features of different contents and improve accuracy.

[0116] The same continuity weights of the audio content feature label, the audio emotion feature label, and the audio background feature label are specifically obtained as follows. Taking the audio content feature label as an example, for example, N i The number is 5, and the audio content feature labels are {1,1,2,3,4}; then the continuity weight of label 1 is 40%, and the continuity weights of 2, 3, and 4 are 20% each, that is, when calculating the proportion of the number of consecutive identical numbers in the total number of labels in the label set; if the audio content feature labels are {1,2,3,1,4}, then the continuity weights of 1,2,3,4 are all 20%;

[0117] In order to improve the accuracy of the transition area of ​​the background sound, the specific range of the content of the same audio theme of the audio and video can be determined. For this purpose, the corresponding label data obtained by the audio and video feature audio extraction model is used to calculate the similarity of each label data, and the data with the same label type is obtained. Based on the similarity calculation result, the timestamp area of ​​the similar label theme of the audio data can be determined, which is convenient for the subsequent determination of the position of the background sound transition adjustment, and effectively improves the accuracy and optimization of the background sound transition adjustment;

[0118] Segment the video image data by segmenting the sequence timestamps to generate a video segmentation sequence;

[0119] Extracting video content labels of each video segmentation sequence; combining audio emotion feature labels, audio background feature labels and audio content feature labels corresponding to each audio segmentation sequence to generate corresponding first topic labels; sorting the first topic labels in the order of segmentation sequence timestamps;

[0120] The extraction of video content labels is to obtain the type data in the picture with the yolov5 model, and to count the type data, and obtain the corresponding video content labels according to the classification of the statistical results; it is mainly judged by the identified targets in yolov5, and the identified targets include people, buildings and scenery in the video. Taking people as an example, a small number of people are marked as 1, a large number of people are marked as 2, and only animals are marked as 3, etc. Figure 4 As shown, yolov5 recognizes that there are a small number of people in the current frame, marked as 1;

[0121] Further, the audio segmentation sequence is segmented according to the audio-video audio similarity to generate new audio emotion feature labels, audio background feature labels and audio content feature labels;

[0122] According to the time sequence of the audio segmentation sequence and the video segmentation sequence, select the video content label T corresponding to the time sequence video , new audio emotion feature label T det , New audio background feature tag T emt and new audio content feature label T back Calculate the generation of the first topic label; take a funny video as an example, after yolov5 recognition, the video content contains 3 people, the screen includes the text effect "Haha", and the video content label is recognized as humorous and funny; the first topic label is generated as follows:

[0123]

[0124]

[0125]

[0126] Among them, δ F For the first topic label; The tag type for the first topic tag; The number of occurrences of the corresponding tag type; CL() is the tag set calculated by the audio and video topic tag clustering model, SimCount() is the number of occurrences of each tag in all tags, N n is the total number of the first topic tag;

[0127] By obtaining topic tags from audio and video, the audio and video content between different types of audio and video clips can be reflected. At the same time, the type of the first topic tag of each audio and video clip can be accurately obtained through clustered content. At the same time, through a large amount of clustered content, the unity of the theme between different types of content can be guaranteed, effectively improving the suitability of background sound adjustment;

[0128] The calculation of the audio and video topic tag clustering model is as follows:

[0129]

[0130]

[0131] Among them, CL is the label set, CL p is the pth tag, Tip() is the distance tag calculation, DIP p is the p-th cluster distance, is the coordinate of the pth cluster center, is the cluster coordinates whose distance needs to be calculated, || || is the distance calculation, DIP pc is the radius of the pth distance center, cn is the total number of all cluster labels, The audio data label coordinates of the cluster center; is the video content label of the cluster center; The label coordinates of the input audio data; is the input video content label; is the audio emotion feature label of the cluster center, is the audio background feature label of the cluster center, is the audio content feature label of the cluster center;

[0132] After determining the main theme of the audio data, in order to further match it with the theme of the video content and enable the background sound adjustment to achieve a more natural transition, the video content tag is combined with the audio tag for evaluation, and the first theme tag is obtained in combination with the evaluation result of the clustering calculation. The main theme feature of the current timestamp area can be determined through the tag, so as to provide the main content data for the subsequent background sound adjustment, so that the transition adjustment of the background sound can achieve a more natural optimization process;

[0133] Traverse the first topic label, calculate the background sound scene relevance of the first topic label of the adjacent audio segmentation sequence; if the background sound scene relevance is greater than or equal to the background sound scene relevance threshold; do not adjust the video and audio data and continue traversing; if the background sound scene relevance is less than the background sound scene relevance threshold; the segmentation interval time length of the adjacent audio segmentation sequence, according to the segmentation interval time length and the adjacent audio change characteristics, make a transition adjustment of the background sound; the background sound scene relevance threshold is set to 80% by default, and the specific value can be modified according to the actual work scenario combined with expert experience;

[0134] Furthermore, all first topic labels are sorted and traversed in the order of timestamps to obtain the first topic labels of the adjacent audio segmentation sequence to be calculated. and

[0135]

[0136]

[0137] Among them, δ rele is the background sound scene relevance, for The total number of tags, for The total number of tags, For the i-th The weight of the label, Tsim() is for identification and Same first topic tag calculation, For the i-th Tags, For the jth Tags, For the i-th The number of times the tag type appears, is the total number of tag types;

[0138] By calculating the correlation between the first theme tags adjacent to each scene, the differences in the themes to be expressed by the audio and video in the adjacent intervals can be obtained, and based on the differences, it can be determined whether the background sound needs to be transitioned and how to transition. By labeling and digitizing these main data, a more accurate judgment process for background sound transition adjustment can be achieved, and a more reasonable background sound transition adjustment optimization can be achieved without violating the audio and video theme.

[0139] Furthermore, the transition adjustment is to determine the length of the t-th background sound segmentation interval; if the segmentation interval is greater than or equal to the background sound adjustment duration threshold, the amplitude data of the audio segmentation sequence to be adjusted is obtained, and the mean of the adjusted amplitude data is generated in combination with the segmentation interval, and the amplitude data is adjusted down to 0 according to the timestamp of the segmentation interval until the amplitude data is 0;

[0140] If the segmentation interval time is less than the background sound adjustment time threshold, the type of the audio change feature of the t+1th background sound is obtained; the background sound material with the same type of audio change feature is called from the material library; the background sound material is added to the background sound transition interval of the tth background sound and the t+1th background sound, and when the tth background sound reaches the background sound transition interval, the amplitude data is lowered to 0; the background sound adjustment time threshold is obtained based on the time average data of a large number of used material data;

[0141] In the process of background sound adjustment, in order to fit the theme content and ensure that the adjustment of the background sound is more reasonable, the duration of the clip is used as the basis for judgment. It is calculated which processing effect is better within the duration. At the same time, the duration can also be used to determine when it is more appropriate to add it. Combined with the audio change characteristics analyzed by the model, similar features in the material library can be found through this feature, and a more natural transition can be made, thereby achieving a natural and accurate optimization process for background sound transition adjustment.

[0142] The background sound transition interval is calculated as follows:

[0143] T start =(1-α t )·T t ;

[0144] T end =α t+1 ·T t+1 ;

[0145] Among them, T start is the starting timestamp of the background sound transition interval, T end is the end timestamp of the background sound transition interval, T t is the total number of timestamps of the tth background sound, T t+1 is the total number of timestamps of the t+1th background sound; α t is the calculation weight of the tth background sound, α t+1 is the calculation weight of the t+1th background sound; α t The default value is 0.8, alpha t+1 The default value is 0.1. The specific situation can be set according to expert experience. In order to ensure the naturalness of the background sound adjustment, the background sound of the audio and video is adjusted by setting the adjustment timestamp to ensure the naturalness of the background sound adjustment and improve the overall quality and audio and video viewing comfort after the audio and video processing is completed.

[0146] Embodiment 2

[0147] This embodiment is described by taking the audio and video as humorous and tragic effects.

[0148] Obtain the video image data and video audio data of the target audio and video, extract the audio change features of the video audio data; obtain the audio emotion feature label, audio background feature label and audio content feature label based on the audio change features;

[0149] The video image data is video image data with image editing effects including humorous and tragic effects, and the image editing effects include text materials, text effect materials and image effect materials; the video audio data is video audio data with audio editing effects; the audio editing effects include unadjusted background music materials, unadjusted background sound effect materials and unadjusted background sound materials; the text materials, text effect materials, image effect materials, unadjusted background music materials, unadjusted background sound effect materials and unadjusted background sound materials all contain humorous and tragic contents, and taking the tragic text materials as an example, for example, a rainstorm special effect is added to the crying text and a sun special effect is added to the happy text;

[0150] Further, the audio change features, audio emotion feature labels, audio background feature labels and audio content feature labels are obtained through an audio-video feature audio extraction classification model;

[0151] The audio and video feature audio extraction model includes an audio and video audio feature processing unit, an audio and video audio input unit, an audio and video audio feature extraction unit, an audio and video audio feature output unit and an audio and video audio label classification output unit;

[0152] The audio, video and audio feature processing unit separates the video and audio data from the audio editing effect to generate a plurality of audio, video and audio pre-input data; the audio, video and audio input unit divides the plurality of audio, video and audio pre-input data into equal parts to obtain an audio, video and audio input data set; the equal division is based on the maximum amplitude and the average frequency of each of the plurality of audio, video and audio pre-input data, and the specific division length is:

[0153]

[0154] Among them, Length is the segmentation length, max() is the maximum value, is the amplitude of the mth data, is the frequency of the mth data;

[0155] The audio-video feature extraction unit extracts the change features of the audio-video audio input data set and generates audio-video audio change feature data. The audio-video audio feature output unit classifies the audio-video audio change feature data and outputs the audio change features. In order to distinguish the audio change features, humorous and funny content and tragic content will be distinguished, for example, humorous and funny content is 1X, tragic content is 2X, etc., which can be modified according to the implementation situation.

[0156] Furthermore, each data in the equally divided audio, video and audio input data set is divided into the same time step, and the divided audio, video and audio input data set is input into the audio, video and audio feature extraction unit; the audio, video and audio feature extraction unit adopts convolutional neural network, Transformer encoder and dilated convolution to obtain; the specific expression is as follows:

[0157]

[0158]

[0159]

[0160] output=fea(output 1 ,output 2 );

[0161] Among them, output is output, input is input, output 1 is the output of the convolutional neural network and Transformer encoder, Encode Transform is the Transformer encoder, Liner is the fused linear transform, CNN is the convolutional neural network, input 1 , input 2 , input n-1 , input n is the input data after division, θ n is the number of data after division, TCN 1 and TCN 2 is dilated convolution, Dropout is the Dropout layer, relu is the activation layer, Nor is the normalization layer, and fea() is feature fusion;

[0162] Classification is to output audio change features through Softmax classification; this embodiment also uses 10 groups of simulation data to identify the accuracy of labels. The 10 groups of data contain humorous and tragic scenes, of which 10 humorous scenes are used; 10 tragic scenes are selected, and 1 humorous scene corresponds to 1 tragic scene. The timestamp position of each tragic scene in the corresponding humorous scene is different. The label recognition accuracy results are shown in Table 2:

[0163] Table 210 groups of humorous and tragic simulation data label accuracy recognition

[0164] Simulating data Tag recognition accuracy Data 1 89.45％ Data 2 92.43％ Data 3 90.62％ Data 4 91.44％ Data 5 92.85％ Data 6 90.89％ Data 7 92.67％ Data 8 91.44％ Data 9 93.15％ Data 10 90.88％

[0165] The audio / video audio label classification output unit further performs feature classification calculation on the audio / video audio change feature data, and outputs audio emotion feature labels, audio background feature labels, and audio content feature labels;

[0166] The audio and video audio label classification output unit uses feature clustering to obtain audio emotion feature labels, audio background feature labels and audio content feature labels. The feature clustering uses the k-means clustering method.

[0167] The audio emotion feature label, audio background feature label and audio content feature label are calculated based on the similarity between the audio and video audio change feature data and the audio and video audio comparison feature data in the standard library, and the one with the highest similarity is taken as the corresponding label data;

[0168] In the case of multiple scene tags such as humor and tragedy, two implementation methods are used to distinguish the content;

[0169] In the first implementation, prefixes are used for distinction, for example, humor is 11, 12, 13, etc., and tragedy is 21, 22, 23, etc.;

[0170] In the second implementation, a numerical range is used for distinction, for example, 0.01-4.99 is a humorous content label, and 5.01-9.99 is a tragic content label;

[0171] Segment the video and audio data according to the audio and video audio similarity of the audio emotion feature label, the audio background feature label and the audio content feature label, and generate an audio segmentation sequence and a segmentation sequence timestamp;

[0172] Furthermore, the audio emotion feature labels, audio background feature labels, and audio content feature labels are sorted in timestamp order, and a fixed number N is selected. i The audio emotion feature labels, audio background feature labels and audio content feature labels are used to calculate the audio-video audio similarity:

[0173]

[0174] Among them, δ N is the audio-video audio similarity, β det is the same continuity weight of the audio content feature label, β emt is the same continuity weight of the audio emotion feature label, β back is the same continuity weight of the audio background feature label, α det , α emt and α back is the audio similarity weight, α det +α emt +α back=1;

[0175] Segment the video image data by segmenting the sequence timestamps to generate a video segmentation sequence;

[0176] Extracting video content labels of each video segmentation sequence; combining audio emotion feature labels, audio background feature labels and audio content feature labels corresponding to each audio segmentation sequence to generate corresponding first topic labels; sorting the first topic labels in the order of segmentation sequence timestamps;

[0177] The extraction of video content labels is the same as the yolov5 model to obtain the type data in the picture, and then to count the type data, and to obtain the corresponding video content labels based on the classification of the statistical results; taking a certain picture as an example, the video content contains 2 people, one of whom has a happy expression and a label of "laughing" on his head; the other has a sad expression and a label of "crying" on his head; the identified content labels are humor and tragedy;

[0178] Further, the audio segmentation sequence is segmented according to the audio-video audio similarity to generate new audio emotion feature labels, audio background feature labels and audio content feature labels;

[0179] According to the time sequence of the audio segmentation sequence and the video segmentation sequence, select the video content label T corresponding to the time sequence video , new audio emotion feature label T det , New audio background feature tag T emt and new audio content feature label T back Calculate the first topic label generation:

[0180]

[0181]

[0182]

[0183] Among them, δ F For the first topic label; The tag type for the first topic tag; The number of occurrences of the corresponding tag type; CL() is the tag set calculated by the audio and video topic tag clustering model, SimCount() is the number of occurrences of each tag in all tags, N n is the total number of the first topic tag;

[0184] The calculation of the audio and video topic tag clustering model is as follows:

[0185]

[0186]

[0187] Among them, CL is the label set, CL p is the pth tag, Tip() is the distance tag calculation, DIP p is the p-th cluster distance, is the coordinate of the pth cluster center, is the cluster coordinates whose distance needs to be calculated, || || is the distance calculation, DIP pc is the radius of the pth distance center, cn is the total number of all cluster labels, The audio data label coordinates of the cluster center; is the video content label of the cluster center; The label coordinates of the input audio data; is the input video content label; is the audio emotion feature label of the cluster center, is the audio background feature label of the cluster center, is the audio content feature label of the cluster center;

[0188] Traverse the first topic tag, calculate the background sound scene relevance of the first topic tag of the adjacent audio segmentation sequence; if the background sound scene relevance is greater than or equal to the background sound scene relevance threshold; do not adjust the video and audio data and continue traversing; if the background sound scene relevance is less than the background sound scene relevance threshold; obtain the segmentation interval time length of the adjacent audio segmentation sequence, and perform background sound transition adjustment according to the segmentation interval time length and the adjacent audio change characteristics;

[0189] Furthermore, all first topic labels are sorted and traversed in the order of timestamps to obtain the first topic labels of the adjacent audio segmentation sequence to be calculated. and

[0190]

[0191]

[0192] Among them, δ rele is the background sound scene relevance, for The total number of tags, for The total number of tags, For the i-th The weight of the label, Tsim() is for identification and Same first topic tag calculation, For the i-th Tags, For the jth Tags, For the i-th The number of times the tag type appears, is the total number of tag types;

[0193] Furthermore, the transition adjustment is to determine the length of the t-th background sound segmentation interval; if the segmentation interval is greater than or equal to the background sound adjustment duration threshold, the amplitude data of the audio segmentation sequence to be adjusted is obtained, and the mean of the adjusted amplitude data is generated in combination with the segmentation interval, and the amplitude data is adjusted down to 0 according to the timestamp of the segmentation interval until the amplitude data is 0;

[0194] If the segmentation interval time is less than the background sound adjustment time threshold, the type of the audio change feature of the t+1th background sound is obtained; the background sound material with the same type of audio change feature is called from the material library; the background sound material is added to the background sound transition interval of the tth background sound and the t+1th background sound, and when the tth background sound reaches the background sound transition interval, the amplitude data is adjusted down to 0; the background sound transition interval is calculated as follows:

[0195] T start =(1-α t )·T t ;

[0196] T end =α t+1 ·T t+1 ;

[0197] Among them, T start is the starting timestamp of the background sound transition interval, T end is the end timestamp of the background sound transition interval, T t is the total number of timestamps of the tth background sound, T t+1 is the total number of timestamps of the t+1th background sound; α t is the calculation weight of the tth background sound, α t+1 is the calculation weight of the t+1th background sound.

[0198] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. An audio and video synthesis processing method based on artificial intelligence, characterized in that: include: Acquire video image data and video audio data of target audio and video, and extract audio change features of the video audio data; Acquire an audio emotion feature label, an audio background feature label, and an audio content feature label according to the audio change feature; Segment the video and audio data according to the audio-video audio similarity of the audio emotion feature tag, the audio background feature tag, and the audio content feature tag, and generate an audio segmentation sequence and a segmentation sequence timestamp; Segmenting the video image data according to the segmentation sequence timestamp to generate a video segmentation sequence; Extracting the video content tags of each of the video segmentation sequences; combining the audio emotion feature tags, the audio background feature tags and the audio content feature tags corresponding to each of the audio segmentation sequences to generate corresponding first topic tags; sorting the first topic tags in the order of the timestamps of the segmentation sequences; Traverse the first topic tag and calculate the background sound scene relevance of the first topic tag of the adjacent audio segmentation sequence; if the background sound scene relevance is greater than or equal to the background sound scene relevance threshold; do not adjust the video audio data and continue traversing; if the background sound scene relevance is less than the background sound scene relevance threshold; obtain the segmentation interval time length of the adjacent audio segmentation sequence, and perform background sound transition adjustment according to the segmentation interval time length and the adjacent audio change characteristics.

2. The method for audio and video synthesis processing based on artificial intelligence according to claim 1, characterized in that: The video image data is video image data with image editing effects added, and the image editing effects include text materials, text effect materials and image effect materials; the video audio data is video audio data with audio editing effects added; the audio editing effects include unadjusted background music materials, unadjusted background sound effect materials and unadjusted background sound materials.

3. The method for audio and video synthesis based on artificial intelligence according to claim 1, characterized in that: The audio change feature, the audio emotion feature label, the audio background feature label, and the audio content feature label are obtained through an audio-video feature audio extraction classification model, and the audio-video feature audio extraction classification model includes: The audio-video feature audio extraction model includes an audio-video audio feature processing unit, an audio-video audio input unit, an audio-video audio feature extraction unit, an audio-video audio feature output unit and an audio-video audio label classification output unit; The audio, video and audio feature processing unit separates the video and audio data from the audio editing effect to generate a plurality of audio, video and audio pre-input data; the audio, video and audio input unit divides the plurality of audio, video and audio pre-input data into equal amounts to obtain an audio, video and audio input data set; The audio, video and audio feature extraction unit extracts the change features of the audio, video and audio input data set to generate audio, video and audio change feature data, and the audio, video and audio feature output unit classifies the audio, video and audio change feature data and outputs the audio change features; The audio-video audio label classification output unit performs further feature classification calculation on the audio-video audio change feature data, and outputs the audio emotion feature label, the audio background feature label and the audio content feature label.

4. The method for audio and video synthesis based on artificial intelligence according to claim 1, characterized in that: The audio-video audio similarity of the audio emotion feature label, the audio background feature label and the audio content feature label is calculated by sorting the audio emotion feature label, the audio background feature label and the audio content feature label in timestamp order and selecting a fixed number N i The audio-video audio similarity is calculated based on the audio emotion feature labels, the audio background feature labels and the audio content feature labels.

5. The method for audio and video synthesis based on artificial intelligence according to claim 4, characterized in that: The audio, video and audio similarity calculation includes: Among them, δ N is the audio / video audio similarity, β det is the same continuity weight of the audio content feature label, β emt is the same continuity weight of the audio emotion feature label, β back is the same continuity weight of the audio background feature label; α det , α emt and α back is the audio similarity weight, and the audio similarity weight relationship is α det +α emt +α back =1.

6. The method for audio and video synthesis based on artificial intelligence according to claim 1, characterized in that: The calculations for evaluating the generation of the first topic label include: Segmenting the audio segmentation sequence according to the audio-video audio similarity to generate new audio emotion feature labels, audio background feature labels and audio content feature labels; According to the time sequence of the audio segmentation sequence and the video segmentation sequence, the video content label T corresponding to the time sequence is selected. video , the new audio emotion feature label T det , the new audio background feature label T emt and the new audio content feature label T back The first topic tag is calculated and generated.

7. The method for audio and video synthesis processing based on artificial intelligence according to claim 6, characterized in that: The first topic tag generation includes: Among them, δ F for the first topic label; a tag type for the first topic tag; The number of occurrences of the corresponding tag type; CL() is the tag set calculated by the audio and video topic tag clustering model, SimCount() is the number of occurrences of each tag in all tags, N n is the total number of the first topic tags.

8. The method for audio and video synthesis based on artificial intelligence according to claim 1, characterized in that: The calculation of the background sound scene relevance of the first topic label of adjacent audio segmentation sequences includes: Sort and traverse all the first topic labels in the order of timestamps to obtain the first topic labels of the adjacent audio segmentation sequence to be calculated and Among them, δ rele is the background sound scene relevance, for The total number of tags, for The total number of tags, For the i-th The weight of the label, Tsim() is for identification and Same first topic tag calculation, For the i-th Tags, For the jth Tags, For the i-th The number of times the tag type appears, The total number of tag types.

9. The method for audio and video synthesis based on artificial intelligence according to claim 1, characterized in that: The transition adjustment of the background sound according to the segmentation interval time length and the adjacent audio change characteristics includes: Determine the length of the segmentation interval of the t-th background sound; if the segmentation interval is greater than or equal to the background sound adjustment duration threshold, obtain the amplitude data of the audio segmentation sequence that needs to be adjusted, combine the segmentation interval, generate the mean of the adjusted amplitude data, and adjust the amplitude data down to 0 according to the timestamp of the segmentation interval; If the duration of the segmentation interval is less than the background sound adjustment duration threshold, the type of the audio change feature of the t+1th background sound is obtained; the background sound material with the same type of audio change feature is called from the material library; the background sound material is added to the background sound transition interval between the tth background sound and the t+1th background sound, and when the tth background sound reaches the background sound transition interval, the amplitude data is lowered to 0.

10. The method for audio and video synthesis based on artificial intelligence according to claim 9, characterized in that: The background sound transition interval calculation includes: T start =(1-α t )·T t ; T end =α t+1 ·T t+1 ; Among them, T start is the starting timestamp of the background sound transition interval, T end is the end timestamp of the background sound transition interval, T t is the total number of timestamps of the tth background sound, T t+1 is the total number of timestamps of the t+1th background sound; α t is the calculation weight of the tth background sound, α t+1 is the calculation weight of the t+1th background sound.

Images