AI-driven audio and video content semantic segmentation method and system

Abstract

The invention relates to an AI-driven audio and video content semantic segmentation method and system, and relates to the technical field of audio and video semantic segmentation, and the method comprises the steps: dividing silent frames in a target audio to obtain target audio content, dividing audio segments, and recognizing audio semantic information; dividing a target video to obtain video segments, identifying and analyzing the similarity with audio semantic information, screening to obtain matched video segments and remaining video frames, and processing to obtain a comprehensive matching degree; a first segmentation synthesis result is generated, the coherence is analyzed after the remaining video frames are randomly inserted, and an optimized insertion sequence is obtained in combination with the comprehensive matching degree; and inserting the left video frame and the silent frame between the matched video segments in the first segmentation synthesis result according to the optimized insertion sequence to obtain an audio-video segmentation synthesis result. According to the method, the problems that the audio and video semantic matching degree is low, the coherence of the synthesized content is poor and the audio and video synthesis quality and rationality are influenced due to the fact that the traditional editing synthesis method does not consider the audio and video semantics sufficiently are solved.

Description

Technical Field

[0001] This application relates to the field of audio and video semantic segmentation, and in particular to an AI-driven method and system for audio and video content semantic segmentation. Background Technology

[0002] With the increasing demands for audio and video presentation quality in media creation, film and television editing, and other fields, the semantic fit and coherence of audio and video editing and synthesis results have become key technical requirements for ensuring content quality.

[0003] Currently, traditional audio and video editing and synthesis methods do not fully consider the semantic relationship between audio and video, and cannot achieve accurate matching between audio semantics and video content. This not only easily leads to low audio and video semantic matching degree and poor coherence of the synthesized content, but also reduces the quality and rationality of the audio and video synthesis results, and increases the risk of the effect when the synthesized content is applied to practical applications. Summary of the Invention

[0004] This application provides an AI-driven semantic segmentation method and system for audio and video content, which improves the problem of low semantic matching degree of audio and video due to insufficient semantic consideration in traditional editing and synthesis, and improves the coherence of synthesized content as well as the quality and rationality of audio and video synthesis.

[0005] The embodiments of this application disclose the following technical solutions: In a first aspect, embodiments of this application provide an AI-driven semantic segmentation method for audio and video content, the method comprising: The target audio and target video are acquired, silent frames within the target audio are segmented, the target audio content is obtained, and it is divided into multiple audio segments. Multiple audio semantic information of multiple audio segments is identified. The target video is divided into video segments, the content of the video segments is identified, and the similarity with the multiple audio semantic information is analyzed. Multiple matching video segments and multiple residual video frames are selected and processed to obtain the comprehensive matching degree. A first segmentation and synthesis result is generated based on multiple matching video segments and multiple audio segments. Multiple residual video frames are randomly inserted between multiple matching video segments in the first segmentation and synthesis result. A coherence analysis is performed to obtain the coherence, and the optimized insertion order is obtained by combining the comprehensive matching degree. According to the optimized insertion order, the multiple legacy video frames and multiple silent frames are inserted between multiple matching video segments in the first segmentation and synthesis result to obtain the audio-visual segmentation and synthesis result.

[0006] Secondly, embodiments of this application provide an AI-driven audio and video content semantic segmentation system, the system comprising: The audio and video acquisition and segmentation module is used to acquire target audio and target video, segment silent frames within the target audio, obtain the target audio content, divide it into multiple audio segments, and identify multiple audio semantic information of multiple audio segments. The video segmentation and matching module is used to segment the target video into video segments, identify the content of the video segments, analyze the similarity with the multiple audio semantic information, filter to obtain multiple matching video segments and multiple residual video frames, and process to obtain the comprehensive matching degree. The legacy frame order optimization module is used to generate a first segmentation and synthesis result based on multiple matching video segments and multiple audio segments, randomly insert multiple legacy video frames between multiple matching video segments in the first segmentation and synthesis result, perform coherence analysis to obtain coherence, and combine the comprehensive matching degree to obtain an optimized insertion order; The audio-visual segmentation and synthesis module is used to insert the multiple legacy video frames and multiple silent frames into the multiple matching video segments within the first segmentation and synthesis result according to the optimized insertion order, so as to obtain the audio-visual segmentation and synthesis result.

[0007] One or more technical solutions provided in this application have at least the following technical effects or advantages: This application proposes an AI-driven semantic segmentation method and system for audio and video content. Through AI technology, it realizes refined processing of audio and video from semantic recognition, matching to synthesis. By combining steps such as audio silent frame separation, video content recognition, semantic similarity matching, legacy frame order optimization and audio-video synchronous synthesis, it achieves accurate segmentation and effective synthesis of audio and video at the semantic level. First, target audio and target video of the same duration are acquired. The target audio is segmented into frames and silent frames are separated. Valid audio segments are divided according to a preset time window, and the audio semantic information of each audio segment is output through an audio semantic recognizer. Next, the target video is randomly segmented into video segments according to the same time window. The video content recognizer built based on deep learning is used to identify the video content, calculate the similarity between the video content and the audio semantics, and filter to obtain multiple matching video segments and a comprehensive matching degree, while retaining unmatched legacy video frames. Then, a preliminary synthesis result is generated based on the matching video segments and audio segments. Legacy video frames are randomly inserted, and the coherence is output through a video coherence analyzer. The segmentation synthesis quality coefficient is calculated based on the comprehensive matching degree. The number of optimizations is adjusted and the insertion order of legacy frames is iteratively optimized based on this segmentation synthesis quality coefficient. Finally, legacy video frames and silent frames are synchronously inserted into the preliminary synthesis result according to the optimized insertion order to ensure that their timelines match, thus obtaining a complete audio and video segmentation synthesis result.

[0008] The technical solution of this application solves the problems of low semantic matching degree, poor continuity of synthesized content, and easy misalignment of audio and video caused by insufficient consideration of semantics in traditional audio and video editing and synthesis. It realizes the all-dimensional quality improvement of audio and video from semantic matching to picture continuity, and provides technical support for ensuring the rationality and high quality of audio and video synthesis in various scenarios. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 A flowchart illustrating an AI-driven semantic segmentation method for audio and video content provided in this application embodiment; Figure 2 This is a schematic diagram of the structure of an AI-driven audio and video content semantic segmentation system provided in an embodiment of this application.

[0011] The components represented by each number in the attached diagram are explained below: Audio and video acquisition and segmentation module 01, video segmentation and matching module 02, legacy frame order optimization module 03, audio and video segmentation and synthesis module 04. Detailed Implementation

[0012] This application provides an AI-driven audio and video content semantic segmentation method and system to solve the technical problem that existing traditional editing and synthesis methods do not adequately consider audio and video semantics, which easily leads to low audio and video semantic matching degree and poor content coherence after synthesis, thus affecting the quality and rationality of audio and video synthesis.

[0013] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0014] In the description of this application, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the stated features. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0015] In the description of this application, the term "for example" is used to mean "used as an example, illustration, or description." Any embodiment described as "for example" in this application is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is provided to enable any person skilled in the art to make and use the invention. Details are set forth in the following description for purposes of explanation. It should be understood that those skilled in the art will recognize that the invention can be made without using these specific details. In other instances, well-known structures and processes will not be described in detail to avoid obscuring the description of the invention with unnecessary detail. Therefore, the invention is not intended to be limited to the embodiments shown, but is consistent with the broadest scope of the principles and features disclosed in this application.

[0016] Example 1, as shown in the appendix Figure 1 As shown, this application provides an AI-driven semantic segmentation method for audio and video content, the method comprising the following steps: S110: Acquire the target audio and target video, divide the silent frames within the target audio, obtain the target audio content, divide it into multiple audio segments, and identify multiple audio semantic information of multiple audio segments; In this embodiment of the application, in the scenario of audio and video editing and synthesis, in order to avoid matching deviations caused by invalid silent information mixed in the audio content or inaccurate audio semantic recognition, it is necessary to preprocess and extract the semantics of the target audio first, while ensuring the consistency of the audio and video time dimension, so as to improve the quality and rationality of subsequent audio and video semantic segmentation and synthesis.

[0017] Specifically, the target audio and target video to be segmented and synthesized are first obtained, and their durations must be the same to ensure that the subsequent audio and video can correspond accurately on the timeline and avoid time misalignment issues.

[0018] Furthermore, the acquired target audio is divided into frames, and the audio intensity is extracted frame by frame. Frames with audio intensity less than a preset audio intensity threshold are determined to be silent frames, while all frames other than silent frames are integrated into the target audio content, so as to separate invalid audio from valid audio.

[0019] Furthermore, the target audio content is divided into multiple independent audio segments according to a preset time window. The size of the preset time window can be adjusted according to the actual length of the audio content and the semantic integrity requirements to ensure that each audio segment can carry a complete semantic unit.

[0020] Finally, the divided audio segments are input into the trained audio semantic recognizer, which performs semantic analysis on each audio segment, such as recognizing the text content within the audio segment, and finally outputs the audio semantic information corresponding to each audio segment.

[0021] This step provides a clear audio semantic benchmark for subsequent semantic matching of video segments by acquiring the audio and video of the matching duration, separating silent frames, dividing audio segments, and identifying audio semantic information. This enables the subsequent matching process to proceed based on the semantics of each audio segment and the corresponding video segment, ensuring more accurate matching results.

[0022] Step S110 in the method provided in this application embodiment includes: Obtain the target audio and target video to be segmented and synthesized, wherein the target audio and target video have the same duration; The target audio is divided into frames, and the audio intensity of each frame is extracted. Frames with audio intensity less than the audio intensity threshold are designated as silent frames, and all frames other than silent frames are designated as target audio content. The target audio content is divided into multiple audio segments according to a preset time window. Multiple audio segments are input into the audio semantic recognizer, and multiple audio semantic information are output.

[0023] In this embodiment of the application, in order to provide an accurate audio benchmark for audio-video semantic matching and avoid invalid silent information mixed in the audio, disordered audio segmentation or fuzzy semantic recognition, it is necessary to first perform systematic preprocessing and semantic extraction on the target audio, while ensuring the consistency of audio and video in the time dimension, so as to improve the quality and rationality of the overall audio-video semantic segmentation and synthesis.

[0024] Specifically, the first step is to obtain the target audio and target video to be segmented and synthesized, such as audio material used for video dubbing and corresponding video material. The duration of both must be the same (the duration setting needs to be determined based on the actual audio and video editing requirements).

[0025] For example, if the target video duration is 5 minutes, the corresponding target audio duration also needs to be adjusted to 5 minutes to ensure that the subsequent audio and video can correspond accurately on the timeline, avoiding time misalignment issues such as the audio finishing playing while the video continues or the video ending while the audio is not finished.

[0026] Furthermore, after acquiring the audio and video data, the target audio is divided into frames, breaking down the continuous audio stream into discrete audio frames, and then the intensity value of each frame of audio is extracted frame by frame.

[0027] Furthermore, an audio intensity threshold is set. This threshold needs to be determined based on the actual type of audio and the requirements of the scenario. For example, in narration-type audio, the audio intensity threshold can be set to 10% of the normal narration volume. Audio frames below this threshold are judged as silent frames, which typically do not carry valid voice information. At the same time, all frames except silent frames are integrated into the target audio content, thereby achieving a clear separation between invalid and valid audio.

[0028] Furthermore, the target audio content is divided according to a preset time window to obtain multiple audio segments.

[0029] The duration of the preset time window also needs to be determined in conjunction with the semantic integrity of the audio content. For example, in dialogue-type audio, the time window can be set to 3 seconds to ensure that each audio segment can fully carry a short dialogue; for long narration-type audio, the time window can be adjusted to 5 seconds, which avoids the increased semantic recognition burden caused by excessively long single audio segments, and also prevents semantic breaks caused by excessively short segments.

[0030] During the segmentation process, the target audio content must be cut into segments according to the time window to ensure that the duration of each audio segment is uniform, providing a regular basic unit for subsequent segment-by-segment semantic recognition and video matching.

[0031] Furthermore, the divided audio segments are input into the audio semantic recognizer to obtain the corresponding audio semantic information.

[0032] The method provided in this application embodiment, which "inputs multiple audio segments into an audio semantic recognizer and outputs multiple audio semantic information", includes: Collect a set of sample audio segments, annotate the audio semantics within each sample audio segment, and obtain a set of sample audio semantic information; An audio semantic recognizer is constructed based on automatic speech recognition networks in machine learning. The audio semantic recognizer is trained using the set of sample audio segments and the set of sample audio semantic information until convergence. The multiple audio segments are input into the input audio semantic recognizer, and multiple audio semantic information is output.

[0033] In this embodiment of the application, in order to make the audio semantic recognition result accurately match the audio and video semantics, it is necessary to first build and train an audio semantic recognizer with high recognition accuracy, and then use the audio semantic recognizer to parse the semantics of the target audio segment to form an audio semantic benchmark, so as to ensure the quality of the overall audio and video semantic segmentation and synthesis.

[0034] Specifically, the first step is to collect a set of sample audio segments. The selection of sample audio segments needs to cover common scenarios in audio and video editing and synthesis, such as film and television dubbing, educational course narration, daily conversations, documentary explanations, and other different types. At the same time, it should take into account audio data with different speaking speeds, timbres, and background noise intensities. The duration of a single sample audio segment should be controlled within the range of 1-10 seconds to ensure that the sample can contain complete semantic units.

[0035] Furthermore, after collecting the set of sample audio segments, the audio semantics within each sample audio segment are labeled to obtain a set of sample audio semantic information.

[0036] Specifically, the annotation should adopt a multi-level semantic description approach, that is, first annotate the core semantic category, and then annotate the specific semantic content. For example, for an audio recording describing "bird activities in the mountains and forests in the early morning", the core category is labeled as "scene description", and the specific content is labeled as "describes the calling characteristics and activity status of various birds in the early morning mountain and forest environment".

[0037] In addition, the annotation process must be completed independently by at least two professional annotators, and the discrepancies in the annotations must be corrected through cross-validation to ensure that the annotation accuracy is not less than 98%, so as to provide accurate supervision signals for subsequent model training.

[0038] Furthermore, an audio semantic recognizer is constructed based on automatic speech recognition networks in machine learning.

[0039] Specifically, a hybrid network architecture of "Convolutional Neural Network (CNN) + Recurrent Neural Network (RNN) + Attention Mechanism" is selected. Among them, the CNN layer is used to extract local features of audio, such as speech frequency distribution, etc., and adopts three convolutional layers with kernel sizes of 3×3, 5×5 and 3×3 respectively, and a stride of 1 for each layer. Each convolutional layer is followed by a batch normalization layer and a ReLU activation function to reduce the gradient vanishing problem.

[0040] In addition, the RNN layer uses a bidirectional long short-term memory network with two hidden layers, each with 256 hidden units, to capture the temporal semantic associations of audio and solve the semantic dependency problem of long audio sequences.

[0041] Meanwhile, the attention mechanism layer employs additive attention, calculating the weight of each temporal feature to make the model focus more on audio segments that contribute more to semantics. The network output layer uses a fully connected layer with Softmax as the activation function, and the output dimension is consistent with the number of semantic categories in the samples, in order to achieve the mapping between semantic category classification and specific semantic content.

[0042] Furthermore, an audio semantic recognizer is trained using a set of sample audio segments and a set of sample audio semantic information.

[0043] First, the sample audio segments are preprocessed by converting the audio to a uniform 16kHz sampling rate and 16-bit mono format. The time-domain audio signal is converted into a Mel spectrogram in the frequency domain through short-time Fourier transform, which serves as the network input feature. Then, the semantic information of the samples is one-hot encoded to convert the semantic annotations in text form into numerical vectors that the model can recognize.

[0044] Secondly, the training process uses a stochastic gradient descent (SGD) optimizer with an initial learning rate of 0.01, which is decayed to 0.5 times the original rate every 10 iterations (i.e., multiplied by 0.5). The cross-entropy loss function is used to measure the difference between the model's predicted semantics and the actual labeled semantics.

[0045] Meanwhile, the training batch size is set to 64, the number of iterations is set to 100 rounds, and the model performance is evaluated using a validation set (divided from the sample set in a 2:8 ratio, with the validation set accounting for 20%) after each round of training. When the semantic recognition accuracy of the validation set no longer improves (fluctuation is less than 0.1%) and the loss value steadily decreases to below 0.05 for 5 consecutive rounds, the model is considered to have converged and training is stopped. At this time, the semantic recognition accuracy of the model must reach more than 95%.

[0046] Furthermore, after the audio semantic recognizer is trained, the previously segmented target audio segments are input into the audio semantic recognizer. After the CNN layer extracts local features, the Bi-LSTM layer captures temporal correlations, and the attention mechanism layer focuses on key segments, the output layer outputs the semantic prediction results, and finally the audio semantic information corresponding to each target audio segment is obtained.

[0047] For example, an audio segment that “introduces the urban subway transfer process” is identified as “core category: operation guidance, specific content: explaining the selection of transfer stations, passageway directions and precautions between different urban subway lines”, providing accurate and clear audio semantic basis for the semantic matching of subsequent video segments.

[0048] S120: Divide the target video into video segments, identify the content of the video segments, analyze the similarity with the multiple audio semantic information, filter to obtain multiple matching video segments and multiple residual video frames, and process to obtain the comprehensive matching degree; In this embodiment of the application, in order to ensure that the video content accurately corresponds to the previously identified audio semantics and to avoid poor synthesis results due to the disconnect between audio and video content, it is necessary to segment the target video, identify its content, and perform semantic matching. At the same time, the overall matching degree is quantified to ensure the rationality and quality of the final audio and video synthesis.

[0049] Specifically, the target video is first randomly divided into video segments according to the same preset time window as the audio segment division to obtain the first video segment.

[0050] The length of the preset time window must be consistent with that of the audio segment division in the early stage to ensure that the audio and video correspond in the time dimension. If the video content has special scene requirements with frequent short-duration scene switching, multiple small video segments can be combined to form a video segment that meets the length of the time window, ensuring that the divided video segment can contain complete picture semantics.

[0051] Furthermore, the first video segment is input into the video content recognizer, which then outputs the specific content of the video segment.

[0052] Furthermore, the similarity between the content of the first video segment and the multiple audio semantic information identified in the previous stage is calculated respectively. The audio segment with the highest similarity is retained and matched with the first video segment to obtain the first matched video segment.

[0053] Similarly, following the above process, the target video is randomly divided, its content is identified, and its similarity is matched. This process is repeated continuously to gradually obtain multiple matching video segments, while recording the maximum similarity score for each matching process.

[0054] During the segmentation and matching process, some video frames cannot be effectively matched with any audio segments and will become unsegmented and unmatched legacy video frames. These legacy video frames need to be processed in conjunction with silent frames in the audio to ensure the continuity of the overall audio and video content.

[0055] Furthermore, after obtaining multiple maximum similarity values ​​corresponding to all matching video segments, the obtained maximum similarity values ​​are summed and then divided by the total number of maximum similarity values. The average value is then used to calculate the comprehensive matching degree, which can intuitively reflect the overall level of audio and video semantic matching.

[0056] This step achieves precise semantic correspondence between audio and video by segmenting and matching video, filtering out residual video frames, and calculating the overall matching degree. It provides a quality reference standard for subsequent insertion of residual video frames and adjustment of the synthesis order, ensuring that subsequent synthesis steps can specifically improve the audio and video segmentation and synthesis effect.

[0057] Step S120 in the method provided in this application embodiment includes: The target video is randomly divided into video segments according to a preset time window to obtain the first video segment; The first video segment is input into the video content recognizer, which recognizes and outputs the video content. The video content recognizer is built based on deep learning and is trained using a set of sample video segments and a set of sample video content. Calculate the similarity between the video content and multiple audio and speech information respectively, retain the audio segment corresponding to the maximum similarity, and match it with the first video segment to obtain the first matched video segment; Continue to randomly divide the video segments, perform similarity calculation and matching, and obtain multiple matching video segments and multiple maximum similarity scores; Calculate the mean of multiple maximum similarities to obtain the overall matching degree.

[0058] In this embodiment of the application, in order to achieve an accurate semantic correspondence between the target video and the audio segment obtained in the preprocessing stage, it is necessary to filter out video segments that match the audio semantics through processes such as video segment division, recognition, and matching, and quantify the comprehensive matching degree to provide a clear basis for subsequent processing of residual video frames and optimization of the synthesis order.

[0059] Specifically, the target video is first randomly divided into video segments according to a preset time window to obtain the first video segment. The length of the preset time window must be consistent with that of the audio segment segmentation in the early stage. For example, if the target audio content is divided into multiple audio segments in the early stage using a 3-second time window, the same time window of the same length is used here to divide the video to ensure that the subsequent audio and video correspond in the time dimension.

[0060] At this point, if the target video contains scenes with frequent short-duration scene transitions, multiple consecutive short video segments can be combined to form a video segment that meets the preset time window length, ensuring that each video segment after division can carry complete image semantics and avoiding semantic fragmentation due to excessively short single-segment duration.

[0061] Furthermore, the obtained first video segment is input into a video content recognizer, which then outputs the specific content of that video segment. The video content recognizer needs to be built based on deep learning and requires systematic training before it can be put into use.

[0062] Specifically, firstly, a collection of sample video segments covering different scenes and objects is collected, such as video segments containing animal activities, human actions, natural scenery, urban architecture, etc. Then, the content of each sample video segment is accurately labeled by human hands to form a sample video content collection.

[0063] Furthermore, the sample video segments are input into the constructed deep learning model, and the model is trained using the sample video content set as a supervision signal. The model parameters are continuously adjusted until the model's recognition accuracy of video content stabilizes at a high level, ensuring that the video images can be accurately converted into textual descriptions that can be compared with audio semantics.

[0064] Specifically, the video content recognizer is based on deep learning technology and is constructed using a combination of Convolutional Neural Network (CNN) and Long Short-Term Memory Network (LSTM). The CNN module employs a lightweight architecture with three convolutional layers, each with a 3×3 kernel and a stride of 1 to ensure the feature map size matches the input. Each convolutional layer is followed by a batch normalization layer and a ReLU activation function to reduce training bias and enhance non-linear fitting capabilities. Finally, a global average pooling layer outputs a 256-dimensional visual feature vector.

[0065] In addition, the LSTM module has a unidirectional structure with one hidden layer and 128 hidden units. A dropout layer with a probability of 0.2 is set between the input layer and the hidden layer to prevent overfitting. The output layer is mapped to the dimension corresponding to the video content category through a fully connected layer, and the activation function is Softmax.

[0066] During the training process, a set of sample video segments covering multiple scenarios is first collected. The duration of each video segment is consistent with the preset time window. Eight key frames are extracted as input, and the video content categories and text descriptions are manually labeled to form a set of sample video content.

[0067] Furthermore, the sample video content set was divided into a training set and a validation set in an 8:2 ratio. Keyframes were preprocessed using normalization, and labels were converted using one-hot encoding. The Adam optimizer was used during training, with an initial learning rate set to 1e. -4 The loss function is cross-entropy loss, the batch size is set to 32, and the training rounds are 50. The accuracy is evaluated using a validation set every 10 training rounds. If the accuracy does not improve for 3 consecutive rounds, the learning rate is halved until the model converges to ensure accurate output of video content recognition results.

[0068] Ultimately, the output video content recognition results can accurately reflect the semantics of the video segment, providing a clear and unified comparison benchmark for subsequent similarity calculation with audio semantic information.

[0069] For example, if the input video segment is "a 3-second scene of flowers blooming in a park", the video content recognizer will output the category "natural landscape - plant growth" and the text description "pink flowers gradually bloom against the background of park greenery, with petals unfolding". The recognition result can be directly used for semantic matching with the corresponding audio segment to ensure semantic consistency of audio and video content.

[0070] Furthermore, the similarity between the recognition result of the first video segment and the multiple audio semantic information obtained in the previous stage is calculated separately. Specifically, the video content description and audio semantic information need to be converted into feature vectors of the same dimension first, and then the similarity between the two is calculated using the cosine similarity algorithm in existing technology.

[0071] For example, if the recognition result of the first video segment is "a white puppy running on the lawn", and the semantic information of a certain audio segment obtained earlier is "describes the scene of the puppy running on the outdoor grass", then the two are first converted into 512-dimensional feature vectors, and then the cosine similarity algorithm is used to calculate the similarity value, which is 0.92, indicating that the two are highly semantically compatible; while if the semantic information of another audio segment is "introducing the architectural layout of the city park", the calculated similarity value is only 0.21, indicating that the semantic correlation between the two is low. Finally, the former with the highest similarity is retained to match the video segment.

[0072] Furthermore, after completing the similarity calculation for all audio segments, the audio segment with the highest similarity value is retained and bound to the first video segment to form the first matching video segment, so as to achieve semantic correspondence between single audio and video segments.

[0073] Similarly, following the above process of "randomly dividing video segments - inputting content into the recognizer - calculating similarity - matching audio segments", the remaining part of the target video is processed again and again to gradually obtain multiple matching video segments, while recording the maximum similarity value corresponding to each matching process.

[0074] During this process, some video frames become legacy video frames because their content cannot be effectively associated with the semantics of any audio segments. For example, meaningless transitional scenes in the video or scene fragments unrelated to the audio theme cannot be classified into any matching video segments. These legacy video frames will be processed in conjunction with silent frames in the audio to avoid video gaps during synthesis.

[0075] Finally, the maximum similarity values ​​corresponding to all matched video segments are statistically calculated. This involves adding these values ​​together and dividing by the total number of maximum similarities. The average value obtained is the overall matching degree.

[0076] The overall matching degree ranges from 0 to 1. The closer the value is to 1, the better the overall semantic matching effect between the target video and audio. Conversely, the lower the value is, the worse the matching effect is. Targeted optimization is needed in subsequent steps to ensure the rationality and coherence of the final synthesized content.

[0077] This step involves randomly dividing the target video into video segments, identifying the video content, calculating the similarity with the audio semantics, filtering matching video segments, and calculating the maximum similarity. The calculated comprehensive matching degree can intuitively grasp the quality of the audio-video semantic correspondence, providing a quantitative reference standard for adjusting the subsequent synthesis results.

[0078] S130: Generate a first segmentation and synthesis result based on multiple matching video segments and multiple audio segments, randomly insert multiple remaining video frames between multiple matching video segments in the first segmentation and synthesis result, perform coherence analysis to obtain coherence, and combine the comprehensive matching degree to obtain an optimized insertion order; In this embodiment of the application, in order to form a preliminary synthesis framework from the matched audio and video segments, and to properly handle the remaining video frames to avoid gaps in the synthesized content, it is also necessary to optimize the insertion order by combining coherence analysis and comprehensive matching degree, so as to improve the coherence and overall quality of the final audio and video synthesis.

[0079] Specifically, the first step is to generate a segmentation and synthesis result based on multiple matching video segments and multiple audio segments. During this process, each matching video segment needs to be time-aligned with its corresponding audio segment to ensure precise time synchronization between audio playback and video display.

[0080] Furthermore, multiple legacy video frames are randomly inserted between multiple matching video segments within the first segmentation and synthesis result to obtain a second adjusted synthesis result.

[0081] The random insertion of legacy video frames must avoid the middle part of the matching video segment and should only be inserted at the junction of two adjacent matching video segments to avoid destroying the content integrity of a single matching video segment.

[0082] Furthermore, the second adjusted synthesis result is input into the video coherence analyzer to output the coherence.

[0083] Among them, the video coherence analyzer needs to be trained in advance using a sample synthesized video set and an labeled sample coherence set. It can judge the coherence of the content based on features such as color transition and scene correlation of the video screen. The coherence value ranges from 0 to 1. The closer the value is to 1, the better the coherence of the synthesized result after inserting the legacy video frame. Conversely, it indicates that there is a break in the connection between the screen.

[0084] Furthermore, after obtaining the coherence, the segmentation and synthesis quality coefficient is calculated by combining it with the previously calculated comprehensive matching degree. This segmentation and synthesis quality coefficient can comprehensively reflect the semantic matching level and the coherence of the synthesis result, and serves as the core basis for optimizing the insertion order of legacy video frames.

[0085] Finally, the insertion order of multiple legacy video frames is optimized based on the segmentation and synthesis quality coefficients to obtain an optimized insertion order. If the segmentation and synthesis quality coefficients are low, it indicates that the current insertion order results in poor image continuity. The distribution of legacy video frames at the junctions of different matching video segments needs to be adjusted until an optimal order that balances continuity and overall matching is found, providing a reasonable insertion scheme for the subsequent final synthesis.

[0086] Step S130 in the method provided in this application embodiment includes: The first segmentation and synthesis result is generated based on multiple matching video segments and multiple audio segments, wherein the multiple matching video segments and multiple audio segments are matched on the time axis; Multiple legacy video frames are randomly inserted between multiple matching video segments in the first segmentation and synthesis result to obtain a second adjusted synthesis result. The second adjusted synthesis result is input into the video coherence analyzer, and the output is the coherence degree. The video coherence analyzer is trained using a sample synthesized video set and a labeled sample coherence degree set, and the coherence degree is greater than or equal to 0 and less than or equal to 1. Based on the coherence and overall matching degree, the segmentation and synthesis quality coefficient is calculated. Based on the segmentation and synthesis quality coefficients, the insertion order of multiple legacy video frames is optimized to obtain an optimized insertion order.

[0087] In this embodiment of the application, in order to form a complete and time-synchronized preliminary synthesis framework by matching audio and video segments, and to make reasonable use of residual video frames to avoid wasting screen resources, it is also necessary to optimize the insertion order through coherence analysis and segmentation synthesis quality coefficient calculation to avoid poor coherence of the synthesized content due to disordered insertion of residual frames, so as to improve the coherence, completeness and overall quality of the final audio and video synthesis.

[0088] First, a first segmentation and synthesis result is generated based on multiple matching video segments and multiple audio segments. This involves precisely matching each matching video segment with its corresponding audio segment along the timeline to ensure that the audio playback progress is completely synchronized with the video display.

[0089] For example, if a matching video segment is "3 seconds of coffee brewing footage" and the corresponding audio segment is "3 seconds of coffee brewing step explanation", the start timestamp of both should be set to 0 seconds and the end timestamp to 3 seconds to ensure that the narration and the on-screen actions correspond perfectly. If there is a slight difference in duration, such as 3.1 seconds for the video segment and 3 seconds for the audio segment, the video segment needs to be trimmed at the frame level to ensure that the duration of each set of matching audio and video is completely consistent, thus forming a preliminary synthesis framework without time misalignment.

[0090] Furthermore, after obtaining the first segmentation and synthesis result, multiple residual video frames are randomly inserted between multiple matching video segments in the result to obtain the second adjusted synthesis result.

[0091] The insertion of legacy video frames must be strictly limited to the junction of two adjacent matching video segments, and cannot be inserted into a single matching video segment, so as to avoid destroying the integrity of the original matching audio and video content.

[0092] For example, between the two matching video segments "coffee brewing" and "cake plating", 2-5 leftover video frames of the "kitchen environment panorama" type can be randomly inserted. The number of insertions needs to be adjusted according to the content correlation between the two matching video segments. If the content correlation is strong, 1-2 fewer frames can be inserted; if the correlation is weak, 3-5 more frames can be inserted as a transition. At the same time, the insertion position of each leftover video frame needs to be recorded.

[0093] Furthermore, the obtained second adjusted synthesis result is input into the video coherence analyzer to output the coherence score. The video coherence analyzer is also based on a deep learning architecture, and is constructed using a combination model of "convolutional neural network (CNN) + long short-term memory network (LSTM)".

[0094] Specifically, the CNN module uses the lightweight MobilNetV2 architecture, with 3 convolutional layers, a kernel size of 3×3, and a stride of 1. It extracts visual features such as color and texture from video frames through a process of "convolution-batch normalization-ReLU activation", and finally outputs a 256-dimensional feature vector.

[0095] In addition, the LSTM module has one hidden layer with 128 hidden units. A dropout layer with a probability of 0.2 is set between the input layer and the hidden layer to prevent overfitting and to capture the temporal logic correlation between adjacent video frames, so as to ensure that the continuity of the scene transition can be recognized.

[0096] Before being put into use, the video coherence analyzer needs to be trained with samples to ensure analysis accuracy. Specifically, a set of sample videos covering different scenes and different cases of legacy frame insertion is first collected. Professionals then label each sample with a coherence level in the range of 0-1 based on dimensions such as the naturalness of color transitions and scene relevance, forming a set of sample coherence levels.

[0097] Furthermore, keyframes are extracted from the synthesized video samples (8 frames per 3 seconds of video) and input into the analyzer model. The sample coherence set is used as the supervision signal, and the Adam optimizer (initial learning rate 1e) is selected. -4 The cross-entropy loss function is used to train the model with a batch size of 32 and 50 training rounds. The model error is evaluated using a validation set every 10 training rounds until the model's prediction error for coherence is less than 0.05 and the accuracy does not improve for 3 consecutive rounds of validation. The model is then considered to have converged, ensuring that it can accurately output the coherence value of the second adjusted synthesis result.

[0098] For example, if the second adjustment synthesis result is "coffee brewing matching video segment + 2 leftover frames of kitchen panorama + cake plating matching video segment", after inputting it into the trained video coherence analyzer, the CNN module will extract the brown color features of the coffee brewing scene, the scene texture features of the kitchen panorama, and the warm color features of the cake plating, and output the corresponding 256-dimensional feature vector; the LSTM module will capture the inter-frame temporal correlation of "coffee brewing scene - kitchen panorama - cake plating scene", judge that the three scenes have strong correlation and natural color transition, and finally output a coherence value of 0.85, indicating that the synthesized video has good picture coherence under this insertion method.

[0099] Furthermore, after obtaining the coherence, the segmentation and synthesis quality coefficient is calculated using a preset formula, based on the comprehensive matching degree calculated in the previous stage.

[0100] The segmentation and synthesis quality coefficient must take into account both the continuity of the image and the semantic matching degree of the audio and video. The calculation formula can be expressed as "segmentation and synthesis quality coefficient = continuity weight × continuity + comprehensive matching weight × comprehensive matching degree" (the weight can be adjusted according to actual needs. If continuity is emphasized, the continuity weight is set to 0.6 and the comprehensive matching weight is set to 0.4).

[0101] For example, if the coherence is 0.8 and the overall matching is 0.85, substituting into the calculation formula, the segmentation and synthesis quality coefficient is 0.8×0.6+0.85×0.4=0.82; if the coherence is only 0.5 and the overall matching is 0.9, the segmentation and synthesis quality coefficient is 0.5×0.6+0.9×0.4=0.66, which intuitively reflects that although the semantic matching of the current synthesis result is good, the coherence of the image is insufficient and needs to be optimized.

[0102] Finally, the insertion order of multiple legacy video frames is optimized based on the calculated segmentation and synthesis quality coefficients to obtain an optimized insertion order.

[0103] The method provided in this application embodiment, "optimizing the insertion order of multiple legacy video frames to obtain an optimized insertion order", includes: Get the preset number of optimizations; Based on the segmentation and synthesis quality coefficient, the preset number of optimizations is adjusted, calculated, and rounded to obtain the number of optimizations. Randomly insert multiple legacy video frames into multiple matching video segments within the first segmentation and synthesis result, analyze the coherence, and perform iterative optimization until the optimization converges. The insertion order with the highest coherence during the optimization process is selected as the optimized insertion order.

[0104] In this embodiment of the application, in order to avoid poor continuity of the synthesized video due to the disordered insertion order of the remaining video frames, and to ensure that the optimization process is efficient and accurate, it is necessary to dynamically adjust the optimization intensity in combination with the segmentation and synthesis quality coefficient, and find the optimal insertion order through iterative optimization in order to improve the continuity and overall quality of the final audio and video synthesis result.

[0105] Specifically, the first step is to determine the preset number of optimizations. The preset number of optimizations needs to be set based on the actual needs of the audio and video synthesis scenario. For example, for everyday short video synthesis (usually 1-5 minutes long with a small number of remaining video frames), the preset number of optimizations can be set to 20 times; for professional film and television clip synthesis (longer in length, with a large number of remaining video frames and high requirements for the correlation between scenes), the preset number of optimizations can be set to 50 times to ensure that the initial optimization intensity can cover the basic scenario requirements and avoid insufficient optimization due to too few times or wasted computing power due to too many times.

[0106] Furthermore, after obtaining the preset number of optimizations, the preset number of optimizations is adjusted and rounded according to the obtained segmentation and synthesis quality coefficients to obtain the number of optimizations.

[0107] Specifically, if the segmentation-synthesis quality coefficient is ≥0.8, it indicates that the current synthesis result has a good foundation and does not require excessive iterative optimization. The preset number of optimizations can be multiplied by 0.5 and rounded to reduce optimization costs. If the segmentation-synthesis quality coefficient is between 0.5 and 0.8, it indicates that the current result needs moderate optimization, and the preset number of optimizations can be kept unchanged. If the segmentation-synthesis quality coefficient is <0.5, it indicates that the current result has obvious problems and the optimization intensity needs to be increased. The preset number of optimizations should be multiplied by 1.5 and rounded to ensure that a better order can be found through sufficient iterations. After adjustment, the number of optimizations should be rounded to ensure that it is an integer to conform to the actual operation logic.

[0108] Furthermore, multiple legacy video frames are randomly inserted between multiple matching video segments in the first segmentation and synthesis result, and the coherence is analyzed. The insertion position of the legacy video frames is continuously adjusted based on the change in coherence, and iterative optimization is performed until the optimization converges, so as to gradually find an insertion method that can improve the coherence of the synthesized video.

[0109] Specifically, in each iteration, the insertion positions of some remaining video frames need to be randomly adjusted first, and then the adjusted synthesis result is input into the video coherence analyzer to recalculate and record the coherence. Iterative optimization needs to continue until the optimization convergence condition is met, that is, after three consecutive iterations, the coherence value fluctuation is less than 0.02, indicating that continuing the iteration can no longer significantly improve the coherence. At this time, the optimization is stopped to avoid invalid iterations.

[0110] Finally, the insertion order with the highest coherence during the optimization process is selected as the optimized insertion order. That is, in all iterations, each adjustment corresponds to a coherence value, and the coherence and corresponding insertion position information of each iteration need to be recorded in real time. When the optimization converges, the insertion positions with the highest coherence value are selected from all records and determined as the final optimized insertion order.

[0111] For example, in 20 iterations, the coherence of the 15th iteration is 0.89 (the highest among all iterations), and the corresponding insertion order is "outdoor landscape frames inserted between 'coffee brewing - cake plating' and kitchen close-up frames inserted between 'cake plating - fruit decoration'". This order is then used as the optimized insertion order to ensure that the coherence of the scene can be maximized when inserting legacy frames in the future.

[0112] S140: According to the optimized insertion order, insert the multiple remaining video frames and multiple silent frames between multiple matching video segments in the first segmentation and synthesis result to obtain the audio-visual segmentation and synthesis result.

[0113] In this embodiment of the application, in order to avoid visual or audio gaps in the synthesized content and to ensure that the audio and video are completely synchronized on the timeline, the remaining video frames and silent frames need to be precisely inserted into the preliminary synthesis framework according to the optimized insertion order in order to generate a complete, coherent and semantically matched audio and video segmentation and synthesis result.

[0114] Specifically, firstly, according to the obtained optimized insertion order, multiple legacy video frames and multiple silent frames are inserted between multiple matching video segments in the first segmentation and synthesis result to fill the content gaps between the matching video segments and make the transition of the synthesized video screen more natural.

[0115] During the insertion process, it is crucial to ensure that the timelines of multiple legacy video frames and multiple silent frames are matched, meaning that the total playback duration of the legacy video frames is completely consistent with the total audio duration of the corresponding silent frames, in order to improve the overall coherence and rationality of the final audio-visual segmentation and synthesis results.

[0116] Finally, after inserting all remaining video frames and silent frames, a complete audio-visual segmentation and synthesis result is formed. This result includes the audio and video content from the initial semantic matching, and fills in content gaps by optimizing the inserted remaining and silent frames. It also avoids gaps between the video and audio, ultimately achieving multi-dimensional alignment of audio and video from semantic to temporal perspectives, meeting the requirements for completeness and coherence of the synthesized content in practical applications.

[0117] Step S140 in the method provided in this application embodiment includes: According to the optimized insertion order, the multiple legacy video frames and multiple silent frames are inserted between multiple matching video segments in the first segmentation and synthesis result to obtain the audio-visual segmentation and synthesis result, wherein the multiple legacy video frames and multiple silent frames are time-axis matched.

[0118] In this embodiment of the application, in order to avoid the problem of image discontinuity or audio gap misalignment in the final audio and video synthesis result, the remaining video frames and silent frames are inserted into the matching video segments according to the determined optimized insertion order, so as to generate a complete, coherent and synchronized audio and video segmentation and synthesis result.

[0119] Specifically, the optimized insertion order is an insertion scheme that maximizes the continuity of the video after multiple rounds of iterative optimization. It must be strictly followed to ensure that the insertion is performed at the junction of different matching video segments within the first segmentation and synthesis result. Insertion should not be performed within a single matching video segment to avoid disrupting the semantic integrity of the original matching audio and video.

[0120] For example, if the optimized insertion order specifies "insert 2 legacy video frames and corresponding silent frames between the matching video segments of 'pet playing' and 'outdoor running', and insert 3 legacy video frames and corresponding silent frames between the matching video segments of 'outdoor running' and 'sunset scenery'", then the insertion operation must be performed exactly according to this position requirement.

[0121] During the insertion process, it is crucial to ensure the timeline matching of multiple legacy video frames and multiple silent frames. The core of timeline matching is that their durations must be completely identical; that is, the total playback duration of the legacy video frame inserted at a certain connection point must be precisely equal to the total audio duration of the corresponding inserted silent frame.

[0122] For example, if the inserted legacy video frames are 2 frames, calculated at a video frame rate of 30fps, the duration of a single frame is approximately 0.033 (1 / 30≈0.033) seconds, and the total duration of 2 frames is approximately 0.066 seconds. Therefore, the corresponding inserted silent frames must also be 2 frames, with a total duration of approximately 0.066 seconds. If the inserted legacy video frames are 3 frames, with a total duration of approximately 0.099 seconds, then the silent frames must also be 3 frames, keeping the total duration consistent.

[0123] By following the above matching requirements, misalignment situations such as "the remaining video frame has finished playing but the audio of the silent frame is still continuing" or "the audio of the silent frame has ended but the remaining video frame has not yet switched" can be avoided, thus ensuring that the audio and video are synchronized in the time dimension.

[0124] Furthermore, after completing all insertion operations, the integrity of the overall composite content must be verified to ensure that all remaining video frames and silent frames have been accurately inserted into their corresponding positions in the optimized insertion order, without any omissions, duplicate insertions, or incorrect insertions.

[0125] Specifically, the verification includes two aspects: first, checking whether all remaining video frames and silent frames have been inserted in the optimized order, with no omissions or duplicate insertions; second, playing the complete composite result to confirm that the transitions between the frames are natural and without obvious breaks, and that the audio part has no noise or premature or delayed endings during the silent frame period.

[0126] For example, when playing to the transition between "pets playing" and "outdoor running", it is necessary to confirm that the two remaining video frames transition smoothly and that there is no audio output during the corresponding silent frame period, and then smoothly connect to the audio content of "outdoor running".

[0127] Finally, through the steps described above—strictly adhering to the optimized insertion order, ensuring timeline matching, and verifying integrity—a complete audio-visual segmentation and synthesis result was formed. This result not only preserves the semantic coherence of the previously matched video and audio segments but also fills content gaps by appropriately inserting residual video frames and silent frames. Simultaneously, it achieves synchronization between the audio and video timelines, resolving the issues of image fragmentation and audio misalignment that easily occur in traditional editing and synthesis, effectively improving the overall quality and rationality of audio-visual synthesis.

[0128] The embodiments of this application, through the specific implementation methods described above, achieve the following technical effects: This application proposes an AI-driven audio and video content semantic segmentation method. First, target audio and target video of consistent duration are acquired. The target audio is segmented into frames to separate silent frames. Effective audio content is divided into multiple audio segments according to a preset time window. Each audio segment is input into a mature audio semantic recognizer trained on samples, and the output is the audio semantic information of each segment. Next, the target video is randomly segmented into video segments according to the same preset time window. These video segments are input into a video content recognizer built based on deep learning to obtain screen content descriptions. The similarity between the video content and the semantic information of each audio segment is calculated. Multiple matching video segments are selected, and the maximum similarity of each match is recorded. The overall matching degree is calculated by averaging the results. Unmatched legacy video frames are retained. Then, timeline matching is performed based on the matching video segments and audio segments to generate the first segmentation and synthesis result. The legacy video frames are randomly inserted between the matching video segments to obtain the second adjusted synthesis result. The video coherence analyzer outputs the coherence degree, and the segmentation and synthesis quality coefficient is calculated based on the comprehensive matching degree. The number of optimizations is adjusted according to the segmentation and synthesis quality coefficient. The insertion position of the legacy video frames is iteratively adjusted and the coherence degree is analyzed until convergence. The optimized insertion order with the highest coherence degree is selected. Finally, according to the optimized insertion order, the legacy video frames and silent frames are inserted between the matching video segments of the first segmentation and synthesis result according to the timeline matching requirements. After completing the integrity verification, the audio and video segmentation and synthesis result is obtained.

[0129] The method provided in this application, through the technical solution of "audio and video preprocessing and semantic recognition - video segment matching and comprehensive matching degree calculation - legacy frame insertion optimization and coherence analysis - audio and video synchronous synthesis and verification", solves the problems of low semantic matching degree, poor picture coherence, and audio and video time axis misalignment caused by neglecting semantic association in traditional audio and video editing and synthesis. It provides accurate and efficient audio and video semantic segmentation and synthesis technology support for short video production, film and television clip editing and other scenarios, and effectively improves the rationality and completeness of the synthesis results.

[0130] Example 2, as shown in the appendix Figure 2 As shown, based on the inventive concept of an AI-driven audio and video content semantic segmentation method provided in Embodiment 1, this application also provides an AI-driven audio and video content semantic segmentation system, specifically including: The audio and video acquisition and segmentation module 01 is used to acquire target audio and target video, segment silent frames within the target audio, obtain target audio content, divide it into multiple audio segments, and identify multiple audio semantic information of multiple audio segments. The video segmentation and matching module 02 is used to segment the target video into video segments, identify the content of the video segments, analyze the similarity with the multiple audio semantic information, filter to obtain multiple matching video segments and multiple residual video frames, and process to obtain the comprehensive matching degree. The legacy frame order optimization module 03 is used to generate a first segmentation and synthesis result based on multiple matching video segments and multiple audio segments, randomly insert multiple legacy video frames between multiple matching video segments in the first segmentation and synthesis result, perform coherence analysis to obtain coherence, and combine the comprehensive matching degree to obtain an optimized insertion order; The audio-visual segmentation and synthesis module 04 is used to insert the multiple legacy video frames and multiple silent frames into the multiple matching video segments in the first segmentation and synthesis result according to the optimized insertion order, so as to obtain the audio-visual segmentation and synthesis result.

[0131] In one embodiment, the audio / video acquisition and segmentation module 01 is further configured to: The target audio and target video to be segmented and synthesized are obtained, wherein the target audio and target video have the same duration; the target audio is divided into frames, and the audio intensity of each frame is extracted. Frames with audio intensity less than the audio intensity threshold are designated as silent frames, and all frames other than silent frames are designated as target audio content; the target audio content is divided according to a preset time window to obtain multiple audio segments; the multiple audio segments are input into an audio semantic recognizer, and multiple audio semantic information is output.

[0132] Furthermore, the audio and video acquisition and segmentation module 01 also includes: A set of sample audio segments is collected, and the audio semantics within each sample audio segment are labeled to obtain a set of sample audio semantic information. An audio semantic recognizer is constructed based on an automatic speech recognition network in machine learning. The audio semantic recognizer is trained using the set of sample audio segments and the set of sample audio semantic information until convergence. The multiple audio segments are respectively input into the input audio semantic recognizer, and multiple audio semantic information is output.

[0133] In one embodiment, the video segmentation and matching module 02 is further configured to: The target video is randomly divided into video segments according to a preset time window to obtain a first video segment. The first video segment is input into a video content recognizer, which recognizes and outputs video content. The video content recognizer is built based on deep learning and trained using a set of sample video segments and a set of sample video content. The similarity between the video content and multiple audio and speech information is calculated, and the audio segment corresponding to the maximum similarity is retained and matched with the first video segment to obtain a first matched video segment. The video segments are randomly divided again, and similarity calculation and matching are performed to obtain multiple matched video segments and multiple maximum similarities. The average of the multiple maximum similarities is calculated to obtain the comprehensive matching degree.

[0134] In one embodiment, the legacy frame order optimization module 03 is further configured to: A first segmentation and synthesis result is generated based on multiple matching video segments and multiple audio segments, wherein the multiple matching video segments and multiple audio segments are time-axis matched; multiple residual video frames are randomly inserted between multiple matching video segments in the first segmentation and synthesis result to obtain a second adjusted synthesis result; the second adjusted synthesis result is input into a video coherence analyzer, and the output is a coherence score, wherein the video coherence analyzer is trained using a sample synthesized video set and a labeled sample coherence score set, and the coherence score is greater than or equal to 0 and less than or equal to 1; a segmentation and synthesis quality coefficient is calculated based on the coherence score and the overall matching score; and the insertion order of multiple residual video frames is optimized based on the segmentation and synthesis quality coefficient to obtain an optimized insertion order.

[0135] Furthermore, the legacy frame order optimization module 03 also includes: Obtain the preset number of optimizations; adjust and round the preset number of optimizations according to the segmentation and synthesis quality coefficients to obtain the number of optimizations; randomly insert multiple residual video frames between multiple matching video segments in the first segmentation and synthesis result, analyze the coherence, and perform iterative optimization until the optimization converges; The insertion order with the highest coherence during the optimization process is selected as the optimized insertion order.

[0136] In one embodiment, the audio-visual segmentation and synthesis module 04 is further configured to: According to the optimized insertion order, the multiple legacy video frames and multiple silent frames are inserted between multiple matching video segments in the first segmentation and synthesis result to obtain the audio-visual segmentation and synthesis result, wherein the multiple legacy video frames and multiple silent frames are time-axis matched.

[0137] It should be noted that the order of the embodiments described above is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, the above description focuses on specific embodiments of this specification. Additionally, the processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired results. In some implementations, multitasking and parallel processing are possible or may be advantageous.

[0138] The above description is only a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

[0139] This specification and accompanying drawings are merely illustrative examples of this application and are intended to cover any and all modifications, variations, combinations, or equivalents within the scope of this application. Clearly, those skilled in the art can make various alterations and modifications to this application without departing from its scope. Therefore, if such modifications and variations fall within the scope of this application and its equivalents, this application intends to include such modifications and variations.

Claims

1. An AI-driven audio-video content semantic segmentation method, characterized in that, The method comprises: obtaining target audio and target video, dividing the silent frames in the target audio, obtaining the target audio content, and dividing into multiple audio segments, identifying multiple audio semantic information of multiple audio segments; video segment division is performed on the target video, the video segment content is identified, the similarity with the multiple audio semantic information is analyzed, multiple matching video segments and multiple remaining video frames are screened and obtained, and a comprehensive matching degree is obtained by processing; based on the multiple matching video segments and the multiple audio segments, a first segmentation synthesis result is generated, the multiple remaining video frames are randomly inserted between the multiple matching video segments in the first segmentation synthesis result, the continuity is analyzed to obtain the continuity, the comprehensive matching degree is combined, and an optimized insertion order is obtained; according to the optimized insertion order, the multiple remaining video frames and the multiple silent frames are inserted between the multiple matching video segments in the first segmentation synthesis result, and an audio-video segmentation synthesis result is obtained.

2. The AI-driven audio-video content semantic segmentation method of claim 1, wherein, obtaining target audio and target video, dividing the silent frames in the target audio, obtaining the target audio content, and dividing into multiple audio segments, identifying multiple audio semantic information of multiple audio segments, comprising: obtaining target audio and target video to be segmented and synthesized, wherein the time length of the target audio and the target video is the same; frame division is performed on the target audio, and the audio intensity of each frame of audio is extracted, the frames with audio intensity less than the audio intensity threshold are taken as silent frames, and all frames other than the silent frames are taken as target audio content; according to a preset time window, the target audio content is divided to obtain multiple audio segments; respectively inputting the multiple audio segments into an audio semantic recognizer to output multiple audio semantic information. 3.The AI-driven audio-video content semantic segmentation method of claim 2, wherein, respectively inputting the multiple audio segments into an audio semantic recognizer to output multiple audio semantic information, comprising: collecting a sample audio segment set, labeling the audio semantics in each sample audio segment to obtain a sample audio semantic information set; based on an automatic speech recognition network in machine learning, an audio semantic recognizer is constructed; using the sample audio segment set and the sample audio semantic information set, the audio semantic recognizer is trained until convergence; the multiple audio segments are respectively input into the input audio semantic recognizer, and multiple audio semantic information is output.

4. The AI-driven audio-video content semantic segmentation method of claim 1, wherein, video segment division is performed on the target video, the video segment content is identified, the similarity with the multiple audio semantic information is analyzed, multiple matching video segments and multiple remaining video frames are screened and obtained, and a comprehensive matching degree is obtained by processing, comprising: randomly dividing the target video into video segments according to a preset time window to obtain a first video segment; the first video segment is input into a video content recognizer to identify and output video content, wherein the video content recognizer is constructed based on deep learning and trained using a sample video segment set and a sample video content set; the similarity of the video content and the multiple audio semantic information is calculated respectively, and the audio segment corresponding to the maximum similarity is retained, which is matched with the first video segment to obtain a first matching video segment; continue to randomly divide the video segments, calculate the similarity and match to obtain multiple matching video segments and multiple maximum similarities; the mean value of the multiple maximum similarities is calculated to obtain a comprehensive matching degree.

5. The AI-driven audio-video content semantic segmentation method of claim 1, wherein, generating a first segmentation synthesis result based on the plurality of matched video segments and the plurality of audio segments, randomly inserting the plurality of remaining video frames between the plurality of matched video segments in the first segmentation synthesis result, performing continuity analysis to obtain continuity, combining the comprehensive matching degree to perform insertion order optimization to obtain an optimized insertion order, comprising: generating a first segmentation synthesis result based on the plurality of matched video segments and the plurality of audio segments, wherein the plurality of matched video segments and the plurality of audio segments are matched in the time axis; randomly inserting the plurality of remaining video frames between the plurality of matched video segments in the first segmentation synthesis result to obtain a second adjusted synthesis result; inputting the second adjusted synthesis result into a video continuity analyzer to output continuity, wherein the video continuity analyzer is trained using a sample synthesized video set and a labeled sample continuity set, and the continuity is greater than or equal to 0 and less than or equal to 1; calculating a segmentation synthesis quality coefficient based on the continuity and the comprehensive matching degree; optimizing the insertion order of the plurality of remaining video frames based on the segmentation synthesis quality coefficient to obtain an optimized insertion order.

6. The AI-driven audio-video content semantic segmentation method of claim 1, wherein, optimizing the insertion order of the plurality of remaining video frames based on the segmentation synthesis quality coefficient to obtain an optimized insertion order, comprising: obtaining a preset optimization number; adjusting and rounding the segmentation synthesis quality coefficient to obtain the optimization number; randomly inserting the plurality of remaining video frames between the plurality of matched video segments in the first segmentation synthesis result, and iteratively optimizing until convergence is achieved; obtaining the insertion order with the maximum continuity during optimization as the optimized insertion order.

7. The AI-driven audio-video content semantic segmentation method of claim 1, wherein, inserting the plurality of remaining video frames and the plurality of silent frames between the plurality of matched video segments in the first segmentation synthesis result according to the optimized insertion order to obtain an audio-visual segmentation synthesis result, comprising: inserting the plurality of remaining video frames and the plurality of silent frames between the plurality of matched video segments in the first segmentation synthesis result according to the optimized insertion order to obtain an audio-visual segmentation synthesis result, wherein the plurality of remaining video frames and the plurality of silent frames are matched in the time axis.

8. An AI-driven audio-video content semantic segmentation system, characterized in that, The system is used to perform the AI-driven audio-visual content semantic segmentation method of any one of claims 1-7, and the system comprises: an audio-video acquisition and division module for acquiring target audio and target video, dividing silent frames in the target audio to obtain target audio content, and dividing the target audio into a plurality of audio segments to identify a plurality of audio semantic information of the plurality of audio segments; a video division and matching module for dividing the target video into video segments, identifying video segment content, analyzing similarity with the plurality of audio semantic information, screening a plurality of matched video segments and a plurality of remaining video frames, and processing a comprehensive matching degree; a remaining frame order optimization module for generating a first segmentation synthesis result based on the plurality of matched video segments and the plurality of audio segments, randomly inserting the plurality of remaining video frames between the plurality of matched video segments in the first segmentation synthesis result, performing continuity analysis to obtain continuity, and combining the comprehensive matching degree to obtain an optimized insertion order; An audio-visual segmentation and synthesis module is configured to insert the plurality of remaining video frames and the plurality of silent frames between the plurality of matched video segments in the first segmentation and synthesis result according to the optimized insertion order to obtain an audio-visual segmentation and synthesis result.

Images