Audio and video joint coding and decoding method and system based on generative artificial intelligence

Abstract

The invention discloses an audio and video joint coding and decoding method based on generative artificial intelligence. The coding comprises the following steps: extracting various modal features from an audio signal and a video; fusion is carried out in cross-modal attention; task identification: firstly identifying a current task type, determining the task type, and analyzing specific requirements of the task by a system; dynamic self-adaptive weight distribution is carried out; and fusing the features, namely fusing the multi-modal features distributed with different weights. The audio and video joint coding method has the advantages that audio and video feature extraction, video feature extraction and AI-based generation technologies are applied to audio and video joint coding for the first time, and the problems that when video coding and decoding are conducted through a traditional method, compression efficiency is low, and the pursuit of higher quality cannot be met are solved. Meanwhile, on the basis of a generation type coding and decoding mode, the style and quality can be flexibly adjusted according to the personal requirements of the user.

Description

Technical Field

[0001] The present invention relates to the field of audio and video generation, and in particular to audio and video joint encoding and decoding technology based on generative artificial intelligence. Background Art

[0002] With the rapid development of multimedia technology, the demand for the generation and processing of audio and video content is increasing. The development of new audio and video generation technologies has provided new solutions for audio and video encoding and decoding. Especially in application scenarios that require high-quality audio and video content generation and real-time processing, these new technologies have shown great potential. At present, with the rapid development of artificial intelligence technology, speech, video recognition and generation technologies based on artificial intelligence have become very mature, laying a solid technical foundation for the joint audio and video encoding of generative artificial intelligence. By utilizing generative artificial intelligence technology, efficient and accurate audio and video encoding and decoding can be achieved to meet higher requirements.

[0003] In the process of identifying and generating audio and video content, AI generation technology can maintain high accuracy in multi-language, multi-emotion and complex environments. Through multimodal data fusion technology, high-level features of audio and video are extracted and fused to generate comprehensive feature vectors to ensure the visual and auditory consistency and coherence of the generated audio and video content. These technological advances not only improve the efficiency of audio and video encoding and the quality of generation, but also provide new technical means for the generation and editing of multimedia content, and have broad application prospects.

[0004] Generative AI technology can generate high-quality audio and video content by learning a large amount of audio and video data. Compared with traditional encoding technology, generative AI technology has higher efficiency and accuracy in processing complex scenes and multimodal data. Through multimodal data fusion technology, generative AI can extract and fuse high-level features of audio and video to generate comprehensive feature vectors, thereby achieving efficient audio and video encoding and decoding.

[0005] With the further development of generative technology, the demand for customized audio and video generation systems is also increasing. Modern users want to be able to generate high-quality, personalized audio and video content based on their personal preferences and needs. This demand is not limited to entertainment and social media, but has also expanded to multiple fields such as education, medical care, advertising, and corporate training. In the entertainment industry, users want to be able to generate customized music videos, movie clips, and game scenes to meet personalized viewing and interactive experiences. In the field of education, teachers and students want to be able to generate customized teaching videos and interactive courseware to improve learning outcomes and engagement. In the medical field, doctors and patients want to be able to generate customized medical images and surgical simulations to improve the accuracy of diagnosis and treatment. In the field of advertising and corporate training, companies want to be able to generate customized promotional videos and training materials to improve brand influence and employee skills. The development of generative technology has made these customized needs feasible. By utilizing generative artificial intelligence technology, audio and video content that meets user needs can be generated efficiently and accurately. This not only improves the efficiency of audio and video content generation and processing, but also provides new technical means for the fusion and analysis of multimodal data to ensure the consistency and coherence of the generated audio and video content in terms of vision and hearing. In short, with the continuous advancement of generation technology, people's demand for customized audio and video generation systems will become increasingly strong, which will promote the further development and application of related technologies and bring more innovations and possibilities.

[0006] However, the current audio and video encoding and decoding technologies are facing severe challenges in today's digital media explosion. Traditional audio and video encoding methods usually process audio and video separately, and this independent encoding method can no longer meet people's growing and diverse needs. Summary of the invention

[0007] The technical problem to be solved by the present invention is how to improve the efficiency and quality of audio and video transmission.

[0008] The present invention solves the above technical problems through the following technical means: a method for joint audio and video encoding based on generative artificial intelligence, comprising the following steps:

[0009] S9, extract various modal features from audio signals and videos;

[0010] S10. Fusion in cross-modal attention, including:

[0011] S101, first, input the extracted video features into the cross-modal attention;

[0012] S102, inputting the audio features into the cross-modal attention through the auxiliary residual auxiliary network, and analyzing the features of different modalities through the cross-modal attention mechanism;

[0013] S103, the cross-modal attention mechanism of audio features is the same as S101 and S102; the features of different modalities are connected with the output results of cross-modal attention;

[0014] S11, task identification, first identify the current task type, and after determining the task type, the system will analyze the specific requirements of the task;

[0015] S12, dynamic adaptive weight allocation;

[0016] S13, feature fusion, fusing multimodal features assigned with different weights.

[0017] As a further optimized technical solution, the step S12, dynamic adaptive weight allocation, specifically includes:

[0018] S121, Feature Importance Assessment: Based on the task requirements, the system assesses the importance of audio and video features, and determines the importance weight of each feature by calculating the contribution of each feature in the current task;

[0019] S122, weight adjustment strategy: the system dynamically adjusts the weights of audio and video features according to the feature importance evaluation results;

[0020] S123, Adaptive weight allocation: During the execution of each task, the system monitors the progress of the task and the performance of the features in real time and dynamically adjusts the weight allocation;

[0021] S124, modality data loss processing: when the data of a certain modality is lost or does not exist, the system generates the data of the missing modality using the data of the existing modality.

[0022] As a further optimization technical solution, various modal features are extracted from the audio signal including:

[0023] S1. Sentiment analysis, including:

[0024] Audio data preprocessing: including noise reduction and normalization; Emotional state recognition: using the sentiment analysis algorithm to recognize the emotional state of the preprocessed audio data and classify the emotional state in the audio;

[0025] S2. Sound event detection, including:

[0026] Sound event recognition: Detect sound events in audio data and identify various sound events in the audio;

[0027] Event annotation: Detailed annotation of detected sound events, including the start time, duration and type of the event;

[0028] S3, timbre recognition, including:

[0029] Voice judgment: through voice recognition technology, determine whether the audio clips are of the same voice. If different voices are detected, send the original voice clips of the predetermined time;

[0030] Timbre consistency processing: If the timbre is consistent, speech recognition is performed to extract the speech content;

[0031] Time marking: marking the time of speech;

[0032] S4, audio feature extraction, including feature parameter extraction and feature integration, integrating time annotation, sound events and sentiment analysis into speech features to form a comprehensive audio feature dataset;

[0033] S5: Data transmission: the original speech segment and the audio feature data set after feature fusion are transmitted.

[0034] As a further optimization technical solution, various modal features are extracted from the video signal including:

[0035] S6, scene switching detection

[0036] S61, for the Nth frame extracted from the video, the scene discriminator compares it with the previous frame, analyzes the contents of the Nth frame and the N-1th frame, and determines whether there is a scene switch;

[0037] S62: If a scene switch is detected, the frame is set as a key frame, the Nth frame is marked as a key frame, and the frame number is initialized to 1;

[0038] S63, extracting and fusing features of the key frame to obtain features of the frame, and extracting various video features from the key frame; preprocessing the extracted features to generate processed features;

[0039] S7. Set keyframes regularly

[0040] S71, if no scene switching is detected, the video encoder increases the frame number by 1 and increases the current frame number N by 1;

[0041] S72, multiple check: determine whether the current frame number is an integer multiple of GOP; if the frame number is an integer multiple of GOP, set the frame as a key frame and reset the frame number to 1;

[0042] S73, performing feature extraction and feature preprocessing on the key frames, extracting various video features from the key frames, preprocessing the extracted features, and generating processed features;

[0043] S8, inter-frame motion vector calculation

[0044] S81, when the number of frames is not an integer multiple of the GOP, the video feature extractor uses an optical flow method to calculate an inter-frame motion vector between the frame and the previous frame;

[0045] S82. Generate features of the Nth frame by combining features of the previous frame, and generate features of the Nth frame by using features of the previous frame and the calculated motion vector.

[0046] The present invention also provides an audio and video joint decoding method based on generative artificial intelligence, comprising the following steps:

[0047] S15. Feature decoding and preprocessing

[0048] The receiving end first needs to preprocess the multimodal features transmitted from the sending end after being processed by the audio and video joint encoding method described in any one of claims 1 to 4. The preprocessing steps include: normalization, denoising, interpolation, and feature alignment and synchronization. The feature alignment includes time alignment and space alignment. Then, the preprocessed multimodal features are decoded.

[0049] S16, joint decoding

[0050] The multimodal features of the audio and video decoded in step S15 are input into the joint decoding module, which processes the decoded audio and video features through an AI-driven multimodal deep learning model. This module combines the audio and video generation modules to generate the final multimodal content through feature fusion and deep network: Video generation module: an encoder-decoder architecture based on deep learning, used to generate video frames from video features. This module uses AI video generation technology, especially the Transformer encoder, to extract key features of the video from the input features to help generate more natural and smooth visual content. Audio generation module: The AI ​​timbre encoder is used to extract and fuse the weighted multimodal features and the timbre features of the original audio. The fused features are converted into a semantic token sequence through an autoregressive Transformer to gradually generate audio content with high consistency and clarity. The final audio passes through the AI ​​synthesis algorithm HiFNet to ensure that the generated audio meets the expected sound quality and semantics;

[0051] S17. Merge audio and video, and use AI-driven synchronization technology to ensure that audio and video are aligned on the timeline.

[0052] As a further optimized technical solution, the step S16, joint decoding, specifically includes:

[0053] S161, the video decoding module first performs word embedding, converts the input video features into vector representation, and then performs position encoding on the vector;

[0054] S162, audio decoding module is based on AI-driven audio generation technology, which converts the fused features into semantic tokens and generates the final audio content. Including: generating timbre features, fusing multimodal features with the timbre features of the original audio through the AI ​​timbre encoder to ensure the consistency and richness of audio generation; the autoregressive Transformer is used to convert the input features into a semantic token sequence. The AI ​​model ensures the coherence and high quality of the audio content through a recursive generation process at this stage; the semantic token sequence generated by the autoregressive Transformer is input into the stream matching model, and the stream matching model is used to convert the generated semantic token sequence into a Mel spectrum map; HiFNet is based on AI spectrum conversion technology to convert the Mel spectrum map into a high-fidelity audio waveform. The model uses an optimization algorithm for the spectral feature and waveform feature generation process to ensure that the output audio meets the effect of natural sound.

[0055] As a further optimized technical solution, the step S17, after merging the audio and video, further includes:

[0056] S18. Objective evaluation: The AI-generated content evaluation model is used to evaluate the quality of audio and video. The evaluation indicators include: peak signal-to-noise ratio. The higher the peak signal-to-noise ratio, the better the quality of the generated video; structural similarity index. The closer the structural similarity index value is to 1, the more similar the structure of the generated video is to the reference video; audio quality evaluation: Use the perceptual speech quality evaluation or speech transmission index indicators to evaluate the quality of the generated audio. If the objective evaluation result is qualified, the video is directly output. If it is unqualified, it enters the weight adjustment module to dynamically adjust the weight distribution of multimodal features until a qualified video is generated;

[0057] S19. Dynamic adjustment.

[0058] As a further optimized technical solution, the step S19, dynamic adjustment includes:

[0059] S191, weight adjustment module: The AI ​​model uses optimization algorithms (such as gradient descent or genetic algorithms) to dynamically adjust the weight distribution of audio and video features to improve content quality;

[0060] S192, regenerate content: After each weight adjustment, the new multimodal features regenerate the audio and video content through the joint decoding module, and the AI ​​model evaluates the output quality again until it meets the expected standards.

[0061] The present invention also provides an audio and video encoding and decoding method based on generative artificial intelligence, comprising the following steps:

[0062] Audio feature extraction;

[0063] Video feature extraction;

[0064] Audio and video joint coding, using any of the above audio and video joint coding methods to jointly encode the extracted audio features and video features;

[0065] Audio and video joint decoding, using any of the above audio and video joint decoding methods to jointly decode the received audio features and video features.

[0066] The present invention also provides an audio and video coding system based on generative artificial intelligence, comprising an audio feature extractor, a video feature extractor, an audio and video joint encoder, and an audio and video joint decoder, wherein the audio and video joint encoder adopts any of the audio and video joint encoding methods described above to jointly encode the extracted audio features and video features; the audio and video joint decoder adopts any of the audio and video joint decoding methods described above to jointly decode the received audio features and video features.

[0067] The advantages of the present invention are: the audio-visual feature extraction, video feature extraction and AI-based generation technology are applied for the first time in the joint encoding of audio and video, which is a new innovation in the audio codec architecture. It mainly solves the problem that the traditional method has low compression efficiency and cannot cope with the pursuit of higher quality when encoding and decoding videos. At the same time, based on the generative encoding and decoding method, the style and quality can be flexibly adjusted according to the personal needs of the user. Moreover, based on the generative audio and video joint encoding and decoding features, the characteristics of audio and video can be fully utilized to complement each other between different modal features, improve compression efficiency, and help to make the original video (decoding) more realistic. It opens up new possibilities for future multimedia applications, especially showing great potential in meeting personalized needs. Specifically including:

[0068] First, the audio feature extractor process optimization proposes an efficient audio feature extraction method that can comprehensively analyze and process audio data. Through steps such as sentiment analysis, sound event detection, timbre recognition, and speech recognition, the accurate extraction and processing of audio features are ensured. This method not only improves the accuracy and efficiency of audio feature extraction, but also provides strong technical support for audio data analysis in multiple fields.

[0069] Secondly, the video feature extractor process optimization improves the efficiency and quality of video encoding through technologies such as scene switching detection, key frame setting, and optical flow calculation. This method ensures the reasonable distribution of key frames during video encoding by regularly setting key frames, thereby improving the compression efficiency and decoding quality of the video. This method is not only applicable to conventional video encoding, but can also be applied to high frame rate and high resolution video encoding scenarios, and has broad application prospects and market value. Through optimization, the pressure on joint encoding and decoding and the generation pressure of generative artificial intelligence are reduced, thereby improving the overall encoding efficiency.

[0070] In addition, multi-task multimodal joint encoding proposes an innovative multimodal fusion framework that aims to dynamically balance and optimize the contribution of each modality. Through the cross-modal attention mechanism and residual auxiliary network, the prediction accuracy and robustness of the model are improved. When other modalities are lost or do not exist, the framework can generate other modalities through this modality to ensure the integrity and consistency of the data. The dynamic weighting strategy can adapt to different task scenarios and enhance the generalization ability of the model, so that it can perform well in different tasks. Specifically, when the data of a certain modality is missing, the system can use the data of the existing modality to generate the data of the missing modality to ensure the integrity of the overall data. In addition, the dynamic adaptive weight adjustment mechanism can automatically adjust the weights of each modality according to the needs of different tasks, so that the model can perform well in various tasks.

[0071] Finally, the audio and video joint decoder based on generative artificial intelligence is an advanced technology that achieves high-quality audio and video generation through decoding and fusion of multimodal features. First, the technology is able to decode the multimodal features transmitted from the sender and perform normalization, denoising and interpolation to improve data quality. Next, it ensures the alignment of audio and video features in time and space to avoid synchronization issues. The decoded audio and video multimodal features are then input into the joint decoding module. The video generation module generates video through steps such as word embedding, position encoding, and multi-head self-attention mechanism, while the audio generation module fuses the weighted multimodal features with the timbre features extracted by the original audio through the timbre encoder, and then generates high-quality audio through autoregressive Transformer stream matching and HiFNet. Finally, the technology ensures that audio and video are aligned on the timeline, using timestamps or other synchronization techniques. Through multiple iterations and optimizations, the quality of the generated content is continuously improved, and finally high-quality audio and video generation is achieved. This technology not only improves the quality and consistency of audio and video generation, but also provides an effective solution for the fusion of multimodal features. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1 It is an overall framework diagram of the audio and video fusion method based on generative artificial intelligence in an embodiment of the present invention;

[0073] Figure 2 is an audio coding flow chart in an embodiment of the present invention;

[0074] Figure 3 is a video encoding flow chart in an embodiment of the present invention;

[0075] Figure 4 is a diagram of an audio and video joint encoding module in an embodiment of the present invention;

[0076] Figure 5It is a diagram of an audio and video joint decoding module in an embodiment of the present invention. DETAILED DESCRIPTION

[0077] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in combination with the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0078] See also Figure 1 , the present invention relates to audio and video coding technology, aiming to significantly improve the efficiency and quality of data processing. By optimizing the audio feature extractor process, the video feature extractor process, the multi-task multi-modal joint coding and the design of the audio-video joint decoder, comprehensive technical support is provided for the analysis and processing of multimedia data. Specifically, the audio feature extractor process optimization proposes an efficient audio feature extraction method that can comprehensively analyze and process audio data; the video feature extractor process optimization improves the efficiency and quality of video coding through techniques such as scene switching detection, key frame setting and optical flow calculation; the audio-video joint coding proposes an innovative multi-modal fusion framework that aims to dynamically balance and optimize the contribution of each modality; the audio-video joint decoder ensures the synchronization and consistency of audio and video signals through feature fusion and time alignment technology.

[0079] The audio feature extractor process includes: first, sentiment analysis is performed on the audio to identify and classify the emotional state in the audio; then, sound event detection is performed on the audio, and the detected events are annotated in detail for subsequent processing; then, the timbre recognition technology is used to determine whether the audio clip has the same timbre. If different timbres are detected, a 3-second original voice clip is sent so that the timbre can be restored according to the original voice later. If the timbre is consistent, voice recognition is performed to extract the voice content. On this basis, the voice is time-annotated to ensure that the time information of each voice clip is accurate; then, voice features are extracted, including multiple parameters such as pitch, frequency, and rhythm, and the time annotation, sound events, and sentiment analysis results are integrated into the voice features to form a comprehensive audio feature data set; finally, multimodal feature processing is performed to comprehensively analyze the audio features to provide more accurate and comprehensive audio data analysis results. This method not only improves the accuracy and efficiency of audio feature extraction, but also provides strong technical support for audio data analysis in multiple fields.

[0080] See also Figure 2 ,The specific steps of audio feature extractor process optimization include:

[0081] S1. Sentiment analysis, including:

[0082] Audio data preprocessing: including noise reduction, normalization and other operations to ensure the quality of audio data;

[0083] Emotional state recognition: Use the sentiment analysis algorithm to recognize the emotional state of the preprocessed audio data and classify the emotional state in the audio, such as happiness, sadness, anger, etc.

[0084] S2. Sound event detection, specifically including:

[0085] Sound event recognition: Detect sound events in audio data and identify various sound events in the audio, such as clapping, laughing, crying, etc.

[0086] Event annotation: Detailed annotation of detected sound events, including information such as the start time, duration and type of the event, for subsequent processing;

[0087] S3, timbre recognition, specifically including:

[0088] Tone judgment: Use tone recognition technology to determine whether the audio clips have the same tone. If different tones are detected, a 3-second original voice clip is sent so that the tone can be restored later based on the original voice.

[0089] Timbre consistency processing: If the timbre is consistent, speech recognition is performed to extract the speech content;

[0090] Time tagging: tag the speech to ensure the accuracy of the time information of each speech segment;

[0091] S4. Audio feature extraction

[0092] S41, feature parameter extraction, including:

[0093] Frequency feature extraction: extract frequency features from audio signals and analyze the frequency distribution of audio;

[0094] Mel frequency extraction: Extract Mel frequency and use Mel frequency cepstral coefficients (MFCC) to analyze audio features;

[0095] Phoneme feature extraction: Identify phoneme features in audio and analyze the distribution and changes of phonemes;

[0096] Energy feature extraction: extract the energy features of audio signals and analyze the energy changes of audio;

[0097] Tone feature extraction: extract the tone features of the audio and analyze the changes in the tone;

[0098] Specific sound pattern recognition: Identify specific sound patterns in audio, such as laughter, crying, etc.;

[0099] Volume feature extraction: extract the volume features of the audio and analyze the changes in volume;

[0100] Speech rate feature extraction: extract speech rate features of audio and analyze changes in speech rate;

[0101] S42, Feature Integration: Integrate time annotation, sound events and sentiment analysis into speech features to form a comprehensive audio feature dataset;

[0102] S5: Data transmission: the 3-second original speech clip and the audio feature data set after feature fusion are transmitted.

[0103] Through the above steps, the audio feature extraction method of the present invention not only improves the accuracy and efficiency of audio feature extraction, but also provides strong technical support for audio data analysis in multiple fields.

[0104] The video feature extractor process includes: first, for the newly arrived Nth frame, the video encoder compares it with the previous frame to determine whether the frame has undergone a scene switch; if a scene switch is detected, the frame is set as a key frame, and feature extraction and feature fusion are performed to obtain the features of the frame. The setting of the key frame helps to maintain the continuity and consistency of the video when the scene switches, ensuring the video quality; if no scene switch is performed, the video encoder will increase the number of frames by one and check whether the number of frames is an integer multiple of GOP (Group of Pictures); if the number of frames is an integer multiple of GOP, the frame is set as a key frame, and feature extraction and fusion are performed. This method ensures the reasonable distribution of key frames in the video encoding process by regularly setting key frames, thereby improving the compression efficiency and decoding quality of the video; for the case where the number of frames is not an integer multiple of GOP, the video encoder will use the optical flow method to calculate the inter-frame motion vector between the frame and the previous frame to obtain a motion description; then, combined with the features of the previous frame, the features of the Nth frame are generated. The optical flow method can accurately describe the changes between frames by analyzing the motion information between frames, thereby improving the accuracy and efficiency of video encoding. Through the above process optimization, the video encoder can improve the encoding efficiency and compression ratio while ensuring the video quality. This method is not only applicable to conventional video encoding, but also can be applied to high frame rate and high resolution video encoding scenarios, and has broad application prospects and market value.

[0105] See also Figure 3 ,The video feature extractor process specifically includes the following steps:

[0106] S6, scene switching detection

[0107] S61, for the Nth frame extracted from the video, the scene discriminator compares it with the previous frame, analyzes the content of the Nth frame and the N-1th frame, and determines whether there is a scene switch. By comparing the color, texture, edge and other features between the frames, the scene discriminator can accurately identify the scene switch;

[0108] S62: If a scene switch is detected, the frame is set as a key frame, the Nth frame is marked as a key frame, and the number of frames is initialized to 1 to ensure that the continuity and consistency of the video are maintained when the scene switches. The setting of the key frame helps to maintain the smoothness and visual coherence of the video when the scene switches;

[0109] S63, extracting and fusing features of the key frame to obtain features of the frame, extracting various video features from the key frame, such as color, texture, edge, etc.; preprocessing the extracted features to generate processed features, which will be used in subsequent encoding and compression processes to improve video quality. The various extracted video features include:

[0110] Text feature extraction: extract text features from the video and analyze the text information in the video;

[0111] Facial and lip shape information extraction: Extract facial and lip shape information from the video, and analyze facial expressions and lip shape changes;

[0112] Scene understanding: Understand the scenes in the video and analyze the changes in the scenes;

[0113] Edge detection: perform edge detection and extract edge features in the video;

[0114] Contextual information understanding: extract contextual information from the video and analyze the contextual relationship of the video;

[0115] Action and posture recognition: Identify actions and postures in videos and analyze changes in actions and postures;

[0116] Light and shadow information extraction: extract light and shadow information from the video and analyze the changes in light and shadow;

[0117] Object detection: perform object detection to identify objects in the video;

[0118] Color and texture feature extraction: Extract color and texture features from videos and analyze color and texture changes;

[0119] S7. Set keyframes regularly

[0120] S71, if no scene switch is detected, the video encoder increases the frame number by 1, increases the current frame number N by 1, and updates the frame number. In this way, it is ensured that each frame is correctly counted and processed;

[0121] S72, multiple check: determine whether the current frame number is an integer multiple of GOP; if the frame number is an integer multiple of GOP, set the frame as a key frame, and reset the frame number to 1 to ensure a reasonable distribution of key frames. Regularly setting key frames helps maintain video quality and consistency during a long video encoding process;

[0122] S73, extracting and preprocessing the key frames, extracting various video features from the key frames, preprocessing the extracted features, generating processed features, which will be used in subsequent encoding and compression processes to improve video quality;

[0123] S8, inter-frame motion vector calculation

[0124] S81. When the number of frames is not an integer multiple of the GOP, the video feature extractor uses an optical flow method to calculate an inter-frame motion vector between the frame and the previous frame. The optical flow method calculates: the motion information between the Nth frame and the N-1th frame is analyzed by the optical flow method to obtain a motion vector. The optical flow method can accurately describe the motion changes between frames, thereby improving the accuracy and efficiency of video encoding.

[0125] S82, combining the features of the previous frame to generate features of the Nth frame, and using the features of the previous frame and the calculated motion vector to generate features of the Nth frame, these features will be used in subsequent encoding and compression processes to improve video quality;

[0126] Through the above process optimization, the video encoder can improve the encoding efficiency and compression ratio while ensuring the video quality. Coding efficiency: optimize the encoding process to improve the efficiency of video encoding. Compression ratio: improve the compression ratio of the video by reasonably setting key frames and accurately describing the motion between frames. The optimized encoding process can significantly reduce the size of the video file while maintaining high-quality visual effects.

[0127] This method is suitable for conventional video encoding, high frame rate and high resolution video encoding scenarios. Application scenarios: This method has broad application prospects and market value and is suitable for a variety of video encoding scenarios. Whether it is conventional video encoding or high frame rate and high resolution video encoding, this method can provide efficient and high-quality encoding solutions.

[0128] The audio and video joint coding method includes: first, extracting various modal features from audio signals and various modal features from videos, and inputting the features of various modalities extracted from audio or video into a cross-modal attention mechanism after residual network learning. The cross-modal attention mechanism can adaptively focus on significant features, improve the efficiency of the overall fusion process, and make the model more robust and flexible; in the feature fusion stage, each modal feature and the feature of the cross-modal attention are fused, which can make full use of the information of each modality and improve the overall performance of the model. The fusion strategy ensures that the features of different modalities can complement each other, thereby improving the prediction ability of the model; dynamically adaptively adjusting the weight of each fused feature according to different tasks, the model can dynamically select the relevant features that need to be emphasized according to the needs of different tasks to optimize the model performance and improve the prediction accuracy. The dynamic weighting strategy can adapt to different task scenarios and enhance the generalization ability of the model; in the modality level feature weighting stage, the features of different modality levels are weighted to ensure that the importance of each modality can be fully considered; then, the weighted features are fused. This fusion strategy can improve the robustness and stability of the model and avoid information loss.

[0129] See also Figure 4 , wherein the audio and video joint encoding method specifically includes the following steps:

[0130] S9. Audio and video feature extraction

[0131] S91. Extract features from audio: frequency features, Mel frequency, phoneme features, energy features, pitch features, specific sound patterns, volume features;

[0132] S92. Extract features from videos: text features, facial and lip information, scene understanding, edge detection, context information understanding, action and posture recognition, light and shadow information extraction, object detection, color and texture features;

[0133] S10. Fusion in Cross-modal Attention

[0134] S101, first, input the extracted video features into the cross-modal attention;

[0135] S102, inputting the audio features into the cross-modal attention through the auxiliary residual auxiliary network, and analyzing the features of different modalities through the cross-modal attention mechanism;

[0136] S103, the cross-modal attention mechanism of audio features is the same as S101 and S102; the features of different modalities are connected with the output results of cross-modal attention, so that the spliced ​​result can not only retain the characteristics of the original features, but also improve the model's ability to focus on significant features, thereby improving the overall fusion efficiency;

[0137] Among them, the residual auxiliary network extracts deep features from the features extracted from audio and video. Since the residual auxiliary network has a deep network structure and residual connections, it can effectively capture and represent complex patterns and features;

[0138] S11. Task Identification

[0139] S111, task type determination: First, the system needs to identify the current task type, which may include emotion expression and reproduction, action recognition and reconstruction, scene understanding and reconstruction, subtitle generation and synchronization, speech separation, abnormal behavior detection, video summary generation, etc. Through the task identification module, the system can automatically determine the specific type of the current task;

[0140] S112, Task Requirements Analysis: Once the task type is determined, the system will analyze the specific requirements of the task. For example, for the emotion expression and reproduction task, the system needs to focus on the emotion features in the audio and the facial expression features in the video; for the action recognition and reconstruction task, the system needs to focus on the action and posture features in the video. Through the task requirements analysis, the system can determine the type of features that need to be emphasized;

[0141] S12. Dynamic Adaptive Weight Allocation

[0142] S121. Feature importance assessment: Based on the task requirements, the system assesses the importance of audio and video features. By calculating the contribution of each feature in the current task, the system can determine the importance weight of each feature. For example, in an emotion expression task, the emotional feature in the audio may be more important than the color feature in the video.

[0143] S122, weight adjustment strategy: The system dynamically adjusts the weights of audio and video features based on the feature importance evaluation results. The specific steps include: first, calculating the initial weight of each feature, and then adjusting the weight of each feature based on task requirements and feature importance, so that important features occupy a larger proportion in the fusion process;

[0144] S123, Adaptive weight allocation: During the execution of each task, the system monitors the progress of the task and the performance of the features in real time and dynamically adjusts the weight allocation. For example, if the contribution of certain features is found to change during the execution of the task, the system will adaptively adjust the weights of these features to ensure that the model can always use the most relevant features for prediction;

[0145] S124, modality data loss processing: When the data of a certain modality is lost or does not exist, the system can use the data of the existing modality to generate the data of the missing modality. The specific steps include: first, identifying the data type of the missing modality; then, using the features of other modalities, generating the data of the missing modality through a generative model (such as a generative adversarial network or a variational autoencoder). This method ensures the integrity and consistency of the data and improves the robustness of the model in the case of incomplete data;

[0146] Through dynamic adaptive weight allocation, the system can selectively emphasize relevant features and optimize the performance of the model. This adaptability not only improves the prediction accuracy of the model, but also enhances the robustness and generalization ability of the model in different task scenarios;

[0147] S13. Fusion features

[0148] Multimodal features assigned with different weights are fused, and the fusion method of each modal feature is flexibly adjusted according to task requirements and data characteristics.

[0149] The audio and video joint decoder includes: first, decoding the multimodal features transmitted from the sender (audio and video joint encoder), and performing normalization, denoising and interpolation processing to improve data quality; then, ensuring the alignment of audio and video features in time and space to avoid synchronization problems; then, the decoded audio and video multimodal features are input into the joint decoding module, the video generation module generates video through word embedding, position encoding, multi-head self-attention mechanism and other steps, and the audio generation module fuses the weighted multimodal features with the timbre features extracted by the original audio through the timbre encoder, and then performs stream matching through the autoregressive Transformer. Combined with HiFNet to generate high-quality audio; finally, ensure that the audio and video are aligned on the timeline, using timestamps or other synchronization technologies to achieve this. The generated audio and video content needs to be objectively evaluated, and standard evaluation indicators (such as PSNR, SSIM, etc.) are used to evaluate the quality of the generated audio and video. If the objective evaluation result is qualified, the video is directly output. If it is unqualified, it enters the weight adjustment module and dynamically adjusts the weight distribution of multimodal features until a qualified video is generated. Through multiple iterations, the feature weights and generation process are continuously adjusted to gradually improve the quality of the generated content. After each iteration, the weight adjustment and generation results are recorded to analyze and optimize the generation strategy.

[0150] See also Figure 5 , wherein the audio and video joint decoder specifically includes the following steps:

[0151] S15. Feature decoding and preprocessing

[0152] The receiving end first needs to decode the multimodal features transmitted from the sending end. These features may be transmitted in a compressed or encrypted form, so they need to be decoded. In the process of transmitting the features from the sending end to the receiving end, they may be affected by various factors, such as noise, signal attenuation, compression loss, etc. These influences will cause the quality of feature data to deteriorate, thus affecting the accuracy of subsequent processing. Therefore, preprocessing is necessary at the receiving end. The preprocessing steps include:

[0153] Normalization: scaling feature data to a standard range to eliminate dimensional differences between different features;

[0154] Denoising: Use filters or other techniques to remove noise from feature data and improve signal purity;

[0155] Interpolation: interpolate missing data points to ensure data integrity;

[0156] Feature alignment and synchronization, including temporal alignment and spatial alignment:

[0157] Temporal alignment: Features of different modalities (such as audio and video) may be out of sync in time. This asynchrony will cause problems in multimodal fusion and affect the final processing effect. For example, in the subtitle generation and synchronization task, the temporal alignment of audio and video is a key step to ensure accurate synchronization of subtitles.

[0158] Spatial alignment: For image and video features, spatial alignment means ensuring that features of different modalities correspond in space. For example, in the fusion of image and depth information, it is necessary to ensure that each pixel in the image can find the corresponding depth value. Spatial alignment can be achieved through geometric transformation, registration and other technologies to ensure the consistency of multimodal features in space.

[0159] S16, joint decoding

[0160] In the joint decoding stage, the decoded audio and video features are processed by an AI-driven multimodal deep learning model. This module combines the audio and video generation modules to generate the final multimodal content through feature fusion and deep networks: Video Generation Module: A deep learning-based encoder-decoder architecture for generating video frames from video features. This module uses AI video generation technology, especially the Transformer encoder, to extract key features of the video from the input features to help generate more natural and smooth visual content. Audio Generation Module: The AI ​​timbre encoder is used to extract and fuse the weighted multimodal features and the timbre features of the original audio. The fused features are converted into a sequence of semantic tokens through an autoregressive Transformer to gradually generate audio content with high consistency and clarity. The final audio passes through the AI ​​synthesis algorithm HiFNet to ensure that the generated audio meets the expected sound quality and semantics.

[0161] S161, the video decoding module uses the Transformer architecture in the AI ​​deep learning model to convert input features into high-bit vectors. The encoder mainly includes:

[0162] Word embedding: embedding video features into a high-dimensional vector space so that deep models can better process the semantic information in the video;

[0163] Positional Encoding: Add position information to video features. Since the Transformer structure itself does not contain timing information, positional encoding is used to enable the model to understand the position of the feature on the timeline.

[0164] Multi-Head Self-Attention Mechanism: At this stage, the AI ​​model uses multiple attention heads to extract the correlation between video features and help generate a high-quality attention matrix. Each attention head independently calculates the feature correlation under different perspectives, and finally splices the results for subsequent video frame generation;

[0165] Feed-Forward Neural Network: Performs nonlinear transformation on the features of each position to enhance the expressiveness of the model;

[0166] Residual Connection and Layer Normalization: Add residual connection after each sublayer and perform layer normalization to prevent gradient disappearance and explosion;

[0167] The decoder samples from the latent space and generates video frames. The decoder is usually composed of multiple deconvolution layers. During the decoding process, in addition to video features, audio features are used as conditional inputs to guide the generation of video frames. The decoder mainly includes masked multi-head self-attention mechanism, encoder-decoder attention, feedforward neural network and residual connection. Masked multi-head self-attention mechanism: similar to the encoder's self-attention mechanism, but masks future information during calculation to ensure that the decoder can only see previously generated content. Encoder-decoder attention: calculates the correlation between the current state of the decoder and the encoder output to generate an attention matrix. Feedforward neural network and residual connection: same as the encoder, nonlinear transformation and layer normalization are performed;

[0168] S162, the audio decoding module converts the fused features into semantic tokens and generates the final audio content based on AI-driven audio generation technology. The specific steps are as follows:

[0169] Timbre feature fusion: Through the AI ​​timbre encoder, multimodal features are fused with the timbre features of the original audio to ensure the consistency and richness of the audio generation.

[0170] Autoregressive Transformer generates semantic token sequences: The autoregressive Transformer gradually generates each semantic token sequence during the generation process, ensuring that the generated audio semantics conforms to the input features. The AI ​​model ensures the coherence and high quality of the audio content through a recursive generation process at this stage.

[0171] The stream matching model generates a mel-spectrogram: The semantic tokens generated by the Transformer are input into the stream matching model, and AI handles the generation tasks of multiple languages ​​and styles and converts the token sequence into a mel-spectrogram.

[0172] HiFNet vocoder generates speech waveforms: HiFNet converts Mel spectrograms into high-fidelity audio waveforms based on AI spectrum conversion technology. The model uses an optimized algorithm for the spectrum feature and waveform feature generation process to ensure that the output audio meets the effect of natural sound.

[0173] S17. When merging audio and video, use AI-driven synchronization technology to ensure that the audio and video are aligned on the timeline.

[0174] AI synchronization algorithm: Use AI-based timestamp and frame synchronization technology to ensure the timing of audio and video is consistent. By comparing the characteristic time points of each frame, the visual content and audio content are accurately synchronized to avoid out-of-sync problems.

[0175] S18. Objective evaluation: AI generates content evaluation models to evaluate audio and video quality, including multiple indicators of video and audio:

[0176] PSNR (Peak Signal-to-Noise Ratio): The peak signal-to-noise ratio of the generated video and the reference video is calculated by the AI ​​model to evaluate the video clarity. A higher PSNR value indicates better quality of the generated video;

[0177] SSIM (Structural Similarity Index): AI-based SSIM evaluates the similarity of video structures to ensure the structural consistency of the generated video results and the reference video. The closer the SSIM value is to 1, the higher the structural similarity.

[0178] Audio quality assessment: Use metrics such as PESQ (perceptual speech quality) or STOI (speech transmission index) to assess the quality of audio generated by the AI ​​model;

[0179] S19, Dynamic Adjustment

[0180] The AI-based optimization module dynamically adjusts feature weights based on the evaluation results and generates audio and video content that meets the requirements through multiple iterations:

[0181] S191, weight adjustment module: The AI ​​model uses optimization algorithms (such as gradient descent or genetic algorithms) to dynamically adjust the weight distribution of audio and video features to improve content quality;

[0182] S192, regenerate content: After each weight adjustment, the new multimodal features regenerate the audio and video content through the joint decoding module, and the AI ​​model evaluates the output quality again until it meets the expected standards;

[0183] Through multiple iterations, the feature weights and generation process are continuously adjusted to gradually improve the quality of the generated content. After each iteration, the weight adjustment and generation results are recorded to analyze and optimize the generation strategy.

[0184] Generative AI can establish deep connections between audio and video data, significantly improve overall coding efficiency by sharing information and features, and generate final audio and video content on demand. This joint coding method uses the power of generative models to more effectively capture redundant information in audio and video data, thereby achieving a higher compression ratio; generative AI also supports the customization of video content, which means that users can flexibly adjust the style, effect and quality of audio and video content according to their personal needs and preferences. This customization capability provides users with greater control, allowing each user to obtain a personalized viewing experience. At the same time, the adaptive ability of generative AI enables it to flexibly adjust the coding strategy according to different content types and scenarios, ensuring that while maintaining audio and video synchronization, quality loss is minimized. In summary, audio and video joint coding based on generative AI not only improves the efficiency and quality of audio and video coding, but also opens up new possibilities for future multimedia applications, especially in meeting personalized needs. It shows great potential.

[0185] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An audio and video joint coding method based on generative artificial intelligence, characterized by: The following steps are involved: S9, extract various modal features from audio signals and videos; S10. Fusion in cross-modal attention, including: S101, first, input the extracted video features into the cross-modal attention; S102, inputting the audio features into the cross-modal attention through the auxiliary residual auxiliary network, and analyzing the features of different modalities through the cross-modal attention mechanism; S103, the cross-modal attention mechanism of audio features is the same as S101 and S102; the features of different modalities are connected with the output results of cross-modal attention; S11, task identification, first identify the current task type, and after determining the task type, the system will analyze the specific requirements of the task; S12, dynamic adaptive weight allocation; S13, feature fusion, fusing multimodal features assigned with different weights.

2. The method for joint audio and video encoding based on generative artificial intelligence according to claim 1, characterized in that: The step S12, dynamic adaptive weight allocation, specifically includes: S121, Feature Importance Assessment: Based on the task requirements, the system assesses the importance of audio and video features, and determines the importance weight of each feature by calculating the contribution of each feature in the current task; S122, weight adjustment strategy: the system dynamically adjusts the weights of audio and video features according to the feature importance evaluation results; S123, Adaptive weight allocation: During the execution of each task, the system monitors the progress of the task and the performance of the features in real time and dynamically adjusts the weight allocation; S124, modality data loss processing: when the data of a certain modality is lost or does not exist, the system generates the data of the missing modality using the data of the existing modality.

3. The method for joint audio and video encoding based on generative artificial intelligence according to claim 1, characterized in that: Extracting various modal features from audio signals includes: S1. Sentiment analysis, including: Audio data preprocessing: including noise reduction and normalization; Emotional state recognition: using the sentiment analysis algorithm to recognize the emotional state of the preprocessed audio data and classify the emotional state in the audio; S2. Sound event detection, including: Sound event recognition: Detect sound events in audio data and identify various sound events in the audio; Event annotation: Detailed annotation of detected sound events, including the start time, duration and type of the event; S3, timbre recognition, including: Voice judgment: through voice recognition technology, determine whether the audio clips are of the same voice. If different voices are detected, send the original voice clips of the predetermined time; Timbre consistency processing: If the timbre is consistent, speech recognition is performed to extract the speech content; Time marking: marking the time of speech; S4, audio feature extraction, including feature parameter extraction and feature integration, integrating time annotation, sound events and sentiment analysis into speech features to form a comprehensive audio feature dataset; S5: Data transmission: the original speech segment and the audio feature data set after feature fusion are transmitted.

4. The method for joint audio and video encoding based on generative artificial intelligence according to claim 1, characterized in that: Extracting various modal features from video signals includes: S6, scene switching detection S61, for the Nth frame extracted from the video, the scene discriminator compares it with the previous frame, analyzes the contents of the Nth frame and the N-1th frame, and determines whether there is a scene switch; S62: If a scene switch is detected, the frame is set as a key frame, the Nth frame is marked as a key frame, and the frame number is initialized to 1; S63, extracting and fusing features of the key frame to obtain features of the frame, and extracting various video features from the key frame; preprocessing the extracted features to generate processed features; S7. Set keyframes regularly S71, if no scene switching is detected, the video encoder increases the frame number by 1 and increases the current frame number N by 1; S72, multiple check: determine whether the current frame number is an integer multiple of GOP; if the frame number is an integer multiple of GOP, set the frame as a key frame and reset the frame number to 1; S73, performing feature extraction and feature preprocessing on the key frames, extracting various video features from the key frames, preprocessing the extracted features, and generating processed features; S8, inter-frame motion vector calculation S81, when the number of frames is not an integer multiple of the GOP, the video feature extractor uses an optical flow method to calculate an inter-frame motion vector between the frame and the previous frame; S82. Generate features of the Nth frame by combining features of the previous frame, and generate features of the Nth frame by using features of the previous frame and the calculated motion vector.

5. A method for joint audio and video decoding based on generative artificial intelligence, characterized in that: The following steps are involved: S15. Feature decoding and preprocessing The receiving end first needs to preprocess the multimodal features transmitted from the sending end after being processed by the audio and video joint encoding method described in any one of claims 1 to 4. The preprocessing steps include: normalization, denoising, interpolation, and feature alignment and synchronization. The feature alignment includes time alignment and space alignment. Then, the preprocessed multimodal features are decoded. S16, joint decoding The multimodal features of the audio and video decoded in step S15 are input into the joint decoding module. The joint decoding processes the decoded audio and video features through an AI-driven multimodal deep learning model. This module combines the audio and video generation modules to generate the final multimodal content through feature fusion and deep network: Video generation module: an encoder-decoder architecture based on deep learning, used to generate video frames from video features. This module uses AI video generation technology, especially the Transformer encoder, to extract key features of the video from the input features. Audio generation module: uses AI's timbre encoder to extract and fuse the weighted multimodal features and the timbre features of the original audio. The fused features are converted into semantic token sequences through an autoregressive Transformer to gradually generate audio content with high consistency and clarity. The final audio passes through the AI ​​synthesis algorithm HiFNet to ensure that the generated audio meets the expected sound quality and semantics. S17. Merge audio and video, and use AI-driven synchronization technology to ensure that audio and video are aligned on the timeline.

6. The method for joint audio and video decoding based on generative artificial intelligence according to claim 5, characterized in that: The step S16, joint decoding specifically includes: S161, the video decoding module first performs word embedding, converts the input video features into vector representation, and then performs position encoding on the vector; S162. The audio decoding module is based on AI-driven audio generation technology, which converts the fused features into semantic tokens and generates the final audio content, including: generating timbre features, fusing the multimodal features with the timbre features of the original audio through the AI ​​timbre encoder to ensure the consistency and richness of the audio generation; the autoregressive Transformer is used to convert the input features into a semantic token sequence. The AI ​​model ensures the coherence and high quality of the audio content through a recursive generation process at this stage; the semantic token sequence generated by the autoregressive Transformer is input into the stream matching model, and the stream matching model is used to convert the generated semantic token sequence into a Mel-spectrogram; HiFNet is based on AI spectrum conversion technology to convert the Mel-spectrogram into a high-fidelity audio waveform.

7. The method for joint audio and video decoding based on generative artificial intelligence according to claim 5, characterized in that: After the step S17, merging the audio and video, the following steps are further included: S18. Objective evaluation: The AI-generated content evaluation model is used to evaluate the quality of audio and video. The evaluation indicators include: peak signal-to-noise ratio. The higher the peak signal-to-noise ratio, the better the quality of the generated video; structural similarity index. The closer the structural similarity index value is to 1, the more similar the structure of the generated video is to the reference video; audio quality evaluation: Use the perceptual speech quality evaluation or speech transmission index indicators to evaluate the quality of the generated audio. If the objective evaluation result is qualified, the video is directly output. If it is unqualified, it enters the weight adjustment module to dynamically adjust the weight distribution of multimodal features until a qualified video is generated; S19. Dynamic adjustment.

8. The method for joint audio and video decoding based on generative artificial intelligence as claimed in claim 7, characterized in that: The step S19, dynamic adjustment comprises: S191, weight adjustment module: adjust the weight distribution of audio and video features according to the objective evaluation results. The AI ​​model uses gradient descent or genetic algorithm to dynamically adjust the weights; S192, regenerate content: After each weight adjustment, the new multimodal features regenerate the audio and video content through the joint decoding module, and the AI ​​model evaluates the output quality again until it meets the expected standards.

9. An audio and video encoding and decoding method based on generative artificial intelligence, characterized in that: The following steps are involved: Audio feature extraction; Video feature extraction; Audio and video joint coding, using the audio and video joint coding method described in any one of claims 1 to 4 to jointly encode the extracted audio features and video features; Audio and video joint decoding, using the audio and video joint decoding method described in any one of claims 5-8 to jointly decode the received audio features and video features.

10. An audio and video coding system based on generative artificial intelligence, characterized in that: It includes an audio feature extractor, a video feature extractor, an audio-video joint encoder and an audio-video joint decoder, wherein the audio-video joint encoder adopts the audio-video joint encoding method described in any one of claims 1-4 to jointly encode the extracted audio features and video features; the audio-video joint decoder adopts the audio-video joint decoding method described in any one of claims 5-8 to jointly decode the received audio features and video features.

Images