An audio and video repairing method and device based on artificial intelligence

Abstract

The application belongs to the technical field of audio and video processing, and provides an audio and video repairing method and device based on artificial intelligence, wherein audio time sequence features and video space features are extracted from damaged audio and video, a joint feature vector is obtained according to the audio time sequence features and the video space features, the joint feature vector is evaluated to obtain a damage evaluation result, a repairing model is adaptively adjusted according to the damage evaluation result to obtain an optimized repairing model, a repairing result is output according to the optimized repairing model, the repairing result is used for feedback to adjust the repairing model, after multiple rounds of iteration, a trained repairing model is obtained and is optimized and deployed, the performance and precision of the repairing model are improved, the occupation of the repairing model is reduced, and thus the repairing effect and the universality for damaged audio and video are improved.

Description

Technical Field

[0001] This invention belongs to the field of audio and video processing technology, and more specifically, relates to an audio and video restoration method and apparatus based on artificial intelligence. Background Technology

[0002] With the widespread application of digital technology, audio and video materials occupy an important place in people's lives, work, and entertainment. However, due to various factors such as storage device failure, interference during transmission, and encoding conversion errors, audio and video files often suffer from problems such as data loss, video stuttering, and sound distortion. Traditional audio and video restoration methods, such as signal processing-based interpolation and filtering methods, are effective in handling simple noise or small-scale data loss, but for complex damage situations, such as large-area video damage or severe audio distortion, the restoration effect is often unsatisfactory, and it is easy to cause the loss of details in the original audio and video content.

[0003] Therefore, overcoming the shortcomings of the existing technology is an urgent problem to be solved in this technical field. Summary of the Invention

[0004] The problem this invention aims to solve is how to improve the repair effect and versatility of audio and video data.

[0005] Firstly, an artificial intelligence-based audio and video restoration method is provided, including: The damaged audio and video data are preprocessed to obtain preprocessed audio data and preprocessed video data. Audio temporal features are obtained from the preprocessed audio data, video spatial features are obtained from the preprocessed video data, and a joint feature vector is obtained from the audio temporal features and the video spatial features. The joint feature vector is evaluated to obtain a damage assessment result. The repair model is then adaptively adjusted based on the damage assessment result to obtain an optimized repair model. The joint feature vector is input into the optimized repair model, and the repair result is output. The optimized repair model is adjusted based on the repair result, and the repair model is trained through multiple iterations. The trained restoration model is then streamlined and optimized, and the streamlined and optimized restoration model is deployed to restore audio and video.

[0006] Preferably, the preprocessing of the damaged audio and video to obtain preprocessed audio data and preprocessed video data specifically includes: Obtain the audio and video data from the damaged audio and video separately; The video data is decoded to obtain a frame-by-frame image sequence. The frame-by-frame image sequence is normalized and the pixel values ​​in the frame-by-frame image sequence are mapped to a preset range to obtain the preprocessed video data. The sampling rate and quantization bit depth of the audio data are standardized, and the standardized audio data is then cut into multiple audio segments of a preset length. All audio segments are then padded, and all padded audio segments are used together as the preprocessed audio data.

[0007] Preferably, the step of obtaining audio temporal features from the preprocessed audio data, obtaining video spatial features from the preprocessed video data, and obtaining a joint feature vector from the audio temporal features and the video spatial features specifically includes: The preprocessed audio data is input into a recurrent neural network to obtain audio temporal features; The preprocessed video data is input into a convolutional neural network to obtain video spatial features; The audio temporal features and the video spatial features are fused according to the corresponding attention weights to obtain the joint feature vector.

[0008] Preferably, the step of fusing the audio temporal features and the video spatial features according to corresponding attention weights to obtain the joint feature vector specifically includes: Obtain the audio attention weights corresponding to the audio temporal features, and obtain the video attention weights corresponding to the video spatial features; The expression for the audio attention weight is: =σ( × + ); The expression for the video attention weight is: =1- ; in, For audio attention weights, Here, σ represents the video attention weights, and σ is the Sigmoid activation function. For trainable video weight matrices, A vector representing the spatial features of the video. A vector representing the temporal features of the audio. The joint feature vector is obtained based on audio temporal features, audio attention weights, video spatial features, and video attention weights.

[0009] Preferably, the evaluation of the joint feature vector to obtain the damage assessment result specifically includes: Based on the joint feature vector, the corresponding frame rate change features, image anomaly features, frequency distribution features, and noise level features are obtained; The damage level is determined in the pixel entropy dimension based on the frame rate change characteristics and image anomaly characteristics. Based on the frequency distribution characteristics and noise level characteristics, the damage level is determined in the signal-to-noise ratio dimension to obtain the damage level in the signal-to-noise ratio dimension. The damage assessment result corresponding to the joint feature vector is obtained based on the damage level in the pixel entropy dimension and the damage level in the signal-to-noise ratio dimension.

[0010] Preferably, the step of adaptively adjusting the repair model based on the damage assessment results to obtain an optimized repair model specifically includes: The proportion of loss types is calculated using a two-dimensional weighting method based on the joint feature vector, which combines damage level weights and modality basis weights. The number of convolutional kernels is dynamically adjusted based on the weight of the loss function. Match the optimal number of iterations based on the number of convolutional kernels; The optimized repair model is obtained by iterative optimization based on the optimal number of iterations.

[0011] Preferably, the step of adjusting the optimized repair model based on the repair results specifically includes: Obtain the reference results corresponding to the damaged audio and video; Compare the reference results with the repair results; Feedback adjustment is performed when the loss value of any mode decreases by less than a preset threshold for a consecutive preset number of rounds, or when the loss value of one mode decreases but the loss value of another mode increases. The feedback adjustment includes: for modes in a stagnant state, increasing the damage level weight and the basic weight of the corresponding mode by a first preset value; increasing the loss weight and the basic weight of the mode with increasing loss value by a second preset value; and decreasing the loss weight and the basic weight of the mode with decreasing loss value by a third preset value, so as to suppress the gradient interference of the mode with decreasing loss value on the mode with increasing loss value. The adjusted video loss weights and audio loss weights are normalized to ensure that the sum of the video loss weights and audio loss weights is 1, thus avoiding weight imbalance that could lead to deviations in the restoration effect.

[0012] Preferably, the step of streamlining and optimizing the trained repair model specifically includes: Obtain the contribution of each connection relationship and the contribution of each parameter in the repair model; Calculate the percentage of the absolute weight of each parameter in the repair model, calculate the percentage of the gradient magnitude of each connection in the repair model, and obtain the contribution based on the percentage of the absolute weight and the percentage of the gradient magnitude. A contribution threshold is set, and redundant connections and invalid parameters are removed based on the contribution threshold to obtain a pre-simplified, optimized, and repaired model.

[0013] Secondly, an artificial intelligence-based audio and video restoration device is provided, comprising at least one processor and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the processor to perform the artificial intelligence-based audio and video restoration method.

[0014] Thirdly, the present invention also provides a non-volatile computer storage medium storing computer-executable instructions that are executed by one or more processors to perform the method described in the first aspect.

[0015] Fourthly, a chip is provided, comprising: a processor and an interface for calling and running a computer program stored in memory, performing the method as described in the first aspect.

[0016] Fifthly, a computer program product containing instructions is provided that, when executed on a computer or processor, causes the computer or processor to perform the method as described in the first aspect.

[0017] In a sixth aspect, an artificial intelligence-based audio and video restoration system is provided, including an artificial intelligence-based audio and video restoration device as described in the second aspect, and using an artificial intelligence-based audio and video restoration method as described in the first aspect.

[0018] Unlike existing technologies, the present invention has at least the following beneficial effects: By extracting audio temporal features and video spatial features from damaged audio and video, a joint feature vector is obtained based on the audio temporal features and the video spatial features. The joint feature vector is evaluated to obtain a damage assessment result. The repair model is adaptively adjusted based on the damage assessment result to obtain an optimized repair model. The repair result is output based on the optimized repair model and used to adjust the repair model. After multiple rounds of iteration, a trained repair model is obtained and optimized for deployment. This improves the performance and accuracy of the repair model while reducing its occupancy, thereby improving the repair effect and versatility of damaged audio and video. Attached Figure Description

[0019] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments of the present invention will be briefly described below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0020] Figure 1 This is a flowchart of an artificial intelligence-based audio and video restoration method provided in an embodiment of the present invention; Figure 2 This is a flowchart illustrating a method for acquiring preprocessed video data and preprocessed audio data in an artificial intelligence-based audio and video restoration method provided in an embodiment of the present invention. Figure 3 This is a flowchart of a method for obtaining joint feature vectors in an artificial intelligence-based audio and video restoration method provided in an embodiment of the present invention; Figure 4 This is a damage assessment result of an artificial intelligence-based audio and video restoration method provided in an embodiment of the present invention; Figure 5 This is a flowchart of a method for optimizing a repair model in an artificial intelligence-based audio and video repair method provided in an embodiment of the present invention; Figure 6 This is a flowchart illustrating the adjustment method of the restoration model in an artificial intelligence-based audio and video restoration method provided in an embodiment of the present invention. Figure 7 This is a flowchart of a method for obtaining a simplified and optimized repair model in an artificial intelligence-based audio and video repair method provided in an embodiment of the present invention; Figure 8 This is a system schematic diagram corresponding to an artificial intelligence-based audio and video restoration method provided in an embodiment of the present invention; Figure 9 This is a schematic diagram of an audio and video restoration device based on artificial intelligence provided in an embodiment of the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0022] Unless the context otherwise requires, throughout the specification and claims, the term "comprising" is interpreted as openly inclusive, meaning "including, but not limited to." In the description of the specification, terms such as "one embodiment," "some embodiments," "exemplary embodiment," "example," "specific example," or "some examples" are intended to indicate that a particular feature, structure, material, or characteristic associated with that embodiment or example is included in at least one embodiment or example of this disclosure. The illustrative representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics mentioned may be included in any suitable manner in any one or more embodiments or examples; that is, although they may be incorporated into embodiments or examples using the above terms for reasons such as order and position, it does not limit them to be incorporated in combination by a single embodiment or example.

[0023] In the description of this invention, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined with "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of embodiments of this disclosure, unless otherwise stated, "a plurality of" means two or more. Furthermore, for example, the description may use the prefix "A" or "B" to describe the same type of nouns as two independent entities. In this case, the corresponding features defined with "A" and "B" are used only to distinguish between similar entities and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features.

[0024] In the description of this invention, the expression “A and / or B” (where A and B are used to formally represent specific features) will be used. The corresponding expression includes the following three combinations: only A, only B, and a combination of A and B.

[0025] As used in this invention, “about,” “approximately,” or “approximately” includes the stated value and the average value within an acceptable range of deviation from a particular value, wherein the acceptable range of deviation is determined by a person skilled in the art taking into account the measurement under discussion and the error associated with the measurement of the particular quantity (i.e., the limitations of the measurement system).

[0026] Furthermore, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0027] Example 1: This embodiment provides an artificial intelligence-based audio and video restoration method, such as... Figure 1 As shown, the method flow includes the following.

[0028] In step 101, the damaged audio and video are preprocessed to obtain preprocessed audio data and preprocessed video data.

[0029] In this embodiment, the damaged audio and video refers to audio and video data in which a large area of ​​the video image is damaged and / or the audio is severely distorted. The provided method is used to train a repair model, which is used to repair the damaged audio and video, and to repair and adjust the image damage and audio distortion in the damaged audio and video. By training the repair model, the repair model can achieve good repair results. The preprocessing is used to separate the audio data and video data in the damaged audio and video and convert them into a form that is easy for neural network processing, that is, to obtain the preprocessed audio data and preprocessed video data.

[0030] In step 102, audio temporal features are obtained based on the preprocessed audio data, video spatial features are obtained based on the preprocessed video data, and a joint feature vector is obtained based on the audio temporal features and the video spatial features.

[0031] Furthermore, in this embodiment, preprocessed audio data and preprocessed video data are respectively input into corresponding neural networks to extract corresponding features from the preprocessed audio data and preprocessed video data to obtain the audio temporal features and video spatial features. The audio temporal features and video spatial features are concatenated and weighted summed to obtain a joint feature vector. The joint feature vector can reflect the complete features of the corresponding damaged audio and video. Subsequently, the joint feature vector is input into the corresponding model for evaluation, definition and repair to obtain the evaluation result and repair result corresponding to the joint feature vector.

[0032] In step 103, the joint feature vector is evaluated to obtain the damage assessment result. Based on the damage assessment result, the repair model is adaptively adjusted to obtain the optimized repair model.

[0033] Furthermore, in this embodiment, the repair model needs to be adjusted specifically according to the degree of damage to the damaged audio and video, i.e., adaptive adjustment. For example, when the degree of damage to the damaged audio and video is low, the complexity of the model needs to be reduced to improve the repair efficiency. When the degree of damage to the damaged audio and video is high, the complexity of the model needs to be increased to obtain an optimized repair model to adapt to repairing the corresponding damage problem.

[0034] Furthermore, in this embodiment, after obtaining the optimized repair model, the repaired data can be obtained through the optimized repair model. Therefore, this embodiment also involves the following steps.

[0035] In step 104, the joint feature vector is input into the optimized repair model, and the repair result is output. The optimized repair model is adjusted based on the repair result, and the repair model is trained through multiple iterations.

[0036] The repair result is the data corresponding to the repaired audio and video data. After obtaining the repair result, the repair effect can be judged by comparing the undamaged data of the damaged audio and video with the repair result, so as to adjust the repair model, which is the feedback adjustment, thereby improving the performance of the repair model.

[0037] In step 105, the trained repair model is simplified and optimized, and the simplified and optimized repair model is deployed to repair the audio and video.

[0038] After the repair model is trained, in order to reduce the performance consumption of the repair model, this embodiment also needs to simplify and optimize the trained repair model, that is, remove the redundant parts in the trained repair model to obtain a relatively simplified model, thereby reducing the size of the repair model. Based on this, deployment can reduce the performance consumption of the repair model.

[0039] In summary, this embodiment extracts audio temporal features and video spatial features from damaged audio and video, and obtains a joint feature vector based on the audio temporal features and the video spatial features. The joint feature vector is evaluated to obtain a damage assessment result. The repair model is adaptively adjusted based on the damage assessment result to obtain an optimized repair model. The optimized repair model outputs a repair result, which is then used to provide feedback and adjust the repair model. After multiple iterations, a trained repair model is obtained and optimized for deployment. This improves the performance and accuracy of the repair model while reducing its occupancy, thereby improving the repair effect and versatility for damaged audio and video.

[0040] Furthermore, in this embodiment, the damaged audio and video need to be preprocessed to divide them into preprocessed audio data and preprocessed video data, which are then used for feature extraction using corresponding models. Therefore, this embodiment also involves the following design: the preprocessing of the damaged audio and video to obtain preprocessed audio data and preprocessed video data, such as... Figure 2 As shown, the method flow includes the following.

[0041] In step 201, the audio data and video data from the damaged audio and video are obtained respectively.

[0042] In this embodiment, the audio data and video data in the damaged audio and video are separated to obtain the audio data and video data.

[0043] In step 202, the video data is decoded to obtain a frame-by-frame image sequence, the frame-by-frame image sequence is normalized, and the pixel values ​​in the frame-by-frame image sequence are mapped to a preset range to obtain the preprocessed video data.

[0044] In this embodiment, the preset interval is set by those skilled in the art according to the actual situation, and the preset interval can be [0,1].

[0045] In step 203, the sampling rate and quantization bit depth of the audio data are standardized, and the standardized audio data is cut into multiple audio segments of a preset length. All audio segments are then filled, and all the filled audio segments are used together as the preprocessed audio data.

[0046] In this embodiment, the preset length is designed by those skilled in the art based on actual conditions. Since the audio segment data obtained after segmenting the audio data may be incomplete, padding of all audio segment data is used to ensure the integrity of the audio segment data.

[0047] Furthermore, after obtaining the preprocessed audio and video data, feature extraction is required for subsequent identification and output of repaired content using the repair model. Therefore, this embodiment also involves the following design: obtaining audio temporal features from the preprocessed audio data, obtaining video spatial features from the preprocessed video data, and obtaining a joint feature vector from the audio temporal features and the video spatial features, such as... Figure 3 As shown, the method flow includes...

[0048] In step 301, the preprocessed audio data is input into a recurrent neural network to obtain audio temporal features.

[0049] In this embodiment, the recurrent neural network can be a long short-term memory network or a gated recurrent unit, etc. The above-mentioned neural network can effectively capture long-term dependencies in audio signals, such as the semantic coherence of speech and the melodic direction of music.

[0050] In step 302, the preprocessed video data is input into a convolutional neural network to obtain video spatial features.

[0051] In this embodiment, the convolutional neural network can be VGGNet or ResNet, etc., which can be fine-tuned according to the needs of audio and video data restoration. Through multi-layer convolution and pooling operations, the convolutional neural network is suitable for spatial structure restoration of images and video frames. It can gradually extract the spatial features of video frames, including the bottom edge and texture features, and the high-level object structure and scene features. It infers the content of the missing area from the known area of ​​the image, fills the damaged area with context information, captures motion information between adjacent frames, and improves the dynamic scene restoration effect.

[0052] In summary, for missing parts in audio segments, the missing content is predicted based on the temporal patterns of preceding and following audio. For missing parts in video segments, temporal information from each frame (such as the motion trends of adjacent frames) can be passed through hidden layers, solving the problem of long-range dependencies that convolutional neural networks struggle to capture. By combining the advantages of convolutional neural networks and recurrent neural networks, multimodal feature fusion of audio and video is achieved, resulting in a more comprehensive and accurate representation of audio and video content, providing rich feature information for subsequent restoration.

[0053] In step 303, the audio temporal features are fused together according to attention weights to obtain the joint feature vector.

[0054] Obtain the audio attention weights corresponding to the audio temporal features, and obtain the video attention weights corresponding to the video spatial features; The expression for the audio attention weight is: =σ( × + ); The expression for the video attention weight is: =1- ; in, For audio attention weights, Here, σ represents the video attention weights, and σ is the Sigmoid activation function. For trainable video weight matrices, A vector representing the spatial features of the video. It is a vector of audio temporal features.

[0055] The joint feature vector is obtained based on audio temporal features, audio attention weights, video spatial features, and video attention weights. Specifically, the audio temporal features and video spatial features are weighted and summed according to the audio attention weights and video attention weights, respectively, to obtain the joint feature vector.

[0056] Furthermore, in this embodiment, considering that the repair model is pre-adjusted according to the degree of damage of the audio and video data for different levels of damage, so that the repair model is more suitable for the audio and video data and improves the repair performance, this embodiment also involves the following design: The joint feature vector is evaluated to obtain the damage assessment result, such as... Figure 4 As shown, the method flow includes...

[0057] In step 401, the corresponding frame rate change features, image anomaly features, frequency distribution features, and noise level features are obtained based on the joint feature vector.

[0058] In this embodiment, the joint feature vector needs to be input into the pre-trained damage analysis model to evaluate the joint feature vector. During the evaluation process, frame rate change features, screen anomaly features, frequency distribution features, and noise level features need to be extracted from the joint feature vector, and the evaluation is performed on different dimensions based on the above features.

[0059] In step 402, the damage level is determined in the pixel entropy dimension based on the frame rate change characteristics and the abnormal image characteristics, and the damage level in the pixel entropy dimension is obtained.

[0060] In this embodiment, the frame rate change features and image anomaly features are video data-related features, and the pixel entropy dimension is a video data-related evaluation dimension used to determine the severity of video data corruption. In this embodiment, the corruption level on the pixel entropy dimension Ep can be divided into three levels: Ep < 1.5 is corruption level L1, 1.5 ≤ Ep ≤ 3.0 is corruption level L2, and Ep > 3.0 is corruption level L3.

[0061] In step 403, the damage level is determined in the signal-to-noise ratio dimension based on the frequency distribution characteristics and noise level characteristics, thus obtaining the damage level in the signal-to-noise ratio dimension.

[0062] In this embodiment, the frequency distribution characteristics and noise level characteristics are features related to the audio data, and the signal-to-noise ratio (SNR) dimension is an evaluation dimension related to the audio data, used to determine the severity of audio data corruption. In this embodiment, the corruption level on the SNR dimension can be divided into three levels: SNR>25dB is corruption level L1, 15dB≤SNR≤25dB is corruption level L2, and SNR<15dB is corruption level L3.

[0063] In step 404, the damage assessment result corresponding to the joint feature vector is obtained based on the damage level in the pixel entropy dimension and the damage level in the signal-to-noise ratio dimension.

[0064] Furthermore, in this embodiment, after obtaining the damage assessment results, the repair model can be adjusted according to the damage assessment results to make the repair model more suitable for the corresponding damaged audio and video, thereby improving the repair performance. Therefore, this embodiment also involves the following design: the repair model is adaptively adjusted according to the damage assessment results to obtain an optimized repair model, such as... Figure 5 As shown, it includes: In step 501, the proportion of loss types is calculated using a two-dimensional weighting method of damage level weight and modality basis weight based on the joint feature vector.

[0065] The expression for calculating the proportion of the loss type is as follows: = ×100%; in, The proportion of the i-th type of loss. For the damage level weight of the i-th type of loss, represents the modal basis weights corresponding to the i-th type of loss.

[0066] In step 502, the number of convolutional kernels is dynamically adjusted according to the proportion of loss weights.

[0067] The expression for calculating the number of convolutional kernels is as follows: =r( × ); in, The number of convolutional kernels allocated to the i-th type of loss. This represents the total number of convolutional kernels in the current convolutional layer.

[0068] In step 503, the optimal number of iterations is matched based on the number of convolution kernels.

[0069] The expression for the optimal number of iterations is: ; in, is the optimal number of iterations, and k is the scaling factor.

[0070] In step 504, the optimized repair model is obtained by iterative optimization based on the optimal number of iterations.

[0071] For example, if the video is judged to be damaged by mild noise interference, the number of convolutional kernels in the convolutional layer of the repair model can be reduced to decrease the model complexity and improve the repair speed; if the data block is severely lost or damaged, the number of training iterations of the repair model can be increased to enhance the model's ability to recover missing information.

[0072] In summary, the above steps involve pre-analyzing damaged audio and video using deep learning algorithms to automatically determine the type of damage (such as noise interference, data block loss, and abnormal frame rate) and its severity (e.g., mild, moderate, and severe). Based on the analysis results, the parameters and repair strategies of the repair model are dynamically adjusted. For example, for mildly damaged audio and video, a simple and fast repair algorithm is used to reduce computation and improve repair efficiency; for severely damaged audio and video, the model complexity and the number of training iterations are increased to recover as much detailed information as possible and ensure the repair effect.

[0073] Furthermore, once the optimized repair model is obtained, the joint feature vector can be input into the optimized repair model to obtain the repair result. The optimized repair model is then adjusted based on the repair result to complete one round of training. The corresponding design is as follows: The optimized repair model is adjusted based on the repair result, such as... Figure 6 As shown, the method flow includes the following.

[0074] In step 601, the reference results corresponding to the damaged audio and video are obtained.

[0075] The reference results are the data of the corresponding damaged audio and video in the theoretically undamaged state, and can be regarded as the result of perfect restoration.

[0076] In step 602, the reference result and the repair result are compared.

[0077] In step 603, when the loss value of any mode decreases by less than a preset threshold for a consecutive preset number of rounds, or when the loss value of any mode decreases but the loss value of another mode increases, feedback adjustment is performed.

[0078] Wherein, the loss value of any modality decreases by less than a preset threshold for a consecutive preset number of rounds: ; Let be the loss value of the k-th mode in the i-th iteration. Let be the loss value of the k-th mode in the (i-1)-th iteration. For the preset threshold, The reduction range is defined as follows: In this embodiment, the preset threshold is set by those skilled in the art based on actual conditions. Both the preset number of rounds and the preset threshold are set by those skilled in the art based on actual conditions.

[0079] When the loss value of one mode decreases but the loss value of another mode increases, cross-modal interference occurs, which is: =( >0)∧( >0),( ≠ 2); in, Let $\frac{i}{k}$ be the loss value of the $k$-th mode during the $i$-th iteration. This is a preset threshold for the magnitude of the loss reduction.

[0080] In step 604, the feedback adjustment includes: for modes in a stagnant state, increasing the damage level weight and the basic weight of the corresponding mode by a first preset value; increasing the loss weight and the basic weight of the mode for modes with increasing loss values ​​by a second preset value; and decreasing the loss weight and the basic weight of the mode for modes with decreasing loss values ​​by a third preset value, so as to suppress the gradient interference of modes with decreasing loss values ​​on modes with increasing loss values.

[0081] In this embodiment, the first preset value, the second preset value, and the third preset value are all set by those skilled in the art according to the actual situation.

[0082] In step 605, the adjusted video loss weights and audio loss weights are normalized to ensure that the sum of the video loss weights and audio loss weights is 1, so as to avoid the imbalance of weights that may lead to deviations in the restoration effect.

[0083] In this embodiment, throughout the training process, the training intensity and learning rate of the repair model and the damage analysis model are continuously adjusted to ensure that the adversarial relationship between the two is balanced, thereby improving the performance of the repair model.

[0084] In summary, by introducing a restoration model and a damage analysis model, the restoration model is responsible for generating restored audio and video data (i.e., the restoration result) based on the characteristics of the damaged audio and video. The restoration model employs an encoder-decoder structure with skip connections. This structure fully utilizes the low-level details of the damaged audio and video during the restoration process, avoiding information loss and improving the realism and accuracy of the restoration result. The damage analysis model compares the restoration result output by the restoration model with the reference result. Through continuous adversarial training, it guides the restoration model to generate restoration results that are closer to reality, thereby improving the overall restoration quality.

[0085] Furthermore, after the repair model is trained, to reduce the space occupied by the model and improve its versatility, it needs to be optimized to obtain a more streamlined model before deployment. Therefore, this embodiment also involves the following design: the streamlined optimization of the trained repair model, such as... Figure 7 As shown, the method flow includes the following.

[0086] In step 701, the contribution of each connection relationship and the contribution of each parameter in the repair model are obtained.

[0087] The parameters include the kernel weights of the convolutional layers and the weight parameters of the attention mechanism. The connectivity relationships include the connectivity relationships between neurons in the convolutional layers, the fusion connectivity relationships between audio feature branches and video feature branches, and the feature connectivity relationships of the multi-scale feature fusion layer.

[0088] In step 702, the absolute weight percentage of each parameter in the repair model is calculated, the gradient magnitude percentage of each connection in the repair model is calculated, and the contribution is obtained based on the absolute weight percentage and the gradient magnitude percentage.

[0089] Calculate the percentage of the absolute weight of the parameter, i.e. Calculate the gradient magnitude ratio for each type of connection relationship, i.e. The contribution is obtained by weighted summation. =0.4 +0.6 .

[0090] In step 703, a contribution threshold is set, and redundant connections and invalid parameters are removed based on the contribution threshold to obtain a pre-simplified, optimized, and repaired model.

[0091] In this embodiment, connections with a contribution value less than the contribution value threshold are treated as redundant links, and parameters with a contribution value less than the contribution value threshold are treated as invalid parameters.

[0092] In this embodiment, the contribution can be the absolute value of the weight. Convolutional channels with a weight absolute value less than 2% can be removed by channel pruning. The pruning operation can be performed using progressive sparse training, and the accuracy of the repaired model decreases by less than or equal to 2% after pruning.

[0093] Specifically, the main layer of the repair model can use FP16 precision, while the output layer of the damaged analysis model can use INT8 precision. By converting the floating-point type of the pre-simplified and optimized repair model from 32 bits to 8 bits, low-precision calculations are used during inference, thereby improving the model's running speed and efficiency.

[0094] Finally, the streamlined and optimized repair model and damage analysis model are deployed to different application platforms, such as video editing software, online video playback platforms, and media players on mobile devices, to achieve real-time repair of various damaged audio and video.

[0095] In summary, the above design steps, considering the requirements for repair efficiency and equipment resources in practical applications, employ pruning algorithms to remove redundant connections and parameters in the neural network, reducing the model's computational load and storage space usage. Quantization techniques are used to convert model parameters from high-precision data types to low-precision types, accelerating the model's inference process with almost no loss in performance. Simultaneously, a lightweight network structure is designed to reduce model complexity while maintaining repair quality, enabling efficient operation on resource-constrained platforms such as mobile devices and embedded systems, thus broadening application scenarios.

[0096] Example 2: In this embodiment, based on embodiment 1, as follows: Figure 8 As shown, an audio and video restoration device is provided, including a preprocessing module, a multimodal fusion module, a restoration module, a real-time monitoring module, an adaptive adjustment module, and a lightweight engine model.

[0097] The system comprises the following modules: a preprocessing module for preprocessing damaged audio and video data to obtain preprocessed audio and video data; a multimodal fusion module for obtaining audio temporal features from the preprocessed audio data, video spatial features from the preprocessed video data, and a joint feature vector from the audio temporal features and video spatial features; an adaptive adjustment module for evaluating the joint feature vector to obtain a damage assessment result, and adaptively adjusting the repair model based on the damage assessment result to obtain an optimized repair model; a repair module for receiving the joint feature vector output to obtain a repair result, and adjusting the optimized repair model based on the repair result through multiple iterations for self-training, wherein the repair module adopts an encoder-decoder structure with skip connections; a lightweight engine model for simplifying and optimizing the trained repair model; and a real-time monitoring module for detecting the audio-visual synchronization offset during repair, for example, triggering a re-repair when the audio-visual synchronization offset is greater than or equal to 50ms.

[0098] Furthermore, the adaptive adjustment module includes a damage analysis submodule and a strategy matching unit, wherein the damage analysis submodule is used to calculate pixel entropy and signal-to-noise ratio, and the strategy matching unit is used to adjust the repair model according to different damage assessment results.

[0099] Furthermore, the multimodal fusion module includes a video feature extraction unit, an audio feature extraction unit, and a cross-modal alignment unit. The video feature extraction unit contains a convolutional neural network with residual connections, the audio feature extraction unit can be a bidirectional gated recurrent unit, and the cross-modal alignment unit is used to calculate the feature mapping matrix based on cosine similarity.

[0100] Example 3: Based on the AI-based audio and video restoration method provided in the foregoing embodiments, the present invention also provides an AI-based audio and video restoration apparatus that can be used to implement the above method, such as... Figure 9 The diagram shown is a schematic representation of the device architecture according to an embodiment of the present invention. The AI-based audio and video restoration device of this embodiment includes one or more processors 41 and a memory 42. Figure 9 Take a processor 41 as an example.

[0101] Processor 41 and memory 42 can be connected via a bus or other means. Figure 9 Taking the example of a connection between China and Israel via a bus.

[0102] The memory 42, as a non-volatile computer-readable storage medium for an AI-based audio and video restoration method, can be used to store non-volatile software programs and non-volatile computer-executable programs, such as the AI-based audio and video restoration method in the aforementioned embodiments. The processor 41 executes various functional applications and data processing of the AI-based audio and video restoration device by running the non-volatile software programs, instructions, and modules stored in the memory 42, thereby implementing the AI-based audio and video restoration method of the aforementioned embodiments.

[0103] Memory 42 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, memory 42 may include memory remotely located relative to processor 41, which can be connected to processor 41 via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0104] The program instructions / modules are stored in memory 42. When executed by one or more processors 41, they perform the AI-based audio and video restoration method described in the foregoing embodiments, for example, the method described above. Figures 1-7 The steps shown.

[0105] This invention also provides a non-volatile computer storage medium storing computer-executable instructions that are executed by one or more processors, for example... Figure 9 One of the processors 41 enables the one or more processors to execute the AI-based audio and video restoration method in the foregoing embodiments, for example, to perform the steps shown in the figures described above.

[0106] It is worth noting that the information interaction and execution process between the modules and units in the above-mentioned device and system are based on the same concept as the processing method embodiment of the present invention. For details, please refer to the description in the method embodiment of the present invention, and will not be repeated here.

[0107] Those skilled in the art will understand that all or part of the steps in the various methods of the embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0108] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An artificial intelligence-based audio and video restoration method, characterized by, include: The damaged audio and video data are preprocessed to obtain preprocessed audio data and preprocessed video data. Audio temporal features are obtained from the preprocessed audio data, video spatial features are obtained from the preprocessed video data, and a joint feature vector is obtained from the audio temporal features and the video spatial features. The joint feature vector is evaluated to obtain a damage assessment result. The repair model is then adaptively adjusted based on the damage assessment result to obtain an optimized repair model. The joint feature vector is input into the optimized repair model, and the repair result is output. The optimized repair model is adjusted based on the repair result, and the repair model is trained through multiple iterations. The trained restoration model is then streamlined and optimized, and the streamlined and optimized restoration model is deployed to restore audio and video. 2.The AI-based audio and video restoration method of claim 1, wherein, The preprocessing of damaged audio and video data to obtain preprocessed audio data and preprocessed video data specifically includes: Obtain the audio and video data from the damaged audio and video separately; The video data is decoded to obtain a frame-by-frame image sequence. The frame-by-frame image sequence is normalized and the pixel values ​​in the frame-by-frame image sequence are mapped to a preset range to obtain the preprocessed video data. The sampling rate and quantization bit depth of the audio data are standardized, and the standardized audio data is then cut into multiple audio segments of a preset length. All audio segments are then padded, and all padded audio segments are used together as the preprocessed audio data. 3.The AI-based audio and video restoration method of claim 1, wherein, The step of obtaining audio temporal features based on the preprocessed audio data, obtaining video spatial features based on the preprocessed video data, and obtaining a joint feature vector based on the audio temporal features and the video spatial features specifically includes: The preprocessed audio data is input into a recurrent neural network to obtain audio temporal features; The preprocessed video data is input into a convolutional neural network to obtain video spatial features; The audio temporal features and the video spatial features are fused according to the corresponding attention weights to obtain the joint feature vector.

4. The AI-based audio and video restoration method of claim 3, wherein, The process of fusing the audio temporal features and the video spatial features according to corresponding attention weights to obtain the joint feature vector specifically includes: Obtain the audio attention weights corresponding to the audio temporal features, and obtain the video attention weights corresponding to the video spatial features; The expression for the audio attention weight is: = σ( × + ); The expression for the video attention weight is: =1- ； in, For audio attention weights, Here, σ represents the video attention weights, and σ is the Sigmoid activation function. For trainable video weight matrices, A vector representing the spatial features of the video. A vector representing the temporal features of the audio. The joint feature vector is obtained based on audio temporal features, audio attention weights, video spatial features, and video attention weights. 5.The AI-based audio and video restoration method of claim 1, wherein, The evaluation of the joint feature vector to obtain the damage assessment result specifically includes: Based on the joint feature vector, the corresponding frame rate change features, image anomaly features, frequency distribution features, and noise level features are obtained; The damage level is determined in the pixel entropy dimension based on the frame rate change characteristics and image anomaly characteristics. Based on the frequency distribution characteristics and noise level characteristics, the damage level is determined in the signal-to-noise ratio dimension to obtain the damage level in the signal-to-noise ratio dimension. The damage assessment result corresponding to the joint feature vector is obtained based on the damage level in the pixel entropy dimension and the damage level in the signal-to-noise ratio dimension. 6.The AI-based audio and video restoration method of claim 1, wherein, The step of adaptively adjusting the repair model based on the damage assessment results to obtain an optimized repair model specifically includes: The proportion of loss types is calculated using a two-dimensional weighting method based on the joint feature vector, which combines damage level weights and modality basis weights. The number of convolutional kernels is dynamically adjusted based on the weight of the loss function. Match the optimal number of iterations based on the number of convolutional kernels; The optimized repair model is obtained by iterative optimization based on the optimal number of iterations. 7.The AI-based audio and video restoration method of claim 1, wherein, The step of adjusting the optimized repair model based on the repair results specifically includes: Obtain the reference results corresponding to the damaged audio and video; Compare the reference results with the repair results; Feedback adjustment is performed when the loss value of any mode decreases by less than a preset threshold for a consecutive preset number of rounds, or when the loss value of one mode decreases but the loss value of another mode increases. The feedback adjustment includes: for modes in a stagnant state, increasing the damage level weight and the basic weight of the corresponding mode by a first preset value; increasing the loss weight and the basic weight of the mode with increasing loss value by a second preset value; and decreasing the loss weight and the basic weight of the mode with decreasing loss value by a third preset value, so as to suppress the gradient interference of the mode with decreasing loss value on the mode with increasing loss value. The adjusted video loss weights and audio loss weights are normalized to ensure that the sum of the video loss weights and audio loss weights is 1, thus avoiding weight imbalance that could lead to deviations in the restoration effect. 8.The AI-based audio and video restoration method of claim 1, wherein, The process of streamlining and optimizing the trained repair model specifically includes: Obtain the contribution of each connection relationship and the contribution of each parameter in the repair model; Calculate the percentage of the absolute weight of each parameter in the repair model, calculate the percentage of the gradient magnitude of each connection in the repair model, and obtain the contribution based on the percentage of the absolute weight and the percentage of the gradient magnitude. A contribution threshold is set, and redundant connections and invalid parameters are removed based on the contribution threshold to obtain a pre-simplified, optimized, and repaired model.

9. An artificial intelligence-based audio and video restoration apparatus, characterized by, The method includes at least one processor and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the processor for performing the artificial intelligence-based audio and video restoration method according to any one of claims 1-8.

10. A non-transitory computer storage medium, comprising, The computer storage medium stores computer program instructions that, when executed by one or more processors, implement the AI-based audio and video restoration method as described in any one of claims 1-8.

Images