Audio and video data processing method and device, equipment and storage medium

Abstract

The invention relates to the field of data processing, in particular to an audio and video data processing method and device, equipment and a storage medium. The method comprises the following steps: acquiring to-be-processed audio data and to-be-processed video data; performing multi-scale wavelet transform decomposition on the to-be-processed audio data, performing nonlinear mapping reconstruction, and constructing distortion correction audio data; carrying out multi-time-window segmentation processing on the distortion correction audio data, carrying out time sequence characteristic evolution fitting, and constructing an audio time sequence semantic characteristic graph; performing local contrast enhancement processing on the to-be-processed video data to generate a local enhanced video; performing super-resolution fusion reconstruction on the local enhanced video to construct a super-definition reconstructed video; frame-by-frame time sequence decomposition is carried out on the super-definition reconstructed video, optical flow displacement tracking between adjacent frames is carried out, and a dynamic video frame sequence is constructed. According to the invention, the time synchronization of audio and video data is improved, and the time deviation delay of audio and video is optimized.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a method, device, equipment and storage medium for processing audio and video data. Background Art

[0002] With the rapid development of multimedia technology, audio and video data have been widely used in various application scenarios, such as smart home, smart security, virtual reality, etc. However, in actual application, there are often deviations and inconsistencies in time synchronization between audio and video data. This is caused by mismatched audio and video acquisition device parameters, differences in transmission channels, and imperfect processing procedures. This audio and video time asynchrony will seriously affect the user experience and bring challenges to subsequent audio and video analysis and applications.

[0003] Traditional audio and video synchronization correction methods usually rely on manual observation and manual adjustment. This method is inefficient and it is difficult to accurately eliminate time deviation. Therefore, there is an urgent need for an intelligent audio and video data processing method to achieve automation and optimization of audio and video time synchronization. Summary of the invention

[0004] In order to solve the above technical problems, the present invention proposes a method, device, equipment and storage medium for processing audio and video data to solve at least one of the above technical problems.

[0005] To achieve the above object, the present invention provides a method for processing audio and video data, comprising the following steps:

[0006] Step S1: obtaining audio data to be processed and video data to be processed; performing multi-scale wavelet transform decomposition on the audio data to be processed, and performing nonlinear mapping reconstruction to construct distortion-corrected audio data;

[0007] Step S2: performing multi-time window segmentation processing on the distortion-corrected audio data, and performing time series feature evolution fitting to construct an audio time series semantic feature graph;

[0008] Step S3: performing local contrast enhancement processing on the video data to be processed to generate a local enhanced video; performing super-resolution fusion reconstruction on the local enhanced video to construct an ultra-clear reconstructed video;

[0009] Step S4: decomposing the ultra-high-definition reconstructed video frame by frame, and tracking the optical flow displacement between adjacent frames to construct a dynamic video frame sequence;

[0010] Step S5: Based on the audio temporal semantic feature graph, the dynamic video frame sequence is positioned and matched frame by frame, and fine positioning and embedding are performed to construct an initial audio embedded video;

[0011] Step S6: Calculate the audio time offset of the initial audio-embedded video to obtain the audio delay value; perform global delay fine-tuning based on the audio delay value, and construct a global delay optimized video to complete the processing of audio and video data.

[0012] The present invention also provides a device for processing audio and video data, comprising:

[0013] A nonlinear reconstruction module is used to obtain the audio data to be processed and the video data to be processed; multi-scale wavelet transform is performed on the audio data to be processed, and nonlinear mapping reconstruction is performed to construct distortion-corrected audio data;

[0014] The semantic feature module is used to perform multi-time window segmentation processing on the distortion-corrected audio data, perform time series feature evolution fitting, and construct an audio time series semantic feature graph;

[0015] The image enhancement module is used to perform local contrast enhancement processing on the video data to be processed to generate a local enhanced video; and to perform super-resolution fusion reconstruction on the local enhanced video to construct an ultra-clear reconstructed video;

[0016] The optical flow displacement tracking module is used to decompose the ultra-high-definition reconstructed video frame by frame, and to track the optical flow displacement between adjacent frames to construct a dynamic video frame sequence;

[0017] A fine positioning embedding module is used to perform frame-by-frame positioning matching of dynamic video frame sequences based on audio temporal semantic feature graphs, and to perform fine positioning embedding to construct an initial audio embedded video;

[0018] The delay optimization module is used to calculate the audio time offset of the initial audio embedded in the video to obtain the audio delay value; based on the audio delay value, the global delay is fine-tuned to construct a global delay optimized video to complete the processing of audio and video data.

[0019] The present invention also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of any of the above-mentioned methods for processing audio and video data when executing the computer program.

[0020] The present invention also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the method for processing audio and video data described in any one of the above are implemented.

[0021] The beneficial effects of the present invention are as follows: decomposing audio data through multi-scale wavelet transform to capture signal features at different time scales; reconstructing through nonlinear mapping to correct distortion in audio data to improve the quality and accuracy of audio data; performing multi-time window segmentation processing on distortion-corrected audio data to capture audio features in different time periods; constructing an audio time series semantic feature map through time series feature evolution fitting to gain a deeper understanding and analysis of audio data; enhancing the local contrast of the video to improve the clarity and detail of the video image; and reconstructing through super-resolution fusion to improve the clarity and detail restoration capability of the video, thereby generating high-quality ultra-clear images. Reconstruct the video, perform frame-by-frame temporal decomposition of the ultra-high-definition reconstructed video to capture the dynamic changes between video frames, reveal the motion trajectory and dynamic change information of objects in the video through optical flow displacement tracking between adjacent frames, and perform frame-by-frame positioning and matching of dynamic video frame sequences based on audio timing semantic feature maps to achieve accurate matching and alignment of audio and video data. Through fine-grained positioning and embedding, embed audio information into video data to achieve a closer association between audio and video data, calculate the audio time offset value to solve the problem of timing asynchrony of audio and video data, and optimize the timing relationship of audio and video data through global delay fine-tuning to achieve better matching and coordinated operation of the overall audio and video data. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 A schematic diagram of the steps of a method for processing audio and video data according to the present invention;

[0023] Figure 2 Detailed implementation flow chart of step S1;

[0024] Figure 3 Detailed implementation flow chart of step S2;

[0025] Figure 4 Detailed implementation flow chart of step S3. DETAILED DESCRIPTION

[0026] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0027] The present application example provides a method, device, equipment and storage medium for processing audio and video data. The execution subjects of the method, device, equipment and storage medium for processing audio and video data include but are not limited to: mechanical equipment, data processing platform, cloud server node, network upload equipment, etc. equipped with the system can be regarded as the general computing node of the present application, and the data processing platform includes but is not limited to: at least one of an audio and image management system, an information management system, and a cloud data management system.

[0028] See also Figures 1 to 4The present invention provides a method for processing audio and video data, the method for processing audio and video data comprising the following steps:

[0029] Step S1: obtaining audio data to be processed and video data to be processed; performing multi-scale wavelet transform decomposition on the audio data to be processed, and performing nonlinear mapping reconstruction to construct distortion-corrected audio data;

[0030] Step S2: performing multi-time window segmentation processing on the distortion-corrected audio data, and performing time series feature evolution fitting to construct an audio time series semantic feature graph;

[0031] Step S3: performing local contrast enhancement processing on the video data to be processed to generate a local enhanced video; performing super-resolution fusion reconstruction on the local enhanced video to construct an ultra-clear reconstructed video;

[0032] Step S4: decomposing the ultra-high-definition reconstructed video frame by frame, and tracking the optical flow displacement between adjacent frames to construct a dynamic video frame sequence;

[0033] Step S5: Based on the audio temporal semantic feature graph, the dynamic video frame sequence is positioned and matched frame by frame, and fine positioning and embedding are performed to construct an initial audio embedded video;

[0034] Step S6: Calculate the audio time offset of the initial audio-embedded video to obtain the audio delay value; perform global delay fine-tuning based on the audio delay value, and construct a global delay optimized video to complete the processing of audio and video data.

[0035] Decompose the audio data through multi-scale wavelet transform to capture the signal characteristics at different time scales. Reconstruct through nonlinear mapping to correct the distortion in the audio data and improve the quality and accuracy of the audio data. Perform multi-time window segmentation on the distortion-corrected audio data to capture the audio characteristics in different time periods. Construct the audio time series semantic feature map through time series feature evolution fitting to gain a deeper understanding and analysis of the audio data, enhance the local contrast of the video, improve the clarity and detail of the video image, and reconstruct through super-resolution fusion to improve the clarity and detail restoration ability of the video, generate high-quality ultra-clear reconstructed video, and The ultra-high-definition reconstructed video is decomposed frame by frame to capture the dynamic changes between video frames. The motion trajectory and dynamic change information of objects in the video are revealed by tracking the optical flow displacement between adjacent frames. The dynamic video frame sequence is positioned and matched frame by frame based on the audio timing semantic feature map to achieve accurate matching and alignment of audio and video data. The audio information is embedded into the video data through fine positioning and embedding to achieve a closer association between the audio and video data. The audio time offset value is calculated to solve the timing synchronization problem of the audio and video data. The global delay fine-tuning is used to optimize the timing relationship of the audio and video data to achieve better matching and coordinated operation of the overall audio and video data.

[0036] In the embodiment of the present invention, refer to Figure 1 , is a flowchart of the steps of a method for processing audio and video data of the present invention. In this example, the steps of the method for processing audio and video data include:

[0037] Step S1: obtaining audio data to be processed and video data to be processed; performing multi-scale wavelet transform decomposition on the audio data to be processed, and performing nonlinear mapping reconstruction to construct distortion-corrected audio data;

[0038] In this embodiment, the original audio data and video data to be processed are obtained, and a multi-scale wavelet transform, such as discrete wavelet transform (DWT), is used to decompose the audio signal into sub-bands of different frequency scales to obtain wavelet coefficients at each scale. A nonlinear mapping function, such as a power function, a hyperbolic tangent function, etc., is designed to perform nonlinear transformation and reconstruction on the wavelet coefficients, thereby suppressing the distortion and noise components in the original audio.

[0039] Step S2: performing multi-time window segmentation processing on the distortion-corrected audio data, and performing time series feature evolution fitting to construct an audio time series semantic feature graph;

[0040] In this embodiment, according to the characteristics of the audio signal and the analysis requirements, the length of the segmentation window (such as 0.5 seconds, 1 second) and the overlap rate (such as 50%) are determined, and the audio processing library (such as l librosa or scipy) to segment the distortion-corrected audio data, segment the audio signal according to the set window length and overlap rate to obtain multiple time window segments, organize the segmented time window segments into a list or array for subsequent processing and analysis, select a suitable feature extraction method, such as short-time Fourier transform (STFT), Mel-frequency cepstral coefficient (MFCC), or time domain features (such as zero-crossing rate, energy, etc.), apply the selected feature extraction method to each time window segment, and calculate the feature vector of each segment. If MFCC is used, use librosa.feature.mfcc function to extract features, organize the feature vectors of all time window segments into a matrix, with rows representing time windows and columns representing feature dimensions, select a suitable time series fitting model, such as linear regression, support vector machine (SVM), neural network, etc., select a suitable model based on the complexity of the features and the amount of data, use the extracted time series feature data to train the selected model, fit the evolution curve of the audio features, and generate an audio time series semantic feature graph based on the fitting results. The graph shows the relationship between time and features, and uses visualization tools (such as Matplotlib ib) drawing a feature map for easy understanding and analysis, and saving the generated audio temporal semantic feature map as an image file (such as PNG or SVG) or storing it in a data structure for subsequent use.

[0041] Step S3: performing local contrast enhancement processing on the video data to be processed to generate a local enhanced video; performing super-resolution fusion reconstruction on the local enhanced video to construct an ultra-clear reconstructed video;

[0042] In this embodiment, local contrast enhancement techniques, such as adaptive histogram equalization (CLAHE), are used to perform local contrast enhancement on each frame of the video to improve the clarity and saliency of image details. Deep learning super-resolution fusion techniques, such as residual convolutional neural network (SRCNN), ESRGAN and other methods are used to upgrade low-resolution video frames to high resolution, and higher-resolution reconstruction is achieved through feature fusion, ultimately constructing ultra-definition reconstructed video data.

[0043] Step S4: decomposing the ultra-high-definition reconstructed video frame by frame, and tracking the optical flow displacement between adjacent frames to construct a dynamic video frame sequence;

[0044] In this embodiment, video frame segmentation technology is used to decompose the video data frame by frame along the time dimension, and independent video frame image data of each frame is extracted. An optical flow estimation algorithm, such as the Lucas-Kanade algorithm and the Farneback algorithm, is used to track the optical flow displacement between adjacent video frames, and the optical flow displacement vector of each pixel is calculated. Displacement vector data describing the dynamic optical flow changes between adjacent frames is generated, and each video frame is connected in time series according to the optical flow motion relationship to construct a dynamic video frame sequence reflecting continuous motion.

[0045] Step S5: Based on the audio temporal semantic feature graph, the dynamic video frame sequence is positioned and matched frame by frame, and fine positioning and embedding are performed to construct an initial audio embedded video;

[0046] In this embodiment, the constructed audio timing semantic feature map is aligned with the dynamic video frame sequence to establish a time synchronization relationship between the two. After the alignment, the similarity between each video frame and the audio timing semantic feature is calculated to obtain the semantic feature similarity data of each video frame. Combined with optimization algorithms, such as dynamic programming and hidden Markov models, the audio timing semantic features and the video frame sequence are finely positioned and matched to obtain the specific positioning data of the audio semantic features in the video frame sequence. According to the positioning data of the semantic features, they are accurately embedded into the corresponding video frames to construct a preliminary audio semantic embedded video.

[0047] Step S6: Calculate the audio time offset of the initial audio-embedded video to obtain the audio delay value; perform global delay fine-tuning based on the audio delay value, and construct a global delay optimized video to complete the processing of audio and video data.

[0048] In this embodiment, an audio synchronization detection algorithm, such as a correlation peak method, a phase difference method, etc., is used to analyze the time difference between the audio and video in a video frame, calculate the specific audio time offset of each video frame, obtain numerical data describing the audio delay state, perform global time calibration on the initial audio embedded in the video, and use technologies such as dynamic time warping (DTW) to dynamically insert or delete the audio signal, so that the time between the audio and video is completely coordinated, and construct a final video output with global delay optimization.

[0049] In this embodiment, refer to Figure 2 , is a flowchart of detailed implementation steps of step S1. In this embodiment, the detailed implementation steps of step S1 include:

[0050] Step S11: Obtaining audio and video data to be processed;

[0051] Step S12: performing deep semantic segmentation processing on the audio and video data to be processed, thereby extracting the audio data to be processed and the video data to be processed;

[0052] Step S13: performing multi-scale wavelet transform decomposition on the audio data to be processed to obtain audio information of different frequency bands;

[0053] Step S14: performing signal-to-noise ratio prediction calculation on audio information of different frequency bands to generate a signal-to-noise ratio for each frequency band;

[0054] Step S15: performing dynamic filtering optimization processing on audio information of different frequency bands based on the signal-to-noise ratio of each frequency band, thereby obtaining multiple noise reduction optimized audio signals;

[0055] Step S16: Perform nonlinear mapping reconstruction on multiple noise reduction optimized audio signals to construct distortion corrected audio data.

[0056] In this embodiment, the audio and video data to be analyzed are obtained through an acquisition device or a database, and a pre-trained deep learning semantic segmentation model, such as YOLO, Mask R-CNN, etc., is used to input the input audio and video data into the segmentation model for forward reasoning. The model will perform semantic-level segmentation and extraction on the input audio and video data, thereby successfully separating and extracting the audio data and the video data. The segmentation result will output two independent data sets: the audio data to be processed and the video data to be processed. The audio signal is multi-scale decomposed along the time dimension using discrete wavelet transform (DWT) or wavelet packet analysis and other technologies to obtain audio sub-signals in different frequency bands. These sub-signals contain frequency feature information of different granularities. Statistical analysis or machine learning methods, such as Gaussian process regression, are used to predict and calculate the signal-to-noise ratio of the audio signal in each frequency band. The signal-to-noise ratio reflects the severity of the noise in the audio signal in this frequency band. These signal-to-noise ratio data will provide a basis for subsequent dynamic filtering optimization. Adaptive filtering algorithms, such as Wiener filtering, Ka lman filtering, etc., dynamically filter the audio sub-signals in each frequency band, reduce noise interference in a targeted manner according to the signal-to-noise ratio of different frequency bands, and finally obtain multiple sections of noise-reduced and optimized audio signals. Use neural networks, Fourier transform and other nonlinear mapping technologies to correct distortion and reconstruct the optimized audio signals to generate high-quality noise-reduced and optimized audio data.

[0057] In this embodiment, refer to Figure 3 , is a flowchart of detailed implementation steps of step S2. In this embodiment, the detailed implementation steps of step S2 include:

[0058] Step S21: performing multi-time window segmentation processing on the distortion-corrected audio data to obtain audio data of multiple time windows;

[0059] Step S22: performing audio frequency analysis on the audio data of multiple time windows one by one, and extracting the frequency features of each time window;

[0060] Step S23: performing key semantic feature recognition on the audio data of multiple time windows to obtain key semantic features of multiple time periods;

[0061] Step S24: performing time series change trend analysis on the frequency characteristics of each time window to generate audio time series change trend data;

[0062] Step S25: performing time series feature evolution fitting on the audio time series change trend data to construct an audio time series trend curve;

[0063] Step S26: Perform dynamic semantic mapping on the audio time series trend curve based on multiple time period key semantic features to construct an audio time series semantic feature graph.

[0064] In this embodiment, a sliding window technology is used to divide the audio signal into multiple time windows, each time window contains a continuous audio data, so that the audio data is segmented in the time dimension. According to the audio data of each time window, Fourier transform or other spectrum analysis methods are used to extract the frequency characteristics of the audio signal in each time window, such as frequency distribution, frequency band energy and other indicators. Speech recognition and natural language processing technology are used to identify key semantic features in each time window, such as keywords, emotional tendencies, etc., to provide input for the construction of a time series semantic feature map. Based on the extracted frequency characteristics of each time window, combined with the time series information of the window, the change trend of each frequency feature over time is analyzed to generate trend data reflecting the law of audio time series change. Time series analysis, curve fitting and other technologies are used to construct a trend curve model that can describe the evolution law of audio time series features for the obtained time series change trend data. The key semantic features are used to dynamically associate the constructed time series trend curves, and a time series semantic feature map of the audio signal is constructed in the time dimension. The feature map can intuitively reflect the time series changes of the semantic content in the audio.

[0065] In this embodiment, refer to Figure 4 , is a flowchart of detailed implementation steps of step S3. In this embodiment, the detailed implementation steps of step S3 include:

[0066] Step S31: Performing edge contour visual recognition on the video data to be processed to extract video edge contour features;

[0067] Step S32: performing local contrast enhancement processing based on the edge contour features of the video to generate a local enhanced video;

[0068] Step S33: performing a multi-sampling convolution operation on the local enhanced video to generate a multi-scale feature map;

[0069] Step S34: performing texture detail generalization learning on the multi-scale feature map to obtain texture details at each scale;

[0070] Step S35: performing fully connected detail mapping on the multi-scale feature map according to the texture details of each scale to generate multiple texture detail feature maps;

[0071] Step S36: Perform super-resolution fusion reconstruction on multiple texture detail feature maps to construct an ultra-clear reconstructed video.

[0072] In this embodiment, a suitable edge detection algorithm is selected, such as Canny edge detection, Sobel operator or Lap Lacian operator, apply the selected edge detection algorithm to each frame of video, extract the edge contour features of the video, combine with contrast enhancement algorithm, such as histogram equalization, perform local contrast enhancement processing on the original video data to highlight the local details and texture information in the video, generate local contrast enhanced video data, design convolution kernels of various sizes to capture features of different scales, apply multiple convolution kernels to each frame of the locally enhanced video, perform multi-sampling convolution operations, generate multi-scale feature maps, select appropriate texture detail learning methods, such as convolutional neural network (CNN) or GAN, input the multi-scale feature map into the selected learning model, prepare for training, train the multi-scale feature map, extract the texture details of each scale, design a fully connected layer, connect the texture details of each scale with the corresponding multi-scale feature map, perform fully connected detail mapping on the multi-scale feature map, generate multiple texture detail feature maps, select appropriate super-resolution reconstruction algorithm, such as SRCNN, ESPCN or GAN, input multiple texture detail feature maps into the super-resolution model for fusion reconstruction, generate ultra-clear reconstructed video, visualize the reconstructed video to ensure that its quality and details meet the requirements.

[0073] In this embodiment, step S4 includes the following steps:

[0074] Step S41: decomposing the ultra-high-definition reconstructed video frame by frame, and extracting each video frame image;

[0075] Step S42: tracking the optical flow displacement between adjacent frames of each video frame image to generate an inter-frame dynamic optical flow displacement vector;

[0076] Step S43: performing continuous frame association mining on the inter-frame dynamic optical flow displacement vector to generate the continuous inter-frame optical flow motion relationship;

[0077] Step S44: Perform dynamic time series processing on each video frame image based on the optical flow motion relationship between consecutive frames to construct a dynamic video frame sequence.

[0078] In this embodiment, a video frame segmentation technique is used to decompose the video data frame by frame along the time dimension, and independent video frame image data of each frame is extracted. An optical flow estimation algorithm, such as the Lucas-Kanade algorithm and the Farneback algorithm, is used to track the optical flow displacement between adjacent video frames, calculate the optical flow displacement vector of each pixel, and generate displacement vector data describing the dynamic optical flow changes between adjacent frames. In combination with the time series analysis technique, the optical flow motion association between consecutive video frames is mined, and a model describing the optical flow change relationship between consecutive frames is established. Using the obtained optical flow motion relationship between consecutive frames, each extracted video frame image is dynamically time-sequenced in series according to the optical flow change relationship to generate a dynamic video frame sequence reflecting the continuous motion changes, which can better describe the dynamic detail information in the video.

[0079] In this embodiment, the specific steps of step S5 are:

[0080] Step S51: aligning the dynamic video frame sequence based on the audio time-series semantic feature graph, and calculating the audio semantic feature similarity to obtain the semantic feature similarity of each video frame;

[0081] Step S52: performing frame-by-frame positioning matching based on the semantic feature similarity of each video frame, thereby obtaining positioning data of the audio semantic feature;

[0082] Step S53: extracting semantic features of multiple audio segments based on the audio time-series semantic feature graph;

[0083] Step S54: finely locate and embed multiple audio semantic features into the dynamic video frame sequence according to the positioning data of the audio semantic features to construct an initial audio-embedded video.

[0084] In this embodiment, a temporal semantic feature graph is extracted from the audio signal, and a method such as MFCC and audio spectrogram is used to synchronize and align the dynamic video frame sequence with the audio temporal feature to ensure that each frame of video corresponds to the correct audio segment, and a similarity calculation method (such as cosine similarity and Euclidean distance) is used to calculate the similarity between each video frame and the corresponding audio feature, and a suitable positioning matching algorithm is selected, such as dynamic time warping (DTW) or simple threshold matching, and frame-by-frame positioning matching is performed on each video frame according to the similarity calculation result to determine the audio feature segment most similar to it, and the audio signal is segmented according to the timing graph of the audio feature, and multiple audio semantic features corresponding to the video frame are extracted, and the extracted multiple audio semantic features are sorted to ensure that each feature is consistent with the positioning data of the video frame, and a strategy for embedding audio into video is designed to determine how to embed the audio feature into the video frame (such as audio superposition and audio editing), and multiple audio semantic features are finely embedded into the corresponding dynamic video frame sequence according to the positioning data, and an initial audio embedded video is generated, and the generated initial audio embedded video is played and quality checked to ensure the synchronization and coordination of audio and video.

[0085] In this embodiment, the specific steps of step S6 are:

[0086] Step S61: performing frame-by-frame audio time deviation recognition on the initial audio-embedded video, and extracting audio time deviation video frames;

[0087] Step S62: performing audio time offset calculation on the audio time deviation video frame to obtain an audio delay value;

[0088] Step S63: performing motion feature distribution analysis on the audio time deviation video frame to obtain the video frame motion distribution feature;

[0089] Step S64: identifying the optimal time optimization point based on the video frame motion distribution characteristics to obtain the optimal delay insertion optimization point in the frame;

[0090] Step S65: Perform dynamic audio offset correction on the optimal delay insertion optimization point in the frame according to the audio delay value to obtain an audio offset correction frame;

[0091] Step S66: fine-tune the global delay of the initial audio-embedded video based on the audio offset correction frame, and construct a global delay optimized video to complete the processing of audio and video data.

[0092] In this embodiment, a suitable time deviation detection method is selected, such as delay estimation based on correlation analysis or signal processing, audio time deviation is identified for each frame, audio time deviation video frames with deviation are marked, the time difference between audio and video in the video frame is analyzed, the specific audio time offset of each target frame is calculated, and numerical data describing the audio delay state is obtained. The motion feature distribution information in each target frame, such as indicators such as motion direction and speed distribution, is extracted by using methods such as optical flow analysis. An optimization algorithm, such as genetic algorithm, particle swarm optimization, etc., is used to identify the optimal position in each target video frame that is most suitable for inserting audio delay, and the optimal delay insertion point that minimizes video distortion is obtained. A suitable audio offset correction method is selected, such as linear interpolation or spline interpolation. According to the calculated audio delay value, the optimal delay insertion optimization point is dynamically corrected for audio offset, and an audio offset correction frame is generated. The global delay fine-tuning strategy is determined to ensure the synchronization of audio and video, and the corresponding frames of the initial audio embedded in the video are selectively replaced to achieve global delay optimization fine-tuning, and finally a global delay optimized video with completely coordinated audio and video time is constructed.

[0093] In this embodiment, the present invention further provides an audio and video data processing device, including:

[0094] A nonlinear reconstruction module is used to obtain the audio data to be processed and the video data to be processed; multi-scale wavelet transform is performed on the audio data to be processed, and nonlinear mapping reconstruction is performed to construct distortion-corrected audio data;

[0095] The semantic feature module is used to perform multi-time window segmentation processing on the distortion-corrected audio data, perform time series feature evolution fitting, and construct an audio time series semantic feature graph;

[0096] The image enhancement module is used to perform local contrast enhancement processing on the video data to be processed to generate a local enhanced video; and to perform super-resolution fusion reconstruction on the local enhanced video to construct an ultra-clear reconstructed video;

[0097] The optical flow displacement tracking module is used to decompose the ultra-high-definition reconstructed video frame by frame, and to track the optical flow displacement between adjacent frames to construct a dynamic video frame sequence;

[0098] A fine positioning embedding module is used to perform frame-by-frame positioning matching of dynamic video frame sequences based on audio temporal semantic feature graphs, and to perform fine positioning embedding to construct an initial audio embedded video;

[0099] The delay optimization module is used to calculate the audio time offset of the initial audio embedded in the video to obtain the audio delay value; based on the audio delay value, the global delay is fine-tuned to construct a global delay optimized video to complete the processing of audio and video data.

[0100] The present invention also provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements any of the above-mentioned audio and video data processing methods when executing the computer program.

[0101] The present invention also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the audio and video data processing method described in any one of the above items are implemented.

[0102] Those skilled in the art clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described systems, systems and units refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0103] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it is stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or partly or all or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for a computer device (a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media for storing program codes such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0104] Therefore, the embodiments should be regarded as illustrative and non-restrictive from all points, and the scope of the present invention is limited by the appended claims rather than the above description, and it is therefore intended that all changes falling within the meaning and range of equivalent elements of the application documents are included in the present invention.

[0105] The above is only a specific embodiment of the present invention, so that those skilled in the art can understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but should conform to the widest scope consistent with the principles and novel features invented herein.

Claims

1. A method for processing audio and video data, characterized in that: The following steps are involved: Step S1: obtaining audio data to be processed and video data to be processed; performing multi-scale wavelet transform decomposition on the audio data to be processed, and performing nonlinear mapping reconstruction to construct distortion-corrected audio data; Step S2: performing multi-time window segmentation processing on the distortion-corrected audio data, and performing time series feature evolution fitting to construct an audio time series semantic feature graph; Step S3: performing local contrast enhancement processing on the video data to be processed to generate a local enhanced video; performing super-resolution fusion reconstruction on the local enhanced video to construct an ultra-clear reconstructed video; Step S4: decomposing the ultra-high-definition reconstructed video frame by frame, and tracking the optical flow displacement between adjacent frames to construct a dynamic video frame sequence; Step S5: Based on the audio temporal semantic feature graph, the dynamic video frame sequence is positioned and matched frame by frame, and fine positioning and embedding are performed to construct an initial audio embedded video; Step S6: Calculate the audio time offset of the initial audio embedded in the video to obtain an audio delay value; Fine-tune the global delay based on the audio delay value and build a global delay optimization video to complete the processing of audio and video data.

2. The method for processing audio and video data according to claim 1, characterized in that: The specific steps of step S1 are: Step S11: Obtaining audio and video data to be processed; Step S12: performing deep semantic segmentation processing on the audio and video data to be processed, thereby extracting the audio data to be processed and the video data to be processed; Step S13: performing multi-scale wavelet transform decomposition on the audio data to be processed to obtain audio information of different frequency bands; Step S14: performing signal-to-noise ratio prediction calculation on audio information of different frequency bands to generate a signal-to-noise ratio for each frequency band; Step S15: performing dynamic filtering optimization processing on audio information of different frequency bands based on the signal-to-noise ratio of each frequency band, thereby obtaining multiple noise reduction optimized audio signals; Step S16: Perform nonlinear mapping reconstruction on multiple noise reduction optimized audio signals to construct distortion corrected audio data.

3. The method for processing audio and video data according to claim 1, characterized in that: The specific steps of step S2 are: Step S21: performing multi-time window segmentation processing on the distortion-corrected audio data to obtain audio data of multiple time windows; Step S22: performing audio frequency analysis on the audio data of multiple time windows one by one, and extracting the frequency features of each time window; Step S23: performing key semantic feature recognition on the audio data of multiple time windows to obtain key semantic features of multiple time periods; Step S24: performing time series change trend analysis on the frequency characteristics of each time window to generate audio time series change trend data; Step S25: performing time series feature evolution fitting on the audio time series change trend data to construct an audio time series trend curve; Step S26: Perform dynamic semantic mapping on the audio time series trend curve based on multiple time period key semantic features to construct an audio time series semantic feature graph.

4. The method for processing audio and video data according to claim 1, characterized in that: The specific steps of step S3 are: Step S31: Performing edge contour visual recognition on the video data to be processed to extract video edge contour features; Step S32: performing local contrast enhancement processing based on the edge contour features of the video to generate a local enhanced video; Step S33: performing a multi-sampling convolution operation on the local enhanced video to generate a multi-scale feature map; Step S34: performing texture detail generalization learning on the multi-scale feature map to obtain texture details at each scale; Step S35: performing fully connected detail mapping on the multi-scale feature map according to the texture details of each scale to generate multiple texture detail feature maps; Step S36: Perform super-resolution fusion reconstruction on multiple texture detail feature maps to construct an ultra-clear reconstructed video.

5. The method for processing audio and video data according to claim 1, characterized in that: The specific steps of step S4 are: Step S41: decomposing the ultra-high-definition reconstructed video frame by frame, and extracting each video frame image; Step S42: tracking the optical flow displacement between adjacent frames of each video frame image to generate an inter-frame dynamic optical flow displacement vector; Step S43: performing continuous frame association mining on the inter-frame dynamic optical flow displacement vector to generate the continuous inter-frame optical flow motion relationship; Step S44: Perform dynamic time series processing on each video frame image based on the optical flow motion relationship between consecutive frames to construct a dynamic video frame sequence.

6. The method for processing audio and video data according to claim 1, characterized in that: The specific steps of step S5 are: Step S51: aligning the dynamic video frame sequence based on the audio time-series semantic feature graph, and calculating the audio semantic feature similarity to obtain the semantic feature similarity of each video frame; Step S52: performing frame-by-frame positioning matching based on the semantic feature similarity of each video frame, thereby obtaining positioning data of the audio semantic feature; Step S53: extracting semantic features of multiple audio segments based on the audio time-series semantic feature graph; Step S54: finely locate and embed multiple audio semantic features into the dynamic video frame sequence according to the positioning data of the audio semantic features to construct an initial audio-embedded video.

7. The method for processing audio and video data according to claim 1, characterized in that: The specific steps of step S6 are: Step S61: performing frame-by-frame audio time deviation recognition on the initial audio-embedded video, and extracting audio time deviation video frames; Step S62: performing audio time offset calculation on the audio time deviation video frame to obtain an audio delay value; Step S63: performing motion feature distribution analysis on the audio time deviation video frame to obtain the video frame motion distribution feature; Step S64: identifying the optimal time optimization point based on the video frame motion distribution characteristics to obtain the optimal delay insertion optimization point in the frame; Step S65: Perform dynamic audio offset correction on the optimal delay insertion optimization point in the frame according to the audio delay value to obtain an audio offset correction frame; Step S66: fine-tune the global delay of the initial audio-embedded video based on the audio offset correction frame, and construct a global delay optimized video to complete the processing of audio and video data.

8. A device for processing audio and video data, characterized in that: The method for processing audio and video data according to claim 1 comprises: A nonlinear reconstruction module is used to obtain the audio data to be processed and the video data to be processed; multi-scale wavelet transform is performed on the audio data to be processed, and nonlinear mapping reconstruction is performed to construct distortion-corrected audio data; The semantic feature module is used to perform multi-time window segmentation processing on the distortion-corrected audio data, perform time series feature evolution fitting, and construct an audio time series semantic feature graph; The image enhancement module is used to perform local contrast enhancement processing on the video data to be processed to generate a local enhanced video; and to perform super-resolution fusion reconstruction on the local enhanced video to construct an ultra-clear reconstructed video; The optical flow displacement tracking module is used to decompose the ultra-high-definition reconstructed video frame by frame, and to track the optical flow displacement between adjacent frames to construct a dynamic video frame sequence; A fine positioning embedding module is used to perform frame-by-frame positioning matching of dynamic video frame sequences based on audio temporal semantic feature graphs, and to perform fine positioning embedding to construct an initial audio embedded video; The delay optimization module is used to calculate the audio time offset of the initial audio embedded in the video to obtain the audio delay value; based on the audio delay value, the global delay is fine-tuned to construct a global delay optimized video to complete the processing of audio and video data.

9. A computer device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor executes the computer program, the steps of the method for processing audio and video data according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for processing audio and video data according to any one of claims 1 to 7 are implemented.

Images