Upmix processing method, system and device for stereo audio and video, and storage medium

Abstract

The invention provides an up-mixing processing method, system and device for a stereo audio video, and a storage medium, and relates to the technical field of audio and video processing, and the method comprises the steps: obtaining a to-be-processed stereo audio video, carrying out the multi-stream separation of the to-be-processed stereo audio video, obtaining a plurality of independent sound source components, the to-be-processed stereo audio video comprises a to-be-processed stereo audio or a to-be-processed stereo video; performing dry and wet sound processing on the to-be-processed stereo sound video to obtain an environment sound component; performing frequency band extension on the environment sound component to obtain a target environment sound component; and carrying out space rendering on each independent sound source component and the target environment sound component by adopting a vector-based amplitude translation method, and carrying out multichannel output fusion to obtain a multichannel audio / video. According to the method, the sound source is accurately positioned and efficiently separated, the dynamic sense and the space sense of the sound source object are enhanced, and the audio and video upmix processing quality and the user experience are improved.

Description

Technical Field

[0001] This invention relates to the field of audio and video processing technology, specifically to a method, system, device, and storage medium for upmixing stereo audio and video. Background Technology

[0002] Currently, there is a large amount of stereo material in audio and video resources, and the presentation of this material is not ideal in multi-speaker environments. Audio and video upmixing technology refers to the process of processing the original audio signal into an audio signal with more channels. By increasing the number of channels, the quality of the user's audiovisual experience can be significantly improved.

[0003] Currently, upmixing technology, which converts traditional stereo to multichannel audio, is widely used in cinema systems, home theaters, and broadcast television. Its core objective is to extract surround sound information from a limited stereo signal to enhance the sense of space. Traditional upmixing techniques are mainly based on simple algorithms, extracting surround information by analyzing phase differences and encoding it into the stereo signal. However, they generally suffer from the following limitations: poor sound source separation, fixed decoding rules, difficulty adapting to complex audio content, limited frequency domain and dynamic range, and a ultimately weak auditory effect. Therefore, how to effectively improve the upmixing effect of stereo audio-video processing to multichannel audio-video has become a key technical challenge that urgently needs to be solved. Summary of the Invention

[0004] In view of the above-mentioned shortcomings of the prior art, the present invention provides a method, system, device and storage medium for upmixing stereo audio and video, which effectively solves the problem of poor upmixing effect when processing stereo audio and video to multi-channel audio and video.

[0005] In a first aspect, the present invention provides a method for upmixing stereo audio and video, the method comprising: Obtain the stereo audio video to be processed, perform multi-stream separation on the stereo audio video to be processed to obtain multiple independent sound source components, the stereo audio video to be processed includes stereo audio to be processed or stereo video to be processed; The stereo audio video to be processed is subjected to dry and wet audio processing to obtain the ambient sound component; The ambient sound components are band-extended to obtain the target ambient sound components; A vector-based amplitude translation method is used to spatially render each independent sound source component and the target environmental sound component, and multi-channel output fusion is performed to obtain multi-channel audio and video.

[0006] In an optional implementation, the step of performing multi-stream separation on the stereo audio-video to be processed to obtain multiple independent sound source components includes: The stereo audio video to be processed is segmented into frames, and the segmented audio segments are normalized to obtain multiple target audio segments. Each of the target audio segments is input into a deep audio source separation model for audio source separation processing to obtain multiple channel signal segments; The overlapping regions of each of the adjacent channel signal segments are fused, and the channel signal segments are spliced ​​together in chronological order to obtain multiple independent sound source components.

[0007] In an optional implementation, the deep audio source separation model includes an encoder unit, a timing modeling unit, and a decoder unit, with the encoder unit and the decoder unit connected in a skip connection. The step of inputting each of the target audio segments into the deep audio source separation model for audio source separation processing to obtain multiple channel signal segments includes: Each of the target audio segments is input into the encoder unit, and features are extracted through the stacked convolutional layers of the encoder unit to obtain a feature map; The feature map is input into the temporal modeling unit for temporal modeling to obtain global features; The global features are input to the decoder unit for layer-by-layer upsampling and then concatenated with the same-layer features of the encoder unit to obtain multiple channel signal segments.

[0008] In an optional implementation, the step of performing wet and dry audio processing on the stereo audio video to be processed to obtain the ambient sound component includes: The minimum mean square algorithm is used to process the left and right channel signals of the stereo audio video to be processed to obtain the minimum mean square weights. An adaptive dynamic programming algorithm is used to adjust the minimum mean square weights to obtain the predicted audio. The environmental sound components are extracted from the predicted and estimated audio to obtain the environmental sound components.

[0009] In an optional implementation, the step of band-spreading the ambient sound components to obtain the target ambient sound components includes: The ambient sound components are preprocessed to obtain a preprocessed signal. The preprocessing includes at least frame splitting, windowing, and short-time Fourier transform. Linear predictive coding envelope extraction and Mel-spectral amplitude envelope extraction are performed on the preprocessed signal to obtain the spectral envelope; The low-frequency band energy is calculated based on the spectral envelope, and the high-frequency band energy is generated based on the low-frequency band energy to obtain the high-frequency amplitude spectrum; The high-frequency phase is inferred by periodically replicating the low-frequency phase of the preprocessed signal, and the transient frames of the preprocessed signal are detected and differentially enhanced to obtain the high-frequency phase spectrum. A complex spectrum is constructed based on the high-frequency amplitude spectrum and the high-frequency phase spectrum. An inverse short-time Fourier transform is performed on the complex spectrum to obtain the target environmental sound component.

[0010] In an optional implementation, the stereo audio video to be processed is the stereo audio video to be processed. The step of using a vector-based amplitude translation method to spatially render each of the independent sound source components and the target environmental sound components, and performing multi-channel output fusion to obtain multi-channel audio and video includes: The speaker layout is determined based on each of the independent sound source components and the target ambient sound components; Multiple virtual sound source positions are set according to each of the independent sound source components and the target environment sound components; The speaker base vector for each virtual sound source location is obtained by calculating based on the location of each virtual sound source and the speaker layout. The speaker gain of each independent sound source component and the target ambient sound component is calculated based on the speaker basis vector, and the speaker gain of each speaker is normalized to obtain multiple target speaker gains; The multi-channel audio and video are obtained by fusing the independent sound source components and the target ambient sound components according to the gain of each target loudspeaker.

[0011] In an optional implementation, the stereo audio video to be processed is a stereo video to be processed. The step of using a vector-based amplitude translation method to spatially render each of the independent sound source components and the target environmental sound components, and performing multi-channel output fusion to obtain multi-channel audio and video includes: Image tracking is performed on the stereo video to be processed to obtain the spherical coordinates of multiple target sound source objects; The speaker layout is determined based on each of the independent sound source components and the target ambient sound components; The speaker basis vectors of each target sound source object are calculated based on the spherical coordinates of each target sound source object and the speaker layout. The speaker gain of each independent sound source component and the target ambient sound component is calculated based on the speaker basis vector, and the speaker gain of each speaker is normalized to obtain multiple target speaker gains; The multi-channel audio and video are obtained by fusing the independent sound source components and the target ambient sound components according to the gain of each target loudspeaker.

[0012] Secondly, the present invention provides a stereo audio-video upmixing processing system, the system comprising: The audio-video separation module is used to acquire the stereo audio video to be processed, perform multi-stream separation on the stereo audio video to be processed, and obtain multiple independent sound source components. The stereo audio video to be processed includes stereo audio to be processed or stereo video to be processed. An ambient sound extraction module is used to perform dry and wet sound processing on the stereo audio video to be processed to obtain ambient sound components. An ambient sound processing module is used to perform frequency band expansion on the ambient sound components to obtain target ambient sound components; The multi-channel fusion module is used to perform spatial rendering of each independent sound source component and the target environmental sound component using a vector-based amplitude translation method, and to perform multi-channel output fusion to obtain multi-channel audio and video.

[0013] Thirdly, the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the stereo sound and video upmixing processing method as described in the first aspect of the present invention.

[0014] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the stereo sound and video upmixing processing method as described in the first aspect of the present invention.

[0015] The method, system, device, and storage medium for upmixing stereo audio and video provided by this invention utilize deep learning-based multi-stream separation technology to intelligently analyze complex scenes, accurately locate and efficiently separate sound sources, greatly improving the upmixing effect. It decomposes stereo audio and video into coherent and ambient sound, enhancing the dynamic range and spatial sense of the sound sources, making the audio more expressive and engaging. Simultaneously, it innovatively employs bandwidth extension technology, effectively compensating for the high-frequency loss problem in traditional methods when audio quality is poor, significantly improving sound quality. By using a vector-based amplitude translation method to spatially render different sound sources, it creates a stereo surround sound effect space, comprehensively improving the quality of audio processing and user experience. Attached Figure Description

[0016] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without paying any creative work.

[0017] Figure 1 This is a first schematic diagram of the process flow of the stereo sound and video upmixing method provided in the embodiment of the present invention; Figure 2 This is a second schematic diagram of the process flow of the stereo sound and video upmixing method provided in the embodiment of the present invention; Figure 3 This is a third schematic diagram of the process flow of the stereo sound and video upmixing method provided in this embodiment of the invention; Figure 4 This is the fourth schematic diagram of the process flow of the stereo sound and video upmixing method provided in the embodiment of the present invention; Figure 5 This is the fifth schematic diagram of the process flow of the stereo sound and video upmixing method provided in the embodiment of the present invention; Figure 6 This is the sixth schematic diagram of the process flow of the stereo sound and video upmixing method provided in the embodiment of the present invention; Figure 7 This is a schematic diagram of the structure of the stereo audio-video upmixing processing system provided in an embodiment of the present invention; Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention.

[0018] Description of main component symbols: 200. Stereo audio and video upmixing system; 210. Audio and video separation module; 220. Ambient sound extraction module; 230. Ambient sound processing module; 240. Multi-channel fusion module; 300. Electronic equipment; 310. Processor; 320. Communication interface; 330. Memory; 340. Communication bus. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be further described clearly and completely below with reference to the accompanying drawings of the embodiments of this invention. It should be noted that the described embodiments are merely some embodiments of this invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0020] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0021] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0022] Stereo audio video is based on two independent channels: the left channel and the right channel. These two channels carry different sound information, creating a sense of depth through the differences in sound between the left and right channels. Multichannel audio video is a more complex audio format that uses more than two channels to present sound. The most common multichannel configurations are 5.1 and 7.1 channels, which include a center channel, front left and front right channels, rear left and rear right channels, and a subwoofer channel. Multichannel audio video achieves more precise sound separation and surround sound effects through the coordinated work of multiple channels. For example, in a movie theater, multichannel audio video allows viewers to perceive sound coming from all directions, providing a more realistic auditory experience. Audio upmixing technology refers to the process of processing the original audio signal into an audio signal with more channels. Increasing the number of channels can significantly improve the quality of the user's audiovisual experience.

[0023] Traditional upmixing techniques are primarily based on simple algorithms, extracting surround information and encoding it into stereo signals by analyzing phase differences. These techniques generally suffer from limitations such as poor source separation, fixed decoding rules, difficulty adapting to complex audio content, limited frequency and dynamic range, and a weak final auditory effect. Therefore, effectively improving the upmixing effect of stereo audio-video processing to multi-channel audio-video has become a critical technical challenge that urgently needs to be addressed.

[0024] Example 1 This invention provides a method for upmixing stereo audio and video, which effectively solves the problem of poor upmixing effect when processing stereo audio and video into multi-channel audio and video. Figure 1 This is a first schematic diagram of the process flow of the stereo audio-video upmixing method provided in this embodiment of the invention, as shown below. Figure 1 As shown, the method includes: S100: Obtain the stereo audio video to be processed, perform multi-stream separation on the stereo audio video to be processed, and obtain multiple independent sound source components.

[0025] In this embodiment of the invention, the stereo audio video to be processed can be either stereo audio or stereo video. For example, the stereo audio to be processed can be stereo audio such as music scenes, and multi-stream separation can be performed for different instrument sound sources and human voice sound sources. The stereo video to be processed can also be stereo video such as movie source scenes. Since movie source scenes have a complex array of objects, it is impossible to completely cover all types of sound source objects. Therefore, objects with a certain degree of motion can be used as sound source objects for multi-stream separation, such as sounds from cars, living things, and the sky. Figure 2 This is a second schematic diagram of the flowchart of the stereo sound and video upmixing processing method provided in the embodiment of the present invention, as shown below. Figure 2 As shown, multi-stream separation specifically includes the following steps: S110. Perform frame segmentation on the stereo audio video to be processed, and normalize the segmented audio segments to obtain multiple target audio segments.

[0026] In this embodiment of the invention, the stereo audio video to be processed is segmented into frames. For example, the stereo audio video to be processed is divided into 8-second audio segments. The duration of 8 seconds is sufficient to cover enough audio information, so that each segment can present relatively complete audio-visual features. Adjacent audio segments are set with an overlap rate of not less than 25%. In this embodiment of the invention, the overlap rate is set to 50%, thereby ensuring the continuity and integrity of the audio information and avoiding information loss or abrupt changes caused by segment cutting.

[0027] In this embodiment of the invention, normalization processing is used to unify the amplitude of all audio segments to the range of [-1, 1] to obtain the target audio segment. Normalization processing not only avoids numerical calculation problems caused by excessive differences in audio and video amplitudes, but also eliminates the influence of different amplitude differences on audio features, allowing subsequent processing to focus more on the intrinsic features of the audio.

[0028] S120. Input each target audio segment into the deep audio source separation model for audio source separation processing to obtain multiple channel signal segments.

[0029] In this embodiment of the invention, the deep audio source separation model adopts a UNet+BiLSTM network model architecture, which includes an encoder unit, a temporal modeling unit, and a decoder unit, wherein: The encoder unit consists of six stacked one-dimensional convolutional layers. In each convolutional layer, the kernel length is fixed at 8, and the stride is set to 4, allowing the convolutional operation to capture local features within a certain length of audio segment. The number of channels increases exponentially with each layer, starting from 64 in the first layer, increasing to 128, 256, 512, and 1024, and remaining at 1024 in the last two layers. This strategy of varying the number of channels helps extract features of different complexities at different levels. Lower levels have fewer channels, mainly capturing simple local features, while higher levels have more channels, enabling the learning of more complex and abstract feature representations. A ReLU activation function is applied after each convolutional operation to introduce non-linearity into the model, enhancing its expressive power and enabling it to learn more complex audio feature patterns.

[0030] Since the stride of each convolutional layer is 4, downsampling is performed on the target audio segment during convolution, compressing the output of each layer to 1 / 4 of its original size in the time dimension. This downsampling mechanism not only reduces the dimensionality of the data and the computational cost, but also expands the receptive field, enabling subsequent layers to capture audio features over a wider range, thereby extracting multi-scale local features. After processing by 6 layers of one-dimensional convolutional groups, the encoder unit finally generates a set of feature maps, denoted as { F 1, F 2,..., F 6}. Among them, F 1 corresponds to the output of the first convolutional layer, representing the simple local features extracted initially. As the number of layers increases, the subsequent feature maps... F 2- F 6 contains increasingly complex and abstract audio feature information.

[0031] The temporal modeling unit employs a two-layer bidirectional long short-term memory (Bi-LSTM) network, which can model the input sequence simultaneously from both past and future perspectives. The forward LSTM processes data starting from the beginning of the sequence, progressively capturing historical information, while the backward LSTM processes data starting from the end of the sequence, focusing on future information. Each Bi-LSTM layer has 1024 hidden units, giving the model powerful representation capabilities and enabling it to capture rich and diverse temporal patterns and features in the audio sequence. The input to the temporal modeling unit is the terminal feature map output from the encoder unit. F 6. At this point, the time step of the input data is T / 4096. The time series modeling unit uses these feature maps as input to further mine the time series information contained therein.

[0032] The decoder unit is constructed sequentially from six layers of transposed convolutional layers, with each layer having a kernel length of 8 and a stride of 4. This configuration allows the transposed convolutions to gradually increase the feature map size while recovering the temporal dimension information lost during audio encoding, making the feature map gradually approach the temporal resolution of the original audio. The number of channels decreases by half with each layer, starting from 1024 in the first layer, decreasing to 512, 256, 128, and 64, and remaining at 64 in the last layer. This channel number variation strategy corresponds to that of the encoder unit; as the feature map size increases, the number of channels gradually decreases, helping to recover audio details while avoiding overfitting, enabling the model to learn more generalizable feature representations.

[0033] Meanwhile, the decoder unit introduces a skip connection mechanism, and the encoder unit... i Layer output F i It will be directly connected to the decoder unit (6- i The input of the layer, such as the second layer output of the encoder unit. F Line 2 connects to the 5th layer input of the decoder unit, and so on. The skip connection mechanism employs a channel-dimensional concatenation method, which overlays the feature map output by the encoder unit with the feature map of the corresponding layer input of the decoder unit along the channel dimension. In this way, low-level and local detailed feature information extracted by the encoder unit, such as high-frequency details like audio edges and textures, can be directly passed to the corresponding layer of the decoder unit. This allows the decoder unit to combine high- and low-level features when reconstructing audio, generating a more refined and accurate audio feature map, thereby significantly improving the quality and accuracy of audio source separation.

[0034] The output layer of the decoder unit uses a 1×1 convolutional kernel. The number of channels can be set according to actual needs, with different channels corresponding to different independent sound source components. The 1×1 convolutional kernel integrates and reduces the dimensionality of the features extracted from previous layers, and then maps them to different channels to achieve the separation and prediction of multiple sound source objects. Simultaneously, the output layer uses a linear activation function. Linear activation functions do not perform non-linear transformations on the output values, ensuring that the range and distribution of the output results are unrestricted, thus more accurately reproducing the audio signals of each sound source object.

[0035] In this embodiment of the invention, the loss function of the deep sound source separation model adopts a composite loss function, including temporal reconstruction loss, multi-scale STFT loss, and deep feature matching loss, wherein: The time-domain reconstruction loss ensures the basic accuracy of the signal and avoids waveform distortion. The formula for the time-domain reconstruction loss is as follows:

[0036]

[0037] In the above formula, L 1 represents the temporal reconstruction loss. K This represents the total number of sampling points in the audio signal. y k This indicates that the original audio signal is at the 1st... k Samples from each sampling point The audio signal generated by the model is in the first... k Samples from each sampling point.

[0038] Multi-scale STFT loss optimizes the spectral characteristics of the signal, making the spectrum of the generated signal closer to the real signal and reducing high-frequency noise or low-frequency distortion. The expression for multi-scale STFT loss is as follows:

[0039] In the above formula, L 2 represents the multi-scale STFT loss. S Represents a set of different STFT scales. Indicated in scale s The following operation performs a short-time Fourier transform on the input signal. This indicates that the amplitude spectrum is taken from the STFT result. This indicates taking the logarithm of the amplitude spectrum. This represents the L1 norm, which is the sum of the absolute values ​​of all elements in a vector. N s Indicated in scale s The number of elements in the STFT result.

[0040] Deep feature matching loss enhances the naturalness of signals at the perceptual level, making the generated signals visually or aurally closer to real signals and avoiding unnatural artifacts or distortions. The expression for deep feature matching loss is as follows:

[0041] In the above formula, L represents the number of layers in the pre-trained model used for feature extraction. (·) indicates that the pre-trained model is in the first... l Feature extraction function of layer The L2 norm is used to calculate the Euclidean distance between two feature vectors or feature maps. ) indicates the first l The dimension of the layer output features.

[0042] Therefore, the expression for the composite loss function is as follows:

[0043] In the above formula, L total Represents the composite loss function. , and represents the coefficients of temporal reconstruction loss, multi-scale STFT loss, and deep feature matching loss, respectively.

[0044] The deep sound source separation model in this invention employs a composite loss function, combining temporal reconstruction loss, multi-scale STFT loss, and deep feature matching loss. This allows for the constraint and optimization of the model output from multiple dimensions, thereby improving the model's performance in complex tasks.

[0045] Optionally, for multi-stream separation of stereo audio such as music scenes, open-source, proprietary, and synthetic data of sound sources such as drums, pianos, basses, and vocals can be acquired to supervise the training of the UNet+BiLSTM network model architecture to obtain a deep sound source separation model. For multi-stream separation of stereo video such as movie source scenes, dynamic data can be collected for customized model training. For example, dynamic sound sources such as car sounds, footsteps, and biological sounds can be used as training data to supervise the training of the UNet+BiLSTM network model architecture to obtain a deep sound source separation model.

[0046] In this embodiment of the invention, for the deep audio source separation model of stereo videos such as movie source scenes, due to the multi-level upsampling model structure and the sampling rate of the training data, fixed frequency band noise will appear at high-frequency nodes. This can be addressed by optimizing the loss function. For example, since the frequency bands of sound sources in various stereo videos such as movie source scenes may be close, aliasing noise may occur in downsampled frequency bands, such as 11kHz, due to spectral folding. This can be optimized by weighting the problematic frequency band using the multi-scale STFT loss of the composite loss function.

[0047] In this embodiment of the invention, each target audio segment is input to an encoder unit, and features are extracted through stacked convolutional layers of the encoder unit to obtain a set of feature maps. F 1, F 2, ..., F 6}. Feature Map F The input is fed to the timing modeling unit for timing modeling, and outputs global features G. The global features G are fed to the decoder unit for layer-by-layer upsampling. After each layer is transposed and convolved, they are concatenated with the features of the same layer in the encoder unit. Finally, the last layer is a 1×1 convolution to generate multiple channel waveforms, thereby obtaining multiple channel signal segments.

[0048] S130. Perform overlapping region fusion processing on adjacent channel signal segments and splice the channel signal segments in time sequence to obtain multiple independent sound source components.

[0049] In this embodiment of the invention, the overlapping portions of adjacent channel signal segments are weighted using a Hamming window. The weighting coefficients are determined based on the shape of the Hamming window function, smoothly transitioning the boundary values ​​of the channel signal segments to zero, thereby reducing distortion during splicing. The Hamming window is a smooth window function whose shape gradually decreases to zero at the boundaries, effectively reducing abrupt changes and discontinuities at segment boundaries. After weighted averaging of all overlapping portions, all channel signal segments are spliced ​​together in chronological order to form multiple independent sound source components. Each independent sound source component is a separated version of a specific sound source in the stereo audio video to be processed. For example, for stereo audio such as music scenes, independent sound source components include piano sound source components, bass sound source components, drum sound source components, human voice sound source components, and other sound source components; for stereo video such as movie source scenes, independent sound source components include car sound source components, footstep sound source components, and biological sound source components.

[0050] As an optional implementation of this invention, side-chain volume control can be applied to the human voice component and other sound source components within an independent sound source component. This adaptively adjusts the gain of the human voice audio in the stereo video based on the energy ratio of the human voice audio to the total audio, ensuring the clarity of the human voice in music scenes. The steps of the side-chain volume control are as follows: First, a low-profile filter is used to... N The audio and stereo video are divided into blocks, with each sampling point grouped together. For each block, the mean values ​​for both the vocal and stereo audio are calculated, along with their respective energy values. Then, a gain value is calculated based on the energy ratio, and the signal is segmented according to the gain value: if the gain is less than 0dB, it is boosted by a factor of 1.1885; if the gain is greater than 9dB, no boost is applied; if the gain is greater than 0dB but less than 9dB, linear interpolation is performed to ensure smooth gain changes. Finally, the calculated linear gain is multiplied by the stereo video signal, thus achieving dynamic adjustment of the vocal audio.

[0051] S200 performs dry and wet audio processing on the stereo audio video to be processed to obtain the ambient sound component.

[0052] In this embodiment of the invention, the least mean square algorithm combined with an adaptive dynamic programming algorithm is used to perform dry and wet sound processing on the stereo audio video to be processed, so as to ensure the basic orientation of the sound. Figure 3 This is a third schematic diagram of the flowchart of the stereo audio-video upmixing processing method provided in this embodiment of the invention, as shown below. Figure 3 As shown, the specific steps for dry and wet sound processing are as follows: S210. The minimum mean square algorithm is used to process the left and right channel signals of the stereo audio video to be processed, and the minimum mean square weight is obtained.

[0053] In this embodiment of the invention, the least mean square algorithm is applied to the left and right channels of the stereo audio video to be processed. Specifically, the right channel signal of each stereo audio video to be processed is first used as the desired signal, and the left channel signal is used as the input signal of the adaptive filter. The adaptive filter continuously adjusts its coefficients so that the output of the filter gradually approaches the desired right channel signal until a predetermined number of iterations is reached. Optionally, the predetermined number of iterations is set to 20, and a 16th-order filter is used. During the 20 iterations, the filter has sufficient opportunities to adjust its coefficients based on the difference between the input signal and the desired signal, and the 16th-order filter can maintain a certain level of complexity without excessively increasing the computational load.

[0054] After 20 predetermined iterations, the training phase for the left channel signal ends. The right channel signal is then used as the input signal to the adaptive filter, while the left channel signal becomes the desired output signal. Similarly, the adaptive filter undergoes another 20 iterations of training based on the new input and desired signals, adjusting its coefficients. Finally, through processing in both directions, two sets of coefficients are obtained, corresponding to the left and right channel signals respectively. These two sets of coefficients, based on signal processing in different directions, serve as minimum mean square weights and can independently characterize the sample features of each independent sound source component.

[0055] S220. An adaptive dynamic programming algorithm is used to adjust the minimum mean square weights to obtain the predicted audio.

[0056] In this embodiment of the invention, an adaptive dynamic programming algorithm is used to fine-tune the minimum mean square weights. The adaptive dynamic programming algorithm is trained with the same filter order and number of iterations as the minimum mean square algorithm, thereby maintaining consistency with the minimum mean square algorithm, while further improving performance by utilizing the fine-tuning characteristics of the adaptive dynamic programming algorithm.

[0057] Specifically, based on the least mean square weighting, an adaptive dynamic programming algorithm is used for prediction and estimation, outputting a predicted audio value y(n). This predicted audio value y(n) is an approximate representation of the original audio signal after processing, comprehensively considering the signal relationships between channels and the fine-tuning effect of the adaptive dynamic programming algorithm. The adaptive dynamic programming algorithm minimizes the dual-channel error by synchronously updating two sets of coefficients to obtain the predicted audio value. In this embodiment of the invention, the adaptive dynamic programming algorithm uses a lower learning rate for fine-tuning, making the coefficient updates more granular and avoiding oscillations or non-convergence problems caused by excessively large step sizes. This allows for more precise adjustment of the coefficients, further reducing channel signal errors and improving the accuracy and stability of the algorithm.

[0058] S230. Extract the environmental sound components from the predicted and estimated audio to obtain the environmental sound components.

[0059] After processing the audio using the least mean square algorithm and adaptive dynamic programming algorithm, the different components in the predicted audio are effectively separated. At this point, the main sound component and the ambient sound component can be extracted from the predicted audio to obtain the ambient sound component. The main sound component usually refers to the sound in the audio that has a clear direction and main information, such as human voices, the playing of main instruments, etc., while the ambient sound component refers to the background sound that creates the atmosphere of the scene, such as the sound of wind, rain, and the noise of a crowd.

[0060] As an optional implementation of this invention, since movie source scenes emphasize dynamism and atmosphere, the directional angle of the sound image can be exaggerated using a correlation-based temporal arrangement method. For example, after calculating the ambient sound channels, the left and right distribution indices can be calculated using the autocorrelation and cross-correlation of the ambient sound channels. Then, the left and right distribution indices are nonlinearly scaled and multiplied with the direct sound to obtain the exaggerated direct sound ambient sound components.

[0061] S300: Bandwidth expansion of the ambient sound component to obtain the target ambient sound component.

[0062] In this embodiment of the invention, since the environmental sound components after dry and wet sound processing may have high-frequency missing issues, the spectrum missing mainly occurs after 7kHz, and the high frequencies can be supplemented by bandwidth expansion. Figure 4 This is the fourth schematic diagram of the stereo audio-video upmixing processing method provided in this embodiment of the invention, as shown below. Figure 4 As shown, bandwidth extension specifically includes the following steps: S310. Preprocess the ambient sound components to obtain a preprocessed signal. The preprocessing includes at least frame processing, windowing processing, and short-time Fourier transform.

[0063] In this embodiment of the invention, the ambient sound component is first processed by frame segmentation. Optionally, a frame length of 25ms and a frame shift of 10ms are used for frame segmentation. Hamming window weighting is applied to each frame signal after segmentation to reduce spectral leakage, thereby obtaining a series of continuous audio frame signals.

[0064] Then, a short-time Fourier transform is performed on each audio frame signal, using a 1024-point fast Fourier transform to obtain the amplitude spectrum and phase spectrum of each audio frame signal. The amplitude spectrum reflects the energy distribution of the signal at different frequencies, while the phase spectrum contains the phase information of the signal. Both are crucial for subsequent high-frequency reconstruction. The amplitude spectrum and phase spectrum constitute the preprocessed signal.

[0065] S320. Perform linear predictive coding envelope extraction and Mel spectrum amplitude envelope extraction on the preprocessed signal to obtain the spectrum envelope.

[0066] First, linear predictive coding envelopes are extracted from each preprocessed signal. The reflection coefficients are calculated using 16th-order linear predictive analysis. The linear predictive coding amplitude envelope can be derived from the reflection coefficients. The linear predictive coding amplitude envelope can describe the overall profile of the signal spectrum and reflect the main frequency components and energy distribution of the signal.

[0067] The amplitude spectrum is then mapped to a 40-dimensional Mel-frequency scale, and a discrete cosine transform is performed to generate Mel-frequency cepstral coefficients. These Mel-frequency cepstral coefficients can extract the spectral envelope information of the audio signal. The first 12 dimensions of the Mel-frequency cepstral coefficients are retained, as these 12 dimensions contain the main feature information of the signal's spectral envelope. An inverse transform is performed on these retained first 12 dimensions of the Mel-frequency cepstral coefficients to reconstruct the Mel-frequency amplitude envelope. The Mel-frequency amplitude envelope describes the spectral characteristics of the signal from another perspective, complementing the linear predictive coding amplitude envelope.

[0068] Finally, the linear predictive coding amplitude envelope and the Mel spectral amplitude envelope are weighted and fused to obtain the spectral envelope. In this embodiment of the invention, the weighting coefficients can be adjusted according to actual needs to achieve the best fusion effect.

[0069] S330. Calculate the low-frequency band energy based on the spectral envelope, and generate the high-frequency band energy based on the low-frequency band energy to obtain the high-frequency amplitude spectrum.

[0070] In this embodiment of the invention, the spectral envelope is nonlinearly stretched in the frequency domain to the target high-frequency band. The purpose of nonlinear stretching is to extend the spectral characteristics of the low-frequency band to the high-frequency band while maintaining the relative structure of the spectrum. The low-frequency band energy is calculated by squaring the spectral envelope, and high-frequency energy is generated according to a certain proportional relationship to obtain the high-frequency amplitude spectrum. In this embodiment of the invention, the proportional relationship can be adjusted according to the statistical characteristics of the signal and the actual application scenario, so that the generated high-frequency energy and low-frequency energy remain coordinated overall, conforming to the energy distribution law of natural sound.

[0071] S340. Based on the low-frequency phase of the preprocessed signal, periodically replicate to infer the high-frequency phase, and detect the transient frames of the preprocessed signal for differential enhancement to obtain the high-frequency phase spectrum.

[0072] In this embodiment of the invention, periodic replication of the low-frequency phase of the preprocessed signal allows for a reasonable estimation of the phase information of the high-frequency components, resulting in a certain degree of continuity and rationality in the phase of the reconstructed high-frequency signal. Simultaneously, the transient position of the preprocessed signal is detected, and transient frames are located. Transients are portions of the signal where energy changes abruptly. Differential enhancement is applied to these transient frames to obtain the high-frequency phase spectrum. Differential enhancement can highlight the signal changes in the transient portion, enhancing the impact and clarity of the transient signal while avoiding adverse effects on other parts of the signal.

[0073] S350: Construct a complex spectrum based on the high-frequency amplitude spectrum and the high-frequency phase spectrum, and perform an inverse short-time Fourier transform on the complex spectrum to obtain the target environmental sound components.

[0074] In this embodiment of the invention, the high-frequency amplitude spectrum and the high-frequency phase spectrum are combined to construct a complex spectrum, which contains complete time-frequency information of the signal and is the basis for time-domain audio reconstruction. An inverse short-time Fourier transform is performed on the complex spectrum to convert the time-frequency domain signal back into a time-domain signal. This signal contains the completed high-frequency components, thereby achieving high-frequency signal completion and audio quality improvement, and obtaining the target environmental sound components.

[0075] As an optional implementation of this invention, since ambient sound typically exhibits strong correlation between the left and right channels, resulting in a comb filtering effect during playback, inter-channel decorrelation processing can be performed on each independent sound source component and the target ambient sound component. Optionally, an all-pass filter can be used for inter-channel decorrelation processing to reduce the adverse effects of comb filtering during audio playback.

[0076] In this embodiment of the invention, the all-pass filter uses two all-pass FIR filters, one on the left and one on the right, whose coefficients are generated based on a random phase response function. Its amplitude response is always 1, and signal characteristics are adjusted only by changing the phase response. The transfer function of the all-pass filter can be expressed as:

[0077] In the above formula, H L (z) This represents the transfer function of the all-pass filter acting on the left channel signal. H R (z) This represents the transfer function of the all-pass filter acting on the right channel signal. N This indicates the order of the all-pass filter. and These represent the filter coefficients for the left and right channels, respectively. k Indicates the sampling point number. Indicates the k The delay of the signal at each sampling point.

[0078] The filter coefficients for the left and right channels are generated using a random phase response function to ensure that the phase characteristics of the left and right channels are independent, thereby reducing inter-channel correlation. The formula for generating the filter coefficients is as follows:

[0079] In the above formula, M Indicates the number of harmonic components. c m Indicates the first m The amplitude weight of each harmonic component Indicates the first m The logarithmic distribution frequency points of each harmonic component Indicates the first m The random phase of each harmonic component.

[0080] Optionally, the number of harmonic components determines the complexity of the phase response. The larger the number of harmonic components, the stronger the phase randomness and the more significant the decorrelation effect. The amplitude weight can be adjusted according to the spectral energy distribution of the ambient sound. Logarithmic frequency points can be distributed at logarithmic intervals to match the human ear's frequency perception characteristics. The random phase is generated by a pseudo-random number generator or noise signal sampling to ensure that the random phase is uniformly distributed within [0, 2π).

[0081] The all-pass filter processes the left and right channels through different transfer functions, introducing independent phase randomness and disrupting the phase consistency of the original signal. Simultaneously, the group delay varies with frequency, and the difference in group delay between the left and right channels further reduces correlation. Since the amplitude response is always 1, the spectral energy distribution of the filtered signal remains unchanged, with only the phase characteristics altered, thus avoiding the introduction of additional distortion.

[0082] As an optional implementation of this invention, for stereo audio such as music scenes, different genres of music can be determined based on music information retrieval, and several preset mixing parameters can be used to perform track-by-track sound effect processing on the independent sound source components of different instruments. For example, for a typical bass audio component, a low-frequency filter below 400Hz is used for enhancement; for the drum audio component, transient enhancement is performed; and for the mid-to-high frequencies of other audio components, a high-frequency filter is used for enhancement.

[0083] The S400 uses a vector-based amplitude translation method to spatially render each independent sound source component and the target environmental sound component, and performs multi-channel output fusion to obtain multi-channel audio and video.

[0084] If the stereo video to be processed is stereo audio such as music scenes... Figure 5This is the fifth schematic diagram of the flowchart of the stereo audio-video upmixing processing method provided in this embodiment of the invention, as shown below. Figure 5 As shown, obtaining multi-channel audio and video specifically includes the following steps: S401. Determine the speaker layout based on each independent sound source component and the target ambient sound component.

[0085] In this embodiment of the invention, the individual sound source components and the target ambient sound component are allocated in the vertical and horizontal dimensions to determine the speaker layout. For example, taking the audio of a rock band's music scene as an example, in the vertical dimension, the bass and drum audio components are allocated to the bottom layer with a vertical coordinate of 0°; the vocal and piano audio components are allocated to the middle layer with vertical coordinates of 15°-30°; and the target ambient sound component is allocated to the top layer with vertical coordinates of 45°-90°. In the horizontal dimension, key components such as the vocal, piano, bass, and drum audio components are concentrated in a 120° arc in front, while the target ambient sound component is allocated to the sides and rear surround.

[0086] Based on the above vertical and horizontal dimension allocation results, an exemplary speaker layout based on a 7.1.4 channel configuration is as follows: Left main channel L =(-30°0°), Right Main Channel R =(30°, 0°), Center Channel C =(0°,0°), Left surround channel Ls =(-110°, 0°), Right Surround Channel Rs =(110°, 0°), upper left front channel Tfl =(-30°, 45°), upper right front channel Tfr =(30°, 45°), upper left rear channel Tbl =(-150°, 45°), upper right rear channel Tbr =(150°, 45°). Where the horizontal axis represents the azimuth angle and the vertical axis represents the elevation angle.

[0087] S402. Set multiple virtual sound source positions based on each independent sound source component and the target environment sound component.

[0088] In this embodiment of the invention, the virtual sound source location is used to represent the virtual location rendered in space of the independent sound source component and the target environmental sound component. The virtual sound source location can be set within the space surrounded by the speaker or within a reasonably extended range to ensure that its location can be effectively simulated by the speaker combination.

[0089] S403. Calculate the speaker base vector for each virtual sound source location based on the location of each virtual sound source and the speaker layout.

[0090] In this embodiment of the invention, for any virtual sound source location, the three loudspeakers closest to that virtual sound source location are selected; in a two-dimensional case, this can be two loudspeakers. A triangle (two-dimensional) or a tetrahedron (three-dimensional) is formed with these three loudspeaker locations as vertices, and the vectors pointing from the virtual sound source location to these three loudspeaker locations are calculated to obtain the loudspeaker basis vectors.

[0091] S404. Calculate the speaker gain of each independent sound source component and the target ambient sound component based on the speaker base vector, and normalize each speaker gain to obtain multiple target speaker gains.

[0092] In this embodiment of the invention, the direction vector of the virtual sound source (the vector pointing from the origin to the virtual sound source position) is projected onto the space spanned by the speaker base vectors, thereby determining the components of the virtual sound source direction in each speaker direction. The amplitude gain of each speaker is calculated based on the projection results, which can be achieved by solving a system of linear equations based on the relationship between the speaker base vectors and the virtual sound source direction vector. For example, if assigned to the left main channel... L Right main channel C Top left front channel Tfl Then solve the following system of linear equations:

[0093] In the above formula, L x , L y and L z Indicates the left main channel L Three-dimensional spatial coordinates, C x , C y and C z Indicates the right main channel C Three-dimensional spatial coordinates, Tfl x , Tfl y and Tfl z Indicates the upper left front channel Tfl Three-dimensional spatial coordinates, , and These represent the left main channel. L Right main channel C Top left front channel Tfl Speaker gain on the channel Indicates the azimuth angle of the corresponding speaker channel. This indicates the elevation angle of the corresponding speaker channel.

[0094] The left main channel was obtained by solving a system of linear equations. L Right main channel C Top left front channel Tfl Speaker gain of the channel , and .

[0095] The calculated speaker gains for each speaker are normalized so that the sum of the squares of all speaker gains is 1, thus obtaining multiple target speaker gains. This ensures that the total energy of the signal remains constant, avoiding problems such as excessively loud or soft sound volume due to gain adjustments.

[0096] S405. Based on the gain of each target loudspeaker, the components of each independent sound source and the target ambient sound component are fused and output to obtain multi-channel audio and video.

[0097] In this embodiment of the invention, the audio signals of each independent sound source component and the target environment sound component are multiplied by the calculated gain of each target speaker. The weighted signals are then output to the corresponding speaker channels to achieve spatial positioning and rendering of the virtual sound source, resulting in multi-channel audio and video, which is in audio format.

[0098] In another aspect of this invention, if the stereo audio video to be processed is a stereo audio video such as a movie source scene, Figure 6 This is the sixth schematic diagram of the flowchart of the stereo audio-video upmixing processing method provided in this embodiment of the invention, as shown below. Figure 6 As shown, obtaining multi-channel audio and video specifically includes the following steps: S410: Perform image tracking on the stereo video to be processed to obtain the spherical coordinates of multiple target sound source objects.

[0099] In this embodiment of the invention, an image tracking method can be used to perform frame-level analysis of target sound source objects such as cars and organisms in the stereo video to be processed, thereby detecting the coordinates of the target sound source objects. Since video frame splitting significantly affects processing speed, and spatial rendering does not require precise positioning, a fixed number of frames can be selected for frame extraction, and the video can be downsampled to a lower resolution. To address the problem of lost target sound source object tracking, smoothing or predictive completion methods can be used. After detecting the coordinates of the target sound source objects in the stereo video to be processed, they are mapped to a spherical coordinate system to obtain the spherical coordinates of multiple target sound source objects for subsequent rendering processes.

[0100] S420: Determine the speaker layout based on each independent sound source component and the target ambient sound component.

[0101] In this embodiment of the invention, the speaker layout can be adaptively set according to each independent sound source component and the target environment sound component. Since the coordinates within the video are calibrated based on the spherical coordinates of the camera position, and the sound source position is basically in the front hemisphere, the speaker layout can be based on the front hemisphere to further enhance the object's dynamism and provide sound source rendering for the sky channel object. For example, assuming a 7.1.4 channel system is used, 7 ground speakers, 1 subwoofer, and 4 sky speakers can be set up, and the spherical coordinate position of each speaker can be defined.

[0102] S430. Calculate the speaker base vector of each target sound source object based on the spherical coordinates and speaker layout of each target sound source object.

[0103] In this embodiment of the invention, two or three nearest loudspeakers containing the target direction are selected based on the spherical coordinates of the target sound source object, and the vectors pointing from the spherical coordinates of the target sound source object to the positions of these three loudspeakers are calculated to obtain the loudspeaker base vectors.

[0104] S440. Calculate the speaker gain of each independent sound source component and the target ambient sound component based on the speaker basis vector, and normalize each speaker gain to obtain multiple target speaker gains.

[0105] In this embodiment of the invention, the direction vector of the target sound source object (the vector pointing from the origin to the position of the target sound source object) is projected onto the space spanned by the speaker basis vectors, thereby determining the components of the target sound source object's direction in each speaker direction. Then, based on a method similar to step S404, the gain of each speaker is calculated by solving a system of linear equations. Finally, the calculated speaker gains of each speaker are normalized to obtain multiple target speaker gains.

[0106] S450: Based on the gain of each target loudspeaker, the components of each independent sound source and the target ambient sound component are fused and output to obtain multi-channel audio and video.

[0107] In this embodiment of the invention, the audio signals of each independent sound source component and the target environment sound component are multiplied by the calculated gain of each target speaker. The weighted signals are then output to the corresponding speaker channels to achieve spatial positioning and rendering of the virtual sound source, resulting in multi-channel audio and video in video format.

[0108] In this embodiment of the invention, specific processing logic can be adopted for different scenarios based on the detected tags. Whether it is a music or movie scene, it can be optimized in a targeted manner, showing strong intelligent adaptability and fully meeting the diverse needs of users.

[0109] The stereo audio-video upmixing method provided in this invention utilizes deep learning-based multi-stream separation technology to accurately locate and efficiently separate sound sources, significantly improving the upmixing effect. It decomposes stereo audio-video into coherent and ambient sound, enhancing the dynamic range and spatial sense of the sound sources, making the audio more expressive and engaging. Simultaneously, it innovatively employs bandwidth extension technology to effectively compensate for the high-frequency loss problem in traditional methods when audio quality is poor, significantly improving sound quality. By using a vector-based amplitude translation method to spatially render different sound sources, it creates a stereo surround sound effect space, comprehensively improving the quality of audio processing and user experience.

[0110] Example 2 Based on the same technical concept as Embodiment 1 above, this embodiment of the invention provides a stereo audio-video upmixing system. Figure 7 This is a schematic diagram of the stereo audio-video upmixing system provided in an embodiment of the present invention, as shown below. Figure 7 As shown, the stereo audio-video upmixing system 200 includes: The audio-video separation module 210 is used to acquire the stereo audio-video to be processed, perform multi-stream separation on the stereo audio-video to be processed, and obtain multiple independent sound source components. The stereo audio-video to be processed includes stereo audio to be processed or stereo video to be processed.

[0111] The ambient sound extraction module 220 is used to perform dry and wet sound processing on the stereo audio video to be processed to obtain the ambient sound component.

[0112] The ambient sound processing module 230 is used to extend the frequency band of the ambient sound components to obtain the target ambient sound components.

[0113] The multi-channel fusion module 240 is used to perform spatial rendering of each independent sound source component and the target environmental sound component using a vector-based amplitude translation method, and to perform multi-channel output fusion to obtain multi-channel audio and video.

[0114] It is understood that the implementation method of the stereo audio-video upmixing processing method described in Embodiment 1 above is also applicable to this embodiment and can achieve the same technical effect, so it will not be described again here.

[0115] Example 3 Based on the same concept, embodiments of the present invention also provide an electronic device. Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention, such as... Figure 8As shown, the electronic device 300 may include a processor 310, a communication interface 320, a memory 330, and a communication bus 340, wherein the processor 310, the communication interface 320, and the memory 330 communicate with each other via the communication bus 340. The processor 310 can call logical instructions in the memory 330 to execute the steps of the stereo sound and video upmixing processing method as described in the above embodiments. For example, it includes: S100. Obtain the stereo audio video to be processed, perform multi-stream separation on the stereo audio video to be processed to obtain multiple independent sound source components. The stereo audio video to be processed includes stereo audio to be processed or stereo video to be processed. S200: Perform dry and wet audio processing on the stereo audio video to be processed to obtain the ambient sound component; S300: Bandwidth expansion of the ambient sound components to obtain the target ambient sound components; The S400 uses a vector-based amplitude translation method to spatially render each independent sound source component and the target environmental sound component, and performs multi-channel output fusion to obtain multi-channel audio and video.

[0116] The processor 310 can be a central processing unit (CPU). The processor can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or combinations of the above types of chips.

[0117] Furthermore, the logical instructions in the aforementioned memory 330 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0118] The memory 330 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created by the processor, etc. Furthermore, the memory may include high-speed random access memory and non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory may optionally include memory remotely located relative to the processor, which can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0119] Example 4 Based on the same concept, embodiments of the present invention also provide a computer-readable storage medium storing a computer program containing at least one piece of code executable by a master control device to control the master control device to implement the steps of the stereo sound and video upmixing processing method as described in the above embodiments. For example, it includes: S100. Obtain the stereo audio video to be processed, perform multi-stream separation on the stereo audio video to be processed to obtain multiple independent sound source components. The stereo audio video to be processed includes stereo audio to be processed or stereo video to be processed. S200: Perform dry and wet audio processing on the stereo audio video to be processed to obtain the ambient sound component; S300: Bandwidth expansion of the ambient sound components to obtain the target ambient sound components; The S400 uses a vector-based amplitude translation method to spatially render each independent sound source component and the target environmental sound component, and performs multi-channel output fusion to obtain multi-channel audio and video.

[0120] Based on the same technical concept, this embodiment of the invention also provides a computer program, which, when executed by a master control device, is used to implement the above-described method embodiments.

[0121] The computer program may be stored, in whole or in part, on a computer-readable storage medium packaged with the processor, or in part or in whole on a memory not packaged with the processor.

[0122] Based on the same technical concept, embodiments of the present invention also provide a processor for implementing the above-described method embodiments. The processor may be a chip.

[0123] In summary, the stereo audio-video upmixing method, system, device, and storage medium provided by this invention utilize deep learning-based multi-stream separation technology to intelligently analyze complex scenes, accurately locate and efficiently separate sound sources, greatly improving the upmixing effect. By decomposing coherent sound and ambient sound into stereo audio-video, the dynamic range and spatial sense of the sound source are enhanced, making the sound in the audio more expressive and engaging. Simultaneously, the innovative use of bandwidth extension technology effectively compensates for the high-frequency loss problem in traditional methods when audio quality is poor, significantly improving sound quality. Furthermore, by employing a vector-based amplitude translation method to spatially render different sound sources, a stereo surround sound effect space is created, comprehensively improving the quality of audio processing and user experience.

[0124] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0125] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

[0126] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for upmixing stereo audio and video, characterized in that, The method includes: Obtain the stereo audio video to be processed, perform multi-stream separation on the stereo audio video to be processed to obtain multiple independent sound source components, the stereo audio video to be processed includes stereo audio to be processed or stereo video to be processed; The stereo audio video to be processed is subjected to dry and wet audio processing to obtain the ambient sound component; The ambient sound components are band-extended to obtain the target ambient sound components; A vector-based amplitude translation method is used to spatially render each independent sound source component and the target environmental sound component, and multi-channel output fusion is performed to obtain multi-channel audio and video.

2. The upmixing method for stereo audio and video according to claim 1, characterized in that, The step of performing multi-stream separation on the stereo audio-video to be processed to obtain multiple independent sound source components includes: The stereo audio video to be processed is segmented into frames, and the segmented audio segments are normalized to obtain multiple target audio segments. Each of the target audio segments is input into a deep audio source separation model for audio source separation processing to obtain multiple channel signal segments; The overlapping regions of each of the adjacent channel signal segments are fused, and the channel signal segments are spliced ​​together in chronological order to obtain multiple independent sound source components.

3. The upmixing method for stereo audio and video according to claim 2, characterized in that, The deep audio source separation model includes an encoder unit, a timing modeling unit, and a decoder unit. The encoder unit and the decoder unit are connected in a skip connection. Each target audio segment is input into the deep audio source separation model for audio source separation processing to obtain multiple channel signal segments, including: Each of the target audio segments is input into the encoder unit, and features are extracted through the stacked convolutional layers of the encoder unit to obtain a feature map; The feature map is input into the temporal modeling unit for temporal modeling to obtain global features; The global features are input to the decoder unit for layer-by-layer upsampling and then concatenated with the same-layer features of the encoder unit to obtain multiple channel signal segments.

4. The method for upmixing stereo audio and video according to claim 1, characterized in that, The step of performing dry and wet audio processing on the stereo audio video to be processed to obtain the ambient sound component includes: The minimum mean square algorithm is used to process the left and right channel signals of the stereo audio video to be processed to obtain the minimum mean square weights. An adaptive dynamic programming algorithm is used to adjust the minimum mean square weights to obtain the predicted audio. The environmental sound components are extracted from the predicted and estimated audio to obtain the environmental sound components.

5. The method for upmixing stereo audio and video according to claim 1, characterized in that, The step of bandwidth spreading the ambient sound components to obtain the target ambient sound components includes: The ambient sound components are preprocessed to obtain a preprocessed signal. The preprocessing includes at least frame splitting, windowing, and short-time Fourier transform. Linear predictive coding envelope extraction and Mel-spectral amplitude envelope extraction are performed on the preprocessed signal to obtain the spectral envelope; The low-frequency band energy is calculated based on the spectral envelope, and the high-frequency band energy is generated based on the low-frequency band energy to obtain the high-frequency amplitude spectrum; The high-frequency phase is inferred by periodically replicating the low-frequency phase of the preprocessed signal, and the transient frames of the preprocessed signal are detected and differentially enhanced to obtain the high-frequency phase spectrum. A complex spectrum is constructed based on the high-frequency amplitude spectrum and the high-frequency phase spectrum. An inverse short-time Fourier transform is performed on the complex spectrum to obtain the target environmental sound component.

6. The method for upmixing stereo audio and video according to claim 1, characterized in that, The stereo audio video to be processed is the stereo audio to be processed. The process involves using a vector-based amplitude translation method to spatially render each independent sound source component and the target environmental sound component, and then performing multi-channel output fusion to obtain multi-channel audio and video, including: The speaker layout is determined based on each of the independent sound source components and the target ambient sound components; Multiple virtual sound source positions are set according to each of the independent sound source components and the target environment sound components; The speaker base vector for each virtual sound source location is obtained by calculating based on the location of each virtual sound source and the speaker layout. The speaker gain of each independent sound source component and the target ambient sound component is calculated based on the speaker basis vector, and the speaker gain of each speaker is normalized to obtain multiple target speaker gains; The multi-channel audio and video are obtained by fusing the independent sound source components and the target ambient sound components according to the gain of each target loudspeaker.

7. The method for upmixing stereo audio and video according to claim 1, characterized in that, The stereo audio video to be processed is a stereo audio video to be processed. The process involves using a vector-based amplitude translation method to spatially render each independent sound source component and the target environmental sound component, and then performing multi-channel output fusion to obtain multi-channel audio and video, including: Image tracking is performed on the stereo video to be processed to obtain the spherical coordinates of multiple target sound source objects; The speaker layout is determined based on each of the independent sound source components and the target ambient sound components; The speaker basis vectors of each target sound source object are calculated based on the spherical coordinates of each target sound source object and the speaker layout. The speaker gain of each independent sound source component and the target ambient sound component is calculated based on the speaker basis vector, and the speaker gain of each speaker is normalized to obtain multiple target speaker gains; The multi-channel audio and video are obtained by fusing the independent sound source components and the target ambient sound components according to the gain of each target loudspeaker.

8. A stereo audio-video upmixing system, characterized in that, The system includes: The audio-video separation module is used to acquire the stereo audio video to be processed, perform multi-stream separation on the stereo audio video to be processed, and obtain multiple independent sound source components. The stereo audio video to be processed includes stereo audio to be processed or stereo video to be processed. An ambient sound extraction module is used to perform dry and wet sound processing on the stereo audio video to be processed to obtain ambient sound components. An ambient sound processing module is used to perform frequency band expansion on the ambient sound components to obtain target ambient sound components; The multi-channel fusion module is used to perform spatial rendering of each independent sound source component and the target environmental sound component using a vector-based amplitude translation method, and to perform multi-channel output fusion to obtain multi-channel audio and video.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the stereo sound and video upmixing processing method as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the stereo audio-video upmixing processing method as described in any one of claims 1-7.

Images