Audio signal mixing and playback method, device, electronic device and storage medium

Abstract

Embodiments of the present invention disclose a method, apparatus, electronic device, and storage medium for audio signal mixing and playback. In these embodiments, correlation analysis is performed on the left and right channel signals in a two-channel signal, and a specific signal processing strategy is determined based on the analysis results. By employing targeted processing methods for the two-channel signal in different states, the accuracy of signal processing is improved. Finally, when played back in a 5.1-channel surround sound system, the sound and image are naturally coherent, achieving a good sense of surround and envelopment.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of audio data technology, and in particular to a method, device, electronic device, and storage medium for mixing and replaying audio signals. Background Art

[0002] With the continuous advancement of electronic technology, audio devices used in media playback scenarios such as movies, music, and more are becoming increasingly powerful and delivering higher-quality sound, enriching users' entertainment experiences. While many existing audio systems are standard two-channel stereo, multi-channel surround sound systems are becoming increasingly popular with the advancement of electronic infrastructure. To make multi-channel surround sound playback systems compatible with stereo audio, upmixing is necessary. Upmixing involves converting a signal with fewer channels into a signal with more channels for playback.

[0003] Many existing two-channel up-mixing playback systems separate directional sound and surround sound based on their statistical characteristics. Two-channel stereo signal model and statistical characteristics: Both the left and right channels are composed of directional sound (also called direct sound, which refers to the sound that travels directly from the sound source to the receiver in a straight line without any reflection) and ambient sound. The directional sounds of the left and right channels are correlated, while the ambient sounds of the left and right channels are uncorrelated, and the ambient sounds are also uncorrelated with each other. However, the up-mixing method based on this model cannot completely separate directional sound and ambient sound. In particular, when the directional sound is fed to a single channel, the directional sound will be mistaken for ambient sound and directly fed to the rear surround sound channel, resulting in jumpy and unnatural sound images. On the other hand, the ambient sound components and energy extracted based on this model are insufficient, resulting in a weak sense of surround and envelopment when the two-channel signal is played in a surround sound system. Summary of the Invention

[0004] The present invention provides an audio signal mixing and replaying method, device, electronic equipment and storage medium to solve the technical problems of sound and image jumping, weak surround and enveloping sense when the existing two-channel signal is played in a surround sound system.

[0005] In a first aspect, an embodiment of the present invention provides an audio signal mixing and playback method, comprising:

[0006] receiving a two-channel signal, the two-channel signal including a left-channel signal and a right-channel signal;

[0007] During the receiving process, whenever the length of the received two-channel signal reaches a first length, the two-channel signal within the first length is used as the target processing signal;

[0008] determining a normalized cross-correlation parameter of a target left channel signal and a target right channel signal in the target processed signal;

[0009] When it is determined, based on the normalized cross-correlation parameter, that the target left channel signal and the target right channel signal are strongly correlated, a first separation of directional / ambient sound is performed on the left channel signal and the right channel signal based on the covariance matrix of the target left channel signal and the right channel signal, and five signals corresponding to the 5.1-channel surround sound are determined based on the result of the first separation;

[0010] When it is confirmed based on the normalized cross-correlation parameter that the target left channel signal and the target right channel signal are weakly correlated, and when it is confirmed based on the target left channel signal and the target right channel signal that there is no directional sound signal, five signals corresponding to the 5.1-channel surround sound are confirmed based on the target left channel signal and the target right channel signal;

[0011] According to the normalized cross-correlation parameters, when it is confirmed that the target left channel signal and the target right channel signal are weakly correlated, and when it is confirmed that there is a directional sound signal based on the target left channel signal and the target right channel signal, a second separation of the directional / ambient sound is performed on the target left channel signal and the target right channel signal, and the 5-way signal corresponding to the 5.1-channel surround sound is confirmed based on the result of the second separation.

[0012] In a second aspect, an embodiment of the present invention further provides an audio signal mixing and playback device, comprising:

[0013] A signal receiving unit, configured to receive a dual-channel signal, the dual-channel signal including a left-channel signal and a right-channel signal;

[0014] a target confirmation unit configured to, during a receiving process, whenever the length of the received two-channel signal reaches a first length, use the two-channel signal within the first length as a target processing signal;

[0015] a parameter confirmation unit, configured to confirm normalized cross-correlation parameters of a target left channel signal and a target right channel signal in a target processed signal;

[0016] A first generating unit is configured to, when it is determined based on the normalized cross-correlation parameter that the target left channel signal and the target right channel signal are strongly correlated, perform a first separation of directional / ambient sound on the left channel signal and the right channel signal based on the covariance matrix of the target left channel signal and the right channel signal, and determine, based on the result of the first separation, five signals corresponding to the 5.1-channel surround sound;

[0017] A second generating unit is configured to, when confirming, based on the normalized cross-correlation parameter, that the target left channel signal and the target right channel signal are weakly correlated and that there is no directional sound signal based on the target left channel signal and the target right channel signal, confirm five signals corresponding to 5.1-channel surround sound based on the target left channel signal and the target right channel signal;

[0018] The third generating unit is used to confirm that the target left channel signal and the target right channel signal are weakly correlated based on the normalized cross-correlation parameter, and when it is confirmed that there is a directional sound signal based on the target left channel signal and the target right channel signal, perform a second separation of the directional / ambient sound on the target left channel signal and the target right channel signal, and confirm the 5-way signal corresponding to the 5.1-channel surround sound based on the result of the second separation.

[0019] In a third aspect, an embodiment of the present invention further provides an electronic device, including:

[0020] one or more processors;

[0021] a memory for storing one or more computer programs;

[0022] When one or more computer programs are executed by one or more processors, the electronic device implements the audio signal mixing and playback method according to the first aspect.

[0023] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the audio signal mixing and playback method of the first aspect.

[0024] In the above-mentioned audio signal mixing and playback method, device, electronic device and storage medium, a dual-channel signal is received, and the dual-channel signal includes a left-channel signal and a right-channel signal; during the receiving process, whenever the length of the received dual-channel signal reaches a first length, the dual-channel signal within the nearest first length is used as the target processing signal; the normalized cross-correlation parameters of the target left-channel signal and the target right-channel signal in the target processing signal are confirmed; when it is confirmed based on the normalized cross-correlation parameters that the target left-channel signal and the target right-channel signal are strongly correlated, the left-channel signal and the right-channel signal are first separated into directional / ambient sounds based on the covariance matrix of the target left-channel signal and the right-channel signal, and the corresponding signals are confirmed based on the result of the first separation. The system then determines the five signals corresponding to 5.1-channel surround sound. When the target left and right channel signals are weakly correlated based on the normalized cross-correlation parameters and the absence of directional sound signals based on the target left and right channel signals, the system then determines the five signals corresponding to 5.1-channel surround sound based on the target left and right channel signals. When the target left and right channel signals are weakly correlated based on the normalized cross-correlation parameters and the presence of directional sound signals based on the target left and right channel signals, the system then performs a second directional / ambient sound separation on the target left and right channel signals and determines the five signals corresponding to 5.1-channel surround sound based on the results of the second separation. By performing a correlation analysis on the left and right channel signals in the two-channel signal and determining a specific signal processing strategy based on the analysis results, the system improves the accuracy of signal processing by applying targeted processing methods to the two-channel signal in different states. Finally, when played back in a 5.1-channel surround sound system, the sound image is naturally coherent, achieving a good sense of surround and envelopment. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 The present invention provides a method flow chart of a mixed audio signal playback method.

[0026] Figure 2 Schematic diagram of the speaker layout for a 5.1-channel surround sound system.

[0027] Figure 3 A schematic structural diagram of a signal processing object in an audio signal mixing and playback method provided by an embodiment of the present invention.

[0028] Figure 4 A schematic diagram of the overall data processing process of an audio signal mixing and playback method provided by an embodiment of the present invention.

[0029] Figure 5 A schematic structural diagram of an audio signal mixing and playback device provided by an embodiment of the present invention.

[0030] Figure 6 A schematic structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0031] The present invention will be further described in detail below with reference to the accompanying drawings and examples. It will be understood that the specific embodiments described herein are intended to explain the present invention, not to limit the present invention. It should also be noted that, for ease of description, the accompanying drawings only illustrate portions relevant to the present invention, not all structures.

[0032] It should be noted that this application specification does not enumerate all optional implementation methods. After reading this application specification, those skilled in the art should be able to understand that as long as the technical features do not contradict each other, any combination of technical features can constitute an optional implementation method.

[0033] With the advancement of technology and improvements in living standards, people are increasingly pursuing a more enjoyable lifestyle. Listening to music and watching movies and TV shows are common daily activities, and more and more people are purchasing surround sound systems. Generally speaking, the more channels and corresponding speakers, the better the surround and enveloping feeling of the sound during playback, and the closer it is to the live experience. However, multi-channel surround sound requires corresponding multi-channel media files to achieve the desired sound effects. Existing media files often only provide a two-channel signal when played. To achieve the best playback effect through surround sound, they are often upmixed and reproduced.

[0034] Surround sound systems with different channels usually have different upward playback strategies, such as Figure 2 As shown, a 5.1-channel surround sound system typically includes five speakers plus a subwoofer: front left (L) and right (R) channels, rear left (LS) and right (RS) surround channels, a center speaker (C), and a 0.1-channel subwoofer. A system can connect a total of six speakers (two main speakers, one center speaker, two surround speakers, and one subwoofer). Existing upmixing solutions for 5.1-channel surround sound systems typically upmix the two channels into a five-way signal (excluding the subwoofer) to achieve a basic surround sound effect. However, existing channel upmixing methods cannot completely separate directional and ambient sounds. In particular, when directional sounds are fed to a single channel, they can be mistaken for ambient sounds and directly fed to the rear surround channels, resulting in a jerky and unnatural sound image. Furthermore, the lack of ambient sound components and energy also results in a weak sense of surround and envelopment when the two-channel signal is played back in a surround sound system.

[0035] To address the above technical issues, this solution proposes an audio signal mixing and playback method, which can perform more accurate correlation analysis on the two-channel signal, adopt different directional / ambient sound separation strategies based on the specific analysis results, and enhance the ambient sound of the separation results, so that the two-channel signal can be mixed and played back in a 5.1-channel surround sound system with natural and coherent sound and image, achieving a good sense of surround and envelopment.

[0036] The following describes in detail the embodiments of the present invention.

[0037] Figure 1 This is a flow chart of a method for mixing and replaying audio signals provided by an embodiment of the present invention. The method for mixing and replaying audio signals is implemented by an electronic device, which can actually be a multi-channel surround sound system device with data processing capabilities. Figure 1 The audio signal mixing and playback method includes steps S110 to S160:

[0038] Step S110: receiving a two-channel signal, where the two-channel signal includes a left-channel signal and a right-channel signal.

[0039] A two-channel signal is a signal composed of two channel signals. When it is played through a two-channel generating device, it can show a certain degree of surround effect. Two channels are the principle of realizing stereo sound. Two speakers are placed at a certain angle to each other in space, and each speaker is provided with a signal by a separate channel. The signal of each channel is processed during recording: the processing principle is to imitate the biological principle of the human ear when hearing sounds in nature (humans have two ears and can judge the specific location of the sound source based on the phase difference between the left and right ears when hearing sounds). In terms of the circuit, the two channel signals are basically different in phase. In this way, when you stand at the intersection of the axis lines of the two speakers and listen to the sound, you can feel the stereo effect.

[0040] For multi-channel surround sound systems, the received two-channel signal is typically sampled from the sound wave at a certain frequency. The sampling frequency must be at least twice the sound wave frequency to restore the actual sound effect during playback. The reception process is a continuous process, each time receiving the left and right channel signals sampled at a specific moment.

[0041] Step S120: During the receiving process, whenever the length of the received two-channel signal reaches a first length, the two-channel signal within the first length is used as a target processing signal.

[0042] When processing the received two-channel signal specifically, it is not processed every time a signal is received, but based on the preset first length, whenever the length of the received two-channel signal reaches the first length, this section of the two-channel signal is used as the target processing signal, and the subsequent actual data processing process of upward mixing and playback begins. While upward mixing and playback, continue to receive subsequent two-channel signals until the received two-channel signal reaches the first length again as the new target processing signal. During this period, the upward mixing and playback of the previous section of the target processing signal is completed. This is equivalent to the final corresponding playback moment of the two-channel signal being slightly later than the moment when the two-channel signal is received, but as a whole it is a continuous playback process. In the specific processing process, the absolute length of the first length is small, and users usually cannot perceive the delay in playback from the beginning.

[0043] like Figure 3 As shown, a two-channel signal, as an audio signal, is actually discrete data obtained by sampling sound. The first length actually refers to the set of signals sampled within the first length at the sampling frequency. Therefore, the first length can also be represented by a number. The multi-channel surround sound system device receives two-channel signals sequentially starting from a certain playback operation initiated by a user. After receiving N signals, from 1 to N, it confirms the first target processing signal for this playback operation, continues receiving subsequent signals, and plays them through the 5.1 multi-channel surround sound system after completing the relevant processing for upmixing and playback. During playback, the N+2 or N+3 signal may have been received. In other words, in terms of time, reception begins at the first signal, but playback of the first signal does not begin until the N+2 or N+3 signal is received. However, the first length is not large, and this playback delay is imperceptible to the user and does not affect the user's auditory enjoyment.

[0044] Step S130: confirming the normalized cross-correlation parameters of the target left channel signal and the target right channel signal in the target processed signal.

[0045] In this embodiment of the present application, the left channel signal in the target processed signal is defined as the target left channel signal, and the right channel signal is defined as the target right channel signal. After confirming the target processed signal, the normalized cross-correlation parameters of the target left channel signal and the target right channel signal are determined. For example, when the first length is N as described above, the normalized cross-correlation parameters of N target left channel signals and target right channel signals are determined.

[0046] When confirming the normalized cross-correlation parameters, i=1,2,...,L confirms the autocorrelation parameters of the target left channel signal, through i=1,2,...,L confirms the autocorrelation parameters of the target right channel signal, through i=1,2,...,L to confirm the cross-correlation parameters of the target left channel signal and the target right channel signal; finally, according to the autocorrelation parameters of the target left channel signal, the autocorrelation parameters of the target right channel signal and the cross-correlation parameters, the normalized cross-correlation parameters of the target left channel signal and the target right channel signal are confirmed, specifically by Confirm that the normalized cross-correlation parameter calculated in this way is between 0 and 1. In the specific confirmation method of the above parameters, L represents the length of the signal (the number of signals within the first length), and Represent the target left channel signal and the target right channel signal, respectively. The autocorrelation function is an average measure of signal characteristics in the time domain, describing the correlation between the values ​​of the same signal at any two different times. The cross-correlation function describes the correlation between the values ​​of two different signals at any two different times. The calculation process for the correlation parameters described above uses an exemplary correlation function; other correlation functions can also be used.

[0047] Step S140: When it is confirmed based on the normalized cross-correlation parameters that the target left channel signal and the target right channel signal are strongly correlated, a first separation of the directional / ambient sound is performed on the left channel signal and the right channel signal based on the covariance matrix of the target left channel signal and the right channel signal, and 5 signals corresponding to the 5.1-channel surround sound are confirmed based on the result of the first separation.

[0048] The normalized cross-correlation parameter represents the degree of correlation between the target left channel signal and the target right channel signal. The specific strength of the relationship is confirmed by a preset threshold. For example, a value above 0.7 indicates a strong correlation, while a value below 0.7 indicates a weak correlation. For another example, a value above 0.3 indicates a strong correlation, while a value below 0.3 indicates a weak correlation.

[0049] Based on the correlation judgment, if it is a strong correlation, the covariance matrix is ​​confirmed according to the target left channel signal, the mean of the target left channel signal, the target right channel signal, and the mean of the target right channel signal;

[0050] The covariance matrix is ​​determined as follows:

[0051]

[0052] Where COV represents the covariance matrix, and Signal E L0 and E R0 The mean of the target left channel signal and the target right channel signal are respectively recorded as [G L ,G R ].

[0053] The left direction sound signal and the left ambient sound signal are confirmed according to the target left channel signal and its corresponding unit eigenvector, and the right direction sound signal and the right ambient sound signal are confirmed according to the target right channel signal and its corresponding unit eigenvector; wherein the left direction sound signal E LD =G L *E L0 ; Left ambient sound signal E LE =E L0 -G L *E L0 ;Right direction acoustic signal E RD =G R *E R0 ; Right ambient sound signal E RE =E R0 -G R *E R0 .

[0054] According to the left direction sound signal and the right direction sound signal, the center channel signal in the 5.1 channel surround sound is confirmed. The center channel signal E C , that is, E C =0.707(E LD +E RD ).

[0055] The left direction sound signal and the right direction sound signal are used as the left front channel signal and the right front channel signal in the 5.1 channel surround sound respectively; the left direction sound signal and the right direction sound signal are passed through the reverberator to obtain the left reverberation signal and the right reverberation signal respectively, and the left surround channel signal in the 5.1 channel surround sound is confirmed according to the left ambient sound signal and the left reverberation signal, and the right surround channel signal in the 5.1 channel surround sound is confirmed according to the right ambient sound signal and the right reverberation signal. LS =G L1 *E LE +G L2 *E′ LE , right surround channel signal E RS =G R1 *E RE +G R2 *E′ RE , where G L1 , G L2 , G R1 and G R2 is the amplitude of signal adjustment, E′ LE and E′ RE Represent the left reverberation signal and the right reverberation signal respectively.

[0056] Step S150: When it is confirmed based on the normalized cross-correlation parameter that the target left channel signal and the target right channel signal are weakly correlated, and when it is confirmed based on the target left channel signal and the target right channel signal that there is no directional sound, 5 signals corresponding to the 5.1-channel surround sound are confirmed based on the target left channel signal and the target right channel signal.

[0057] According to the time difference and amplitude difference between the target left channel signal and the target right channel signal, the channel time difference and channel sound level difference are obtained; when the channel time difference is within the first time difference range or the channel sound level difference is within the first level difference range, it is confirmed that there is no directional sound signal, for example, the channel time difference is close to 0.1ms or the channel sound level difference is close to 0dB; when the channel time difference is within the second time difference range or the channel sound level difference is within the second level difference range, it is confirmed that there is a directional sound signal, for example, the channel time difference is much greater than 0.1ms or the channel sound level difference is much greater than 0dB; wherein, the value within the second time difference range is greater than the value within the first time difference range, and the value within the second level difference range is greater than the value within the first level difference range.

[0058] In this state, the five signals corresponding to the 5.1-channel surround sound are specifically confirmed based on the target left channel signal and the target right channel signal, mainly by setting the center channel signal, the left front channel signal and the right front channel signal in the 5.1-channel surround sound to 0; and the target left channel signal and the target right channel signal are respectively passed through the reverberator to obtain the left channel reverberation signal and the right channel reverberation signal, and the left surround channel signal in the 5.1-channel surround sound is confirmed based on the target left channel signal and the left channel reverberation signal, and the right surround channel signal in the 5.1-channel surround sound is confirmed based on the target right channel signal and the right channel reverberation signal. The final confirmation process at this time is generally consistent with the process of confirming the left surround channel signal and the right surround channel signal in step S140 described above, and will not be repeated here.

[0059] Step S160: When it is confirmed based on the normalized cross-correlation parameter that the target left channel signal and the target right channel signal are weakly correlated, and when it is confirmed that there is directional sound based on the target left channel signal and the target right channel signal, a second separation of directional / ambient sound is performed on the target left channel signal and the target right channel signal, and 5 signals corresponding to 5.1-channel surround sound are confirmed based on the result of the second separation.

[0060] In this case, when confirming the 5-way signal, first confirm the energy values ​​corresponding to the target left channel signal and the target right channel signal, and take the smaller value of the two energy values ​​as the energy value of the ambient sound. Assume that the energy of the target left channel signal is PE L0 , the energy of the target right channel signal is PE R0 , the energy of environmental sound PE E =min(PE L0 ,PE R0), that is, to the smaller value. According to the target left channel signal, the energy value of the target left channel signal and the energy value of the ambient sound, confirm the left front channel signal in the 5.1 channel surround sound, specifically E L =E LD =(1-PE E / PE L0 )*E L0 According to the target right channel signal, the energy value of the target right channel signal and the energy value of the ambient sound, confirm the right front channel signal in the 5.1 channel surround sound, specifically E R =E RD =(1-PE E / PE E0 )*E R0 . Set the center channel signal in the 5.1-channel surround sound to 0. Pass the left front channel signal and the right front channel signal through the reverberator to obtain the left front reverberation signal and the right front reverberation signal respectively. According to the target right channel signal, the energy value of the target right channel signal and the energy value of the ambient sound, the right ambient sound signal is confirmed. According to the target left channel signal, the energy value of the target left channel signal and the energy value of the ambient sound, the left ambient sound signal is confirmed. According to the left ambient sound signal and the left reverberation signal, the left surround channel signal in the 5.1-channel surround sound is confirmed. According to the right ambient sound signal and the right reverberation signal, the right surround channel signal in the 5.1-channel surround sound is confirmed. Specifically, there can be E LE =PE E / PE L0 *E L0 , E RE =PE E / PE E0 *E R0 The processing of the left surround channel signal and the right surround channel signal is generally the same as that in other cases, and will not be repeated here.

[0061] No matter how the five-way signals are obtained, the left direction sound signal is ultimately fed to the left front channel of the 5.1-channel surround sound, the right direction sound is fed to the right front channel of the 5.1-channel surround sound, the center channel signal obtained based on the left direction sound signal and the right direction sound signal is fed to the center channel of the 5.1-channel surround sound, the left surround sound signal obtained based on the left direction sound signal and the left ambient sound signal is fed to the left surround channel of the 5.1-channel surround sound, and the right surround sound signal obtained based on the right direction sound signal and the right ambient sound signal is fed to the right surround channel of the 5.1-channel surround sound, thereby completing the normal playback of the two-channel signal in the 5.1 multi-channel surround sound system and achieving the ultimate playback goal of good results.

[0062] The above overall processing process can be referred to Figure 4, it can be seen that this solution improves the problem of sound and image jumps through correlation analysis, making the image display more coherent and natural. The directional / ambient sound separation module designed for different situations, as well as the final signal confirmation and ambient sound enhancement based on this separation, provide a good music playback experience.

[0063] In summary, a two-channel signal is received, and the two-channel signal includes a left channel signal and a right channel signal; during the receiving process, whenever the length of the received two-channel signal reaches a first length, the two-channel signal within the nearest first length is used as the target processing signal; the normalized cross-correlation parameters of the target left channel signal and the target right channel signal in the target processing signal are confirmed; when it is confirmed that the target left channel signal and the target right channel signal are strongly correlated according to the normalized cross-correlation parameters, the left channel signal and the right channel signal are first separated from the direction / ambient sound according to the covariance matrix of the target left channel signal and the right channel signal, and the 5 corresponding to the 5.1 channel surround sound is confirmed according to the result of the first separation. The system then uses the left and right channel signals to identify the target, weakly correlated signal based on the normalized cross-correlation parameter, and confirms the absence of directional sound signals based on the target left and right channel signals. Five signals corresponding to 5.1-channel surround sound are then identified based on the target left and right channel signals. Furthermore, when the left and right channel signals are weakly correlated based on the normalized cross-correlation parameter, and confirm the presence of directional sound signals based on the target left and right channel signals, a second directional / ambient sound separation is performed on the target left and right channel signals, and five signals corresponding to 5.1-channel surround sound are identified based on the results of the second separation. By performing a correlation analysis on the left and right channel signals in the two-channel signal and determining a specific signal processing strategy based on the analysis results, the system employs targeted processing methods for the two-channel signal in different states, thereby improving signal processing accuracy. Ultimately, when played back in a 5.1-channel surround sound system, the sound image is naturally coherent, achieving a good sense of surround and envelopment.

[0064] Figure 5 This is a schematic diagram of the structure of the audio signal mixing and playback device provided by an embodiment of the present invention. Figure 5 The audio signal mixing and reproducing apparatus includes: a signal receiving unit 210, a target confirmation unit 220, a parameter confirmation unit 230, a first generation unit 240, a second generation unit 250 and a third generation unit 260.

[0065] The signal receiving unit 210 is used to receive a two-channel signal, which includes a left channel signal and a right channel signal; the target confirmation unit 220 is used to, during the receiving process, whenever the length of the received two-channel signal reaches a first length, use the two-channel signal within the nearest first length as the target processing signal; the parameter confirmation unit 230 is used to confirm the normalized cross-correlation parameters of the target left channel signal and the target right channel signal in the target processing signal; the first generation unit 240 is used to, when it is confirmed that the target left channel signal and the target right channel signal are strongly correlated based on the normalized cross-correlation parameters, perform a first separation of the direction / ambient sound on the left channel signal and the right channel signal based on the covariance matrix of the target left channel signal and the right channel signal, and confirm the corresponding direction / ambient sound based on the result of the first separation. 5-way signals corresponding to 5.1-channel surround sound; a second generating unit 250, for confirming, based on the normalized cross-correlation parameters, that the target left channel signal and the target right channel signal are weakly correlated, and when it is confirmed that there is no directional sound signal based on the target left channel signal and the target right channel signal, confirming the 5-way signals corresponding to 5.1-channel surround sound based on the target left channel signal and the target right channel signal; a third generating unit 260, for confirming, based on the normalized cross-correlation parameters, that the target left channel signal and the target right channel signal are weakly correlated, and when it is confirmed that there is a directional sound signal based on the target left channel signal and the target right channel signal, performing a second separation of directional / ambient sound on the target left channel signal and the target right channel signal, and confirming the 5-way signals corresponding to 5.1-channel surround sound based on the result of the second separation.

[0066] On the basis of the above embodiment, according to the covariance matrix of the target left channel signal and the right channel signal, the left channel signal and the right channel signal are first separated into directional / ambient sound, and according to the result of the first separation, 5-way signals corresponding to 5.1-channel surround sound are confirmed, including:

[0067] determining a covariance matrix based on the target left channel signal, the mean of the target left channel signal, the target right channel signal, and the mean of the target right channel signal;

[0068] Confirm the unit eigenvectors corresponding to the target left channel signal and the target right channel signal in the covariance matrix;

[0069] Confirm the left directional sound signal and the left ambient sound signal based on the target left channel signal and its corresponding unit eigenvector, and confirm the right directional sound signal and the right ambient sound signal based on the target right channel signal and its corresponding unit eigenvector;

[0070] Confirm the center channel signal in the 5.1-channel surround sound according to the left direction sound signal and the right direction sound signal;

[0071] The left directional sound signal and the right directional sound signal are respectively used as the left front channel signal and the right front channel signal in the 5.1 channel surround sound;

[0072] The left directional sound signal and the right directional sound signal are respectively passed through the reverberator to obtain the left reverberation signal and the right reverberation signal. The left surround channel signal in the 5.1-channel surround sound is confirmed according to the left ambient sound signal and the left reverberation signal, and the right surround channel signal in the 5.1-channel surround sound is confirmed according to the right ambient sound signal and the right reverberation signal.

[0073] On the basis of the above embodiment, the five-way signal corresponding to the 5.1-channel surround sound is determined according to the target left channel signal and the target right channel signal, including:

[0074] Set the center channel signal, left front channel signal and right front channel signal in the 5.1 channel surround sound to 0;

[0075] The target left channel signal and the target right channel signal are respectively passed through the reverberator to obtain a left channel reverberation signal and a right channel reverberation signal. The left surround channel signal in the 5.1-channel surround sound is confirmed according to the target left channel signal and the left channel reverberation signal, and the right surround channel signal in the 5.1-channel surround sound is confirmed according to the target right channel signal and the right channel reverberation signal.

[0076] Based on the above embodiment, a second separation of directional / ambient sound is performed on the target left channel signal and the target right channel signal, and five signals corresponding to 5.1-channel surround sound are confirmed according to the result of the second separation, including:

[0077] Determine the energy values ​​corresponding to the target left channel signal and the target right channel signal, and take the smaller of the two energy values ​​as the energy value of the ambient sound;

[0078] confirming a left front channel signal in a 5.1-channel surround sound according to a target left channel signal, an energy value of the target left channel signal, and an energy value of the ambient sound;

[0079] confirming a right front channel signal in a 5.1-channel surround sound according to a target right channel signal, an energy value of the target right channel signal, and an energy value of the ambient sound;

[0080] Set the center channel signal in the 5.1-channel surround sound to 0;

[0081] The left front channel signal and the right front channel signal are respectively passed through the reverberator to obtain the left front reverberation signal and the right front reverberation signal. The right ambient sound signal is confirmed according to the target right channel signal, the energy value of the target right channel signal and the energy value of the ambient sound. The left ambient sound signal is confirmed according to the target left channel signal, the energy value of the target left channel signal and the energy value of the ambient sound. The left surround channel signal in the 5.1-channel surround sound is confirmed according to the left ambient sound signal and the left reverberation signal. The right surround channel signal in the 5.1-channel surround sound is confirmed according to the right ambient sound signal and the right reverberation signal.

[0082] Based on the above embodiment, the directional sound signals of the target left channel signal and the target right channel signal are confirmed by the following method:

[0083] According to the time difference and amplitude difference between the target left channel signal and the target right channel signal, the channel time difference and the channel sound level difference are obtained;

[0084] When the channel time difference is within the first time difference range or the channel sound level difference is within the first level difference range, it is confirmed that there is no directional sound signal;

[0085] When the channel time difference is within the second time difference range or the channel upgrade difference is within the second level difference range, it is confirmed that there is a directional sound signal;

[0086] The value within the second time difference range is greater than the value within the first time difference range, and the value within the second level difference range is greater than the value within the first level difference range.

[0087] Based on the above embodiment, determining the normalized cross-correlation parameters of the target left channel signal and the target right channel signal in the target processed signal includes:

[0088] Determining an autocorrelation parameter of a target left channel signal, an autocorrelation parameter of a target right channel signal, and a cross-correlation parameter of the target left channel signal and the target right channel signal;

[0089] According to the autocorrelation parameter of the target left channel signal, the autocorrelation parameter of the target right channel signal, and the cross-correlation parameter, a normalized cross-correlation parameter of the target left channel signal and the target right channel signal is determined.

[0090] The audio signal mixing and playback device provided in the embodiment of the present invention is included in an electronic device and can be used to execute any of the audio signal mixing and playback methods provided in the above embodiments, and has corresponding functions and beneficial effects.

[0091] It is worth noting that in the embodiment of the above-mentioned audio signal mixing and playback device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0092] Figure 6 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. Figure 6 As shown, the electronic device includes a processor 310 and a memory 320, and may also include an input device 330, an output device 340 and a communication device 350; the number of processors 310 in the electronic device may be one or more, Figure 6In the figure, a processor 310 is used as an example; the processor 310, memory 320, input device 330, output device 340 and communication device 350 in the electronic device can be connected via a bus or other means. Figure 6 The bus connection is taken as an example.

[0093] The memory 320, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as the program instructions / modules corresponding to the audio signal mixing and playback method in the embodiments of the present invention. The processor 310 executes the software programs, instructions, and modules stored in the memory 320 to execute various functional applications and data processing of the electronic device, thereby implementing the aforementioned audio signal mixing and playback method.

[0094] The memory 320 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created based on the use of the electronic device, etc. In addition, the memory 320 may include a high-speed random access memory and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 320 may further include a memory remotely located relative to the processor 310, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0095] The input device 330 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the electronic device. The output device 340 may include a display device such as a display screen.

[0096] An embodiment of the present invention also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, it is used to perform the relevant operations in the audio signal mixing and playback method provided in any embodiment of the present application, and has corresponding functions and beneficial effects.

[0097] Those skilled in the art should understand that the embodiments of the present application may be provided as methods, systems, or computer program products.

[0098] Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including the instruction device, which implements the function specified in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0099] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory. Memory may include non-permanent storage in a computer-readable medium, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0100] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0101] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0102] Note that the above are only preferred embodiments of the present invention and the technical principles employed. Those skilled in the art will appreciate that the present invention is not limited to the specific embodiments herein, and that various obvious changes, readjustments, and substitutions are possible for those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments and may include many other equivalent embodiments without departing from the scope of the present invention. The scope of the present invention is determined by the scope of the appended claims.

Claims

1. An audio signal mixing and playback method, characterized in that: include: receiving a two-channel signal, the two-channel signal comprising a left-channel signal and a right-channel signal; During the receiving process, whenever the length of the received two-channel signal reaches a first length, the two-channel signal within the first length is used as the target processing signal; Determining normalized cross-correlation parameters of a target left channel signal and a target right channel signal in the target processed signal; When it is determined, based on the normalized cross-correlation parameter, that the target left channel signal and the target right channel signal are strongly correlated, performing a first separation of directional / ambient sound on the left channel signal and the right channel signal based on a covariance matrix of the target left channel signal and the right channel signal, and determining, based on a result of the first separation, five signals corresponding to 5.1-channel surround sound; When it is confirmed based on the normalized cross-correlation parameter that the target left channel signal and the target right channel signal are weakly correlated, and when it is confirmed based on the target left channel signal and the target right channel signal that there is no directional sound signal, five signals corresponding to 5.1-channel surround sound are confirmed based on the target left channel signal and the target right channel signal; According to the normalized cross-correlation parameter, when it is confirmed that the target left channel signal and the target right channel signal are weakly correlated, and when it is confirmed that there is a directional sound signal according to the target left channel signal and the target right channel signal, a second separation of directional / ambient sound is performed on the target left channel signal and the target right channel signal, and 5 signals corresponding to 5.1-channel surround sound are confirmed according to the result of the second separation.

2. The audio signal mixing and playback method according to claim 1, wherein: The method further comprises performing a first separation of directional / ambient sound on the left channel signal and the right channel signal according to the covariance matrix of the target left channel signal and the right channel signal, and confirming five signals corresponding to 5.1-channel surround sound according to the result of the first separation, including: Determining a covariance matrix based on the target left channel signal, the mean of the target left channel signal, the target right channel signal, and the mean of the target right channel signal; Determining the unit eigenvectors corresponding to the target left channel signal and the target right channel signal in the covariance matrix; Determine the left directional sound signal and the left ambient sound signal according to the target left channel signal and its corresponding unit eigenvector, and determine the right directional sound signal and the right ambient sound signal according to the target right channel signal and its corresponding unit eigenvector; Determine the center channel signal in the 5.1-channel surround sound according to the left direction sound signal and the right direction sound signal; The left direction sound signal and the right direction sound signal are used as the left front channel signal and the right front channel signal in the 5.1 channel surround sound respectively; The left direction sound signal and the right direction sound signal are respectively passed through a reverberator to obtain a left reverberation signal and a right reverberation signal. The left surround channel signal in the 5.1-channel surround sound is confirmed according to the left ambient sound signal and the left reverberation signal, and the right surround channel signal in the 5.1-channel surround sound is confirmed according to the right ambient sound signal and the right reverberation signal.

3. The audio signal mixing and playback method according to claim 1, wherein: The step of confirming five signals corresponding to 5.1-channel surround sound according to the target left channel signal and the target right channel signal includes: The center channel signal, the left front channel signal and the right front channel signal in the 5.1-channel surround sound are set to 0; The target left channel signal and the target right channel signal are respectively passed through a reverberator to obtain a left channel reverberation signal and a right channel reverberation signal, and the left surround channel signal in the 5.1-channel surround sound is confirmed according to the target left channel signal and the left channel reverberation signal, and the right surround channel signal in the 5.1-channel surround sound is confirmed according to the target right channel signal and the right channel reverberation signal.

4. The audio signal mixing and playback method according to claim 1, wherein: The performing a second separation of directional / ambient sound on the target left channel signal and the target right channel signal, and confirming five signals corresponding to 5.1-channel surround sound according to a result of the second separation, includes: Determine the energy values ​​corresponding to the target left channel signal and the target right channel signal, and take the smaller of the two energy values ​​as the energy value of the ambient sound; confirming a left front channel signal in the 5.1-channel surround sound according to the target left channel signal, the energy value of the target left channel signal, and the energy value of the ambient sound; confirming a right front channel signal in the 5.1-channel surround sound according to the target right channel signal, the energy value of the target right channel signal, and the energy value of the ambient sound; The center channel signal in the 5.1-channel surround sound is set to 0; The left front channel signal and the right front channel signal are respectively passed through a reverberator to obtain a left front reverberation signal and a right front reverberation signal; the right ambient sound signal is confirmed according to the target right channel signal, the energy value of the target right channel signal and the energy value of the ambient sound; the left ambient sound signal is confirmed according to the target left channel signal, the energy value of the target left channel signal and the energy value of the ambient sound; the left surround channel signal in the 5.1-channel surround sound is confirmed according to the left ambient sound signal and the left reverberation signal; and the right surround channel signal in the 5.1-channel surround sound is confirmed according to the right ambient sound signal and the right reverberation signal.

5. The audio signal mixing and playback method according to any one of claims 1 to 4, characterized in that: The directional sound signals of the target left channel signal and the target right channel signal are confirmed by the following method: Obtaining a channel time difference and a channel level difference according to a time difference and an amplitude difference between the target left channel signal and the target right channel signal; When the channel time difference is within a first time difference range or the channel sound level difference is within a first level difference range, confirming that there is no directional sound signal; When the channel time difference is within the second time difference range or the channel upgrade difference is within the second level difference range, it is confirmed that there is a directional sound signal; The value within the second time difference range is greater than the value within the first time difference range, and the value within the second level difference range is greater than the value within the first level difference range.

6. The audio signal mixing and playback method according to claim 1, wherein: The determining of the normalized cross-correlation parameters of the target left channel signal and the target right channel signal in the target processed signal includes: Determining an autocorrelation parameter of the target left channel signal, an autocorrelation parameter of the target right channel signal, and a cross-correlation parameter of the target left channel signal and the target right channel signal; According to the autocorrelation parameter of the target left channel signal, the autocorrelation parameter and the cross-correlation parameter of the target right channel signal, a normalized cross-correlation parameter of the target left channel signal and the target right channel signal is determined.

7. An audio signal mixing and reproducing device, characterized in that: include: A signal receiving unit, configured to receive a dual-channel signal, wherein the dual-channel signal includes a left-channel signal and a right-channel signal; a target confirmation unit configured to, during a receiving process, whenever the length of the received two-channel signal reaches a first length, use the two-channel signal within the first length as a target processing signal; a parameter confirmation unit, configured to confirm normalized cross-correlation parameters of a target left channel signal and a target right channel signal in the target processed signal; A first generating unit is configured to, when it is determined, based on the normalized cross-correlation parameter, that the target left channel signal and the target right channel signal are strongly correlated, perform a first separation of directional / ambient sound on the left channel signal and the right channel signal based on a covariance matrix of the target left channel signal and the right channel signal, and determine, based on a result of the first separation, five signals corresponding to 5.1-channel surround sound; A second generating unit is configured to, when confirming, based on the normalized cross-correlation parameter, that the target left channel signal and the target right channel signal are weakly correlated and confirming, based on the target left channel signal and the target right channel signal, that there is no directional sound signal, confirming, based on the target left channel signal and the target right channel signal, 5-way signals corresponding to 5.1-channel surround sound; The third generating unit is used to confirm that the target left channel signal and the target right channel signal are weakly correlated based on the normalized cross-correlation parameter, and when it is confirmed that there is a directional sound signal based on the target left channel signal and the target right channel signal, perform a second separation of directional / ambient sound on the target left channel signal and the target right channel signal, and confirm 5 signals corresponding to 5.1-channel surround sound based on the result of the second separation.

8. The audio signal mixing and reproducing device according to claim 7, characterized in that: The method further comprises performing a first separation of directional / ambient sound on the left channel signal and the right channel signal according to the covariance matrix of the target left channel signal and the right channel signal, and confirming five signals corresponding to 5.1-channel surround sound according to the result of the first separation, including: Determining a covariance matrix based on the target left channel signal, the mean of the target left channel signal, the target right channel signal, and the mean of the target right channel signal; Determining the unit eigenvectors corresponding to the target left channel signal and the target right channel signal in the covariance matrix; Determine the left directional sound signal and the left ambient sound signal according to the target left channel signal and its corresponding unit eigenvector, and determine the right directional sound signal and the right ambient sound signal according to the target right channel signal and its corresponding unit eigenvector; Determine the center channel signal in the 5.1-channel surround sound according to the left direction sound signal and the right direction sound signal; The left direction sound signal and the right direction sound signal are used as the left front channel signal and the right front channel signal in the 5.1 channel surround sound respectively; The left direction sound signal and the right direction sound signal are respectively passed through a reverberator to obtain a left reverberation signal and a right reverberation signal. The left surround channel signal in the 5.1-channel surround sound is confirmed according to the left ambient sound signal and the left reverberation signal, and the right surround channel signal in the 5.1-channel surround sound is confirmed according to the right ambient sound signal and the right reverberation signal.

9. An electronic device, characterized in that: include: one or more processors; a memory for storing one or more computer programs; When the one or more computer programs are executed by the one or more processors, the electronic device implements the audio signal mixing and playback method according to any one of claims 1 to 6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the audio signal mixing and playback method according to any one of claims 1 to 6 is implemented.

Images