Signal generation method and device, readable storage medium and computer program product

Abstract

The invention discloses a signal generation method and device, a readable storage medium and a computer program product, and relates to the technical field of signal processing, the signal generation method and device are applied to audio equipment, the audio equipment comprises an omnidirectional microphone and a splayed directional microphone, and the method comprises the following steps: acquiring a picked original audio signal, the original audio signal comprises a first audio signal and a second audio signal, the first audio signal is an audio signal picked up by an omnidirectional microphone, and the second audio signal is an audio signal picked up by a splayed directional microphone; sound channel separation processing is carried out on the original audio signal to obtain a dual-sound-channel signal, and the dual-sound-channel signal comprises a left sound channel signal and a right sound channel signal; sound source parameters corresponding to the sound source are calculated based on the dual-track signal, the original audio signal is rendered according to the sound source parameters, a generated target audio signal is obtained, and the sound source parameters comprise the sound source orientation and / or the sound source size. According to the invention, the universality of stereo recording is improved.

Description

Technical Field

[0001] The present application relates to the field of signal processing technology, and in particular to a signal generation method, device, readable storage medium and computer program product. Background Art

[0002] The basic characteristics of sound include pitch, timbre, intensity, etc. Stereo (also known as 3D stereo) refers to sound that, in addition to these basic characteristics, also has spatial characteristics such as directionality, sense of distance, and sense of layering. For example, in a concert hall, each instrument has its own position, and the listener can distinguish the direction and distance of the sound source based on the sound they hear. This is the directionality and sense of distance of stereo.

[0003] Traditional stereo recording usually requires two or more microphones for recording, and during the recording process, the two microphones need to be relatively far apart (usually 2 to 3 meters) to achieve stereo recording with obvious left and right separation. Usually, stereo recording can only be performed through larger audio devices, while for smaller audio devices that require tightly integrated microphones, such as mobile phones, tablets, VR (Virtual Reality) glasses, AR (Augmented Reality) glasses, etc., due to the small distance between the microphones, the recording effect may be similar to that of a single microphone, lacking a sense of three-dimensionality, and the universality of stereo recording is low.

[0004] Therefore, how to improve the universality of stereo recording is a technical problem to be solved urgently in this technical field. Summary of the invention

[0005] The main purpose of this application is to provide a signal generation method, device, readable storage medium and computer program product, aiming to solve the technical problem of how to improve the universality of stereo recording.

[0006] To achieve the above object, the present application provides a signal generation method, which is applied to an audio device, wherein the audio device includes an omnidirectional microphone and a figure-of-eight directional microphone, and the signal generation method includes the following steps:

[0007] Acquire the picked-up original audio signal, wherein the original audio signal includes a first audio signal and a second audio signal, the first audio signal is the audio signal picked up by the omnidirectional microphone, and the second audio signal is the audio signal picked up by the figure-of-eight directional microphone;

[0008] Performing channel separation processing on the original audio signal to obtain a dual-channel signal, wherein the dual-channel signal includes a left channel signal and a right channel signal;

[0009] The sound source parameters corresponding to the sound source are calculated based on the two-channel signal, and the original audio signal is rendered according to the sound source parameters to obtain a generated target audio signal, wherein the sound source parameters include the sound source orientation and / or the sound source size.

[0010] In one embodiment, the step of performing channel separation processing on the original audio signal to obtain a dual-channel signal includes:

[0011] Performing alignment processing on the first audio signal and the second audio signal to obtain an aligned audio signal, wherein the alignment processing includes time alignment processing and / or frequency response alignment processing, the aligned audio signal includes a third audio signal and a fourth audio signal, the third audio signal is the first audio signal after the alignment processing, and the fourth audio signal is the second audio signal after the alignment processing;

[0012] Adding the fourth audio signal to the third audio signal obtains a left channel signal, and subtracting the fourth audio signal with an inverted phase from the third audio signal obtains a right channel signal;

[0013] It is determined that the left channel signal and the right channel signal are dual channel signals.

[0014] In one embodiment, after the step of aligning the first audio signal and the second audio signal to obtain an aligned audio signal, the method further includes:

[0015] For the third audio signal and the fourth audio signal of the initial frame, a left channel signal is obtained by adding the third audio signal to the fourth audio signal, and a right channel signal is obtained by subtracting the fourth audio signal with an inverted phase from the third audio signal;

[0016] Determine that the signal value of the third audio signal is a first signal value, determine that the signal value of the left channel signal is a second signal value, and determine that the signal value of the right channel signal is a third signal value;

[0017] Calculating a weighted sum of the second signal value and the third signal value to obtain a target signal value;

[0018] determining a target gain based on the first signal value and the target signal value, wherein the target gain is negatively correlated with the first signal value and the target gain is positively correlated with the target signal value;

[0019] adjusting the third audio signal of the next frame by the target gain to obtain an intermediate audio signal, determining the intermediate audio signal to be a new third audio signal, and returning to the steps of obtaining a left channel signal by adding the fourth audio signal to the third audio signal and obtaining a right channel signal by subtracting the fourth audio signal with an inverted phase from the third audio signal based on the new third audio signal and the fourth audio signal of the next frame;

[0020] After a preset separation end condition is met, it is determined that all the left channel signals and all the right channel signals obtained are dual channel signals.

[0021] In one embodiment, the step of calculating the sound source parameters corresponding to the sound source based on the binaural signal includes:

[0022] Performing frequency band division processing on the dual-channel signal to obtain a plurality of sub-band signals;

[0023] For each of the sub-band signals, a target frequency band to which the sub-band signal belongs is determined, and a sound source parameter corresponding to the sound source in the target frequency band is calculated based on the sub-band signal.

[0024] In one embodiment, the step of calculating the sound source parameter corresponding to the target frequency band based on the sub-band signal includes:

[0025] Calculating spatial cue parameters based on the left channel sub-signal and the right channel sub-signal in the sub-band signal, and determining sound source parameters corresponding to the sound source in the target frequency band based on the spatial cue parameters;

[0026] The left channel sub-signal and the right channel sub-signal belong to the same frequency band.

[0027] In one embodiment, the step of determining the sound source parameters corresponding to the sound source in the target frequency band based on the spatial cue parameters includes:

[0028] If the spatial cue parameter includes interaural time difference and the target frequency band belongs to a preset low-frequency band, determining the sound source direction corresponding to the sound source in the target frequency band based on the interaural time difference;

[0029] If the spatial cue parameter includes binaural level difference and the target frequency band does not belong to the preset low-frequency band, determining the sound source direction corresponding to the sound source in the target frequency band based on the binaural level difference;

[0030] If the spatial cue parameter includes a binaural hearing cross-correlation coefficient, the sound source size corresponding to the sound source in the target frequency band is determined based on the binaural hearing cross-correlation coefficient.

[0031] In one embodiment, the step of calculating the spatial cue parameter based on the left channel sub-signal and the right channel sub-signal in the sub-band signal comprises:

[0032] If there are a plurality of the left channel sub-signals or a plurality of the right channel sub-signals in the sub-band signal, selecting a target left channel sub-signal and a target right channel sub-signal;

[0033] A spatial cue parameter is calculated based on the target left channel sub-signal and the target right channel sub-signal.

[0034] In addition, to achieve the above-mentioned purpose, the present application also provides an audio device, which includes an omnidirectional microphone, an eight-shaped directional microphone and a processor, the omnidirectional microphone and the eight-shaped directional microphone are respectively connected to the processor, and the processor is used to execute the steps of the signal generation method as described above.

[0035] In addition, to achieve the above-mentioned purpose, the present application also provides a readable storage medium, which is a computer-readable storage medium, and a program for implementing the signal generation method is stored on the computer-readable storage medium. The program for implementing the signal generation method is executed by a processor to implement the steps of the signal generation method as described above.

[0036] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned signal generation method when executed by a processor.

[0037] One or more technical solutions proposed in this application have at least the following technical effects:

[0038] An omnidirectional microphone and an eight-shaped directional microphone are integrated in an audio device for recording, and the original audio signals picked up by the omnidirectional microphone and the eight-shaped directional microphone are obtained; the original audio signal is subjected to channel separation processing to obtain a dual-channel signal, wherein the dual-channel signal includes a left channel signal and a right channel signal; the sound source parameters corresponding to the sound source are calculated based on the dual-channel signal, and the original audio signal is rendered according to the sound source parameters to obtain a generated target audio signal, wherein the sound source parameters include the sound source orientation and / or the sound source size. In this way, the embodiment of the present application separates the left channel signal and the right channel signal from the original audio signal picked up by the omnidirectional microphone and the eight-shaped directional microphone, so as to simulate the sound heard by human ears through the left channel signal and the right channel signal, so that the eight-shaped directional microphone and the omnidirectional microphone can be closely arranged, and a very compact miniature directional recording device can be constructed through the eight-shaped directional microphone and the omnidirectional microphone, which occupies a small space and can be easily integrated into a smaller audio device, thereby improving the universality of stereo recording. In addition, the sound source direction and / or sound source size are calculated based on the dual-channel signal, and the sound source direction enables the listener to determine the specific position of the sound source, thereby increasing the listener's spatial directionality perception, and the sound source size can increase the sense of depth of the sound, thereby increasing the listener's spatial depth perception. Therefore, the sound source direction and size can both be used to restore the spatial sense of stereo, and thus render the original audio signal based on the sound source parameters to simulate the sound heard by the listener according to the sound source parameters, so that the target audio signal generated by the rendering can better restore the spatial sense of the sound, that is, the stereo effect of the generated signal is guaranteed. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0040] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0041] Figure 1 This is a flow chart of the first embodiment of the signal generation method of the present application;

[0042] Figure 2 This is a schematic diagram of a calculation ITDG scenario involved in an embodiment of a signal generation method of the present application;

[0043] Figure 3 This is a schematic diagram of an ITDG variation curve involved in an embodiment of a signal generation method of the present application;

[0044] Figure 4 This is a schematic diagram of the structure of the signal generating device of the present application;

[0045] Figure 5 Schematic diagram of the device structure of the hardware operating environment involved in the signal generating device in the embodiment of the present application.

[0046] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0047] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the scope of protection of the present invention.

[0048] With the widespread use of open audio products and equipment such as AR (Augmented Reality), VR (Virtual Reality), and smart audio glasses, users have an increasing demand for immersive near-ear open audio devices. At present, in the application scenario of stereo recording, users usually wear audio equipment, and then use the device's pickup unit (such as a microphone) to pick up sound signals in the surrounding environment. Then, the audio signal obtained after picking up the sound is saved to obtain a stereo recording signal. However, due to the limited spatial perception that the hardware device can present, the recording signal obtained in the above manner cannot accurately express the spatial information of the sound signal, and cannot achieve the effect of human hearing. During subsequent playback, the sense of space cannot be restored, and the user experience is poor.

[0049] Traditional stereo recording usually requires two or more microphones, and the two microphones need to be placed relatively far apart (usually 2 to 3 meters) to achieve stereo recording with clear left and right separation. When the microphones are closely mounted (such as on mobile phones, tablets, VR glasses, AR glasses, etc.), the recording effect may be similar to that of a single microphone, lacking a sense of three-dimensionality.

[0050] Based on this, the main solution of the present application is: integrating an omnidirectional microphone and a figure-of-eight directional microphone in an audio device for recording, obtaining the original audio signals picked up by the omnidirectional microphone and the figure-of-eight directional microphone; performing channel separation processing on the original audio signal to obtain a dual-channel signal, wherein the dual-channel signal includes a left channel signal and a right channel signal; calculating the sound source parameters corresponding to the sound source based on the dual-channel signal, rendering the original audio signal according to the sound source parameters, and obtaining a generated target audio signal, wherein the sound source parameters include the sound source orientation and / or the sound source size.

[0051] The present application separates the left channel signal and the right channel signal from the original audio signal picked up by the omnidirectional microphone and the figure-of-eight directional microphone, so as to simulate the sound heard by human ears through the left channel signal and the right channel signal, so that the figure-of-eight directional microphone and the omnidirectional microphone can be closely arranged. Through the figure-of-eight directional microphone and the omnidirectional microphone, a very compact miniature directional recording device can be constructed, which occupies a small space and can be easily integrated into a small audio device, thereby improving the universality of stereo recording. And the sound source orientation and / or sound source size are calculated based on the two-channel signal, and the sound source orientation enables the listener to judge the specific position of the sound source, thereby increasing the listener's spatial directionality perception, and the sound source size can increase the depth of the sound, thereby increasing the listener's spatial depth perception, so the sound source orientation and size can be used to restore the spatial sense of stereo, thereby rendering the original audio signal based on the sound source parameters, so as to simulate the rendering of the sound heard by the listener according to the sound source parameters, so that the target audio signal generated by the rendering can better restore the spatial sense of the sound, that is, to ensure the stereo effect of the generated signal.

[0052] It should be noted that the executor of each embodiment of the signal generation method of the present application can be an audio device that can realize the above functions, such as headphones, AR glasses, AR helmets, VR glasses, VR helmets, smart audio glasses, etc. This embodiment does not impose any specific restrictions on this.

[0053] Based on this, the present application proposes a signal generation method of the first embodiment, which is applied to an audio device, wherein the audio device includes an omnidirectional microphone and an eight-shaped directional microphone. Figure 1 , the signal generating method comprises steps S10 to S30:

[0054] Step S10, acquiring the picked-up original audio signal, wherein the original audio signal includes a first audio signal and a second audio signal, the first audio signal is the audio signal picked up by the omnidirectional microphone, and the second audio signal is the audio signal picked up by the figure-of-eight directional microphone;

[0055] An omnidirectional microphone is a microphone that is able to capture sound coming from all directions. A figure-of-eight microphone, also called a supercardioid or bicardioid microphone, is a microphone with a specific directivity that is more sensitive in front of and behind the microphone, and less sensitive on the sides.

[0056] The omnidirectional microphone and the figure-eight directional microphone are respectively set at different positions to pick up direct sound signals and ambient sound signals. Among them, the direct sound signal refers to the signal that is directly transmitted from the sound source to the pickup unit without any reflection or diffraction. The ambient sound signal refers to the signal after the sound is reflected, diffracted and scattered during the propagation process. It includes the multiple reflections of the sound wave on the walls, ceilings, floors and other surfaces in the room, as well as the acoustic effects caused by the room geometry and materials.

[0057] Furthermore, the audio device may include one or more omnidirectional microphones and one or more figure-of-eight directional microphones. The present embodiment does not impose any specific limitation on the specific number of the omnidirectional microphones and the figure-of-eight directional microphones.

[0058] After the original audio signal is acquired, the original audio signal may be subjected to noise reduction processing, and subsequent processing may be performed based on the noise-reduced original audio signal to improve the quality of the original audio signal.

[0059] Step S20, performing channel separation processing on the original audio signal to obtain a dual-channel signal, wherein the dual-channel signal includes a left channel signal and a right channel signal;

[0060] It should be noted that the left channel signal and the right channel signal are respectively used to simulate the sound signals heard by human ears.

[0061] As one implementation manner, the step of performing channel separation processing on the original audio signal to obtain a dual-channel signal includes steps S201 to S203:

[0062] Step S201, aligning the first audio signal and the second audio signal to obtain an aligned audio signal, wherein the alignment process includes time alignment process and / or frequency response alignment process, and the aligned audio signal includes a third audio signal and a fourth audio signal, the third audio signal is the first audio signal after the alignment process, and the fourth audio signal is the second audio signal after the alignment process;

[0063] Considering that the omnidirectional microphone and the figure-of-eight directional microphone are set at different positions of the audio device, the different setting positions may cause the signals picked up by the omnidirectional microphone and the figure-of-eight directional microphone to have a certain relative delay. In order to avoid regarding the relative delay of the signal caused by the difference in the position of the omnidirectional microphone and the figure-of-eight directional microphone as caused by the direction of the sound source, this embodiment performs alignment processing on the first audio signal and the second audio signal. Specifically, the relative delay delay between the two can be calculated, and the time alignment processing can be ended after the first audio signal or the second audio signal is delayed by the delay time.

[0064] Specifically, delay=d / c*fs, wherein d represents the distance between the omnidirectional microphone and the figure-of-eight directional microphone, c represents the speed of sound, and fs represents the sampling rates of the omnidirectional microphone and the figure-of-eight directional microphone (the sampling rates of the two are set to be consistent).

[0065] Frequency response refers to the response characteristics of signals with different frequency components. It describes the relationship between the amplitude and phase of the output signal relative to the frequency of the input signal. Specifically, frequency response includes amplitude response and phase response. Amplitude response refers to the characteristic that the amplitude of the output signal changes with the frequency of the input signal. Phase response refers to the characteristic that the phase of the output signal changes with the frequency of the input signal.

[0066] The frequency response alignment process can specifically be to align the amplitude response and phase response of the first audio signal and the second audio signal, that is, to align the amplitude response and phase response of the omnidirectional microphone and the figure-of-eight directional microphone. In a specific embodiment, the output of the figure-of-eight microphone (that is, the second audio signal) is subjected to filter processing in the time-frequency domain, and the amplitude response and phase response of the figure-of-eight microphone output are calibrated, and aligned with the amplitude response and phase response of the omnidirectional microphone, so as to avoid the influence of the directivity and difference between different microphones, and ensure that the amplitude response of the omnidirectional microphone and the figure-of-eight microphone is consistent in the range of 0 to 180 degrees, and there is only a phase difference. Among them, the filtering process in the time-frequency domain includes but is not limited to NLMS (Normalized Least Mean Square) filtering, convolution, IIR (Infinite Impulse Response) filtering, FIR (Finite Impulse Response) filtering, Fourier transform filtering, wavelet transform filtering, equalization processing, etc.

[0067] Preferably, the first audio signal and the second audio signal are time aligned and frequency responded to be aligned to eliminate signal differences caused by non-sound source factors such as the position difference between the omnidirectional microphone and the figure-of-eight microphone and its own frequency response characteristics, thereby improving the quality of subsequent signal separation.

[0068] Step S202, adding the fourth audio signal to the third audio signal to obtain a left channel signal, and subtracting the fourth audio signal with an inverted phase from the third audio signal to obtain a right channel signal;

[0069] Let the third audio signal be Mid and the fourth audio signal be Side, then the left channel signal and the right channel signal can be expressed by the formulas left channel signal = Mid + Side (Mid signal plus the Side signal with no phase inversion), right channel signal = Mid - Side (Mid signal minus the Side signal with phase inversion).

[0070] Step S203: determine whether the left channel signal and the right channel signal are dual channel signals.

[0071] In addition, after obtaining the dual-channel signal, the first audio signal can be gain-controlled based on the weighted average of the amplitude (i.e., signal value) of the left channel signal and the amplitude of the right channel signal as the target amplitude (e.g., if the target amplitude is A and the amplitude of the first audio signal is B, then the first audio signal can be gain-adjusted by A / B) to obtain an intermediate signal of appropriate relative strength. In this way, the left channel signal, the right channel signal, and the intermediate signal are separated from the original audio signal, so that the relevant personnel can achieve the best stereo effect by adjusting the left channel signal, the right channel signal, and the intermediate signal during subsequent mixing.

[0072] As another implementation manner, as one implementation manner, the step of performing channel separation processing on the original audio signal to obtain a dual-channel signal includes steps S204 to S210:

[0073] Step S204, aligning the first audio signal and the second audio signal to obtain an aligned audio signal;

[0074] This alignment process is the same as the above alignment process and will not be described again here.

[0075] Step S205: for the third audio signal and the fourth audio signal of the initial frame, obtain a left channel signal by adding the third audio signal to the fourth audio signal, and obtain a right channel signal by subtracting the fourth audio signal after inverting the phase from the third audio signal;

[0076] For the third audio signal and the fourth audio signal of the initial frame, that is, for the third audio signal of the first frame and the fourth audio signal of the first frame, the first frame left channel signal is obtained by adding the third audio signal of this frame to the fourth audio signal of this frame. Similarly, the first frame right channel audio signal is obtained by subtracting the fourth audio signal of this frame after inverting the phase from the third audio signal of this frame.

[0077] Step S206, determining that the signal value of the third audio signal is a first signal value, determining that the signal value of the left channel signal is a second signal value, and determining that the signal value of the right channel signal is a third signal value;

[0078] The first signal value may specifically be the average value of the signal values ​​corresponding to all sampling points of the first audio signal. Similarly, the second signal value may be the average value of the signal values ​​corresponding to all sampling points of the left channel signal, and the third signal value may be the average value of the signal values ​​corresponding to all sampling points of the right channel signal.

[0079] Step S207, calculating a weighted sum of the second signal value and the third signal value to obtain a target signal value;

[0080] Step S208, determining a target gain based on the first signal value and the target signal value, wherein the target gain is negatively correlated with the first signal value, and the target gain is positively correlated with the target signal value;

[0081] Specifically, the target gain may be a ratio of the first signal value to the target signal value, such as the target signal value divided by the first signal value as the target gain.

[0082] Step S209, adjusting the third audio signal of the next frame by the target gain to obtain an intermediate audio signal, determining the intermediate audio signal to be a new third audio signal, and returning to the step of obtaining a left channel signal by adding the fourth audio signal to the third audio signal and obtaining a right channel signal by subtracting the fourth audio signal with an inverted phase from the third audio signal based on the new third audio signal and the fourth audio signal of the next frame;

[0083] The third audio signal of the next frame is adjusted by the target gain. Specifically, the adjustment may be to multiply the target gain by the signal value, such as multiplying the signal value corresponding to each sampling point by the target gain.

[0084] The adjusted intermediate audio signal is used as a new third audio signal, that is, a third audio signal of the next frame, to perform sum and difference calculation with the fourth audio signal of the next frame to obtain a left channel signal of the next frame and a right channel signal of the next frame.

[0085] Step S210: After a preset separation end condition is met, it is determined that all the left channel signals and all the right channel signals are dual channel signals.

[0086] The preset separation end condition can be a condition set by relevant personnel according to actual needs, such as a certain number of frames of dual-channel signals have been obtained, or the third audio signal or the fourth audio signal of the current frame (the signal frame for which the sum and difference calculation is performed at the time) is the last frame signal, etc. This embodiment does not impose specific restrictions on this.

[0087] It should be noted that one frame of dual-channel signal includes one frame of left-channel signal and one frame of right-channel signal.

[0088] Step S30, calculating sound source parameters corresponding to the sound source based on the two-channel signal, rendering the original audio signal according to the sound source parameters, and obtaining a generated target audio signal, wherein the sound source parameters include the sound source orientation and / or the sound source size.

[0089] The sound source parameters can be used to simulate the sound heard by the human ear, including but not limited to the sound source orientation and / or the sound source size, such as the sound source intensity. Among them, the sound source orientation refers to the spatial position of the sound source relative to the listener, which includes the position of the sound source on the horizontal plane (left and right and front and back) and the vertical plane (up and down); the sound source size refers to the volume sense of the sound source. The sound source size is not completely consistent with the physical size of the sound source, but is related to factors such as the spectral distribution, dynamic range, and reverberation effect of the sound. The sound source size can affect the listener's perception of the spatial characteristics of the sound. The spatial characteristics of the sound include the width, depth, and height of the sound. The sound source intensity usually refers to the loudness of the sound, that is, the strength or volume of the sound. It is a measure of the energy of the sound and is closely related to the sound pressure level (SPL). The sound source intensity determines the degree of influence of the sound on the listener's senses, which is commonly referred to as the "volume size".

[0090] Considering that the orientation of the sound source enables the listener to determine the specific location of the sound source, this is the basis of the stereo system. Through stereo imaging, the sound source can be located in front of, behind, on the left, on the right, above and below the listener. The size of the sound source can increase the sense of depth of the sound. For example, a sound with a longer reverberation time will be perceived as coming from a farther space, thereby increasing the listener's perception of spatial depth. Combining the orientation and size of the sound source, a more realistic and immersive audio signal can be rendered. For example, in a movie theater or home theater, through precise sound source positioning and reverberation effects, the sense of space in a movie scene can be simulated, making the audience feel as if they are in the environment where the story takes place. Based on this, the various embodiments of the present application select the size of the sound source and the orientation of the sound source as sound source parameters so that the generated signal has a better spatial restoration effect.

[0091] The sound source parameters can be calculated based on the two signals in a known manner. For example, the size of the sound source can be estimated based on the frequency components of the two signals (a sound with a wide spectrum will usually be perceived as a larger sound source). The direction of the sound source can be calculated based on the time difference, intensity difference, phase difference and other characteristics of the two signals. This embodiment does not describe in detail the specific calculation process of the sound source parameters.

[0092] After obtaining the sound source parameters, the original audio signal is rendered according to the sound source parameters to obtain a generated target audio signal.

[0093] Signal rendering generally refers to the process of processing a sound signal into a sound that can produce a desired auditory effect on a specific playback device. In this embodiment, in order to restore the spatial sense of the sound, a stereo rendering method is used. Stereo rendering is an audio processing technology that uses two or more channels to create a spatial sense of sound. It controls the volume, frequency content, and time difference (phase difference) between different channels based on sound source parameters, so that when the generated signal is played, the listener feels that the sound comes from a sound source with this sound source parameter.

[0094] In this embodiment, an omnidirectional microphone and an eight-shaped directional microphone are integrated in an audio device for recording, and the original audio signals picked up by the omnidirectional microphone and the eight-shaped directional microphone are obtained; the original audio signal is subjected to channel separation processing to obtain a dual-channel signal, wherein the dual-channel signal includes a left channel signal and a right channel signal; the sound source parameters corresponding to the sound source are calculated based on the dual-channel signal, and the original audio signal is rendered according to the sound source parameters to obtain a generated target audio signal, wherein the sound source parameters include the sound source orientation and / or the sound source size. In this way, this embodiment separates the left channel signal and the right channel signal from the original audio signal picked up by the omnidirectional microphone and the eight-shaped directional microphone, so as to simulate the sound heard by human ears through the left channel signal and the right channel signal, so that the eight-shaped directional microphone and the omnidirectional microphone can be closely arranged, and a very compact miniature directional recording device can be constructed through the eight-shaped directional microphone and the omnidirectional microphone, which occupies a small space and can be easily integrated into a smaller audio device, thereby improving the universality of stereo recording. In addition, the sound source direction and / or sound source size are calculated based on the dual-channel signal, and the sound source direction enables the listener to determine the specific position of the sound source, thereby increasing the listener's spatial directionality perception, and the sound source size can increase the sense of depth of the sound, thereby increasing the listener's spatial depth perception. Therefore, the sound source direction and size can both be used to restore the spatial sense of stereo, and thus render the original audio signal based on the sound source parameters to simulate the sound heard by the listener according to the sound source parameters, so that the target audio signal generated by the rendering can better restore the spatial sense of the sound, that is, the stereo effect of the generated signal is guaranteed.

[0095] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. On this basis, the step of calculating the sound source parameters corresponding to the sound source based on the binaural signal includes:

[0096] Step A10, performing frequency band division processing on the dual-channel signal to obtain a plurality of sub-band signals;

[0097] It should be noted that the audio signal initially picked up by the microphone is usually a time-domain signal. Therefore, if the stereo signal is a time-domain signal, a Fourier transform is performed on the stereo signal to obtain a frequency-domain stereo signal, and a frequency band division process is performed based on the frequency-domain stereo signal.

[0098] One or more specific frequency boundaries can be set in advance, and the stereo signal is subjected to a frequency band division process according to the pre-set frequency boundaries. Exemplarily, assume that the pre-set frequency boundaries are A, B, C, and D, where A < B < C < D. Then, according to this frequency boundary, the stereo signal can be divided into four frequency bands f1, f2, f3, and f4. Specifically, f1, f2, f3, and f4 can be less than A, [A, B), [B, C), and greater than or equal to C, respectively.

[0099] It should be noted that the stereo signal includes a left-channel signal and a right-channel signal. Performing a frequency band division process on the stereo signal means performing a frequency band division process on both the left-channel signal and the right-channel signal. Each sub-band signal obtained by the frequency band division process includes a left-channel sub-signal and a right-channel signal belonging to the same frequency band.

[0100] Exemplarily, denote the stereo signal as M, the left-channel signal in M as L, and the right-channel signal in M as R. After performing a frequency band division process on M according to the frequency boundary in the above example, four sub-band signals S1, S2, S3, and S4 belonging to f1, f2, f3, and f4 are obtained. Then, S1 includes [L1, R1], S2 includes [L2, R2], S3 includes [L3, R3], and S4 includes [L4, R4]. Among them, L1 represents the left-channel sub-signal in L belonging to the f1 frequency band, L2 represents the left-channel sub-signal in L belonging to the f2 frequency band, L3 represents the left-channel sub-signal in L belonging to the f3 frequency band, L4 represents the left-channel sub-signal in L belonging to the f4 frequency band, R1 represents the right-channel sub-signal in R belonging to the f1 frequency band, R2 represents the right-channel sub-signal in R belonging to the f2 frequency band, R3 represents the right-channel sub-signal in R belonging to the f3 frequency band, and R4 represents the right-channel sub-signal in R belonging to the f4 frequency band.

[0101] Step A20: For each of the sub-band signals, determine the target frequency band to which the sub-band signal belongs, and calculate the sound source parameters corresponding to the sound source in the target frequency band based on the sub-band signal.

[0102] It should be noted that based on the target frequency band to which each sub-band signal belongs, the sound source parameters corresponding to the sound source in each target audio are calculated, that is, the sound source parameters corresponding to the sound source in each frequency band are calculated to generate the target audio signal of the frequency band based on the sound source parameters corresponding to each frequency band. Exemplarily, let the sound source parameter be I, and the sub-band signals are S1, S2, S3 and S4, and the target frequency bands they belong to are f1, f2, f3 and f4 respectively. Then, the sound source parameter I1 corresponding to the sound source in the f1 frequency band is calculated based on S1, the sound source parameter I2 corresponding to the sound source in the f2 frequency band is calculated based on S2, the sound source parameter I3 corresponding to the sound source in the f3 frequency band is calculated based on S3, and the sound source parameter I4 corresponding to the sound source in the f4 frequency band is calculated based on S4. The final sound source parameter I specifically includes I1, I2, I3 and I4. During rendering, I1 is used to generate the target audio signal in the f1 frequency band, I2 is used to generate the target audio signal in the f2 frequency band, I3 is used to generate the target audio signal in the f3 frequency band, and I4 is used to generate the target audio signal in the f4 frequency band.

[0103] Taking into account that the sound source parameters depend on the frequency response characteristics of the sound source, the calculated sound source parameters may vary with the frequency. Based on this, this embodiment performs frequency band division processing on the original audio signal picked up by the pickup unit, calculates the sound source parameters corresponding to the sound source in each frequency band, and renders the original audio signal based on the sound source parameters corresponding to each frequency band, so that the sound in each frequency band has a better spatial restoration effect, further improving the spatial restoration effect of the sound.

[0104] Based on the first embodiment and / or the second embodiment of the present application, in the third embodiment of the present application, the same or similar contents as those in the first and second embodiments can be referred to the above introduction, and will not be described in detail later. On this basis, the step of calculating the sound source parameter corresponding to the target frequency band based on the sub-band signal includes:

[0105] Step B10, calculating spatial cue parameters based on the left channel sub-signal and the right channel sub-signal in the sub-band signal, and determining sound source parameters corresponding to the sound source in the target frequency band based on the spatial cue parameters; wherein the left channel sub-signal and the right channel sub-signal belong to the same frequency band.

[0106] Spatial cue parameters refer to various factors that are used to describe the perception of the direction and size of the sound source when the sound propagates in three-dimensional space, and may specifically include interaural time difference (ITD), interaural level difference (ILD), and interaural cross-correlation coefficient (IACC), which are not specifically limited in this embodiment.

[0107] Among them, interaural time difference refers to the time difference between the sound waves reaching the two ears. Since human ears are located on both sides of the head, when a sound source is located on the left or right side of the head, the time when the sound wave reaches the near ear will be earlier than the time when it reaches the far ear. By detecting this time difference, the brain can determine the position of the sound source relative to the head. By measuring the time difference between the sound waves reaching the two ears, the horizontal position of the sound source can be estimated. Specifically, the ITD can be calculated in a known way, such as Where n represents the index of the sampling point (the index of the i-th sampling point is i-1), N represents the total number of sampling points, x1 and x2 represent the left channel sub-signal and the right channel sub-signal respectively, x1(n) represents the signal value of the sampling point with index n of the left channel sub-signal, x2(n) represents the signal value of the sampling point with index n of the right channel sub-signal, d represents the number of delayed samples, argmax d Indicates the value of d when the maximum value is returned.

[0108] Binaural level difference refers to the difference in intensity or volume when sound waves reach the two ears. Since human ears are located on both sides of the head, when a sound source is located on the left or right side of the head, the sound waves will have intensity differences during propagation due to factors such as distance, head occlusion, and auricle filtering effect. The intensity difference of the sound waves reaching the two ears provides the horizontal position and distance information of the sound source. When the sound source is biased to one side, the sound intensity received by the ear far away from the sound source will be reduced. Specifically, the ILD can be calculated in a known manner, such as

[0109] The binaural hearing correlation coefficient is a parameter that measures the similarity of the sound signals received by the two ears. It is a dimensionless parameter with a value range of +1 to -1. Specifically, the IACC can be calculated in a known manner, such as Wherein, p1 represents the sound pressure signal of the left channel sub-signal, and p2 represents the sound pressure signal of the right channel sub-signal. represents the signal mean of the sound pressure signal of the left channel sub-signal, Indicates the signal mean of the sound pressure signal of the right channel sub-signal.

[0110] As for the specific calculation method of the sound source parameters, as one of the implementation methods, a parameter calculation model can be pre-trained by machine learning, so that the spatial clue parameters are input into the pre-trained parameter calculation model to output the sound source parameters.

[0111] As another implementation method, a mapping table can be set based on the correlation between sound source parameters and spatial clue parameters, and the sound source parameters can be obtained based on the mapping table. For example, binaural time difference and binaural sound level difference both have a certain correlation with the sound source orientation. Based on this correlation, the mapping relationship between binaural time difference and sound source orientation, and binaural sound level difference and sound source orientation can be set respectively. The binaural hearing mutual correlation coefficient has a certain correlation with the sound source size. Based on this correlation, the mapping relationship between the binaural hearing mutual correlation coefficient and the sound source size can be set.

[0112] The above are only two feasible implementation methods of calculating sound source parameters based on spatial cue parameters provided by this embodiment. This embodiment does not specifically limit the specific implementation method of calculating sound source parameters based on spatial cue parameters.

[0113] In a possible implementation, the step of determining the sound source parameters corresponding to the sound source in the target frequency band based on the spatial cue parameters includes:

[0114] Step C10, if the spatial cue parameter includes interaural time difference and the target frequency band belongs to a preset low-frequency band, determining the sound source direction corresponding to the sound source in the target frequency band based on the interaural time difference;

[0115] Step C20, if the spatial cue parameter includes binaural level difference and the target frequency band does not belong to the preset low-frequency band, determining the sound source direction corresponding to the sound source in the target frequency band based on the binaural level difference;

[0116] The preset low-frequency band is a frequency band set in advance. Considering that the correlation between the binaural sound level difference and the sound source orientation in the frequency band above 1.5 kHz is stronger, and the correlation between the binaural time difference and the sound source orientation in the frequency band below 1.5 kHz is stronger, preferably, the preset low-frequency band is a frequency band less than or equal to 1.5 kHz.

[0117] It should be noted that in order to avoid the phenomenon that part of a target frequency band belongs to the preset low-frequency band while the other part of the frequency band does not belong to the preset low-frequency band, such as the preset low-frequency band is less than or equal to 1.5kHz, and a target frequency band is 0.8kHz~2kHz, when setting the frequency boundary for frequency band division processing of the original audio signal, the set frequency boundary at least includes the frequency boundary of the preset low-frequency band.

[0118] Step C30: If the spatial cue parameter includes a binaural hearing cross-correlation coefficient, determine the sound source size corresponding to the target frequency band based on the binaural hearing cross-correlation coefficient.

[0119] In this embodiment, considering that in the low-frequency band, the binaural time difference is more sensitive to the change of the sound source orientation and has a strong correlation with the sound source orientation; in the high-frequency band, the binaural level difference is more sensitive to the change of the sound source orientation and has a strong correlation with the sound source orientation, therefore, when the target frequency band belongs to the preset low-frequency band, the sound source orientation corresponding to the sound source in the target frequency band is determined based on the binaural time difference, and when the target frequency band does not belong to the preset low-frequency band, the sound source orientation corresponding to the sound source in the target frequency band is determined based on the binaural level difference, so that the accuracy of the determined sound source orientation can be improved. In addition, the binaural hearing mutual correlation coefficient is more sensitive to the change of the sound source size and has a certain correlation with the sound source size, therefore, the sound source size is determined based on the binaural hearing mutual correlation coefficient.

[0120] In a possible implementation, the step of calculating the spatial cue parameter based on the left channel sub-signal and the right channel sub-signal in the sub-band signal includes:

[0121] Step D10, if there are a plurality of the left channel sub-signals or a plurality of the right channel sub-signals in the sub-band signal, selecting a target left channel sub-signal and a target right channel sub-signal;

[0122] It can be understood that when there are multiple omnidirectional microphones or figure-eight directional microphones, there will be multiple audio signals picked up, and there will also be multiple corresponding sub-audio signals in the sub-band signal. At this time, a target left channel sub-signal and a target right channel sub-signal can be randomly selected.

[0123] Step D20: calculating spatial cue parameters based on the target left channel sub-signal and the target right channel sub-signal.

[0124] In this embodiment, when there are multiple left channel sub-signals or multiple right channel sub-signals, one target left channel sub-signal and one target right channel sub-signal are selected to calculate the spatial cue parameters, thereby reducing the number of audio signals involved in the calculation and thereby reducing the amount of calculation.

[0125] In addition, in another feasible embodiment, a left channel sub-signal and a right channel sub-signal can be combined in sequence, and a spatial cue parameter can be calculated based on the currently combined left channel sub-signal and right channel sub-signal, and finally the average parameter of the spatial cue parameter calculated for all combinations is determined, and the average parameter is used as the final calculated spatial cue parameter. In this way, accidental errors in a certain audio signal, such as accidental failure to pick up a signal, etc., which may cause the calculated spatial cue parameter to be incorrect, can be reduced, thereby improving the accuracy of the final calculated spatial cue parameter.

[0126] For example, assume that the audio device has two omnidirectional microphones and two figure-of-eight directional microphones. The left channel audio signals picked up by the two omnidirectional microphones are L1 and L2 respectively, and the right channel audio signals picked up by the two figure-of-eight directional microphones are R1 and R2 respectively. The original audio signal M includes L1, L2, R1 and R2. After frequency band division processing of M based on two frequency bands f1 and f2, two sub-band signals S1 and S2 are obtained. S1 includes [L1_1, L2_1, R1_1, R2 _1], S2 includes [L1_2, L2_2, R1_2, R2_2], then for the subband signal S1, the spatial clue parameter D1 can be calculated based on L1_1 and R1_1, the spatial clue parameter D2 can be calculated based on L1_1 and R2_1, the spatial clue parameter D3 can be calculated based on L2_1 and R1_1, and the spatial clue parameter D4 can be calculated based on L2_1 and R2_1. The average value of D1, D2, D3 and D4 is used as the spatial clue parameter corresponding to the final S1. It should be noted that when the spatial clue parameter includes multiple parameters, the average value of each parameter is taken.

[0127] In a preferred embodiment, the step of determining the sound source parameters corresponding to the sound source in the target frequency band based on the spatial cue parameters includes:

[0128] Step E10: determining the sound source parameters corresponding to the sound source in the target frequency band based on the preset mapping relationship and the spatial cue parameters.

[0129] Specifically, corresponding mapping relationships can be set according to the corresponding relationships between different spatial clues and different sound source parameters. For example, different mapping relationships between interaural time differences and sound source orientations, different mapping relationships between interaural sound level differences and sound source orientations, and different mapping relationships between interaural hearing correlation coefficients and sound source sizes can be set. In this way, the sound source parameters can be found based on the mapping relationships, which reduces the amount of calculation and improves the calculation efficiency of the sound source parameters.

[0130] Based on the first embodiment, the second embodiment and / or the third embodiment of the present application, in the fourth embodiment of the present application, the same or similar contents as those in the first embodiment, the second embodiment and the third embodiment can be referred to the above introduction, and will not be repeated later. On this basis, the audio device further includes an intermediate sound pickup unit, and after the step of calculating the sound source parameters corresponding to the sound source based on the original audio signal, the method further includes:

[0131] Step F10, obtaining the distance between the intermediate sound pickup unit and the sound source, and obtaining the distance between the sound source and the reflection surface;

[0132] The reflection surface may be a plane where the audio played by the sound source is emitted for the first time during the propagation process. For the convenience of subsequent explanation and description, the distance between the sound source and the reflection surface is recorded as d b The distance between the middle pickup unit and the sound source is d d .

[0133] For example, refer to Figure 2 As shown in the figure, the user wears an audio device, the sound source is a speaker, and the audio played by the speaker is emitted on the wall for the first time during the propagation process. Then the wall is the reflection surface, d b is the distance between the speaker and the wall. The distance between the middle pickup unit and the speaker is close to the distance between the user and the speaker. The distance between the user and the speaker is taken as d d .

[0134] Step F20, taking the distance between the middle sound pickup unit and the sound source as the first distance, and taking the distance between the sound source and the reflecting surface as the second distance;

[0135] Step F30, calculating a first acoustic reflection time difference based on the first distance and the second distance;

[0136] The first acoustic reflection time difference is the rate of change of the time difference between the sound wave reaching the two ears. This parameter reflects how the time difference between the sound wave reaching the two ears changes with the change of the sound source direction due to factors such as the position and shape of the sound source and the shape of the listener's head and ears when the sound wave propagates in space.

[0137] Specifically, the first acoustic reflection time difference (Initial Time-delay Gap, ITDG) can be calculated based on a preset formula, and the preset formula can be: Here, c represents the speed of sound.

[0138] Step F40, determining the spatial size of the space where the sound source is currently located based on the first acoustic reflection time difference;

[0139] For example, refer to Figure 3 As shown, Figure 3 The vertical axis represents ITDG in milliseconds, and the horizontal axis represents d d , unit meter, different d b ( Figure 3 The ITDG changes with d d The change curve of d d Under certain circumstances, ITDG and d b positively correlated, and d bIt is usually positively correlated with the spatial size of the current space, so based on this correlation, the spatial size can be determined according to ITDG.

[0140] The step of rendering the original audio signal according to the sound source parameters to obtain a generated target audio signal comprises:

[0141] Step F50: Rendering the original audio signal according to the space size and the sound source parameter to obtain a generated target audio signal.

[0142] The original audio signal is rendered according to the spatial size and sound source parameters to further improve the spatial restoration effect of the generated target audio signal.

[0143] For example, in order to help understand the technical concept or technical principle of signal generation after the present embodiment is combined with the above-mentioned embodiment 1, embodiment 2 and embodiment 3, a specific embodiment is listed. In this specific embodiment, the signal generation process is as follows:

[0144] (1) First, the omnidirectional microphone picks up the omnidirectional microphone signal (i.e., the first audio signal).

[0145] (2) The figure-of-eight microphone pickup module obtains the figure-of-eight microphone signal (ie, the second audio signal), and calculates the relative delay between the figure-of-eight microphone and the omnidirectional microphone through the positions of the omnidirectional microphone and the figure-of-eight microphone as follows: delay = d / c*fs.

[0146] (3) After the delay processing, the output of the figure-of-eight microphone is subjected to time-frequency domain equalization filter processing. The amplitude response and phase response of the figure-of-eight microphone output are calibrated and aligned with the amplitude response and phase response of the omnidirectional microphone to ensure that the amplitude response of the omnidirectional microphone and the figure-of-eight microphone is consistent within the range of 0 to 180 degrees, with only a phase difference. The filtering processing methods in the time-frequency domain include but are not limited to NLMS filtering, convolution, IIR filtering, FIR filtering, Fourier transform filtering, wavelet transform filtering, equalization processing, etc.

[0147] (4) The processed omnidirectional microphone signal and the figure-of-eight microphone signal are summed and subtracted. When the amplitude-frequency response and phase response of the two are consistent in front, the two signals are added to obtain a beam pointing to the front (i.e., the left channel signal). When the two signals are reversed at 180 degrees, the two signals are subtracted to obtain a beam pointing to the rear (i.e., the right channel signal), thereby achieving left and right channel separation. The information in the 90-degree direction can be provided by the omnidirectional microphone after adjusting the amplitude.

[0148] The signals in the left and right directions can be obtained by amplitude processing through the omnidirectional microphone signal. Based on the weighted average of the left and right channel amplitudes as the target amplitude, the omnidirectional microphone output is gain controlled to obtain the middle channel output information with appropriate relative strength.

[0149] (6) The two-channel signal is subjected to time-frequency transformation and frequency band division processing to obtain sub-band signals belonging to four frequency bands, namely low frequency (less than 200 Hz), mid-low frequency (200 Hz to 1.5 kHz), mid-high frequency (1.5 kHz to 4 kHz) and high frequency (greater than 4 kHz).

[0150] (7) Extracting spatial cue parameters from the sub-band signals of the four frequency bands respectively; the spatial cue parameters include interaural time difference ITD, interaural level difference ILD, and interaural auditory correlation coefficient IACC.

[0151] (8) The size of the sound source is calculated by IACC. The direction of the sound source is calculated based on the correspondence between ITD and the direction of the sound source at low and medium-low frequencies. The direction of the sound source is calculated based on the correspondence between ILD and the direction of the sound source at medium-high and high frequencies.

[0152] (9) Calculate ITDG and estimate the spatial size based on ITDG.

[0153] (10) Rendering the original audio signal according to the sound source orientation, sound source size and spatial size to generate a target audio signal.

[0154] It can be seen that the target audio signal finally generated fully integrates the azimuth information of the sound source and can well express the spatial information of the sound signal. After playback, it can achieve the effect of human hearing and provide a better user experience.

[0155] It should be noted that the above specific embodiments are only used to understand the present application and do not constitute a limitation on the signal generation process of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0156] In addition, the present application also provides a signal generating device, referring to Figure 4 , the signal generating device includes an omnidirectional microphone and an eight-shaped directional microphone, and the signal generating device also includes:

[0157] An acquisition module 10 is used to acquire the picked-up original audio signal, wherein the original audio signal includes a first audio signal and a second audio signal, the first audio signal is the audio signal picked up by the omnidirectional microphone, and the second audio signal is the audio signal picked up by the figure-of-eight directional microphone;

[0158] A separation module 20, configured to perform channel separation processing on the original audio signal to obtain a dual-channel signal, wherein the dual-channel signal includes a left channel signal and a right channel signal;

[0159] The rendering module 30 is used to calculate the sound source parameters corresponding to the sound source based on the two-channel signal, and render the original audio signal according to the sound source parameters to obtain a generated target audio signal, wherein the sound source parameters include the sound source orientation and / or the sound source size.

[0160] In one embodiment, the separation module 20 is further used for:

[0161] Performing alignment processing on the first audio signal and the second audio signal to obtain an aligned audio signal, wherein the alignment processing includes time alignment processing and / or frequency response alignment processing, the aligned audio signal includes a third audio signal and a fourth audio signal, the third audio signal is the first audio signal after the alignment processing, and the fourth audio signal is the second audio signal after the alignment processing;

[0162] Adding the fourth audio signal to the third audio signal obtains a left channel signal, and subtracting the fourth audio signal with an inverted phase from the third audio signal obtains a right channel signal;

[0163] It is determined that the left channel signal and the right channel signal are dual channel signals.

[0164] In one embodiment, the separation module 20 is further used for:

[0165] For the third audio signal and the fourth audio signal of the initial frame, a left channel signal is obtained by adding the third audio signal to the fourth audio signal, and a right channel signal is obtained by subtracting the fourth audio signal with an inverted phase from the third audio signal;

[0166] Determine that the signal value of the third audio signal is a first signal value, determine that the signal value of the left channel signal is a second signal value, and determine that the signal value of the right channel signal is a third signal value;

[0167] Calculating a weighted sum of the second signal value and the third signal value to obtain a target signal value;

[0168] determining a target gain based on the first signal value and the target signal value, wherein the target gain is negatively correlated with the first signal value and the target gain is positively correlated with the target signal value;

[0169] adjusting the third audio signal of the next frame by the target gain to obtain an intermediate audio signal, determining the intermediate audio signal to be a new third audio signal, and returning to the steps of obtaining a left channel signal by adding the fourth audio signal to the third audio signal and obtaining a right channel signal by subtracting the fourth audio signal with an inverted phase from the third audio signal based on the new third audio signal and the fourth audio signal of the next frame;

[0170] After a preset separation end condition is met, it is determined that all the left channel signals and all the right channel signals obtained are dual channel signals.

[0171] In one embodiment, the rendering module 30 is further configured to:

[0172] Performing frequency band division processing on the dual-channel signal to obtain a plurality of sub-band signals;

[0173] For each of the sub-band signals, a target frequency band to which the sub-band signal belongs is determined, and a sound source parameter corresponding to the sound source in the target frequency band is calculated based on the sub-band signal.

[0174] In one embodiment, the rendering module 30 is further configured to:

[0175] Calculating spatial cue parameters based on the left channel sub-signal and the right channel sub-signal in the sub-band signal, and determining sound source parameters corresponding to the sound source in the target frequency band based on the spatial cue parameters;

[0176] The left channel sub-signal and the right channel sub-signal belong to the same frequency band.

[0177] In one embodiment, the rendering module 30 is further configured to:

[0178] If the spatial cue parameter includes interaural time difference and the target frequency band belongs to a preset low-frequency band, determining the sound source direction corresponding to the sound source in the target frequency band based on the interaural time difference;

[0179] If the spatial cue parameter includes binaural level difference and the target frequency band does not belong to the preset low-frequency band, determining the sound source direction corresponding to the sound source in the target frequency band based on the binaural level difference;

[0180] If the spatial cue parameter includes a binaural hearing cross-correlation coefficient, the sound source size corresponding to the sound source in the target frequency band is determined based on the binaural hearing cross-correlation coefficient.

[0181] In one embodiment, the rendering module 30 is further configured to:

[0182] If there are a plurality of the left channel sub-signals or a plurality of the right channel sub-signals in the sub-band signal, selecting a target left channel sub-signal and a target right channel sub-signal;

[0183] A spatial cue parameter is calculated based on the target left channel sub-signal and the target right channel sub-signal.

[0184] In addition, an embodiment of the present application also proposes an audio device, which includes an omnidirectional microphone, an eight-shaped directional microphone and a processor, wherein the omnidirectional microphone and the eight-shaped directional microphone are respectively connected to the processor, and the processor is used to execute the steps of the signal generation method as described above.

[0185] like Figure 5 As shown, the audio device may also include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the audio device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the audio device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows an audio device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0186] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0187] The audio device provided in the embodiment of the present application adopts the signal generation method in the above embodiment to solve the technical problem of how to improve the universality of stereo recording. Compared with the prior art, the beneficial effects of the audio device provided in the present application are the same as the beneficial effects of the signal generation method provided in the above embodiment, and other technical features in the audio device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0188] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0189] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0190] In addition, to achieve the above-mentioned purpose, an embodiment of the present application also provides a readable storage medium having computer-readable program instructions (ie, computer program) stored thereon, and the computer-readable program instructions are used to execute the signal generation method in the above-mentioned embodiment.

[0191] The computer-readable storage medium provided in the embodiment of the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0192] The computer-readable storage medium may be included in the audio device; or may exist independently without being installed in the audio device.

[0193] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by the audio device, the audio device: obtains the picked-up original audio signal, wherein the original audio signal includes a first audio signal and a second audio signal, the first audio signal is the audio signal picked up by the omnidirectional microphone, and the second audio signal is the audio signal picked up by the figure-of-eight directional microphone; performs channel separation processing on the original audio signal to obtain a dual-channel signal, wherein the dual-channel signal includes a left channel signal and a right channel signal; calculates sound source parameters corresponding to the sound source based on the dual-channel signal, renders the original audio signal according to the sound source parameters, and obtains a generated target audio signal, wherein the sound source parameters include the sound source orientation and / or the sound source size.

[0194] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0195] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0196] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0197] The readable storage medium provided in the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned signal generation method, and can solve the technical problem of how to improve the universality of stereo recording. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in the present application are the same as the beneficial effects of the signal generation method provided in the above-mentioned embodiment, and will not be described in detail here.

[0198] In addition, an embodiment of the present application also proposes a computer program product, including a signal generation program, which implements the steps of the signal generation method described above when executed by a processor.

[0199] The specific implementation of the computer program product of the present application is basically the same as the above-mentioned embodiments of the signal generation method, and will not be repeated here.

[0200] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0201] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0202] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software sensor, which is stored in a storage medium (such as ROM / RAM, disk, CD) as described above, including a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0203] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A signal generation method, characterized in that: Applied to an audio device, the audio device includes an omnidirectional microphone and a figure-of-eight directional microphone, and the signal generation method includes the following steps: Acquire the picked-up original audio signal, wherein the original audio signal includes a first audio signal and a second audio signal, the first audio signal is the audio signal picked up by the omnidirectional microphone, and the second audio signal is the audio signal picked up by the figure-of-eight directional microphone; Performing channel separation processing on the original audio signal to obtain a dual-channel signal, wherein the dual-channel signal includes a left channel signal and a right channel signal; The sound source parameters corresponding to the sound source are calculated based on the two-channel signal, and the original audio signal is rendered according to the sound source parameters to obtain a generated target audio signal, wherein the sound source parameters include the sound source orientation and / or the sound source size.

2. The signal generating method according to claim 1, characterized in that: The step of performing channel separation processing on the original audio signal to obtain a dual-channel signal comprises: Performing alignment processing on the first audio signal and the second audio signal to obtain an aligned audio signal, wherein the alignment processing includes time alignment processing and / or frequency response alignment processing, the aligned audio signal includes a third audio signal and a fourth audio signal, the third audio signal is the first audio signal after the alignment processing, and the fourth audio signal is the second audio signal after the alignment processing; Adding the fourth audio signal to the third audio signal obtains a left channel signal, and subtracting the fourth audio signal with an inverted phase from the third audio signal obtains a right channel signal; It is determined that the left channel signal and the right channel signal are dual channel signals.

3. The signal generating method according to claim 2, characterized in that: After the step of aligning the first audio signal and the second audio signal to obtain an aligned audio signal, the method further includes: For the third audio signal and the fourth audio signal of the initial frame, a left channel signal is obtained by adding the third audio signal to the fourth audio signal, and a right channel signal is obtained by subtracting the fourth audio signal with an inverted phase from the third audio signal; Determine that the signal value of the third audio signal is a first signal value, determine that the signal value of the left channel signal is a second signal value, and determine that the signal value of the right channel signal is a third signal value; Calculating a weighted sum of the second signal value and the third signal value to obtain a target signal value; determining a target gain based on the first signal value and the target signal value, wherein the target gain is negatively correlated with the first signal value and the target gain is positively correlated with the target signal value; adjusting the third audio signal of the next frame by the target gain to obtain an intermediate audio signal, determining the intermediate audio signal to be a new third audio signal, and returning to the steps of obtaining a left channel signal by adding the fourth audio signal to the third audio signal and obtaining a right channel signal by subtracting the fourth audio signal with an inverted phase from the third audio signal based on the new third audio signal and the fourth audio signal of the next frame; After a preset separation end condition is met, it is determined that all the left channel signals and all the right channel signals obtained are dual channel signals.

4. The signal generating method according to claim 1, characterized in that: The step of calculating the sound source parameters corresponding to the sound source based on the binaural signal comprises: Performing frequency band division processing on the dual-channel signal to obtain a plurality of sub-band signals; For each of the sub-band signals, a target frequency band to which the sub-band signal belongs is determined, and a sound source parameter corresponding to the sound source in the target frequency band is calculated based on the sub-band signal.

5. The signal generating method according to claim 4, characterized in that: The step of calculating the sound source parameter corresponding to the target frequency band based on the sub-band signal comprises: Calculating spatial cue parameters based on the left channel sub-signal and the right channel sub-signal in the sub-band signal, and determining sound source parameters corresponding to the sound source in the target frequency band based on the spatial cue parameters; The left channel sub-signal and the right channel sub-signal belong to the same frequency band.

6. The signal generating method according to claim 5, characterized in that: The step of determining the sound source parameters corresponding to the sound source in the target frequency band based on the spatial cue parameters comprises: If the spatial cue parameter includes interaural time difference and the target frequency band belongs to a preset low-frequency band, determining the sound source direction corresponding to the sound source in the target frequency band based on the interaural time difference; If the spatial cue parameter includes binaural level difference and the target frequency band does not belong to the preset low-frequency band, determining the sound source direction corresponding to the sound source in the target frequency band based on the binaural level difference; If the spatial cue parameter includes a binaural hearing cross-correlation coefficient, the sound source size corresponding to the sound source in the target frequency band is determined based on the binaural hearing cross-correlation coefficient.

7. The signal generating method according to claim 5, characterized in that: The step of calculating the spatial cue parameter based on the left channel sub-signal and the right channel sub-signal in the sub-band signal comprises: If there are a plurality of the left channel sub-signals or a plurality of the right channel sub-signals in the sub-band signal, selecting a target left channel sub-signal and a target right channel sub-signal; A spatial cue parameter is calculated based on the target left channel sub-signal and the target right channel sub-signal.

8. An audio device, characterized in that: The audio device includes an omnidirectional microphone, a figure-of-eight directional microphone and a processor, wherein the omnidirectional microphone and the figure-of-eight directional microphone are respectively connected to the processor, and the processor is used to execute the steps of the signal generation method as described in any one of claims 1 to 7.

9. A readable storage medium, characterized in that: The readable storage medium is a computer-readable storage medium, on which is stored a program for implementing the signal generating method, and the program for implementing the signal generating method is executed by a processor to implement the steps of the signal generating method as described in any one of claims 1 to 7.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the signal generation method according to any one of claims 1 to 7 are implemented.

Images