Pickup method, microphone module, portable device, computer storage medium and program product

Abstract

Embodiments of the present application provide a pickup method, a microphone module, a portable device, a computer storage medium and a program product, wherein the pickup method comprises: collecting a sound signal in an environment by a microphone module, wherein the microphone module comprises a pair of double-directional microphones and an omnidirectional microphone, a first double-directional microphone and a second double-directional microphone in the pair of double-directional microphones are close to each other in space and their directional axis lines are perpendicular to each other, and the omnidirectional microphone is arranged at or near the intersection of the directional axis lines; based on the sound signal, performing sound source positioning to determine a target sound direction; adjusting a pickup beam direction of the microphone module to the target sound direction to perform a pickup operation for the target sound direction to obtain a target sound signal. Through the embodiments of the present application, the effect of effectively picking up the target speaker sound by the portable device is realized.

Description

Technical Field

[0001] This application relates to the field of audio processing technology, and in particular to a sound pickup method, microphone module, portable device, computer storage medium, and computer program product. Background Technology

[0002] With the widespread adoption of intelligent technologies, there is an increasing demand for convenient, real-time audio data acquisition and subsequent intelligent processing based on that data. Consequently, portable devices capable of real-time audio data acquisition and translation or intelligent summarization are becoming increasingly common.

[0003] However, real-world sound environments are often complex, especially in open spaces or noisy environments. Noise and interference make it difficult for portable devices to clearly pick up the voice of the target speaker. Traditional methods using microphone arrays to achieve beamforming, thereby more focused pickup of the target speaker's voice and suppression of ambient noise and other interference, result in large microphone arrays that are difficult to deploy in miniaturized devices. Therefore, how to enable portable devices to effectively pick up the voice of the target speaker has become an urgent problem to be solved. Summary of the Invention

[0004] In view of this, embodiments of this application provide a sound pickup scheme to at least partially solve the above-mentioned problems.

[0005] According to a first aspect of the embodiments of this application, a sound pickup method is provided, comprising: acquiring sound signals in the environment through a microphone module, wherein the microphone module includes a pair of bidirectional microphones and an omnidirectional microphone, wherein the first bidirectional microphone and the second bidirectional microphone in the pair of bidirectional microphones are spatially close to each other and their directional axes are orthogonal to each other, and the omnidirectional microphone is disposed at or near the intersection of the directional axes; performing sound source localization based on the sound signals to determine the direction of a target sound; adjusting the pickup beam of the microphone module to the direction of the target sound to perform a sound pickup operation for the direction of the target sound, thereby obtaining a target sound signal.

[0006] According to a second aspect of the embodiments of this application, a microphone module is provided, comprising: a pair of bidirectional microphones and an omnidirectional microphone, wherein the pair of bidirectional microphones includes a first bidirectional microphone and a second bidirectional microphone; the first bidirectional microphone and the second bidirectional microphone in the pair of bidirectional microphones are spatially close to each other and their directional axes are orthogonal to each other; and the omnidirectional microphone is disposed at or near the intersection of the directional axes.

[0007] According to a third aspect of the embodiments of this application, a portable device is provided, comprising at least: an audio processing chip, a microphone module as described in the second aspect, and a memory; the audio processing chip is communicatively connected to the microphone module and the memory; the microphone module is used to collect sound signals from the environment; the microphone module includes a pair of bidirectional microphones and an omnidirectional microphone, wherein the first bidirectional microphone and the second bidirectional microphone in the pair of bidirectional microphones are spatially close to each other and their directional axes are orthogonal to each other, and the omnidirectional microphone is disposed at or near the intersection of the directional axes; the memory is used to store at least one executable instruction, the executable instruction causing the audio processing chip to perform an operation corresponding to the sound pickup method described in the first aspect based on the sound signals collected by the microphone module.

[0008] According to a fourth aspect of the embodiments of this application, a computer storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect.

[0009] According to a fifth aspect of the embodiments of this application, a computer program product is provided, including computer instructions that instruct a computing device to perform an operation corresponding to the method described in the first aspect.

[0010] According to the embodiments of this application, on one hand, a microphone module is provided, which includes a pair of bidirectional microphones and an omnidirectional microphone. The first and second bidirectional microphones in the bidirectional microphone pair are spatially close to each other and their directional axes are orthogonal, forming an omnidirectional orthogonal directional base. Since the directionality of the bidirectional microphones originates from their own acoustic structure rather than the physical aperture of the microphone array, their directionality has natural frequency consistency within the effective pickup frequency range, eliminating the need to rely on large physical spacing to maintain low-frequency directionality. This allows the entire microphone module to be deployed in a very compact size in a miniaturized portable device. The addition of the omnidirectional microphone effectively reduces interference outside the target sound direction, enabling the entire microphone module to more effectively pick up sound from the target sound direction. On the other hand, based on the microphone module, by performing sound source localization to determine the target sound direction and dynamically adjusting the pickup beam of the microphone module to the target sound direction, it is possible to adaptively track the target speaker, achieve focused pickup of the target sound signal, and simultaneously spatially suppress noise and interference from other directions. This enables portable devices to effectively pick up the voice of the target speaker. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings.

[0012] Figure 1A This is a flowchart illustrating the steps of a sound pickup method according to an embodiment of this application; Figure 1B for Figure 1A A schematic diagram of the acoustic deployment of a microphone module in the illustrated embodiment; Figure 1C for Figure 1A A schematic diagram of the microphone module's pickup beam pointing adjustment process in the illustrated embodiment; Figure 1D for Figure 1A A schematic diagram illustrating an example of a sound pickup scenario in the illustrated embodiment; Figure 2 This is a schematic diagram of the structure of a portable device according to an embodiment of this application. Detailed Implementation

[0013] To enable those skilled in the art to better understand the technical solutions in the embodiments of this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art should fall within the protection scope of the embodiments of this application.

[0014] The specific implementation of the embodiments of this application will be further described below with reference to the accompanying drawings.

[0015] Reference Figure 1A The diagram illustrates a flowchart of a sound pickup method according to an embodiment of this application. The sound pickup method of this embodiment includes the following steps: Step S102: Collect sound signals from the environment using the microphone module.

[0016] The microphone module includes a pair of bidirectional microphones and an omnidirectional microphone. The first and second bidirectional microphones in the bidirectional microphone pair are close to each other in space and their directional axes are orthogonal to each other. The omnidirectional microphone is located at or near the intersection of the directional axes.

[0017] Bidirectional microphones, also known as figure-eight microphones, have a pickup pattern resembling the number "8". Their acoustic sensitivity is highest in the positive and negative directions of the axis (the two poles of the "8"), while sensitivity approaches zero in the direction perpendicular to the axis. This directivity stems from the microphone's acoustic design; the diaphragm can simultaneously sense sound pressure from both the front and back, generating output when there is a difference in sound pressure levels on both sides, rather than relying on the arrival time difference caused by the physical distance between the microphones. Therefore, the directivity of bidirectional microphones remains consistent across a wide frequency range, unlike traditional differential microphone arrays which suffer from directivity degradation at low frequencies due to the microphone spacing-to-sound-wave-wave ratio approaching zero. Omnidirectional microphones, on the other hand, have a spherical (three-dimensional) or circular (two-dimensional) pickup pattern, exhibiting nearly uniform sensitivity to sound from all directions. Their output signal contains omnidirectional sound information from the microphone's location.

[0018] In this embodiment, a pair of bidirectional microphones, namely the aforementioned bidirectional microphone pair, is provided, comprising a first bidirectional microphone and a second bidirectional microphone. These two bidirectional microphones are spatially close to each other, and their directional axes are orthogonal to each other. Thus, the figure-eight directional patterns of the first and second bidirectional microphones form a set of mutually perpendicular orthogonal bases in space. In a planar coordinate system, the axial direction of the first bidirectional microphone can be considered as the horizontal axis (X-axis), and the axial direction of the second bidirectional microphone can be considered as the vertical axis (Y-axis). Sound signals in any azimuth angle θ can be decomposed into components in these two orthogonal directions, corresponding to the respective pickup responses of the two bidirectional microphones. The spatial proximity of the two bidirectional microphones, meaning their spatial positions are close, makes their distance much smaller than the wavelength, ideally allowing them to be considered as sampling from the same point. This close deployment ensures that the spatial reference points corresponding to the signals collected by the two bidirectional microphones are consistent, avoiding the impact of phase differences introduced by the microphone spacing on subsequent processing such as beamforming, and also making the physical size of the entire microphone module more compact.

[0019] In one alternative approach, a first bidirectional microphone is positioned along the lateral direction of the device in which the microphone module is located, and a second bidirectional microphone is positioned along the longitudinal direction of the device, thereby aligning the orthogonal directional base of the microphone module with the geometric direction of the portable device itself, which facilitates subsequent calculation of the target sound direction based on the device's orientation.

[0020] In another alternative approach, both the first and second bidirectional microphones are implemented using MEMS (Micro-Electro-Mechanical System) microphones. MEMS microphones have advantages such as small size, low power consumption, good consistency, and ease of integration with signal processing circuits, making them suitable for deployment in portable devices. However, this is not the only option; other bidirectional microphones that can be deployed in portable devices are also applicable to the solutions described in this application.

[0021] In this embodiment, the omnidirectional microphone is positioned at or near the intersection of the directional axes of the two bidirectional microphones to ensure that the signal acquired by the omnidirectional microphone is consistent with the signal acquired by the bidirectional microphone at the spatial reference point, thereby providing an omnidirectional sound reference for subsequent beamforming.

[0022] An exemplary acoustic deployment of a microphone module, such as Figure 1B As shown, in Figure 1B In the diagram, the first bidirectional microphone, mic1, is positioned horizontally (X-axis). The figure-eight-shaped dashed line on its outer side represents its pickup pattern. The two elliptical lobes extend to the left and right ends of mic1's axis, indicating that mic1 has high sensitivity to sound from the horizontal sides, while its sensitivity to sound from the vertical direction approaches zero. The second bidirectional microphone is... Figure 1B The diagram shows mic2, which is deployed along the longitudinal direction (Y-axis). The figure-eight-shaped dashed line on its outer side represents its pickup pattern. The two elliptical lobes extend upwards and downwards respectively, indicating that mic2 has high sensitivity to sounds from the front and rear longitudinally, and its sensitivity to lateral sounds is close to zero.

[0023] Depend on Figure 1B As can be seen, the directional axes of mic1 and mic2 are perpendicular and orthogonal to each other, and the two sets of figure-eight patterns form a set of orthogonal directional bases covering the entire plane in space. Omnidirectional microphones... Figure 1B The symbol mic3, located at the center of the intersection of two bidirectional microphones, mic1 and mic2, is marked as a small circle. The dashed line surrounding it forms a nearly perfect circle, indicating that the omnidirectional microphone mic3 has almost uniform sensitivity to sound from all directions. As can be seen, mic3 is positioned at the intersection of the two directional axes corresponding to mic1 and mic2, ensuring that the omnidirectional sound signal it collects is highly aligned spatially with the sound signals collected by mic1 and mic2, providing a precise reference for subsequent beamforming.

[0024] In practical sound pickup scenarios, the sound signals collected by the microphone module are usually composite signals superimposed from multiple sound sources, including the target speaker's voice signal, sound signals from other interfering speakers from other directions, environmental background noise, and various reflection and reverberation components. The microphone module of this application embodiment can simultaneously acquire multiple sound signals carrying omnidirectional directional information, providing sufficient acoustic information input for subsequent sound source localization and beam pointing adjustment.

[0025] Step S104: Based on the collected sound signals, perform sound source localization to determine the direction of the target sound.

[0026] After acquiring multiple sound signals through the microphone module, it is necessary to extract the directional information of the target sound source from these signals, i.e., sound source localization, so that the subsequent pickup beam can be accurately pointed to the target sound source.

[0027] Sound source localization is a process of inferring the direction or position of a sound source (such as a target speaker) relative to the microphone module by analyzing the sound signals collected by the microphone module. Traditional sound source localization algorithms, such as the TDOA algorithm based on time difference of arrival, the MUSIC algorithm based on spatial spectrum, and the ESPRIT algorithm, mostly rely on the physical characteristics of the microphone, such as the microphone spacing, the number of microphones, and the microphone array aperture. They are either inaccurate in localization or require large sizes, making them unsuitable for portable devices.

[0028] In one optional embodiment of this application, a lightweight deep learning model is employed. This approach allows for the use of a deep learning model with appropriate functionality, tailored to the specific needs of the scenario, to perform multi-dimensional feature learning based on the sound signals collected by the microphone module, outputting information that can be used to determine the direction of the target sound. This overcomes the dependence of traditional sound source localization methods on the physical characteristics of the microphone, enabling high-precision, full-bandwidth, and robust sound source localization on miniaturized portable devices.

[0029] In one optional embodiment of this application, the target sound direction is picked up by adjusting the weights of the first and second bidirectional microphones. Based on this, when determining the target sound direction, optionally, the collected sound signal can be input into a first deep learning model. The first deep learning model extracts multi-channel spatial acoustic features from the sound signal to determine the target sound direction, and outputs weights for adjusting the output signals of the first and second bidirectional microphones based on the target sound direction. For example, the first deep learning model can extract features from multiple dimensions such as signal amplitude, phase, and spectrum to obtain a complex mapping relationship between the sound source direction and signal features, achieving high-precision sound source localization on portable devices based on microphone modules.

[0030] In one example, the first deep learning model takes multiple audio signals (including signals from two bidirectional microphones and an omnidirectional microphone) collected by a microphone module as input, i.e., multi-channel audio signals. Spatial acoustic features are extracted from these multi-channel audio signals, such as spatial features reflecting sound sources from different directions, like cross-channel correlation, phase difference, and energy ratio, from the corresponding time-domain or frequency-domain signals. After extracting the spatial acoustic features, the first deep learning model further analyzes these features to determine the target sound direction and directly outputs weights for adjusting the output signals of the first and second bidirectional microphones. This integrates sound source localization and weight calculation into the same model, simplifying model processing, reducing system computational latency, and making it more suitable for portable device applications with high real-time requirements.

[0031] The first deep learning model can be any appropriate lightweight deep learning model adopted by those skilled in the art according to actual needs, including but not limited to the UNET model. In one example, the computational cost of the UNET model is less than 10M computing power units.

[0032] However, not limited to the above methods, in another alternative method, the collected sound signal can be input into a second deep learning model, the second deep learning model can extract the spatial location features of the sound signal, and output spatial direction information to indicate the direction of the target sound.

[0033] Unlike the first deep learning model, the second deep learning model extracts spatial location features from the sound signal and outputs a quantitative description of the target sound direction (i.e., spatial direction information), rather than outputting numerical values ​​used to adjust the weights of the bidirectional microphones. Spatial location features reflect the position of the sound source in space and may include, but are not limited to, phase difference features, amplitude ratio features, and cross-channel power spectral ratio features of the sound signals from each channel. The second deep learning model can perform deep nonlinear modeling based on spatial location features, outputting spatial direction information such as azimuth and elevation angles. Subsequently, the weights of the output signals from the first and second bidirectional microphones are calculated based on this spatial direction information. This approach makes the model structure relatively simple and provides better versatility, applicable to a wider range of beamforming schemes. Similar to the first deep learning model, the specific implementation of the second deep learning model can also be achieved by those skilled in the art using any appropriate lightweight deep learning model according to actual needs, including but not limited to the UNET model.

[0034] The results of sound source localization can serve as the basis for subsequent adjustments to the microphone module's pickup beam direction. Accurate sound source localization in this step can improve the precision of adjusting the pickup beam direction, thereby enhancing the pickup effect on the target speaker's voice and the ability to suppress interference sounds.

[0035] Step S106: Adjust the microphone module's pickup beam direction to the target sound direction to perform pickup operation in the target sound direction and obtain the target sound signal.

[0036] After determining the direction of the target sound, this step adjusts the microphone module's pickup beam to that direction, thereby achieving focused pickup of the target speaker's voice while spatially suppressing noise and interference from non-target directions.

[0037] The pickup beam is the directional pattern of a microphone module's pickup sensitivity in space, that is, the distribution of sensitivity to sound signals in different directions. The beam direction is the principal axis direction of this sensitivity pattern, that is, the direction with the highest sound sensitivity. By adjusting the pickup beam direction, it is possible to prioritize the pickup of sound from a specific direction while suppressing sound from other directions.

[0038] In this embodiment, the direction of the microphone beam can be adjusted by adjusting the weights of the output signals of the two bidirectional microphones. That is, the weights of the output signals of the first and second bidirectional microphones are adjusted according to the target sound direction so that the microphone module's microphone beam is directed towards the target sound direction. Because the directional axes of the first and second bidirectional microphones are orthogonal, they exhibit orthogonal decomposition characteristics. Therefore, by adjusting the weights of the two microphone output signals, the microphone module's microphone beam can be effectively directed towards the target sound direction.

[0039] In one alternative approach, adjusting the weights of the output signals of the first and second bidirectional microphones according to the target sound direction, so that the pickup beam of the microphone module is directed towards the target sound direction, may include: determining a first weight corresponding to the output signal of the first bidirectional microphone and a second weight corresponding to the output signal of the second bidirectional microphone according to the target sound direction; multiplying the first bidirectional microphone output signal by the first weight to obtain a first adjustment result; multiplying the second bidirectional microphone output signal by the second weight to obtain a second adjustment result; and directing the pickup beam of the microphone module towards the target sound direction according to the first and second adjustment results.

[0040] In one example, the azimuth angle of the target sound direction is set as θ, the first weight is set as w1, and the second weight is set as w2. w1 is then proportional to cos(θ), and w2 is proportional to sin(θ). The output signal of the first bidirectional microphone can be multiplied by w1 to obtain the first adjustment result; the output signal of the second bidirectional microphone can be multiplied by w2 to obtain the second adjustment result. Thus, the output signals of the two bidirectional microphones can be directionally weighted at the signal level, amplifying the component corresponding to the target sound direction in each output signal, while components in other directions gradually attenuate as the deviation from the target sound direction increases. However, the calculation method for the weights is not limited to the above method. In practical applications, those skilled in the art can choose other forms of weight calculation methods according to actual needs, and this application embodiment does not impose such limitations. Furthermore, the weights can be real values ​​or complex values ​​(to accommodate both amplitude and phase adjustments), depending on the system requirements.

[0041] If the output signal of the first bidirectional microphone is multiplied by w1, and the output signal of the second bidirectional microphone is multiplied by w2, and then the multiplication results are superimposed, the response amplitude of the superimposed synthesized signal in the direction of the azimuth angle θ is proportional to (w1·cos(θ) + w2·sin(θ)). It can be seen that by selecting appropriate w1 and w2, the spatial response of the synthesized signal can be maximized at the target sound direction, thus pointing the synthesized beam towards the target sound direction. This allows for effective adjustment of the pickup beam direction without relying on the physical characteristics of the microphones.

[0042] Based on this, in one optional approach, making the microphone module's pickup beam point towards the target sound direction according to the first adjustment result and the second adjustment result may include: superimposing the first adjustment result and the second adjustment result to obtain a virtual bidirectional signal pointing towards the target sound direction; and performing beamforming on the virtual bidirectional signal and the output signal of the omnidirectional microphone according to a preset ratio so that the microphone module's pickup beam points towards the target sound direction.

[0043] In this embodiment, the virtual bidirectional signal is a synthesized signal obtained by superimposing (summing) the weighted first adjustment result and the second adjustment result. This signal does not physically originate from any real bidirectional microphone with its directional axis pointing towards the target sound direction. Instead, it is synthesized by mathematical operations on the output signals of two spatially fixed bidirectional microphones. The effect is equivalent to rotating the directional axis of a bidirectional microphone to the azimuth angle corresponding to the target sound direction. Assuming the output signals of the first and second bidirectional microphones are S1 (which can be implemented as cos(θ)) and S2 (which can be implemented as sin(θ)), respectively, the superimposed synthesized signal is (w1·S1 + w2·S2). This synthesized signal is equivalent to the output signal of a virtual bidirectional microphone with its directional axis pointing towards the azimuth angle θ. Its pickup pattern remains a figure-eight shape, but the main axis direction now points towards the target sound direction.

[0044] As can be seen, by changing the values ​​of w1 and w2, the directional axis of the virtual bidirectional signal can be rotated to any azimuth angle in the plane, thereby achieving omnidirectional beam steering without any mechanical moving parts. Because it is a beam steering method at the purely signal processing level, it not only has an extremely fast response speed but is also completely wear-free, making it very suitable for portable device applications that require real-time tracking of dynamic sound sources.

[0045] Furthermore, the virtual bidirectional signal can be beamformed with the output signal of the omnidirectional microphone. Although the virtual bidirectional signal can point the main axis of the pickup beam toward the direction of the target sound, its figure-eight pickup pattern also has high sensitivity in the opposite direction of the target direction (i.e., the back lobe direction), which means that sound from the back of the target sound direction will also be picked up. To eliminate the back lobe, the output signal of the omnidirectional microphone is introduced to participate in beamforming in this embodiment.

[0046] An omnidirectional microphone has the same sensitivity to sound from all directions. Its output signal includes sound from both the target sound direction (main lobe) and the back lobe direction, with equal response amplitudes in both directions. By beamforming (e.g., adding) a virtual bidirectional signal to the omnidirectional microphone's output signal at an appropriate preset ratio, the two signals can be mutually enhanced in the target sound direction, while canceling each other out in the opposite direction (back lobe direction). The amplitude ratio of the omnidirectional microphone's output signal to the virtual bidirectional signal in the preset ratio can be selected within the range of 1:3 to 1:1. When the ratio is 1:1, the resulting supercardioid pickup pattern (i.e., the beam-combined pickup beam is a cardioid pickup beam pointing towards the target sound) has near-zero sensitivity at the rear, exhibiting the strongest suppression capability against interference from both the front and back of the target sound direction. When the ratio is approximately 1:1.73, the resulting supercardioid pickup pattern (i.e., the beam-combined pickup beam is a supercardioid pickup beam pointing towards the target sound) has good random energy efficiency, making it suitable for complex open acoustic environments where background noise is diffusely distributed. When the ratio is 1:3, the resulting supercardioid pickup pattern (i.e., the beam-combined pickup beam is a supercardioid pickup beam pointing towards the target sound) has the largest directivity factor and the most concentrated main lobe, exhibiting the strongest suppression capability against lateral interference. Those skilled in the art can flexibly select preset ratios within the above ranges based on the directional distribution characteristics of interference in actual application scenarios; this application does not impose such limitations.

[0047] Furthermore, in this embodiment, the microphone module's pickup beam exhibits consistent directivity across the effective pickup frequency range. Consistent directivity means that the directional pattern of the pickup beam remains stable across the entire effective pickup frequency range (such as the main frequency range of the speech signal), without significant drift or distortion with frequency changes. Because the figure-eight directivity of a bidirectional microphone originates from a differential acoustic structure, it naturally maintains consistency across a wide frequency band, independent of the ratio between the microphone's physical spacing and the sound wave wavelength. This is substantially different from the inherent defect of inconsistent directivity in traditional differential microphone arrays across different frequency bands. Based on this, in the pickup scheme of this embodiment, the virtual bidirectional signal synthesized through weight adjustment of the bidirectional microphone pair inherits the consistent directivity characteristic of the bidirectional microphone across the entire frequency band, thereby ensuring that the final synthesized pickup beam maintains consistent directivity across the effective pickup frequency range. Therefore, in the context of the speaker's voice signal, the target speaker's voice can be picked up with uniform directional selectivity across the entire voice frequency range. There is no problem of insufficient low-frequency noise suppression caused by weakened directivity in the low-frequency band. Thus, a uniform and stable noise suppression effect is obtained within the effective pickup frequency range of the voice signal (such as the full frequency band), improving the overall consistency of the pickup quality.

[0048] It should be noted that the target sound signal formed after beamforming is a single-channel sound signal.

[0049] The following, combined with Figure 1C The process of adjusting the pickup beam direction of the microphone module described above is illustrated by way of example.

[0050] Figure 1C It contains three polar coordinate pickup patterns. The first pickup pattern shows the beam pattern of an omnidirectional microphone, the second shows the beam pattern of a virtual bidirectional microphone, and the third is the beam pattern formed by combining the beams of the omnidirectional and bidirectional microphones. Figure 1C The process of microphone module beamforming is illustrated using the "+" and "=" symbols. Figure 1C As can be seen, the output signal of the omnidirectional microphone and the signal of the virtual bidirectional microphone are beamformed to form a supercardioid pickup beam. Each pickup pattern shows the pickup directionality at five representative frequencies: 200 Hz, 1000 Hz, 2000 Hz, 4000 Hz, and 7500 Hz, using different curves.

[0051] As seen in the first pickup pattern, the omnidirectional microphone's pickup pattern at all frequencies is nearly circular, indicating its almost uniform sensitivity to sound from all 360° directions. It can be observed that the curves corresponding to the five different frequencies have a high degree of overlap and similarity in shape, and the overall pattern remains stable across all frequencies, characterizing the omnidirectional microphone's acoustic characteristics of non-selectivity towards sound direction and uniform pickup in all directions. This acoustic characteristic of the omnidirectional microphone enables it to provide an omnidirectional sound reference signal in beamforming.

[0052] As can be seen from the second pickup pattern, after weight adjustment, the pickup direction of the superimposed virtual bidirectional signal still forms an "8" shape, which is particularly pronounced at frequencies of 2000 Hz and below, and slightly less pronounced at 4000 Hz and 7500 Hz. Specifically, the sensitivity is highest at 0° (directly in front) and 180° (directly behind), while the sensitivity approaches zero at ±90° (both sides). As mentioned earlier, this virtual bidirectional signal is obtained by multiplying the output signals of the first and second bidirectional microphones by the first and second weights determined according to the target sound direction, respectively, and then superimposing them. Its pickup axis has been adjusted by weights to point towards the target sound direction (corresponding to the 0° direction in the diagram). Because the directivity of the bidirectional microphone originates from its own pressure-difference acoustic structure, rather than relying on the ratio of microphone spacing to sound wavelength, it can maintain an "8" shaped directivity within the effective frequency band from 200 Hz to 7500 Hz. This is significantly different from the directivity degradation of traditional differential microphone arrays in the low-frequency range due to the spacing being much smaller than the wavelength of the sound wave.

[0053] As seen in the third pickup pattern, beamforming the omnidirectional microphone output signal from the first pickup pattern with the virtual bidirectional signal from the second pickup pattern at a preset ratio creates a supercardioid pickup pattern. Compared to the figure-eight pattern in the second pickup pattern, the synthesized supercardioid pattern exhibits a concentrated main lobe with high sensitivity at 0° (target sound direction); sensitivity decreases significantly in the ±90° lateral and 135°-225° rear regions; a small back lobe exists directly behind at 180°, but its sensitivity is much lower than the main lobe directly in front. This indicates that by introducing the omnidirectional microphone signal into beamforming, the back lobe, which is also highly sensitive to the rear in the figure-eight pattern, is effectively suppressed. The pickup directionality changes from bidirectional to a unidirectional supercardioid pattern dominated by the target sound direction, achieving spatial suppression of interfering sounds from the sides and rear, allowing the target speaker's voice to be picked up with a high signal-to-noise ratio.

[0054] Based on the above, the solution of this embodiment provides, on the one hand, a microphone module comprising a bidirectional microphone pair and an omnidirectional microphone. The first and second bidirectional microphones in the bidirectional microphone pair are spatially close to each other and their directional axes are orthogonal, forming an omnidirectional orthogonal directional base. Since the directionality of the bidirectional microphone originates from its own acoustic structure rather than the physical aperture of the microphone array, its directionality has natural frequency consistency within the effective pickup frequency range, eliminating the need for a large physical spacing to maintain low-frequency directionality. This allows the entire microphone module to be deployed in a very compact size within a miniaturized portable device. The addition of the omnidirectional microphone effectively reduces interference outside the target sound direction, enabling the entire microphone module to more effectively pick up sound from the target sound direction. On the other hand, based on the microphone module, by performing sound source localization to determine the target sound direction and dynamically adjusting the microphone module's pickup beam direction to the target sound direction, it can adaptively track the target speaker, achieving focused pickup of the target sound signal while spatially suppressing noise and interference from other directions. This enables portable devices to effectively pick up the voice of the target speaker.

[0055] Although the above process can effectively capture the target speaker's voice signal, in order to achieve high-definition sound pickup, in one alternative method, after step S106, the following step S108 may also be optionally included.

[0056] Step S108: Perform speech enhancement processing on the single-channel target sound signal to obtain an enhanced target sound signal.

[0057] After obtaining the target sound signal through beamforming, selective sound pickup from the target sound direction has been completed in the spatial dimension, effectively suppressing interference sounds from non-target sound directions. However, in real-world sound environments, there may still be environmental noise (such as background white noise, air conditioning noise, and other wide-directional noise) similar to or the same as the source direction of the target speaker's voice, as well as room reverberation effects (the delay superposition phenomenon caused by multiple reflections of sound within a room, leading to temporal ambiguity in the speech signal). These components can be eliminated through further speech enhancement processing to obtain a more effective target sound signal.

[0058] Therefore, optionally, after beamforming the target sound signal, speech enhancement processing can be applied to the target sound signal. As mentioned earlier, the target sound signal at this time is a single-channel target sound signal, and the speech enhancement processing uses this single-channel signal as input. Because beamforming performs preliminary screening of the sound signal in the spatial dimension, while speech enhancement processing can further process the signal in the time-frequency domain, the two complement each other technically. That is, beamforming is good at suppressing interference sounds from specific spatial directions, while speech enhancement processing is good at removing background noise and reverberation that overlap with the target sound signal in space. By using the two in cascade, the target sound signal can be jointly optimized in both spatial and time-frequency dimensions, resulting in a clearer sound pickup result than using either one alone.

[0059] In one alternative approach, speech enhancement processing is implemented using a third deep learning model. This third deep learning model takes a single-channel target audio signal as input, performs feature extraction and mapping, and outputs an enhanced target audio signal after speech enhancement processing. This deep learning-based speech enhancement model, or third deep learning model, can learn the statistical differences between speech signals and noise / reverberation signals from a large amount of real-world audio data. Therefore, it can effectively extract clean speech signals from noisy and reverberant environments during the inference stage. Its performance and effectiveness are superior to traditional statistical model-based speech enhancement, especially in scenarios with complex or non-stationary noise types.

[0060] Optionally, the third deep learning model performs denoising and dereverberation on the single-channel target audio signal based on time-frequency domain feature reconstruction to obtain an enhanced target audio signal. Denoising separates and removes additive noise components from the target audio signal, improving its signal-to-noise ratio. Deeverberation removes the time delay additive components introduced by room reverberation, restoring the temporal clarity of the target audio signal. The denoising and dereverberation tasks are correlated and can be jointly trained. The third deep learning model obtained through joint training can share features between the two tasks, achieving better results than processing them independently. Furthermore, through feature reconstruction, the third deep learning model can learn and reconstruct the speech features of the input target audio signal, restoring sound details masked by noise and reverberation. Besides suppressing noise components, it can maintain high speech readability and naturalness even in noisy environments. For example, the third deep learning model can also be implemented as a lightweight UNET-based model.

[0061] A third deep learning model is used to perform joint speech enhancement processing on the single-channel target sound signal output by beamforming, involving denoising and reverberation feature reconstruction. This further suppresses and removes residual noise and reverberation components in the time-frequency domain, and restores speech details masked by noise and reverberation through feature reconstruction, resulting in an enhanced target sound signal. Speech enhancement complements the spatial domain processing of beamforming; the speech enhancement performs deep cleaning of the signal in the time-frequency domain, ensuring that the final enhanced target sound signal achieves noise suppression in both spatial and time-frequency dimensions. This enables high-definition pickup of the target speaker's voice in complex acoustic environments, further improving the sound pickup quality of portable devices in open or noisy environments.

[0062] In one example, lightweight UNET architecture models can all be implemented as encoder-decoder structures. The encoder extracts features from the input signal through a series of convolutional downsampling layers, while the decoder reconstructs the target data through a series of deconvolutional upsampling layers combined with skip connections. Taking the third deep learning model as an example, the encoder extracts time-frequency domain features from the input signal through a series of convolutional downsampling layers, while the decoder reconstructs a clean speech spectrogram through a series of deconvolutional upsampling layers combined with skip connections.

[0063] The following, combined with Figure 1D The process of the above sound pickup method will be explained using a scenario example.

[0064] Figure 1D The example given is a face-to-face conversation between a salesperson and a customer. However, those skilled in the art should understand that the solutions in this application are also applicable to other conversation scenarios, such as small roundtable meetings, one-on-one conversations, interviews, and other multi-person conversations.

[0065] In this example, the portable device is implemented as a smart card equipped with the aforementioned microphone module. The smart card can be manually triggered, such as by gesture operation (e.g., double-clicking), voice command, or physical switch; it can also be automatically detected for activation, such as by Voice Activity Detection (VAD), which automatically enters the working state after detecting a continuous voice activity signal. This application embodiment does not limit the triggering method of the smart card.

[0066] In this example of a face-to-face conversation, a smart card is worn on the salesperson's chest. The customer is positioned directly in front of the salesperson (approximately at 0°) and conversing face-to-face. In addition to the conversation between the salesperson and the customer, the location (such as a shopping mall or store) also contains conversations from other customers in the surrounding area, background music, and environmental noise from the store's air conditioning or moving shelves. The acoustic environment is quite complex, and the noise sources are distributed from all directions.

[0067] Once the smart card is triggered, the microphone module collects ambient sound signals in real time, including voice signals from salespeople and customers, as well as the various types of interference sounds mentioned above.

[0068] Suppose that between 0:10 and 0:25, a customer is asking a question to a salesperson. The smart card inputs multiple audio signals collected by the microphone module into a first deep learning model. The first deep learning model extracts multi-channel spatial acoustic features from the multiple audio signals, identifies the current main speaker, i.e., the customer, located at approximately 0° (directly in front), and outputs corresponding weight adjustment instructions. If the directional axis of the first bidirectional microphone is aligned with the longitudinal direction of the salesperson's body (i.e., facing forward), the weight adjustment instruction can instruct the first weight to be at its maximum value and the second weight to be at zero (or close to zero), so that the directional axis of the synthesized virtual bidirectional signal is aligned directly forward. Furthermore, this virtual bidirectional signal is beamformed with the output signal of the omnidirectional microphone according to a preset ratio to form a supercardioid pickup beam pointing in the direction of the customer, while effectively suppressing background music and other customer conversations from the sides and rear. Based on this, the single-channel target sound signal obtained by beamforming is processed by a third deep learning model for denoising and dereverberation feature reconstruction to eliminate residual background noise in the shopping mall and output clear customer inquiry voice.

[0069] Now suppose that between 0 minutes 28 seconds and 0 minutes 45 seconds, the salesperson begins introducing the product to the customer. At this time, the microphone module and the speaker (salesperson) are on the same side. The first deep learning model detects that the sound energy from the direction directly in front has switched from being dominated by the customer's voice to being dominated by the salesperson's voice. The first deep learning model updates the sound source localization and weight output processing, switching the current target sound direction to the salesperson's side (approximately 180° directly in front) to pick up the sound signal from the salesperson. After speech enhancement processing, the clear sound signal of the salesperson is output.

[0070] Between 0 minutes 50 seconds and 1 minute 10 seconds, loud sales pitches from nearby salespeople suddenly came from the customer's side (approximately 75°). These pitches were quite loud and significantly interfered with the conversation between the customer and the salespeople. Since the customer was still at approximately 0° and continuing their conversation, the first deep learning model maintained the target sound direction at approximately 0°, and the main axis of the supercardioid pickup beam was stably pointed in the customer's direction, spatially suppressing the sales pitches from the approximately 75° direction. Because this direction is located in the attenuation region of the supercardioid beam, its sensitivity is much lower than that of the main lobe direction, effectively suppressing the sales pitches in the spatial dimension. Building on this, the third deep learning model further removed residual sales pitch interference components from the obtained target sound signal, outputting clear dialogue from the customer. Throughout the process of the sudden interference appearing and being effectively suppressed, the beam direction remained stable without any direction switching, thus preserving the continuity of the conversation.

[0071] Throughout the entire dialogue, since the first and second bidirectional microphones maintain consistent directionality within the effective frequency range of the speech signal, both the low-frequency fundamental tone components in the customer's voice and the high-frequency consonants are subject to consistent directional filtering. There is no problem of low-frequency background noise residue in the mall caused by low-frequency directional degradation, and the sound pickup quality remains stable.

[0072] After obtaining the enhanced target audio signal, subsequent processing can be performed as needed, including but not limited to: speech transcription, real-time translation, and dialogue summarization.

[0073] This example demonstrates how portable devices can effectively pick up sound in real-world application scenarios.

[0074] Refer again Figure 1B The embodiments of this application may also provide a separate microphone module, namely, a microphone module including a pair of bidirectional microphones and an omnidirectional microphone.

[0075] The bidirectional microphone pair includes a first bidirectional microphone and a second bidirectional microphone. The first and second bidirectional microphones in the pair are spatially close to each other and their directional axes are orthogonal to each other.

[0076] An omnidirectional microphone is positioned at or near the intersection of the directional axes.

[0077] Compared to traditional differential microphone arrays that require at least 2-4 cm spacing, the microphone module of this application embodiment can control the overall size to within 1 cm². Traditional differential microphone arrays suffer severe degradation in directivity below 500 Hz due to the physical spacing being much smaller than the sound wave wavelength, resulting in poor low-frequency noise suppression. However, this application embodiment utilizes the inherent pressure difference principle of bidirectional microphones, maintaining consistent directivity across the entire effective speech frequency band from 200 Hz to 8000 Hz. For example, at 250 Hz, the directivity index of the microphone module of this application embodiment can be maintained above 4 dB, while a differential microphone array of the same size is almost 0 dB. This gives the microphone module of this application embodiment a greater advantage in suppressing common low-frequency noises such as those from air conditioners and fans.

[0078] Reference Figure 2 This application also provides a portable device. The portable device includes at least: an audio processing chip 202, a microphone module 204, and a memory 206.

[0079] The microphone module 204 is integrated into the portable device according to the aforementioned acoustic deployment method, including a first bidirectional microphone, a second bidirectional microphone (forming a bidirectional microphone pair), and an omnidirectional microphone. The microphone module 204 is used to acquire sound signals in real time from the physical environment where the portable device is located. The first and second bidirectional microphones are spatially close to each other, and their directional axes are orthogonal to each other; the omnidirectional microphone is located at or near the intersection of the two directional axes. The specific installation positions and spacing of the three microphones can be flexibly set by those skilled in the art according to the specific shape and size constraints of the portable device, and this embodiment does not limit this.

[0080] The audio processing chip 202 is communicatively connected to the microphone module 204 and is used to process the sound signals acquired by the microphone module 204 in real time. The audio processing chip 202 can read corresponding executable instructions from the memory 206 and execute the sound pickup method described in the foregoing method embodiments. This includes locating the sound source based on the sound signals acquired by the microphone module 204 to determine the direction of the target sound, adjusting the pickup beam of the microphone module 204 to the direction of the target sound according to the direction of the target sound, and performing a pickup operation to obtain the target sound signal. For example, in an optional embodiment, the weights of the output signals of the first bidirectional microphone and the second bidirectional microphone can be adjusted according to the direction of the target sound, and then a virtual bidirectional signal can be obtained based on these weights. The pickup beam can then be directed to the direction of the target sound by beamforming the virtual bidirectional signal and the output signal of the omnidirectional microphone. Optionally, speech enhancement processing can also be performed on the target sound signal.

[0081] The audio processing chip 202 can be a dedicated audio digital signal processor, or a general-purpose processor, a system-on-a-chip with integrated audio processing functions, an application-specific integrated circuit, or a field-programmable gate array, etc. The specific implementation of the audio processing chip 202 is not limited in the embodiments of this application.

[0082] The memory 206 is communicatively connected to the audio processing chip 202 and stores at least one executable instruction. This executable instruction enables the audio processing chip 202 to perform the operation corresponding to the aforementioned sound pickup method based on the sound signal collected by the microphone module 204. The memory 206 may include embedded non-volatile memory or the like for persistently storing executable instructions and algorithm parameters. This application embodiment does not limit the specific type and storage capacity of the memory 206.

[0083] The specific implementation of the audio processing chip 202 processing the sound signal collected by the microphone module 204 can be referred to the relevant description in the foregoing method embodiments, and will not be repeated here.

[0084] In this embodiment, the portable device may include, but is not limited to, miniaturized portable devices such as smart cards, portable recording cards, AI glasses, smartwatches, smart earphones, and AI voice recorders. Those skilled in the art can flexibly adjust the installation position and deployment method of the microphone module 204 according to the specific form of the portable device, and select the appropriate specifications for the audio processing chip 202 and memory 206. This embodiment does not limit the specific form of the portable device or the specific specifications of each component.

[0085] The portable device provided by the embodiments of this application can integrate the sound source localization and beamforming signal processing capabilities of the aforementioned sound pickup method into a portable device under extremely compact hardware constraints. This enables effective directional pickup of the target speaker's voice in complex acoustic environments such as open spaces or noisy environments, solving the problem that portable devices have difficulty clearly picking up the target speaker's voice. This provides high-quality voice input for subsequent intelligent applications such as speech recognition, real-time translation, and intelligent summarization.

[0086] This application also provides a computer storage medium storing a computer program that, when executed by a processor, implements the method described in any of the foregoing method embodiments. This computer storage medium includes, but is not limited to, embedded non-volatile memory, embedded static random access memory, etc.

[0087] This application also provides a computer program product, including computer instructions that instruct a computing device to perform an operation corresponding to any of the sound pickup methods in the above-described multiple method embodiments.

[0088] Furthermore, it should be noted that the user-related information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to user voice data, data for analysis, stored data, displayed data, etc.) involved in the embodiments of this application are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0089] It should be noted that, depending on the implementation needs, the various components / steps described in the embodiments of this application can be broken down into more components / steps, or two or more components / steps or parts of the operation of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of this application.

[0090] Those skilled in the art will recognize that the units and method steps of the various examples described in conjunction with the embodiments of this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this application.

[0091] The above embodiments are only used to illustrate the embodiments of this application, and are not intended to limit the embodiments of this application. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the embodiments of this application. Therefore, all equivalent technical solutions also fall within the scope of the embodiments of this application, and the patent protection scope of the embodiments of this application should be defined by the claims.

Claims

1. A sound pickup method, comprising: The microphone module collects sound signals from the environment. The microphone module includes a pair of bidirectional microphones and an omnidirectional microphone. The first and second bidirectional microphones in the pair of bidirectional microphones are close to each other in space and their directional axes are orthogonal to each other. The omnidirectional microphone is located at or near the intersection of the directional axes. Based on the sound signal, sound source localization is performed to determine the direction of the target sound; The microphone module's pickup beam is adjusted to the direction of the target sound to perform a pickup operation in the direction of the target sound and obtain the target sound signal.

2. The method according to claim 1, wherein, The step of adjusting the microphone module's pickup beam direction to the target sound direction includes: Based on the target sound direction, the weights of the output signals of the first bidirectional microphone and the second bidirectional microphone are adjusted so that the pickup beam of the microphone module is directed towards the target sound direction.

3. The method according to claim 2, wherein, The step of adjusting the weights of the output signals of the first and second bidirectional microphones according to the target sound direction, so that the pickup beam of the microphone module is directed towards the target sound direction, includes: Based on the target sound direction, determine the first weight corresponding to the output signal of the first bidirectional microphone and the second weight corresponding to the output signal of the second bidirectional microphone; The first bidirectional microphone output signal is multiplied by the first weight to obtain the first adjustment result; the second bidirectional microphone output signal is multiplied by the second weight to obtain the second adjustment result. Based on the first adjustment result and the second adjustment result, the microphone module's pickup beam is directed towards the target sound direction.

4. The method according to claim 3, wherein, The step of directing the microphone module's pickup beam toward the target sound direction based on the first adjustment result and the second adjustment result includes: By superimposing the first adjustment result and the second adjustment result, a virtual bidirectional signal pointing towards the target sound direction is obtained; The virtual bidirectional signal and the output signal of the omnidirectional microphone are beamformed according to a preset ratio so that the microphone module's pickup beam is directed toward the target sound direction.

5. The method according to claim 4, wherein, The synthesized pickup beam is a cardioid or supercardioid pickup beam pointing in the direction of the target sound.

6. The method according to claim 3, wherein, The first weight is proportional to the cosine of the azimuth angle of the target sound direction, and the second weight is proportional to the sine of the azimuth angle.

7. The method according to any one of claims 1-6, wherein, The microphone module's pickup beam has consistent directionality within its effective pickup frequency range.

8. The method according to any one of claims 2-6, wherein, The step of locating the sound source and determining the direction of the target sound based on the sound signal includes: The sound signal is input into a first deep learning model, which extracts multi-channel spatial acoustic features from the sound signal to determine the direction of the target sound. Based on the direction of the target sound, the model outputs weights for adjusting the output signals of the first and second bidirectional microphones.

9. The method according to claim 8, wherein, The step of extracting multi-channel spatial acoustic features from the sound signal using the first deep learning model includes: The first deep learning model is used to extract multi-channel spatial acoustic features from the sound signal, including cross-channel correlation and phase difference.

10. The method according to any one of claims 1-6, wherein, The step of locating the sound source and determining the direction of the target sound based on the sound signal includes: The sound signal is input into a second deep learning model, which extracts spatial location features from the sound signal and outputs spatial direction information to indicate the direction of the target sound.

11. The method according to any one of claims 1-6, wherein, The target sound signal obtained by performing a sound pickup operation in the direction of the target sound is a single-channel sound signal. After performing a sound pickup operation targeting the direction of the target sound to obtain the target sound signal, the method further includes: The target sound signal in a single channel is subjected to speech enhancement processing to obtain an enhanced target sound signal.

12. The method according to claim 11, wherein, The process of performing speech enhancement processing on the single-channel target audio signal to obtain an enhanced target audio signal includes: The target sound signal in a single channel is enhanced using a third deep learning model to obtain an enhanced target sound signal.

13. The method according to claim 12, wherein, The process of performing speech enhancement processing on the single-channel target sound signal using a third deep learning model to obtain an enhanced target sound signal includes: The target sound signal in a single channel is enhanced by denoising and de-reverberation based on time-frequency domain feature reconstruction using a third deep learning model to obtain an enhanced target sound signal.

14. The method according to any one of claims 1-6, wherein, The first bidirectional microphone is arranged along the lateral direction of the device in which the microphone module is located, and the second bidirectional microphone is arranged along the longitudinal direction of the device.

15. The method according to any one of claims 1-6, wherein, Both the first and second bidirectional microphones are MEMS microphones.

16. A microphone module, comprising: A pair of bidirectional microphones and an omnidirectional microphone, the pair of bidirectional microphones including a first bidirectional microphone and a second bidirectional microphone; The first and second bidirectional microphones in the bidirectional microphone pair are spatially close to each other and their directional axes are orthogonal to each other; and The omnidirectional microphone is located at or near the intersection of the directional axes.

17. A portable device comprising at least: An audio processing chip, a microphone module as described in claim 16, and a memory; The audio processing chip is communicatively connected to the microphone module and the memory. The microphone module is used to collect sound signals from the environment; The memory is used to store at least one executable instruction, which causes the audio processing chip to perform the operation corresponding to the sound pickup method as described in any one of claims 1-15 based on the sound signal collected by the microphone module.

18. A computer storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in any one of claims 1-15.

19. A computer program product comprising computer instructions that instruct a computing device to perform an operation corresponding to any one of the methods described in claims 1-15.

Images