Audio interaction device, data processing method, and computer storage medium

By setting up a hidden interface microphone array in the smart speaker and a speaker layout away from the microphone, combined with the sound-transmitting hole and reflective surface design, the problem of taking into account the appearance and sound quality of the speaker is solved, achieving higher signal-to-noise ratio and aesthetics.

CN109698984BActive Publication Date: 2024-11-05LITTLE BIRD CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN201810608620.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2018-06-13
Publication Date
2024-11-05
Estimated Expiration
2038-06-13

AI Technical Summary

Technical Problem

When designing microphone arrays and speaker layouts, existing smart speakers are difficult to take into account both the appearance and sound quality performance, and the microphone is easily touched by mistake and causes noise problems.

Method used

Design an audio interactive device. The microphone is set at the bottom close to the space surface and adopts a hidden interface layout. The speaker is away from the microphone. The signal processing is optimized through the sound-transmissive hole and reflective surface. The microphone array signal processing algorithm is used to locate and weight the sound source. , improve signal-to-noise ratio and directionality.

Benefits of technology

It improves the appearance aesthetics and sound quality performance of audio interactive devices, avoids noise problems caused by accidental touch of microphones, and improves the signal-to-noise ratio and array performance of microphones.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN109698984B_ABST
    Figure CN109698984B_ABST
Patent Text Reader

Abstract

The embodiment of the present invention discloses an audio interaction device. The device comprises: a shell, a plurality of microphones located in a housing space of the shell, at least one processor, and a memory for storing a computer program that can be run on the processor, wherein the processor is used to execute the following when running the computer program: identifying audio signals obtained by the plurality of microphones and processing the audio signals; wherein the distance between the plurality of microphones and a first surface of the shell of the audio interaction device is less than a first threshold; and the first surface is parallel to the plane where the plurality of microphones are located and is between the plane where the plurality of microphones are located and a placement surface.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of speakers, and in particular to an audio interaction device, a data processing method, and a computer storage medium. Background Art

[0002] With the intelligentization of audio output devices, audio output devices not only have audio output functions, but also have audio input functions, becoming voice interaction devices to facilitate voice interaction with users. More and more voice interaction devices use microphone arrays instead of single microphones to improve the quality of voice input, such as clarity and signal-to-noise ratio.

[0003] However, microphone arrays bring difficulties to appearance design. The arrangement of microphones may conflict with the arrangement of other devices, requiring more compromises, and the appearance will also be affected.

[0004] Taking common smart speakers as an example, common products on the market usually place the microphone array near the upper surface of the product, and have obvious sound-transmitting openings or sound-transmitting meshes on the outer shell, while the product's speakers are placed in the lower half of the product, which restricts both the appearance design and sound quality.

[0005] In conventional designs, in order to make the microphones respond consistently and to minimize the impact of reflections and their own acoustic structures, it is usually required that there is no obstruction between the microphones. The microphone module has a large sound-permeable opening. This usually causes the microphone array to be arranged at the top or the most protruding outside of the device, with a basically flat outer surface and a large sound-permeable hole at the microphone. In interactive devices such as smart speakers, in order to avoid overload and distortion of the microphone signal due to excessive sound, the speakers of smart speakers must be away from the microphone array, so they have to be at the bottom of the speaker, and therefore close to the interface where the smart speaker is placed (such as a desktop or the ground). The speakers placed at the bottom limit the sound playback effect of the smart speaker, and the top has to have a sound-permeable hole, which affects the appearance. In addition, the top or outside of the device is generally the part that users see and touch more often, and the large sound-permeable hole also makes it easy to accidentally touch the microphone during operation, causing noise. Summary of the invention

[0006] In order to solve the existing technical problems, the embodiments of the present invention provide an audio interaction device, a data processing method, and a computer storage medium.

[0007] To achieve the above object, the technical solution of the embodiment of the present invention is implemented as follows:

[0008] An embodiment of the present invention provides an audio interaction device, the device comprising: a housing, a plurality of microphones located in a housing space of the housing, at least one processor, and a memory for storing a computer program that can be run on the processor, wherein the processor is configured to execute, when running the computer program, the following operations: identifying audio signals obtained by the plurality of microphones and processing the audio signals;

[0009] Among them, the distance between the multiple microphones and the first surface of the shell of the audio interaction device is less than a first threshold; the first surface is parallel to the plane where the multiple microphones are located and between the plane where the multiple microphones are located and the placement surface.

[0010] In the above solution, the shell is provided with a first sound-permeable hole corresponding to each of the multiple microphones; wherein the multiple first sound-permeable holes are located at the connection between the first surface and the side surface of the audio interaction device.

[0011] In the above solution, the shell provided with the first sound-transmitting hole forms a receiving portion having at least one reflecting surface, and the microphone is located in the receiving portion.

[0012] In the above solution, each of the multiple microphones corresponds to a accommodating portion; and the multiple accommodating portions have the same structure.

[0013] In the above solution, the first sound-permeable hole forms a centrally symmetrical opening shape on the shell.

[0014] In the above solution, the number of the multiple microphones is associated with the attribute parameters of the audio signal to be received and the audio interaction device.

[0015] In the above solution, the angles formed between any two adjacent microphones among the multiple microphones and the center point of the audio interaction device are equal.

[0016] In the above solution, the device also includes at least one speaker; the distance between the at least one speaker and the plane where the multiple microphones are located is greater than a second threshold.

[0017] In the above solution, the shell is provided with second sound-permeable holes respectively corresponding to each speaker of the at least one speaker; at least one second sound-permeable hole is located on a second surface of the shell away from the first surface.

[0018] In the above solution, the memory stores an application program including a microphone array signal processing algorithm;

[0019] When the processor is used to run the application program including the microphone array signal processing algorithm, the processor executes: determining the first sound source position by delay estimation and / or amplitude estimation for a microphone pair formed by any two microphones among the multiple microphones; performing weighted processing on the determined multiple first sound source positions to obtain the sound source position;

[0020] The step of performing weighted processing on the determined multiple first sound source positions to obtain the sound source position comprises: determining a weight value corresponding to the first sound source position of the microphone pair based on at least one of the following information, and performing weighted processing based on the weight value and the corresponding first sound source position to obtain the sound source position;

[0021] The information includes: an amplitude relationship of audio signals received by two microphones in a microphone pair;

[0022] the energy of the audio signal received by any microphone in the microphone pair;

[0023] The spacing between the two microphones in a microphone pair;

[0024] The attribute parameters of the audio signal received by any microphone in the microphone pair include at least one of the following: frequency, period, and wavelength.

[0025] The embodiment of the present invention further provides a data processing method, which is applied to the audio interaction device described in the embodiment of the present invention; the method comprises:

[0026] Acquire audio signals through multiple microphones;

[0027] Determine the first sound source position by using a delay estimation and / or amplitude estimation method for a microphone pair formed by any two microphones among the plurality of microphones;

[0028] The determined multiple first sound source positions are weighted to obtain the sound source position.

[0029] In the above solution, the step of performing weighted processing on the determined multiple first sound source positions to obtain the sound source position includes:

[0030] Determine a weight value corresponding to a first sound source position of the microphone pair based on at least one of the following information, and perform weighted processing based on the weight value and the corresponding first sound source position to obtain the sound source position;

[0031] The information includes: an amplitude relationship of audio signals received by two microphones in a microphone pair;

[0032] the energy of the audio signal received by any microphone in the microphone pair;

[0033] The spacing between the two microphones in a microphone pair;

[0034] The attribute parameters of the audio signal received by any microphone in the microphone pair include at least one of the following: frequency, period, and wavelength.

[0035] The embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the data processing method described in the embodiment of the present invention are implemented.

[0036] The audio interaction device and data processing method and computer storage medium provided by the embodiment of the present invention include: a shell, a plurality of microphones located in the accommodating space of the shell, at least one processor and a memory for storing a computer program that can be run on the processor, wherein the processor is used to execute the following when running the computer program: identifying the audio signals obtained by the plurality of microphones and processing the audio signals; wherein the distance between the plurality of microphones and the first surface of the shell of the audio interaction device is less than a first threshold; the first surface is parallel to the plane where the plurality of microphones are located and between the plane where the plurality of microphones are located and the placement surface. According to the technical solution of the embodiment of the present invention, the microphone is arranged at the bottom of the audio interaction device close to the placement surface, and a hidden interface microphone array is adopted to improve the freedom and aesthetics of the interactive device design; the aesthetics of the overall appearance of the audio interaction device is improved, and the noise caused by accidentally touching the microphone during operation is avoided; on the other hand, the signal-to-noise ratio and directivity of the microphone are improved without increasing the cost, and better array performance is obtained. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 A schematic diagram of the structure of an audio interaction device according to an embodiment of the present invention;

[0038] Figure 2 A schematic diagram of an audio interaction device according to an embodiment of the present invention at an upward viewing angle;

[0039] Figure 3 A partial cross-sectional schematic diagram of the location of a microphone of an audio interaction device according to an embodiment of the present invention;

[0040] Figure 4a A schematic diagram of an audio transmission path of an existing audio interaction device;

[0041] Figure 4b A schematic diagram of an audio transmission path of an audio interaction device according to an embodiment of the present invention;

[0042] Figure 5 A schematic diagram of an audio interaction device according to an embodiment of the present invention using a delay estimation method to determine a sound source position;

[0043] Figure 6 A schematic diagram of the sensitivity of a microphone facing toward a sound source and a microphone facing away from the sound source of an audio interaction device according to an embodiment of the present invention;

[0044] Figure 7 The figure is a schematic diagram of the sensitivity of the microphone of the audio interaction device in various directions according to an embodiment of the present invention. DETAILED DESCRIPTION

[0045] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0046] An embodiment of the present invention provides an audio interaction device. Figure 1 A schematic diagram of the structure of an audio interaction device according to an embodiment of the present invention; Figure 2 is a schematic diagram of an audio interaction device according to an embodiment of the present invention at an upward viewing angle; Figure 1 and Figure 2 As shown, the device includes: a housing, a plurality of microphones located in a housing space of the housing, at least one processor, and a memory for storing a computer program that can be run on the processor, wherein the processor is used to execute, when running the computer program: identifying audio signals obtained by the plurality of microphones and processing the audio signals;

[0047] Among them, the distance between the multiple microphones and the first surface of the shell of the audio interaction device is less than a first threshold; the first surface is parallel to the plane where the multiple microphones are located and between the plane where the multiple microphones are located and the placement surface.

[0048] In this embodiment, the audio interaction device has an audio input function. In actual applications, the audio interaction device can be a terminal device such as a smart speaker, a loudspeaker, a telephone, a mobile phone, an interface microphone, etc. Among them, the audio interaction device has at least one plane, and the at least one plane includes a first surface. As an implementation method, when the audio interaction device is placed on a storage surface, the first surface is in contact with or close to the storage surface; the storage surface is a plane on which the audio interaction device is placed, and the storage surface can be a plane such as the ground or a desktop. The storage surface can also be a vertical wall or a roof wall. No matter how the audio interaction device is placed on the storage surface, the first surface is a plane that is in contact with the storage surface, or a plane close to the storage surface (that is, the first plane in the audio interaction device is the closest to the storage surface).

[0049] As another implementation manner, the first surface may also be an interface of the interface microphone itself when the microphone is an interface microphone type, such as an interface formed by an interface microphone holder.

[0050] In this embodiment, the plane where the multiple microphones are located is parallel to the first surface, or considering that there may be a certain error in the microphone setting process, the plane where the multiple microphones are located is approximately parallel to the first surface. In addition, the first surface is between the plane where the multiple microphones are located and the placement surface, and when the distance between the multiple microphones and the first surface is less than the first threshold, it can be understood that the multiple microphones are set at the bottom of the audio interaction device.

[0051] Taking the first surface as a surface that is in contact with or close to the placement surface as an example, the audio interaction device is in contact with or close to the placement surface through the first surface. Since the distance between the multiple microphones and the first surface is less than the first threshold, the multiple microphones are close to the placement surface. Among them, the placement surface can also be called a first interface. The path for the sound source to reach the microphone may include: a first path for the audio signal to be emitted by the sound source and directly reach the microphone, and this audio signal can be called a direct audio signal; and a second path for the audio signal to reach the first interface, be reflected by the first interface and reach the microphone, and this audio signal can be called a reflected audio signal. When the first interface is close to the microphone, since the first interface is close to the microphone, the reflected audio signal of the first interface and the direct audio signal reach the microphone almost at the same time, so that the audio signal received by the microphone is enhanced, that is, the acoustic reflection effect of the first interface can improve the signal-to-noise ratio and sensitivity of the microphone in a wider frequency band.

[0052] It can be understood that when a user speaks, the voice audio emitted by the user reaches the microphone via multiple paths and is picked up by the microphone. These paths include the shortest path and the reflection path. If the distance between the interface and the microphone is very close, the distance is much smaller than the wavelength of the sound wave of the voice audio, and the length of the shortest path and the reflection path are close, the voice audio reaching the microphone via the two paths is completely correlated and superimposed in almost the same phase, the amplitude is increased to two times, and the energy is increased to four times, which enhances the sensitivity by 6 dB (10log(4)).

[0053] The interface also has an enhancement effect on environmental noise, but because environmental noise is isotropic random noise, it does not increase the sensitivity by 6dB like voice audio, but only by 3dB (10log(2)). Such an interface increases the sensitivity of voice by 6dB and noise by 3dB, so the overall signal-to-noise ratio increases by 3dB (10log(2)).

[0054] Similar principle and multi-interface effect can further improve the signal-to-noise ratio. Two interfaces can improve it by nearly 5dB (10log(3)).

[0055] In addition to the placement surface, a second interface or more interfaces can be provided around the microphone through reasonable appearance design. As an embodiment, the shell of the audio interaction device forms a housing portion having at least one reflective surface, and the microphone is located in the housing portion. Among them, the at least one reflective surface of the housing portion for accommodating the microphone can be called a second interface. Similar to the first interface, since the distance between the second interface and the microphone is relatively close, the reflected audio signal of the second interface and the direct audio signal reach the microphone almost at the same time, thereby enhancing the audio signal received by the microphone. In other application scenarios, the audio interaction device can also be placed with the first surface close to the wall. When the distance between the wall and the microphone is relatively close, the wall can also be used as an interface to enhance the audio signal received by the microphone.

[0056] Therefore, when the interface has the same impact on multiple microphones of the device (for example, the interface serves as the placement surface, the structure of the microphones is consistent, the angle formed by the microphones and the interface is the same, etc.), the improvement in microphone sensitivity is positively correlated with the number of interfaces. For example, under the premise that the distance between the interface and the microphone is much smaller than the wavelength of the audio signal to be collected, one interface can improve the signal-to-noise ratio of the audio signal relative to the ambient background noise by 3dB, and two interfaces can improve it by 5dB, etc.

[0057] In this embodiment, the audio interactive device has a shell, which can be a centrally symmetrical shell or an asymmetrical shell; when the shell is a shell with a centrally symmetrical relationship. The first surface of the shell can be a shape with a centrally symmetrical relationship, such as a circle, a regular polygon, etc. The side of the audio interactive device and the first surface can be perpendicular to each other, or the inner wall of the side of the audio interactive device and the first surface have an acute angle or an obtuse angle, such as Figure 2 As shown, the inner wall of the side of the audio interaction device forms an obtuse angle with the ground.

[0058] In this embodiment, the audio interaction device is provided with a microphone array formed by a plurality of microphones, and the plurality of microphones are used to collect audio signals. The plurality of microphones are arranged at the bottom of the audio interaction device. It can be understood that the plurality of microphones are close to the first surface of the audio interaction device, that is, the distance between the plurality of microphones and the first surface of the shell of the audio interaction device is less than a first threshold. Among them, the distance between the plurality of microphones and the first surface of the shell of the audio interaction device can be zero, that is, the plurality of microphones are arranged at the connection between the first surface of the audio interaction device and the side surface of the audio interaction device, which can be specifically as follows Figure 2As shown. As an embodiment, the housing is provided with a first sound-permeable hole corresponding to each of the plurality of microphones; as an embodiment, the plurality of first sound-permeable holes may be located on a side surface of the audio interaction device; as another embodiment, the plurality of first sound-permeable holes are located at the connection between the first surface and the side surface of the audio interaction device. The microphone receives the audio signal through the corresponding first sound-permeable hole.

[0059] Based on the above embodiment, in another embodiment, the audio interaction device may also have an audio output function, that is, the device may also include at least one speaker; the distance between the at least one speaker and the plane where the multiple microphones are located is greater than the second threshold value, and it can be understood that the at least one speaker is far away from the first surface of the shell. Then the shell is also provided with a second sound-permeable hole corresponding to each speaker of the at least one speaker; at least one second sound-permeable hole is located on the second surface of the shell away from the first surface, that is, at least one second sound-permeable hole can be located on the second surface of the shell, which can be understood as the top surface relative to the bottom surface. The speaker outputs an audio signal through the corresponding second sound-permeable hole. Among them, the shell is provided with a second sound-permeable hole corresponding to the at least one speaker; at least one second sound-permeable hole is located on the second surface of the shell away from the first surface. For example, when the first surface is the bottom surface, the second surface can be the top surface. Alternatively, the second surface can also be a partial area in the side surface away from the first surface.

[0060] In actual applications, the distance between the microphone and the speaker is much smaller than the distance between the microphone and the user. In the audio signal received by the microphone, the audio signal component emitted by the speaker is much larger than the audio signal component of the user, thereby drowning the audio signal of the user. Although conventional echo cancellation algorithms can eliminate most of the audio signal components of the speaker, the performance of the echo cancellation algorithm has physical limits. When the speaker quality is very good and the upper limit of the measurable sound pressure level of the microphone is higher than the sound pressure level of the speaker signal at the microphone, the speaker sound component can be reduced by about 30dB, and in most cases it can only be reduced by 20dB to 25dB. In order to better restore the user's audio signal, the proportion of the speaker's audio signal component to the microphone's received signal should be as small as possible, that is, the speaker's audio signal should be as weak as possible when it reaches the microphone. Based on this, the distance between the plane where the multiple microphones are located and the speaker is greater than the second threshold, which can also be said that the distance between the microphone and the speaker should be as large as possible. In one embodiment, the microphone and the speaker are set at both ends of the long axis of the device. And the measured value of the audio signal emitted by the speaker at the microphone is lower than the upper limit of the measurable sound pressure level of the microphone.

[0061] In practical applications, as an implementation method, the distance between the microphone and the speaker is maximized within the size range of the audio interaction device, that is, the microphone is arranged on the first surface of the audio interaction device, and the speaker is arranged on the second surface of the audio interaction device, that is, the distance between the at least one speaker and the plane where the multiple microphones are located is the height of the audio interaction device.

[0062] As another embodiment, the layout of the speaker and the microphone can also be adapted to the internal layout design of the audio interaction device, and the second threshold is related to the maximum volume of the speaker, the upper limit of the sound pressure level measurement of the multiple microphones, and the size of the audio interaction device itself. For example, when the speaker plays at maximum volume, the audio signal received at the microphone is lower than the upper limit of the sound pressure level measurement of the microphone. For example, if the speaker plays at maximum volume, the sound pressure level is 110 decibels at a distance of 10 cm and 104 decibels at a distance of 20 cm. If the upper limit of the sound pressure level measurement of a certain type of microphone used in the device is 104 decibels, when this type of microphone is used, the distance between the microphone and the speaker should not be less than 20 cm to be used normally. If the product size is limited and the distance between the microphone and the speaker is 10 cm, other types of microphones with a measurement upper limit of not less than 110 decibels must be used.

[0063] Based on this, in an embodiment of the present invention, when the distance between the microphone and the speaker reaches the maximum within the size range of the audio interaction device and the audio signal of the maximum volume emitted by the speaker received by the microphone is lower than the upper limit of the sound pressure level measurement of the microphone (that is, when the upper limit of the sound pressure level measurement of the microphone can meet the maximum volume of the speaker), the first distance can be determined based on the maximum volume of the speaker and the upper limit of the sound pressure level measurement of the microphone. The first distance is the minimum distance allowed between the speaker and the microphone when the speaker is in normal use. Then the second threshold is greater than or equal to the first distance. Correspondingly, the distance between the multiple microphones and the first surface of the shell of the audio interaction device is less than the first threshold. The first threshold can be determined based on the size of the audio interaction device (specifically the height of the device) and the second threshold.

[0064] It can be understood that, on the basis that the size of the audio interaction device (specifically, the height of the device) is greater than the second threshold, the layout of the multiple microphones and speakers can be adapted to the internal layout design on the basis that the distance between the multiple microphones and speakers is greater than the second threshold. For example, the multiple microphones can be located near the first surface of the audio interaction device, or even on the first surface; accordingly, the first sound-permeable holes corresponding to the multiple microphones can be located on the first surface, on the side surface close to the first surface, or even at the connection between the first surface and the side surface, such as Figure 2In the scenario where the first sound-permeable hole is located on the side surface close to the first surface, the side inner wall of the housing of the audio interaction device and the first surface form the following Figure 2 The obtuse angle shown, no matter how the audio interaction device is placed, the first sound-permeable hole is facing away from the user's line of sight. Compared with setting the first sound-permeable hole at the connection between the first surface and the side surface of the shell, both solutions do not affect the appearance of the device. It can be understood that, as a first embodiment, multiple first sound-permeable holes can be set at the connection between the first surface and the side surface of the audio interaction device; as a second embodiment, when the side inner wall of the shell of the audio interaction device forms an obtuse angle greater than a threshold with the first surface, multiple first sound-permeable holes can be set on the side surface of the shell of the audio interaction device. In another embodiment, the first surface of the audio interaction device can be provided with at least three supporting members, and the audio interaction device is placed on the placement surface by means of the at least three supporting members. In this application scenario, the first sound-permeable hole can also be set on the first surface, and this implementation scheme also does not affect the appearance of the device.

[0065] In this embodiment, the first sound-transmitting holes form a centrally symmetrical opening shape on the shell, and the opening shapes formed by the plurality of first sound-transmitting holes on the shell are the same. Specifically, the opening shape formed by the first sound-transmitting holes on the shell is, for example, at least one of the following: a centrally symmetrical opening shape such as a slit, a circular opening, a regular polygonal opening, etc.

[0066] In practical applications, as an implementation mode, the layout positions of multiple microphones are close to the first surface of the shell of the audio interaction device, or close to the side surface of the shell; in another embodiment, the shell provided with a first sound-permeable hole forms a accommodating portion having at least one reflective surface, and the microphone is located in the accommodating portion. Figure 3 FIG. 1 is a partial cross-sectional schematic diagram of the position of the microphone of the audio interaction device according to an embodiment of the present invention. Figure 3 As shown, it can be understood that, taking the layout position of the microphone close to the first surface of the shell of the audio interaction device as an example, the microphone is at a certain distance from the first surface or from the connection between the first surface and the side surface; the shell of the audio interaction device forms a groove or chamfer to form a accommodating portion with at least one reflective surface, and the microphone is located in the accommodating portion. Since the accommodating portion has at least one reflective surface, the reflective surface can be called the aforementioned second interface, thereby improving the signal-to-noise ratio of the microphone, such as the signal-to-noise ratio of the microphone in the mid- and high-frequency range can be improved by about 3dB to 5dB.

[0067] In this embodiment, each microphone among the plurality of microphones corresponds to a receiving portion; the plurality of receiving portions have the same structure, that is, each microphone corresponds to the same receiving portion structure.

[0068] In this embodiment, as an implementation method, the angles formed between any two adjacent microphones of the multiple microphones and the center point of the audio interaction device are equal, that is, the microphone array formed by the multiple microphones is evenly arranged, which is convenient for all-round (i.e. 360°) sound collection, avoiding the situation where multiple microphones are concentrated on one side. If the sound source is facing away from the side where the microphones are concentrated, the audio signal emitted from the sound source must bypass the audio interaction device to reach the microphone due to the occlusion of the audio interaction device body itself. This diffraction transmission method will cause certain losses to the high-frequency signal in the audio signal, and lacks direct audio signals, which is not conducive to the positioning processing of the sound source and the enhancement processing of the audio signal in the specified direction. It can be understood that the multiple microphones are evenly distributed on the edge of the cross section of the audio interaction device. Taking the number of multiple microphones as six as an example, they are placed at the bottom of the audio interaction device and arranged at equal intervals. The line between the center of the circle of the plane where the six microphones are located and each microphone makes the angle formed between any two microphones and the center of the circle be 60°.

[0069] As another implementation, the microphone array formed by multiple microphones may also be arranged unevenly, that is, the irregularly arranged microphone array is adapted to the shape of the housing of the audio interactive device and / or the layout structure inside the device. For example, if there are many screws or wires in the device, the microphone array cannot be evenly arranged.

[0070] In the present embodiment, the types of all microphones and the directivity of microphone array elements (microphone array elements represent microphones and structures around microphones) are known. This is because the audio signals received by the microphones need to subsequently perform sound source localization and signal enhancement processing in a specified direction, which requires that the receiving effect of each microphone be known, and the properties and parameters of each microphone be known, such as sensitivity and frequency response indicators, the reflection enhancement effect produced by each microphone housing, and the structure of each microphone combined with the housing has known directivity and sensitivity.

[0071] In this embodiment, the number of the multiple microphones is associated with the attribute parameters of the audio signal to be received and the product features of the audio interaction device. In one example, due to the constraints of product cost, the number of microphones used in the microphone array is relatively small, and a smaller number of microphones corresponds to fewer analog-to-digital conversion chips, thereby having a lower computing load. In another example, a larger number of microphones can be used. As the number of microphones increases, the directivity of the microphone array improves, and the processing effect will also improve. However, after the number of microphones increases to a certain number, the improvement in effect will not be obvious. There are two main reasons: 1. For audio processing, since the main energy of audio is distributed within [0,4000] Hz, and the commonly used audio transmission frequency band does not exceed [0,8000] Hz, if the microphones are too dense so that the minimum spacing between microphones is less than 2 cm (1 / 4 wavelength of 4kHz sound waves, 1 / 2 wavelength of 8kHz sound waves), then increasing the distribution density and number of microphones will not significantly improve the array directivity (this is the common 1 / 2 wavelength spacing criterion in arrays); on the other hand, the array directivity of the microphone does not need to be very sharp, because the speaker's vocalization site is not a single point, but occupies a certain angle range in space. The array response should be flat within this angle range. Too sharp directivity will instead lose some audio.

[0072] Based on this, in an embodiment of the present invention, the distance between any two microphones among the multiple microphones is determined based on the attribute parameters of the audio signal to be received, and the number of the multiple microphones is determined based on the distance between the any two microphones and the characteristics of the audio interaction device (the characteristics may specifically be the device cost constraint and the size of the device). The distance between any two microphones satisfies 1 / 2 wavelength of the audio signal to be received, and the distance between any two microphones is greater than or equal to 2 cm.

[0073] In this embodiment, the memory stores an application program including a microphone array signal processing algorithm; when the processor executes the application program including the microphone array signal processing algorithm, the processing of sound source localization and signal enhancement of the sound source based on the audio signals received by the multiple microphones is implemented. The processing of sound source localization includes the processing of sound source orientation and the processing of determining the distance from the sound source, that is, the sound source localization is related to the sound source orientation and the distance from the sound source.

[0074] Normally, the direction of the sound source is determined based on the time delay relationship or amplitude relationship between the audio signal reaching each microphone in the microphone array, and the sound source orientation result is obtained, and the signal of the sound source is enhanced according to the sound source orientation result. Among them, the method of determining the sound source position based on the time delay relationship can be called the time delay estimation method, and the method of determining the sound source position based on the amplitude relationship can be called the amplitude estimation method. Among them, under the premise that the wavelength is less than twice the microphone spacing (that is, the spacing between two adjacent microphones), the time delay relationship can be calculated from the phase relationship of the audio signal.

[0075] On the other hand, when the audio signal is radiated from the sound source to a single microphone, the audio signal received at the microphone will experience amplitude attenuation and transmission delay. The audio received by each microphone in the microphone array will experience corresponding transmission delay and amplitude attenuation. The sound source position can also be inferred from the amplitude relationship or transmission delay relationship. Since each microphone in the microphone array has spatial directivity, the signal in the direction of the sound source can be enhanced, while other audio signals except the direction of the sound source can be attenuated.

[0076] In actual use, the distance between the sound source and each microphone is often much larger than the aperture of the microphone array, and the amplitude difference is small, so the time delay relationship is often used to determine the direction of the sound source. Among them, there is more than one path for the sound source to reach the microphone, including the shortest path (usually a direct path) and many longer reflection paths. The audio signal received by the microphone is usually composed of a direct audio signal and a reflected audio signal. The transmission delay also includes the shortest delay and the reflection delay. The shortest delay is usually the direct delay corresponding to the direct path, and the reflection delay is the delay corresponding to the reflection path; the relationship between the shortest delay and the sound source position is relatively simple and unique, and the relationship between the reflection delay and the sound source position is complex and not unique. If there are many reflective surfaces and the reflected sound is strong, errors will occur in the delay calculation, which will affect the positioning accuracy.

[0077] In order to determine the location of the sound source with the shortest delay, the layout of the microphone array in common product design will also try to increase the proportion of direct audio signals. Therefore, common microphone arrays are arranged on the top of the audio interaction device. There is no obstruction between the microphones. The audio signal is mainly direct audio signals, and the calculation of direct delay is more accurate. Figure 4a shown.

[0078] However, in the embodiment of the present invention, the microphone array is arranged near the first surface. The side facing the sound source has a stronger direct audio signal, while the side facing away from the sound source has no direct audio signal transmission path. The path with the shortest transmission delay is diffracted from the device surface. Figure 4bAs shown in the figure, the loss of high-frequency audio signals in diffraction is very large. The attenuation of reflected audio signals is weak, so the total energy of the audio signals received by the microphone on the back side is reduced, especially in the high-frequency part. The reflected audio signals have similar or even stronger energy than the audio signals corresponding to the shortest delay path. There will be a large error in calculating the delay and positioning based on the delay. Moreover, the diffraction attenuation is related to the length / radian of the diffraction path and the sound energy absorption characteristics of the product's outer surface.

[0079] Based on this, in an embodiment of the present invention, when the processor is used to run the application program including the microphone array signal processing algorithm, the processor executes: determining the first sound source position by means of delay estimation and / or amplitude estimation for a microphone pair formed by any two microphones among the multiple microphones; performing weighted processing on the multiple determined first sound source positions to obtain the sound source position; wherein, performing weighted processing on the multiple determined first sound source positions to obtain the sound source position comprises: determining a weight value corresponding to the first sound source position of the microphone pair based on at least one of the following information, performing weighted processing based on the weight value and the corresponding first sound source position to obtain the sound source position; the information comprises: the amplitude relationship of the audio signals received by the two microphones in the microphone pair; the energy of the audio signal received by any microphone in the microphone pair; the distance between the two microphones in the microphone pair; the attribute parameters of the audio signal received by any microphone in the microphone pair, the attribute parameters comprising at least one of the following: frequency, period, wavelength.

[0080] Among them, the method of determining the position of the first sound source by using time delay estimation includes: obtaining a first audio signal received by a first microphone, and obtaining a second audio signal received by a second microphone; determining a receiving delay based on the first audio signal and the second audio signal; determining a distance difference between the sound source and the first microphone and the second microphone based on the receiving delay; and determining the position of the first sound source based on the distance difference and the distance between the first microphone and the second microphone.

[0081] Specifically, refer to Figure 5 As shown in the figure, the speed of audio signal propagation in the air is a constant c. When sound s is transmitted from the sound source to microphone A at a distance LA from the sound source, the audio signal received by microphone A can be expressed as HA.s(t-LA / c); when sound s is transmitted from the sound source to microphone B at a distance LB from the sound source, the signal received by microphone B can be expressed as HB.s(t-LB / c); where HA and HB represent the energy attenuation of transmission respectively. If there is background noise in the environment, the microphone signal can be expressed as HA.s(t-LA / c)+nA(t) and HB.s(t-LB / c)+nB(t); nA and nB are independent and identically distributed random noise signals.

[0082] The relative receiving delay between the audio signals received by microphone A and microphone B is LA / c–LB / c. If LA / c-LB / c can be calculated, when the propagation speed c of the audio signal in the air is a constant, the distance difference (LA-LB) between the sound source and the first microphone and the second microphone can be determined, and this distance is less than or equal to the spacing L between microphone A and microphone B; and (LA-LB) / L represents the cosine function value of the angle between the sound source and the line connecting microphone A and microphone B. Then, based on the cosine function value, the spacing L and the distance difference (LA-LB), the direction of the sound source and the angle between the line connecting microphone A and microphone B can be determined. The array composed of two microphones can determine the direction of the sound source in the half plane of 0 to 180 degrees. When the number of microphones increases to three or more and the microphones are not arranged in a colinear manner, the microphone array can use the time delay method to determine the accurate direction of the sound source in the full plane; multiple microphone pairs can be formed in the microphone array, and the sound source directions calculated by the multiple microphone pairs are weighted combined to obtain the final sound source direction.

[0083] The calculation of the receiving delay can generally be done by using the cross-correlation method, the phase method, etc. Under the condition that the noise is not greater than the audio signal and the period of the audio signal is greater than twice the relative receiving delay between any two microphones, the conventional cross-correlation method, the cross-power spectrum phase method, etc. can be used to calculate a relatively accurate receiving delay.

[0084] When the period of the audio signal is less than twice the receiving delay between any two microphones (that is, the wavelength of the audio signal is less than twice the product of the distance between the microphones and the cosine of the angle between the microphone connection and the direction of the sound source), there will be multiple numerical solutions when calculating the delay from the phase of the cross-power spectrum, and the relative delay may have a large deviation and cannot be used for direction finding. For multiple microphone pairs in the microphone array, if the distance between some microphone pairs is large, greater than twice the wavelength, the relative delay can only be guaranteed to be less than half the period within a limited range of the incident direction of the audio signal. Beyond this range, errors will occur in the relative delay calculation and angle calculation, resulting in invalid values. If there is no effective way to eliminate invalid directions, these invalid directions will be mixed into the final result, causing errors.

[0085] When the microphone is unidirectional and points at different angles, the amplitude information can be used for directionality, which helps to eliminate these invalid directions.

[0086] Assume that the sensitivity of the microphone in all directions theta at a certain frequency f can be expressed by d(theta–thetak,f). d(alpha,f) means that in the direction with an angle alpha to the direction of the microphone, when alpha=0, the sensitivity is maximum. Function d can also be called the directivity function. If microphone A and microphone B are not pointing in the same direction but at an angle beta, and the angles between the incident direction of the sound source signal and the directions of the two microphones are betaA and betaB respectively, then the directivity functions of microphone A and microphone B are d_A and d_B respectively; when the audio signal reaches the two microphones, the ratio of transmission attenuation HA and HB conforms to the formula HA / HB=d_A(betaA) / d_B(betaB). When the value of the directivity function d(alpha,f) changes significantly with the angle alpha, the direction of the audio signal relative to microphone A and microphone B can be obtained through the amplitude information. When the wavelength of the audio signal is shorter and the frequency is higher, the directivity of the microphone is more obvious, and the change of d(alpha,f) with direction is more obvious.

[0087] Take a device with a hidden interface microphone with six microphones as an example. The shape of the device is close to a cylinder with a diameter of about 8cm. The microphones are arranged on the bottom of the product, close to the surface where the object is placed. The structural design of each microphone is the same. Microphones ABCDEF are arranged at equal intervals in a counterclockwise direction. Due to the shielding effect of the cylindrical shell, each microphone has obvious directivity, and because the structure of each microphone is the same, the directivity function of each microphone is also the same, and the direction is the line connecting the center of the circle to the microphone.

[0088] In an embodiment of the present invention, the sound source position can be calculated using the amplitude relationship of the audio signals received by the microphones and the relative receiving delay. Taking the example of an audio interaction device having six microphones, the six microphones can have 15 different microphone pairs, each microphone pair can calculate the receiving delay based on the audio signals received by the two microphones, and determine the first sound source position based on the receiving delay; further weighted processing is performed based on the first sound source position determined by each microphone pair. Among them, the weight value is related to at least one of the following information: the amplitude relationship of the audio signals received by the two microphones in the microphone pair; the energy of the audio signal received by any microphone in the microphone pair; the distance between the two microphones in the microphone pair; the attribute parameters of the audio signal received by any microphone in the microphone pair, and the attribute parameters include at least one of the following: frequency, period, wavelength.

[0089] In practical applications, the weight values ​​of N microphone pairs can be pre-set to 1 / N, where N is a positive integer greater than 1; 1 / N is further adjusted based on at least one of the aforementioned information, and after the adjustment is completed, the N weight values ​​are normalized so that the sum of the weight values ​​of the N microphones is 1.

[0090] In one embodiment, if the distance between two microphones in the microphone pair is greater than half the wavelength of the audio signal, the distance between the two microphones in the microphone pair is inversely correlated with the corresponding weight value, that is, the greater the distance between the two microphones in the microphone pair, the smaller the corresponding weight value.

[0091] In one embodiment, when the incident direction of the audio signal can be roughly determined within an angle range, for each microphone pair, the acoustic path difference within this angle range is calculated. When the incident direction of the audio signal is within the region corresponding to the angle range, the distance between the two microphones in the microphone pair is multiplied by the cosine of the approximate direction of the audio signal determined within the region and the direction of the line connecting the microphone pair. The product represents the acoustic path difference, that is, the difference in the distance traveled by the sound source of the audio signal to reach the two microphones in the microphone pair. It can be understood that the acoustic path difference is determined based on the distance between the two microphones of the microphone pair; and the corresponding weight value is adjusted according to the comparison result of the acoustic path difference and the wavelength.

[0092] As an example, if the sound path difference exceeds 1 / 2 wavelength of the audio signal, the weight of the corresponding microphone pair is reduced to 0.

[0093] As another example, the sound path difference is compared with 3 / 8 wavelength of the audio signal; if the sound path difference exceeds 3 / 8 wavelength of the audio signal, the weight value of the corresponding microphone pair is reduced to 1 / 2 of the initial weight value 1 / N.

[0094] As another example, when the incident direction of the sound source has no or difficult to have a clear range, if the distance between two microphones in a microphone pair exceeds 1 / 2 wavelength of the audio signal, the weight value of the corresponding microphone pair is reduced to 0.

[0095] In one embodiment, if the energy of the audio signal received by a microphone is lower than the energy of the audio signals received by other microphones, the weight value of the microphone pair to which the microphone belongs is smaller than the weight values ​​of other microphone pairs.

[0096] As an example, the energy of the audio signals received by the microphones is checked and sorted by size; the maximum energy value is determined; if the energy of the audio signal received by a certain microphone is 6dB or more lower than the maximum energy value, the weight of the microphone pair is reduced to 1 / 2 of the initial weight value 1 / N.

[0097] In one embodiment, if the frequency of audio signals received by all microphones among a plurality of microphones is lower than a first preset threshold, so that the distance between a microphone pair formed by any two microphones among the plurality of microphones is lower than half the wavelength of the audio signal, and the energy difference between audio signals received by two microphones in the microphone pair with the largest distance is less than a first value, the weight values ​​of all microphone pairs are equal.

[0098] In one embodiment, if the frequency of audio signals received by all microphones among a plurality of microphones is greater than a first preset threshold and less than a second preset threshold, so that the spacing between a microphone pair formed by any two microphones among the plurality of microphones is less than half the wavelength of the audio signal, and the energy difference between audio signals received by two microphones in the microphone pair with the largest spacing is greater than a first value and less than a second value, the weight values ​​of the microphone pair formed by any two microphones among the plurality of microphones are different, but the difference between the weight values ​​is within the preset threshold range. It can be understood that, although the weight values ​​are different, the difference is small and the weight values ​​are similar.

[0099] As an example, when the distance between a certain microphone pair is greater than half of the wavelength of the audio signal, the relative delay of the microphone pair is likely to be higher than half of the audio signal period, and the risk of invalid calculation results is also high. Based on this, the weight value corresponding to the first sound source position corresponding to the microphone pair is small. As another example, when the energy of the audio signal received by a certain microphone is lower than the energy of the audio signal received by other microphones, the signal-to-noise ratio of the audio signal received by the microphone is also low, and the first sound source position corresponding to the microphone pair including this microphone is greatly affected by noise. Based on this, the weight value corresponding to the first sound source position corresponding to the microphone pair is small. In order to reduce the influence of environmental reflections and calculation errors, the amplitude estimation method can also be used to eliminate wild points. As another example, if the distance between the two microphones in the microphone pair is less than half of the wavelength of the received audio signal, or the energy of the audio signal received by each microphone is close (such as the difference between the received energies is within a preset threshold range), the weight values ​​corresponding to the first sound source positions determined by each microphone pair are the same or similar.

[0100] Specifically, take the number of microphones as six, i.e., microphone A, microphone B, microphone C, microphone D, microphone E, and microphone F, and assume that the audio signal is incident from a 15-degree direction, and the directions of microphones ABCDEF are 0, 60, 120, 180, 240, and 300 degrees, respectively. The direction of the audio signal is closest to the direction of microphone array element A. Here, the microphone may be an omnidirectional microphone, and the microphone and its surrounding structure (including the direction of the microphone) form a microphone array element, and the microphone array element is unidirectional.

[0101] When the frequency of the audio signal is high, such as 3000Hz, its signal wavelength is 11.3cm. Combined with the diameter of the bottom surface of the device and the arrangement information of the microphones, it can be seen that among all microphone pairs, the wavelength of the audio signal is less than twice the spacing between microphone pairs AD, BE, and CF, but greater than twice the spacing between other microphone pairs. By comparing the energy of the audio signals received by the six microphones, the microphone closest to the direction of the microphone can be determined. For example, the energy of the audio signal received by the microphones is sorted, the energy of microphone A is the largest, the energy of microphone B is second, and the energy of microphone F is third. It can be determined that the incident angle of the audio signal is closest to the direction of microphone A, followed by microphone B, and then microphone F. At this time, the sound source corresponding to the audio signal can be roughly located based on microphone A and microphone B or microphone A, microphone B and microphone F. Among all microphone pairs, the reception delay of microphone pair AD may be greater than 1 / 2 of the signal period. The delay calculation value is not unique and cannot be used for orientation. Its weight is reset to 0. Other microphone pairs can avoid this risk. The receiving delay of microphone pairs AB, AF, and FB is the smallest, and the energy of the received audio signal is strong and the signal-to-noise ratio is high. The sound source positions calculated by these three microphone pairs based on the receiving delay have corresponding higher weight values. The sound source positions calculated by other microphone pairs based on the receiving delay have corresponding weight values ​​that are smaller than the higher weight values. In addition, if the direction calculated by a certain microphone pair deviates from the approximate area determined based on microphone A and microphone B or microphone A, microphone B, and microphone F, this microphone pair may encounter abnormal reflection interference or noise interference and should be eliminated, and its corresponding weight value should be set to 0. Similarly, when the frequency of the audio signal is higher, more microphone pairs will be eliminated.

[0102] When the audio signal frequency is low, such as 1500Hz, the audio signal wavelength is 22.6cm, so that the distance between all microphone pairs is less than half of the wavelength, and the sound source positions calculated by all microphone pairs can be used for weighted calculation of the final sound source position. The directivity of each microphone is more obvious at this frequency. From the energy comparison between the microphone array elements and the microphone pairs, it can be known that the energy of microphone D is the lowest, and the energy difference between microphone pair AD is the largest. Therefore, in the weighted processing, the sound source positions calculated by all microphone pairs have the lowest weight value from microphone pair AD, the second lowest weight value from other microphone pairs including microphone D, and the highest weight value from microphone pair AB, microphone pair AF, and microphone pair BF, which have the largest energy and the smallest energy difference.

[0103] When the audio signal frequency is lower, such as 500Hz, the audio signal wavelength is 67.8cm, so that the distance between all microphone pairs is less than half of the wavelength, and the directivity of the microphone array element is not very obvious at this frequency. Even for the microphone pair with the largest difference, the energy difference does not exceed 3dB. At this time, the sound source direction weights calculated by each microphone pair are close. When the audio signal frequency is lower, such as 200Hz, the directivity of the microphone array element is very weak, and the sound source direction weights calculated by each microphone pair are equal.

[0104] It should be noted that the above method is a method of localizing the sound source using an interface microphone with an occluding effect inside the device. The embodiment of the present invention is to make full use of the occluding effect and avoid the error problem caused by the microphone receiving delay being higher than half a period as much as possible.

[0105] The embodiment of the present invention can calculate the sound sources in multiple different directions one by one. When it is determined that the sound source in a certain direction needs to be enhanced, the direction of the sound source and a certain angle range to the left and right can be set as a protection area, and other directions can be set as a suppression area. The audio signal from the protection area is enhanced, and the audio signal in the suppression area is weakened, so as to improve the clarity and quality of the audio signal. The audio signal enhancement method can include super-directional array filter, minimum variance distortion-free response array filter, blind source separation method, etc.

[0106] In one embodiment, the memory further stores an audio instruction recognition program; when the processor executes the audio instruction recognition program, it realizes the recognition of audio data obtained based on audio signal conversion, and obtains the audio instruction contained in the audio data.

[0107] Specifically, the user can control the audio interaction device through voice, for example, controlling the audio interaction device to play music files, pause playing music files, switch to play the "previous" or "next" music file, etc. Based on this, the audio interaction device is also provided with microphone-related components, such as an analog-to-digital conversion module, which is used to perform analog-to-digital conversion on audio signals to obtain audio data; when the processor executes the audio instruction recognition program, it recognizes the audio data and obtains the audio instruction contained in the audio data.

[0108] In one embodiment, the audio interaction device may further include a communication component that supports communication between the audio interaction device and other devices through a wired network or a wireless network. The audio interaction device can access a wireless network based on a communication standard, and the communication standard includes at least one of the following: wireless fidelity (WiFi), mobile phone action communication standards (such as 2G, 3G, 4G, 5G, etc.). In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be based on radio frequency identification (RFID) technology, infrared data organization (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology or other technologies.

[0109] In one embodiment, the audio interactive device may further include a power supply component for providing power to various components in the audio interactive device. The power supply component may include a power management system, at least one power supply, and other components associated with generating, managing, and distributing power to the audio interactive device.

[0110] In this embodiment, the processor is used to control the overall operation of the audio interaction device, such as audio output control, audio input control, volume adjustment, control of audio output content, etc. The number of processors may include at least one module to facilitate interaction with other components. For example, the processor may include a microphone module to facilitate interaction with a microphone.

[0111] In this embodiment, the memory can be implemented by any type of volatile or non-volatile storage device, or a combination thereof. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disk, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk memory or a tape memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), direct memory bus random access memory (DRRAM). The memory described in the embodiments of the present invention is intended to include, but is not limited to, these and any other suitable types of memory.

[0112] By adopting the technical solution of the embodiment of the present invention, on the one hand, the microphone is arranged at the bottom of the audio interaction device close to the placement surface, which improves the aesthetics of the overall appearance of the audio interaction device and avoids accidental touching of the microphone to generate noise during operation; on the other hand, in this embodiment, the speaker is arranged on the other side away from the microphone, that is, it is arranged on the top of the audio interaction device, which improves the audio output effect of the audio interaction device. Figure 6 Schematic diagram of the sensitivity of the microphone facing the sound source and the microphone facing away from the sound source of the audio interaction device according to an embodiment of the present invention; Figure 6 As shown, there is an amplitude difference of more than 5dB above 1500Hz, and an amplitude difference of more than 8dB above 3000Hz. Figure 7 FIG. 1 is a schematic diagram of the sensitivity of a microphone of an audio interaction device in various directions according to an embodiment of the present invention. Figure 7 As shown, when the signal source is at 0 degrees and 180 degrees, the sensitivity difference exceeds 5dB.

[0113] The embodiment of the present invention further provides a data processing method, which is applied to the aforementioned audio interaction device and is used to process an audio signal received by the audio interaction device. The method comprises:

[0114] Step 101: Obtain audio signals through multiple microphones.

[0115] Step 102: Determine a first sound source position by using a delay estimation and / or amplitude estimation method for a microphone pair formed by any two microphones among the plurality of microphones.

[0116] Step 103: performing weighted processing on the determined multiple first sound source positions to obtain the sound source position.

[0117] The data processing method of this embodiment is mainly used to perform sound source localization processing on audio signals received by multiple microphones.

[0118] As an implementation mode, the microphone pair formed by any two microphones among the multiple microphones uses a delay estimation method to determine the first sound source position, including: obtaining a first audio signal received by the first microphone, obtaining a second audio signal received by the second microphone; determining a receiving delay based on the first audio signal and the second audio signal; determining a distance difference between the sound source and the first microphone and the second microphone based on the receiving delay; determining the first sound source position based on the distance difference and the distance between the first microphone and the second microphone. The specific implementation process can be referred to the description in the aforementioned embodiment, which will not be repeated here.

[0119] In one embodiment, it is assumed that the sensitivity of the microphone in each direction theta at a certain frequency f can be represented by d(theta–thetak,f). d(alpha,f) indicates that in the direction with an angle alpha to the direction of the microphone, when alpha=0, the sensitivity is maximum. Function d can also be called a directivity function. If the direction of microphone A and microphone B is not in the same direction but at an angle beta, and the angles between the incident direction of the sound source signal and the directions of the two microphones are betaA and betaB respectively, then the directivity functions of microphone A and microphone B are d_A and d_B respectively; when the audio signal reaches the two microphones, the ratio of transmission attenuation HA and HB conforms to the formula HA / HB=d_A(betaA) / d_B(betaB). When the value of the directivity function d(alpha,f) changes significantly with the angle alpha, the direction of the audio signal relative to microphone A and microphone B can be obtained through the amplitude information. When the wavelength of the audio signal is shorter and the frequency is higher, the directivity of the microphone is more obvious, and the change of d(alpha,f) with direction is also more obvious.

[0120] In one embodiment, performing weighted processing on the determined multiple first sound source positions to obtain the sound source position includes: determining a weight value corresponding to the first sound source position of the microphone pair based on at least one of the following information, and performing weighted processing based on the weight value and the corresponding first sound source position to obtain the sound source position;

[0121] The information includes: an amplitude relationship of audio signals received by two microphones in a microphone pair;

[0122] the energy of the audio signal received by any microphone in the microphone pair;

[0123] The spacing between the two microphones in a microphone pair;

[0124] The attribute parameters of the audio signal received by any microphone in the microphone pair include at least one of the following: frequency, period, and wavelength.

[0125] In practical applications, the weight values ​​of N microphone pairs can be pre-set to 1 / N, where N is a positive integer greater than 1; 1 / N is further adjusted based on at least one of the aforementioned information, and after the adjustment is completed, the N weight values ​​are normalized so that the sum of the weight values ​​of the N microphones is 1.

[0126] In one embodiment, if the distance between two microphones in a microphone pair is greater than half the wavelength of the audio signal, the distance between the two microphones in the microphone pair is inversely correlated with the corresponding weight value, that is, the greater the distance between the two microphones in the microphone pair, the smaller the corresponding weight value. If, when the region where the signal incident direction is located is known, the distance between the two microphones in the microphone pair multiplied by the cosine of a certain incident direction in the region and the direction of the line connecting the microphone pair has an absolute value greater than half the wavelength of the audio signal, the weight value of the microphone pair is reduced to zero.

[0127] In one embodiment, when the incident direction of the audio signal can be roughly determined within an angle range, for each microphone pair, the acoustic path difference within this angle range is calculated. When the incident direction of the audio signal is within the region corresponding to the angle range, the distance between the two microphones in the microphone pair is multiplied by the cosine of the approximate direction of the audio signal determined within the region and the direction of the line connecting the microphone pair. The product represents the acoustic path difference, that is, the difference in the distance traveled by the sound source of the audio signal to reach the two microphones in the microphone pair. It can be understood that the acoustic path difference is determined based on the distance between the two microphones of the microphone pair; and the corresponding weight value is adjusted according to the comparison result of the acoustic path difference and the wavelength.

[0128] As an example, if the sound path difference exceeds 1 / 2 wavelength of the audio signal, the weight of the corresponding microphone pair is reduced to 0.

[0129] As another example, the sound path difference is compared with 3 / 8 wavelength of the audio signal; if the sound path difference exceeds 3 / 8 wavelength of the audio signal, the weight value of the corresponding microphone pair is reduced to 1 / 2 of the initial weight value 1 / N.

[0130] As another example, when the incident direction of the sound source has no or difficult to have a clear range, if the distance between two microphones in a microphone pair exceeds 1 / 2 wavelength of the audio signal, the weight value of the corresponding microphone pair is reduced to 0.

[0131] In one embodiment, if the energy of the audio signal received by a microphone is lower than the energy of the audio signals received by other microphones, the weight value of the microphone pair to which the microphone belongs is smaller than the weight values ​​of other microphone pairs.

[0132] As an example, the energy of the audio signals received by the microphones is checked and sorted by size; the maximum energy value is determined; if the energy of the audio signal received by a certain microphone is 6dB or more lower than the maximum energy value, the weight of the microphone pair is reduced to 1 / 2 of the initial weight value 1 / N.

[0133] In one embodiment, if the frequency of audio signals received by all microphones among a plurality of microphones is lower than a first preset threshold, so that the distance between a microphone pair formed by any two microphones among the plurality of microphones is lower than half the wavelength of the audio signal, and the energy difference between audio signals received by two microphones in the microphone pair with the largest distance is less than a first value, the weight values ​​of all microphone pairs are equal.

[0134] In one embodiment, if the frequency of audio signals received by all microphones among a plurality of microphones is greater than a first preset threshold and less than a second preset threshold, so that the spacing between a microphone pair formed by any two microphones among the plurality of microphones is less than half the wavelength of the audio signal, and the energy difference between audio signals received by two microphones in the microphone pair with the largest spacing is greater than a first value and less than a second value, the weight values ​​of the microphone pair formed by any two microphones among the plurality of microphones are different, but the difference between the weight values ​​is within the preset threshold range. It can be understood that, although the weight values ​​are different, the difference is small and the weight values ​​are similar.

[0135] As an example, when the distance between a certain microphone pair is greater than half of the wavelength of the audio signal, the relative delay of the microphone pair is likely to be higher than half of the audio signal period, and the risk of invalid calculation results is also high. Based on this, the weight value corresponding to the first sound source position corresponding to the microphone pair is small. As another example, when the energy of the audio signal received by a certain microphone is lower than the energy of the audio signal received by other microphones, the signal-to-noise ratio of the audio signal received by the microphone is also low, and the first sound source position corresponding to the microphone pair including this microphone is greatly affected by noise. Based on this, the weight value corresponding to the first sound source position corresponding to the microphone pair is small. In order to reduce the influence of environmental reflections and calculation errors, the amplitude estimation method can also be used to eliminate wild points. As another example, if the distance between the two microphones in the microphone pair is less than half of the wavelength of the received audio signal, or the energy of the audio signal received by each microphone is close (such as the difference between the received energies is within a preset threshold range), the weight values ​​corresponding to the first sound source positions determined by each microphone pair are the same or similar.

[0136] An embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the data processing method described in the embodiment of the present invention are implemented.

[0137] In the several embodiments provided in the present application, it should be understood that the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0138] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0139] A person skilled in the art can understand that: all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), disks or optical disks, etc. Various media that can store program codes.

[0140] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention can be essentially or partly reflected in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

[0141] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0142] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art who is familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. An audio interaction device, characterized in that: The device comprises: a housing, a plurality of microphones located in a housing space of the housing, at least one processor and a memory for storing a computer program that can be run on the processor, wherein the processor is used to execute, when running the computer program: identifying audio signals obtained by the plurality of microphones and processing the audio signals; Among them, the multiple microphones are interface microphones and the distance between the multiple microphones and the first surface of the shell of the audio interaction device is less than a first threshold, and the first surface is parallel to and the closest to the placement surface on which the audio interaction device is placed; the shell forms a accommodating portion having at least one reflective surface; each of the multiple microphones is located in a corresponding accommodating portion, and the multiple accommodating portions have the same structure.

2. The device according to claim 1, characterized in that The housing is provided with a first sound-transmitting hole corresponding to each microphone of the plurality of microphones; Among them, the plurality of first sound-permeable holes are located at the connection between the first surface and the side surface of the audio interaction device.

3. The device according to claim 2, characterized in that The first sound-permeable hole forms a centrally symmetrical opening shape on the shell.

4. The device according to claim 1, characterized in that The number of the plurality of microphones is associated with attribute parameters of the audio signal to be received; the attribute parameters at least include the wavelength and / or transmission frequency band of the audio signal.

5. The device according to any one of claims 1 to 4, characterized in that The angles formed between any two adjacent microphones among the multiple microphones and the center point of the audio interaction device are equal.

6. The device according to claim 1, characterized in that The device also includes at least one speaker; the at least one speaker is arranged on a second surface of a shell close to the audio interaction device, wherein the distance between the second surface and the first surface is greater than a second threshold.

7. The device according to claim 6, characterized in that The second surface of the housing is provided with second sound-permeable holes respectively corresponding to each speaker of the at least one speaker.

8. The device according to claim 1, characterized in that The processor further executes: determining a first sound source position by delay estimation and / or amplitude estimation for a microphone pair formed by any two microphones among the multiple microphones; and performing weighted processing on the determined multiple first sound source positions to obtain the sound source position.

9. The device according to claim 8, characterized in that The step of performing weighted processing on the determined multiple first sound source positions to obtain the sound source position comprises: determining a weight value corresponding to the first sound source position of the microphone pair based on at least one of the following information, and performing weighted processing based on the weight value and the corresponding first sound source position to obtain the sound source position; The information includes: an amplitude relationship of audio signals received by two microphones in a microphone pair; the energy of the audio signal received by any microphone in the microphone pair; The spacing between the two microphones in a microphone pair; The attribute parameters of the audio signal received by any microphone in the microphone pair include at least one of the following: frequency, period, and wavelength.

10. A data processing method, characterized in that: The method is applied to the audio interaction device according to any one of claims 1 to 9; the method comprises: Acquire audio signals through multiple microphones; Determine the first sound source position by using a delay estimation and / or amplitude estimation method for a microphone pair formed by any two microphones among the plurality of microphones; The determined multiple first sound source positions are weighted to obtain the sound source position.

11. The method according to claim 10, characterized in that The step of performing weighted processing on the determined multiple first sound source positions to obtain the sound source positions includes: Determine a weight value corresponding to a first sound source position of the microphone pair based on at least one of the following information, and perform weighted processing based on the weight value and the corresponding first sound source position to obtain the sound source position; The information includes: an amplitude relationship of audio signals received by two microphones in a microphone pair; the energy of the audio signal received by any microphone in the microphone pair; The spacing between the two microphones in a microphone pair; The attribute parameters of the audio signal received by any microphone in the microphone pair include at least one of the following: frequency, period, and wavelength.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the data processing method according to claim 10 or 11 are implemented.