Target object positioning method and apparatus

Through the multi-subarray acquisition and denoising processing of the target microphone array, combined with cross-power spectral density and scanning algorithm, the precise positioning of speakers in a noisy environment is achieved, solving the problems of positioning accuracy and audio clarity in video conferences.

WO2024230747A1PCT designated stage expired Publication Date: 2024-11-14DINGTALK (CHINA) INFORMATION TECH CO LTD
View PDF 7 Cites 0 Cited by

Patent Information

Application Number
PCT/CN2024/091767
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-11
Filing Date
2024-05-08
Publication Date
2024-11-14

AI Technical Summary

Technical Problem

In scenarios such as video conferencing, it is difficult for the prior art to accurately locate the speaker, especially in noise and reverberation environments, which affects the clarity and positioning accuracy of the audio signal.

Method used

The target microphone array is adopted to collect the target audio signals through multiple subarrays, perform denoising processing and cross-power spectral density analysis, and combine preset scanning algorithms to determine the target angle and sub-angle to accurately locate the position of the target object.

Benefits of technology

It improves the accuracy of speaker positioning and clarity of audio signals, reduces noise interference, and enhances the multi-party participation and director effect of video conferences.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2024091767_14112024_PF_FP_ABST
    Figure CN2024091767_14112024_PF_FP_ABST
Patent Text Reader

Abstract

Embodiments of the present description provide a target object positioning method and apparatus. The method comprises: determining a target microphone array, wherein the target microphone array comprises at least two sub-arrays, and each sub-array comprises microphones; determining target audio signals of a target object which are acquired by the plurality of microphones; processing the target audio signals, and according to a processing result, determining a target angle of the target object relative to the target microphone array; according to the target angle, determining a sub-angle of the target object relative to each sub-array; and according to the target angle and the sub-angle of the target object relative to each sub-array, determining the position of the target object relative to the target microphone array. By combining the target angle and the plurality of sub-angles, more calculation data can be provided when the position of the target object relative to the target microphone array is determined, so as to position the target object according to the sound emitted by the target object.
Need to check novelty before this filing date? Find Prior Art

Description

Target object positioning method and device

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on May 11, 2023, with application number 202310537612.3 and application name “Target Object Positioning Method and Device”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The embodiments of this specification relate to the field of speech processing technology, and in particular to a method for locating a target object. Background Art

[0003] With the development of electronic information technology and acoustics, sound source localization technology based on microphone arrays has emerged. A microphone array can be understood as an array consisting of several microphones arranged in a specific spatial geometry. For example, during a video conference, a microphone array can often be used to capture the speaker's audio signal, thereby enabling other participants in the video conference to understand the speaker's audio content.

[0004] However, in video conferencing, in order to ensure clear and accurate capture of the speaker's audio and video, it is often necessary to locate the speaker. Therefore, an effective technical solution is urgently needed to achieve speaker location.

[0005] Summary of the Invention

[0006] In view of this, embodiments of this specification provide a method for locating a target object. One or more embodiments of this specification also relate to a target object locating device, a conferencing device, a target object processing method, a target object processing device, a computing device, a computer-readable storage medium, and a computer program to address technical deficiencies in the prior art.

[0007] According to a first aspect of the embodiments of this specification, a method for locating a target object is provided, including:

[0008] Determining a target microphone array, wherein the target microphone array includes at least two sub-arrays, and each sub-array includes a microphone;

[0009] determining a target audio signal of a target object collected by the plurality of microphones;

[0010] processing the target audio signal, and determining a target angle of the target object relative to the target microphone array according to a processing result;

[0011] determining, according to the target angle, a sub-angle of the target object relative to each sub-array;

[0012] The position of the target object relative to the target microphone array is determined according to the target angle and the sub-angles of the target object relative to each sub-array.

[0013] According to a second aspect of the embodiments of this specification, a target object positioning device is provided, including:

[0014] A first determining module is configured to determine a target microphone array, wherein the target microphone array includes at least two sub-arrays, and each sub-array includes a microphone;

[0015] A second determining module is configured to determine a target audio signal of a target object collected by the plurality of microphones;

[0016] a processing module configured to process the target audio signal and determine a target angle of the target object relative to the target microphone array according to a processing result;

[0017] a third determining module, configured to determine a sub-angle of the target object relative to each sub-array according to the target angle;

[0018] The fourth determination module is configured to determine a position of the target object relative to the target microphone array according to the target angle and the sub-angles of the target object relative to each sub-array.

[0019] According to a third aspect of the embodiments of this specification, a conference device is provided, including:

[0020] target microphone array, memory, and processor;

[0021] Each microphone in the target microphone array is used to collect a target audio signal of a target object;

[0022] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the above method are implemented.

[0023] According to a fourth aspect of the embodiments of this specification, a target object processing method is provided, which is applied to a terminal-side device, including:

[0024] Receiving an initial image of a target object set captured by a camera device, and displaying the initial image on a display interface;

[0025] According to the target object positioning method described in the embodiment of this specification, a position of a target object relative to a target microphone array is determined, wherein the target object is any one object in the target object set;

[0026] A target image of the target object captured by the camera device is received, and the target image is displayed on a display interface, wherein the target image is captured by the camera device based on the position of the target object relative to the microphone array.

[0027] According to a fifth aspect of an embodiment of this specification, a target object processing apparatus is provided, which is applied to a terminal-side device and includes:

[0028] A first display module is configured to receive an initial image of a target object set captured by a camera device and display the initial image on a display interface;

[0029] a determination module configured to determine a position of a target object relative to a target microphone array according to the target object positioning method described in an embodiment of this specification, wherein the target object is any one object in the target object set;

[0030] The second display module is configured to receive a target image of the target object captured by the camera device and display the target image on a display interface, wherein the target image is captured by the camera device based on the position of the target object relative to the microphone array.

[0031] According to a sixth aspect of the embodiments of this specification, a computing device is provided, including:

[0032] memory and processor;

[0033] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the above method are implemented.

[0034] According to a seventh aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores computer-executable instructions, and the instructions implement the steps of the above method when executed by a processor.

[0035] According to an eighth aspect of the embodiments of this specification, a computer program is provided, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of the above method.

[0036] One embodiment of the present specification provides a method for locating a target object, which includes determining a target microphone array, wherein the target microphone array includes at least two sub-arrays, and each sub-array includes a microphone; determining a target audio signal of a target object collected by multiple microphones; processing the target audio signal, and determining a target angle of the target object relative to the target microphone array based on the processing result; determining a sub-angle of the target object relative to each sub-array based on the target angle; and determining a position of the target object relative to the target microphone array based on the target angle and the sub-angle of the target object relative to each sub-array.

[0037] When determining the position of the target object relative to the target microphone array, the above method not only takes into account the target angle of the target object relative to the target microphone array, but also takes into account the sub-angles of the target object relative to each sub-array. By combining the target angle and multiple sub-angles, more calculation data can be provided when determining the position of the target object relative to the target microphone array, thereby realizing the positioning of the target object based on the sound emitted by the target object, and further ensuring the accuracy of the positioning of the target object. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] FIG1 is a schematic diagram of an application scenario of a target object positioning method provided by an embodiment of this specification;

[0039] FIG2 is a flow chart of a target object positioning method provided by one embodiment of this specification;

[0040] FIG3 is a schematic diagram of the structure of a target microphone array in a target object positioning method provided by one embodiment of this specification;

[0041] FIG4 is a schematic diagram of reference coordinate axes of a target microphone array in a target object processing method provided by one embodiment of this specification;

[0042] FIG5 is a schematic diagram of a scanning method in a target object positioning method provided in an embodiment of this specification;

[0043] FIG6 is a schematic diagram of a positioning result of a target object positioning method provided by an embodiment of this specification;

[0044] FIG7 is a schematic diagram of another positioning result of a target object positioning method provided by an embodiment of this specification;

[0045] FIG8 is a flowchart of a processing process of a target object positioning method provided by one embodiment of this specification;

[0046] FIG9 is a schematic diagram of a target object positioning method provided by an embodiment of this specification applied to a teaching scenario;

[0047] FIG10 is a schematic structural diagram of a target object positioning device provided by one embodiment of this specification;

[0048] FIG11 is a flowchart of a target object processing method provided by one embodiment of this specification;

[0049] FIG12 is a schematic diagram of an application scenario of a target object processing method provided by an embodiment of this specification;

[0050] FIG13 is a schematic structural diagram of a target object processing device provided by one embodiment of this specification;

[0051] FIG14 is a structural block diagram of a computing device provided in one embodiment of this specification. DETAILED DESCRIPTION

[0052] The following description sets forth many specific details to facilitate a thorough understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the scope of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0053] The terms used in one or more embodiments of this specification are for the purpose of describing specific embodiments only and are not intended to limit one or more embodiments of this specification. The singular forms "a," "the," and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0054] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0055] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0056] First, the terms involved in one or more embodiments of this specification are explained.

[0057] Cross-power spectral density can be used to describe the frequency information between two random vibration processes. It not only provides the energy size distributed by frequency, but also provides the relationship between the two signals.

[0058] In this specification, a target object positioning method is provided. This specification also involves a target object positioning device, a conference device, a target object processing method, a target object processing device, a computing device, and a computer-readable storage medium, which are described in detail one by one in the following embodiments.

[0059] Referring to FIG. 1 , FIG. 1 shows a schematic diagram of an application scenario of a target object positioning method provided according to an embodiment of this specification.

[0060] FIG. 1 includes a conference device 102 .

[0061] During specific implementation, in an online live broadcast scenario, the speaker (such as a course lecturer or a conference speaker) usually uses the conference device 102 to collect the speaker's audio and image. When the speaker speaks, he or she can face the conference device 102 to facilitate the conference device 102 to collect the speaker's audio and image, thereby realizing the live broadcast of the meeting or online course. The conference device 102 is deployed with a target microphone array, which can be used to collect the speaker's voice signal. After determining the target audio signal of the speaker collected by each microphone in the target microphone array, the conference device 102 can determine the target angle of the speaker relative to the target microphone array based on the target audio signal, and can also determine the sub-angle of the speaker relative to each sub-array included in the target microphone array. The position of the speaker relative to the target microphone array is determined based on the target angle and the sub-angle of each sub-array.

[0062] After determining the position of the speaker relative to the target microphone array, the conference device 102 can use the position to zoom or track the video image of the speaker to achieve multi-party participation in the video conference.

[0063] Furthermore, conference device 102 can also communicate with cloud-side devices. After the target microphone array collects the speaker's voice signal, it can send the signal to the cloud-side device. The cloud-side device removes noise from the signal and locates the speaker based on the signal. The cloud-side device then sends the location result to conference device 102, which then uses it to perform subsequent operations.

[0064] Referring to FIG. 2 , FIG. 2 shows a flow chart of a target object positioning method provided according to an embodiment of this specification, which specifically includes the following steps.

[0065] Step 202: Determine a target microphone array, wherein the target microphone array includes at least two sub-arrays, and each sub-array includes a microphone.

[0066] Specifically, the target object positioning method provided in the embodiments of this specification can be applied to the conference terminal equipment in the audio and video conferencing system. The target object can be, for example, a speaker speaking in a video conference. Using this target object positioning method, the speaker's audio can be collected through the conference terminal equipment, and the speaker can be positioned based on the audio, so that the speaker's partial image can be enlarged based on the speaker's position information, so that other participants in the video conference can see the speaker more clearly, thereby achieving the speaker's guidance. Alternatively, when the speaker's position changes, the speaker can be tracked and guided based on the positioning result of the speaker. This target object positioning method can locate one speaker, or it can locate or distinguish two or more speakers. In addition, this target object positioning method can also be applied to audio interaction scenarios such as voice calls. In voice call scenarios, it is necessary to locate the speaker, so that the speaker's audio can be collected based on the positioning result of the speaker, making the collected audio clearer.

[0067] For ease of understanding, the embodiments of this specification are described in detail by taking the target object positioning method applied to a video conferencing scenario as an example, but this does not affect the implementation of the target object positioning method in other feasible scenarios.

[0068] Among them, the target microphone array can be understood as an array composed of microphones arranged according to a spatial geometric structure. In practical applications, the target microphone array can be set in a conference device to collect sound signals of the video conference site during a video conference. The target microphone array can collect multi-channel sound signals. From the perspective of microphone type, the target microphone array can be an array composed of omnidirectional microphones, or an array composed of directional microphones. From the perspective of array shape, the microphone array can be a linear array or a circular array, or an array of irregular shape. The embodiments of this specification do not limit this. The subarray can be understood as a subarray obtained by dividing the microphones included in the target microphone array.

[0069] Specifically, as shown in Figure 3, Figure 3 shows a schematic diagram of the structure of a target microphone array in a target object localization method provided according to one embodiment of this specification. Referring to Figure 3, target microphone array 300 includes a first sub-array 310, a second sub-array 320, and a third sub-array 330. Each sub-array includes a microphone M.

[0070] It is understood that Figure 3 is only a schematic diagram of a target microphone array. The number of subarrays included in the target microphone array can be determined according to actual needs. For example, the target microphone array can include 2 subarrays, 3 subarrays, 4 subarrays, etc., and this embodiment of the specification is not limited here.

[0071] Step 204: Determine the target audio signal of the target object collected by the plurality of microphones.

[0072] Specifically, after the target microphone array is determined, the target audio signal of the target object collected by each microphone in the target microphone array may be determined.

[0073] The term "multiple microphones" may be understood as referring to some or all microphones in a target microphone array. For example, if the target microphone array includes 10 microphones, the target audio signal collected by 5 of the 10 microphones may be determined, or the target audio signal collected by each of the 10 microphones may be determined. The number of microphones to be determined may be determined based on actual needs and is not limited in this embodiment of the present specification.

[0074] The target object can be understood as a speaker. Then, the target audio signal of the target object can be understood as an audio signal corresponding to the sound emitted by the speaker when speaking.

[0075] Based on this, the audio signal corresponding to the sound emitted by the speaker when speaking, collected by each microphone in the target microphone array, can be determined.

[0076] For example, when the target microphone array includes three microphones m1, m2 and m3, the target audio signal 1 of the target object collected by microphone m1 can be determined; the target audio signal 2 of the target object collected by microphone m2 can be determined; and the target audio signal 3 of the target object collected by microphone m3 can be determined.

[0077] In specific implementations, in video conferencing scenarios, since there may be other sound sources in the venue, noise may be generated. When using the target microphone array to collect audio signals, the background noise generated by the electrical system may also cause noise impact. In large venues, reverberation may also occur. Based on this, in order to ensure more accurate subsequent positioning, it is necessary to denoise the audio signals collected by the microphones to obtain the denoised target audio signals. The specific implementation method is as follows:

[0078] The determining of the target audio signal of the target object collected by the plurality of microphones includes:

[0079] determining an initial audio signal of a target object collected by the plurality of microphones;

[0080] The initial audio signal is subjected to denoising processing to obtain a target audio signal of a target object collected by the plurality of microphones.

[0081] The initial audio signal can be understood as an audio signal collected by a microphone before de-noising, and the target audio signal can be understood as a denoised audio signal obtained by denoising the initial audio signal collected by the microphone.

[0082] Specifically, the initial audio signal of the speaker collected by each microphone in the target microphone array may be determined, and denoising processing may be performed on the initial audio signal to obtain the target audio signal of the speaker collected by each microphone.

[0083] Continuing with the above example, for the initial audio signal of the speaker collected by microphone m1, the initial audio signal can be denoised to obtain the denoised target audio signal 1; correspondingly, for the initial audio signal of the speaker collected by microphone m2, the initial audio signal can be denoised to obtain the denoised target audio signal 2; for the initial audio signal of the speaker collected by microphone m3, the initial audio signal can be denoised to obtain the denoised target audio signal 3.

[0084] In summary, by denoising the initial audio signal collected by each microphone in the target microphone array to obtain the denoised target audio signal, the interference of reverberation and noise can be reduced when locating the speaker based on the audio signal, further ensuring the accuracy of the positioning result.

[0085] Specifically, when denoising the initial audio signal, a filter may be used to perform separation processing on the initial audio signal to separate the noise signal from the audio signal to obtain a denoised target audio signal.

[0086] In one embodiment provided in this specification, the initial audio signal may be denoised by noise estimation, which is specifically implemented as follows:

[0087] The performing denoising on the initial audio signal to obtain the target audio signal of the target object collected by the plurality of microphones includes:

[0088] Determine a signal-to-noise ratio of a first initial audio signal collected by any one microphone, and determine a first cross-power spectral density between the first initial audio signal and the initial audio signal;

[0089] determining, based on the signal-to-noise ratio, a second cross-power spectral density between the noise of the first initial audio signal and the noise of the initial audio signal;

[0090] The initial audio signal is subjected to denoising processing according to the first cross-power spectral density and the second cross-power spectral density to obtain a target audio signal of a target object collected by the plurality of microphones.

[0091] Among them, since the cross power spectrum density includes the phase difference between the audio signals collected from different channels, the phase difference can be used to represent the time difference between the sound emitted by the target object reaching the microphone, and the time difference can be used for subsequent target object positioning. The first initial audio signal can be understood as any initial audio signal. For example, for three initial audio signals collected by three microphones respectively, the first initial audio signal is any one of the three initial audio signals. The signal-to-noise ratio of the first initial audio signal can be understood as the ratio between the first initial audio signal and the noise of the first initial audio signal. According to the signal-to-noise ratio of the first initial audio signal, the second cross power spectrum density between the noise of the initial audio signal collected by each microphone and the noise of the first initial audio signal can be determined.

[0092] Specifically, the first cross power spectral density between the first initial audio signal and the initial audio signal can be obtained based on a vector of the initial audio signal in the frequency domain. The vector of the initial audio signal in the frequency domain can be obtained by performing a short-time Fourier transform on the initial audio signal. Correspondingly, the second cross power spectral density can be obtained based on the vector of the initial audio signal in the frequency domain.

[0093] Based on this, the initial audio signals of the target object collected by each microphone can be determined, and the initial audio signals can be short-time Fourier transformed to realize the conversion processing of the initial audio signals from the time domain to the frequency domain, and obtain the vector of the initial audio signal in the frequency domain. Among the initial audio signals collected by each microphone, any first initial audio signal is selected, the noise of the first initial audio signal is estimated, and the signal-to-noise ratio of the first initial audio signal is determined. According to the signal-to-noise ratio of the first initial audio signal and the vector of the initial audio signal in the frequency domain, the second cross-power spectral density between the noise of the first initial audio signal and the noise of the initial audio signal is obtained. According to the vector of the initial audio signal in the frequency domain, the first cross-power spectral density between the first initial audio signal and the initial audio signal is obtained. The first cross-power spectral density and the second cross-power spectral density are subtracted to obtain the denoised target audio signal. In this way, the audio signal is denoised.

[0094] In practical applications, noise estimation of the first initial audio signal can be performed using a minimum statistical noise estimation algorithm, which can be used to estimate the power spectral density of steady-state noise. Alternatively, other noise estimation algorithms can be used, which are not limited in this embodiment of the present disclosure.

[0095] When the short-time Fourier transform is used to convert the initial audio signal from the time domain to the frequency domain, the vector of the initial audio signals collected by M microphones converted to the frequency domain is shown in the following formula (1). M (k,n)] (1)

[0096] Where k is the frequency domain index, n is the time domain index, and M is the number of microphones. y1(k,n) is the frequency domain vector of the initial audio signal collected by the first microphone. y(k,n) is the frequency domain vector of the initial audio signal collected by M microphones.

[0097] The first cross power spectrum density between the initial audio signal and the first initial audio signal is expressed as the following formula (2): Y(k,n)=λY(k,n-1)+(1-λ)y(k,n) (2)

[0098] Where Y(k,n)=[Y 1,r (k,n),……,Y m,r (k,n),……,Y M,r (k,n)], is the cross power spectral density of the initial audio signal collected by M microphones. λ is the smoothing coefficient, which ranges from 0 to 1. m is the serial number of the microphone. r is the serial number of the microphone that collects the first initial audio signal (i.e., the serial number of the microphone used for noise estimation). Y1,r (k,n) is the cross power spectrum density between the initial audio signal collected by the first microphone and the first initial audio signal. m,r (k,n) is the cross power spectrum density between the initial audio signal collected by the mth microphone and the first initial audio signal. M,r (k, n) is the cross power spectral density between the initial audio signal collected by the Mth microphone and the first initial audio signal.

[0099] The second cross power spectrum density between the noise of the first initial audio signal and the noise of the initial audio signal is expressed as the following formula (3).

[0100] Where N(k,n)=[N 1,r (k,n),……,N m,r (k,n),……,N M,r (k, n)], is the second cross power spectrum density between the noise of the initial audio signal collected by M microphones and the noise of the first initial audio signal, S r (k,n) is the signal-to-noise ratio of the first initial audio signal. N 1,r (k,n) is the second cross power spectrum density between the noise of the initial audio signal collected by the first microphone and the noise of the first initial audio signal. m,r (k,n) is the second cross power spectrum density between the noise of the initial audio signal collected by the mth microphone and the noise of the first initial audio signal. M,r (k, n) is a second cross power spectral density between the noise of the initial audio signal collected by the Mth microphone and the noise of the first initial audio signal.

[0101] The target audio signal obtained by subtracting the second cross power spectrum density from the first cross power spectrum density is shown in the following formula (4). Φ(k,n)=Y(k,n)-N(k,n)=[Φ1(k,n),……,Φ m (k,n),……,Φ M (k,n)] (4)

[0102] Among them, Φ m (k,n)=Y m,r (k,n)-N m,r (k,n) is the target audio signal obtained after denoising the initial audio signal collected by the mth microphone. Φ(k,n) is the target audio signal obtained after denoising the initial audio signals collected by the M microphones. Φ M (k, n) is the target audio signal obtained after denoising the initial audio signal collected by the Mth microphone.

[0103] Continuing with the above example, if the target microphone array includes three microphones m1, m2, and m3, determine the frequency domain vector X1 of the initial audio signal m1 collected by microphone m1, the frequency domain vector X2 of the initial audio signal m2 collected by microphone m2, the frequency domain vector X3 of the initial audio signal m3 collected by microphone m3, and the frequency domain vector (X1, X2, X3) consisting of the three initial audio signals collected by the three microphones. Of these three initial audio signals, the initial audio signal m1 collected by microphone m1 is used as the first initial audio signal. Noise estimation is performed on the first initial audio signal m1 to determine the signal-to-noise ratio of the first initial audio signal m1. Based on the signal-to-noise ratio of the first initial audio signal m1 and the frequency domain vector X1 of the initial audio signal m1, determine the second cross-power spectral density mX1 between the noise of the initial audio signal m1 and the noise of the first initial audio signal m1. Based on the frequency domain vector X1 of the initial audio signal m1, determine the first cross-power spectral density XX1 between the initial audio signal m1 and the first initial audio signal m1. Subtract the first cross power spectral density XX1 from the second cross power spectral density mX1 to obtain the denoised target audio signal 1 corresponding to the initial audio signal m1. When denoising the initial audio signal m2, determine the second cross power spectral density mX2 between the noise of the initial audio signal m2 and the noise of the first initial audio signal m1 based on the signal-to-noise ratio of the first initial audio signal m1 and the vector X2 of the initial audio signal m2 in the frequency domain. Determine the first cross power spectral density XX2 between the initial audio signal m2 and the first initial audio signal m1 based on the vector X2 of the initial audio signal m2 in the frequency domain. Subtract the first cross power spectral density XX2 from the second cross power spectral density mX2 to obtain the denoised target audio signal 2 corresponding to the initial audio signal m2. Similarly, perform a similar operation on the initial audio signal m3 to obtain the denoised target audio signal 3.

[0104] In summary, denoising the audio signal through noise estimation reduces the impact of noise and reverberation on target object positioning. Furthermore, since the audio signal collected by each microphone can be assumed to be identical, the cross-power spectral density can be calculated based solely on the signal-to-noise ratio of the audio signal collected by any one microphone, reducing computing power consumption and further improving target object positioning efficiency.

[0105] Step 206: Process the target audio signal, and determine a target angle of the target object relative to the target microphone array according to the processing result.

[0106] Specifically, after determining the target audio signal collected by each microphone and after denoising, the target audio signal may be processed, and a target angle of the target object relative to the target microphone array may be determined based on the processing result.

[0107] The target angle of the target object relative to the target microphone array can be understood as the target angle of the target object under the target reference coordinate axis of the target microphone array. The target angle can be understood as the direction angle of the target object relative to the target reference coordinate axis of the target microphone array.

[0108] The target reference coordinate axis of the target microphone array can be understood as a coordinate system with the linear array of microphones as the x-axis and the direction facing the target object as the y-axis. The target reference point of this target reference coordinate axis can be located anywhere in the target microphone array. In practical applications, to ensure that the camera coordinate system and the microphone coordinate system match, the target reference point can be located at the camera itself.

[0109] Specifically, referring to FIG. 4 , FIG. 4 shows a schematic diagram of reference coordinate axes of a target microphone array in a target object processing method provided according to an embodiment of this specification.

[0110] As shown in FIG4 , target microphone array 400 includes a first sub-array 410, a second sub-array 420, and a third sub-array 430. xO1y1 is the first reference coordinate axis of first sub-array 410, xOy is the target reference coordinate axis of target microphone array 400 and third sub-array 430, and xO2y2 is the second reference coordinate axis of second sub-array 420. θ1 is the first sub-angle of target object p relative to first sub-array 410, i.e., the first sub-angle of the target object in the first reference coordinate axis. θ2 is the second sub-angle of target object p relative to second sub-array 420, i.e., the second sub-angle of the target object in the second reference coordinate axis. θ0 is the target angle of target object p relative to target microphone array 400, i.e., the target angle of the target object in the target reference coordinate axis. θ3 is the third sub-angle of target object p relative to third sub-array 430, i.e., the third sub-angle of the target object in the target reference coordinate axis. The vertical distance between target object p and the target microphone array is D. When the target object is subsequently positioned, the vertical distance D and the target angle θ0 need to be determined.

[0111] In specific implementation, in order to calculate the target angle of the target object relative to the target microphone array, a preset scanning algorithm can be used to scan the preset scanning range with the target audio signal as the input signal to obtain the target angle that meets the preset scanning conditions. The specific implementation method is as follows:

[0112] The processing of the target audio signal and determining a target angle of the target object relative to the target microphone array according to the processing result includes:

[0113] Using the target audio signal as an input signal, scanning a preset scanning range using a preset scanning algorithm, wherein the preset scanning range is determined according to a preset scanning angle and a preset scanning distance;

[0114] A target angle of the target object relative to the target microphone array is determined according to the scanning result.

[0115] The preset scanning distance may be determined according to the actual size of the target microphone array.

[0116] In practical applications, the preset scanning algorithm can be understood as any algorithm for estimating the location of a sound source. The preset scanning angle can be from 0 degrees to 180 degrees. The preset scanning angle can be defined in the target reference coordinate axis of the target microphone array. Referring to FIG5 , FIG5 is a schematic diagram of a scanning method in a target object positioning method provided according to an embodiment of the present specification. As shown in FIG5 , the preset scanning angle θ can be defined in the target reference coordinate axis xOy of the target microphone array. l The system scans at the preset scanning angle and distance R, and determines a target angle that satisfies the scanning conditions within the preset scanning angles. This target angle is the target angle of the target object relative to the target microphone array. Figure 5 also includes steady-state noise sources. Due to the aforementioned denoising of the initial audio signal, steady-state noise sources within the preset scanning range do not affect the scanning results.

[0117] Based on this, the target audio signal collected by each microphone in the target microphone array can be used as the input signal, and the preset scanning algorithm can be used to scan the preset scanning range composed of the preset scanning angle and the preset scanning distance, and the target angle of the target object relative to the target microphone array can be determined according to the scanning results.

[0118] In a specific implementation, each subarray may include at least two microphones. The target angle of the target object relative to the target microphone array may be calculated based on the phase and phase matching coefficient of each target audio signal. The phase matching coefficient may be generated by a beamforming method based on a preset scanning distance and a preset scanning angle. Target angle As shown in the following formula (5).

[0119] in,

[0120] Among them, θ l is the preset scanning angle, m is the microphone, U1 is the first subarray, U2 is the second subarray, U3 is the third subarray, U is the target microphone array, is the phase matching coefficient of the target audio signal collected by the mth microphone, is the phase of the target audio signal collected by the mth microphone. l ,n) is the intermediate result for calculating the first sub-angle, S2(k,θ l ,n) is the intermediate result for calculating the second sub-angle, S3(k,θ l ,n) is the intermediate result used to calculate the third sub-angle.

[0121] In summary, by utilizing the preset scanning algorithm, a target angle that meets the scanning conditions is determined in the preset scanning angles, and the target angle is used as the target angle of the target object relative to the target microphone array, providing a data basis for subsequent positioning of the target object.

[0122] Step 208: Determine the sub-angles of the target object relative to each sub-array according to the target angle.

[0123] Specifically, after the target angle is determined, the sub-angles of the target angle relative to each sub-array may be determined based on the target angle.

[0124] In specific implementation, since it is difficult to ensure the accuracy of the positioning result by locating the target object only based on the target angle, based on this, the sub-angles of the target object relative to each sub-array can be used in combination with the target angle to ensure the accuracy of the positioning result. The specific implementation method is as follows:

[0125] The at least two sub-arrays include a first sub-array and a second sub-array, wherein the first sub-array and the second sub-array are respectively located at two ends of the target microphone array;

[0126] Accordingly, determining the sub-angles of the target object relative to each sub-array according to the target angle includes:

[0127] According to the target angle, a first sub-angle of the target object relative to the first sub-array and a second sub-angle of the target object relative to the second sub-array are determined.

[0128] It can be seen from the above formula (5) that when performing a full-angle scan on the target audio signal collected by each microphone included in the target microphone array, since the target microphone array includes each sub-array, the scanning result obtained by performing the full-angle scan also includes the scanning result of the microphone included in each sub-array. The first sub-angle and the second sub-angle can be directly obtained by reusing the intermediate result of the full-angle scanning operation in formula (5).

[0129] In summary, by directly obtaining the first sub-angle and the second sub-angle according to the target angle, the computing power consumption in the sub-array scanning operation can be omitted, thereby saving computing power.

[0130] In a specific implementation, determining, based on the target angle, a first sub-angle of the target object relative to the first sub-array and a second sub-angle of the target object relative to the second sub-array includes:

[0131] Obtaining, according to the target angle, a first coordinate angle of the target object relative to the first subarray under a target reference coordinate axis of the target microphone array, and a second coordinate angle of the target object relative to the second subarray under the target reference coordinate axis;

[0132] Converting the first coordinate angle into a first sub-angle of the target object relative to the first sub-array under a first reference coordinate axis of the first sub-array;

[0133] The second coordinate angle is converted into a second sub-angle of the target object relative to the second sub-array under the second reference coordinate axis of the second sub-array.

[0134] Specifically, the first coordinate angle is shown in the following formula (6), and the second coordinate angle is shown in the following formula (7).

[0135] in, is the first coordinate angle, is the second coordinate angle.

[0136] In the aforementioned full-angle scanning of the target audio signals collected by each microphone in the target microphone array, the scanning is performed within the target reference coordinate axis of the target microphone array. Therefore, the first coordinate angle of the target object relative to the first sub-array under the target reference coordinate axis and the second coordinate angle of the target object relative to the second sub-array under the target reference coordinate axis are determined based on the target angle. In order to locate the target object, it is also necessary to convert the first coordinate angle and the second coordinate angle to the first sub-angle under the first reference coordinate axis and the second coordinate angle to the second sub-angle under the second reference coordinate axis.

[0137] Accordingly, the first sub-angle is shown in the following formula (8), and the second sub-angle is shown in the following formula (9).

[0138] in, All are intermediate variables. c1 is the distance between the first reference point O1 of the first reference coordinate axis and the target reference point O of the target reference coordinate axis. R is the preset scanning distance. c2 is the distance between the second reference point O2 of the second reference coordinate axis and the target reference point O of the target reference coordinate axis.

[0139] In summary, by transforming the coordinate axes, the first sub-angle of the target object relative to the first sub-array and the second sub-angle of the target object relative to the second sub-array can be determined, which facilitates the subsequent determination of the position of the target object based on the first sub-angle and the second sub-angle.

[0140] Step 210: Determine the position of the target object relative to the target microphone array according to the target angle and the sub-angles of the target object relative to each sub-array.

[0141] Specifically, after determining the target angle and the sub-angles of the target object relative to each sub-array, the position of the target object relative to the target microphone array may be determined according to the target angle and the sub-angles of the target object relative to each sub-array.

[0142] The target angle and the sub-angle can both be understood as the angle of the estimated direction of arrival of the sound emitted by the target object (ie, the arrival direction angle).

[0143] In a specific implementation, determining the position of the target object relative to the target microphone array according to the target angle and the sub-angles of the target object relative to each sub-array includes:

[0144] Calculating a vertical distance of the target object relative to the target microphone array according to the first sub-angle and the second sub-angle;

[0145] A position of the target object relative to the target microphone array is determined according to the target angle and the vertical distance.

[0146] The position of the target object relative to the target microphone array can be understood as the position of the target object in the target reference coordinate axis.

[0147] Specifically, the vertical distance D of the target object relative to the target microphone array can be calculated according to the following formula (10).

[0148] Specifically, the vertical distance and target angle of the target object relative to the target microphone array may be used as the position of the target object relative to the target microphone array.

[0149] In practical applications, if the target microphone array is large and the microphones are unevenly distributed, the closer the target object is to the target microphone array, the larger the target angle will be, leading to inaccurate positioning results. To address this, a third subarray, located in the same target reference coordinate axis as the target microphone array, can be used to correct the target angle, thereby improving the accuracy of the arrival direction angle estimation. The specific implementation method is as follows:

[0150] The at least two sub-arrays further include a third sub-array, wherein the third sub-array is located between the first sub-array and the second sub-array;

[0151] Accordingly, determining the position of the target object relative to the target microphone array according to the target angle and the vertical distance includes:

[0152] determining a third sub-angle of the target object relative to the third sub-array according to the target angle;

[0153] If it is determined that the vertical distance is less than or equal to the preset distance threshold, using the third sub-angle as the adjusted target angle;

[0154] A position of the target object relative to the target microphone array is determined according to the adjusted target angle and the vertical distance.

[0155] The preset distance threshold may be determined according to the actual size of the target microphone array.

[0156] Similar to the above, the third sub-angle of the target object relative to the third sub-array can be directly obtained according to the target angle. The third sub-angle is shown in the following formula (11).

[0157] in, It is the third sub-angle.

[0158] Specifically, if it is determined that the vertical distance between the target object and the target microphone array is less than or equal to a preset distance threshold, the third sub-angle of the target object relative to the third sub-array can be used as the adjusted target angle. The adjusted target angle (i.e., the third sub-angle) and the vertical distance are used as the position of the target object relative to the target microphone array.

[0159] Accordingly, determining the position of the target object relative to the target microphone array according to the target angle and the vertical distance further includes:

[0160] When it is determined that the vertical distance is greater than the preset distance threshold, the position of the target object relative to the target microphone array is determined according to the target angle and the vertical distance.

[0161] In practical applications, the target angle can be corrected according to the following formula (12).

[0162] Wherein, D0 is the preset distance threshold.

[0163] In summary, the target angle is corrected by the third sub-direction angle, which avoids the deviation of the target angle estimation caused by the close distance between the target object and the target microphone array, and further ensures the accuracy of the target object positioning result.

[0164] In practical applications, this target object localization method is illustrated using the example of capturing speaker audio in a video conferencing scenario. The target microphone array has a maximum size of 0.72 meters, with a total of 19 microphones. The first subarray has four microphones, c1 = 0.39 meters; the second subarray has three microphones, c2 = 0.175 meters; and the third subarray has four microphones. During speaker localization, the scanning distance R = 7.0 meters, the preset distance threshold D0 = 2.1 meters, and the smoothing coefficient λ = 0.9.

[0165] Referring to FIG6 , FIG6 illustrates a schematic diagram of a positioning result of a target object positioning method provided according to an embodiment of the present disclosure. FIG6 illustrates the positioning result when a speaker is positioned to the left front of a target microphone array. FIG6(a) illustrates the positioning result of the vertical distance between the speaker and the target microphone array. FIG6(b) illustrates the positioning result of the azimuth angle between the speaker and the target microphone array. The true value of the vertical distance to the speaker is 0.86 meters, and the true value of the azimuth angle to the speaker is 60 degrees. The true value can be understood as the actual value. The first 2 seconds of the positioning result include a period of algorithm startup time, which can be ignored. In the vertical distance estimation results, the vertical estimation error is within 10%. In the speaker azimuth angle results, the speaker azimuth angle (uncorrected) of the main array (target microphone array) has a deviation of more than 10 degrees and significant fluctuations, while the azimuth angle of the middle subarray (third subarray) is stable with only a deviation of 2 degrees. Using this to correct the speaker azimuth angle of the main array will significantly improve the positioning accuracy within close range.

[0166] Refer to Figure 7, which shows another schematic diagram of positioning results of a target object positioning method provided according to an embodiment of the present specification. Figure 7 shows the positioning result when the speaker is directly in front of the target microphone array. Figure 7(a) shows the positioning result of the vertical distance between the speaker and the target microphone array. Figure 7(b) shows the positioning result of the azimuth angle between the speaker and the target microphone array. The true value of the vertical distance of the speaker is 1.0 meter, and the true value of the speaker azimuth angle is 88 degrees. The first 2 seconds of the positioning result contain a period of algorithm startup time which can be ignored. The results are the same as above: the error of the vertical distance estimation result is small (within 10%), and the speaker azimuth angle is corrected by using the intermediate subarray, and the angle deviation in the close range is controlled within 2 degrees, which significantly improves the accuracy.

[0167] To sum up, when determining the position of the target object relative to the target microphone array, the above method not only takes into account the target angle of the target object relative to the target microphone array, but also takes into account the sub-angles of the target object relative to each sub-array. By combining the target angle and multiple sub-angles, more calculation data can be provided when determining the position of the target object relative to the target microphone array, thereby realizing the positioning of the target object according to the sound emitted by the target object, and further ensuring the accuracy of the positioning of the target object.

[0168] The following further illustrates the target object positioning method provided in this specification using the application of the target object positioning method in a video conferencing scenario as an example, in conjunction with FIG8 . FIG8 shows a flowchart of a target object positioning method provided in one embodiment of this specification, specifically including the following steps.

[0169] Step 802: Determine a target microphone array, wherein the target microphone array includes at least two sub-arrays, and each sub-array includes a microphone.

[0170] Specifically, in a video conferencing scenario, a conference device can be deployed. The conference device is equipped with a target microphone array, which includes a left terminal array (i.e., a first subarray), a middle subarray (i.e., a third subarray), and a right terminal array (i.e., a second subarray). Each subarray includes at least two microphones. The target microphone array is used to collect the speaker's audio, and the speaker can be subsequently located based on the collected audio.

[0171] Step 804: Determine the initial audio signal of the target object collected by each microphone.

[0172] In a video conference scenario, the target object may be the speaker in the video conference.

[0173] Specifically, the initial audio signal of the speaker collected by each microphone in the target microphone array may be determined.

[0174] Step 806: performing denoising processing on the initial audio signal to obtain the target audio signal of the target object collected by each microphone.

[0175] Specifically, when denoising the initial audio signal, a short-time Fourier transform can be performed on each initial audio signal collected by each microphone to achieve a conversion from the time domain to the frequency domain, thereby obtaining a vector of each initial audio signal in the frequency domain. The short-time Fourier transform is a conversion method for converting a time domain signal into a frequency domain signal. In addition, other methods such as the Fourier transform can also be used to achieve the conversion from the time domain to the frequency domain, which is not limited in the embodiments of this specification. Furthermore, among the initial audio signals collected by each microphone, any initial audio signal is selected for noise estimation to determine the signal-to-noise ratio of the initial audio signal. Based on the signal-to-noise ratio and the vector of each initial audio signal in the frequency domain, the cross-power spectral density between each initial audio signal and the initial audio signal for which noise estimation is performed, as well as the cross-power spectral density between the noise signal of each initial audio signal and the initial audio signal for which noise estimation is performed, can be calculated. Based on the calculated cross-power spectral density, each target audio signal after denoising is determined.

[0176] Step 808: Using the target audio signal as an input signal, a preset scanning algorithm is used to scan a preset scanning range, and a target angle of the target object relative to the target microphone array is determined based on the scanning result.

[0177] The preset scanning range can be determined according to a preset scanning distance and a preset scanning angle.

[0178] Specifically, when the preset scanning range is scanned using the preset scanning algorithm, an angle that meets the preset scanning condition may be determined in the preset scanning range as the target angle.

[0179] Step 810: Determine the sub-angles of the target object relative to each sub-array according to the target angle.

[0180] Specifically, since the target audio signals collected by all microphones in the target microphone array are scanned at full angles, based on this, the first sub-angle of the target object relative to the left terminal array, the second sub-angle of the target object relative to the right terminal array, and the third sub-angle of the target object relative to the middle sub-array can be obtained according to the scanning results of the aforementioned target angles.

[0181] Step 812: Calculate a vertical distance of the target object relative to the target microphone array based on the first sub-angle and the second sub-angle.

[0182] Step 814: Adjust the target angle according to the third sub-angle to obtain an adjusted target angle.

[0183] Specifically, when the vertical distance is less than or equal to the distance threshold (ie, the preset distance threshold), the third sub-angle is used as the adjusted target angle. When the vertical distance is greater than the distance threshold, the target angle is not adjusted.

[0184] Step 816: Determine the position of the target object relative to the target microphone array based on the adjusted target angle and vertical distance.

[0185] When determining the position of the target object relative to the target microphone array, the above method not only takes into account the target angle of the target object relative to the target microphone array, but also takes into account the sub-angles of the target object relative to each sub-array. By combining the target angle and multiple sub-angles, more calculation data can be provided when determining the position of the target object relative to the target microphone array, thereby realizing the positioning of the target object based on the sound emitted by the target object, and further ensuring the accuracy of the positioning of the target object.

[0186] Referring to FIG9 , FIG9 shows a schematic diagram of a target object positioning method provided by an embodiment of the present specification applied to a teaching scenario. In actual application, when a teacher (i.e., the target object) is recording a video course or broadcasting an online course in a recording room, a conference device 902 is placed in the recording room. The conference device 902 includes a target microphone array and a camera. Before the video course recording or online course broadcast begins, the teacher can stand at any position in the recording room. The display interface of the terminal-side device 904 in the recording room can display a position map of the interior of the recording room, that is, the display interface displays the position of the target microphone array in the recording room, and when the teacher speaks, the target microphone array collects the audio signal emitted by the teacher, thereby determining the teacher's position, and displaying the teacher's position through the display interface of the terminal-side device 904. As the teacher moves, the teacher's position on the display interface of the terminal-side device 904 will also change. The teacher speaks at position 1 in the recording room, moves from position 1 to position 2, speaks at position 2, and moves from position 2 to position 3, and speaks at position 3. The conference device 902 or the client-side device 904 can use the target object positioning method described above to determine the teacher's location 1, location 2, and location 3 each time they speak. The target location is determined based on locations 1, 2, and 3, as well as the audio signals collected by the target microphone array each time the teacher speaks. When the teacher stands at the target location, the target microphone array receives clearer sound. This target location is then displayed to the teacher via the display interface of the client-side device 904. The teacher can use the target location displayed on the client-side device 904 to determine their speaking position during a video course recording or online live course.

[0187] Corresponding to the above method embodiment, this specification also provides an embodiment of a target object positioning device. FIG10 shows a schematic structural diagram of a target object positioning device provided in one embodiment of this specification. As shown in FIG10 , the device includes:

[0188] A first determining module 1002 is configured to determine a target microphone array, wherein the target microphone array includes at least two sub-arrays, and each sub-array includes a microphone;

[0189] A second determining module 1004 is configured to determine a target audio signal of a target object collected by the plurality of microphones;

[0190] a processing module 1006 configured to process the target audio signal and determine a target angle of the target object relative to the target microphone array according to a processing result;

[0191] A third determining module 1008 is configured to determine a sub-angle of the target object relative to each sub-array according to the target angle;

[0192] The fourth determining module 1010 is configured to determine the position of the target object relative to the target microphone array according to the target angle and the sub-angles of the target object relative to each sub-array.

[0193] In an optional embodiment, the at least two subarrays include a first subarray and a second subarray, and the first subarray and the second subarray are respectively located at two ends of the target microphone array; and the third determining module 1008 is further configured to:

[0194] According to the target angle, a first sub-angle of the target object relative to the first sub-array and a second sub-angle of the target object relative to the second sub-array are determined.

[0195] In an optional embodiment, the fourth determining module 1010 is further configured to:

[0196] Calculating a vertical distance of the target object relative to the target microphone array according to the first sub-angle and the second sub-angle;

[0197] A position of the target object relative to the target microphone array is determined according to the target angle and the vertical distance.

[0198] In an optional embodiment, the at least two subarrays further include a third subarray, and the third subarray is located between the first subarray and the second subarray; and the fourth determining module 1010 is further configured to:

[0199] determining a third sub-angle of the target object relative to the third sub-array according to the target angle;

[0200] If it is determined that the vertical distance is less than or equal to the preset distance threshold, using the third sub-angle as the adjusted target angle;

[0201] A position of the target object relative to the target microphone array is determined according to the adjusted target angle and the vertical distance.

[0202] In an optional embodiment, the fourth determining module 1010 is further configured to:

[0203] When it is determined that the vertical distance is greater than the preset distance threshold, the position of the target object relative to the target microphone array is determined according to the target angle and the vertical distance.

[0204] In an optional embodiment, the third determining module 1008 is further configured to:

[0205] Obtaining, according to the target angle, a first coordinate angle of the target object relative to the first subarray under a target reference coordinate axis of the target microphone array, and a second coordinate angle of the target object relative to the second subarray under the target reference coordinate axis;

[0206] Converting the first coordinate angle into a first sub-angle of the target object relative to the first sub-array under a first reference coordinate axis of the first sub-array;

[0207] The second coordinate angle is converted into a second sub-angle of the target object relative to the second sub-array under the second reference coordinate axis of the second sub-array.

[0208] In an optional embodiment, the processing module 1006 is further configured to:

[0209] Using the target audio signal as an input signal, scanning a preset scanning range using a preset scanning algorithm, wherein the preset scanning range is determined according to a preset scanning angle and a preset scanning distance;

[0210] A target angle of the target object relative to the target microphone array is determined according to the scanning result.

[0211] In an optional embodiment, the second determining module 1004 is further configured to:

[0212] determining an initial audio signal of a target object collected by the plurality of microphones;

[0213] The initial audio signal is subjected to denoising processing to obtain a target audio signal of a target object collected by the plurality of microphones.

[0214] In an optional embodiment, the second determining module 1004 is further configured to:

[0215] Determine a signal-to-noise ratio of a first initial audio signal collected by any one microphone, and determine a first cross-power spectral density between the first initial audio signal and the initial audio signal;

[0216] determining, based on the signal-to-noise ratio, a second cross-power spectral density between the noise of the first initial audio signal and the noise of the initial audio signal;

[0217] The initial audio signal is subjected to denoising processing according to the first cross-power spectral density and the second cross-power spectral density to obtain a target audio signal of a target object collected by the plurality of microphones.

[0218] To sum up, when determining the position of the target object relative to the target microphone array, the above-mentioned device not only takes into account the target angle of the target object relative to the target microphone array, but also takes into account the sub-angles of the target object relative to each sub-array. By combining the target angle and multiple sub-angles, it is possible to provide more calculation data when determining the position of the target object relative to the target microphone array, thereby realizing the positioning of the target object according to the sound emitted by the target object, and further ensuring the accuracy of the positioning of the target object.

[0219] The above is a schematic diagram of a target object positioning device according to this embodiment. It should be noted that the technical solution of the target object positioning device and the technical solution of the target object positioning method described above are based on the same concept. For details not described in detail in the technical solution of the target object positioning device, please refer to the description of the technical solution of the target object positioning method described above.

[0220] Corresponding to the above method embodiments, an embodiment of this specification also provides a target object processing method, which is applied to a terminal-side device. FIG11 shows a flowchart of a target object processing method provided according to an embodiment of this specification. The specific steps are as follows.

[0221] Step 1102: receiving an initial image of the target object set captured by a camera device, and displaying the initial image on a display interface;

[0222] Step 1104: Determine the position of a target object relative to a target microphone array according to a target object positioning method provided in one embodiment of this specification, wherein the target object is any one of the target object set;

[0223] Step 1106: receiving a target image of the target object captured by the camera device, and displaying the target image on a display interface, wherein the target image is captured by the camera device based on the position of the target object relative to the microphone array.

[0224] The initial image of the target object set captured by the camera device can be understood as the initial image of the target object set captured by the camera device based on a preset initial position. For example, if the camera device is initially facing forward and the target object set is located directly in front of the camera device, then the initial image of the target object set is the initial image of the target object set captured by the camera device facing forward. The target microphone array and camera device are both provided in the conference device.

[0225] Specifically, an initial image of a target object set captured by a camera device based on a preset initial position can be received and displayed on a display interface. According to the target object positioning method provided above, any object in the target object set is located to determine the position of the object relative to the target microphone array. A target image of the object captured by the camera device based on the position of the object relative to the target microphone array is received and displayed on a display interface.

[0226] Taking a video conference scenario as an example, refer to Figure 12, which shows a schematic diagram of an application scenario of a target object processing method provided by an embodiment of this specification. Figure 12 includes a conference device 1202 and an end-side device 1204. Specifically, when multiple participants (i.e., a set of target objects) participate in a video conference, the multiple participants can sit in front of the conference device 1202, the camera of the conference device 1202 can be used to capture images of the participants, and the target microphone array of the conference device 1202 can be used to capture audio signals emitted by the participants. When one of the multiple participants, participant A (i.e., the target object), speaks, the conference device 1202 or the end-side device 1204 can determine the position of participant A relative to the target microphone array (which can be understood as relative to the conference device) based on the audio signal emitted by participant A when speaking and collected by the target microphone array, and based on the position, control the camera of the conference device 1202 to turn to participant A. Accordingly, before participant A speaks, the display interface of the terminal device 1204 displays an image of multiple participants captured by the camera at its initial position. After participant A speaks, the camera turns to focus on participant A, and the display interface of the terminal device 1204 displays participant A's image. Alternatively, the display interface of the terminal device 1204 can continue to display multiple participant images, and a light reminder can be displayed at participant A's position. If there are many participants, this can inform other participants who the current speaker is.

[0227] It is understandable that the above method can also be applied to virtual conference scenarios. The specific process is similar and will not be repeated in this manual.

[0228] Corresponding to the above method embodiment, this specification also provides an embodiment of a target object processing device. FIG13 shows a schematic diagram of the structure of a target object processing device provided by one embodiment of this specification. As shown in FIG13, the device includes:

[0229] A first display module 1302 is configured to receive an initial image of a target object set captured by a camera device and display the initial image on a display interface;

[0230] The determination module 1304 is configured to determine a position of a target object relative to the target microphone array according to the target object positioning method described in the embodiments of this specification, wherein the target object is any one object in the target object set;

[0231] The second display module 1306 is configured to receive a target image of the target object captured by the camera device and display the target image on a display interface, wherein the target image is captured by the camera device based on the position of the target object relative to the microphone array.

[0232] The above is a schematic diagram of a target object processing device according to this embodiment. It should be noted that the technical solution of the target object processing device and the technical solution of the target object processing method described above are based on the same concept. For details not described in detail in the technical solution of the target object processing device, please refer to the description of the technical solution of the target object processing method described above.

[0233] Corresponding to the above method embodiment, the embodiment of this specification further provides a conference device, including:

[0234] target microphone array, memory, and processor;

[0235] Each microphone in the target microphone array is used to collect a target audio signal of a target object;

[0236] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the above method are implemented.

[0237] When determining the position of a target object relative to a target microphone array, the conferencing device not only takes into account the target angle of the target object relative to the target microphone array, but also takes into account the sub-angles of the target object relative to each sub-array. By combining the target angle and multiple sub-angles, the device can provide more calculation data when determining the position of the target object relative to the target microphone array, thereby locating the target object based on the sound emitted by the target object, and further ensuring the accuracy of the target object positioning.

[0238] The above is a schematic scheme of a conference device of this embodiment. It should be noted that the technical scheme of the conference device and the technical scheme of the target object positioning method described above are of the same concept. For details not described in detail in the technical scheme of the conference device, please refer to the description of the technical scheme of the target object positioning method described above.

[0239] Figure 14 shows a block diagram of a computing device 1400 according to one embodiment of this specification. Components of computing device 1400 include, but are not limited to, a memory 1410 and a processor 1420. Processor 1420 is connected to memory 1410 via a bus 1430, and a database 1450 is used to store data.

[0240] Computing device 1400 also includes an access device 1440 that enables computing device 1400 to communicate via one or more networks 1460. Examples of such networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. Access device 1440 may include one or more of any type of network interface (e.g., a network interface controller (NIC)) whether wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and the like.

[0241] In one embodiment of the present application, the aforementioned components of the computing device 1400 and other components not shown in FIG14 may also be connected to each other, for example, via a bus. It should be understood that the computing device structure block diagram shown in FIG14 is for illustrative purposes only and does not limit the scope of the present application. Those skilled in the art may add or replace other components as needed.

[0242] Computing device 1400 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, personal digital assistant, laptop computer, notebook computer, netbook computer, etc.), a mobile phone (e.g., a smartphone), a wearable computing device (e.g., a smartwatch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or personal computer (PC). Computing device 1400 may also be a mobile or stationary server.

[0243] The processor 1420 is configured to execute the following computer-executable instructions, which implement the steps of the above method when executed by the processor.

[0244] The above is a schematic solution of a computing device of this embodiment. It should be noted that the technical solution of the computing device and the technical solution of the above method belong to the same concept. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the above method.

[0245] An embodiment of the present specification further provides a computer-readable storage medium storing computer-executable instructions, which implement the steps of the above method when executed by a processor.

[0246] The above is a schematic solution of a computer-readable storage medium of this embodiment. It should be noted that the technical solution of the storage medium and the technical solution of the above method belong to the same concept. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solution of the above method.

[0247] An embodiment of the present specification further provides a computer program, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of the above method.

[0248] The above is an illustrative solution of a computer program of this embodiment. It should be noted that the technical solution of the computer program and the technical solution of the above method belong to the same concept, and any details not described in detail in the technical solution of the computer program can be referred to the description of the technical solution of the above method.

[0249] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0250] The computer instructions include computer program code, which may be in source code form, object code form, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal, and software distribution medium. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media does not include electric carrier signals and telecommunication signals.

[0251] It should be noted that for the aforementioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0252] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0253] The preferred embodiments disclosed above are intended only to help illustrate this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific embodiments described. Obviously, many modifications and variations can be made based on the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that those skilled in the art can better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A method for locating a target object, comprising: Determining a target microphone array, wherein the target microphone array includes at least two sub-arrays, and each sub-array includes a microphone; Determining a target audio signal of a target object collected by the plurality of microphones; Processing the target audio signal, and determining a target angle of the target object relative to the target microphone array according to a processing result; Determining sub-angles of the target object relative to each sub-array according to the target angle; The position of the target object relative to the target microphone array is determined according to the target angle and the sub-angles of the target object relative to each sub-array.

2. The method according to claim 1, wherein the at least two sub-arrays include a first sub-array and a second sub-array, and the first sub-array and the second sub-array are respectively located at two ends of the target microphone array; Correspondingly, determining the sub-angles of the target object relative to each sub-array according to the target angle includes: According to the target angle, a first sub-angle of the target object relative to the first sub-array and a second sub-angle of the target object relative to the second sub-array are determined.

3. The method according to claim 2, wherein determining the position of the target object relative to the target microphone array according to the target angle and the sub-angles of the target object relative to each sub-array comprises: Calculating a vertical distance of the target object relative to the target microphone array according to the first sub-angle and the second sub-angle; A position of the target object relative to the target microphone array is determined according to the target angle and the vertical distance.

4. The method according to claim 3, wherein the at least two sub-arrays further include a third sub-array, and the third sub-array is located between the first sub-array and the second sub-array; Accordingly, determining the position of the target object relative to the target microphone array according to the target angle and the vertical distance includes: determining, according to the target angle, a third sub-angle of the target object relative to the third sub-array; In the case where it is determined that the vertical distance is less than or equal to the preset distance threshold, taking the third sub-angle as the adjusted target angle; The position of the target object relative to the target microphone array is determined according to the adjusted target angle and the vertical distance.

5. The method according to claim 4, wherein determining the position of the target object relative to the target microphone array according to the target angle and the vertical distance further comprises: When it is determined that the vertical distance is greater than the preset distance threshold, the position of the target object relative to the target microphone array is determined according to the target angle and the vertical distance.

6. The method according to claim 2, wherein determining, according to the target angle, a first sub-angle of the target object relative to the first sub-array and a second sub-angle of the target object relative to the second sub-array comprises: According to the target angle, obtaining a first coordinate angle of the target object relative to the first subarray under a target reference coordinate axis of the target microphone array, and a second coordinate angle of the target object relative to the second subarray under the target reference coordinate axis; Converting the first coordinate angle into a first sub-angle of the target object relative to the first sub-array under a first reference coordinate axis of the first sub-array; The second coordinate angle is converted into a second sub-angle of the target object relative to the second sub-array under a second reference coordinate axis of the second sub-array.

7. The method according to claim 1, wherein the processing the target audio signal and determining the target angle of the target object relative to the target microphone array according to the processing result comprises: Using the target audio signal as an input signal, and using a preset scanning algorithm to scan a preset scanning range, wherein the preset scanning range is determined according to a preset scanning angle and a preset scanning distance; A target angle of the target object relative to the target microphone array is determined according to the scanning result.

8. The method according to claim 1, wherein determining the target audio signal of the target object collected by the plurality of microphones comprises: Determining initial audio signals of a target object collected by the plurality of microphones; The initial audio signal is subjected to denoising processing to obtain a target audio signal of a target object collected by the plurality of microphones.

9. The method according to claim 8, wherein the denoising process is performed on the initial audio signal to obtain the target audio signal of the target object collected by the plurality of microphones, comprising: Determine a signal-to-noise ratio of a first initial audio signal collected by any microphone, and determine a first cross power spectral density between the first initial audio signal and the initial audio signal; determining, according to the signal-to-noise ratio, a second cross power spectral density between the noise of the first initial audio signal and the noise of the initial audio signal; The initial audio signal is denoised according to the first cross power spectral density and the second cross power spectral density to obtain a target audio signal of a target object collected by the plurality of microphones.

10. A target object positioning device, comprising: A first determination module is configured to determine a target microphone array, wherein the target microphone array includes at least two sub-arrays, and each sub-array includes a microphone; A second determination module is configured to determine a target audio signal of a target object collected by the plurality of microphones; a processing module, configured to process the target audio signal and determine a target angle of the target object relative to the target microphone array according to a processing result; A third determination module is configured to determine a sub-angle of the target object relative to each sub-array according to the target angle; The fourth determination module is configured to determine the position of the target object relative to the target microphone array according to the target angle and the sub-angles of the target object relative to each sub-array.

11. A conference device comprising: target microphone array, memory, and processor; Each microphone in the target microphone array is used to collect a target audio signal of a target object; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the method described in any one of claims 1 to 9 are implemented.

12. A target object processing method, applied to a terminal device, comprising: Receiving an initial image of a target object set captured by a camera device, and displaying the initial image on a display interface; According to the target object positioning method according to any one of claims 1 to 9, determining the position of the target object relative to the target microphone array, wherein the target object is any one object in the target object set; A target image for the target object captured by the camera device is received, and the target image is displayed on a display interface, wherein the target image is captured by the camera device based on the position of the target object relative to the microphone array.

13. A computing device comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the method described in any one of claims 1 to 9 are implemented.

14. A computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions, when executed by a processor, implement the steps of the method according to any one of claims 1 to 9.

Citation Information

Patent Citations

  • Acoustic pointing device, pointing method of sound source position, and computer system

    CN101515197A

  • Sound source positioning method and device based on microphone array and computer equipment

    CN111060872A

  • Sound source positioning method and device

    CN113203988A

  • Acoustic imaging method supporting zooming

    CN114994607A

  • Sound source localization method and device based on microphone linear double arrays and medium

    CN115128544A