Sound source positioning method, electronic device, and storage medium

Abstract

The application discloses a sound source positioning method, an electronic device and a storage medium, and relates to the technical field of acoustic signal processing. The sound source positioning method is applied to a head-mounted device, a first microphone array is arranged in the head-mounted device, a second microphone array is arranged in a necklace in communication connection with the head-mounted device, and the sound source positioning method comprises the following steps: acquiring a first microphone array signal collected by the first microphone array; acquiring a relative pose between the necklace and the head-mounted device, and acquiring sound source positioning auxiliary data sent by the necklace, wherein the sound source positioning auxiliary data is obtained based on a second microphone array signal collected by the second microphone array; and performing sound source positioning according to the first microphone array signal, the sound source positioning auxiliary data and the relative pose to obtain a sound source positioning result. The application realizes more accurate sound source positioning in the full spatial range of the head-mounted device.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of acoustic signal processing, in particular to a sound source positioning method, an electronic device and a storage medium. BACKGROUND

[0002] At present, head-mounted devices such as AR (Augmented Reality) / VR (Virtual Reality) devices, smart glasses and the like have limited number of microphones due to size and appearance design limitations, and the layout spacing between the microphones is narrow. This leads to the geometry of the microphone array usually close to a planar circular array, which is difficult to form an effective aperture in the vertical direction, so that the height information of the sound source cannot be accurately distinguished, that is, there is a problem of ambiguity in the up-down direction positioning. Secondly, the head-mounted device is close to the user's head when worn, and the user's head will have a significant shielding effect on sound waves. This shielding will especially affect the sound signals from the rear of the wearer, causing the rear sound source signal received by the microphone array to be distorted or weakened, thereby significantly reducing the positioning accuracy of the rear sound source.

[0003] The above content is only used to assist in understanding the technical solutions of the present application and does not represent the acknowledgement of the above content as prior art. SUMMARY

[0004] The main purpose of the present application is to provide a sound source positioning method, an electronic device and a storage medium, which aims to solve the technical problem of low sound source positioning accuracy of the current head-mounted device.

[0005] To achieve the above purpose, the present application provides a sound source positioning method, which is applied to a head-mounted device, a first microphone array is arranged in the head-mounted device, a second microphone array is arranged in a necklace in communication connection with the head-mounted device, and the sound source positioning method comprises:

[0006] Obtaining a first microphone array signal collected by the first microphone array;

[0007] Obtaining the relative pose between the necklace and the head-mounted device, and obtaining sound source positioning auxiliary data sent by the necklace, wherein the sound source positioning auxiliary data is obtained based on a second microphone array signal collected by the second microphone array;

[0008] Performing sound source positioning according to the first microphone array signal, the sound source positioning auxiliary data and the relative pose to obtain a sound source positioning result.

[0009] Optionally, the sound source positioning auxiliary data includes a candidate direction set obtained based on the second microphone array signal for sound source positioning.

[0010] The step of performing sound source positioning according to the first microphone array signal, the sound source positioning auxiliary data and the relative pose to obtain a sound source positioning result comprises:

[0011] Performing sound source positioning based on the first microphone array signal to obtain a preliminary positioning direction;

[0012] After converting the preliminary positioning direction and each candidate direction in the candidate direction set to the same coordinate system according to the relative pose, performing weighted summation on the preliminary positioning direction and each candidate direction to obtain a sound source positioning result.

[0013] Optionally, the sound source positioning auxiliary data further comprises a second TDOA matrix calculated based on the second microphone array signal, the second TDOA matrix comprising a corresponding measured time difference of arrival of each microphone pair in the second microphone array;

[0014] The step of performing weighted summation on the preliminary positioning direction and each candidate direction to obtain a sound source positioning result comprises:

[0015] For each direction pair composed of the preliminary positioning direction and each candidate direction, calculating a consistency score of the direction pair according to the second TDOA matrix and a first TDOA matrix calculated based on the first microphone array signal, wherein the first TDOA matrix comprises a corresponding measured time difference of arrival of each microphone pair in the first microphone array;

[0016] Taking the consistency score of the direction pair as a weight of the candidate direction in the direction pair, performing weighted summation on each candidate direction to obtain a first fused direction;

[0017] Performing weighted summation on the first fused direction and the preliminary positioning direction to obtain a sound source positioning result.

[0018] Optionally, the step of performing weighted summation on the first fused direction and the preliminary positioning direction to obtain a sound source positioning result comprises:

[0019] If both the first fused direction and the preliminary positioning direction fall within a target direction range, taking a first weight as a weight corresponding to the first fused direction, and performing weighted summation on the first fused direction and the preliminary positioning direction to obtain a sound source positioning result, wherein the target direction range is a conical or fan-shaped region around a target direction in a coordinate system of the head-mounted device, and is a subset of a full direction range of sound source positioning of the head-mounted device, and the target direction is oriented towards the back of the wearer's head when the head-mounted device is worn.

[0020] If at least one of the first fusion direction and the preliminary positioning direction does not fall within the target direction range, a second weight is used as a weight corresponding to the first fusion direction, and a weighted sum of the first fusion direction and the preliminary positioning direction is obtained to obtain a sound source positioning result, wherein the second weight is less than the first weight.

[0021] Optionally, the step of obtaining a sound source positioning result by performing a weighted sum of the preliminary positioning direction and each candidate direction comprises:

[0022] performing a weighted sum of the preliminary positioning direction and each candidate direction to obtain a preliminary fusion direction;

[0023] calculating a third TDOA matrix according to the preliminary fusion direction, wherein the third TDOA matrix comprises theoretical time difference of arrival of each target microphone pair assuming that the sound source comes from the preliminary fusion direction, the each target microphone pair being a microphone pair formed by a microphone in the first microphone array and a microphone in the second microphone array;

[0024] calculating a frequency spectrum feature, a spatial filter response map and an ambient noise spectrum according to the first microphone array signal;

[0025] inputting the third TDOA matrix, the frequency spectrum feature, the spatial filter response map and the ambient noise spectrum into a preset deep learning model for processing to obtain a sound source positioning result.

[0026] Optionally, the sound source positioning method further comprises:

[0027] in a case where the preliminary positioning direction is obtained and the sound source positioning auxiliary data sent by the necklace has not been received, outputting the preliminary positioning direction as a sound source positioning result;

[0028] in a case where the sound source positioning auxiliary data sent by the necklace is received, outputting the sound source positioning result obtained by performing a weighted sum of the preliminary positioning direction and each candidate direction.

[0029] Optionally, the sound source positioning method further comprises:

[0030] obtaining a clock offset between the necklace and the necklace;

[0031] The step of performing sound source positioning according to the first microphone array signal, the sound source positioning auxiliary data and the relative pose to obtain a sound source positioning result comprises:

[0032] Data time alignment is performed according to a collection time of the first microphone array signal, a timestamp carried in the sound source positioning auxiliary data, and the clock offset, sound source positioning is performed based on the aligned data and the relative pose, and a sound source positioning result is obtained.

[0033] In addition, to achieve the above object, the application further provides a sound source positioning method applied to a necklace, wherein a second microphone array is arranged in the necklace, and a first microphone array is arranged in a head-mounted device in communication connection with the necklace, and the sound source positioning method comprises the following steps of:

[0034] obtaining a second microphone array signal collected by the second microphone array;

[0035] obtaining sound source positioning auxiliary data based on the second microphone array signal;

[0036] sending the sound source positioning auxiliary data to the head-mounted device, so that the head-mounted device performs sound source positioning according to a first microphone array signal, the sound source positioning auxiliary data and a relative pose between the head-mounted device and the necklace, and obtains a sound source positioning result, wherein the first microphone array signal is obtained based on the first microphone array.

[0037] In addition, to achieve the above object, the application further provides an electronic device, which further comprises a memory, a processor and a computer program stored in the memory and executable on the processor, and the computer program is configured to implement the steps of the sound source positioning method as described above.

[0038] In addition, to achieve the above object, the application further provides a storage medium, which is a computer readable storage medium, and the storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the sound source positioning method as described above.

[0039] In addition, to achieve the above object, the application further provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the steps of the sound source positioning method as described above.

[0040] The one or more technical solutions provided by the application have at least the following technical effects:

[0041] The first microphone array signal of the head-mounted device is acquired to establish the basic data for sound source positioning; the relative pose of the necklace and the head-mounted device and the sound source positioning auxiliary data are acquired to establish a unified spatial reference system and introduce the auxiliary spatial information of the necklace; the sound source positioning is performed based on the first microphone array signal, the sound source positioning auxiliary data and the relative pose, and the microphone arrays of the head-mounted device and the necklace are virtually integrated into an enhanced array with a more optimal spatial layout; then, the natural height difference of the necklace in the vertical direction can be used to solve the problem that the head-mounted device cannot distinguish the sound from above and below, and the microphone unit located behind the neck of the necklace can be used to avoid the head shielding, so as to significantly improve the positioning accuracy of the sound source behind, and finally realize more accurate sound source positioning in the whole space. BRIEF DESCRIPTION OF DRAWINGS

[0042] The accompanying drawings, which are incorporated herein and form a part of the specification, illustrate embodiments consistent with the present application and, together with the description, further serve to explain the principles of the application.

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, for those skilled in the art, other drawings can also be obtained based on these drawings without creative labor.

[0044] Figure 1 A schematic diagram of the layout position of the microphone array in a head-mounted device related to the sound source positioning method of the present application;

[0045] Figure 2 A schematic diagram of the layout position of the microphone array in a necklace related to the sound source positioning method of the present application;

[0046] Figure 3 A flowchart provided by the first embodiment of the sound source positioning method of the present application;

[0047] Figure 4 A flowchart provided by the fourth embodiment of the sound source positioning method of the present application;

[0048] Figure 5 A device structure diagram of the hardware running environment related to the sound source positioning method in the embodiments of the present application.

[0049] The object implementation, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0050] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application, and are not used to limit the present application.

[0051] For better understanding of the technical solutions of the present application, the following will be described in detail in combination with the accompanying drawings and specific embodiments.

[0052] The geometry of the microphone array is usually close to a planar ring array, which is difficult to form an effective aperture in the vertical direction, so as to accurately distinguish the height information of the sound source, that is, there is a problem of ambiguity in the up-down direction positioning. Secondly, the head-mounted device is close to the user's head when worn, and the user's head will have a significant shielding effect on sound waves. This shielding will especially affect the sound signals from the rear of the wearer, causing the rear sound source signal received by the microphone array to be distorted or weakened, thereby causing the positioning accuracy of the rear sound source to be significantly reduced.

[0053] The embodiments of the present application provide a solution, which establishes the basic data for sound source positioning by acquiring the first microphone array signal of the head-mounted device; establishes a unified spatial reference system and introduces the auxiliary spatial information of the necklace by acquiring the relative pose of the necklace and the head-mounted device and the sound source positioning auxiliary data; performs sound source positioning based on the first microphone array signal, the sound source positioning auxiliary data and the relative pose, and virtually integrates the microphone arrays of the head-mounted device and the necklace into an enhanced array with a more optimal spatial layout; and then the problem that the head-mounted device cannot distinguish up-down sound can be solved by using the natural height difference of the necklace in the vertical direction, and the head shielding can be avoided by using the microphone unit located behind the neck of the necklace, which significantly improves the positioning accuracy of the rear sound source, and finally realizes more accurate sound source positioning in the full spatial range.

[0054] That is, the embodiments of the present application form a large three-dimensional microphone array system by the combination of the head-mounted device and the smart necklace, which can overcome the above problems caused by using the head-mounted device alone, realize high-precision sound source positioning in all directions in space, and solve the problems of being unable to distinguish up-down and insufficient accuracy of rear positioning.

[0055] The first embodiment of the sound source positioning method of the present application is proposed below. In the present embodiment, the execution subject of the sound source positioning method is a head-mounted device. It can be understood that there are many types of head-mounted devices at present, such as smart glasses, VR (Virtual Reality) / AR (Augmented Reality) devices, etc., and there are many types of hardware architecture and software system implementation of each head-mounted device. The embodiments of the present application do not limit the type, hardware architecture and software system implementation of the head-mounted device to which the sound source positioning method is applied. In the present embodiment, a microphone array (hereinafter referred to as a first microphone array for distinction) is arranged in the head-mounted device, and a second microphone array (hereinafter referred to as a second microphone array for distinction) is arranged in the necklace. The head-mounted device can establish a communication connection with the necklace, and the second microphone array in the necklace is used to assist in sound source positioning, thereby improving the accuracy of sound source positioning. A microphone array is a set composed of at least two microphones arranged in a certain manner. In the present embodiment, the number of microphones included in the first microphone array and the second microphone array is not limited, and the arrangement of the first microphone array in the head-mounted device and the arrangement of the second microphone array in the necklace are not limited. In a feasible implementation manner, the first microphone array on the head-mounted device can adopt a ring layout, for example, 3 to 5 microphones are arranged on the frame of smart glasses. These microphones are arranged in a ring shape in the horizontal plane, without deliberately creating a height difference in the vertical direction. This layout can achieve accurate sound source positioning with only 3 microphones, thereby reducing the constraints on the appearance design of the glasses, allowing more flexible and aesthetic design, and meeting the appearance requirements of the glasses as a decorative item. Exemplarily, Figure 1 An example of arranging 4 microphones on the frame of smart glasses is given, Figure 1 A, B, C and D are four positions respectively arranged with a microphone, constituting the first microphone array. In a feasible implementation manner, the second microphone array on the necklace can be designed to include 8 to 12 microphones, which are distributed on the necklace and around the user's neck when worn. Since the necklace is worn around the neck, some microphones are located behind the neck, thereby being able to capture sounds from the rear, making up for the defect of the head-mounted device in positioning the sound source from the rear. At the same time, the necklace naturally forms a height difference in the vertical direction when worn, which helps to distinguish the up and down directions of the sound and solves the problem of ambiguous positioning of the head-mounted device in the vertical direction. Exemplarily, Figure 2 An example of arranging 8 microphones on the necklace is given.

[0056] The material and appearance design of the necklace are not limited in the present embodiment, and a flexible material or a rigid material can be used.

[0057] In an implementable embodiment, a high-performance AI chip can be mounted on a head-mounted device such as smart glasses to support real-time neural network inference, multi-microphone signal fusion processing, and user interaction, and a 3-5 mic ring planar array can be arranged. The necklace can use an integrated low-power MCU and a small DSP, and an 8-12 mic ring planar array can be arranged to be responsible for raw audio acquisition and front-end preprocessing.

[0058] Referring to Figure 3 , Figure 3 is a flowchart of a first embodiment of a sound source positioning method of the present application. In this embodiment, the sound source positioning method comprises steps S10-S30:

[0059] Step S10, obtaining a first microphone array signal collected by the first microphone array.

[0060] The microphone array signal collected by the first microphone array is referred to as the first microphone array signal for distinction. The first microphone array signal can include sound signals collected by each microphone, which are raw audio data represented as a time series and used for subsequent sound source positioning processing, such as estimating the direction of the sound source by analyzing the time difference or phase difference between the microphones.

[0061] Step S20, obtaining the relative pose between the necklace and the head-mounted device, and obtaining the sound source positioning auxiliary data sent by the necklace, wherein the sound source positioning auxiliary data is obtained based on the second microphone array signal collected by the second microphone array.

[0062] The relative pose refers to the relative position and attitude between the head-mounted device and the necklace, which can be represented by a translation vector and a rotation matrix. The translation vector describes the spatial displacement between the two, and the rotation matrix describes the difference in orientation between the two.

[0063] There are various ways to obtain the relative pose: for example, a default value can be set in advance, assuming that the relative pose of the head-mounted device and the necklace is fixed when the user wears them; or joint calibration through inertial measurement units (IMUs) and ultra-wideband (UWB) modules in the head-mounted device and the necklace. The joint calibration process can include: measuring the three-dimensional spatial distance and orientation between the head-mounted device and the necklace using UWB, obtaining the position coordinates of the necklace in the head-mounted device coordinate system based on the three-dimensional spatial distance and orientation, i.e., the translation vector; using the direction assistance information provided by UWB and the attitude of the head-mounted device and the necklace measured by IMU respectively, and calculating the relative rotation matrix required to convert the direction in the necklace coordinate system to the head-mounted device coordinate system through sensor fusion algorithm.

[0064] In an implementable embodiment, in the case of joint calibration by IMU and UWB, the IMU in the head-mounted device can monitor the intensity of user motion, for example, according to the acceleration amplitude to judge the intensity of motion. When the motion is intense, the relative pose change can be fast, and therefore the update frequency is increased. Conversely, the update frequency is reduced to save resources. At the same time, the power of the head-mounted device can also affect the update frequency: when the power is low, the update frequency is reduced to prolong the battery life. The accuracy of the relative pose is crucial for sound source positioning, and therefore regular calibration or adjustment according to environmental changes is necessary.

[0065] The microphone array signal collected by the second microphone array is referred to as a second microphone array signal for distinction. The second microphone array signal can include sound signals collected by each microphone.

[0066] In an implementable embodiment, the sound source positioning auxiliary data can be the second microphone array signal itself, i.e., the original audio data collected by the second microphone array in the necklace; or, the sound source positioning auxiliary data can be a sound source direction estimate value calculated based on the second microphone array signal, for example, a set of candidate directions.

[0067] In an implementable embodiment, Bluetooth 5.3 can be used to realize high-speed and low-latency data transmission between the head-mounted device and the necklace. The data transmission from the necklace to the head-mounted device adopts LC3 encoding of LE Audio, which is used to transmit sound source positioning auxiliary data (such as candidate direction set, TDOA matrix, timestamp, etc.), to realize the transmission of low-code-rate high-fidelity compressed voice files, and the data channel can use GATT+L2CAP custom service to transmit structured data. LE Audio refers to low-power audio, which is a new generation of Bluetooth audio technology standard formulated by Bluetooth Special Interest Group. LC3 encoding is a new type of audio encoder introduced in the LE Audio standard. GATT (Generic Attribute Profile) defines a structured data service model for creating custom services (such as “sound source positioning auxiliary data service”) and feature values to encapsulate and identify structured data to be transmitted (such as candidate direction set, TDOA matrix, etc.). L2CAP (Logical Link Control and Adaptation Protocol) is responsible for managing data channels between Bluetooth devices and providing reliable or unreliable packet transmission. By combining the use of GATT to define data structures and L2CAP to establish efficient data channels, reliable and orderly transmission of sound source positioning auxiliary data can be realized.

[0068] In an implementation, the necklace can run a voice activity detection (VAD) algorithm that analyzes the audio signals captured by the second microphone array in real time to determine whether there is human voice in the current environment. Only when the VAD algorithm detects voice activity, the necklace initiates the subsequent sound source positioning auxiliary data calculation process (such as generating a candidate direction set or a TDOA matrix), and sends these data to the headset; while in silence or only environmental noise, it remains dormant or low-power listening state. Since there is no need for continuous, high-computational-load sound source positioning data generation and wireless transmission during voiceless periods, the necklace's processor and communication module can be in low-power state most of the time, effectively extending the battery life of the necklace as a portable device. Secondly, it also reduces unnecessary wireless data transmission between the headset and the necklace.

[0069] The necklace end only performs simple preprocessing work, reducing power consumption; while the main data processing task is undertaken by the headset end, which not only ensures efficient operation of the system, but also prolongs the battery life.

[0070] Step S30, sound source positioning is performed according to the first microphone array signal, the sound source positioning auxiliary data and the relative pose to obtain a sound source positioning result.

[0071] The sound source positioning result can be the direction of the sound source finally positioned, for example, represented by a unit vector in three-dimensional space, or represented by spherical coordinates such as azimuth and elevation.

[0072] There are many specific implementations of sound source positioning according to the first microphone array signal, the sound source positioning auxiliary data and the relative pose, which are not limited in this embodiment. For example, in an implementation, the sound source positioning auxiliary data can be the second microphone array signal itself, and the relative pose can be a translation vector and a rotation matrix: using the relative pose, especially the rotation matrix, the spatial position coordinates of all microphones in the second microphone array are converted and unified into the coordinate system of the headset. In this way, all microphones in the first microphone array and the second microphone array form a "virtual microphone array" with a larger layout in the coordinate system of the headset; the first microphone array signal and the second microphone array signal are jointly regarded as a whole signal, and a sound source positioning algorithm (such as beamforming or TDOA method based on the whole virtual array) is used to calculate the sound source direction, thereby obtaining the sound source positioning result.

[0073] The sound source positioning auxiliary data can be used to obtain more accurate sound source positioning results, because the second microphone array provides additional spatial information, especially capturing sound from different positions, reducing the blind area or error when positioning alone with the headset, for example, the microphone array of the necklace covers the rear and vertical direction, making up for the deficiency of the headset. Fusion of multi-microphone array data can improve robustness, especially in noisy environments.

[0074] In an embodiment, the sound source positioning auxiliary data includes a candidate direction set obtained by sound source positioning based on the second microphone array signal. Each candidate direction in the candidate direction set can be obtained by applying a sound source positioning algorithm such as generalized cross-correlation phase transform (GCC-PHAT) to the second microphone array signal, which calculates a set of possible sound source directions based on the time difference or phase difference between microphones, each direction corresponding to an estimated value. The step S30 includes S301-S302:

[0075] Step S301, sound source positioning based on the first microphone array signal to obtain a preliminary positioning direction.

[0076] The preliminary positioning direction is calculated by applying a sound source positioning algorithm such as SRP-PHAT (Steered Response Power-Phase Transform) to the first microphone array signal. The first microphone array can generate multiple candidate directions, and the preliminary positioning direction can be the direction with the highest confidence, for example, selected by signal strength or consistency score, to ensure that a usable positioning result can be provided when using the headset alone.

[0077] Step S302, after converting the preliminary positioning direction and each candidate direction in the candidate direction set to the same coordinate system according to the relative pose, the preliminary positioning direction and each candidate direction are weighted and summed to obtain the sound source positioning result.

[0078] In an implementation, the relative pose can be represented by a translation vector and a rotation matrix, and when converting the candidate direction set to the head-mounted device coordinate system, the rotation matrix is applied to rotate each candidate direction so that all directions are represented in the head-mounted device coordinate system. Then, the preliminary positioning direction and each candidate direction are weighted and summed, for example, according to a pre-set weight: let the preliminary positioning direction be P, the candidate direction set be D = {d1, d2,..., dn}, the weight of P be w_p, and the weight of each candidate direction be w_di, and w_p + ∑w_di = 1, then the sound source positioning result R = w_p * P + ∑(w_di * di); the weight w_p can be set to be greater than ∑w_di, and the weight w_di of each candidate direction can be determined according to the confidence of each candidate direction, and the higher the confidence of the candidate direction, the greater the weight corresponding to the candidate direction. Setting the weight w_p of the preliminary positioning direction P to be greater than the sum of the weights of all candidate directions ∑w_di means that the system tends to trust the preliminary result calculated by the head-mounted device itself, which helps to maintain a low processing delay, because the data acquisition and processing of the head-mounted device are usually more direct and fast, avoiding the communication overhead or delay that may be introduced by waiting for the necklace auxiliary data, thereby ensuring the real-time response of the sound source positioning, which is particularly important in application scenarios that require fast interaction (such as AR rendering). High-confidence candidate directions often result from more reliable signal processing results (such as direction estimates calculated by TDOA algorithms), which can compensate for the positioning deficiencies of the head-mounted device in certain directions (such as the rear or vertical direction), thereby calibrating and refining the preliminary positioning direction and improving the overall positioning accuracy.

[0079] In other implementations, the weights can also be dynamically adjusted according to the consistency between the candidate directions and the preliminary positioning direction to improve accuracy.

[0080] By combining the candidate direction set provided by the second microphone array of the necklace, the head-mounted device can calibrate the preliminary positioning direction. Since the microphone array of the necklace has a better spatial distribution, such as coverage in the rear and vertical directions, the candidate direction set can compensate for the deficiencies of the head-mounted device alone, thereby improving the overall accuracy of the sound source positioning, especially in the rear and vertical directions.

[0081] In an implementation, the sound source positioning method further comprises S40-S50:

[0082] Step S40, in the case where the preliminary positioning direction is obtained and the sound source positioning auxiliary data sent by the necklace has not been received, outputting the preliminary positioning direction as the sound source positioning result.

[0083] In a specific application, there can be a situation where the headset has obtained a preliminary orientation direction based on the first microphone array, while the necklace has not sent, sent delayed or failed to send the sound source positioning auxiliary data, for example due to unstable communication connection, insufficient necklace power or network delay. Outputting the preliminary orientation direction as the sound source positioning result ensures that the basic function is available and the real-time performance is maintained.

[0084] Step S50, in the case where the necklace sends the sound source positioning auxiliary data, the preliminary orientation direction and each candidate direction are weighted and summed to obtain the sound source positioning result output.

[0085] Outputting the sound source positioning result is to provide it to the subsequent processing module, i.e. as a sound direction parameter for subsequent processing such as AR rendering, but the result is more accurate at this time. This case-based output mechanism balances the real-time performance and accuracy of sound source positioning. When the necklace data is not available, the preliminary result is output in time to ensure real-time performance; when the data is available, the fusion result is output to improve accuracy, thereby optimizing user experience and avoiding delay caused by waiting for data. In addition, it can provide a gradual and clear process from preliminary rapid positioning to final accurate calibration, not only improving the accuracy of sound source positioning, but also enhancing the user's immersion and use experience. Users can feel the process of sound source from vague to gradually clear, and this progressive accurate positioning improves the authenticity and naturalness of interaction.

[0086] In a feasible implementation, the sound source positioning method further comprises step S60:

[0087] Step S60, obtain the clock offset between the headset and the necklace.

[0088] The clock offset is obtained through a two-way timestamp synchronization mechanism: the headset sends a synchronization request (including a local timestamp) at time t1, the necklace receives and records the timestamp at t2, then sends a response (including t2 and t3) at t3, and the headset receives the response at t4. Calculate the communication round-trip delay: delay=(t4-t1)-(t3-t2), and the clock offset: offset=[(t2-delay / 2)-t1]. The system can perform timestamp alignment periodically, for example, every 5 seconds, to compensate for the error caused by the clock and ensure data time synchronization.

[0089] The step S30 comprises:

[0090] In step S303, data time alignment is performed according to the collection time of the first microphone array signal, the timestamp carried in the sound source positioning auxiliary data, and the clock offset, and sound source positioning is performed based on the aligned data and the relative pose to obtain a sound source positioning result.

[0091] The data time alignment process can include: adjusting the time axis of the sound source positioning auxiliary data according to the collection time T1 of the first microphone array signal, the timestamp T2 carried in the sound source positioning auxiliary data, and the clock offset offset, for example, correcting T2 to T2' = T2 - offset to align with T1. Then, sound source positioning is performed using the aligned data and the relative pose, for example, calculating the sound source direction through a TDOA or beamforming algorithm. For example, in a feasible implementation, when the sound source positioning auxiliary data is the second microphone array signal itself, the aligned first microphone array signal and the second microphone array signal can be combined, and the sound source direction can be calculated after being converted to the same coordinate system using the relative pose. For another example, in a feasible implementation, when the sound source positioning auxiliary data includes a candidate direction set and a second TDOA matrix calculated based on the second microphone array signal, the sound source positioning auxiliary data can further include a timestamp of the candidate direction set and the second TDOA matrix, which is determined according to the collection time of the second microphone array signal; the head-mounted device can also determine a timestamp of a preliminary positioning direction and a first TDOA matrix calculated based on the first microphone array signal according to the collection time of the first microphone array signal, and align the candidate direction set and the second TDOA matrix with the preliminary positioning direction and the first TDOA matrix according to the timestamp of the candidate direction set and the second TDOA matrix and the timestamp of the preliminary positioning direction and the first TDOA matrix. Subsequent sound source positioning is performed based on the aligned candidate direction set, second TDOA matrix, preliminary positioning direction, and first TDOA matrix.

[0092] In this embodiment, a distributed computing architecture is adopted, and the head-mounted device and the necklace each process the data collected by itself, and then transmit the data for comprehensive analysis through a wireless manner. This manner not only guarantees real-time response speed, but also can complete complex sound source positioning calculation tasks under limited hardware resources. Due to the size limitation, the head-mounted device has a small number of microphones and a layout close to a ring shape, resulting in ambiguous vertical direction positioning and inaccurate backward positioning. The second microphone array of the necklace makes up for these defects by providing rear microphones and vertical height differences. Combined with the relative pose and the sound source positioning auxiliary data, the head-mounted device can obtain more comprehensive sound source information, and significantly improve the positioning accuracy, especially in the rear and up-down directions.

[0093] Based on the first embodiment, the second embodiment of the sound source positioning method is proposed. In this embodiment, the same or similar contents as the first embodiment can be referred to the above description, and will not be repeated hereinafter. In this embodiment, the sound source positioning auxiliary data further includes a second TDOA matrix calculated based on the second microphone array signal, and the second TDOA matrix includes the corresponding measured time difference of arrival of each microphone pair in the second microphone array. The second TDOA matrix is a matrix composed of the measured time difference of arrival between each pair of microphones in the second microphone array, and the calculation method is to estimate the time difference of arrival of each pair of microphones by cross-correlation analysis on the second microphone array signal, for example, using a generalized cross-correlation function to calculate the time difference, and organizing it into a matrix form.

[0094] The step S302 of weighting and summing the preliminary positioning direction and each candidate direction to obtain the sound source positioning result includes S3021-S3023:

[0095] In step S3021, for each direction pair composed of the preliminary positioning direction and each candidate direction, a consistency score of the direction pair is calculated according to the second TDOA matrix and a first TDOA matrix calculated based on the first microphone array signal, wherein the first TDOA matrix includes the corresponding measured time difference of arrival of each microphone pair in the first microphone array.

[0096] The first TDOA matrix is a matrix of measured time difference of arrival corresponding to each microphone pair in the first microphone array, and the calculation method is the same as the second TDOA matrix, which is obtained by cross-correlation analysis. The consistency score is calculated by comparing the theoretical time difference of the first TDOA matrix and the second TDOA matrix under a given direction pair, for example, using a multi-view TDOA fusion algorithm: let the direction pair be (P, di), where P is the preliminary positioning direction and di is the candidate direction, the theoretical time difference is calculated based on the sound wave propagation model, and the consistency score s_i=exp(-Σ|Δt_actual-Δt_theoretical|^2), where Δt_actual is the measured time difference, Δt_theoretical is the theoretical time difference, and the higher the score, the better the consistency of the direction pair in the two arrays, i.e. the candidate direction is more reliable.

[0097] In step S3022, the consistency score of the direction pair is taken as the weight of the candidate direction in the direction pair, and each candidate direction is weighted and summed to obtain a first fusion direction.

[0098] Let the candidate direction set be D = {d1, d2,..., dn}, and the consistency score of each candidate direction di be s_i, then the first fusion direction F = Σ(s_i * d_i) / Σs_i, i.e. the weighted average direction.

[0099] Step S3023, weighted sum of the first fusion direction and the preliminary positioning direction to obtain the sound source positioning result.

[0100] Let the preliminary positioning direction be P, the first fusion direction be F, and the weight be w, then the sound source positioning result R = w * F + (1-w) * P.

[0101] Through the second TDOA matrix and the candidate direction set, the consistency score is calculated to weight the fusion, so that the sound source positioning result is more reliable. This calibrates the preliminary positioning direction, especially in backward positioning, which takes advantage of the necklace data to improve accuracy, because the microphone array of the necklace covers better in the backward direction.

[0102] In a feasible implementation, the step S3023 includes S30231-S30232:

[0103] Step S30231, if the first fusion direction and the preliminary positioning direction both fall within the target direction range, then the first weight is used as the weight corresponding to the first fusion direction, and the first fusion direction and the preliminary positioning direction are weighted and summed to obtain the sound source positioning result, wherein the target direction range is a conical or sector region around the target direction in the coordinate system of the head-mounted device, and is a subset of the full direction range of sound source positioning of the head-mounted device, and the target direction is directed to the rear of the wearer's head when the head-mounted device is worn.

[0104] The target direction is a specific direction in the coordinate system of the head-mounted device, which is directed to the rear of the wearer's head when the head-mounted device is worn, and the target direction range is a conical region centered on the direction, with an angle range such as ±30 degrees. In the case where the first fusion direction and the preliminary positioning direction both fall within the target direction range, the weight of the preliminary positioning direction is 1-w1 when weighted and summed, where w1 is the first weight, for example w1=0.7, indicating that the necklace data is more trusted in the accuracy of the rear direction.

[0105] Step S30232, if at least one of the first fusion direction and the preliminary positioning direction does not fall within the target direction range, then the second weight is used as the weight corresponding to the first fusion direction, and the first fusion direction and the preliminary positioning direction are weighted and summed to obtain the sound source positioning result, wherein the second weight is less than the first weight.

[0106] In the case that at least one of the first fused direction and the preliminary orientation direction does not fall within the target direction range, the weight of the preliminary orientation direction in the weighted summation is 1-w2, where w2 is a second weight, and w2w1, for example w2=0.3. The second weight is set smaller than the first weight, because the second microphone array on the necklace has a microphone at the back of the neck, so when the preliminary orientation direction and the first fused direction of the positioning are both from the back of the wearer's head, the first fused direction is a direction determined with the aid of data of the necklace, and increasing its weight can more accurately compensate for the problem of inaccurate positioning of the head-mounted device in the back direction; and when the direction does not fall within the back direction range, the data of the head-mounted device can be more reliable, so the weight of the necklace data is reduced to give priority to the data of the head-mounted device.

[0107] Based on the first and / or second embodiments described above, a third embodiment of the sound source positioning method of the present application is proposed. In this embodiment, the same or similar contents as the first and second embodiments described above can be referred to the above introduction, and will not be repeated hereinafter. In this embodiment, the step S302 of weighting and summing the preliminary orientation direction and each candidate direction to obtain a sound source positioning result includes S3024-S3027:

[0108] Step S3024, weighting and summing the preliminary orientation direction and each candidate direction to obtain a preliminary fused direction.

[0109] Compared with the direction obtained by weighting and summing the preliminary orientation direction and each candidate direction in the foregoing embodiments as the final sound source positioning result, in this embodiment, the direction is taken as the preliminary fused direction, and on the basis of the preliminary fused direction, further correction is made through the deep learning model.

[0110] Step S3025, calculating a third TDOA matrix according to the preliminary fused direction, wherein the third TDOA matrix includes theoretical time difference of arrival of each target microphone pair assuming that the sound source comes from the preliminary fused direction, and the each target microphone pair is a microphone pair composed of a microphone in the first microphone array and a microphone in the second microphone array.

[0111] The calculation of the third TDOA matrix is based on the preliminary fusion direction as the hypothesized sound source direction, using the sound wave propagation model and the geometric relationship of the microphone pairs. Specifically, let the preliminary fusion direction be a unit vector d, and the relative pose between the headset and the necklace is known (e.g., represented by a translation vector and a rotation matrix), convert all microphone positions in the first microphone array and the second microphone array to the same coordinate system (e.g., the headset coordinate system). For each target microphone pair (i.e., each pair of a microphone in the first microphone array and a microphone in the second microphone array), calculate the propagation path difference of the sound wave from the hypothesized sound source direction to the two microphones: path difference Δl = d · (p_i - p_j), where p_i and p_j are the position vectors of the two microphones, and · represents the dot product. Then, the theoretical time difference of arrival Δt_ij = Δl / c, where c is the speed of sound (e.g., 340 m / s). The third TDOA matrix is composed of all these Δt_ij, forming a matrix or vector representing the expected time difference (theoretical time difference of arrival) between the microphone pairs under the hypothesized sound source direction.

[0112] At step S3026, the spectral feature, spatial filter response map, and ambient noise spectrum are calculated based on the first microphone array signal.

[0113] The spectral feature is obtained by performing frequency domain analysis on the first microphone array signal, such as using short-time Fourier transform (STFT) to calculate the power spectrum or Mel-frequency cepstral coefficients (MFCC) of the signal, to capture the spectral characteristics of the sound source, such as frequency distribution and timbre information. The spatial filter response map is generated by a beamforming algorithm, such as using a delay-and-sum beamformer, to calculate the spatial response based on the geometric layout of the first microphone array and the preliminary fusion direction, forming a map representing the signal intensity in different directions, highlighting the energy distribution in the sound source direction. The ambient noise spectrum is obtained by statistically analyzing the spectrum of the first microphone array signal during sound source-free activities or silent periods, such as calculating the average power spectrum of the background noise, to characterize the environmental noise characteristics for noise suppression in subsequent processing.

[0114] At step S3027, the third TDOA matrix, the spectral feature, the spatial filter response map, and the ambient noise spectrum are input into a pre-trained deep learning model for processing to obtain the sound source localization result.

[0115] The deep learning model includes Conv-TasNet and Spatial Transformer Network. Conv-TasNet is an end-to-end speech separation network, but is used for feature extraction here: the first microphone array signal is input into Conv-TasNet, the original output layer is removed, the encoder and separation module are retained, and a high-dimensional signal feature vector is output, which captures the time-frequency and spatial information of the signal. Then, this signal feature is spliced with the third TDOA matrix (flattened into a vector), the spectral feature (such as the MFCC vector), the spatial filter response map (flattened into a vector), and the ambient noise spectrum (flattened into a vector) to form a comprehensive feature vector. The spliced feature vector is input into the Spatial Transformer Network, which learns spatial transformation parameters through convolutional layers and fully connected layers, and finally outputs the sound source direction (such as a unit vector or spherical coordinates). The Spatial Transformer Network is improved here into a regression network, which outputs continuous direction values instead of traditional classification outputs, to adapt to the continuous direction estimation of sound source positioning. Specifically, the improvements to Conv-TasNet include removing the output layer and adjusting the input dimension to accept multi-modal features; the improvements to the Spatial Transformer Network include adding a regression head (such as a fully connected layer) to output the direction vector.

[0116] The deep learning model can be pre-trained and deployed in the head-mounted device. The deep learning model can be trained through a supervised learning method. The training data can be constructed by setting multiple sound source directions, playing specific audio signals from various sound source directions, and collecting microphone array signals under different sound source direction conditions through the first microphone array and the second microphone array as samples. For each sample, the corresponding third TDOA matrix, spectral feature, spatial filter response map, and ambient noise spectrum are generated as input features of the model in the same way as in the foregoing embodiments; the sound source direction corresponding to the sample (i.e., the real sound source direction) is used as the training label. The training goal is to optimize the loss function such as mean square error to make the model's predicted sound source direction as close as possible to the real label, so as to learn the complex mapping relationship from multi-modal input features to continuous sound source direction.

[0117] In this embodiment, in combination with the deep learning model, the third TDOA matrix, the spectral feature, the spatial filtering response map and the ambient noise spectrum can further calibrate the preliminary fusion direction, so as to obtain more accurate sound source positioning results. This is because the third TDOA matrix provides theoretical constraints based on geometric relationships, the spectral feature and the spatial filtering response map capture the frequency domain and spatial characteristics of the signal, the ambient noise spectrum helps to denoise, and the deep learning model fuses these multi-modal features through end-to-end learning, can model the nonlinear relationship (such as multipath effect and noise interference) in the complex acoustic environment, compensate for possible errors in the preliminary fusion direction, and thus improve the robustness and accuracy of positioning, especially when the sound source direction changes or the environment is noisy.

[0118] Based on the above first, second and / or third embodiments, a fourth embodiment of the sound source positioning method of the present application is proposed. In this embodiment, the same or similar contents as the above first, second and third embodiments can be referred to the above introduction, and will not be repeated hereinafter. In this embodiment, the execution subject of the sound source positioning method is a necklace. The necklace is provided with a second microphone array, and the head-mounted device in communication connection with the necklace is provided with a first microphone array. Referring to Figure 4 , Figure 4 The flowchart of the first embodiment of the sound source positioning method of the present application is shown in the figure. The sound source positioning method comprises steps A10-A30:

[0119] Step A10, obtaining the second microphone array signal collected by the second microphone array.

[0120] Step A20, obtaining sound source positioning auxiliary data based on the second microphone array signal.

[0121] Step A30, sending the sound source positioning auxiliary data to the head-mounted device, so that the head-mounted device performs sound source positioning according to the first microphone array signal, the sound source positioning auxiliary data and the relative pose between the head-mounted device and the necklace, to obtain a sound source positioning result, wherein the first microphone array signal is collected based on the first microphone array.

[0122] The explanation and specific implementation of steps A10-A30 in this embodiment can refer to the explanation and specific implementation of steps S10-S30 described above, which will not be repeated here.

[0123] The electronic device provided in the embodiment of the present application. The electronic device can be a head-mounted device or a necklace. The electronic device comprises at least one processor and a memory in communication connection with the at least one processor. The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the sound source positioning method in the above embodiments.

[0124] Reference will now be made to the following description Figure 5 which shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present application. Figure 5 The electronic device shown is merely an example and should not bring any limitation to the functions and use range of the embodiments of the present application.

[0125] As shown in Figure 5 , the electronic device can include a processing device 1001 (e.g. a central processor, a graphics processor, etc.) which can perform various appropriate actions and processes according to programs stored in a read-only memory 1002 or loaded from a storage device 1003 into a random access memory 1004. Various programs and data required for the operation of the electronic device are also stored in the random access memory 1004. The processing device 1001, the read-only memory 1002 and the random access memory 1004 are connected to each other through a bus 1005. An input / output interface 1006 is also connected to the bus. Generally, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; the storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although the electronic device with various systems is shown in the figure, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be alternatively implemented or provided.

[0126] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program codes for executing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network through the communication device, or installed from the storage device 1003, or installed from the read-only memory 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the methods of the embodiments disclosed in the present application are performed.

[0127] Compared with the prior art, the electronic device provided by the embodiments of the present application has the same beneficial effects as the sound source positioning method provided by the above-mentioned embodiments, and other technical features in the electronic device are the same as the features disclosed in the previous embodiment method, which will not be repeated here.

[0128] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0129] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, which are used to execute the sound source localization method in the above embodiments.

[0130] The computer-readable storage medium provided in this application embodiment may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0131] The aforementioned computer-readable storage medium may be included in an electronic device or may exist independently without being assembled into an electronic device.

[0132] The aforementioned computer-readable storage medium carries one or more programs, which, when executed by an electronic device, cause the electronic device to perform the functions defined in the methods of the embodiments disclosed in this application.

[0133] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0134] The flow diagrams and the block diagrams in the drawings are illustrations of architectures, functionalities, and operations of possible implementations of systems, methods, and computer program products according to various embodiments of the present application. In this regard, each block in the flow diagrams or block diagrams can represent a module, a segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may

[0135] The modules involved in the embodiments of the present application can be implemented in the form of software or in the form of hardware. In some cases, the name of the module does not constitute a limitation on the module itself.

[0136] The readable storage medium provided by the embodiments of the present application is a computer readable storage medium, which stores computer readable program instructions (i.e. computer programs) for executing the above-mentioned sound source positioning method. Compared with the prior art, the computer readable storage medium provided by the embodiments of the present application has the same beneficial effects as the sound source positioning method provided by the above-mentioned embodiments, which will not be described here.

[0137] The embodiment of the present application further provides a computer program product comprising a computer program, which, when executed by a processor, implements the steps of the sound source positioning method as described above.

[0138] Compared with the prior art, the computer program product provided by the embodiment of the present application has the same beneficial effects as the sound source positioning method provided by the above-mentioned embodiment, which will not be repeated here.

[0139] The above only describes some embodiments of the present application, and does not limit the patent scope of the present application. Any equivalent structural transformation made by using the content of the present application specification and drawings, or direct / indirect application in other related technical fields is included in the patent protection scope of the present application.

Claims

1. A method for locating a sound source, characterized in that, The sound source localization method is applied to a head-mounted device, which is equipped with a first microphone array, and a necklace that establishes a communication connection with the head-mounted device is equipped with a second microphone array. The sound source localization method includes: Acquire the first microphone array signal collected by the first microphone array; The relative pose between the necklace and the head-mounted device is obtained, as well as the sound source localization assistance data sent by the necklace is obtained, wherein the sound source localization assistance data is obtained based on the second microphone array signal collected by the second microphone array; The sound source is located based on the first microphone array signal, the sound source localization auxiliary data, and the relative pose, and the sound source localization result is obtained.

2. The sound source localization method as described in claim 1, characterized in that, The sound source localization auxiliary data includes a candidate direction set obtained by sound source localization based on the second microphone array signal; The step of performing sound source localization based on the first microphone array signal, the sound source localization auxiliary data, and the relative pose to obtain the sound source localization result includes: Based on the signal from the first microphone array, the sound source is located to obtain a preliminary positioning direction; After transforming the preliminary positioning direction and each candidate direction in the candidate direction set to the same coordinate system based on the relative pose, the preliminary positioning direction and each candidate direction are weighted and summed to obtain the sound source localization result.

3. The sound source localization method as described in claim 2, characterized in that, The sound source localization auxiliary data also includes a second TDOA matrix calculated based on the signal of the second microphone array, the second TDOA matrix including the measured time difference of arrival for each microphone pair in the second microphone array; The step of obtaining the sound source localization result by weighted summation of the preliminary localization direction and each of the candidate directions includes: For each direction pair formed by combining the initial positioning direction with each candidate direction, a consistency score for the direction pair is calculated based on the second TDOA matrix and the first TDOA matrix calculated based on the signal of the first microphone array. The first TDOA matrix includes the measured time difference of arrival corresponding to each microphone pair in the first microphone array. The consistency score of the direction pair is used as the weight of the candidate direction in the direction pair, and the candidate directions are weighted and summed to obtain the first fusion direction; The sound source localization result is obtained by weighted summation of the first fusion direction and the preliminary positioning direction.

4. The sound source localization method as described in claim 3, characterized in that, The step of weighted summing of the first fusion direction and the preliminary positioning direction to obtain the sound source localization result includes: If both the first fusion direction and the preliminary positioning direction fall within the target direction range, then the first weight is used as the weight corresponding to the first fusion direction, and the first fusion direction and the preliminary positioning direction are weighted and summed to obtain the sound source localization result. The target direction range is a cone-shaped or fan-shaped region around the target direction in the coordinate system of the head-mounted device, and is a subset of the full-directional range of the sound source localization of the head-mounted device. The target direction faces the back of the wearer's head when the head-mounted device is worn. If at least one of the first fusion direction and the preliminary positioning direction does not fall within the target direction range, then the second weight is used as the weight corresponding to the first fusion direction, and the first fusion direction and the preliminary positioning direction are weighted and summed to obtain the sound source localization result, wherein the second weight is less than the first weight.

5. The sound source localization method as described in claim 2, characterized in that, The step of obtaining the sound source localization result by weighted summation of the preliminary localization direction and each of the candidate directions includes: The preliminary fusion direction is obtained by weighted summation of the preliminary positioning direction and each of the candidate directions; The third TDOA matrix is ​​calculated based on the initial fusion direction, wherein the third TDOA matrix includes the theoretical arrival time difference of each target microphone pair under the assumption that the sound source comes from the initial fusion direction, and each target microphone pair is a microphone pair composed of two microphones in the first microphone array and two microphones in the second microphone array; The spectral characteristics, spatial filter response map, and ambient noise spectrum are calculated based on the signals from the first microphone array. The third TDOA matrix, the spectral features, the spatial filter response map, and the environmental noise spectrum are input into a preset deep learning model for processing to obtain the sound source localization result.

6. The sound source localization method as described in claim 2, characterized in that, The sound source localization method further includes: If the preliminary positioning direction is obtained, but the sound source positioning auxiliary data sent by the necklace has not yet been received, the preliminary positioning direction is output as the sound source positioning result. Upon receiving the sound source localization assistance data sent by the necklace, the sound source localization result is output by weighted summation of the preliminary localization direction and each of the candidate directions.

7. The sound source localization method according to any one of claims 1 to 6, characterized in that, The sound source localization method further includes: Obtain the clock offset from the necklace; The step of performing sound source localization based on the first microphone array signal, the sound source localization auxiliary data, and the relative pose to obtain the sound source localization result includes: Data time alignment is performed based on the acquisition time of the first microphone array signal, the timestamp carried in the sound source localization auxiliary data, and the clock offset. Sound source localization is then performed based on the aligned data and the relative pose to obtain the sound source localization result.

8. A method for locating a sound source, characterized in that, The sound source localization method is applied to a necklace, which is equipped with a second microphone array, and a head-mounted device that establishes a communication connection with the necklace is equipped with a first microphone array. The sound source localization method includes: Acquire the second microphone array signal collected by the second microphone array; Sound source localization auxiliary data is obtained based on the signals from the second microphone array; The sound source localization auxiliary data is sent to the head-mounted device so that the head-mounted device can perform sound source localization based on the first microphone array signal, the sound source localization auxiliary data, and the relative pose between the head-mounted device and the necklace, and obtain the sound source localization result, wherein the first microphone array signal is obtained based on the first microphone array.

9. An electronic device, characterized in that, The electronic device further includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the sound source localization method as described in any one of claims 1 to 8.

10. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the sound source localization method as described in any one of claims 1 to 8.

Images