Sound source positioning method, device and equipment and computer readable storage medium

Abstract

The application discloses a sound source positioning method, device and equipment and a computer readable storage medium. The sound source positioning method comprises the following steps: collecting a scene image and an audio signal in an environment, and detecting all human body images included in the scene image; determining a current position corresponding to each human body image, and determining a sound source angle of the audio signal; detecting a target position matched with the sound source angle in all the current positions, and outputting a human body image corresponding to the target position. The application improves the accuracy of sound source positioning.

Description

Technical Field

[0001] This application relates to the field of positioning technology, and in particular to a sound source positioning method, apparatus, device, and computer-readable storage medium. Background Technology

[0002] Sound source localization is a technique that uses sound waves emitted by a sound source to determine its location. Sound source localization technology contributes to the development of machine intelligence.

[0003] Traditional sound source localization technology uses a microphone array to receive sound source signals. Based on the correlation between the sound source signals received by each microphone, the time delay difference between the received sound source signals of each microphone is estimated. Then, the distance difference between the sound source and different microphones is calculated based on the time delay difference and the speed of sound. Then, a system of equations is established and solved based on each distance difference and the distance between each microphone and the sound source to calculate the coordinates of the sound source's location.

[0004] However, the sound source signals received by the microphone often include signals other than those emitted by the sound source, such as reverberation. This leads to significant errors in the estimated time delay difference based on the correlation of the various sound source signals, resulting in inaccurate sound source localization. Summary of the Invention

[0005] The main objective of this application is to provide a sound source localization method, apparatus, device, and computer-readable storage medium, aiming to solve the technical problem of how to improve the accuracy of sound source localization.

[0006] To achieve the above objectives, this application provides a sound source localization method, which includes the following steps:

[0007] Acquire scene images and audio signals in the environment, and detect all human images included in the scene images;

[0008] Determine the current position corresponding to each of the human body images, and determine the sound source angle of the audio signal;

[0009] Detect the target position that matches the angle of the sound source among all the current positions, and output the human body image corresponding to the target position.

[0010] Optionally, the step of detecting the target position that matches the sound source angle among all the current positions includes:

[0011] The calibration position corresponding to the sound source angle is obtained from the first preset calibration relationship, wherein the first preset calibration relationship includes the correspondence between different sound source angles and different calibration positions in the scene image;

[0012] Iterate through all the current positions in sequence and check whether the current position matched the calibration position.

[0013] After a successful match, the current position is used as the target position.

[0014] Optionally, the step of detecting whether the current position being traversed matches the calibrated position includes:

[0015] Determine the distance from the human image corresponding to the current position to the preset camera;

[0016] The target location range is determined based on the calibrated position and the distance, and it is detected whether the current traversed position is within the target location range.

[0017] After the current position is traversed within the target position range, it is determined that the current position traversed is successfully matched with the calibrated position.

[0018] Optionally, the step of detecting whether the current position being traversed is within the range of the target position includes:

[0019] Determine the first horizontal coordinate of the current position in the preset coordinate system, and detect whether the coordinate value of the first horizontal coordinate is within the range of the target position;

[0020] If the coordinate value of the first horizontal coordinate is within the target location range, then it is determined that the current traversed position is within the target location range.

[0021] Optionally, the step of determining the target location range based on the calibrated location and the distance includes:

[0022] The position error value corresponding to the distance is obtained from the second preset calibration relationship, wherein the second preset calibration relationship includes the correspondence between different distances from the human image to the preset camera and different position error values;

[0023] Determine the second horizontal coordinate of the calibration position in the preset coordinate system, and determine the target position range based on the second horizontal coordinate and the position error value.

[0024] Optionally, the step of determining the target position range based on the second horizontal coordinate and the position error value includes:

[0025] The sum of the second horizontal coordinate value and the position error value is used as the upper limit position, and the difference between the second horizontal coordinate value and the position error value is used as the lower limit position.

[0026] The range between the upper limit position and the lower limit position is taken as the target position range.

[0027] Optionally, the step of determining the sound source angle of the audio signal includes:

[0028] The time difference information of the audio signal acquired by multiple microphones in a preset microphone array is statistically analyzed, and the time difference information is input into a pre-trained sound source localization model to output the sound source angle.

[0029] Furthermore, to achieve the above objectives, this application also provides a sound source localization device, the sound source localization device comprising:

[0030] The acquisition module is used to acquire scene images and audio signals in the environment, and to detect all human images included in the scene images;

[0031] The determination module is used to determine the current position corresponding to each of the human body images and to determine the sound source angle of the audio signal;

[0032] The output module is used to detect the target position that matches the angle of the sound source among all the current positions, and output the human body image corresponding to the target position.

[0033] In addition, to achieve the above objectives, this application also provides a sound source localization device, which includes: a memory, a processor, and a sound source localization program stored in the memory; the processor is used to execute the sound source localization program to implement the steps of the above-described sound source localization method.

[0034] In addition, to achieve the above objectives, this application also provides a computer-readable storage medium storing one or more programs, which can be executed by one or more processors to implement the steps of the above-described sound source localization method.

[0035] This application acquires scene images and audio signals from the environment to be located, performs human detection on the scene images to detect all human images included in the scene images, and determines the current position of each human image in the environment to be located. Combined with the sound source angle of the audio signal, it automatically matches all target positions that match the sound source angle. The human image corresponding to the target position is output as the final sound source localization result to the display device for display, thus achieving sound source localization. This avoids the problem of calculating the position coordinates of the sound source solely based on the audio signals received by the microphone array. Since the audio signals received by the microphone include interference signals, the sound source position calculation based on the audio signals has a large error, leading to inaccurate sound source localization. This application improves the accuracy of sound source localization. Furthermore, by combining audio signals with human images to achieve sound source localization, it does not require face recognition or recognition of the speaker's voiceprint features. It only requires simple human image detection, which requires low computing power and low requirements for computer hardware capabilities, thus reducing the cost of using sound source localization. Attached Figure Description

[0036] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings.

[0037] Figure 1 This is a schematic diagram of the terminal / device structure of the hardware operating environment involved in the embodiments of this application;

[0038] Figure 2 This is a flowchart illustrating the first embodiment of the sound source localization method of this application;

[0039] Figure 3 This is an illustrative schematic diagram of the positioning device for the sound source localization method of this application;

[0040] Figure 4 This is an illustrative schematic diagram of the sound source range in the sound source localization method of this application;

[0041] Figure 5 This is an illustrative diagram showing the human body image and sound source range map in the sound source localization method of this application;

[0042] Figure 6 This is a schematic diagram of the device module of the sound source localization device of this application. Detailed Implementation

[0043] It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application.

[0044] Reference Figure 1 , Figure 1 This is a schematic diagram of the sound source localization device structure in the hardware operating environment involved in the embodiments of this application.

[0045] like Figure 1 As shown, the sound source localization device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen and an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be a high-speed random access memory (RAM) or a stable non-volatile memory (NVM), such as a disk drive. The memory 1005 may also optionally be a storage device independent of the aforementioned processor 1001.

[0046] Those skilled in the art will understand that Figure 1 The structure shown does not constitute a limitation on the sound source localization device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0047] like Figure 1 As shown, the memory 1005, as a storage medium, may include an operating system, a data storage module, a network communication module, a user interface module, and a sound source localization program. Figure 1 In the sound source localization device shown, the network interface 1004 is mainly used for sound source localization with other devices; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the sound source localization device of this application can be set in the sound source localization device, and the sound source localization device calls the sound source localization program stored in the memory 1005 through the processor 1001 and executes the sound source localization method provided in the embodiment of this application.

[0048] This application proposes a sound source localization method. In the first embodiment of the sound source localization method of this application, referring to... Figure 2 Sound source localization methods include:

[0049] Step S10: Acquire scene images and audio signals in the environment, and detect all human images included in the scene images;

[0050] Currently, sound source localization is achieved by establishing a one-to-one mapping between array microphones and preset camera positions on the elliptical conference seats. When the sound intensity of one array microphone is strongest, the corresponding preset position is located, and the camera pan-tilt unit is driven to move to that microphone position. However, this method is not flexible enough and can only be used for fixed seats. For different conference rooms, the preset camera positions need to be re-acquired, and the mapping between the preset positions and the microphones on each seat needs to be re-established, resulting in high labor costs.

[0051] Another approach involves using multiple array microphones, both vertically and horizontally, to capture digital audio signals. The DOA (Direction of Arrival) algorithm is then applied to these digital signals to determine the angle between the sound source's location and the array microphone lines. While audio algorithms are used in both the horizontal and vertical directions, both are susceptible to noise interference. Currently, noise reduction algorithms can be employed to denoise the audio signals captured by the microphones. However, the denoised digital signal can impair the sound source signal, resulting in inaccurate localization in both directions and low positioning accuracy.

[0052] This involves combining facial recognition and speech recognition to pinpoint the location of a speaker's voice, using facial recognition to correct inaccurate audio localization. However, the algorithms used are quite complex. Facial recognition algorithms require face detection before identifying the person. Speech recognition algorithms calculate the angle of the sound source and identify the voiceprint. Both algorithms require significant hardware support; ordinary low-cost embedded hardware lacks sufficient computing power, resulting in high hardware costs. In many scenarios, this type of sound source localization device is unsuitable because the speaker is not authorized. Furthermore, facial detection in facial recognition is less robust; accuracy is low when image quality is poor, leading to low localization precision.

[0053] Therefore, in this embodiment, to avoid the above-mentioned defects and improve the accuracy of sound source localization, the problem of inaccurate sound source localization by detecting the position of the human body in the image is corrected by using only the microphone array. At the same time, compared with face recognition, it only needs to detect the human body image and does not need to recognize the face, so the required computing power is lower and can be supported by ordinary embedded hardware. Compared with face detection, human body detection is more robust and has a longer detection distance, making it suitable for more scenarios. Compared with voiceprint recognition, it only needs to detect the angle of the sound source and does not need to recognize the voiceprint features of the speaker, so the required computing power is lower and can be supported by ordinary embedded hardware.

[0054] In this embodiment, the first step is to acquire scene images and audio signals from the environment to be located. Both scene images and audio signals can be acquired using conventional techniques in the art. Scene images can be acquired using a camera, and audio signals can be acquired using a microphone array. The microphone array consists of an array of at least one microphone, and the audio signal includes both the sound source signal and interference signals. Furthermore, in this embodiment, the positioning results of the microphone array are corrected by the position of the human body in the scene image. Therefore, a human detection algorithm is needed to detect all human images included in the scene image. This human detection algorithm can be a commonly used human image detection algorithm, such as runtime-based detection algorithms and machine learning-based detection algorithms.

[0055] In this embodiment, a positioning device is provided that includes at least a microphone array and a camera. The positioning device is placed in the environment to be located. Audio signals are collected by the microphone array to determine the direction of the audio source, performing preliminary positioning of the audio signal. Scene images are captured by the camera, and human images included in the scene images are detected to correct the positioning results of the microphone array. The position of the human images is matched with the preliminary positioning results of the audio signal obtained by the microphone array to determine the target user of the sound source signal. For example, referring to… Figure 3 A positioning device includes a camera C1 and a microphone array. The camera C1 does not have a pan-tilt head, has a wide-angle lens with a horizontal FOV (field of angle) of 120 degrees, and can capture 4K images. The microphone array consists of two symmetrical microphones m1 and m2 for sound pickup. The distance between the two microphones can be close or far, depending on the actual positioning situation. Here, 10 centimeters is preferred. The 4K panoramic image is captured by the C1 camera, and the 4K image is downsized to a 4CIF image to obtain the panoramic image D1, which is the scene image. A human detection algorithm is used to detect human bodies in the panoramic image D1, and the detected images are labeled as human images, such as D2.

[0056] Step S20: Determine the current position corresponding to each of the human body images, and determine the sound source angle of the audio signal;

[0057] In this embodiment, after determining all human images included in the scene image, since it is necessary to use human images to correct the positioning results based solely on audio signals, it is necessary to determine the position of each human image in the scene image, i.e., the current position of each human image. Optionally, each human image can be regarded as a rectangle, and the center position of the rectangle of each human image can be taken as the current position of each human image. At the same time, the current position of each human image will be updated periodically to ensure the accuracy of the current position of the human image.

[0058] Furthermore, the DOA algorithm can be used to calculate the sound source angle of the audio signal to complete the initial localization. In one embodiment, the step of determining the sound source angle of the audio signal includes:

[0059] Step a: Calculate the time difference information of the audio signals acquired by multiple microphones in the preset microphone array, input the time difference information into the pre-trained sound source localization model, and output the sound source angle.

[0060] For example, the time difference information of audio signals acquired by multiple microphones in a preset microphone array is statistically analyzed. Based on this time difference information, a pre-trained sound source localization model is used to obtain the sound source angle. Specifically, the sound source localization model uses the known sound source angle to test the time difference of the audio signals received by the microphone arrays, obtaining a time difference matrix to complete the training of the sound source localization model. After training, for any microphone array input into the sound source localization model, the time difference of the audio signals acquired can be used to determine the azimuth and elevation angles of the sound source in the microphone coordinate system, i.e., the sound source angle of the audio signal.

[0061] In another scenario, the audio signal can first be converted into an electrical signal by a microphone array and then subjected to noise reduction and amplification. Multi-channel data synchronous acquisition of the signal is then used, and finally the sound source angle of the audio signal is obtained by the host computer system based on the MUSIC algorithm.

[0062] In this embodiment, the time difference information of the audio signal acquired by multiple microphones in a preset microphone array is statistically analyzed. The pre-trained sound source localization model determines the sound source angle of the audio signal using this time difference information, and performs preliminary localization of the sound source of the audio signal. This provides a data foundation for subsequent correction of the preliminary localization result of the sound source using the position of the human body image. Furthermore, the specific localization algorithm used in the pre-trained sound source localization model is not limited. Users can choose different algorithms to train the sound source localization model according to the actual application scenario. It can be the DOA algorithm, the MUSIC algorithm, or other algorithms, etc., thereby ensuring that the determination of the sound source angle of the audio signal is not limited by the specific application scenario and improving the robustness of determining the sound source angle of the audio signal.

[0063] Step S30: Detect the target position that matches the angle of the sound source among all the current positions, and output the human body image corresponding to the target position.

[0064] In this embodiment, after determining the current position of each character image in the scene image, it is necessary to determine the current position (i.e., the target position) among all the current positions of the character images that matches the sound source angle of the audio signal. By combining the human image with the sound source angle of the audio signal, just like the human eye and ear, the sound source's position is located together, thereby achieving a more accurate real-time sound source localization effect.

[0065] After the human image corresponding to the target location is determined by combining image and sound, the image is cropped to a suitable size that includes the human body and then sent to the display module for display. At this point, the image seen by the display module only contains the image of the human body that is speaking, without including other images, thus achieving sound source localization.

[0066] This embodiment acquires scene images and audio signals from the environment to be located, performs human detection on the scene images to detect all human images included in the scene images, and determines the current position of each human image in the environment to be located. Combined with the sound source angle of the audio signal, it automatically matches all target positions that match the sound source angle. The human image corresponding to the target position is output as the final sound source localization result to the display device for display, thus achieving sound source localization. This avoids the problem of calculating the position coordinates of the sound source solely based on the audio signals received by the microphone array. Since the audio signals received by the microphone include interference signals, the calculation of the sound source position based on the audio signals has a large error, leading to inaccurate sound source localization. This embodiment improves the accuracy of sound source localization. Furthermore, by combining audio signals with human images to achieve sound source localization, it does not require face recognition or identification of the speaker's voiceprint features. It only requires simple human image detection, which requires low computing power and low requirements for computer hardware capabilities, thus reducing the cost of using sound source localization.

[0067] Furthermore, based on the first embodiment of this application described above, a second embodiment of the sound source localization method of this application is proposed. A refinement of step S30 of the above embodiment, which involves detecting target positions among all current positions that match the sound source angle and outputting the human image corresponding to the target position, includes:

[0068] Step b: Obtain the calibration position corresponding to the sound source angle from the first preset calibration relationship, wherein the first preset calibration relationship includes the correspondence between different calibration positions of different sound source angles in the scene image;

[0069] In this embodiment, a positioning device comprising at least a camera and a microphone array is provided. During the initial development of the positioning device, calibration is required before it can be used for sound source localization. The calibration includes a first calibration relationship and a second calibration relationship. The first calibration relationship converts the sound source angle into horizontal coordinates on a panoramic image (i.e., a scene image), i.e., the calibration position. The second calibration relationship uses the distance from the human image detected by the human detection algorithm to the camera to determine the error in the sound source range, i.e., the position error value. Based on this calibration, the positioning device in this embodiment achieves sound source localization.

[0070] In this embodiment, the positioning device completes the calibration of the first calibration relationship and the second calibration relationship. Then, for the sound source angle corresponding to the audio signal, the horizontal coordinate (i.e., the calibration position) of the sound source angle in the scene image can be determined according to the first calibration relationship (i.e., the first preset calibration relationship) in the positioning device. For example, the positioning device uses the audio signals picked up by array microphones m1 and m2 to perform DOA positioning algorithm processing to obtain the angle of the sound source as a. The angle a is transformed into the horizontal coordinate Xa on the scene image through the calibration relationship, i.e., the first preset calibration relationship. Then Xa is the calibration position of the sound source angle.

[0071] Furthermore, in another scenario, the first calibration relationship can be to convert the sound source angle into a two-dimensional coordinate position in the scene image, followed by subsequent position matching in two-dimensional coordinates; alternatively, it can be to convert the sound source angle into a three-dimensional coordinate position in the scene image. In short, the first calibration relationship can be set by the user according to the actual situation, and the specific coordinate axis selected for position matching can also be chosen according to the actual situation; no restrictions are imposed here.

[0072] Step c: Iterate through all the current positions in sequence and check whether the current position being iterated matches the calibrated position;

[0073] In this embodiment, the current position of the human body image and the calibrated position of the sound source angle are both definite position coordinates. Therefore, in most cases, the current position of the human body image and the calibrated position of the sound source angle will not perfectly match. Thus, it is necessary to determine a current position that matches the calibrated position from all current positions; that is, to match all current positions with the calibrated position one by one. Optionally, each time, a current position is selected from all current positions (i.e., the traversed current positions), and it is checked whether this current position matches the calibrated position. If the match is successful, this current position is taken as the target position, and the human body image corresponding to the target position is output to the display module to achieve sound source localization. If the match is unsuccessful, the matching process continues.

[0074] Step d: After a successful match, the current position that has been traversed is taken as the target position.

[0075] In this embodiment, the sound source angle of the audio signal is converted into a calibrated position in the scene image through a pre-set calibration relationship in the positioning device. Then, the current position of each human image is sequentially matched with the calibrated position to determine the target position that matches the calibrated position. The current position of the human image is used to correct the calibrated position of the sound source angle, thereby improving the accuracy of sound source localization. Moreover, the process of converting the sound source angle into a calibrated position and matching the calibrated position with all current positions is automatically completed in the positioning device. It requires low computing power, is less restricted by the scene, has a wide range of applications, and exhibits high robustness in sound source localization.

[0076] In one embodiment, the step of detecting whether the current position being traversed matches the calibrated position includes:

[0077] Step e: Determine the distance from the human image corresponding to the current position to the preset camera;

[0078] In this embodiment, all current positions are sequentially matched with calibrated positions. Each time, one current position is selected for matching. Since the distance between the human body and the camera in the positioning device is not the same for each current position, the sound source error corresponding to each current position is also different. Therefore, after traversing to a current position from all current positions, the distance between the human body image corresponding to this traversed current position and the camera (i.e., the preset camera) in the positioning device within the environment to be located is first determined. The sound source error is determined based on this distance. Optionally, the distance between the human body and the preset camera in the actual environment (i.e., the environment to be located) can be determined by calculating the area of ​​the human body image. Calculating the distance based on the area is simple and effective.

[0079] Step f: Determine the target location range based on the calibrated position and the distance, and detect whether the current traversed position is within the target location range;

[0080] Step g: After the current position being traversed is within the range of the target position, it is determined that the current position being traversed is successfully matched with the calibrated position.

[0081] In this embodiment, by traversing all current positions, each current position is matched with a calibrated position. The target position range of the calibrated position is determined by the distance from the human image corresponding to the traversed current position to the preset camera in the environment to be located. It is then checked whether the traversed current position is within the target position range. If the traversed current position is within the target position range, it means that the human image corresponding to this current position is the target user emitting the sound source signal. If the traversed current position is not within the target position range, the matching continues. The target position range varies depending on the distance from the human body to the preset camera. Only when the traversed current position is within the target position range is the match considered successful, and the successfully matched current position is taken as the target position, ensuring the validity and accuracy of the finally determined target position.

[0082] In one embodiment, the step of determining the target location range based on the calibrated location and the distance includes:

[0083] Step h: Obtain the position error value corresponding to the distance from the second preset calibration relationship, wherein the second preset calibration relationship includes the correspondence between different distances from the human image to the preset camera and different position error values;

[0084] Step i: Determine the second horizontal coordinate of the calibration position in the preset coordinate system, and determine the target position range based on the second horizontal coordinate and the position error value.

[0085] In this embodiment, the second calibration relationship of the positioning device determines the sound source range error, i.e., the position error value, based on the distance between the human image detected by the human detection algorithm and the camera. After determining the distance between the human image corresponding to the current traversed position and the camera in the actual scene, the second preset calibration relationship in the positioning device automatically obtains the sound source range error, i.e., the position error value, corresponding to the distance. Furthermore, the calibrated position is transformed into a preset coordinate system, and the horizontal coordinate of the calibrated position in the preset coordinate system, i.e., the second horizontal coordinate, is determined. The target position range is then determined based on the second horizontal coordinate and the position error value.

[0086] In this embodiment, the position error value corresponding to the distance is determined by the second preset calibration relationship in the positioning device, and the calibration position is transformed into a preset coordinate system to determine the second horizontal coordinate of the calibration position, and further determine the target position range. Since the subsequent position matching also involves matching the horizontal coordinate of the current position, the target range is determined by using the second horizontal coordinate of the calibration position and the position error value, thus ensuring the effectiveness of determining the target position range.

[0087] In one embodiment, the step of determining the target position range based on the second horizontal coordinate and the position error value includes:

[0088] Step j: The sum of the coordinate value of the second horizontal coordinate and the position error value is taken as the upper limit position, and the difference between the coordinate value of the second horizontal coordinate and the position error value is taken as the lower limit position.

[0089] Step k: The range between the upper limit position and the lower limit position is taken as the target position range.

[0090] In this example, after determining the second horizontal coordinate of the calibrated position within a preset coordinate system and determining the position error value corresponding to the current traversed position, the sum of the second horizontal coordinate and the position error value is used as the upper limit position, and the difference between the second horizontal coordinate and the position error value is used as the lower limit position, thereby determining the target position range. For example, refer to... Figure 4A positioning device includes a camera C1 and a microphone array, wherein the microphone array consists of two symmetrical microphones m1 and m2 for sound pickup. A panoramic image D1 (i.e., a scene image) is acquired through the camera C1. The audio data picked up by the array microphones m1 and m2 is processed by the DOA positioning algorithm to obtain the angle of the sound source as a. The angle a is transformed into coordinates Xa on the panoramic image D1 through a first preset calibration relationship. Based on the distance from the human image corresponding to the current position in the actual scene to the camera C1, the error range b is obtained through a second preset calibration relationship. Finally, the D3 sound source range image in the panoramic image D1 is obtained through the horizontal coordinate range [Xa-b, Xa+b].

[0091] In this embodiment, the sum of the coordinate value of the second horizontal coordinate and the position error value is used as the upper limit position, and the difference between the coordinate value of the second horizontal coordinate and the position error value is used as the lower limit position, thus obtaining the target position range. The position error value is determined by the second preset relationship in the positioning device. The second horizontal coordinate is the horizontal coordinate of the calibration position in the preset coordinate system. Therefore, the determination of the target position range involves two aspects: firstly, determining the distance from the sound source angle of the audio signal to the calibration position corresponding to the sound source angle, and then determining the second horizontal coordinate of the calibration position in the preset coordinate system; secondly, determining the distance from the human image corresponding to the current position to the preset camera, and then determining the position error value corresponding to this distance. Finally, the target position range is determined by the second horizontal coordinate and this position error value, ensuring the accuracy and effectiveness of the target position range and providing an effective data foundation for determining the target position.

[0092] In one embodiment, the step of detecting whether the current position being traversed is within the range of the target position includes:

[0093] Step 1: Determine the first horizontal coordinate of the current position in the preset coordinate system, and detect whether the coordinate value of the first horizontal coordinate is within the range of the target position;

[0094] Step m: If the coordinate value of the first horizontal coordinate is within the target position range, then it is determined that the current position being traversed is within the target position range.

[0095] In this embodiment, the horizontal axis direction of the preset coordinate system is used as the matching direction for position matching. Therefore, only the coordinate values in the horizontal direction are matched in this embodiment. The preset coordinate system can be any coordinate system set by the user. Preferably, in this embodiment, a coordinate system is established with the lower left vertex of the scene image as the origin, the horizontal direction as the X-axis, and the vertical direction of the gravitational acceleration as the Y-axis. The X-axis can be called the horizontal coordinate, and the Y-axis can be called the vertical coordinate. After establishing the coordinate system, the current position being traversed can be transformed into the preset coordinate system, and the horizontal coordinate (i.e., the first horizontal coordinate) of the current position being traversed in the X-axis direction can be determined. Then, it is determined whether the coordinate value of the first horizontal coordinate is within the target position range. If it is, it is considered that the current position being traversed is also within the target position range, and the vertical coordinate of the current position being traversed is not considered, and only the horizontal coordinate of the current position being traversed is matched. If the coordinate value of the first horizontal coordinate is not within the target position range, it is considered that the current position being traversed is not within the target position range, and a new current position is re-traversed from all the current positions for the next round of matching.

[0096] In addition, to assist in understanding the matching process between the current position being traversed and the calibrated position in this example, the following is an example. Refer to Figure 5 , a positioning device including a camera C1 and a microphone array. The microphone array consists of two symmetric microphones m1 and m2 for sound pickup. The panoramic image D1 (i.e., the scene image) is collected through the camera C1. The coordinates (x0, y0) of the D2 human body image corresponding to a current position being traversed on the D1 panoramic image. The audio signals collected by the microphones m1 and m2 can obtain the horizontal coordinate Xa of the calibrated position in the panoramic image D1 through the DOA sound source localization algorithm and the first preset calibration relationship, and the distance d from the human body image corresponding to the current position being traversed to the preset camera in the actual scene is calculated. The sound source error range for the distance d is b. Then, the D3 sound source range map, that is, the target position range, is [Xa-b, Xa+b]. By comparing whether x0 is within the range of [Xa-b, Xa+b], it is determined whether the matching is successful and whether this current position being traversed is the target position. If x0 < Xa - b or x0 > Xa + b, the matching fails, and it is not the D2 human body making a sound. A new current position is re-traversed from all the current positions for the next round of matching. If x0 >= Xa - b and x0 <= Xa + b, the matching is successful. At this time, it can be known that this current position being traversed is the target position and the human body corresponding to this current position being traversed is the target user making a sound, that is, the D2 human body is making a sound.

[0097] In this embodiment, the first horizontal coordinate of the current position in a preset coordinate system is matched with the target position range. If the coordinate value of the first horizontal coordinate is not within the target position range, the current position is traversed again for matching. If the coordinate value of the first horizontal coordinate is within the target position range, the match is successful. This traversed current position is then taken as the target position, and the human image corresponding to the target position is output to the display device, thus completing the sound source localization. This achieves accurate and effective sound source localization. Furthermore, the entire calculation and matching process is simple, requires little computing power, has low hardware requirements, is less restricted by the scene, has a wide range of applications, and exhibits high robustness in sound source localization.

[0098] Successful matching ensures the accuracy of target location determination, and the entire matching process is completed automatically, requiring only coordinate value comparison to complete the matching. This requires low computing power, and the matching process is efficient, accurate, and fast.

[0099] In addition, this application also provides a sound source localization device. Referring to the figure, the sound source localization device includes:

[0100] The acquisition module is used to acquire scene images and audio signals in the environment, and to detect all human images included in the scene images;

[0101] The determination module is used to determine the current position corresponding to each of the human body images and to determine the sound source angle of the audio signal;

[0102] The output module is used to detect the target position that matches the angle of the sound source among all the current positions, and output the human body image corresponding to the target position.

[0103] In addition, this application also provides a sound source localization device, which includes: a memory, a processor, and a sound source localization program stored in the memory; the processor is used to execute the sound source localization program to implement the steps of the above-described embodiments of the sound source localization method.

[0104] This application also provides a computer-readable storage medium storing one or more programs, which can be executed by one or more processors to implement the steps of the various embodiments of the above-described sound source localization method.

[0105] The specific implementation of the storage medium in this application is basically the same as the embodiments of the sound source localization method described above, and will not be repeated here.

[0106] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0107] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0108] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0109] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method for locating a sound source, characterized in that, The sound source localization method includes: Acquire scene images and audio signals in the environment, and detect all human images included in the scene images; Determine the current position corresponding to each of the human body images, and determine the sound source angle of the audio signal; The calibration position corresponding to the sound source angle is obtained from the first preset calibration relationship, wherein the first preset calibration relationship includes the correspondence between different sound source angles and different calibration positions in the scene image; The system iterates through all the current positions in sequence, and determines the distance from the human image corresponding to the current position to the preset camera based on the area of ​​the human image corresponding to the current position. The position error value corresponding to the distance is obtained from the second preset calibration relationship, wherein the second preset calibration relationship includes the correspondence between different distances from the human image to the preset camera and different position error values; Determine the second horizontal coordinate of the calibration position in the preset coordinate system, determine the target position range based on the second horizontal coordinate and the position error value, and detect whether the current position being traversed is within the target position range; After the current position is within the range of the target position, it is determined that the current position matched the calibrated position successfully. After a successful match, the current position is taken as the target position, and the human image corresponding to the target position is output.

2. The sound source localization method as described in claim 1, characterized in that, The step of detecting whether the current position being traversed is within the range of the target position includes: Determine the first horizontal coordinate of the current position in the preset coordinate system, and detect whether the coordinate value of the first horizontal coordinate is within the range of the target position; If the coordinate value of the first horizontal coordinate is within the target location range, then it is determined that the current traversed position is within the target location range.

3. The sound source localization method as described in claim 1, characterized in that, The step of determining the target location range based on the second horizontal coordinate and the position error value includes: The sum of the second horizontal coordinate value and the position error value is used as the upper limit position, and the difference between the second horizontal coordinate value and the position error value is used as the lower limit position. The range between the upper limit position and the lower limit position is taken as the target position range.

4. The sound source localization method as described in claim 1, characterized in that, The step of determining the sound source angle of the audio signal includes: The time difference information of the audio signal acquired by multiple microphones in a preset microphone array is statistically analyzed, and the time difference information is input into a pre-trained sound source localization model to output the sound source angle.

5. A sound source localization device, characterized in that, The sound source localization device includes: The acquisition module is used to acquire scene images and audio signals in the environment, and to detect all human images included in the scene images; The determination module is used to determine the current position corresponding to each of the human body images and to determine the sound source angle of the audio signal; Output module, used for: The calibration position corresponding to the sound source angle is obtained from the first preset calibration relationship, wherein the first preset calibration relationship includes the correspondence between different sound source angles and different calibration positions in the scene image; The system iterates through all the current positions in sequence, and determines the distance from the human image corresponding to the current position to the preset camera based on the area of ​​the human image corresponding to the current position. The position error value corresponding to the distance is obtained from the second preset calibration relationship, wherein the second preset calibration relationship includes the correspondence between different distances from the human image to the preset camera and different position error values; Determine the second horizontal coordinate of the calibration position in the preset coordinate system, determine the target position range based on the second horizontal coordinate and the position error value, and detect whether the current position being traversed is within the target position range; After the current position is within the range of the target position, it is determined that the current position matched the calibrated position successfully. After a successful match, the current position is taken as the target position, and the human image corresponding to the target position is output.

6. A sound source localization device, characterized in that, The sound source localization device includes: a memory, a processor, and a sound source localization program stored in the memory and executable on the processor. When the sound source localization program is executed by the processor, it implements the steps of the sound source localization method as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a sound source localization program, which, when executed by a processor, implements the steps of the sound source localization method as described in any one of claims 1 to 4.

Images