A method and device for locating sound sources on a traffic road

Abstract

The embodiment of the present application provides a method and device for locating a sound source on a traffic road, which relates to the field of data processing technology. The method includes: obtaining channel data output by each microphone in the microphone array of each sonar, wherein each channel data output by each microphone includes: audio data collected by the microphone and a count value of the audio data collected by the microphone; matching the count values ​​included in each channel data, arranging the audio data with the same count value in sequence in units of sonar, and obtaining an audio array corresponding to the count value; locating the sound source in the traffic road according to the positional relationship between the obtained audio array and the microphones in the microphone array between different sonars, and obtaining the sound source position. Applying the sound source localization solution provided by the embodiment of the present application can improve the accuracy of sound source localization.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method and device for locating a traffic road sound source. Background Art

[0002] Sonar is used to locate sound sources, and its positioning accuracy is related to the array area of ​​the microphone array integrated inside and the number of microphone channels. Since sonar is limited by weight and volume, the array area and number of microphone channels in the sonar are limited, which leads to limited positioning accuracy of sonar. Summary of the invention

[0003] The purpose of the embodiment of the present application is to provide a method and device for localizing a sound source on a traffic road to improve the accuracy of localizing the sound source. The specific technical solution is as follows:

[0004] In a first aspect, an embodiment of the present application provides a method for locating a sound source on a traffic road, wherein at least two sonars are set up above the traffic road, and the detection ranges of the at least two sonars have an overlapping area, and the overlapping area includes a road area, and microphone arrays in different sonars synchronously collect data, and the method includes:

[0005] Obtain channel data output by each microphone in the microphone array of each sonar, wherein each channel data output by each microphone includes: audio data collected by the microphone and a count value of the audio data collected by the microphone;

[0006] Matching the count values ​​included in each channel data, arranging the audio data with the same count value in sequence in sonar units, and obtaining an audio array corresponding to the count value;

[0007] According to the obtained audio array and the positional relationship between microphones in different sonar microphone arrays, the sound source in the traffic road is located to obtain the sound source position.

[0008] In a second aspect, an embodiment of the present application provides a traffic road sound source positioning device, wherein at least two sonars are set up above the traffic road, and the detection ranges of the at least two sonars have overlapping areas, and the overlapping areas include road areas, and microphone arrays in different sonars synchronously collect data, and the device includes:

[0009] A data acquisition module is used to obtain channel data output by each microphone in the microphone array of each sonar, wherein each channel data output by each microphone includes: audio data collected by the microphone and a count value of the audio data collected by the microphone;

[0010] An array acquisition module is used to match the count values ​​included in each channel data, and arrange the audio data with the same count value in sequence in sonar units to obtain an audio array corresponding to the count value;

[0011] The sound source localization module is used to locate the sound source in the traffic road according to the obtained audio array and the position relationship between the microphones in the microphone arrays of different sonars to obtain the sound source position.

[0012] In a third aspect, an embodiment of the present application provides an electronic device, including:

[0013] Memory, used to store computer programs;

[0014] The processor is used to implement the method described in the first aspect when executing the program stored in the memory.

[0015] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0016] Beneficial effects of the embodiments of the present application:

[0017] It can be seen from the above that when the solution provided in the embodiment of the present application is used to locate the sound source, the microphone arrays of each sonar synchronously collect data, and the channel data obtained by the electronic device includes audio data and count values. It can be seen that the count value corresponds to the audio data, so that the count value also corresponds to the collection time of the audio data. The count values ​​included in each channel data are matched, and the audio data with the same count value are arranged in sequence in units of sonar to obtain an audio array corresponding to the count value. The audio array can be regarded as: at the collection time corresponding to the corresponding count value, the microphone array of each sonar contains an array of audio data synchronously collected by the microphone, which is equivalent to increasing the number of microphone channels in the microphone array, and the array area is also equivalent to increasing to the area determined according to multiple sonars. Therefore, sound source positioning is performed according to the positional relationship between the audio array and the microphones in the microphone array between different sonars, which can improve the positioning accuracy of sound source positioning. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0019] Figure 1 A schematic diagram of a sound source localization scenario provided in an embodiment of the present application;

[0020] Figure 2a A schematic diagram of the structure of a sonar provided in an embodiment of the present application;

[0021] Figure 2b A schematic diagram of the distribution of data collected by microphones in the first sonar microphone array provided in an embodiment of the present application;

[0022] Figure 3 A schematic diagram of a flow chart of a method for localizing a traffic road sound source provided in an embodiment of the present application;

[0023] Figure 4 A schematic diagram of the position relationship of a sonar provided in an embodiment of the present application;

[0024] Figure 5 A schematic diagram of a target coordinate system provided in an embodiment of the present application;

[0025] Figure 6 A schematic diagram of the distribution of data collected by microphones in the second sonar microphone array provided in an embodiment of the present application;

[0026] Figure 7 A schematic diagram of the structure of a traffic road sound source localization device provided in an embodiment of the present application;

[0027] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field based on the present application belong to the scope of protection of the present application.

[0029] First, the execution entity of the traffic road sound source localization solution provided in the embodiment of the present application is described.

[0030] The execution subject of this solution is an electronic device. In one case, the electronic device can be one of the sonars. In this case, the sonar can be connected to other sonars for communication to obtain audio data generated by other sonars, and the microphone array of the sonar itself can collect data. The sonar can generate audio data based on the data collected by the microphone array. After the sonar obtains the audio data generated by each sonar, it can locate the sound source based on the audio data generated by each sonar.

[0031] In one embodiment of the present application, the sonar can be connected to other sonars via USB, Ethernet or other transmission interfaces.

[0032] In another case, the electronic device may be a device with data processing capability other than the sonars. In this case, the electronic device may be connected to communicate with the sonars to obtain the audio data generated by the sonars, thereby locating the sound source based on the audio data generated by the sonars.

[0033] In one embodiment of the present application, the electronic device can be connected to each sonar communication via USB, Ethernet or other transmission interface.

[0034] Secondly, the application scenario of the traffic road sound source localization solution provided in the embodiment of the present application is explained.

[0035] The sound source localization solution provided in the embodiment of the present application can be applied to the scenario of locating vehicles that produce roaring sounds on traffic roads.

[0036] In this scenario, at least two sonars may be set up above the traffic road, and the detection ranges of the at least two sonars have an overlapping area, and the overlapping area includes the road area, so that each sonar can detect the sound emitted by vehicles in the road area. In addition, in the at least two sonars, the microphone arrays in different sonars collect data synchronously.

[0037] The relative position relationship between the sonars installed can be found in the subsequent embodiments and will not be described in detail here.

[0038] like Figure 1 , showing a scenario of locating a vehicle generating a roaring sound on a traffic road. Figure 1 In the scene shown, there are two sonars 101 and 102 that collect data synchronously. The two sonars are installed side by side on a pole 103 above the road. The two sonars are installed independently, and the change of the installation position of one sonar will not affect the other sonar. There is an overlapping area in the detection range of the two sonars, such as Figure 1 The detection range of the sonar 101 and the detection range of the sonar 102 shown in FIG. 1 overlap, that is, Figure 1 The overlapping area of ​​the detection ranges of the sonars 101 and 102 is an effective detection area for sound source positioning, and the overlapping area includes a road area. The two sonars can be used to locate the vehicle that generates the roar in the road area in the effective detection area.

[0039] The traffic road sound source localization solution provided in the embodiment of the present application can also be applied to the scenario of localizing a speaker, or the scenario of localizing other sound sources, which is not limited in the embodiment of the present application.

[0040] in addition, Figure 1Only the case of two sonars is shown. In addition, the number of sonars in the sound source localization solution provided in the embodiment of the present application can also be other numbers, such as 3, 4, etc., and the embodiment of the present application is not limited to this.

[0041] Next, the sonar structure will be described.

[0042] See also Figure 2a , Figure 2a A schematic diagram of the structure of a sonar. Figure 2a The center of the sonar shown is integrated with an image sensor 201, and there are 32 microphones around the center of the sonar, which form the microphone array of the sonar. Each microphone in the microphone array is called a microphone channel, and the number of microphone channels is the number of microphones included in the microphone array. Figure 2a The number of microphone channels is 32.

[0043] In the sonar microphone array, the data collected by each microphone is called audio data.

[0044] Each microphone in the microphone array collects data synchronously. The data collected by each microphone once can be called a sampling point data. The number of channel data contained in the sampling point data is the number of microphone channels in the microphone array. Figure 2a In the example, the number of microphone channels is 32, so the number of audio data contained in one sampling point data is 32.

[0045] Each microphone in the microphone array collects data synchronously, and the sampling frequency of the microphone can be regarded as the sampling frequency of the microphone array. For example, if the sampling frequency of the microphone is 16000 times / second, and each microphone in the microphone array collects data synchronously, the sampling frequency of the microphone array can also be regarded as 16000 times / second.

[0046] In one embodiment of the present application, an audio acquisition module is integrated in the sonar to obtain data collected by the microphone array at a preset sampling frequency during the operation of the microphone array.

[0047] Each time the microphone array collects data, the data collected by each microphone in the microphone array constitutes a sampling point data. The microphone array continuously collects data according to the preset sampling frequency, and multiple sampling point data will be obtained. The audio acquisition module can collect and arrange the sampling point data in the order of the sampling points.

[0048] If A represents the sampling point, B i represents the data collected by the i-th microphone in the microphone array, N represents the total number of microphones in the microphone array, and i is any integer from 1 to N. Then a sampling point data can be expressed as: A[B1 B2 B3 …BN ]. During the operation of the microphone array, the microphone array continuously collects data to obtain sampling point data. After the sonar sorts the data, the obtained sampling point data can be expressed as: A1[B1 B2 B3 … B N ]、A2[B1 B2 B3 … B N ]、A3[B1B2 B3 … B N ], ..., A j [B1 B2 B3 ... B N ], where A j [B1 B2 B3 ... B N ] represents the data of the jth sampling point, which contains the data collected synchronously for the jth time by the N microphones in the microphone array.

[0049] like Figure 2b , shows the data of the approximate sinusoidal signal obtained by collecting data with a sampling frequency of 16000 times / second by a single microphone, Figure 2b Each dot in the figure represents the data of the microphone at a sampling point, that is, the channel data obtained by one acquisition, where the first dot is the data of the microphone at sampling point 1, the second dot is the data of the microphone at sampling point 2, the third dot is the data of the microphone at sampling point 3, and the jth dot is the data of the microphone at sampling point j. The data of each microphone in the microphone array at the same sampling point constitutes the data of the same sampling point.

[0050] The following describes the relationship between the array area of ​​the microphone array and the number of microphone channels in the sonar and the positioning accuracy of the sonar.

[0051] In beamforming technology, the main indicators for measuring positioning accuracy performance are angular resolution and noise suppression level.

[0052] The angular resolution is related to the main lobe width of the beam. The narrower the main lobe width, the higher the angular resolution. The relationship between the two can be expressed as the following expression:

[0053]

[0054]

[0055] in, represents the main lobe width, represents the signal beam wavelength, c represents the speed of sound, f represents the signal beam frequency, and D represents the size of the microphone array.

[0056] It can be seen from the above expression that the larger the microphone array area, the larger D is, and the smaller B is, that is, the narrower the main lobe width is, the higher the angular resolution is, and the higher the positioning accuracy is.

[0057] The noise suppression level is related to the sidelobe dynamic range. The larger the sidelobe dynamic range, the smaller the impact of interference noise and the higher the noise suppression level.

[0058] The sidelobe dynamic range can be replaced by the white noise gain, and its relationship with the number of microphone channels can be expressed as follows:

[0059]

[0060] in, represents the white noise gain, Indicates the number of microphone channels.

[0061] It can be seen from the above expressions that the larger the number of microphone channels, the higher the white noise gain, the better the noise resistance, and the higher the robustness, which is equivalent to the larger sidelobe dynamic range, the smaller the impact of interference noise, the higher the noise suppression level, and the higher the positioning accuracy.

[0062] Next, the method and device for localizing traffic road sound sources are described in detail through specific embodiments.

[0063] See also Figure 3 , a flow chart of a method for locating a sound source on a traffic road is provided. The sound source locating solution provided in this embodiment is applied to a scenario of locating a vehicle that generates a roaring sound on a traffic road. In this scenario, at least two sonars are set up above the traffic road. The detection ranges of the at least two sonars have an overlapping area, and the overlapping area includes a road area. The microphone arrays in different sonars synchronously collect data.

[0064] First, combining the two situations of the above-mentioned electronic devices, the method of realizing synchronous data collection of microphone arrays in different sonars is explained.

[0065] 1. The electronic device is one of the sonars mentioned above.

[0066] In this case, the above-mentioned electronic device can generate a synchronous acquisition signal and send the synchronous acquisition signal to other sonars. After receiving the synchronous acquisition signal, the other sonars start data acquisition. In addition, when the electronic device sends the synchronous acquisition signal to other devices, it can also start data acquisition itself.

[0067] Since the time taken for the electronic device to send the synchronous acquisition signal to other sonars and for other sonars to receive the synchronous acquisition signal is extremely small, each sonar can be considered to collect data synchronously.

[0068] 2. The above-mentioned electronic equipment is equipment with data processing capabilities other than sonars

[0069] In this case, the electronic device can generate a synchronous acquisition signal and send the synchronous acquisition signal to each sonar synchronously. After each sonar receives the synchronous acquisition signal, it can synchronously perform data acquisition.

[0070] In addition, regardless of whether the electronic device is a sonar or other device, the microphone array in each sonar collects data synchronously, and each microphone included in the microphone array in each sonar also collects data synchronously, so all microphones included in the microphone array in each sonar collect data synchronously. For example, if two sonars are set up above a traffic road, and the microphone array of each sonar includes 32 microphones, then the microphone array in the two sonars includes 64 microphones, and the 64 microphones collect data synchronously.

[0071] Below Figure 3 Steps S301-S303 in the flowchart are illustrated.

[0072] Step S301: obtaining channel data output by each microphone in the microphone array of each sonar.

[0073] The data of each channel output by each microphone includes: audio data collected by the microphone and a count value of the audio data collected by the microphone.

[0074] In the process of synchronously collecting data by each microphone in the microphone array of each sonar, since each microphone in the microphone array of each sonar collects data synchronously, each microphone in the microphone array of each sonar collects data synchronously according to the same sampling frequency.

[0075] When each sonar collects data synchronously, for each sonar, the sonar can start from the first audio data collected by the microphone included in its own microphone array, and count the audio data collected by each microphone included in its own microphone. Each audio data output by each microphone corresponds to a count value, and the count value can be regarded as the collection order of the corresponding audio data, and the collection order of each audio data can be counted from zero or from one.

[0076] For example, when the collection order of audio data is counted from one, if the count value corresponding to an audio data is 3, it means that the audio data is the data obtained by the microphone collecting data for the third time.

[0077] The channel data output by the microphone is composed of audio data and its corresponding count value, that is, each channel data output by each microphone includes: audio data collected by the microphone and the count value of the audio data collected by the microphone. Therefore, when the electronic device locates the sound source of the traffic road, it uses the two types of information, audio data and its corresponding count value, to realize the sound source location, rather than only using the audio data to locate the sound source.

[0078] Specifically, each sonar can collect channel data output by each microphone in its own microphone array. In the case where the electronic device is one of the sonars, each microphone in the microphone array of the electronic device itself can output channel data, and other sonars can send the collected channel data to the electronic device after collecting the channel data.

[0079] In the case that the electronic device is a device other than each sonar, after collecting the channel data, each sonar can send the collected channel data to the electronic device.

[0080] Step S302: Match the count values ​​included in each channel data, and arrange the audio data with the same count value in sequence in sonar units to obtain an audio array corresponding to the count value.

[0081] All microphones included in the microphone array in each sonar collect data synchronously. Each time all microphones collect audio data synchronously, the sonar where the microphone is located will count it. The count value of the audio data in the same synchronous collection process is the same. Therefore, the audio data with the same count value is the audio data obtained by all microphones collecting data synchronously once. The audio data with the same count value is arranged to obtain an audio array, which can be considered as a one-dimensional array of audio data obtained by all microphones collecting data synchronously once.

[0082] The count value corresponds to the audio array, and reflects the number of times that the audio data contained in the corresponding audio array is obtained by all microphones synchronously collecting data.

[0083] For example, if the count value is 5, it means that the audio data contained in the audio array corresponding to the count value is the data obtained by all microphones synchronously collecting data for the fifth time.

[0084] Specifically, the channel data output by each microphone includes audio data and count values. For each count value, the channel data containing the count value can be determined in the channel data from each sonar, and the audio data in the channel data can be determined, and the determined audio data constitutes the audio data subarray corresponding to the sonar.

[0085] After the audio data subarrays corresponding to each sonar are determined from the channel data from each sonar, the audio data subarrays corresponding to each sonar can be arranged one by one in units of sonar, with the audio data subarray corresponding to one sonar being followed by the audio data subarray corresponding to another sonar, to obtain an audio array.

[0086] For example, the audio data in the channel data output by the microphone can be recorded as S xyz , indicating the The microphone array of the sonar The corresponding count value of the microphone output is audio data.

[0087] If there are three sonars, namely sonar 1, sonar 2 and sonar 3, the microphone array of each sonar includes four microphones, and the three sonars synchronously collect data twice, then the audio data contained in the channel data output by each microphone in the microphone array of each sonar is as follows:

[0088] The audio data contained in the channel data output by the four microphones in the microphone array of sonar 1 is: S 110 , S 111 , S 120 , S 121 , S 130 , S 131 , S 140 , S 141 ;

[0089] The audio data contained in the channel data output by the four microphones in the microphone array of Sonar 2 is: S 210 , S 211 , S 220 , S 221 , S 230 , S 231 , S 240 , S 241 ;

[0090] The audio data contained in the channel data output by the four microphones in the microphone array of Sonar 3 is: S 310 , S 311 , S 320 , S 321 , S 330 , S 331 , S 340 , S 341 .

[0091] When matching the count values ​​included in each channel data, , among the audio data contained in the channel data output by each microphone of the sonar 1, it can be determined that the audio data in the channel data containing the count value is S 110 , S 120 , S 130 , S 140 In the audio data contained in the channel data output by each microphone of the sonar 2, it can be determined that the audio data in the channel data containing the count value is S 210 , S 220 , S 230 , S 240In the audio data contained in the channel data output by each microphone of the sonar 3, it can be determined that the audio data in the channel data containing the count value is S 310 , S 320 , S 330 , S 340 .

[0092] After the audio data is determined, the determined audio data is arranged in sequence in sonar units to obtain an audio array corresponding to the count value [S 110 , S 120 , S 130 , S 140 , S 210 , S 220 , S 230 , S 240 , S 310 , S 320 , S 330 , S 340 ].

[0093] Similarly, for the count value , you can get the count value The corresponding audio array [S 111 , S 121 , S 131 , S 141 , S 211 , S 221 , S 231 , S 241 , S 311 , S 321 , S 331 , S 341 ].

[0094] For example, if two sampling point data generated by sonar are obtained, for a count value, the electronic device determines a sampling point data generated by the sonar and corresponding to the count value, which is expressed as: A 11 [B 11 , B 12 , B 13 …… B 1N ], the sampling point data corresponding to the count value generated by another sonar is expressed as: A 21 [B 21 , B 22 , B 23 …… B 2N ], then the determined sampling point data are arranged in sequence to obtain the audio array [B 11 , B 12 , B 13 …… B 1N , B 21 , B22 , B 23 …… B 2N ].

[0095] Among them, B 1i is: the first sonar microphone array The audio data collected by microphones, B 2i The microphone array of the second sonar includes The audio data collected by microphones, i is any integer in 1-N.

[0096] Step S303: According to the obtained audio array and the positional relationship between microphones in different sonar microphone arrays, the sound source in the traffic road is located to obtain the sound source position.

[0097] In the process of sound source positioning, after each sonar is installed and set up, the relative positions between the sonars remain unchanged, and the positions of the microphones in the microphone array of the sonar remain unchanged. Therefore, after each sonar is installed and set up, the positional relationship between the microphones in the microphone array of each sonar can be calibrated, that is, the microphone position relationship can be calibrated.

[0098] The calibration method of the positional relationship between the microphones in the microphone array of each sonar can be found in the subsequent embodiments and will not be described in detail here.

[0099] Specifically, the channel data collected by the microphone includes the audio data collected by the microphone and the count value, and the audio data also has a corresponding relationship with the collection time. Therefore, the count value has a corresponding relationship with the collection time. The audio array corresponding to a count value can be understood as: the array of audio data collected by each microphone in the microphone array of each sonar at the collection time corresponding to the count value.

[0100] According to the positional relationship between the audio array corresponding to each count value and the microphones in the microphone arrays between different sonars, the sound source in the traffic road is located, and the position of the sound source at the collection time corresponding to the count value can be obtained.

[0101] The specific implementation method of locating the sound source position according to the audio array corresponding to a count value can be found in the subsequent embodiments and will not be described in detail here.

[0102] It can be seen from the above that when the solution provided in the embodiment of the present application is used to locate the sound source, the microphone arrays of each sonar synchronously collect data, and the channel data obtained by the electronic device includes audio data and count values. It can be seen that the count value corresponds to the audio data, so that the count value also corresponds to the collection time of the audio data. The count values ​​included in each channel data are matched, and the audio data with the same count value are arranged in sequence in units of sonar to obtain an audio array corresponding to the count value. The audio array can be regarded as: at the collection time corresponding to the corresponding count value, the microphone array of each sonar contains an array of audio data synchronously collected by the microphone, which is equivalent to increasing the number of microphone channels in the microphone array, and the array area is also equivalent to increasing to the area determined according to multiple sonars. Therefore, sound source positioning is performed according to the positional relationship between the audio array and the microphones in the microphone array between different sonars, which can improve the positioning accuracy of sound source positioning.

[0103] like Figure 4 As shown in the figure, when two sonars with 32 channels are used for sound source positioning, it can be regarded as using a sonar with 64 channels and an array area approximately equal to the area of ​​an ellipse with a major diameter of 2L+d for sound source positioning, wherein the diameter L represents the diameter of a single 32-channel sonar, and the interval d represents the interval between two 32-channel sonars. Compared with a 32-channel sonar, this sonar has more channels and a larger array area, and therefore, the sound source positioning accuracy is higher.

[0104] When three sonars with 32 channels are used to locate the sound source, it can be regarded as using a 96-channel sonar with a larger array area to locate the sound source, so the sound source localization accuracy is further increased. It can be seen that the sound source localization accuracy changes with the number of sonars used. In this way, for scenes with different positioning accuracy requirements, the number of sonars can be flexibly selected to meet the scene's requirements for positioning accuracy.

[0105] In this scheme, each sonar is installed and set up independently. As long as the overlapping detection area of ​​each sonar includes the traffic road, the position change of one sonar will not affect the position of other sonars. Therefore, it can be considered that this scheme collects split sonars for sound source positioning. In the scenario where the sonar needs to be installed outdoors, the installation of the sonar can be completed more conveniently, and it is convenient for the later maintenance of the installed sonar, and the ability to cope with complex weather is better.

[0106] In addition, the application of the sound source localization solution provided in the embodiment of the present application does not require changing the hardware structure of the sonar, and only improves the sonar at the software level, thereby reducing the impact on the production of the sonar.

[0107] The relative position relationship between the sonars when they are set up is explained below.

[0108] In the process of installing each sonar, a sonar can be installed first as a reference sonar. The installation position of the reference sonar can be determined based on previous manual experience, or any position can be determined as the installation position of the reference sonar. After the reference sonar is installed, the sonar adjacent to the reference sonar is installed according to the position of the reference sonar. After the sonar is installed, the sonar is also used as a reference sonar. Other sonars are installed in the same way until the installation of each sonar is completed.

[0109] In one embodiment of the present application, the sonar adjacent to the installed reference sonar may be installed according to at least one of the following two installation standards.

[0110] The first installation standard is that in two adjacent sonars, the distance between the first microphone and the second microphone, which are the shortest distances apart, is less than or equal to a first set distance.

[0111] The first microphone and the second microphone belong to two adjacent sonars respectively.

[0112] The first set distance may be determined according to the wavelength of the sound emitted by the sound source to be located.

[0113] In different sound source positioning scenarios, the sound emitted by the sound source to be positioned is different, and the wavelength of the sound is also different, so the first set distance is also different.

[0114] For example, in a scenario of locating a vehicle that generates a roar on a road, the sound emitted by the sound source to be located is the roar of the vehicle, and the wavelength of the sound emitted by the sound source to be located is the wavelength of the roar of the vehicle. Since the roar of the vehicle can be regarded as a sound with a frequency of 700 Hz, the wavelength of the sound emitted by the sound source to be located is the wavelength of the sound of 700 Hz, so the first set distance is a distance determined according to the wavelength of the sound of 700 Hz.

[0115] In the scenario of locating a speaker, the sound emitted by the sound source to be located is a human voice, and the wavelength of the sound emitted by the sound source to be located is the wavelength of the human voice, so the first set distance is a distance determined according to the wavelength of the human voice.

[0116] The first set distance is determined according to the wavelength of the sound emitted by the sound source to be located, and each sonar is installed according to the first set distance. This can make the sound source positioning using the installed sonars more sensitive to the sound of the wavelength of the sound emitted by the sound source to be located, so that the sound source emitting the sound of this wavelength can be positioned more accurately, thereby improving the accuracy of sound source positioning.

[0117] In addition, the first set distance may also be determined based on previous experience in manually installing sonar.

[0118] When installing the sonar to be installed adjacent to the reference sonar, for the outermost microphone in the microphone array of the reference sonar, a spatial range with the microphone as the center and the first set distance as the radius can be determined. In this way, when installing the sonar to be installed, as long as at least one of the microphones included in the sonar to be installed is ensured to exist within the spatial range, the distance between the microphone belonging to the reference sonar and serving as the center of the circle and the microphone belonging to the sonar to be installed and located within the spatial range can be made less than or equal to the first set distance.

[0119] Since each sonar is used to locate the sound source of the traffic road, there is a certain distance between the traffic road and each sonar. Among the adjacent sonars of each sonar, the smaller the distance between the first microphone and the second microphone, which are the shortest apart, the smaller the distance between adjacent sonars. Then, each sonar can be regarded as a sonar containing more microphones and a larger number of microphone channels. Therefore, by using a sonar with more microphones and a larger number of microphone channels to locate the sound source, the accuracy of sound source positioning can be improved. Therefore, when installing each sonar, the accuracy of sound source positioning can be improved by setting the distance between the first microphone and the second microphone, which respectively belong to adjacent sonars, to be less than or equal to the first set distance.

[0120] In one embodiment of the present application, the first set distance is half the wavelength of the sound emitted by the sound source to be located. When installing each sonar, setting the distance between the first microphone and the second microphone, which are the shortest in two adjacent sonars, to be less than or equal to half the wavelength of the sound emitted by the sound source to be located can improve the accuracy of sound source location.

[0121] The second installation standard is that the installation distance between the centers of adjacent sonars is less than or equal to the second set distance.

[0122] The second set distance may be determined according to the first set distance.

[0123] In this case, the first set distance can be regarded as the maximum distance between the first microphone and the second microphone in two adjacent sonars, and the relative positions between the first microphone and the second microphone and the center of the sonar to which they belong are fixed and can be measured in advance. Therefore, the maximum distance between the centers of adjacent sonars can be calculated based on the first set distance and the relative positions between the first microphone and the second microphone and the center of the sonar to which they belong, as the above-mentioned second set distance, or a distance smaller than the calculated maximum distance between the centers of adjacent sonars can be used as the above-mentioned second set distance.

[0124] For example, the sum of the first set distance, the distance between the first microphone and the center of the sonar to which it belongs, and the distance between the second microphone and the center of the sonar to which it belongs may be calculated as the maximum distance between the centers of adjacent sonars.

[0125] Since the second set distance is determined according to the first set distance, when the installation distance between the centers of adjacent sonars satisfies the standard of being less than or equal to the second set distance, the distance between the first microphone and the second microphone of the adjacent sonars can also meet the standard of being less than or equal to the first set distance. Therefore, each sonar is installed according to the standard that the installation distance between the centers of adjacent sonars is less than or equal to the second set distance, thereby improving the accuracy of sound source positioning.

[0126] The second set distance may also be determined based on previous experience in manually installing sonar.

[0127] When installing each sonar, when the sonar rotates a certain angle about its own center, the position of each microphone in the sonar will change, but the position of the sonar center will not change. Therefore, using the position of the center of each sonar to install the sonar, there is no need to consider the rotation of the sonar about its own center. This is more convenient than using the position of the microphone of each sonar to install the sonar.

[0128] In one embodiment of the present application, in a scenario of locating a vehicle that generates a roaring sound on a road, the second set distance is 40 centimeters.

[0129] Two calibration methods for the positional relationship between microphones in the microphone array of each sonar used in the above step S303 are described below.

[0130] In the first calibration method, the sonar is integrated with an image sensor. This type of sonar is also called a sonar-video all-in-one machine, an audio-visual all-in-one machine, or a sonar all-in-one machine.

[0131] The image sensor may be located at the center of the sonar, and the center of the image sensor may overlap with the center of the sonar, as described above. Figure 2a ; The image sensor can also be located at other positions of the sonar so that the center of the image sensor does not overlap with the center of the sonar.

[0132] In this case, the microphone position relationship is obtained by AD calibration according to the following steps:

[0133] Step A: Obtain calibration images collected by at least two sonar-integrated image sensors.

[0134] Among them, calibration objects are placed in the overlapping area of ​​the field of view of the image sensors integrated by each sonar.

[0135] When a calibration object is placed in the overlapping area of ​​the field of view of each sonar integrated image sensor, each sonar integrated image sensor can collect multiple calibration images of the calibration object in different postures, and the electronic device can obtain the multiple calibration images collected by each sonar integrated image sensor.

[0136] In addition, the surface of the calibration object may have a plurality of calibration balls. When the calibration object is placed in the overlapping area, the image sensors integrated by each sonar may capture one or more calibration images of the calibration object in one or more postures, and the electronic device then obtains the calibration images captured by the image sensors integrated by each sonar.

[0137] Step B: Detect the image position of the calibration object in each calibration image.

[0138] Specifically, object detection may be performed on each calibration image using an object detection model, algorithm, etc., to obtain the image position of the calibration object in each calibration image.

[0139] In the case where the surface of the calibration object has a plurality of calibration spheres, the image position of each calibration sphere may be detected as the image position of the calibration object.

[0140] Step C: According to the image position of the calibration object in each calibration image, the center position relationship between the centers of each sonar is obtained.

[0141] Specifically, the central position relationship between the centers of each sonar may include the position relationship between the centers of two sonars. For any two sonars in each sonar, the first position relationship between the image coordinate systems of the calibration images corresponding to the two sonars can be calculated based on the image positions of the calibration objects in the calibration images corresponding to the two sonars. The calibration images are acquired by image sensors. Based on the first position relationship, the second position relationship between the image sensors integrated in the two sonars can be calculated. Since the position of the image sensor in the sonar is fixed, the relative position between the image sensor and the center of the sonar remains unchanged. Thus, after calculating the second position relationship, the position relationship between the centers of the two sonars can be calculated based on the second position relationship and the third position relationship between the image sensor and the center of the sonar.

[0142] From the foregoing content, it can be known that there are two situations for the position of the sonar-integrated image sensor in the sonar. In one situation, the center of the image sensor overlaps with the center of the sonar. In this case, after calculating the second positional relationship between the sonar-integrated image sensors, the second positional relationship can be directly used as the positional relationship between the centers of the sonars.

[0143] In another case, the center of the image sensor does not overlap with the center of the sonar. In this case, the third positional relationship between the center of the image sensor and the sonar can be obtained in advance. In this way, after calculating the second positional relationship between the image sensors integrated with the sonar, the positional relationship between the centers of the sonars can be calculated based on the second positional relationship and the third positional relationship.

[0144] Step D: Based on the center position relationship and the component position relationship, determine the position relationship between the microphones in the microphone array of the second sonar and the center of the first sonar as the position relationship between the microphones in the microphone array of the second sonar and the first sonar.

[0145] The first sonar is one of the sonars, the second sonar is the sonar other than the first sonar, and the component position relationship is the position relationship between the center of the second sonar and the microphones in the microphone array.

[0146] Specifically, during the calibration process, one of the sonars can be selected as the first sonar and the other sonars can be selected as the second sonar. In this way, when the center position relationship between the centers of the sonars is obtained, the center position relationship between the center of the first sonar and the center of the second sonar can be determined for each second sonar according to the image position of the calibration object in the calibration image corresponding to the first sonar and the image position of the calibration object in the calibration image corresponding to the second sonar. Then, based on the center position relationship and the component position relationship corresponding to the second sonar, the position relationship between the center of the first sonar and the microphone in the microphone array of the second sonar is determined as the position relationship between the second sonar and the microphone in the microphone array of the first sonar.

[0147] After such processing is performed for each second sonar, the positional relationship between the center of the first sonar and the microphones in the microphone array of each second sonar can be obtained as the positional relationship between each second sonar and the microphones in the microphone array of the first sonar.

[0148] In this method, calibration images collected by image sensors integrated with each sonar are obtained, and the center position relationship between the centers of each sonar can be accurately obtained according to the image position of the calibration object in each calibration image, so that based on the center position relationship and the component position relationship, the position relationship between the center of the first sonar and the microphone in the microphone array of the second sonar can be accurately determined. It can be seen that by applying this calibration method, the position relationship between the microphones in the microphone array between the second sonar and the first sonar can be accurately calibrated, so that the position relationship can be used in the sound source localization process to accurately locate the sound source in the traffic road and improve the sound source localization accuracy.

[0149] In the second calibration method, after each sonar is installed and set up, the position of each sonar remains unchanged. In this case, an image captured by an image acquisition device and including each sonar within a field of view can be obtained, and the image positions of the first sonar and the second sonar in the image can be detected. According to the detected image positions, the relative position relationship between the first sonar and the second sonar in the image coordinate system is determined, and then according to the conversion relationship between the image coordinate system and the world coordinate system, the relative position relationship between the first sonar and the second sonar in the image coordinate system is converted into the relative position relationship between the first sonar and the second sonar in the world coordinate system. Finally, according to the relative position relationship between the first sonar and the second sonar in the world coordinate system, and the position relationship between the center of the sonar and the sonar, the position relationship between the center of the first sonar and the microphone in the microphone array of the second sonar is determined as the position relationship between the second sonar and the microphone in the microphone array between the first sonar.

[0150] The specific implementation method of locating the sound source position according to the audio array corresponding to a count value in the above step S303 is described below.

[0151] In one embodiment of the present application, the electronic device can determine a first position of a microphone of a main sonar among at least two sonars relative to a main sonar center of the main sonar; based on a positional relationship between microphones in a microphone array between a main sonar and an auxiliary sonar among at least two sonars, transform the first position to obtain a second position of the microphone of the auxiliary sonar relative to the main sonar center; integrate the first position and the second position to obtain a position array; and use the obtained audio array and position array to locate the angular coordinates of a sound source in a traffic road relative to the main sonar center.

[0152] The principle of sound source localization is to calculate the angle of the sound source position relative to the array center of the microphone array by the time difference when the sound source reaches each microphone.

[0153] The main sonar mentioned above can be considered as: the sonar with the sonar center as the center of the array, and the auxiliary sonar can be considered as the other sonars among the sonars except the main sonar.

[0154] When selecting the main and auxiliary sonars, you can select one sonar from each sonar as the main sonar and the other sonars as auxiliary sonars.

[0155] The above position array is obtained by integrating the first position and the second position, and can be considered as a one-dimensional array of the positions of all microphones in the microphone array of each sonar relative to the main sonar center of the main sonar.

[0156] Specifically, the position of the microphone in the sonar is fixed, and the electronic device can obtain in advance the position of the microphone in the main sonar relative to the center of the sonar as the first position.

[0157] After calibrating the positional relationship between the microphones in the microphone arrays of each sonar, the relative positional relationship between the microphones in the microphone array of the main sonar and the microphones in the microphone array of the auxiliary sonar can be obtained from the calibrated positional relationship. In this way, the second position of the microphones in the microphone array of the auxiliary sonar relative to the center of the main sonar can be determined based on the first position of the microphones of the main sonar relative to the center of the main sonar and the relative positional relationship between the microphones in the microphone array of the main sonar and the microphones in the microphone array of the auxiliary sonar.

[0158] When the first position and the second position are combined, the positions corresponding to the sonars can be arranged one by one in a sonar unit in such a manner that the position corresponding to one sonar is arranged after the position corresponding to another sonar, thereby obtaining a position array.

[0159] In the above position array, the position of each microphone relative to the center of the main sonar can be expressed as coordinates in a target coordinate system. The target coordinate system can be a three-dimensional space coordinate system established with the position of the center of the main sonar as the coordinate origin. The above angular coordinates can also be coordinates in the target coordinate system.

[0160] For example, see Figure 5 , shows a target coordinate system XYZ, in which the angular coordinates of point Q include the pitch angle and horizontal angle , where the pitch angle is the angle between the line OQ between the origin and point Q and the X-axis, and the horizontal angle is the angle between the orthographic projection of the line OQ on the plane where the XY axis is located and the X-axis.

[0161] After integrating the first position and the second position to obtain the position array, the pitch angle and horizontal angle of the sound source in the traffic road in the target coordinate system can be calculated according to the position array and the audio array, based on the principle of sound source positioning (calculating the angle of the sound source position relative to the center of the main sonar by the time difference between the sound source reaching each microphone). The angular coordinates formed by the calculated pitch angle and horizontal angle are the angular coordinates of the sound source in the traffic road in the target coordinate system.

[0162] As can be seen from the above, when the solution provided in the embodiment of the present application is used to locate the sound source on the traffic road, the first position is the position of the microphone of the main sonar relative to the center of the main sonar, and the second position is the position of the microphone of the auxiliary sonar relative to the center of the main sonar. The first position and the second position are integrated to obtain the position array, and the position of the microphone of each sonar relative to the center of the main sonar is obtained. In this way, according to the position array and the audio array, the angular coordinates of the sound source in the traffic road relative to the center of the main sonar can be accurately located. Therefore, the traffic road sound source localization solution provided in the embodiment of the present application can improve the accuracy of sound source localization.

[0163] When the sonar is integrated with an image sensor, the steps mentioned in the following embodiments may also be performed.

[0164] In one embodiment of the present application, the sonar is integrated with an image sensor, and the field of view of the image sensor covers the road area.

[0165] In this case, after the sonar is set up, the detection area of ​​the microphone in the sonar is fixed, and the field of view of the image sensor in the sonar is also fixed. The detection area of ​​the microphone and the field of view of the image sensor both include the road area. If there is an object on the traffic road, the image of the object on the sensor target surface formed by the image sensor with a fixed focal length, fixed pixel point physical size and fixed pixels is fixed and unique. In this way, after the sonar is set up, the mapping relationship between the sound source position on the traffic road located by audio data and the image position of the sound source in the image collected by the image sensor can be calibrated.

[0166] During the sound source localization process, the electronic device can, on the one hand, locate the sound source position of the sound source in the traffic road based on the positional relationship between the audio array corresponding to a count value and the microphones in the microphone arrays between different sonars; on the other hand, it can obtain the target image captured by the target image sensor at the acquisition time corresponding to the audio data in the audio array, wherein the target image sensor is an image sensor integrated in the main sonar of at least two sonars.

[0167] After the electronic device locates the sound source position corresponding to the audio array based on the positional relationship between the audio array and the microphones in the microphone arrays of different sonars, it can map the sound source position corresponding to the audio array to the target image according to the above mapping relationship to obtain the image position.

[0168] Object recognition is performed on the object candidate area containing the image position in the target image. If the object is successfully recognized, the recognized object is determined as the sound source in the traffic road.

[0169] Specifically, when the main sonar is set up, the size of the vehicle area in the image captured by the target image sensor of the main sonar can be considered unchanged. After the image position is obtained through the mapping, the object candidate area in the target image can be determined according to the image position and the size of the vehicle area, so as to perform object recognition on the object candidate area. If the object is successfully recognized, the recognized object is determined as the sound source in the traffic road.

[0170] For example, when determining the object candidate area in the target image, the first pixel column located to the left of the image position and at a distance of the width from the image position, and the second pixel column located to the right of the image position and at a distance of the width from the image position can be determined in the target image based on the width of the vehicle area; and the first pixel row located above the image position and at a distance of the height from the image position, and the second pixel row located below the image position and at a distance of the height from the image position can be determined in the target image based on the height of the vehicle area, so that the area surrounded by the first pixel column, the second pixel column, the first pixel row and the second pixel row is the object candidate area.

[0171] From the above, it can be seen that when the solution provided in the embodiment of the present application is used to locate the sound source on the traffic road, the sound source position corresponding to the audio array can be mapped to the target image, and object recognition can be performed on the object candidate area containing the image position in the target image. If the object is successfully recognized, the recognized object can be determined as the sound source in the traffic road. In this way, the sound source in the traffic road can be clearly displayed in the target image, which is convenient for users to check.

[0172] During the sound source localization process, the sonar may also be processed in the manners mentioned in the following embodiments.

[0173] In one embodiment of the present application, during the sound source localization process, a synchronous reset signal may be periodically sent to the sonar, so that the sonar receiving the reset signal resets the count value of the audio data collected by its own microphone array.

[0174] The above-mentioned synchronous reset signal can be a preset high or low level signal or other high-speed synchronous trigger signal.

[0175] After the sonar receives the synchronization reset signal, the sonar can start counting the audio data collected by the microphones included in its own microphone array from the beginning. In this case, the count value corresponding to the audio data can be regarded as: the collection order of the audio data in the synchronization reset period where the audio data is located, wherein the synchronization reset period where the audio data is located is the synchronization reset period from the first response time before the collection time of the audio data to the second response time after the collection time of the audio data.

[0176] Specifically, in the case where the electronic device is a device other than each sonar, the device is connected to each sonar for communication, and can periodically send a synchronization reset signal to the sonar. Each time the sonar receives the synchronization reset signal, it can respond to the synchronization reset signal and reset the count value of the audio data collected by the microphone array of the sonar after the response signal, that is, count the audio data collected by the microphone array of the sonar after the response signal from the beginning.

[0177] The above device periodically sends a synchronization reset signal to the sonar, and each time the sonar receives the synchronization reset signal, it starts counting the audio data collected by the microphone array from the beginning. In this way, when the audio data generated by each sonar is obtained, the audio data generated by each sonar in the same cycle is obtained.

[0178] In addition, the above-mentioned device can be connected to each sonar through an IO lead or other physical connection. The delay difference of the above-mentioned device sending a synchronous trigger signal to each sonar is much lower than the collection interval of the sonar's microphone array. In this way, after each sonar receives the synchronous reset signal, it can be considered that each sonar synchronously resets the count value of the audio data collected by its own microphone array.

[0179] In the case where the above-mentioned electronic device is one of the sonars, the above-mentioned electronic device can periodically send a synchronous reset signal to other sonars, so that the sonar receiving the reset signal resets the count value of the audio data collected by its own microphone array, and the electronic device also synchronously resets the count value of the audio data collected by its own microphone array.

[0180] In this case, the master sonar and the slave sonars can be connected through IO leads or other physical connections. Each time the master sonar sends a synchronous reset signal to each slave sonar, each slave sonar resets the count value of the audio data collected by its own microphone array, and the master sonar also resets the count value of the audio data collected by its own microphone array. Similarly, this also allows each sonar to synchronously reset the count value of the audio data collected by its own microphone array. The master sonar periodically performs the above operation, and each sonar also periodically and synchronously resets the count value of the audio data collected by its own microphone array, thereby improving the consistency of each sonar counting the data collected by its own microphone array, and then improving the consistency of the audio data generated by each sonar. In this way, the audio data generated by each sonar is obtained, and the sound source is located based on the audio data, which can improve the accuracy of sound source location.

[0181] Similarly, in the case where the above-mentioned electronic device is a device other than each sonar, the device can also achieve the purpose of periodically and synchronously resetting the count value of the audio data collected by each microphone array by each sonar by periodically sending a synchronization reset signal to each sonar. Therefore, no matter whether the above-mentioned electronic device is one of the sonars or other devices other than each sonar, the method mentioned in the embodiment of the present application can be used to periodically and synchronously reset the count value of the audio data collected by each microphone array by each sonar, thereby improving the consistency of each sonar counting the audio data collected by its own microphone array, and then improving the consistency of the audio data generated by each sonar. In this way, the audio data generated by each sonar is obtained, and the sound source is located based on the audio data, which can improve the accuracy of sound source positioning.

[0182] like Figure 6 , shows the data collected by the microphones in the microphone arrays of the two sonars after they receive the synchronization trigger signal (i.e., the synchronization reset signal). After the two sonars receive the synchronization trigger signal, the count value of the audio data collected by the microphones in the subsequent microphone arrays can be reset, that is, the audio data collected by the microphones in the microphone arrays can be counted from the beginning after the synchronization trigger signal is received.

[0183] The sending cycle of the above reset signal is described below.

[0184] In one embodiment of the present application, a sending period of the above reset signal is less than or equal to a set period.

[0185] If the sending period of the reset signal is longer, the more data the sonar microphone array collects in one period, the more bits of the count value, and the more resources the count value needs to occupy; conversely, the shorter the sending period of the reset signal is, the less data the sonar microphone array collects in one period, the fewer bits of the count value, and the fewer resources the count value needs to occupy. Therefore, setting the sending period of the reset signal to be less than or equal to the set period can save resources.

[0186] The above-mentioned setting period can be determined according to any one of the following two implementation methods.

[0187] In a first implementation, the set period is determined according to the number of bits of the count value included in the channel data and the frequency at which the microphones in the microphone array of the sonar collect audio data.

[0188] The number of bits of the count value contained in the above channel data determines the upper limit of the count, which can be regarded as the maximum number of data collected by the sonar microphone array in one cycle. If this number is exceeded, the sonar can only start counting from the beginning again, which will cause different data collected by the microphone array of the same sonar in one cycle to have the same count, resulting in matching errors during subsequent synchronization matching.

[0189] For example, if the number of count digits is 1, the counts are only 0 and 1, and a maximum of two data can be counted, so the upper limit of the count value is 2. If the number of count digits is K, the maximum number of counts is 2. K The upper limit of the count value is 2. K .

[0190] The acquisition frequency of the microphone array of the sonar mentioned above refers to the amount of data collected by the microphone array within 1 second.

[0191] According to the number of bits of the count value contained in the above-mentioned channel data and the frequency of the microphones in the sonar microphone array collecting audio data, the maximum sending period of the above-mentioned reset signal can be calculated. In this way, a period not greater than the maximum sending period is determined as the above-mentioned set period. This can avoid the problem that different data collected by the microphone array of the same sonar within one period have the same count, thereby ensuring that subsequent synchronous matching can be accurate and error-free, thereby improving the accuracy of sound source positioning.

[0192] For example, the above-mentioned setting period T is determined according to the following expression:

[0193]

[0194] in, Indicates the number of bits of the count value contained in the channel data. Indicates the acquisition frequency of the sonar microphone array.

[0195] The set period determined by the above expression is smaller than the maximum sending period of the reset signal. Therefore, using the period determined by the above expression as the set period can ensure that subsequent synchronization matching can be accurate, thereby improving the accuracy of sound source positioning.

[0196] In a second implementation, the above-mentioned set period may be determined manually based on past experience.

[0197] Corresponding to the above-mentioned sound source localization method, an embodiment of the present application further provides a sound source localization device.

[0198] In one embodiment of the present application, see Figure 7, a structural schematic diagram of a sound source localization device is provided. In this embodiment, at least two sonars are set up above the traffic road, and the detection ranges of the at least two sonars have overlapping areas, and the overlapping areas include road areas. The microphone arrays in different sonars synchronously collect data, including:

[0199] The data acquisition module 701 is used to obtain channel data output by each microphone in the microphone array of each sonar, wherein each channel data output by each microphone includes: audio data collected by the microphone and a count value of the audio data collected by the microphone;

[0200] The array acquisition module 702 is used to match the count values ​​included in each channel data, and arrange the audio data with the same count value in sequence in sonar units to obtain an audio array corresponding to the count value;

[0201] The sound source localization module 703 is used to locate the sound source in the traffic road according to the obtained audio array and the position relationship between the microphones in the microphone arrays of different sonars to obtain the sound source position.

[0202] It can be seen from the above that when the solution provided in the embodiment of the present application is used to locate the sound source, the microphone arrays of each sonar synchronously collect data, and the channel data obtained by the electronic device includes audio data and count values. It can be seen that the count value corresponds to the audio data, so that the count value also corresponds to the collection time of the audio data. The count values ​​included in each channel data are matched, and the audio data with the same count value are arranged in sequence in units of sonar to obtain an audio array corresponding to the count value. The audio array can be regarded as: at the collection time corresponding to the corresponding count value, the microphone array of each sonar contains an array of audio data synchronously collected by the microphone, which is equivalent to increasing the number of microphone channels in the microphone array, and the array area is also equivalent to increasing to the area determined according to multiple sonars. Therefore, sound source positioning is performed according to the positional relationship between the audio array and the microphones in the microphone array between different sonars, which can improve the positioning accuracy of sound source positioning.

[0203] In one embodiment of the present application, the sound source localization module 703 is specifically used to:

[0204] Determine a first position of a microphone of a main sonar among the at least two sonars relative to a main sonar center of the main sonar; transform the first position based on a positional relationship of microphones in a microphone array between the main sonar and the auxiliary sonar among the at least two sonars to obtain a second position of the microphone of the auxiliary sonar relative to the main sonar center; integrate the first position and the second position to obtain a position array; and locate the angular coordinate of a sound source in the traffic road relative to the main sonar center using the obtained audio array and the position array.

[0205] As can be seen from the above, when the solution provided in the embodiment of the present application is used to locate the sound source on the traffic road, the first position is the position of the microphone of the main sonar relative to the center of the main sonar, and the second position is the position of the microphone of the auxiliary sonar relative to the center of the main sonar. The first position and the second position are integrated to obtain the position array, and the position of the microphone of each sonar relative to the center of the main sonar is obtained. In this way, according to the position array and the audio array, the angular coordinates of the sound source in the traffic road relative to the center of the main sonar can be accurately located. Therefore, the traffic road sound source localization solution provided in the embodiment of the present application can improve the accuracy of sound source localization.

[0206] In one embodiment of the present application, the sonar is integrated with an image sensor, and the field of view of the image sensor covers the road area.

[0207] The device also includes:

[0208] An image acquisition module, used to obtain a target image captured by a target image sensor at a capture time corresponding to the audio data in the audio array, wherein the target image sensor is: an image sensor integrated in a main sonar of the at least two sonars;

[0209] A position mapping module, used to map the sound source position corresponding to the audio array to the target image to obtain an image position;

[0210] An object recognition module, configured to perform object recognition on an object candidate region in the target image that includes the image position;

[0211] The sound source determination module is used to determine the identified object as the sound source in the traffic road if the object is successfully identified.

[0212] From the above, it can be seen that when the solution provided in the embodiment of the present application is used to locate the sound source on the traffic road, the sound source position corresponding to the audio array can be mapped to the target image, and object recognition can be performed on the object candidate area containing the image position in the target image. If the object is successfully recognized, the recognized object can be determined as the sound source in the traffic road. In this way, the sound source in the traffic road can be clearly displayed in the target image, which is convenient for users to check.

[0213] In one embodiment of the present application, in two adjacent sonars, the distance between a first microphone and a second microphone, which are the shortest apart, is less than or equal to a first set distance, wherein the first microphone and the second microphone belong to two adjacent sonars respectively, and the first set distance is determined based on the wavelength of the sound emitted by the sound source to be located.

[0214] In this solution, since each sonar is used to locate the sound source of the traffic road, there is a certain distance between the traffic road and each sonar. Among the adjacent sonars of each sonar, the smaller the distance between the first microphone and the second microphone, which are the shortest apart, the smaller the distance between adjacent sonars. Then, each sonar can be regarded as a sonar containing more microphones and a larger number of microphone channels. Therefore, by using a sonar with more microphones and a larger number of microphone channels to locate the sound source, the accuracy of sound source positioning can be improved. Therefore, when installing each sonar, the accuracy of sound source positioning can be improved by setting the installation distance between the centers of adjacent sonars to be less than or equal to the first set distance.

[0215] The first set distance is determined according to the wavelength of the sound emitted by the sound source to be located, and each sonar is installed according to the first set distance. This can make the sound source positioning using the installed sonars more sensitive to the sound of the wavelength of the sound emitted by the sound source to be located, so that the sound source emitting the sound of this wavelength can be positioned more accurately, thereby improving the accuracy of sound source positioning.

[0216] In one embodiment of the present application, the first set distance is half the wavelength of the sound emitted by the sound source to be located. When installing each sonar, setting the distance between the first microphone and the second microphone, which are the shortest distances between two adjacent sonars, to be less than or equal to half the wavelength of the sound emitted by the sound source to be located can improve the accuracy of sound source location.

[0217] In one embodiment of the present application, the installation distance between the centers of adjacent sonars is less than or equal to a second set distance, and the second set distance is determined according to the first set distance.

[0218] When installing each sonar, when the sonar rotates a certain angle about its own center, the position of each microphone in the sonar will change, but the position of the sonar center will not change. Therefore, using the position of the center of each sonar to install the sonar, there is no need to consider the rotation of the sonar about its own center. This is more convenient than using the position of the microphone of each sonar to install the sonar.

[0219] Since the second set distance is determined according to the first set distance, when the installation distance between the centers of adjacent sonars satisfies the standard of being less than or equal to the second set distance, the distance between the first microphone and the second microphone of the adjacent sonars can also meet the standard of being less than or equal to the first set distance. Therefore, each sonar is installed according to the standard that the installation distance between the centers of adjacent sonars is less than or equal to the second set distance, thereby improving the accuracy of sound source positioning.

[0220] In one embodiment of the present application, the device further includes:

[0221] The signal sending module is used to periodically send a synchronous reset signal to the sonar, so that the sonar receiving the reset signal resets the count value of the audio data collected by its own microphone array, and the sending period of the reset signal is less than or equal to the set period.

[0222] In this scheme, by periodically sending a synchronous reset signal to the sonar, each sonar can periodically and synchronously reset the count value of the audio data collected by its own microphone array, thereby improving the consistency of each sonar's counting of the audio data collected by its own microphone array, and further improving the consistency of the audio data generated by each sonar. In this way, the audio data generated by each sonar is obtained, and the sound source is located based on the audio data, which can improve the accuracy of sound source positioning.

[0223] If the sending cycle of the above reset signal is longer, the more data the sonar microphone array collects in one cycle, the more bits used to represent the data count, and the more resources are needed for data counting; conversely, the shorter the sending cycle of the above reset signal is, the less data the sonar microphone array collects in one cycle, the fewer bits used to represent the data count, and the fewer resources are needed for data counting. Therefore, setting the sending cycle of the above reset signal to be less than or equal to the set cycle can save resources.

[0224] In one embodiment of the present application, the electronic device is one of the at least two sonars;

[0225] The signal sending module is specifically used for:

[0226] Periodically sends a synchronization reset signal to other sonars and resets the count value of the audio data collected by its own microphone array.

[0227] In this solution, each sonar can periodically and synchronously reset the count value of the audio data collected by its own microphone array, thereby improving the consistency of each sonar's counting of the data collected by its own microphone array, and further improving the consistency of the audio data generated by each sonar. In this way, the audio data generated by each sonar is obtained, and the sound source is located based on the audio data, which can improve the accuracy of sound source positioning.

[0228] In one embodiment of the present application, the set period is determined according to the number of bits of the count value included in the channel data and the frequency at which the microphones in the microphone array collect audio data.

[0229] In this scheme, the maximum sending period of the reset signal can be calculated based on the number of bits of the count value contained in the above-mentioned channel data and the frequency of the microphones in the sonar microphone array collecting audio data. In this way, a period not greater than the maximum sending period is determined as the above-mentioned set period. This can avoid the problem that different data collected by the same sonar microphone array within one period have the same count, thereby ensuring that subsequent synchronous matching can be accurate and error-free, thereby improving the accuracy of sound source positioning.

[0230] In one embodiment of the present application, the set period T is determined according to the following expression:

[0231]

[0232] in, represents the number of bits of the count value contained in the channel data, Indicates the acquisition frequency of the microphone array of the sonar.

[0233] In this solution, the set period determined by the above expression is smaller than the maximum sending period of the reset signal. Therefore, using the period determined by the above expression as the set period can ensure that subsequent synchronization matching can be accurate, thereby improving the accuracy of sound source positioning.

[0234] In one embodiment of the present application, the sonar is integrated with an image sensor;

[0235] The position relationship of the microphones is calibrated in the following manner:

[0236] Obtaining calibration images collected by the at least two sonar-integrated image sensors, wherein calibration objects are placed in overlapping areas of the field of view of the sonar-integrated image sensors;

[0237] Detecting the image position of the calibration object in each calibration image;

[0238] According to the image position of the calibration object in each calibration image, the center position relationship between the centers of each sonar is obtained;

[0239] Based on the center position relationship and the component position relationship, the position relationship between the microphone in the microphone array of the second sonar and the center of the first sonar is determined as the position relationship between the microphones in the microphone array between the second sonar and the first sonar, wherein the first sonar is one of the sonars, the second sonar is the sonar other than the first sonar among the sonars, and the component position relationship is the position relationship between the center of the second sonar and the microphones in the microphone array.

[0240] In this scheme, calibration images collected by image sensors integrated by each sonar are obtained, and the center position relationship between the centers of each sonar can be accurately obtained according to the image position of the calibration object in each calibration image, so that based on the center position relationship and the component position relationship, the position relationship between the center of the first sonar and the microphone in the microphone array of the second sonar can be accurately determined. It can be seen that by applying this calibration method, the position relationship between the microphones in the microphone array of the second sonar and the first sonar can be accurately determined, so that the position relationship can be used in the sound source localization process to accurately locate the sound source in the traffic road and improve the sound source localization accuracy.

[0241] The present application also provides an electronic device, such as Figure 8 As shown, including:

[0242] Memory 801, used for storing computer programs;

[0243] The processor 802 is used to execute the program stored in the memory 801, and implements the following steps:

[0244] Obtain channel data output by each microphone in the microphone array of each sonar, wherein each channel data output by each microphone includes: audio data collected by the microphone and a count value of the audio data collected by the microphone;

[0245] Matching the count values ​​included in each channel data, arranging the audio data with the same count value in sequence in sonar units, and obtaining an audio array corresponding to the count value;

[0246] According to the obtained audio array and the positional relationship between microphones in different sonar microphone arrays, the sound source in the traffic road is located to obtain the sound source position.

[0247] The description of the processor 802 executing the program stored in the memory 801 to implement other sound source localization methods can be found in the aforementioned method embodiments, which will not be repeated here.

[0248] Furthermore, the electronic device may further include a communication bus and / or a communication interface, and the processor 802, the communication interface, and the memory 801 communicate with each other via the communication bus.

[0249] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0250] The communication interface is used for communication between the above electronic device and other devices.

[0251] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0252] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0253] In another embodiment provided in the present application, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned sound source localization methods are implemented.

[0254] In another embodiment provided by the present application, a computer program product including instructions is also provided, and when the computer is run on the computer, the computer is enabled to execute any of the sound source localization methods in the above embodiments.

[0255] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a solid-state hard disk (SSD), etc.

[0256] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0257] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, electronic device, computer-readable storage medium, and computer product embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0258] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.

Claims

1. A method for localizing a traffic road sound source, characterized in that: At least two sonars are set up above the traffic road, and the detection ranges of the at least two sonars have overlapping areas, and the overlapping areas include road areas. The microphone arrays in different sonars synchronously collect data, and the method includes: Obtain channel data output by each microphone in the microphone array of each sonar, wherein each channel data output by each microphone includes: audio data collected by the microphone and a count value of the audio data collected by the microphone. For each sonar, the sonar starts from the first audio data collected by the microphone included in its own microphone array, and counts the audio data collected by each microphone included in its own microphone, and each audio data output by each microphone corresponds to a count value; Matching the count values ​​included in each channel data, arranging the audio data with the same count value in sequence in sonar units, and obtaining an audio array corresponding to the count value; According to the obtained audio array and the positional relationship between microphones in different sonar microphone arrays, the sound source in the traffic road is located to obtain the sound source position.

2. The method according to claim 1, characterized in that: The method of locating the sound source in the traffic road according to the obtained audio array and the positional relationship between microphones in different sonar microphone arrays includes: Determining a first position of a microphone of a master sonar of the at least two sonars relative to a master sonar center of the master sonar; Based on the positional relationship between microphones in a microphone array between a main sonar and an auxiliary sonar in the at least two sonars, transforming the first position to obtain a second position of the microphone of the auxiliary sonar relative to the center of the main sonar; Integrating the first position and the second position to obtain a position array; The obtained audio array and the position array are used to locate the angular coordinates of the sound source in the traffic road relative to the center of the main sonar.

3. The method according to claim 1, characterized in that The sonar is integrated with an image sensor, and the field of view of the image sensor covers the road area; the method further includes: Obtaining a target image captured by a target image sensor at a capture time corresponding to the audio data in the audio array, wherein the target image sensor is: an image sensor integrated in a main sonar of the at least two sonars; Mapping the sound source position corresponding to the audio array to the target image to obtain the image position; Performing object recognition on an object candidate region in the target image that includes the image position; If the object is successfully identified, the identified object is determined as the sound source in the traffic road.

4. The method according to any one of claims 1 to 3, characterized in that The deployment positions of the at least two sonars meet the following conditions: In two adjacent sonars, a distance between a first microphone and a second microphone, which are the shortest between the two microphones, is less than or equal to a first set distance, wherein the first microphone and the second microphone belong to two adjacent sonars respectively, and the first set distance is determined according to a wavelength of a sound emitted by a sound source to be located; or The installation distance between the centers of adjacent sonars is less than or equal to a second set distance, wherein the second set distance is determined based on the shortest distance between microphones in adjacent sonars.

5. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: Periodically sending a synchronous reset signal to the sonar, so that the sonar receiving the reset signal resets the count value of the audio data collected by each microphone in its microphone array, and the sending period of the reset signal is less than or equal to the set period; If the sonar is one of the at least two sonars, the count value of the audio data collected by each microphone in the microphone array of the sonar is reset.

6. The method according to claim 5, characterized in that The set period is determined according to the number of bits of the count value included in the channel data and the frequency of the microphones in the microphone array collecting audio data.

7. The method according to claim 6, characterized in that The set period T is determined according to the following expression: ； in, represents the number of bits of the count value contained in the channel data, Indicates the frequency at which the microphones in the microphone array collect audio data.

8. The method according to claim 1 or 2, characterized in that: The sonar is integrated with an image sensor; The position relationship of the microphones is calibrated in the following manner: Obtaining calibration images collected by the at least two sonar-integrated image sensors, wherein calibration objects are placed in overlapping areas of the field of view of the sonar-integrated image sensors; Detecting the image position of the calibration object in each calibration image; According to the image position of the calibration object in each calibration image, the center position relationship between the centers of each sonar is obtained; Based on the center position relationship and the component position relationship, the position relationship between the microphone in the microphone array of the second sonar and the center of the first sonar is determined as the position relationship between the microphones in the microphone array between the second sonar and the first sonar, wherein the first sonar is one of the sonars, the second sonar is the sonar other than the first sonar among the sonars, and the component position relationship is the position relationship between the center of the second sonar and the microphones in the microphone array.

9. A traffic road sound source localization device, characterized in that: At least two sonars are set up above the traffic road, and the detection ranges of the at least two sonars have overlapping areas, and the overlapping areas include road areas. The microphone arrays in different sonars synchronously collect data, and the device includes: A data acquisition module is used to obtain channel data output by each microphone in the microphone array of each sonar, wherein each channel data output by each microphone includes: audio data collected by the microphone and a count value of the audio data collected by the microphone. For each sonar, the sonar starts from the first audio data collected by the microphone included in its own microphone array, and counts the audio data collected by each microphone included in its own microphone, and each audio data output by each microphone corresponds to a count value; An array acquisition module is used to match the count values ​​included in each channel data, and arrange the audio data with the same count value in sequence in sonar units to obtain an audio array corresponding to the count value; The sound source localization module is used to locate the sound source in the traffic road according to the obtained audio array and the position relationship between the microphones in the microphone arrays of different sonars to obtain the sound source position.

10. The device according to claim 9, characterized in that The sound source localization module is specifically used to determine a first position of a microphone of a main sonar among the at least two sonars relative to a main sonar center of the main sonar; based on a positional relationship between microphones in a microphone array between a main sonar and an auxiliary sonar among the at least two sonars, transform the first position to obtain a second position of the microphone of the auxiliary sonar relative to the main sonar center; and integrate the first position and the second position to obtain a position array; Using the obtained audio array and the position array, locate the angular coordinates of the sound source in the traffic road relative to the center of the main sonar; and / or The sonar is integrated with an image sensor, and the field of view of the image sensor covers the road area; the device also includes: an image acquisition module, which is used to obtain a target image captured by a target image sensor at the acquisition time corresponding to the audio data in the audio array, wherein the target image sensor is: an image sensor integrated in the main sonar of the at least two sonars; a position mapping module, which is used to map the sound source position corresponding to the audio array to the target image to obtain the image position; an object recognition module, which is used to perform object recognition on the object candidate area containing the image position in the target image; a sound source determination module, which is used to determine the recognized object as the sound source in the traffic road if the object is successfully recognized; and / or In two adjacent sonars, the distance between a first microphone and a second microphone, which are the shortest in distance, is less than or equal to a first set distance, wherein the first microphone and the second microphone belong to two adjacent sonars respectively, and the first set distance is determined according to the wavelength of the sound emitted by the sound source to be located; or the installation distance between the centers of adjacent sonars is less than or equal to a second set distance, wherein the second set distance is determined according to the shortest distance between the microphones in the adjacent sonars; and / or The device further includes: a signal sending module, which is used to periodically send a synchronous reset signal to the sonar, so that the sonar that receives the reset signal resets the count value of the audio data collected by its own microphone array, and the sending period of the reset signal is less than or equal to the set period; and / or The device is one of the at least two sonars; the signal sending module is specifically used to: periodically send a synchronization reset signal to other sonars, and reset the count value of the audio data collected by its own microphone array; and / or The setting period is determined according to the number of bits of the count value contained in the channel data and the frequency of the microphones in the microphone array collecting audio data; and / or The set period T is determined according to the following expression: ； in, represents the number of bits of the count value contained in the channel data, Indicates the frequency at which the microphones in the microphone array collect audio data; and / or The sonar is integrated with an image sensor; the positional relationship of the microphones is calibrated in the following manner: obtaining calibration images collected by the at least two sonar-integrated image sensors, wherein calibration objects are placed in overlapping areas of the field of view of the image sensors integrated with each sonar; detecting the image positions of the calibration objects in each calibration image; obtaining the center positional relationship between the centers of each sonar according to the image positions of the calibration objects in each calibration image; and determining the positional relationship between the microphones in the microphone array of the second sonar and the center of the first sonar based on the center positional relationship and the component positional relationship as the positional relationship between the microphones in the microphone array of the second sonar and the first sonar, wherein the first sonar is one of the sonars, the second sonar is the sonar other than the first sonar among the sonars, and the component positional relationship is the positional relationship between the center of the second sonar and the microphones in the microphone array.

11. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, for implementing any of the methods described in claims 1-8 when executing a program stored in a memory.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images