Sound source target positioning method and device and computer readable storage medium

Abstract

This application discloses a method, apparatus, and computer-readable storage medium for sound source target localization. The method includes: acquiring real-time images and audio data synchronized with the real-time images; determining whether speech data exists in the audio data; if so, performing audio localization processing on the speech data to determine the sound source location corresponding to the speech data in the real-time image; determining a sound source region within a preset range of the sound source location based on error compensation data; determining whether the sound source region includes a target object; if not, adjusting and expanding the error compensation data, and returning to the step of determining the sound source region within the preset range of the sound source location based on the error compensation data; if so, tracking and displaying the tracking area determined by the target object and the sound source region. Through the above methods, this application can improve the accuracy of sound source target localization.

Description

Technical Field

[0001] This application relates to the field of audio processing technology, and in particular to a method, apparatus and computer-readable storage medium for sound source target localization. Background Technology

[0002] In scenarios such as multi-person video conferences and multiple car horns, many factors can prevent accurate location of the sound source. For example, noisy environments and errors in audio acquisition and localization algorithms can affect the localization results. These interferences can often be eliminated by adding multi-directional audio acquisition devices or auxiliary equipment. However, this increases equipment costs. Simpler, more common scenarios don't require high costs, such as video conferences. Audio acquisition devices mounted on a whiteboard can only provide unidirectional sound source localization information. During video conferences, it's necessary to locate the speaker and project the image. If the audio acquisition and localization algorithm has a large error or the ambient noise is high, the obtained localization information may be even less accurate, failing to correctly track and amplify the speaker. Summary of the Invention

[0003] This application provides a method, apparatus, and computer-readable storage medium for locating sound sources, which solves the problem of low accuracy in sound source target localization in the prior art.

[0004] To address the aforementioned technical problems, this application provides a sound source target localization method, comprising: acquiring a real-time image and audio data synchronized with the real-time image; determining whether speech data exists in the audio data; if so, performing audio localization processing on the speech data to determine the sound source location corresponding to the speech data in the real-time image; determining a sound source region within a preset range of the sound source location based on error compensation data; determining whether the sound source region includes a target object; if not, adjusting and expanding the error compensation data, and returning to the step of determining the sound source region within the preset range of the sound source location based on the error compensation data; if so, tracking and displaying the tracking area determined by the target object and the sound source region.

[0005] Optionally, determining the sound source region within a preset range of the sound source location based on error compensation data includes: if moving the sound source location a predetermined distance toward the first boundary of the real-time image results in it exceeding the first boundary, then defining the region in the real-time image between the first boundary and a position at the predetermined distance from the first boundary as the sound source region; otherwise, determining whether moving the sound source location a predetermined distance toward the second boundary exceeds the second boundary; wherein the second boundary and the first boundary are positioned opposite each other in the real-time image; if it exceeds the second boundary, then defining the region in the real-time image between the second boundary and a position at the predetermined distance from the second boundary as the sound source region; if it does not exceed the second boundary, defining the region between the position of the sound source location after moving the predetermined distance toward the first boundary and the position of the sound source location after moving the predetermined distance toward the second boundary as the sound source region.

[0006] Optionally, the tracking display of the tracking area determined by the target object and the sound source region includes: using the width of the sound source region as the target screen width of the tracking area, and using the height of the region where the target object is located as the target screen height of the tracking area; and tracking display of the tracking area determined by the target screen width and the target screen height.

[0007] Optionally, the error compensation data is the result of dividing the product of the algorithm error of the audio positioning algorithm and the range expansion factor by the compensation factor, wherein the initial value of the range expansion factor is 1; the adjustment and expansion of the error compensation data includes: adjusting the range expansion factor by one, and updating the error compensation data using the adjusted range expansion factor.

[0008] Optionally, the tracking display of the tracking area determined by the target object and the sound source region includes: when the number of target objects is at least two, then the tracking display of the tracking area determined by at least two target objects and the sound source region.

[0009] Optionally, the tracking display of the tracking area determined by at least two target objects and the sound source region includes: determining the area enclosed by the maximum and minimum vertical coordinates and the maximum and minimum horizontal coordinates of the at least two target objects as the tracking area.

[0010] Optionally, determining whether speech data exists in the audio data includes: if audio data is detected in a state without speech, then storing audio sample points, truncated to a set length, sequentially into various storage locations of a cache file, and after each storage, determining whether the sample rate without speech in the cache file exceeds a set probability threshold; if it exceeds, then determining that there is no speech data in the audio data; otherwise, after the storage location is full, determining whether the sample point in the last storage location is without speech data; if so, then moving the audio sample point forward one storage location and storing a new audio sample point in the last empty storage location; if not, then determining that there is speech data in the audio data.

[0011] Optionally, determining whether there is speech data in the audio data includes: if audio data is detected in the presence of speech, storing audio sample points extracted from the audio data according to a set length into each storage location of the cache file in sequence, and determining whether all audio sample points in the storage locations are without speech data; if not, determining that there is speech data in the audio data.

[0012] To address the aforementioned technical problems, a second aspect of this application provides a sound source target localization device, comprising a processor and a memory coupled to each other; the memory stores a computer program, and the processor executes the computer program to implement the sound source target localization method provided in the first aspect.

[0013] To address the aforementioned technical problems, a third aspect of this application provides a computer-readable storage medium storing program data, which, when executed by a processor, implements the sound source target localization method provided in the first aspect.

[0014] The beneficial effects of this application are as follows: Unlike existing technologies, this application acquires real-time images and audio data synchronized with the real-time images. It determines whether speech data exists in the audio data. If so, it performs audio localization processing on the speech data to determine the sound source location corresponding to the speech data in the real-time image. Based on error compensation data, it determines the sound source region within a preset range of the sound source location and whether the sound source region includes the target object. If not, it adjusts and expands the error compensation data and returns to the step of determining the sound source region within the preset range of the sound source location based on the error compensation data. If the target object is included, it tracks and displays the tracking area determined by the target object and the sound source region. This application can further determine the sound source region including the target object based on the initial sound source location obtained through audio localization processing, using error compensation data, thereby improving the accuracy of sound source target localization. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a flowchart illustrating an embodiment of the sound source target localization method of this application;

[0017] Figure 2 This is a flowchart illustrating an embodiment of step S12 of this application;

[0018] Figure 3 This is a schematic diagram of an embodiment of audio sample point storage according to this application;

[0019] Figure 4 This is a schematic diagram of another embodiment of audio sample point storage in this application;

[0020] Figure 5 This application Figure 4 A schematic diagram of an embodiment of shifting the middle sample point to the left;

[0021] Figure 6 This is a flowchart illustrating another embodiment of step S12 of this application;

[0022] Figure 7 This is a flowchart illustrating an embodiment of step S14 of this application;

[0023] Figure 8 This is a schematic diagram of an embodiment of the real-time image of this application;

[0024] Figure 9 This is a schematic diagram of another embodiment of the real-time image of this application;

[0025] Figure 10 This is a schematic block diagram of an embodiment of the sound source target localization device of this application;

[0026] Figure 11 This is a schematic block diagram of an embodiment of a computer-readable storage medium of this application. Detailed Implementation

[0027] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0028] The terms "first" and "second" in this application are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such processes, methods, products, or apparatus.

[0029] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0030] To address the interference caused by errors in audio localization algorithms, this application provides a method for compensating for errors in one-way voice localization and tracking targets, as described in the following embodiments.

[0031] Please see Figure 1 , Figure 1 This is a schematic flowchart illustrating an embodiment of the sound source target localization method of this application. It should be noted that if substantially the same result is obtained, this embodiment does not necessarily reflect that outcome. Figure 1 The illustrated process sequence is limited. This embodiment includes the following steps:

[0032] Step S11: Acquire real-time images and audio data synchronized with the real-time images.

[0033] Images and audio data, for example, are acquired using camera and recording equipment to capture real-time images and audio data including the sound source target. Multiple consecutive frames of real-time images can be combined to form a continuous video. The sound source target is, for example, a person, another designated animal, or an object, and the audio data may include speech data of a person speaking, noise data, etc.

[0034] One implementation scenario is in a video conferencing setting, where real-time image and audio data can be collected image and audio data of the participants. Sometimes, there may be multiple participants on a single screen, meaning multiple potential sound source targets exist within a single real-time image. Among these potential sound source targets, only one or a few may speak at a time. This embodiment aims to accurately locate the sound source target in the real-time image.

[0035] Step S12: Determine whether there is speech data in the audio data.

[0036] This step determines whether the audio data contains speech data generated by a person speaking.

[0037] If voice data exists, proceed to step S13; if no voice data exists, do not perform tracking and zoom display on the real-time image.

[0038] Optionally, before step S12, a target detection network can be used to perform target detection in the real-time image to determine whether the target object is included. If no target object is detected in the real-time image, it indicates that there is no target object in the current real-time image or the target object has left the range of the real-time image. In this case, the real-time image is not tracked and displayed, and the original real-time image is displayed directly. If a target object is detected in the real-time image, the process proceeds to step S12 to continue audio tracking.

[0039] Please see Figure 2 If audio data is detected in the absence of voice, this step may include the following steps:

[0040] Step S21: Store the audio sample points, which are cut off according to the set length, into the respective storage locations of the cache file in sequence.

[0041] Create an audio data sample point cache buff file, which includes n storage bits, a relative time of 1.5 seconds, a voice acquisition misrecognition rate of H%, initialize sample point i = 0, and initialize the buff to zero.

[0042] For each audio sample point extracted, it is saved sequentially to the buff storage location, i.e., buff[i] = a (a represents the audio sample point), and the count i is incremented by 1.

[0043] Step S22: Determine whether the sample rate of no speech in the cache file exceeds the set probability threshold.

[0044] After each storage, it is determined whether the percentage of samples without speech in all storage bits exceeds H%, that is, whether the number of sample points without speech data exceeds n*H%. If it exceeds, it is determined that there is no speech data in the audio data. If it does not exceed, it proceeds to step S23.

[0045] like Figure 3 As shown, assuming n is 10, H = 2, 1000 represents the location value with voice, 0 is the buff initialization zero value, and -1 represents the value without voice, then when i = 7, it can be determined that there are 3 samples without voice, which exceeds the set probability threshold H, so the no-voice judgment process can be terminated in advance, and the judgment result is no voice data.

[0046] Understandably, if the horizontal coordinate of the location is 0 or x, it is considered to be an audio signal outside the screen, and it can also be judged as no audio data.

[0047] Step S23: After the storage bits are full, determine whether the sample point of the last storage bit is without voice data.

[0048] When the audio sample point buffer is full, the last sample point may contain a signal that transitions from speech to no speech. Therefore, it is necessary to determine whether the last sample point contains no speech data.

[0049] If yes, proceed to step S24; otherwise, proceed to step S25.

[0050] Step S24: Move the audio sample point forward by one storage bit, and store the new audio sample point in the last empty storage bit.

[0051] To rule out the possibility of anomalies, the audio sample point buffer window is shifted to the left, i.e., buff[i] = buff[i+1], i∈[0,n-1), such as buff[0] = buff[1], buff[n-2] = buff[n-1]. The number of voice data samples is updated, i = n-1, and the process returns to step S22 to continue to determine whether there is voice data.

[0052] like Figure 4 , Figure 5 As shown, the last sample point is a no-speech value of -1. The cached data is shifted to the left as a whole, and the last data point is cleared to 0. The speech status is then assessed.

[0053] Step S25: Determine that there is voice data in the audio data.

[0054] Step S26: Determine that there is no speech data in the audio data.

[0055] Please see Figure 6 If audio data is detected in a voice state, step S12 may include the following steps:

[0056] S31: Store the audio sample points, which are cut off according to the set length, into the various storage locations of the cache file in sequence.

[0057] In this embodiment, a cache file containing voice acquisition data sample points is created, with a size of N, a relative time of 5 seconds, a noise duration threshold of K, and no continuous voice sample data P (P being an integer greater than 0). The cache is initialized to j = 0, cleared, and the count variable count is set to 0. Acquired audio sample points are saved to the cache, i.e., cache[j] = b (b representing an audio sample point), and the count j is incremented by 1.

[0058] S32: Determine whether all audio sample points in the previously set number of storage bits are without voice data.

[0059] Determine whether the data without speech is continuous, that is, determine whether the stored audio sample point j is greater than the continuous data without speech P. This step is to determine whether the first P data in the audio sample point cache are all data without speech, that is, to determine whether cache[0] to cache[P-1] are all data without speech.

[0060] If yes, proceed to step S34; otherwise, proceed to step S33.

[0061] Step S33: Determine that there is speech data in the audio data.

[0062] Step S34: Continue to determine whether subsequent audio sample points contain speech data.

[0063] That is, determine whether cache[j] (j is greater than or equal to P) contains voice data. If it contains voice data, increment the voice count by 1, return and repeat step S34; otherwise, execute step S35.

[0064] Step S35: Determine whether the number of audio sample points with speech data exceeds the noise duration threshold.

[0065] If the number of voice data sample points exceeds the limit, the count is reset to zero, and the process returns to step S34; if the count does not exceed the limit, step S36 is executed.

[0066] Step S36: Determine that all collected audio sample points are sample points without speech data.

[0067] This step updates the number of sample points without speech data to P+count+1, meaning that the current sample point and the previous count sample points are considered to be noise interference.

[0068] Step S37: Determine whether there are any audio sample points with sound exceeding three times the noise duration threshold among the currently collected audio sample points.

[0069] That is, whether jP is greater than 3*K. If yes, then jump to step S33; otherwise, jump to step S38.

[0070] Step S38: When the voice data sample is full, determine whether the last audio sample point contains voice data.

[0071] That is, if j>N is true, then determine whether the last audio sample point has speech data. If the last sample point has speech data, execute step S39; otherwise, execute step S40.

[0072] Step S39: Move the audio sample point buffer window to the left and return to step S34.

[0073] Whether the last audio sample point contains speech data may involve a transition from no speech to speech data. In this step, the audio sample point cache window is shifted to the left, i.e., cache[j] = cache[j+1], j∈[0,N-1), for example, cache[0] = cache[1], cache[n-2] = cache[N-1], and the process returns to step S34 to continue determining whether the collected audio sample point contains speech data.

[0074] Step S40: Determine that there is no speech data in the audio data.

[0075] That is, if the system is determined to be in a no-voice state, the cache is cleared.

[0076] This embodiment uses continuous audio data to continuously utilize speech data for sound source localization, thereby reducing the misjudgment and misidentification rate of sound source localization.

[0077] Step S13: Perform audio localization processing on the speech data to determine the location of the sound source corresponding to the speech data in the real-time image.

[0078] This step uses a voice localization algorithm to process the voice data in order to determine the location information of the corresponding voice data in the real-time image.

[0079] Step S14: Determine the sound source region within a preset range of the sound source location based on the error compensation data.

[0080] Please see Figure 7 , Figure 7 This is a schematic flowchart illustrating an embodiment of step S14 of this application. It should be noted that if substantially the same result is achieved, this embodiment does not necessarily follow that approach. Figure 1 The illustrated process sequence is limited. Step S14 may specifically include the following steps:

[0081] Step S141: Move the sound source position a set distance toward the first boundary of the real-time image.

[0082] Please refer to the following: Figure 8 , Figure 8It is a schematic diagram of an embodiment of a real-time image. x represents the sound source position obtained by processing with a voice localization algorithm. In this embodiment, a two-dimensional rectangular coordinate system XOY is established with the upper left corner of the real-time image as the origin. The coordinates of the lower right corner point of the real-time image can be expressed as (X, Y). The sound source position is reflected by its abscissa A in the real-time image. In other embodiments, the sound source position can also be reflected by its ordinate in the real-time image, and the two-dimensional rectangular coordinate system can also be different from that in this embodiment. O1 and O2 are two target objects.

[0083] Among them, the first boundary is the left boundary of the real-time image. The set distance is determined by dividing the product of the algorithm error of the audio localization algorithm and the expansion range factor by the compensation factor, and can be expressed as: W*m / T, where W represents the algorithm error of the audio localization algorithm and W < X, T represents the compensation factor and T > 1, m represents the expansion range factor, and the initial value of m is 1.

[0084] Step S142: Determine whether it exceeds the first boundary.

[0085] Determine whether the moved sound source position exceeds the first boundary. If it exceeds, execute step S143; otherwise, execute step S144. [[ID=,11]]

[0086] Among them, if it is determined that A < W*m / T holds, it can be determined that the moved sound source position exceeds the first boundary.

[0087] Step S143: In the real-time image, determine the area between the first boundary and the position separated by the set distance between the first boundaries as the sound source area.

[0088] That is, determine the area between 0 and W*m / T in the horizontal axis direction of the real-time image as the sound source area.

[0089] Step S144: Determine whether moving the sound source position by the set distance towards the second boundary exceeds the second boundary.

[0090] Among them, the second boundary is set opposite to the first boundary in the real-time image. In this embodiment, the second boundary is the right boundary of the real-time image. If it is determined that A + W*m / T > X holds, it can be determined that the moved sound source position exceeds the second boundary.

[0091] If the moved sound source position exceeds the second boundary, execute step S145; if it does not exceed, execute step S146. [[ID=,28]]

[0092] Step S145: In the real-time image, determine the area between the second boundary and the position separated by the set distance between the second boundaries as the sound source area. <,

[0093] That is, determine the area between X - W*m / T and X as the sound source area.

[0094] Step S146: The area between the position of the sound source after moving it a set distance toward the first boundary and the position of the sound source after moving it a set distance toward the second boundary is defined as the sound source area.

[0095] That is, the region between AW*m / T and A+W*m / T in the horizontal direction of the real-time image is defined as the sound source region.

[0096] Step S15: Determine whether the target object is included in the sound source region.

[0097] This step utilizes an object detection network to detect objects in the sound source region to determine whether it includes a target object, such as a human or an animal. The object detection network can be a face detection network, animal detection network, etc., pre-trained using training data; no specific limitation is made here.

[0098] If not included, proceed to step S16; if included, proceed to step S17.

[0099] Step S16: Adjust the magnified error compensation data and return to step S14.

[0100] This involves adjusting the range expansion factor by one and then using the adjusted range expansion factor to update the error compensation data.

[0101] That is, based on the previous range factor, it is increased by one unit, and the updated error compensation data can be expressed as: W*(m+1) / T.

[0102] Optionally, after adjusting the range factor, it is determined whether m is equal to T. If the target object is still not located when the range factor m is increased to T, it is determined that the location has failed and the process can proceed to step S18.

[0103] Step S17: Tracking display of the tracking area determined by the target object and the sound source region.

[0104] The horizontal coordinate range of the sound source region can be determined as the width range of the tracking region, and the height of the target object in the sound source region can be used as the height of the tracking region, thereby determining the tracking region in the real-time image.

[0105] Optionally, when there are at least two target objects, the tracking display shows the tracking area determined by the at least two target objects and the sound source region. Specifically, the tracking area is defined as the region enclosed by the maximum and minimum vertical coordinate values ​​and the maximum and minimum horizontal coordinate values ​​of the at least two target objects.

[0106] Please see Figure 9In this embodiment, the region between AW*m / T and A+W*m / T is defined as the sound source region. Within the sound source region, there are two target objects O1 and O2. The coordinates of the upper left corner of O1 are (x1, y1), the height is height1, and the width is width1. The coordinates of the upper left corner of O2 are (x2, y2), the height is height2, and the width is width2. Therefore, the height of the tracking region O3 is x2+width2-x1, and the height is y1+height1-y2.

[0107] Optionally, during the display of the tracking area, it is detected whether the target object in the tracking area has left the tracking area and whether the target object in the tracking area continues to output voice data. If the target object in the tracking area has no voice data for a set time (e.g., 3 seconds or 5 seconds) or the target object has left the tracking area, the original size of the real-time image is restored.

[0108] Step S18: No processing.

[0109] If step S12 determines that there is no voice data in the audio data, then no tracking and display processing is performed on the real-time image, and the original real-time image is displayed directly.

[0110] Unlike existing technologies, this application initially locates the sound source region in a real-time image based on error compensation data, then adjusts the error compensation data by determining whether the sound source region includes the target object, thereby redetermining the appropriate sound source region, and finally tracks and displays the tracking area determined by the target object and the sound source region, thus improving the accuracy and flexibility of sound localization.

[0111] Please see Figure 10 , Figure 10 This is a schematic block diagram of an embodiment of the sound source target localization device of this application. The sound source target localization device 200 includes a processor 210 and a memory 220 coupled to each other. The memory 220 stores a computer program, and the processor 210 is used to execute the computer program to implement the sound source target localization method described in the above embodiments.

[0112] For a description of each step of the processing, please refer to the description of each step in the above embodiment of the sound source target localization method of this application, and it will not be repeated here.

[0113] The memory 220 can be used to store program data and modules. The processor 210 executes various functional applications and data processing by running the program data and modules stored in the memory 220. The memory 220 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as voice positioning processing function, tracking display function, etc.), etc.; the data storage area may store data created based on the use of the sound source target positioning device 200 (such as image data, audio data, error compensation data, etc.). In addition, the memory 220 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 220 may also include a memory controller to provide the processor 210 with access to the memory 220.

[0114] In the various embodiments of this application, the disclosed methods and apparatus can be implemented in other ways. For example, the embodiments of the sound source target localization device 200 described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling or direct coupling or communication connection may be through some interfaces, indirect coupling or communication connection of devices or units, and may be electrical, mechanical, or other forms.

[0115] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0116] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0117] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium.

[0118] See Figure 11 , Figure 11 This is a schematic block diagram of a computer-readable storage medium according to an embodiment of the present application. The computer-readable storage medium 300 stores program data 310, which, when executed, implements the steps of the sound source target localization method embodiments described above.

[0119] For a description of each step of the processing, please refer to the description of each step in the above embodiment of the sound source target localization method of this application, and it will not be repeated here.

[0120] The computer-readable storage medium 300 can be any medium capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0121] The above description is merely an embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method for locating a sound source target, characterized in that, The method includes: Acquire real-time images and audio data synchronized with the real-time images; Determine whether there is voice data in the audio data; If so, then audio localization processing is performed on the voice data to determine the location of the sound source corresponding to the voice data in the real-time image; The sound source region is determined within a preset range at the sound source location based on error compensation data; Determine whether the sound source region includes the target object; If not, adjust and expand the error compensation data, and return to the step of determining the sound source region within a preset range of the sound source location based on the error compensation data; If so, the tracking area determined by the target object and the sound source region is displayed; Determining the sound source region within a preset range at the sound source location based on error compensation data includes: If the sound source location is moved a set distance toward the first boundary of the real-time image and then exceeds the first boundary, then the area between the first boundary and the location between the first boundary and the set distance is defined as the sound source area in the real-time image. Otherwise, determine whether moving the sound source location by the set distance toward the second boundary exceeds the second boundary; wherein the second boundary and the first boundary are set relative to each other in the real-time image; If it exceeds the limit, the region between the second boundary and the position between the second boundary and the set distance from the second boundary in the real-time image is determined as the sound source region; If the distance is not exceeded, the area between the position after the sound source location is moved to the first boundary by the set distance and the position after the sound source location is moved to the second boundary by the set distance is defined as the sound source area; The error compensation data is the product of the algorithm error of the audio positioning algorithm and the range expansion factor, divided by the compensation factor, and the initial value of the range expansion factor is 1. The adjustment to expand the error compensation data includes: The range expansion factor is increased by one, and the error compensation data is updated using the adjusted range expansion factor.

2. The method according to claim 1, characterized in that, The tracking display shows the tracking area determined by the target object and the sound source region, including: The width of the sound source region is taken as the target image width of the tracking region, and the height of the region where the target object is located is taken as the target image height of the tracking region. The tracking display shows the tracking area determined by the target screen width and the target screen height.

3. The method according to claim 1, characterized in that, The tracking display shows the tracking area determined by the target object and the sound source region, including: When the number of target objects is at least two, the tracking display shows the tracking area determined by the at least two target objects and the sound source region.

4. The method according to claim 3, characterized in that, The tracking display shows the tracking area determined by at least two of the target objects and the sound source region, including: The tracking area is defined as the region enclosed by the maximum and minimum ordinate values ​​and the maximum and minimum abscissa values ​​of at least two of the target objects.

5. The method according to claim 1, characterized in that, Determining whether speech data exists in the audio data includes: If audio data is detected in the absence of speech, the audio data is cut into audio sample points of a set length and stored sequentially in each storage location of the cache file. After each storage, it is determined whether the sample rate of no speech in the cache file exceeds the set probability threshold. If the number exceeds the limit, it is determined that there is no voice data in the audio data. Otherwise, after the storage space is full, it is determined whether the sample point of the last storage space has no voice data. If so, the audio sample point is moved forward by one storage space and a new audio sample point is stored in the last empty storage space. If not, it is determined that there is voice data in the audio data.

6. The method according to claim 1, characterized in that, Determining whether speech data exists in the audio data includes: If audio data is detected while there is voice, the audio data is truncated into audio sample points of a set length and stored sequentially in each storage location of the cache file. It is then determined whether the audio sample points in the previous set number of storage locations are all without voice data. If not, it is determined that there is voice data in the audio data. If so, it is determined whether the subsequent audio sample points contain voice data. If the number of audio sample points with voice data does not exceed the noise duration threshold, it is determined that all the collected audio sample points are without voice data. When the voice data sample is full and the number of voice sample points among the currently collected audio sample points does not exceed three times the noise duration threshold, it is determined whether the last audio sample point contains voice data. If it does not contain voice data, it is determined that there is no voice data in the audio data.

7. A sound source target localization device, characterized in that, The apparatus includes a processor and a memory coupled to each other; the memory stores a computer program, and the processor executes the computer program to implement the steps of the method as described in any one of claims 1-6.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program data that, when executed by a processor, implements the steps of the method as described in any one of claims 1-6.

Images