Audio processing method and audio device

Abstract

The present disclosure provides an audio processing method and an audio device, which can be applied to the field of audio and video processing. The audio processing method is applied to an audio acquisition device, and the audio acquisition device comprises a multi-layer microphone. The method comprises the following steps: receiving audio data of a target object based on the multi-layer microphone, determining position information of the target object relative to the audio acquisition device, the position information comprising horizontal position information and vertical position information; determining a first tracking strategy according to the position information; and tracking and picking up sound of the target object based on the first tracking strategy.

Description

Technical Field

[0001] This disclosure relates to the field of audio and video processing, and more specifically, to an audio processing method and an audio device. Background Technology

[0002] With the increasing prevalence of applications such as video conferencing and remote collaboration, the market demand for audio acquisition and tracking technologies is growing. However, most traditional audio devices lack tracking capabilities, making it difficult to meet the growing demand for intelligent audio. Summary of the Invention

[0003] In view of this, the present disclosure provides an audio processing method and an audio device.

[0004] According to one aspect of this disclosure, an audio processing method is provided, characterized in that it is applied to an audio acquisition device, the audio acquisition device comprising: a multi-layer microphone; the method comprising: receiving audio data of a target object based on the multi-layer microphone, determining position information of the target object relative to the audio acquisition device, the position information including horizontal position information and vertical position information; determining a first tracking strategy based on the position information; and tracking and picking up sound from the target object based on the first tracking strategy.

[0005] According to another aspect of this disclosure, an audio device is provided, comprising: one or more processors; and a memory for storing one or more instructions, wherein, when executed by the one or more processors, the one or more processors cause the one or more processors to perform the method as described in this disclosure.

[0006] According to embodiments of this disclosure, by receiving audio data of a target object through multiple microphones, the horizontal and vertical position information of the target object relative to the audio acquisition device can be accurately determined, enabling the audio acquisition device to possess omnidirectional spatial perception capabilities. Based on this, by determining a first tracking strategy according to the position information and performing tracking and sound pickup based on this strategy, real-time and accurate audio tracking of the speaker and other target objects can be achieved. Regardless of how the target object moves, the audio acquisition device can always focus on the target object, weakening noise from other angles, thereby improving the speaker's signal-to-noise ratio, ensuring audio quality, and enabling the audio acquisition device to automatically follow the speaker, thus enhancing the user experience in scenarios such as meetings. Attached Figure Description

[0007] The above and other objects, features and advantages of this disclosure will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0008] Figure 1 This illustration schematically shows a system architecture to which audio processing methods can be applied according to embodiments of the present disclosure;

[0009] Figure 2 A flowchart illustrating an audio processing method according to an embodiment of the present disclosure is shown schematically.

[0010] Figure 3 This illustration schematically shows an example diagram of the audio data determination process for a target object according to an embodiment of the present disclosure;

[0011] Figure 4A An example schematic diagram of an audio acquisition device according to an embodiment of the present disclosure is shown;

[0012] Figure 4B An example schematic diagram of an audio acquisition device according to another embodiment of the present disclosure is shown;

[0013] Figure 4C An example schematic diagram of a one-dimensional array according to an embodiment of the present disclosure is shown;

[0014] Figure 4D An example schematic diagram of a one-dimensional array according to an embodiment of the present disclosure is shown;

[0015] Figure 4E An example schematic diagram of a two-dimensional array according to an embodiment of the present disclosure is shown;

[0016] Figure 4F An example schematic diagram of a three-dimensional array according to an embodiment of the present disclosure is shown;

[0017] Figure 5A This illustration schematically shows an example diagram of a first tracking strategy determination process according to an embodiment of the present disclosure;

[0018] Figure 5B This illustration schematically shows an example diagram of a first tracking strategy determination process according to another embodiment of the present disclosure;

[0019] Figure 6A This illustration schematically shows an example of a process for establishing a connection between an audio acquisition device and a video acquisition device according to an embodiment of the present disclosure;

[0020] Figure 6B This illustration schematically shows an example of a process for establishing a connection between an audio acquisition device and a video acquisition device according to another embodiment of the present disclosure;

[0021] Figure 6C The illustration shows an example of an applicable scenario for the conference equipment control method according to an embodiment of the present disclosure;

[0022] Figure 7AThis illustration schematically shows an example of a data transmission process from an audio acquisition device to a video acquisition device according to an embodiment of the present disclosure;

[0023] Figure 7B This schematic diagram illustrates an example of a data transmission process from an audio acquisition device to a video acquisition device according to another embodiment of the present disclosure;

[0024] Figure 7C This schematically illustrates an example diagram of a process for adjusting a video capture device to track a target object according to an embodiment of the present disclosure; and

[0025] Figure 8 A block diagram of an audio device suitable for implementing an audio processing method according to an embodiment of the present disclosure is shown schematically. Detailed Implementation

[0026] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0027] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0028] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0029] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0030] In the technical solution of this invention, the user information (including but not limited to user personal information, user image information, user device information, such as location information) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of related data all comply with relevant laws, regulations, and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entry points for users to choose to authorize or refuse.

[0031] Currently, although some audio devices have sound source localization capabilities, these devices often only achieve horizontal localization and cannot obtain vertical information, resulting in an inability to achieve accurate tracking.

[0032] Therefore, this disclosure provides an audio processing method and an audio device that can be applied to the field of audio and video processing. The audio processing method is applied to an audio acquisition device, which includes a multi-layer microphone. The method includes: receiving audio data of a target object based on the multi-layer microphone; determining the position information of the target object relative to the audio acquisition device, the position information including horizontal and vertical position information; determining a first tracking strategy based on the position information; and tracking and picking up sound from the target object based on the first tracking strategy.

[0033] Figure 1 The illustration schematically depicts a system architecture to which audio processing methods can be applied according to embodiments of this disclosure. It should be noted that... Figure 1 The examples shown are merely examples of system architectures that can be applied to the embodiments of this disclosure, in order to help those skilled in the art understand the technical content of this disclosure, but do not mean that the embodiments of this disclosure cannot be used in other devices, systems, environments or scenarios.

[0034] like Figure 1 As shown, the system architecture 100 according to this embodiment may include an audio acquisition device 101, a video acquisition device 102, a first terminal device 104, and a second terminal device 105.

[0035] A connection is established between audio acquisition device 101 and video acquisition device 102, and the network serves as the medium for providing a communication link between them. Audio acquisition device 101 and video acquisition device 102 are devices located in the same environment, connected via wired means or short-range communication; that is, they are in the same location.

[0036] Audio acquisition devices refer to conference equipment with a display screen and sound pickup capabilities. In another embodiment, the audio acquisition device can also be a conference equipment integrating sound pickup and amplification functions. The audio acquisition device can be used independently as a conference device; for example, it can be connected to a terminal device, allowing information exchange between the terminal device and remote devices (i.e., participants in an online meeting or cloud devices) to achieve its conference function. Video acquisition devices refer to conference equipment with a camera, capable of capturing images using the camera, and can also be used independently as a conference device. For example, a video acquisition device can be connected to a terminal device, allowing information exchange between the terminal device and remote devices (i.e., participants in an online meeting or cloud devices) to achieve its conference function. However, it does not have a display screen, making user control less convenient, and generally requires the use of other devices for control.

[0037] In one embodiment, the audio acquisition device 101 and video acquisition device 102 can be used directly after installation, and the video acquisition device 102 can be controlled through the display screen of the audio acquisition device 101. For example, the audio acquisition device 101 has built-in conferencing software, and users can initiate online meetings through the audio acquisition device 101. During the online meeting, the audio acquisition device 101 picks up audio, the video acquisition device 102 captures video, and the captured video data is transmitted to the audio acquisition device 101. The audio acquisition device 101 then sends the picked-up audio data and the received video data together to a remote device (i.e., the participating device in the online meeting or a cloud device).

[0038] After the audio acquisition device 101 and the video acquisition device 102 are installed, the audio acquisition device 101 can also establish a connection with the first terminal device 104 and / or the second terminal device 105. The network is used as a medium to provide a communication link between the audio acquisition device 101, the first terminal device 104 and / or the second terminal device 105.

[0039] In one embodiment, the first terminal device 104 and / or the second terminal device 105 have built-in conferencing software. An online meeting is initiated through the first terminal device 104 and / or the second terminal device 105. The audio acquisition device 101 is used to pick up audio, and the video acquisition device 102 is used to capture video. The picked-up audio data and captured video data are sent to the first terminal device 104 and / or the second terminal device 105, and then sent by the first terminal device 104 and / or the second terminal device 105 to a remote device.

[0040] In one example, the audio acquisition device 101 may include: a multi-layer microphone, which can receive audio data of the target object 103 based on the multi-layer microphone, determine the position information of the target object 103 relative to the audio acquisition device 101, the position information including horizontal position information and vertical position information; determine a first tracking strategy for the audio acquisition device 101 based on the position information; and track and pick up sound from the target object 103 based on the first tracking strategy.

[0041] In another example, the audio acquisition device 101 can also establish a connection with the video acquisition device 102; the audio acquisition device 101 sends location information to the video acquisition device 102 so that the video acquisition device 102 determines a second tracking strategy for itself based on the location information, or the audio acquisition device 101 sends the second tracking strategy determined based on the location information to the video acquisition device 102; the video acquisition device 102 is adjusted based on the second tracking strategy to track the target object 103 so that a preset position of the target object 103 is within the shooting range.

[0042] It should be noted that the audio processing method provided in this embodiment can generally be executed by the audio acquisition device 101.

[0043] It should be understood that Figure 1 The number of audio acquisition devices, video acquisition devices, and terminal devices shown is merely illustrative. Depending on the implementation requirements, any number of audio acquisition devices, video acquisition devices, and terminal devices can be included.

[0044] It should be noted that the sequence numbers of the operations in the following methods are for descriptive purposes only and should not be considered as indicating the execution order of the operations. Unless explicitly stated otherwise, the method does not need to be executed in the exact order shown.

[0045] The foregoing has described the system architecture for which audio processing methods can be applied, as disclosed in this disclosure. The following will use... Figure 2 The audio processing procedure of this disclosure will be further explained as an example.

[0046] Figure 2 A flowchart illustrating an audio processing method according to an embodiment of the present disclosure is shown schematically.

[0047] like Figure 2 As shown, the audio processing method 200 includes steps S210 to S230.

[0048] In step S210, based on the audio data of the target object received by the multi-layer microphone, the position information of the target object relative to the audio acquisition device is determined. The position information includes horizontal position information and vertical position information.

[0049] In step S220, a first tracking strategy is determined based on the location information.

[0050] In step S230, the target object is tracked and its sound is picked up based on the first tracking strategy.

[0051] The audio processing method 200 disclosed herein can be applied to an audio acquisition device. The audio acquisition device includes a multi-layer microphone (two or more layers). Multi-layer microphones enable spatial localization of the sound source, i.e., they can locate not only horizontal azimuth information but also height information. This multi-layer microphone can function as a microphone array to process the received audio data, forming a directional beam, thereby enhancing the signal in a specific direction while suppressing interference from other directions. The target object refers to the speaker, i.e., the person speaking.

[0052] After receiving the audio data of the target object, before processing the audio data, it is possible to detect whether there is human speech activity in the audio data. The specific detection method can be configured according to actual business needs and is not limited here. For example, VAD (Voice Activity Detection) technology can be used to detect the audio data to determine whether there is human speech activity in the audio data by distinguishing the speech signal from background noise.

[0053] If human speech activity is confirmed in the audio data, analysis is performed to obtain the location information of the target object relative to the audio acquisition device. The specific analysis method can be configured according to actual business needs and is not limited here. For example, DOA (Direction of Arrival) technology can be used to analyze the direction of arrival of the audio data to determine the location information of the target object relative to the audio acquisition device. Alternatively, the model can be trained using sample audio data with location annotations to obtain a pre-trained model. This pre-trained model can then be used to analyze the spectrogram features of the audio data to output the location information of the target object relative to the audio acquisition device.

[0054] Location information can include horizontal and vertical location information. After obtaining the location information, a first tracking strategy can be determined based on it. The first tracking strategy refers to the strategy used to instruct the audio acquisition device how to adjust the microphone parameters to achieve effective tracking and pickup of the target object's audio. It should be noted that different location information may correspond to different first tracking strategies.

[0055] After obtaining the first tracking strategy, the target object can be tracked and picked up based on the first tracking strategy. That is, the directivity and receiving sensitivity of the multi-layer microphone can be directly controlled according to the parameter settings in the first tracking strategy to pick up sound. For example, the microphone's pickup beam can be pointed in the direction of the target object according to the position information, and the beamwidth and gain can be adjusted according to the requirements of the first tracking strategy to enhance the pickup quality of the target audio.

[0056] According to embodiments of this disclosure, by receiving audio data of a target object through multiple microphones, the horizontal and vertical position information of the target object relative to the audio acquisition device can be accurately determined, enabling the audio acquisition device to possess omnidirectional spatial perception capabilities. Based on this, by determining a first tracking strategy according to the position information and performing tracking and sound pickup based on this strategy, real-time and accurate audio tracking of the speaker and other target objects can be achieved. Regardless of how the target object moves, the audio acquisition device can always focus on the target object, weakening noise from other angles, thereby improving the speaker's signal-to-noise ratio, ensuring audio quality, and enabling the audio acquisition device to automatically follow the speaker, thus enhancing the user experience in scenarios such as meetings.

[0057] The above provides a preliminary description of the audio processing method provided in this disclosure. The following will use... Figure 3 As an example, the process for determining the audio data of the target object of this disclosure will be further explained.

[0058] Figure 3 The illustration shows an example schematic diagram of the process for determining audio data of a target object according to an embodiment of the present disclosure.

[0059] like Figure 3 As shown, in 300, in response to the audio acquisition device 101 receiving the audio stream 301, speech activity detection can be performed on the audio stream 301. The audio stream 301 refers to a sequence of audio signals that changes continuously over time. Speech activity detection is a technique that helps identify valid speech portions by distinguishing between speech signals and background noise, thereby determining whether human voices are present in the audio stream 301.

[0060] In step S310, is there a human voice in the audio stream 301? If not, the audio acquisition device 101 can remain in standby mode; if yes, the audio data 303 of the target object 103 can be determined. In one example, the method for determining whether there is a human voice in the audio stream 301 may also include at least one of the following: energy-based detection method, spectrum analysis-based detection method, and deep learning-based detection method.

[0061] Energy-based detection methods determine the presence of speech activity in audio stream 301 by comparing the energy of the audio signal in audio stream 301 with a set threshold. For example, if the energy of the audio signal exceeds the threshold for a continuous period, it can be determined that human voice is present in audio stream 301. Spectrum analysis-based detection methods determine the presence of speech activity in audio stream 301 by analyzing its spectral characteristics. For example, if audio stream 301 exhibits specific distribution characteristics within certain frequency ranges, it can be determined that human voice is present in audio stream 301. Deep learning-based detection methods use a trained model to determine the presence of human voice in audio stream 301. For example, the features of audio stream 301 are input into a trained model, and the model outputs the probability of whether human voice is present in audio stream 301.

[0062] In one example, the method of obtaining audio data 303 can be configured according to actual business needs and is not limited here. For instance, audio data 303 can be extracted directly from audio stream 301. For example, when it is determined that audio stream 301 within a certain time period belongs to target object 103, that portion of audio data can be copied as the audio data of target object 103. Alternatively, audio data 303 can be extracted by first marking the audio portion of target object 103 in audio stream 301, and then extracting it based on the markings. For example, tags can be added to audio stream 301 to identify which parts belong to target object 103, and then the corresponding audio data 303 can be extracted based on the tags.

[0063] According to embodiments of this disclosure, by responding to the input of an audio stream and performing voice activity detection, the human voice portion can be detected in a timely manner. This accurate recognition capability can effectively separate the audio data of the target object from a complex audio environment, improving the targeting and efficiency of audio processing.

[0064] In one embodiment, in response to receiving an audio stream, voice activity detection is performed on the audio stream; in response to detecting the presence of a human voice in the audio stream, it is determined whether the currently speaking human voice belongs to the target object based on a preset detection strategy, the preset detection strategy including at least one of the number of human voices and the sound pressure level of the human voice; in response to the human voice belonging to the target object, audio data of the target object is determined.

[0065] In this embodiment, the presence of human voices in the audio stream is first detected, followed by the number of human voices. If there is only one human voice, that person is selected as the target. If there are multiple human voices, the person with the loudest voice is selected as the target. The sound pressure level represents the volume of the sound.

[0066] The foregoing has provided a preliminary explanation of the process for determining the audio data of the target object provided in this disclosure. The following will describe the process in detail using... Figure 4A and Figure 4B As an example, the structure of the audio acquisition device for acquiring audio data of a target object disclosed herein will be further described.

[0067] Figure 4A An example schematic diagram of an audio acquisition device according to an embodiment of the present disclosure is shown.

[0068] like Figure 4A As shown, the audio acquisition device 101 may include a display screen 410 and a lifting mechanism 420.

[0069] In 400A, the display screen 410 is illustrated as being mounted on the base of the audio acquisition device 101, but this is not a limitation. The embodiments of this application do not restrict the installation position of the display screen 410 within the audio acquisition device 101. The display screen 410 can be used for user operation and information display. For example, the display screen 410 can be used to allow users to operate the audio acquisition device 101 and display information related to the audio acquisition device 101.

[0070] The lifting mechanism 420 can be used to adjust the height of the audio acquisition device 101. The specific structure of the lifting mechanism 420 can be configured according to actual business needs and is not limited here. For example, the lifting mechanism 420 can be at least one of the following: a push rod structure, a screw structure, a hydraulic structure, a pneumatic structure, etc.

[0071] In one example, the audio acquisition device 101 may further include a light-emitting component 430. In 400A, the light-emitting component 430 is illustrated as being disposed on the upper part of the audio acquisition device 101, and / or disposed on the side housing of the audio acquisition device 101, but is not limited thereto. This application embodiment does not limit the installation position of the light-emitting component 430 in the audio acquisition device 101. For example, the light-emitting component 430 may be at least one of the following: an LED light, a light strip, or other components with light-emitting capabilities.

[0072] Figure 4B An example schematic diagram of an audio acquisition device according to another embodiment of the present disclosure is shown.

[0073] like Figure 4B As shown, in 400B, the audio acquisition device 101 may include at least two layers of audio acquisition unit groups, each of which includes multiple audio acquisition units. For example, the audio acquisition device 101 includes an audio acquisition unit group 441 and an audio acquisition unit group 442, where the audio acquisition unit group 441 includes multiple audio acquisition units and the audio acquisition unit group 442 includes multiple audio acquisition units.

[0074] In one example, the multiple audio acquisition units include at least one of the following structures: a one-dimensional array, a two-dimensional array, and a three-dimensional array. These will be described below using... Figure 4C and Figure 4D The one-dimensional array provided in this disclosure will be illustrated as an example. Figure 4E The two-dimensional array provided in this disclosure will be illustrated using an example. Figure 4F The three-dimensional array provided in this disclosure is illustrated as an example.

[0075] Figure 4C An example schematic diagram of a one-dimensional array according to an embodiment of the present disclosure is shown; Figure 4D The illustration shows an example schematic diagram of a one-dimensional array according to an embodiment of the present disclosure.

[0076] like Figure 4C and Figure 4D As shown, a one-dimensional array is an array composed of multiple audio acquisition units arranged in a one-dimensional straight line. One-dimensional arrays can be used for audio acquisition in a specific direction.

[0077] In 400C, M audio acquisition units are arranged in a horizontal one-dimensional straight line to form a one-dimensional array 401. In 400D, N audio acquisition units are arranged in a vertical one-dimensional straight line to form a one-dimensional array 402.

[0078] It should be noted that the spacing between the audio acquisition units in a one-dimensional array can be configured according to actual business needs and is not limited here. For example, the audio acquisition units in a one-dimensional array can be arranged at equal intervals, thereby achieving uniform directional enhancement; or the audio acquisition units in a one-dimensional array can be arranged at unequal intervals, thereby optimizing the response to specific frequencies or directions.

[0079] Figure 4E An example schematic diagram of a two-dimensional array according to an embodiment of the present disclosure is shown.

[0080] like Figure 4E As shown, a two-dimensional array is an array composed of multiple audio acquisition units arranged in a two-dimensional plane. Two-dimensional arrays can be used for audio acquisition in multiple directions.

[0081] In the 400E, N rows * M columns of audio acquisition units are arranged in a two-dimensional plane to form a two-dimensional array 403.

[0082] It should be noted that the arrangement of the audio acquisition units in the two-dimensional array can be configured according to actual business needs and is not limited here. For example, the audio acquisition units in the two-dimensional array can be arranged in a rectangle, thereby achieving uniform coverage; or they can be arranged in a circle, thereby achieving 360-degree coverage, etc.

[0083] Figure 4F An example schematic diagram of a three-dimensional array according to an embodiment of the present disclosure is shown.

[0084] like Figure 4F As shown, a three-dimensional array is an array composed of multiple audio acquisition units distributed in three-dimensional space. Three-dimensional arrays are suitable for omnidirectional, stereo audio acquisition.

[0085] In the 400F, multiple audio acquisition units are arranged in a cylindrical shape in three-dimensional space to form a three-dimensional array 404.

[0086] It should be noted that the arrangement of the audio acquisition units in the 3D array can be configured according to actual business needs and is not limited here. For example, the audio acquisition units in the 3D array can be arranged in a spherical shape, thereby achieving omnidirectional audio coverage.

[0087] According to embodiments of this disclosure, by employing at least two layers of audio acquisition unit groups in the audio acquisition device, the multi-layer audio acquisition unit group design enables the audio acquisition device to acquire audio signals from different heights or angles, thereby enhancing its adaptability to complex environments. Furthermore, by allowing the audio acquisition units to be arranged in various forms such as one-dimensional arrays, two-dimensional arrays, or three-dimensional arrays, the flexibility and performance of audio acquisition are improved.

[0088] In embodiments of this disclosure, after the multi-layer microphones included in the audio acquisition device receive audio data of the target object, the position information of the target object relative to the audio acquisition device can be determined based on the audio data. The position information may include horizontal position information and vertical position information.

[0089] In one example, horizontal delay estimation can be performed on multiple audio acquisition units to obtain the horizontal position information of the target object relative to the audio acquisition device. Horizontal delay estimation refers to estimating the horizontal position information of the target object relative to the audio acquisition device by analyzing the arrival time difference of the audio signals received by multiple audio acquisition units in the horizontal direction. For example, when the target object is horizontally located to the left of the audio acquisition device, the left microphone will receive the audio signal earlier than the right microphone; by calculating this time difference, the horizontal position information of the target object relative to the audio acquisition device can be estimated.

[0090] In another example, vertical delay estimation can be performed on multiple audio acquisition units to obtain the vertical position information of the target object relative to the audio acquisition device. Vertical delay estimation refers to estimating the vertical position information of the target object relative to the audio acquisition device by analyzing the arrival time difference of the audio signals received by multiple audio acquisition units in the vertical direction. For example, when the target object is vertically above the audio acquisition device, the microphone above will receive the audio signal earlier than the microphone below. By calculating this time difference, the vertical position information of the target object can be estimated.

[0091] It should be noted that the methods for determining the horizontal and vertical position information can be configured according to actual business needs and are not limited here. For example, the determination methods may include at least one of the following: a method based on generalized cross-correlation, a method based on beamforming, and a method based on machine learning. The method based on generalized cross-correlation refers to using the cross-correlation function of the audio data to estimate the time delay and obtain the horizontal and / or vertical position information. The method based on beamforming refers to forming beams pointing in different directions and obtaining the horizontal and / or vertical position information based on the intensity and time difference of the beam outputs. The method based on machine learning refers to using a trained model to process the audio data and obtain the horizontal and / or vertical position information.

[0092] According to embodiments of this disclosure, by estimating the time delay of multiple audio acquisition units in the horizontal or vertical direction, the position information of the target object relative to the audio acquisition device can be accurately determined. This precise positioning capability enables the audio acquisition device to accurately track and pick up the target object, thereby effectively focusing on the speaker, suppressing background noise, improving audio quality, and enhancing the user experience.

[0093] After the audio acquisition device acquires the audio data of the target object, the first tracking strategy can be further determined, which will be discussed below. Figure 5A and Figure 5B The process for determining the first tracking strategy provided in this disclosure is illustrated as an example.

[0094] Figure 5A This illustration schematically shows an example diagram of a first tracking strategy determination process according to an embodiment of the present disclosure; Figure 5B The illustration shows an example schematic diagram of a first tracking strategy determination process according to another embodiment of the present disclosure.

[0095] like Figure 5A and Figure 5BAs shown, after the audio acquisition device 101 acquires the audio data of the target object 103, it can determine the angle information 501 or the position information 502 of the target object 103 relative to the audio acquisition device 101. Based on this, a first tracking strategy 503 can be determined according to the angle information 501 or the position information 502.

[0096] In embodiments of this disclosure, the first tracking strategy 503 may correspond to a beam adjustment type. For example, the beam adjustment type may include a first adjustment type and a second adjustment type.

[0097] In 500A, the first adjustment type can refer to beam adjustment achieved using a delay summation method. The delay summation method means that, given the ability of each audio acquisition unit to rotate, the delay is compensated for by the relative position information of the target object relative to the audio acquisition device, thereby enabling the horizontal and pitch rotation of each audio acquisition unit. In this case, the first tracking strategy 503 may include: forming a target beam 506 pointing towards the target object 103.

[0098] Taking an audio acquisition device 101 comprising P audio acquisition units as an example, for instance, the audio acquisition device 101 includes audio acquisition units 504_1, ..., audio acquisition units 504_P, where P is a positive integer greater than 1.

[0099] After obtaining the target beam 506, the compensation delay amount for each audio acquisition unit can be determined based on the target beam. The compensation delay amount refers to the amount of delay that needs to be compensated, calculated based on the direction of the target beam 506 and the position of each audio acquisition unit, in order to align the signals of multiple audio acquisition units in time. For example, the compensation delay amount 505_1 for audio acquisition unit 504_1, ..., and the compensation delay amount 505_P for audio acquisition unit 504_P.

[0100] Based on this, beam adjustment can be performed according to the compensation delay amount used for each audio acquisition unit to form a target beam 506 for receiving sound from the target object 103. Beam adjustment refers to processing the signals of each audio acquisition unit according to the compensation delay amount to form the target beam 506 pointing towards the target object. For example, by applying corresponding delay compensation to the signals of each audio acquisition unit, these signals are coherently added in the target direction, thereby enhancing the audio signal of the target object.

[0101] According to embodiments of this disclosure, by determining a first tracking strategy based on angle information or position information, and forming a target beam pointing towards the target object based on the strategy, the beam direction can be dynamically adjusted, thereby ensuring that the audio acquisition device always points towards the target object, suppressing background noise and other interfering sound sources, and improving the audio acquisition device's ability to track and pick up sound from the target object.

[0102] In 500B, the second adjustment type can refer to beam adjustment implemented using a differential method. The differential method refers to switching between multiple preset audio acquisition unit groups based on the relative position information of the target object relative to the audio acquisition device, when the individual audio acquisition units cannot rotate. In this case, the audio data of the target object 103 can be acquired by the currently used audio acquisition unit group 507_1 in the audio acquisition device 101, and the first tracking strategy 503 can include: selecting the target audio acquisition unit group 507_2 pointing towards the target object 103.

[0103] After determining the target audio acquisition unit group 507_2, an acquisition command can be generated based on the target audio acquisition unit group 507_2. The acquisition command can be used to instruct the audio acquisition device 101 to switch from the currently used audio acquisition unit group 507_1 to the target audio acquisition unit group 507_2. After switching to the target audio acquisition unit group 507_2, audio can be acquired from the target object 103 using the target audio acquisition unit group 507_2.

[0104] It should be noted that the above switching process can be achieved by directly controlling the hardware switching circuit of the audio acquisition device 101 and switching the signal acquisition path to the target audio acquisition unit group 507_2; or it can be achieved by modifying the software configuration parameters of the audio acquisition device 101.

[0105] Furthermore, the aforementioned switching process can be completed instantly to meet real-time requirements; the aforementioned switching process can also be completed gradually to reduce interruption or distortion of audio signals during the switching process; the aforementioned switching process can also be performed in advance by predicting the movement trend of the target object 103 and based on the obtained future position to ensure the continuity of audio acquisition.

[0106] According to embodiments of this disclosure, by determining a first tracking strategy based on angle information or position information, and selecting a target audio acquisition unit group pointing to the target object based on the strategy, the audio acquisition unit group can be dynamically switched, thereby ensuring that the audio acquisition device always points to the target object, suppressing background noise and other interfering sound sources, and improving the audio acquisition device's ability to track and pick up sound from the target object.

[0107] In the embodiments of this disclosure, the audio acquisition device provided above can also be connected to a video acquisition device, which will be described below using... Figure 6A and Figure 6B As an example, the connection establishment process between the audio acquisition device and the video acquisition device disclosed herein will be further explained.

[0108] Figure 6A The illustration shows an example of a process for establishing a connection between an audio acquisition device and a video acquisition device according to an embodiment of the present disclosure.

[0109] like Figure 6A As shown in Figure 600A, the connection between the audio acquisition device 101 and the video acquisition device 102 can include a physical connection and a communication connection. The physical connection can refer to the mechanical and electrical connections between the audio acquisition device 101 and the video acquisition device 102, while the communication connection can refer to the data transmission channel between them. The display screen 301 of the audio acquisition device 101 can also be used by the user to operate the video acquisition device 102 and display information related to the video acquisition device 102.

[0110] In one example, a physical connection can be established between the audio acquisition device 101 and the video acquisition device 102 via a magnetic attraction component 601_1 and a positioning component 601_2. The magnetic attraction component 601_1 can be used to quickly fix and initially position the audio acquisition device 101 and the video acquisition device 102 through magnetic attraction. For example, the magnetic attraction component 601_1 can be a magnet or a magnetic metal sheet. The positioning component 601_2 can be used for precise positioning and stable support of the audio acquisition device 101 and the video acquisition device 102. For example, the positioning component 601_2 can be a positioning post, a slot, a mechanical latch, a guide rail, or a slide.

[0111] For example, the audio acquisition device 101 may have a magnet on its top, and the video acquisition device 102 may have a metal plate on its bottom. When the user brings the video acquisition device 102 close to the audio acquisition device 101, the magnet attracts the metal plate, initially fixing the video acquisition device 102 onto the audio acquisition device 101. Furthermore, matching positioning posts and slots may be provided on the top of the audio acquisition device 101 and the bottom of the video acquisition device 102. When the video acquisition device 102 is attracted by the magnet, the positioning post inserts into the slot, ensuring that the video acquisition device 102 is correctly aligned on the audio acquisition device 101 and establishing a stable physical connection.

[0112] In another example, when audio acquisition device 101 detects that video acquisition device 102 is in a present state via connection component 602, audio acquisition device 101 supplies power to video acquisition device 102 via connection component 602, and a communication connection can be established through message handshake. Connection component 602 can be used to provide power supply and communication functions between audio acquisition device 101 and video acquisition device 102. For example, connection component 602 can be a pogo pin.

[0113] For example, when the video acquisition device 102 is correctly installed on the audio acquisition device 101, the connection component 602 can detect the presence of the video acquisition device 102 through electrical contacts. Upon detecting the presence of the video acquisition device 102, the audio acquisition device 101 can provide power to the video acquisition device 102 through the power supply line in the connection component 602. Furthermore, the audio acquisition device 101 and the video acquisition device 102 perform a message handshake through the communication line in the connection component 602. For example, the audio acquisition device 101 can send an inquiry message to the video acquisition device 102. After receiving and verifying the message, the video acquisition device 102 returns a response message containing information such as device type and version number. Through this handshake process, the audio acquisition device 101 and the video acquisition device 102 negotiate communication parameters and establish a stable communication connection.

[0114] In one example, the video capture device 102 has a camera function. By deploying the video capture device 102 and the audio capture device 101 in the same scene, it is easy to achieve the tracking of the speaker's image.

[0115] In one example, the video capture device 102 includes a camera.

[0116] In one example, the video capture device 102 may also include a pan-tilt unit, which can be used to rotate the camera horizontally and / or tilt it.

[0117] In one example, the video capture device 102 may also include a connecting shaft connected to a camera, which can be used to control the camera's pitch movement.

[0118] In one example, the audio acquisition device 101 can be connected to the video conferencing device 102 via a wired and / or wireless connection. For example, the video acquisition device 102 can be connected to the top of the audio acquisition device 101 via a wired connection. Alternatively, other connection methods may be used depending on the actual scenario, and this embodiment of the application does not limit this.

[0119] In one example, the number of video capture devices 102 can be one or more. The deployment positions of different video capture devices 102 can vary to capture video from different angles. For example, video capture device 102 can capture 360° video, enabling the tracking of speakers from various angles within a meeting setting without losing them. In one possible implementation, when audio capture device 101 is connected to video capture device 102, if the current height is insufficient for optimal sound pickup and capture, such as when the target object is obstructed, the height of audio capture device 101 can be adjusted to the target height using lifting mechanism 420. This ensures that there are no obstructions between the microphone array of audio capture device 101 and video capture device 102 and the target object, resulting in better sound pickup and capture.

[0120] According to embodiments of this disclosure, a rapid physical connection between an audio acquisition device and a video acquisition device is achieved through a magnetic attachment component and a positioning component. After the connection component detects the presence status, it automatically supplies power and establishes a communication connection. This design simplifies and speeds up the connection process between devices, allowing users to easily connect video acquisition devices to audio acquisition devices while ensuring stable power supply and data transmission, thus enhancing the ease of use of the devices and the efficiency of collaborative work.

[0121] Figure 6B The illustration shows an example diagram of a process for establishing a connection between an audio acquisition device and a video acquisition device according to another embodiment of the present disclosure.

[0122] like Figure 6B As shown in Figure 600B, in addition to establishing a physical connection 601 and a communication connection 602 with the video acquisition device 102, the audio acquisition device 101 may also include a lifting mechanism 603, which can be used to adjust the height of the audio acquisition device 101. The specific structure of the lifting mechanism 603 can be configured according to actual business needs and is not limited here. For example, the lifting mechanism 603 can be at least one of the following: a push rod structure, a screw structure, a hydraulic structure, a pneumatic structure, etc.

[0123] In one example, the current height of the audio acquisition device 101 can be obtained, and the position information of the target object can be determined based on the current height and angle information. The angle information may include horizontal and pitch angles. Based on this, a first tracking strategy can be determined.

[0124] The method for obtaining the current height can be configured according to actual business needs and is not limited here. For example, a linear displacement sensor installed on the lifting mechanism 603 can be used to measure the height of the audio acquisition device 101. Alternatively, a rotary encoder can be used to measure the rotation angle of the lifting mechanism 603, and the height of the audio acquisition device 101 can be calculated by combining parameters such as the screw pitch. Alternatively, a laser rangefinder can be used to measure the distance between the audio acquisition device 101 and a reference point to obtain the height of the audio acquisition device 101.

[0125] According to embodiments of this disclosure, by introducing a lifting mechanism to adjust the height of the audio acquisition device, the audio acquisition device can adapt to sound sources at different heights and positions. Based on this, by combining the current height and angle information of the audio acquisition device to determine the position information of the target object, it is possible to ensure that the speech of the target object is clearly captured, while suppressing background noise and other interfering sound sources, thereby improving the quality of audio acquisition.

[0126] In one example, the target height can also be determined based on the target object's location information. The method for determining the target height can be configured according to actual business needs and is not limited here.

[0127] For example, the target height can be determined using a pre-established mapping table. For instance, the location information of different possible target objects can be pre-configured, and the target height of the corresponding lifting mechanism 603 can be pre-determined through experimentation or calculation, storing these correspondences in the mapping table. Once the location information of the target object is obtained, the target height can be directly obtained by querying the mapping table based on the location information.

[0128] For example, the target height can be calculated using a mathematical model. For instance, the target height that the lifting mechanism 603 needs to reach can be calculated based on the position information of the target object using geometric relationships or kinematic models.

[0129] For example, the target height can be predicted using a trained model. For instance, a trained model can be obtained by training samples including sample location information and sample height. Once the location information of the target object is obtained, this information can be input into the trained model to predict the target height that the lifting mechanism 603 needs to reach.

[0130] After obtaining the target height, the lifting mechanism 603 can be adjusted to the target height to ensure that the audio and video acquisition devices are aligned with the target object in the vertical direction. The specific adjustment method can be adapted to the structure of the lifting mechanism 603.

[0131] In one example, when the current height of the audio acquisition device 101 is insufficient for good sound pickup, such as when the target object is obstructed, the height of the audio acquisition device 101 can be adjusted to the target height by the lifting mechanism 603, so that there are no obstructions between the microphone array of the audio acquisition device 101 and the target object, thus achieving better sound pickup.

[0132] In one example, when the audio acquisition device 101 is connected to the video acquisition device, if the current height cannot meet the requirements for good sound pickup and shooting, such as when the target object is blocked, the height of the audio acquisition device 101 can be adjusted to the target height through the lifting mechanism 603. This ensures that there are no obstructions between the microphone array of the audio acquisition device 101 and the video acquisition device and the target object, thus achieving better sound pickup and shooting.

[0133] According to embodiments of this disclosure, since the lifting mechanism can automatically adjust to a suitable target height based on the position information of the target object, it ensures accurate spatial matching between the audio acquisition device and the target object, adapts to different application scenarios and changes in the target object, and improves the convenience and accuracy of operation.

[0134] In the embodiments of this disclosure, the audio acquisition device 101 and video acquisition device 102 provided above can be applied to conference scenarios. The following will use... Figure 6C As an example, the applicable scenarios for the audio processing method disclosed herein will be further explained.

[0135] Figure 6C The illustration shows an example of an applicable scenario for the conference equipment control method according to an embodiment of the present disclosure.

[0136] like Figure 6C As shown, in 600C, in addition to establishing a physical connection 601 and a communication connection 602 with the video acquisition device 102, the audio acquisition device 101 also establishes a connection with the terminal device 104 to act as a peripheral device of the terminal device 104. The terminal device 104 can be, for example, a computer, a mobile phone, a tablet, etc. The connection between the audio acquisition device 101 and the terminal device 104 can be a wired connection or a wireless connection, which is not limited here. For example, the audio acquisition device 101 and the terminal device 104 can be connected via a USB cable; alternatively, the audio acquisition device 101 and the terminal device 104 can also be connected via WiFi or Bluetooth.

[0137] In one example, since the audio acquisition device 101 and the video acquisition device 102 have established a physical connection 601 and a communication connection 602, the audio acquisition device 101 and the video acquisition device 102 have been put into use. By connecting the audio acquisition device 101 to the terminal device 104, the audio data acquired by the audio acquisition device 101 and the video data acquired by the video acquisition device 102 can be transmitted to the terminal device 104.

[0138] For example, audio acquisition device 101 can be used to receive video data acquired by video acquisition device 102, for example, the video data can be transmitted via communication connection 602. Audio acquisition device 101 can send the video data and the audio data acquired by audio acquisition device 101 to terminal device 104.

[0139] Taking terminal devices 104 and 107 as examples, both of which have installed and connected to a certain conference application A, this conference application A belongs to a certain conference platform.

[0140] Each terminal device can provide online meeting services to one or more users by accessing the meeting application of the meeting platform. For example, terminal device 104 can provide meeting services to participant 106 using terminal device 104 through the meeting application, and terminal device 107 can provide meeting services to participant 108 using terminal device 107 through the meeting application.

[0141] For terminal device 104, audio acquisition device 101 and video acquisition device 102 connected to terminal device 104 participate in the same conference. In this case, the audio data acquired by audio acquisition device 101 can be audio data for the conference application A and user 106, and the video data acquired by video acquisition device 102 can be video data for the conference application A and user 106. Audio acquisition device 101 can send the audio data and video data of application A and user 106 to terminal device 104.

[0142] For terminal device 107, the audio data collected by terminal device 107 may be audio data for the conference application A and user 108, and the video data collected by terminal device 107 may be video data for the conference application A and user 108.

[0143] The network can provide a medium for a communication link between terminal device 104 and terminal device 107. That is, terminal device 104 can send the audio and video data of application A and user 106 to terminal device 107 through the network, so that terminal device 107 can play the audio and video data of application A and user 106.

[0144] Terminal device 107 can also send the audio and video data of conference application A and user 108 to terminal device 104 via the network, so that audio acquisition device 101 can play the audio data of conference application A and user 108, and video acquisition device 102 can play the video data of conference application A and user 108.

[0145] According to embodiments of this disclosure, by connecting the audio acquisition device to the terminal device, the integration and transmission of audio data and video data are realized, giving full play to the sound pickup capability of the audio acquisition device and the video acquisition capability of the video acquisition device, providing users with a complete and high-quality audio and video experience.

[0146] After the connection between the audio acquisition device and the video acquisition device is established, this disclosure allows data transmission from the audio acquisition device to the video acquisition device. The following will describe this process in detail. Figure 7A and Figure 7B As an example, the data transmission process from the audio acquisition device to the video acquisition device is explained.

[0147] Figure 7A This schematic diagram illustrates an example of a data transmission process from an audio acquisition device to a video acquisition device according to an embodiment of the present disclosure. Figure 7B The illustration shows an example diagram of a data transmission process from an audio acquisition device to a video acquisition device according to another embodiment of the present disclosure.

[0148] like Figure 7A and Figure 7B As shown, in 700A, after the audio acquisition device 101 acquires the audio data of the target object 103, it can determine the position information 701 of the target object 103 relative to the audio acquisition device 101. The method for determining the position information 701 may include at least one of the following: based on sound source localization technology, based on microphone array technology, based on spatial audio analysis technology, etc.

[0149] For example, location information based on sound source localization (DOA) technology refers to analyzing the audio data of the target object 103 to obtain location information 701. Alternatively, location information based on microphone array technology refers to using a microphone array combined with signal processing algorithms to analyze the time difference and phase difference of audio data arriving at different microphones to obtain location information 701. Another alternative is location information based on spatial audio analysis technology, which uses speech recognition technology to first identify valid speech in the audio data, and then combines this with spatial audio analysis to obtain location information 701.

[0150] After obtaining location information 701, audio acquisition device 101 can send location information 701 to video acquisition device 102 based on the connection between audio and video acquisition device 102, so that video acquisition device 102 can determine a second tracking strategy 702 based on location information 701. The second tracking strategy 702 can be used to adjust video acquisition device 102 to track the target object 103.

[0151] For example, the video acquisition device 102 can calculate the required orientation and focal length of the pan-tilt unit and / or camera based on the position information 701, forming a second tracking strategy 702. The video acquisition device 102 can accurately track the target object 103 and capture images of it by executing the second tracking strategy 702, i.e., continuously adjusting the orientation and focal length of the pan-tilt unit and / or camera.

[0152] It should be noted that when determining the second tracking strategy 702 based on the location information 701, a prediction algorithm can also be used to predict the movement trajectory of the target object 103 and adjust the acquisition direction in advance. The acquisition direction can be adjusted by various mechanical or electronic control methods, such as motor-driven gimbal rotation or lens module movement.

[0153] Compared to 700A, in 700B, after obtaining location information 701, the audio acquisition device 101 can also determine a second tracking strategy 702 based on the location information 701, and send the second tracking strategy 702 to the video acquisition device 102 based on the connection between the audio acquisition device 101 and the video acquisition device 102. The second tracking strategy 702 can be used to adjust the video acquisition device 102 to track the target object 103.

[0154] After the video capture device 102 obtains the second tracking strategy 702, it can jump to track the target object 103 based on the second tracking strategy 702. The following will use... Figure 7C As an example, the process of adjusting video capture equipment to track a target object is explained.

[0155] Figure 7C The illustration shows an example schematic diagram of a process of adjusting a video capture device to track a target object according to an embodiment of the present disclosure.

[0156] like Figure 7C As shown in Figure 700C, taking the video capture device 102 capturing video frame 705 at time t1, video frame 707 at time t2, and video frame 708 at time t3 as an example, the process of adjusting the video capture device 102 to track the target object 103 is explained. t1 <t2<t3。

[0157] The preset location refers to the area of ​​focus during video capture. In one example, the preset location could be the lip location or the face location including the lips, meaning the camera's focus is on the lip location or the entire face area of ​​the target object, with the lips as the core region. The lip location refers to the specific area where the lips of the target object 103 are located. The lip location can be obtained, for example, by performing facial key point detection on the video frame. The face location including the lips can be obtained, for example, by performing facial recognition on the video frame to examine the face area, and then precisely determining the lip location within the face area.

[0158] For video frame 705 at time t1, the preset position 706 of the target object 103 is within the shooting range; for video frame 707 at time t2, the preset position of the target object 103 is not within the shooting range.

[0159] In this situation, since the preset position of the target object 103 in video frame 707 is shifted to the left compared to the preset position of the target object 103 in video frame 705, the target tracking function built into the video acquisition device 102 can be used. To maintain continuous acquisition of the preset position, the acquisition direction of the video acquisition device 102 can be adjusted to the left to readjust the preset position of the target object 103 back into the shooting range. Thus, for video frame 708 at time t3, the preset position of the target object 103 is back within the shooting range.

[0160] According to embodiments of this disclosure, the audio acquisition device accurately acquires audio data and location information of the target object by leveraging its own advantages, and transmits the location information or a second tracking strategy determined based on the location information to the video acquisition device. The video acquisition device adjusts the acquisition direction accordingly. Through the collaborative work of the audio acquisition device and the video acquisition device, the video acquisition device can intelligently track the target object, ensuring that the preset position of the target object is always within the shooting range, thereby improving the tracking efficiency and accuracy of the target object.

[0161] In one example, a video capture device can extract lip movement features from the lip region using a lip movement detection algorithm. A lip movement detection algorithm is an algorithm used to analyze and extract the lip movement features of an object; that is, it analyzes consecutive frames of the lip region in a video stream captured by the video capture device to capture changes in lip movement. Lip movement features refer to the characteristics exhibited by the lips during movement, such as the degree of lip opening and closing, movement speed, and movement trajectory. For example, when a person speaks, the degree of lip opening and closing and the speed of lip movement change with the speech; these changes are lip movement features.

[0162] Specific lip movement detection algorithms can be configured according to actual business needs and are not limited here. For example, lip movement detection algorithms may include at least one of the following: optical flow-based methods, deep learning-based methods, and facial keypoint-based methods. Optical flow-based methods detect lip movements by calculating the motion vectors of pixels in consecutive video frames. For example, optical flow can be used to calculate the motion field of the lip region. Deep learning-based methods use a trained deep learning model to extract lip movement features. For example, a 3D convolutional neural network can be used to process video sequences and extract the spatiotemporal features of the lips. Facial keypoint-based methods analyze lip movements by detecting and tracking facial keypoints. For example, a facial keypoint detection model can be used to track changes in the coordinates of keypoints on the lips.

[0163] After obtaining lip movement features, the video capture device can identify target objects based on these features. That is, it can compare the objects in the image based on the lip movement features to determine whether they are target objects. If they are target objects, the target image information of the target objects can be extracted.

[0164] The specific object comparison method can be configured according to actual business needs and is not limited here. For example, object analysis methods may include at least one of the following: template matching-based methods, deep learning-based methods, and facial key point-based methods. Template matching-based methods compare the lip movement features extracted from the image with a pre-stored lip movement feature template library to find the most similar template, thereby identifying the target object. Deep learning-based methods use a trained second deep learning model to model and analyze the lip movement feature sequence in the image, thereby identifying the target object. Facial key point-based methods identify the target object from the image by using multimodal features that fuse lip movement features with other facial key point features.

[0165] After obtaining the target image information of the target object, the voiceprint features of the target object can be associated with the target image information to achieve a correspondence between the voiceprint features and the target image information. This ensures that the same target object has both voiceprint features in the audio modality and target image information in the image modality. Voiceprint features refer to speech parameters such as speech waveform, frequency, intensity, and timing in audio data that reflect the physiological and behavioral characteristics of the target object, used to identify and distinguish the identity of different speakers. Target image information refers to the image content in video frames or video clips containing the target object captured by video acquisition equipment.

[0166] According to embodiments of this disclosure, by combining lip movement feature extraction with target object recognition and associating voiceprint features with image content, multimodal information fusion is achieved, which helps to improve the accuracy and reliability of target object recognition.

[0167] The specific association method can be configured according to actual business needs and is not limited here. For example, the association method may include at least one of the following: a mapping-based method and a fusion-based method. The mapping-based method refers to establishing a mapping table between voiceprint features and target image information in the target database, and associating the two through the unique identifier of the target object. The fusion-based method refers to fusing voiceprint features and target image information to form a new fused feature, thereby achieving the association between the two.

[0168] For example, if the target database is a relational database, two tables can be created. The first table can be used to store voiceprint features, such as feature vectors and target object identifiers. The second table can be used to store target image information, such as image data, target object identifiers, and timestamps. The unique identifier of the target object can be used as the association key for both tables, thereby associating the voiceprint features stored in the first table with the target image information stored in the second table.

[0169] For example, if the target database is a non-relational database, the voiceprint features and image information can be stored as different fields in a data table. The data table contains a unique identifier for the target object, thereby enabling the association between each voiceprint feature and each target image information.

[0170] According to embodiments of this disclosure, by associating the voiceprint features of the target object with the target image information, information from both sound and image modalities can be comprehensively utilized, which helps to improve the accuracy and reliability of object recognition.

[0171] In one example, the target database includes multiple sets of associations between voiceprint features and target image information. When a voice signal is received based on a multi-layer microphone, the voiceprint features of the voice signal can be extracted. Based on the voiceprint features of the voice signal, multiple voiceprint features in the target database are matched, and the target image information corresponding to the voiceprint features that match the voiceprint features of the voice signal is taken as the target image information corresponding to the voiceprint features of the voice signal.

[0172] By acquiring the current image information captured by the video capture device, it can be determined whether the current image information matches the target image information. The current image information refers to the image information of the current scene captured in real time by the video capture device, which may include image content such as objects currently engaged in activity. If they do not match, it can be determined that the captured object is not the target speaker corresponding to the voice signal. If they match, it can be determined that the captured object is the target speaker corresponding to the voice signal.

[0173] The method for determining whether the current screen information matches the target screen information can be configured according to actual business needs and is not limited here. For example, the determination method may include at least one of the following: image feature matching-based methods and deep learning-based methods, etc.

[0174] For example, the image feature matching method refers to using an image feature matching algorithm to extract each first feature point in the current image information and each second feature point in the target image information, and determine the matching degree between each first feature point and each second feature point. If the matching degree exceeds a preset matching degree threshold, it can be determined that the current image information and the target image information are consistent.

[0175] Regarding the aforementioned preset matching threshold, a higher threshold increases the standard for determining whether the current image information matches the target image information, reducing the possibility of misjudgment. Conversely, a lower threshold decreases the standard for determining whether the current image information matches the target image information, increasing the possibility of misjudgment. Therefore, the aforementioned preset matching threshold can be adapted to the application scenario of the audio processing method.

[0176] For example, a deep learning-based method refers to using a deep information model to extract and classify features of the current image information and the target image information respectively, to obtain a first classification result of the current image information and a second classification result of the target image information. If the first classification result and the second classification result are the same, it can be determined that the current image information and the target image information are consistent.

[0177] According to embodiments of this disclosure, by combining voiceprint features and image information, voiceprint features are extracted and corresponding target image information is obtained when the multi-layer microphone receives the voice signal. This information is then compared with the current image information captured by the video acquisition device. This allows for the comprehensive judgment of information from both sound and image modalities, overcoming the errors that may occur with single-modal recognition, enhancing the adaptability to complex environments and various interference factors, and improving the accuracy and reliability of target speaker identification.

[0178] The above are merely exemplary embodiments, but are not limited thereto. Other audio processing methods known in the art may also be included, as long as they can achieve real-time and accurate audio and video tracking of the speaker or other target objects.

[0179] Based on the above audio processing method, the present invention also provides an audio device. The following will be combined with... Figure 8 The device is described in detail.

[0180] Figure 8 A block diagram of an audio device suitable for implementing an audio processing method according to an embodiment of the present disclosure is shown schematically. Figure 8The audio device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments disclosed herein.

[0181] like Figure 8 As shown, a computer audio device 800 according to an embodiment of the present disclosure includes a processor 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage portion 809 into a random access memory (RAM) 803. The processor 801 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 801 may also include onboard memory for caching purposes. The processor 801 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0182] RAM 803 stores various programs and data required for the operation of audio device 800. Processor 801, ROM 802, and RAM 803 are interconnected via bus 804.

[0183] According to embodiments of this disclosure, the audio device 800 may further include an input / output (I / O) interface 805, which is also connected to a bus 804. The audio device 800 may also include one or more of the following components connected to the input / output (I / O) interface 805: an input section 806 including a keyboard, mouse, etc.; an output section 807 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the input / output (I / O) interface 805 as needed. A removable medium 811, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 810 as needed so that computer programs read from it can be installed into the storage section 808 as needed.

[0184] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the audio processing method according to the embodiments of this disclosure.

[0185] In this disclosure, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0186] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods provided in the embodiments of this disclosure. When the computer program product is run on an audio device, the program code is used to enable the audio device to implement the audio processing methods provided in the embodiments of this disclosure.

[0187] When the computer program is executed by the processor 801, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0188] According to embodiments of this disclosure, program code for executing computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages.

[0189] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. It should also be noted that in some alternative implementations, the functions indicated in the boxes may occur in a different order than those shown in the drawings.

[0190] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of this disclosure is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. An audio processing method, characterized in that, The method is applied to an audio acquisition device, the audio acquisition device comprising: a multi-layer microphone; the method comprising: Based on the audio data of the target object received by the multi-layer microphone, the position information of the target object relative to the audio acquisition device is determined, and the position information includes horizontal position information and vertical position information. Based on the location information, a first tracking strategy is determined; The target object is tracked and its sound is picked up based on the first tracking strategy.

2. The method according to claim 1, characterized in that, The audio acquisition device establishes a connection with the video acquisition device; The method further includes: The audio acquisition device sends the location information to the video acquisition device, so that the video acquisition device determines a second tracking strategy based on the location information, or the audio acquisition device sends the second tracking strategy determined based on the location information to the video acquisition device; The second tracking strategy is used to adjust the video capture device to track and capture the target object, so that the preset position of the target object is within the shooting range.

3. The method according to claim 2, characterized in that, The preset position is the lip position or a face position that includes the lips; the method further includes: The video acquisition device extracts lip movement features in the lip region using a lip movement detection algorithm; The video acquisition device identifies the target object based on the lip movement features and extracts the image information of the target object; it then associates the voiceprint features of the target object with the image content of the target object.

4. The method according to claim 2, characterized in that, The method further includes: Extract the voiceprint features of the target object, receive the target image information of the target object sent by the video acquisition device, and associate and store the voiceprint features of the target object with the target image information in the target database.

5. The method according to claim 4, characterized in that, The target database includes multiple sets of association relationships between voiceprint features and target image information; the method further includes: When a voice signal is received based on the multi-layer microphone, the voiceprint features of the voice signal are extracted, and the target image information corresponding to the voiceprint features is obtained. The system acquires the current image information captured by the video acquisition device, determines whether the current image information is consistent with the target image information, and if they are consistent, determines that the captured object is the target speaker corresponding to the voice signal.

6. The method according to claim 2, characterized in that, The audio acquisition device and the video acquisition device establish a communication connection; The communication connection is established when the audio acquisition device detects that the video acquisition device is in a present state through the connection component, and the audio acquisition device supplies power to the video acquisition device through the connection component and establishes the connection via message handshake.

7. The method according to claim 6, characterized in that, The connection component of the audio acquisition device includes a magnetic component and a positioning component, and the audio acquisition device establishes a connection with the video acquisition device through the magnetic component and the positioning component.

8. The method according to claim 1, characterized in that, The audio acquisition device includes a lifting mechanism, which is used to adjust the height of the audio acquisition device. Determining the first tracking strategy based on the location information includes: The current height of the audio acquisition device is obtained, and the position information of the target object is determined based on the current height and angle information, wherein the angle information includes: horizontal angle and pitch angle; the first tracking strategy is determined based on the position information.

9. The method according to claim 8, characterized in that, The step of determining the first tracking strategy based on the location information and tracking and picking up sound from the target object based on the first tracking strategy includes: The lifting mechanism is adjusted to a target height based on the location information of the target object, wherein the target height is determined based on the location information of the target object.

10. The method according to any one of claims 1 to 9, characterized in that, When the beam adjustment type is the first adjustment type, determining the first tracking strategy includes: Based on the location information, a first tracking strategy is determined, the first tracking strategy including: forming a target beam pointing towards the target object; The method further includes: The compensation delay amount for each of the audio acquisition units is determined based on the target beam; Beam adjustment is performed based on the compensation delay amount used for each of the audio acquisition units to form a target beam for receiving sound from the target object.

11. The method according to any one of claims 1 to 9, characterized in that, When the beam adjustment type is the second adjustment type, determining the first tracking strategy includes: Based on the location information, the first tracking strategy is determined, and the first tracking strategy includes: selecting a target audio acquisition unit group pointing to the target object; The method further includes: Based on the target audio acquisition unit group, an acquisition instruction is generated. The acquisition instruction is used to instruct the audio acquisition device to switch from the currently used audio acquisition unit group to the target audio acquisition unit group, and to use the target audio acquisition unit group to acquire audio from the target object.

12. The method according to claim 1, characterized in that, The audio acquisition device includes at least two layers of audio acquisition unit groups, and the audio acquisition unit groups include multiple audio acquisition units. The multiple audio acquisition units include at least one of the following structures: one-dimensional array, two-dimensional array, and three-dimensional array. The one-dimensional array is an array composed of multiple audio acquisition units arranged in a one-dimensional straight line.

13. The method according to claim 12, characterized in that, Determining the position information of the target object relative to the audio acquisition device includes one of the following: Horizontal time delay estimation is performed on the plurality of audio acquisition units to obtain the horizontal position information of the target object relative to the audio acquisition device; or Vertical time delay estimation is performed on the multiple audio acquisition units to obtain the vertical position information of the target object relative to the audio acquisition device.

14. The method according to claim 1, further comprising: In response to receiving an audio stream, voice activity detection is performed on the audio stream; In response to the detection of human voice in the audio stream, a preset detection strategy is used to determine whether the currently speaking human voice belongs to the target object. The preset detection strategy includes at least one of the number of human voices and the sound pressure level of the human voice. In response to the human voice belonging to the target object, the audio data of the target object is determined.

15. An audio device, comprising: One or more processors; Memory, used to store one or more computer programs. The characteristic feature is that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 14.

Images