Pickup equipment, sound signal processing method and equipment thereof and medium

Abstract

The invention discloses a pickup device, a sound signal processing method and device thereof, and a medium, so that the same pickup device can be switched to only have a pure pickup function, and can also be switched to have functions of playing sound and collecting sound at the same time. The invention provides pickup equipment. The pickup equipment comprises a microphone array, an output module, an input module, a loudspeaker and a mode switching module, wherein the mode switching module is used for controlling the pickup equipment to be switched between a first working mode and a second working mode; when the pickup equipment is in the first working mode, the microphone array is controlled to collect sound signals, the sound signals are output to an external host through the output module, and signal transmission between the input module and the loudspeaker is controlled to be cut off; when the pickup equipment is in the second working mode, the microphone array is controlled to collect sound signals, the sound signals are output to the external host through the output module, and the loudspeaker is controlled to play the sound signals from the external host.

Description

Technical Field

[0001] The present application relates to the field of video technology, and in particular to a sound pickup device and a sound signal processing method, device and medium thereof. Background Art

[0002] With the popularity of remote work and online collaboration, real-time communication technology, especially video conferencing, has become an important part of daily office and business. The rapid development of technology has brought about the demand for higher standards of video conferencing systems, especially in terms of sound quality. High-quality audio is essential to ensure the clarity and effectiveness of communication. Summary of the invention

[0003] The embodiments of the present application provide a sound pickup device and a sound signal processing method, device and medium thereof, which are used to realize a dual-mode sound pickup device, so that the same sound pickup device can be switched to have only a pure sound pickup function, and can also be switched to have the functions of playing sound and collecting sound at the same time.

[0004] An embodiment of the present application provides a sound pickup device, comprising: a microphone array, an output module, an input module, a speaker, and a mode switching module; wherein:

[0005] The microphone array is used to collect sound signals and output them to an external host through the output module;

[0006] The input module is used to input a sound signal to be played provided by an external host and play it through the speaker;

[0007] The mode switching module is used to control the sound pickup device to switch between a first working mode and a second working mode; when the sound pickup device is in the first working mode, the microphone array is controlled to collect sound signals and output them to the external host through the output module, and the signal transmission between the input module and the speaker is disconnected; when the sound pickup device is in the second working mode, the microphone array is controlled to collect sound signals and output them to the external host through the output module, and the speaker is controlled to play the sound signals from the external host.

[0008] The embodiment of the present application provides a sound pickup device including a microphone array, an output module, an input module, a loudspeaker, and a mode switching module; wherein, sound signals are collected by the microphone array and output to an external host through the output module; through the input module, a sound signal to be played provided by the external host can be input and played through the loudspeaker; through the mode switching module, the sound pickup device can be controlled to switch between a first working mode and a second working mode; wherein, when the sound pickup device is in the first working mode, the microphone array is controlled to collect sound signals and output to the external host through the output module, and the signal transmission between the input module and the loudspeaker is controlled to be disconnected; when the sound pickup device is in the second working mode, the microphone array is controlled to collect sound signals and output to the external host through the output module, and the loudspeaker is controlled to play the sound signals from the external host. Therefore, the embodiment of the present application realizes a dual-mode sound pickup device, so that the same sound pickup device can be switched to have only a pure sound pickup function, and can also be switched to have both the function of playing sound and collecting sound.

[0009] In some embodiments, the sound pickup device further comprises:

[0010] The echo cancellation module connected between the microphone array and the output module is used to perform echo cancellation processing on each sound signal collected by the microphone array based on the reference signal.

[0011] In some embodiments, the sound pickup device further comprises:

[0012] The reference signal switching module is used to control the reference signal to come from the input signal provided by the external host through the input module when the sound pickup device is in the first working mode; and to control the reference signal to come from the echo signal of the sound signal played by the speaker when the sound pickup device is in the second working mode.

[0013] In some embodiments, the input signal provided by the external host through the input module is the same as the sound signal played by the external speaker of the external host;

[0014] The sound pickup device also includes:

[0015] The delay calibration module is used to align the delay of the input signal provided by the external host through the input module with the sound signal collected by the microphone array, and use them as the reference signal.

[0016] In some embodiments, the sound pickup device further comprises:

[0017] A noise reduction and dereverberation module is used to perform noise reduction and dereverberation processing on the signal after the echo cancellation processing.

[0018] In some embodiments, the sound pickup device further comprises:

[0019] A beamforming module is used to perform beamforming processing on the signal that has undergone the noise reduction and dereverberation processing to obtain a single-channel signal.

[0020] In some embodiments, the sound pickup device further includes: a gain processing module, and / or, an equalizer;

[0021] The gain processing module is used to perform gain processing on the single-channel signal;

[0022] The equalizer is used to perform equalization processing on the single-channel signal.

[0023] An embodiment of the present application provides a sound signal processing method applied to a sound pickup device, wherein the sound pickup device includes: a microphone array, an output module, an input module, a speaker, and a mode switching module; the method includes:

[0024] Controlling the sound pickup device to be in a first working mode or a second working mode through the mode switching module;

[0025] When the sound pickup device is in the first working mode, the microphone array is controlled to collect sound signals and output them to an external host through the output module, and the signal transmission between the input module and the speaker is controlled to be disconnected;

[0026] When the sound pickup device is in the second working mode, the microphone array is controlled to collect sound signals and output them to the external host through the output module, and the speaker is controlled to play the sound signals from the external host.

[0027] Another embodiment of the present application provides an electronic device, which includes a memory and a processor, wherein the memory is used to store program instructions, and the processor is used to call the program instructions stored in the memory and execute any of the above methods according to the obtained program.

[0028] Another embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable the computer to execute any of the above methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0030] Figure 1 A schematic diagram of the structure of a sound pickup device provided in an embodiment of the present application;

[0031] Figure 2 A schematic diagram of the connection relationship between a sound pickup device and a conference host provided in an embodiment of the present application;

[0032] Figure 3 to Figure 8 Schematic diagrams of various specific structures of the sound pickup device provided in the embodiments of the present application;

[0033] Fig. 9 A schematic flow chart of a method for processing a sound signal of a sound pickup device in a first working mode provided by an embodiment of the present application;

[0034] Fig.10 A schematic flow chart of a method for processing sound signals of a sound pickup device in a second working mode provided by an embodiment of the present application;

[0035] Fig.11 A schematic diagram of the overall process of a sound signal processing method of a sound pickup device provided in an embodiment of the present application;

[0036] Fig.12 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0037] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0038] The embodiments of the present application provide a sound pickup device and a sound signal processing method, device and medium thereof, which are used to realize a dual-mode sound pickup device, so that the same sound pickup device can be switched to have only a pure sound pickup function, and can also be switched to have the functions of playing sound and collecting sound at the same time.

[0039] Among them, the method, device, and medium are based on the same application concept. Since the principles of solving problems by the method, device, and medium are similar, the implementation of the device, medium, and method can refer to each other, and the repeated parts will not be repeated.

[0040] The terms "first", "second", etc. (if any) in the specification and claims of the embodiments of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments described herein can be implemented in an order other than the content illustrated or described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0041] The following examples and embodiments are to be understood as illustrative examples only. Although this specification may refer to "one", "an" or "some" examples or embodiments in several places, this does not mean that each such reference relates to the same example or embodiment, nor does it mean that the feature applies only to a single example or embodiment. Individual features of different embodiments may also be combined to provide other embodiments. In addition, terms such as "including" and "comprising" should be understood as not limiting the described embodiments to only those features that have been mentioned; such examples and embodiments may also include features, structures, units, modules, etc. that are not specifically mentioned.

[0042] The following is a detailed description of each embodiment of the present application in conjunction with the accompanying drawings. It should be noted that the display order of the embodiments of the present application only represents the order of the embodiments, and does not represent the advantages and disadvantages of the technical solutions provided by the embodiments.

[0043] Voice pickup equipment, such as high-quality, versatile microphones, plays a key role in improving the sound quality of video conferences. They can effectively capture sound, reduce background noise interference, and provide a clear call experience through advanced noise reduction technology. In addition, high-end voice pickup equipment can also have automatic gain control and echo cancellation functions, which can further improve the efficiency and participation of the meeting.

[0044] As users' requirements for sound quality increase, more advanced audio processing technologies need to be continuously integrated to meet market demand. The development of these technologies not only improves the sound quality of meetings, but also improves the overall efficiency of meetings, making remote collaboration smoother and more efficient.

[0045] As the requirements for conference sound quality increase, many conference rooms or individuals will be equipped with dedicated conference microphones with algorithm processing. Different conference scenarios have different requirements for microphone functions. The current microphone functions are relatively simple. Some microphones have a built-in speaker, but these microphones only use the built-in speaker to play sound. They do not support the use of other external speakers to play sound, which limits the use scenarios of these microphones.

[0046] In summary, how to design a sound pickup device that supports both pure sound collection but no sound playback (i.e. pure sound pickup mode) and integrated sound collection and playback (i.e. integrated playback and collection mode, referred to as integrated playback and collection mode) with good sound pickup effect has become an urgent problem to be solved.

[0047] The sound pickup device provided in the embodiment of the present application has both a pure sound pickup mode and an integrated playback and collection mode, and can be applied to scenarios such as video conferencing, audio conferencing, Voice over Internet Protocol (VoIP) calls, game interactions, education and training, emergency command, and telemedicine.

[0048] It should be noted that the sound pickup device described in the embodiment of the present application is a general term, which not only has the function of collecting sound, but also has the function of playing sound.

[0049] See also Figure 1 , a sound pickup device 01 provided in an embodiment of the present application includes: a microphone array 001, an output module 002, an input module 003, a speaker 004, and a mode switching module 005; wherein:

[0050] The microphone array 001 is used to collect sound signals and output them to an external host through the output module 002;

[0051] The input module 003 is used to input the sound signal to be played provided by the external host 02 and play it through the speaker 004;

[0052] The mode switching module 005 is used to control the sound pickup device 01 to switch between a first working mode (also called a pure sound pickup mode) and a second working mode (also called a playback and collection integrated mode); wherein, when the sound pickup device is in the first working mode, the microphone array is controlled to collect sound signals and output them to an external host through the output module, and the signal transmission between the input module and the speaker is controlled to be disconnected; when the sound pickup device is in the second working mode, the microphone array is controlled to collect sound signals and output them to an external host through the output module, and the speaker is controlled to play sound signals from the external host.

[0053] Among them, when the sound pickup device is in the first working mode, controlling the input module to disconnect the signal transmission between the speaker can be achieved by hardware disconnection (for example, disconnecting the connection between the input module and the speaker) or software disconnection (for example, controlling the input module to stop sending signals to the speaker).

[0054] That is to say, in the embodiment of the present application, when the dual-mode sound pickup device is in the first working mode, the microphone array is controlled to collect sound signals and output them to the external host through the output module, and the speaker is controlled not to play the sound signals; and when the dual-mode sound pickup device is in the second working mode, the microphone array can be controlled to collect sound signals and output them to the external host through the output module, and the speaker can be controlled to play the sound signals from the external host.

[0055] The connection relationship between the dual-mode sound pickup device and the external host provided in the embodiment of the present application is as follows: Figure 2 As shown, the microphone array for collecting sound signals in the sound pickup device described in the embodiment of the present application may include one microphone or multiple microphones, for example Figure 2 As shown, a microphone array consisting of 8 microphones may be included to collect 8-channel sound signals.

[0056] The dual-mode sound pickup device provided in the embodiment of the present application is connected to an external host through an input module and an output module. Specifically, there are several connection modes, for example: USB connection, POE (Power Over Ethernet) network cable connection, wireless connection, etc.

[0057] In pure sound pickup mode, the external host transmits one channel of playback sound data to the sound pickup device through the input module, and at the same time transmits one channel to the external speaker connected to the external host;

[0058] The external speaker may be, for example, a speaker or a television set or other device with a sound output function.

[0059] In the integrated broadcast and collection mode, the external host only needs to transmit one channel of sound data to the pickup device, and no longer needs to transmit the sound data to the external speaker. The sound can be played externally through the speaker in the pickup device.

[0060] In the above two working modes, the sound pickup device provided in the embodiments of the present application will transmit sound signals to the connected external host through the output module.

[0061] In some embodiments, the sound signal transmitted by the output module of the embodiment of the present application to the connected external host may be a processed sound signal. For example, the sound pickup device of the embodiment of the present application performs echo cancellation, noise reduction, reverberation, beamforming, automatic gain, equalizer (EQ) and other processing on the collected sound signal, and then sends it to the external host, which can be sent to the opposite device (for example, the opposite host participating in the same video conference) through the external host. Specifically:

[0062] In some embodiments, see Figure 3 , the sound pickup device also includes:

[0063] The echo cancellation module 006 connected between the microphone array and the output module is used to perform echo cancellation processing on each sound signal collected by the microphone array based on the reference signal.

[0064] In some embodiments, see Figure 4 , the sound pickup device also includes:

[0065] The reference signal switching module 007 is used to control the reference signal to come from the input signal provided by the external host through the input module when the sound pickup device is in the first working mode; and to control the reference signal to come from the echo signal of the sound signal played by the speaker when the sound pickup device is in the second working mode.

[0066] In some embodiments, the input signal provided by the external host through the input module is the same as the sound signal played by the external speaker of the external host;

[0067] See also Figure 5 , the sound pickup device also includes:

[0068] The delay calibration module 008 is used to align the delay of the input signal provided by the external host through the input module with the sound signal collected by the microphone array, and use them as the reference signal.

[0069] In some embodiments, see Figure 6 , the sound pickup device also includes:

[0070] The noise reduction and reverberation removal module 009 is used for performing noise reduction and reverberation removal processing on the signal after the echo cancellation processing.

[0071] In some embodiments, see Figure 7 , the sound pickup device also includes:

[0072] The beamforming module 010 is used to perform beamforming processing on the signal that has undergone the noise reduction and dereverberation processing to obtain a single-channel signal.

[0073] In some embodiments, see Figure 8 , the sound pickup device further includes: a gain processing module 011, and / or, an equalizer 012;

[0074] The gain processing module 011 is used to perform gain processing on the single-channel signal;

[0075] The equalizer 012 is used to perform equalization (EQ) processing on the single-channel signal.

[0076] It should be noted that the embodiments of the present application are Figure 8 The figure shows that the sound pickup device further includes a gain processing module 011 and an equalizer 012, but in actual application, only one of them may be included, and the connection order of the gain processing module 011 and the equalizer 012 may also be interchanged.

[0077] That is to say, the sound pickup device provided in the embodiment of the present application not only supports the above two working modes, but also can process the sound signal, that is, through the audio processing algorithm, it can realize the following functions: echo cancellation, noise cancellation, dereverberation, beamforming, automatic gain, EQ, etc.

[0078] The following describes examples of the working process of the sound pickup device provided in the embodiments of the present application in different working modes.

[0079] Embodiment 1:

[0080] The dual-mode sound pickup device provided in the embodiment of the present application, if the pure sound pickup mode is used, see Fig. 9 , then the sound signal processing process includes the following steps:

[0081] S901: Control the microphone array to start collecting original sound array signals;

[0082] That is, the microphone array is controlled to collect at least one sound signal, and each sound signal is collected by one microphone;

[0083] S902: Acquire a playback signal provided by an external host, and use the playback signal as a reference signal after delay calibration, and use the reference signal to perform echo cancellation processing on each channel of collected signals;

[0084] That is to say, in this embodiment, the reference signal used in the echo cancellation processing comes from the external host. The external host can transmit the reference signal to the microphone via a USB cable, wireless transmission, POE cable, 3.5mm audio cable, etc. The reference signal provided by the external host is the sound signal that has been played by the external host through a speaker or TV, etc. (i.e., the external speaker of the external host). This sound signal needs to be delayed aligned with the sound signal collected by the microphone array (i.e., delay calibration processing), and after alignment, it is used as a reference signal for echo cancellation processing.

[0085] S903: Performing artificial intelligence (AI) noise reduction and reverberation removal processing on the signal after the echo cancellation processing;

[0086] After processing in this step, the signal collected by the microphone array has noise eliminated and reverberation reduced or eliminated.

[0087] S904: performing beamforming processing on the signal after noise reduction and dereverberation processing;

[0088] After beamforming processing, multi-channel signals become single-channel signals. For example, originally 8-channel signals become 1-channel signals after beamforming processing, that is, 8 channels become single-channel signals.

[0089] S905: performing automatic gain processing and equalization processing on the signal after the beamforming processing;

[0090] An equalizer may be used to perform the equalization processing.

[0091] S906: Output the signal after the automatic gain processing and equalization processing to an external host.

[0092] That is, the data processed as above is finally sent to the external host via USB cable, wireless transmission, POE cable, 3.5mm audio cable, etc., and can be sent to other terminal devices participating in the meeting through the external host.

[0093] Embodiment 2:

[0094] If the dual-mode sound pickup device provided in the embodiment of the present application is used in the broadcasting and collecting integrated mode, see Fig.10 , the signal processing process includes the following steps:

[0095] S101: Control the microphone array to start collecting original sound array signals;

[0096] S102: Acquire a signal to be played provided by an external host, play the signal through a speaker in the dual-mode sound pickup device, and echo-collect the playback signal of the speaker, use the echo-collected signal as a reference signal, and perform echo cancellation processing on each sound signal collected by the microphone array using the reference signal;

[0097] That is to say, in this embodiment, the reference signal used in the echo cancellation process is the echo signal of the sound played by the speaker of the sound pickup device;

[0098] Specifically, the external host can send the sound signal to be played to the sound pickup device through a USB cable, wireless transmission, POE cable, 3.5mm audio cable, etc. After the sound pickup device receives the sound signal to be played sent by the host, it first plays it through the speaker of the sound pickup device itself, and at the same time, through the AD chip, it picks up the sound signal played by the speaker of the sound pickup device itself to obtain the picked up signal, and uses the picked up signal as the reference signal needed in the echo cancellation processing process.

[0099] S103: performing artificial intelligence (AI) noise reduction and reverberation removal processing on the signal after the echo cancellation processing;

[0100] S104: performing beamforming processing on the signal after noise reduction and dereverberation processing;

[0101] S105: performing automatic gain processing and equalization processing on the signal after the beamforming processing;

[0102] S106: Outputting the signal after the automatic gain processing and equalization processing to an external host.

[0103] That is, the data processed as above is finally sent to the host via USB cable, wireless transmission, POE cable, 3.5mm audio cable, etc.

[0104] In summary, see Fig.11 The sound signal processing method of the dual-mode sound pickup device (the sound pickup device includes: a microphone array, an output module, an input module, a speaker, and a mode switching module) provided in the embodiment of the present application includes:

[0105] S111, controlling the sound pickup device to be in a first working mode (i.e., pure sound pickup mode) or a second working mode (i.e., integrated broadcasting and collection mode) through the mode switching module;

[0106] S112, when the sound pickup device is in the first working mode, controlling the microphone array to collect sound signals and outputting them to an external host through the output module, and controlling the input module to disconnect signal transmission from the speaker;

[0107] When the user selects the pure sound pickup mode, the speaker of the sound pickup device stops working, and the external host sends a playback signal to the sound pickup device. The sound pickup device can receive the audio signal from the host in both wired and wireless ways;

[0108] Among them, wired methods include: USB cable, POE network cable, 3.55mm audio cable;

[0109] Wireless methods such as Bluetooth, Star Flash, 2.4G and other transmission protocols are all available.

[0110] S113. When the sound pickup device is in the second working mode, control the microphone array to collect sound signals and output them to the external host through the output module, and control the speaker to play the sound signals from the external host.

[0111] In some embodiments, the method further comprises:

[0112] Through the echo cancellation module connected between the microphone array and the output module, echo cancellation processing is performed on each sound signal collected by the microphone array based on the reference signal.

[0113] For example, adaptive filtering may be used to eliminate echoes. Specifically, echo elimination may be performed using an NLMS algorithm, an RLS algorithm, a Kalman filter algorithm, a deep learning algorithm, or the like.

[0114] In some embodiments, the method further comprises:

[0115] Through the reference signal switching module, when the sound pickup device is in the first working mode, the reference signal is controlled to come from the input signal provided by the external host through the input module; when the sound pickup device is in the second working mode, the reference signal is controlled to come from the echo signal of the sound signal played by the speaker.

[0116] That is to say, when the user selects the integrated broadcast and collection mode for the sound pickup device, the delay calibration module stops working, the speaker of the sound pickup device starts playing sound, and the sound played back is used as a reference signal.

[0117] In some embodiments, the input signal provided by the external host through the input module is the same as the sound signal played by the external speaker of the external host;

[0118] The method further comprises:

[0119] The delay calibration module is used to align the input signal provided by the external host through the input module with the sound signal collected by the microphone array, and then use them as the reference signal.

[0120] Among them, since there is a great uncertainty in the echo path delay between the host and the pickup device, in the pure pickup mode, the sound signal is played through the external speaker connected to the host, and the echo delay will be affected by the playback device itself and the host system. Therefore, it is necessary to perform delay calibration on the reference signal of echo cancellation, and align the delay of the signal sent by the external host to the pickup device with the signal collected by the microphone array of the pickup device. For example, delay detection can be performed through audio fingerprinting algorithm (Audio Fingerprinting), autocorrelation algorithm (Autocorrelation Method), cross-correlation method (Cross-correlation Method), deep learning algorithm (delay estimation based on machine learning), phase difference-based algorithm, etc., and delay alignment processing can be performed according to the detection results.

[0121] The autocorrelation method estimates the delay by calculating the correlation between the signal and itself at different time delays. When the autocorrelation value reaches a peak, the delay can be regarded as the delay of the signal.

[0122] The cross-correlation method calculates the correlation between two signals to find the best alignment point between them. This method is very effective for determining the relative delay between two signals.

[0123] Audio fingerprinting algorithms identify and align signals by analyzing their unique characteristics. This approach can look for specific frequency patterns, rhythmic patterns, or other audio features.

[0124] Deep learning algorithms use machine learning techniques to train models to identify delay features in audio signals. This approach can adapt to a variety of complex audio environments and provide high-precision delay estimation.

[0125] Phase difference based algorithms focus on analyzing the phase difference of signals to estimate delay, especially when two signals have similar amplitudes but are offset in time.

[0126] In some embodiments, the method further comprises:

[0127] The signal after the echo cancellation processing is subjected to noise reduction and dereverberation processing by the noise reduction and dereverberation module.

[0128] For example, multi-channel AI noise reduction and dereverberation can be performed, wherein the algorithm adopted by the AI ​​noise reduction is to use deep learning to perform speech noise reduction processing, such as mask-based AI noise reduction, and end-to-end AI noise reduction based on matching (mapping). The specific neural networks used are, for example, convolutional neural networks (CNN), recurrent neural networks (RNN) or convolutional recurrent networks (CRN); the dereverberation can, for example, use the WPE algorithm, or the WPE algorithm combined with a deep learning algorithm.

[0129] In some embodiments, the method further comprises:

[0130] The signal that has undergone the noise reduction and dereverberation processing is subjected to beamforming processing by a beamforming module to obtain a single-channel signal.

[0131] The beamforming process can be implemented using algorithms such as DS, minimum variance unbiased estimation (Minimum Variance Distortionless Response, MVDR), and GSC.

[0132] In some embodiments, the method further comprises:

[0133] Performing gain processing on the single-channel signal through a gain processing module;

[0134] The single-channel signal after gain processing is equalized by the equalizer.

[0135] The gain processing, such as automatic gain control, can automatically adjust the output volume.

[0136] The equalization processing, such as adjusting the timbre of the speech through EQ, can be implemented using a time domain equalization or a frequency domain equalization algorithm.

[0137] Finally, the processed audio data is transmitted to the external host through the output module and sent to the other end through the host's network module.

[0138] The external speakers described in the embodiments of the present application include, for example, televisions, whiteboards, active speakers, etc.

[0139] In addition, in some embodiments, multiple sound pickup devices can also be cascaded with the host, that is, the same external host can be connected to one sound pickup device or multiple sound pickup devices, thereby achieving comprehensive sound collection and processing for the target scene. The sound signal processing process of each sound pickup device is similar and will not be repeated one by one.

[0140] The following is an introduction to the equipment or device provided in the embodiments of the present application, in which the explanations or examples of technical features that are the same or corresponding to those described in the above method will not be repeated later.

[0141] An electronic device provided in an embodiment of the present application is used to process sound signals of a sound pickup device, wherein the sound pickup device includes: a microphone array, an output module, an input module, a speaker, and a mode switching module; see Fig.12 , the electronic device includes, for example:

[0142] The processor 600 is used to read the program in the memory 620 and execute the following process:

[0143] Controlling the sound pickup device to be in a first working mode or a second working mode through the mode switching module;

[0144] When the sound pickup device is in the first working mode, the microphone array is controlled to collect sound signals and output them to an external host through the output module, and the signal transmission between the input module and the speaker is controlled to be disconnected;

[0145] When the sound pickup device is in the second working mode, the microphone array is controlled to collect sound signals and output them to the external host through the output module, and the speaker is controlled to play the sound signals from the external host.

[0146] In some embodiments, the processor 600 is further configured to read a program in the memory 620 and execute the following process:

[0147] For each sound signal collected by the microphone array, echo cancellation processing is performed based on the reference signal.

[0148] In some embodiments, the processor 600 is further configured to read a program in the memory 620 and execute the following process:

[0149] When the sound pickup device is in the first working mode, the reference signal is controlled to come from the input signal provided by the external host through the input module; when the sound pickup device is in the second working mode, the reference signal is controlled to come from the echo signal of the sound signal played by the speaker.

[0150] In some embodiments, the input signal provided by the external host through the input module is the same as the sound signal played by the external speaker of the external host;

[0151] The processor 600 is further configured to read the program in the memory 620 and execute the following process:

[0152] The input signal provided by the external host through the input module and the sound signal collected by the microphone array are delayed and aligned to serve as the reference signal.

[0153] In some embodiments, the processor 600 is further configured to read a program in the memory 620 and execute the following process:

[0154] The signal after the echo cancellation processing is subjected to noise reduction and reverberation removal processing.

[0155] In some embodiments, the processor 600 is further configured to read a program in the memory 620 and execute the following process:

[0156] The signal after the noise reduction and dereverberation processing is subjected to beamforming processing to obtain a single-channel signal.

[0157] In some embodiments, the processor 600 is further configured to read a program in the memory 620 and execute the following process:

[0158] Perform gain processing and / or equalization processing on the single-channel signal.

[0159] The transceiver 610 is configured to receive and send data under the control of the processor 600 .

[0160] Among them, Fig.12 In the embodiment, the bus architecture may include any number of interconnected buses and bridges, specifically one or more processors represented by processor 600 and various circuits of memory represented by memory 620 are linked together. The bus architecture may also link together various other circuits such as peripherals, voltage regulators, and power management circuits, which are well known in the art and are therefore not further described herein. The bus interface provides an interface. The transceiver 610 may be a plurality of components, namely, a transmitter and a receiver, providing a unit for communicating with various other devices on a transmission medium, including wireless channels, wired channels, optical cables, and other transmission media. For different user devices, the user interface 630 may also be an interface capable of externally and internally connecting required devices, and the connected devices include but are not limited to a keypad, a display, a speaker, a microphone, a joystick, and the like.

[0161] The processor 600 is responsible for managing the bus architecture and general processing, and the memory 620 can store data used by the processor 600 when performing operations.

[0162] In some embodiments, the processor 600 may be a CPU (central processing unit), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array) or a CPLD (Complex Programmable Logic Device), and the processor may also adopt a multi-core architecture.

[0163] The processor calls the computer program stored in the memory to execute any of the methods provided in the embodiments of the present application according to the obtained executable instructions. The processor and the memory can also be arranged physically separately.

[0164] It should be noted here that the above-mentioned device provided in the embodiment of the present application can implement all the method steps implemented in the above-mentioned method embodiment, and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as those in the method embodiment will not be described in detail here.

[0165] It should be noted that the division of modules in the embodiments of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, each functional module in each embodiment of the present application may be integrated into a processing unit, or each module may exist physically separately, or two or more modules may be integrated into one unit. The above-mentioned integrated modules may be implemented in the form of hardware or in the form of software functional units.

[0166] If the integrated module is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program codes.

[0167] Any of the devices or apparatuses provided in the embodiments of the present application may specifically be a desktop computer, a portable computer, a smart phone, a tablet computer, a personal digital assistant (PDA), etc. It may include a central processing unit (CPU), a memory, an input / output device, etc. The input device may include a keyboard, a mouse, a touch screen, etc. The output device may include a display device, such as a liquid crystal display (LCD), a cathode ray tube (CRT), etc.

[0168] The memory may include a read-only memory (ROM) and a random access memory (RAM), and provides the processor with program instructions and data stored in the memory. In an embodiment of the present application, the memory may be used to store the program of any of the methods provided in the embodiments of the present application.

[0169] The processor calls the program instructions stored in the memory, and the processor is used to execute any of the methods provided in the embodiments of the present application according to the obtained program instructions.

[0170] The present application embodiment also provides a computer program product or computer program, which includes a computer instruction, which is stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction so that the computer device executes any of the methods described in the above embodiments. The program product can use any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, - but not limited to - an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0171] The embodiment of the present application provides a computer-readable storage medium for storing computer program instructions used by the apparatus provided in the embodiment of the present application, which includes a program for executing any of the methods provided in the embodiment of the present application. The computer-readable storage medium may be a non-transitory computer-readable medium.

[0172] The computer-readable storage medium can be any available medium or data storage device that can be accessed by a computer, including but not limited to magnetic storage (such as floppy disks, hard disks, magnetic tapes, magneto-optical disks (MO), etc.), optical storage (such as CD, DVD, BD, HVD, etc.), and semiconductor storage (such as ROM, EPROM, EEPROM, non-volatile memory (NANDFLASH), solid-state drive (SSD)), etc.

[0173] It should be understood that:

[0174] The access technology through which entities in the communication network transmit traffic can be any suitable current or future technology, such as WLAN (Wireless Local Access Network), WiMAX (Worldwide Interoperability for Microwave Access), LTE, LTE-A, 5G, Bluetooth, infrared, etc.; in addition, the embodiments can also apply wired technology, for example, IP-based access technology, such as a wired network or a fixed line.

[0175] Embodiments suitable for being implemented as software code or a portion thereof and running using a processor or processing functionality are independent of the software code and may be specified using any known or future developed programming language, such as a high-level programming language such as objective-C, C, C++, C#, Java, Python, Javascript, other scripting languages, etc., or a low-level programming language such as machine language or assembler.

[0176] The implementation of the embodiments is hardware independent and may be implemented using any known or future developed hardware technology or any mixture thereof, such as a microprocessor or CPU (central processing unit), MOS (metal oxide semiconductor), CMOS (complementary MOS), BiMOS (bipolar MOS), BiCMOS (bipolar CMOS), ECL (emitter coupled logic) and / or TTL (transistor-transistor logic).

[0177] Embodiments may be implemented as separate devices, apparatuses, units, components, or functions, or in a distributed manner, for example, one or more processors or processing functions may be used or shared in a process, or one or more processing segments or processing portions may be used and shared in a process, wherein one physical processor or more than one physical processor may be used to implement one or more processing portions dedicated to a specific process as described.

[0178] The device may be implemented by a semiconductor chip, a chipset, or a (hardware) module including such a chip or chipset.

[0179] The embodiments may also be implemented as any combination of hardware and software, such as ASIC (Application Specific IC (Integrated Circuit)) components, FPGA (Field Programmable Gate Array) or CPLD (Complex Programmable Logic Device) components or DSP (Digital Signal Processor) components.

[0180] The embodiments may also be implemented as a computer program product including a computer usable medium having computer readable program code embodied therein, the computer readable program code being adapted to perform the processes as described in the embodiments, wherein the computer usable medium may be a non-transitory medium.

[0181] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) that contain computer-usable program code.

[0182] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0183] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0184] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0185] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A sound pickup device, characterized in that: The sound pickup device includes: a microphone array, an output module, an input module, a speaker, and a mode switching module; wherein: The microphone array is used to collect sound signals and output them to an external host through the output module; The input module is used to input a sound signal to be played provided by an external host and play it through the speaker; The mode switching module is used to control the sound pickup device to switch between a first working mode and a second working mode; when the sound pickup device is in the first working mode, the microphone array is controlled to collect sound signals and output them to the external host through the output module, and the signal transmission between the input module and the speaker is disconnected; when the sound pickup device is in the second working mode, the microphone array is controlled to collect sound signals and output them to the external host through the output module, and the speaker is controlled to play the sound signals from the external host.

2. The sound pickup device according to claim 1, characterized in that The sound pickup device also includes: The echo cancellation module connected between the microphone array and the output module is used to perform echo cancellation processing on each sound signal collected by the microphone array based on the reference signal.

3. The sound pickup device according to claim 2, characterized in that: The sound pickup device also includes: The reference signal switching module is used to control the reference signal to come from the input signal provided by the external host through the input module when the sound pickup device is in the first working mode; and to control the reference signal to come from the echo signal of the sound signal played by the speaker when the sound pickup device is in the second working mode.

4. The sound pickup device according to claim 3, characterized in that: The input signal provided by the external host through the input module is the same as the sound signal played by the external speaker of the external host; The sound pickup device also includes: The delay calibration module is used to align the delay of the input signal provided by the external host through the input module with the sound signal collected by the microphone array, and use them as the reference signal.

5. The sound pickup device according to claim 2, characterized in that: The sound pickup device also includes: A noise reduction and dereverberation module is used to perform noise reduction and dereverberation processing on the signal after the echo cancellation processing.

6. The sound pickup device according to claim 5, characterized in that: The sound pickup device also includes: A beamforming module is used to perform beamforming processing on the signal that has undergone the noise reduction and dereverberation processing to obtain a single-channel signal.

7. The sound pickup device according to claim 6, characterized in that The sound pickup device further includes: a gain processing module, and / or an equalizer; The gain processing module is used to perform gain processing on the single-channel signal; The equalizer is used to perform equalization processing on the single-channel signal.

8. A sound signal processing method applied to a sound pickup device, characterized in that: The sound pickup device comprises: a microphone array, an output module, an input module, a speaker, and a mode switching module; the method comprises: Controlling the sound pickup device to be in a first working mode or a second working mode through the mode switching module; When the sound pickup device is in the first working mode, the microphone array is controlled to collect sound signals and output them to an external host through the output module, and the signal transmission between the input module and the speaker is controlled to be disconnected; When the sound pickup device is in the second working mode, the microphone array is controlled to collect sound signals and output them to the external host through the output module, and the speaker is controlled to play the sound signals from the external host.

9. An electronic device, characterized in that: include: A memory for storing program instructions; A processor is used to call the program instructions stored in the memory and execute the method according to claim 8 according to the obtained program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to cause the computer to execute the method of claim 8.

Images