Fusion system and audio data processing method for microphone array, electronic equipment and storage medium

Abstract

The invention discloses a fusion system for a microphone array, an audio data processing method, an electronic device and a storage medium wherein the system comprises: a multiplexer for collecting audio data from a plurality of audio processing units, the multiplexer employing a model of independent writing by a plurality of producers and reading by a consumer, each producer manages a respective data block linked list, and each producer has a respective mutual exclusion lock, so that writing competition is reduced; the sound mixing matrix is connected to the multiplexer and used for carrying out sound mixing processing on the collected audio data, the sound mixing matrix comprises a channel switch table and a gain coefficient table, and sound mixing calculation is achieved in a table look-up product addition mode; and the synchronous triggering mechanism is used for enabling the plurality of audio processing units to run at the same rhythm and ensuring that the data collected by the multiplexer is continuous. According to the embodiment of the invention, through the fusion design of the multiplexer and the sound mixing matrix, the problems of data synchronization, calculation efficiency and clock consistency in the microphone array system are effectively solved.

Description

Technical Field

[0001] This invention belongs to the field of audio processing technology, and particularly relates to a fusion system for microphone arrays, an audio data processing method, an electronic device, and a storage medium. Background Technology

[0002] In existing audio processing systems, microphone arrays typically contain multiple audio processing units for acquiring and processing multiple audio signals. A multiplexer, acting as a data collector, is responsible for aligning multiple audio data streams and sending them to subsequent processing modules; a mixing matrix is ​​used to mix the multiple audio signals to generate the required output channel signals.

[0003] However, existing technologies have the following drawbacks: 1. Data contention and latency issues in multiplexers: In a multi-producer single-consumer model, traditional multiplexers may experience data writing contention among producers, leading to data blocking or latency and affecting real-time performance.

[0004] 2. Low computational efficiency of mixing matrix: Existing mixing matrices mostly use conditional judgment (such as if / else) to control channel switching and gain, which has low computational efficiency and is not suitable for high real-time scenarios.

[0005] 3. Poor synchronization of audio processing units: Due to inconsistent audio data source clocks, the execution rhythms of each processing unit are different, resulting in discontinuous data collected by the multiplexer, requiring additional buffering and increasing system latency.

[0006] 4. Insufficient handling of data volume fluctuations: Clock drift or jitter in audio data sources leads to unstable input data volume. Existing resampling technologies are mostly fixed ratios, which cannot adapt to dynamic fluctuations and affect output quality.

[0007] These shortcomings stem from the contention mechanisms of multiplexers, inefficient calculations of mixing matrices, lack of global synchronization triggering mechanisms, and fixed resampling methods in existing technologies. These problems have long existed in the field of audio processing, and are particularly prominent in microphone array systems with multi-channel, low-latency requirements. Summary of the Invention

[0008] The embodiments of the present invention are intended to solve at least one of the above-mentioned technical problems.

[0009] In a first aspect, embodiments of the present invention provide a fusion system for a microphone array, comprising: a multiplexer for collecting audio data from multiple audio processing units, the multiplexer employing a model of multiple producers independently writing and one consumer reading, each producer managing its own data block linked list, and each producer having its own mutex lock to reduce write contention; a mixing matrix connected to the multiplexer for mixing the collected audio data, the mixing matrix including a channel switch table and a gain coefficient table, and mixing calculations being performed by looking up the table and adding the products; a synchronization triggering mechanism for ensuring that the multiple audio processing units operate at the same rhythm, thereby ensuring the continuity of the data collected by the multiplexer; and a dynamic resampling mechanism for handling data fluctuations caused by inconsistent audio data source clocks, the dynamic resampling mechanism using upper and lower watermarks to control the resampling timing, performing dynamic resampling when the input data exceeds the watermark to stabilize the output data.

[0010] Secondly, embodiments of the present invention provide an audio data processing method, comprising: enabling multiple audio processing units to execute in lockstep via a synchronous triggering mechanism; collecting audio data from the multiple audio processing units via a multiplexer, wherein the multiplexer adopts a model of multiple producers independently writing and one consumer reading, with each producer independently writing to its own channel; performing mixing processing on the collected audio data via a mixing matrix, wherein the mixing matrix uses a channel switching table and a gain coefficient table for mixing calculations; and handling audio data source clock inconsistencies via a dynamic resampling mechanism, using a water level line to control the resampling process, and performing dynamic resampling when the input data volume exceeds the water level line to stabilize the output data volume.

[0011] Thirdly, embodiments of the present invention provide an electronic device comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform any of the above-described audio data processing methods of the present invention.

[0012] Fourthly, embodiments of the present invention provide a storage medium storing one or more programs including execution instructions, the execution instructions being readable and executable by electronic devices (including but not limited to computers, servers, or network devices, etc.) to perform any of the above-described audio data processing methods of the present invention.

[0013] Fifthly, embodiments of the present invention also provide a computer program product, the computer program product including a computer program stored on a storage medium, the computer program including program instructions, which, when executed by a computer, cause the computer to perform any of the above-described audio data processing methods.

[0014] This invention improves system real-time performance and computational efficiency by optimizing contention writing in the multiplexer and efficient calculation of the mixing matrix; the synchronous triggering mechanism reduces data discontinuity; and the dynamic resampling mechanism adapts to clock fluctuations, ensuring output stability, effectively solving the problems of data synchronization, computational efficiency, and clock consistency in microphone array systems. Attached Figure Description

[0015] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 A schematic diagram of a fusion system for a microphone array provided by the present invention; Figure 2 A flowchart of an audio data processing method provided by the present invention; Figure 3 This is a schematic representation of the mixing channel switch provided by the present invention; Figure 4 This invention provides a schematic representation of the gain coefficient. Figure 5 This is a schematic diagram of the audio system architecture provided by the present invention; Figure 6 This is a schematic diagram of the muxer structure provided by the present invention; Figure 7 A schematic diagram of the upper and lower water level line mechanism provided by the present invention; Figure 8 This is a schematic diagram of the structure of an embodiment of the electronic device of the present invention. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other.

[0019] This invention can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, elements, data structures, etc., that perform a specific task or implement a specific abstract data type. This invention can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0020] In this invention, terms such as "module," "device," and "system" refer to relevant entities applied to a computer, such as hardware, combinations of hardware and software, software, or software in execution. More specifically, for example, an element can be, but is not limited to, a process running on a processor, a processor, an object, an executable element, an execution thread, a program, and / or a computer. Furthermore, an application program or script running on a server, and the server itself, can also be an element. One or more elements may be in an execution process and / or thread, and elements may be localized on a single computer and / or distributed across two or more computers, and may be run on various computer-readable media. Elements can also communicate via local and / or remote processes based on signals having one or more data packets, for example, signals from data interacting with another element in a local system, a distributed system, and / or interacting with other systems via signals over a network on the Internet.

[0021] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising" or "including" include not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0022] This embodiment provides a fusion system for microphone arrays, including a multiplexer, a mixing matrix, a synchronization triggering mechanism, and a dynamic resampling mechanism.

[0023] Please refer to Figure 1 The diagram illustrates a fusion system for a microphone array according to an embodiment of the present invention.

[0024] like Figure 1 As shown, 101 is a multiplexer used to collect audio data from multiple audio processing units. The multiplexer adopts a model of multiple producers writing independently and one consumer reading. Each producer manages its own linked list of data blocks, and each producer has its own mutex lock to reduce write contention. 102, Mixing matrix, connected to the multiplexer, used to mix the collected audio data. The mixing matrix includes a channel switch table and a gain coefficient table. Mixing calculation is achieved by looking up the table, multiplying and adding the products. 103, Synchronous triggering mechanism, used to make the multiple audio processing units run at the same rhythm to ensure the continuity of data collected by the multiplexer; 104. Dynamic resampling mechanism is used to handle data volume fluctuations caused by inconsistent audio data source clocks. The dynamic resampling mechanism uses upper and lower water level lines to control the resampling timing. Dynamic resampling is performed when the input data volume exceeds the water level line in order to stabilize the output data volume.

[0025] In this embodiment, 101, a multiplexer, is used to collect audio data from multiple audio processing units (such as algorithms A, B, C, and D). The multiplexer adopts a model of multiple producers writing independently and one consumer reading. Each producer manages its own data block linked list (idle / active linked list), and each producer has its own mutex lock to reduce write contention.

[0026] 102. Mixing Matrix: Connected to the multiplexer, used for mixing the collected audio data. The mixing matrix includes a channel switch table and a gain coefficient table, and the mixing calculation is performed by looking up the table, multiplying and adding the products.

[0027] 103. Synchronous Trigger Mechanism: Used to enable multiple audio processing units to run at the same rhythm, such as synchronously triggering all processing units based on the rhythm of microphone recording (e.g., triggering once every 4ms) to ensure data continuity.

[0028] 104. Dynamic resampling mechanism: used to handle data volume fluctuations caused by inconsistent audio data source clocks. The dynamic resampling mechanism uses upper and lower watermarks to control the resampling timing. Dynamic resampling is performed when the input data volume exceeds the watermark to stabilize the output data volume.

[0029] This embodiment improves system real-time performance and computational efficiency by optimizing contention writing in the multiplexer and efficient calculation of the mixing matrix; the synchronous triggering mechanism reduces data discontinuity; and the dynamic resampling mechanism adapts to clock fluctuations, ensuring output stability.

[0030] In some optional embodiments, the channel switch table of the mixing matrix is ​​used to control the input channel switch state of each output channel, and the gain coefficient table is used to control the global gain of each output channel and the independent gain of each input channel. The mixing calculation is implemented through a three-level loop: the first level loop iterates through the output channels, the second level loop iterates through the data frames, and the third level loop iterates through the input channels, performing multiplicative and additive calculations. The channel switch table controls the input channel switch state of each output channel (0 indicates off, 1 indicates on); the gain coefficient table controls the global gain of each output channel and the independent gain of each input channel; the mixing calculation is implemented through a three-level loop: the first level loop iterates through the output channels, extracting the output channel switch values ​​and global gain values; the second level loop iterates through the data frames (e.g., 4ms data per frame); and the third level loop iterates through the input channels, performing multiplicative and additive calculations.

[0031] The mixing matrix logic is implemented using a three-layer loop. The first layer loop extracts the output channel switch value and global gain value for each output channel. The second layer loop handles each data point of the input channel. The third layer loop performs independent switching and gain coefficient control on a single data point of each input channel and then sums the results. This lookup-table multiplication and addition calculation method significantly accelerates the calculation speed (multipliers and adders are typically single-cycle instructions). This application implements the mixing matrix calculation function through an efficient data organization and calculation method. This embodiment, through lookup-table multiplication and addition, utilizes the single-cycle instruction characteristics of hardware multipliers and adders to greatly accelerate the mixing calculation speed, making it suitable for high real-time scenarios.

[0032] In some optional embodiments, the synchronous triggering mechanism triggers all audio processing units based on the rhythm of microphone recording, causing the audio processing units to execute in lockstep, thereby reducing link latency. The synchronous triggering mechanism triggers all audio processing units based on the rhythm of microphone recording (e.g., reading one frame of data every 4ms), causing the audio processing units to execute in lockstep. For example, the execution rhythm of audio processing units one and two is synchronized with the microphone recording, ensuring that the data collected by the multiplexer is time-aligned. Through lockstep execution, the multiplexer can process data immediately after all data arrives (e.g., execution can begin after 3.5ms), achieving data alignment. This embodiment reduces system latency and improves the real-time performance and continuity of data processing through the synchronous triggering mechanism.

[0033] In some optional embodiments, in the multiplexer, the consumer checks the readable data volume of each producer when reading data. If the readable data volume of any producer is insufficient, it marks the read as blocked. After writing data, the producer checks the read blocking flag and issues a readable notification to release the consumer's read blocking. This embodiment further defines the read blocking mechanism of the multiplexer. When the consumer reads data, it checks the readable data volume of each producer. If the readable data volume of any producer is insufficient, it marks the read as blocked, thus putting the entire consumer into a read blocking state. After writing data, the producer checks the read blocking flag. If it is in a read blocking state, it issues a readable notification to release the consumer's read blocking. After receiving the notification, the consumer re-checks the data volume and performs reading and processing. This embodiment, through the read blocking and notification mechanism, avoids consumer idleness or data loss, improving the reliability and efficiency of data processing.

[0034] In some optional embodiments, the multiplexer supports chained write and read operations. When the write rates of producers are inconsistent, a dynamic data block linked list is used to adapt, whereby each producer acquires a corresponding number of idle data blocks for writing based on its own rate. For example, the multiplexer supports chained write and read operations, and when the write rates of producers are inconsistent, a dynamic data block linked list is used to adapt. Each producer acquires a corresponding number of idle data blocks for writing based on its own rate; for example, producer A acquires 1 data block per 1ms / time, producer B acquires 2 data blocks per 2ms / time, and producer C acquires 8 data blocks per 8ms / time. The consumer reads a fixed number of data blocks each time (e.g., 4ms of data) and ensures data integrity through a readable data volume check. This embodiment, through a chained write and read mechanism, adapts to audio processing units with different rates, improving the system's flexibility and resource utilization.

[0035] In some optional embodiments, the channel switching table and gain coefficient table of the mixing matrix are (M + 1) x N in size, where M is the number of input channels, N is the number of output channels, and the last column is used to control the overall switching and global gain of the output channels. For example, in a 3x4 matrix, the table size is 4x4, and the last column controls the switching and gain of the output channels. This embodiment simplifies the control logic of the mixing matrix through a unified table structure, improving configuration flexibility and computational efficiency.

[0036] The mixing matrix implemented in the microphone array of this application needs to support the following functions: 1. Supports independent switching of each output channel; 2. Supports independent control of input channels involved in each output channel; 3. Supports global gain control for each output channel and independent gain control for each input channel. Please refer to Figure 2The application presents an audio data processing method for use in the aforementioned microphone array fusion system.

[0037] like Figure 2 As shown, in step 201, multiple audio processing units are executed in a lockstep manner through a synchronous triggering mechanism; In step 202, audio data is collected from the plurality of audio processing units through a multiplexer, wherein the multiplexer adopts a model of multiple producers writing independently and one consumer reading, with each producer writing independently to its own channel. In step 203, the collected audio data is mixed using a mixing matrix, wherein the mixing matrix is ​​calculated using a channel switch table and a gain coefficient table. In step 204, the inconsistency of the audio data source clock is handled by a dynamic resampling mechanism. A water level line is used to control the resampling process. Dynamic resampling is performed when the amount of input data exceeds the water level line in order to stabilize the amount of output data.

[0038] For step 201, a synchronous triggering mechanism enables multiple audio processing units to execute in lockstep. For example, audio processing unit one obtains audio from another data source, processes it using an algorithm, and then outputs it to the Muxer. (Multi-input source lockstep execution, also called synchronous execution) means that audio processing units one and two run at the same pace. The data source and microphone recording of audio processing unit one are not completely synchronized. Without this mechanism, audio processing units one and two cannot execute in lockstep, resulting in discontinuous Muxer data.

[0039] Then, for step 202, audio data is collected from the plurality of audio processing units via a multiplexer, wherein the multiplexer adopts a model of multiple producers writing independently and one consumer reading, with each producer writing independently to its own channel; for example, multiple producers write independently with little or no contention between them; only one consumer reads, minimizing contention between the consumer and the producer; chained writing and reading are supported (suitable for situations where the producer and consumer rates are inconsistent), and the write rates between producers can also be inconsistent; producers use their own mutex locks, meaning there is no contention between producers; each producer manages its own list of idle / active data blocks.

[0040] Next, in step 203, the collected audio data is mixed using a mixing matrix, wherein the mixing matrix uses a channel switching table and a gain coefficient table for mixing calculations. This table lookup and product addition method can greatly speed up the calculation (multipliers and adders are usually single-cycle instructions). Channel mixing calculations can also use if / else statements, which are less efficient than product addition calculations. This is why this application uses two tables to implement matrix calculations.

[0041] Finally, for step 204, an audio data source clock inconsistency is handled through a dynamic resampling mechanism. A watermark is used to control the resampling process; dynamic resampling is performed when the input data exceeds the watermark to stabilize the output data. This application's dynamic resampling uses an upper and lower watermark mechanism; appropriate processing is only performed when the watermark is exceeded, ensuring that the final algorithm's data output is always stable.

[0042] This embodiment achieves low-latency, high-efficiency audio data processing through methodological optimization, making it suitable for various microphone array application scenarios.

[0043] In some optional embodiments, the mixing calculation of the mixing matrix includes: traversing the output channels, extracting the output channel switching values ​​and global gain values; traversing the data frames, for each data frame traversing the input channels, performing independent switching and gain coefficient control of the input channels and accumulating the results to obtain the output channel data. This embodiment improves the efficiency and scalability of the mixing calculation through standardized multiplication and accumulation calculations.

[0044] Please refer to Figure 3 and Figure 4 The application presents schematic diagrams of channel switch representation and channel gain coefficient representation.

[0045] like Figure 3 , Figure 4 As shown, the mixing matrix implemented in the microphone array of this application needs to support the following functions: 1. Support independent switching of each output channel; 2. Support independent control of the input channels involved in each output channel; 3. Support global gain control of each output channel and independent gain control of each input channel.

[0046] To address this requirement, this application implements a two-table structure: a mixing channel switch table and a channel gain coefficient table.

[0047] refer to Figure 3 0 represents the channel being closed, and 1 represents the channel being open.

[0048] For an M x N matrix, where M represents the input channel and N represents the input channel, the final matrix switch table size is (M + 1) x N.

[0049] Figure 3 It is a 3x4 matrix, and the last column is the master switch for the output channel.

[0050] refer to Figure 4 For an MxN matrix, where M represents the input channels and N represents the output channels, the final matrix table size is (M + 1) x N.

[0051] Figure 4 The middle column is a 3x4 matrix, and the last column is the output gain control of the output channel.

[0052] The three-level loop implements the mixing matrix logic. The first level loop extracts the output channel switch value and global gain value for each output channel. The second level loop is for each data point of the input channel. The third level loop performs independent switching and gain coefficient control on a data point of each input channel and adds them together.

[0053] This method of multiplication and addition based on a lookup table can greatly speed up the calculation (multipliers and adders are usually single-cycle instructions).

[0054] This patent achieves the mixing matrix calculation function through an efficient data organization and calculation method.

[0055] In engineering implementation, before multi-channel audio data enters the mixing matrix for processing, a multi-channel data collector, also known as a multiplexer, is needed. This multiplexer is responsible for collecting audio from different sources (working threads) in the thread where the mixing matrix is ​​located, thus achieving data alignment.

[0056] (The producer-consumer model in software engineering is equivalent to a multiplexer model with multiple producers and one consumer.) Please refer to Figure 5 The document presents a schematic diagram of the audio system architecture provided in this application.

[0057] like Figure 5As shown, this application describes a real scenario in a microphone array system (audio system architecture): There are two audio processing units, one and two, in the system. Audio processing unit two obtains audio data from the microphone and then performs three different parallel execution flows (worker threads) represented by algorithms ABC. The processing time required for each algorithm is random, that is, the time interval between the output audio after the three algorithms are executed is random. From the perspective of the muxer, the output audio of A / B / C may arrive at the same time or one after the other. However, since ABC is triggered by audio processing unit two, their processing rhythm is consistent (if a frame of data is obtained from the microphone (e.g., 4ms), then ABC of audio processing unit two can always be completed when the next 4ms of data is readable).

[0058] The audio processing unit 1 obtains audio from another data source, processes it using algorithm D, and then outputs it to the Muxer.

[0059] Figure 5 The red line segment in the image represents an innovation of this application (multi-input source lockstep execution, also known as synchronous execution), which means that audio processing units one and two run at the same rhythm.

[0060] The data source and microphone recording of audio processing unit one are not fully synchronized. Without this mechanism, audio processing units one and two cannot perform lockstep execution, resulting in discontinuous Muxer data.

[0061] Through the multi-input source synchronous triggering mechanism, Muxer can run exactly according to the rhythm of microphone recording (if the microphone recording reads every 4ms, then the muxer also collects and processes the data of all ABCD channels every 4ms, and there is only a lag relationship between them).

[0062] By synchronously triggering the execution of ABCD, all data can reach the muxer within 4ms, but the order of arrival varies. The muxer can start execution as long as data is available for ABCD (it is not necessary to wait until 4ms. For example, if the data for ABCD arrives at 3.5ms, 3.4ms, 2.5ms, and 3.2ms respectively, the muxer can start execution immediately at 3.5ms, which can reduce the link latency by 4 - 3.5 = 0.5ms).

[0063] Please refer to Figure 6 The application presents a schematic diagram of the muxer structure provided in this application.

[0064] like Figure 6 As shown, for this data communication model involving multiple producers and a single consumer, the more efficient muxer structure designed in this application needs to meet the following characteristics: 1. Multiple producers write independently, with little or no competition among them; 2. Only one consumer reads the data, minimizing competition between the consumer and the producer; 3. Supports chained writing and reading (suitable for situations where the producer and consumer speeds are inconsistent), and the write speeds between producers can also be inconsistent; 4. Each producer uses its own mutex lock, meaning there is no competition between producers; 5. Each producer manages its own list of idle / active data blocks.

[0065] Figure 6 Each consumer requires 4ms of data to process, while producers A / B / C have rates of 1ms / 2ms / 8ms respectively. Therefore, this application requires the use of 1ms data blocks as the basic data block unit. Producer A acquires (Takes slot) one free block and writes data each time, producer B acquires (Takes slot link) two free blocks and writes data each time, and producer C acquires 8 data blocks and writes data each time. From the consumer's perspective, each time a data is read, the current readable data volume of each producer is recorded. When a producer's readable data volume is insufficient, it is marked as read-blocked, causing the entire consumer to enter a read-blocked state. Each producer checks the read-blocking flag after writing data. If it is in a read-blocked state, it issues a readable notification, allowing the consumer to unblock the data and perform data reading and processing. In this application, the read multi-channel data is sent to the mixing matrix for processing.

[0066] Please refer to this again. Figure 7 They presented a schematic diagram of the water level mechanism provided in this application.

[0067] like Figure 7 As shown, although this application uses a synchronous triggering mechanism to achieve synchronous reading, due to the main reason that the microphone recording and audio data source are not synchronized by clock, each time the microphone recording triggers the audio processing unit to read data, it may not be able to read the desired amount of data (4ms in this example). Therefore, the output of algorithm D may not always be 4ms of data. Thus, this application uses a dynamic resampling mechanism. When the input data is small, resampling generates more data. When the audio processing unit is triggered for the first time and reads less data (e.g., less than 4ms), the next time it is triggered, it will generally read more data (e.g., greater than 4ms). Therefore, the dynamic resampling in this application uses an upper and lower watermark mechanism. Only when the watermark is exceeded will appropriate processing be performed, and the final data output of algorithm D is always stable.

[0068] It should be noted that the resampling technology provided in this application is an audio processing technology, which is generally a fixed ratio, such as resampling 48kHz audio to 16kHz (downsampling), or resampling 16kHz audio to 32kHz (upsampling).

[0069] This application integrates the producer-consumer model, operating system scheduling, multi-threaded synchronization and communication technology, matrix calculation, and audio system data processing methods, and proposes an optimization scheme for the actual needs of microphone arrays.

[0070] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of combined actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, as some steps can be performed in other orders or simultaneously according to the present invention. Secondly, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to the present invention. In the above embodiments, the descriptions of each embodiment have their own emphasis; for parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0071] In some embodiments, the present invention provides a non-volatile computer-readable storage medium storing one or more programs including execution instructions, which can be read and executed by an electronic device (including but not limited to a computer, server, or network device, etc.) to perform any of the above-described audio data processing methods of the present invention.

[0072] In some embodiments, the present invention also provides a computer program product, the computer program product including a computer program stored on a non-volatile computer-readable storage medium, the computer program including program instructions, which, when executed by a computer, cause the computer to perform any of the above-described audio data processing methods.

[0073] In some embodiments, the present invention also provides an electronic device comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform an audio data processing method.

[0074] Figure 8 This is a schematic diagram of the hardware structure of an electronic device for performing an audio data processing method according to another embodiment of this application, as shown below. Figure 8 As shown, the device includes: One or more processors 810 and memory 820, Figure 8Take the 810 processor as an example.

[0075] The device for performing the audio data processing method may further include an input device 830 and an output device 840.

[0076] The processor 810, memory 820, input device 830, and output device 840 can be connected via a bus or other means. Figure 8 Taking the example of a connection between China and Israel via a bus.

[0077] The memory 820, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the audio data processing method in the embodiments of this application. The processor 810 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions, and modules stored in the memory 820, thereby implementing the audio data processing method of the above-described method embodiments.

[0078] The memory 820 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the microphone auto-configuration device. Furthermore, the memory 820 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 820 may optionally include memory remotely located relative to the processor 810, and these remote memories can be connected to the microphone auto-configuration device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0079] The input device 830 can receive input digital or character information and generate signals related to user settings and function control of the pickup area automatic configuration device. The output device 840 may include a display device such as a display screen.

[0080] The one or more modules are stored in the memory 820, and when executed by the one or more processors 810, they perform the audio data processing method in any of the above method embodiments.

[0081] The above-described product can perform the methods provided in the embodiments of this application, and has the corresponding functional modules and beneficial effects for performing the methods. Technical details not described in detail in this embodiment can be found in the methods provided in the embodiments of this application.

[0082] The electronic devices in this application embodiments exist in various forms, including but not limited to: (1) Mobile communication devices: These devices are characterized by their mobile communication capabilities and primarily aim to provide voice and data communication. These terminals include smartphones, multimedia phones, feature phones, and low-end phones.

[0083] (2) Ultra-mobile personal computer devices: These devices fall under the category of personal computers, possessing computing and processing capabilities, and generally also have mobile internet access features. These terminals include: PDAs, MIDs, and UMPCs, etc.

[0084] (3) Portable entertainment devices: These devices can display and play multimedia content. This category includes audio and video players, handheld game consoles, e-book readers, as well as smart toys and portable car navigation devices.

[0085] (4) Other airborne electronic devices with data interaction capabilities, such as vehicle-mounted systems installed on vehicles.

[0086] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0087] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software plus a general-purpose hardware platform, or of course, using hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0088] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A fusion system for microphone arrays, characterized in that, include: A multiplexer is used to collect audio data from multiple audio processing units. The multiplexer adopts a model of multiple producers writing independently and one consumer reading. Each producer manages its own linked list of data blocks and each producer has its own mutex lock to reduce write contention. A mixing matrix, connected to the multiplexer, is used to mix the collected audio data. The mixing matrix includes a channel switch table and a gain coefficient table, and the mixing calculation is achieved by looking up the table, multiplying and adding the products. A synchronous triggering mechanism is used to make the multiple audio processing units run at the same rhythm, ensuring the continuity of data collected by the multiplexer; A dynamic resampling mechanism is used to handle data fluctuations caused by inconsistent audio data source clocks. The dynamic resampling mechanism uses upper and lower watermarks to control the resampling timing. Dynamic resampling is performed when the input data exceeds the watermark to stabilize the output data.

2. The system according to claim 1, characterized in that, The channel switch table of the mixing matrix is ​​used to control the input channel switch state of each output channel, and the gain coefficient table is used to control the global gain of each output channel and the independent gain of each input channel. The mixing calculation is implemented through three loops: the first loop iterates through the output channels, the second loop iterates through the data frames, and the third loop iterates through the input channels, performing multiplication and accumulation calculations.

3. The system according to claim 1, characterized in that, The synchronous triggering mechanism triggers all audio processing units based on the rhythm of microphone recording, enabling the audio processing units to execute in lockstep, thereby reducing link latency.

4. The system according to claim 1, characterized in that, In the multiplexer, when the consumer reads data, it checks the amount of readable data of each producer. When the amount of readable data of any producer is insufficient, it marks the read as blocked. After the producer writes data, it checks the read blocking flag and issues a readable notification to release the consumer's read blocking.

5. The system according to claim 1, characterized in that, The multiplexer supports chained writing and reading. When the writing rates of producers are inconsistent, the data block linked list is dynamically managed to adapt, in which the producer obtains a corresponding number of free data blocks for writing according to its own rate.

6. The system according to claim 1, characterized in that, The channel switch table and gain coefficient table of the mixing matrix are of size (M + 1) x N, where M is the number of input channels, N is the number of output channels, and the last column is used to control the master switch and global gain of the output channels.

7. An audio data processing method, used in the system according to any one of claims 1-6, characterized in that, include: Multiple audio processing units are executed in lockstep via a synchronous triggering mechanism; Audio data is collected from the plurality of audio processing units through a multiplexer, wherein the multiplexer adopts a model of multiple producers writing independently and one consumer reading, with each producer writing independently to its own channel. The collected audio data is mixed using a mixing matrix, wherein the mixing matrix is ​​used to perform mixing calculations using a channel switching table and a gain coefficient table. The inconsistent audio data source clock is handled by a dynamic resampling mechanism. A water level is used to control the resampling process. Dynamic resampling is performed when the amount of input data exceeds the water level to stabilize the amount of output data.

8. The method according to claim 7, characterized in that, The mixing calculation of the mixing matrix includes: traversing the output channels, extracting the output channel switch values ​​and global gain values; traversing the data frames, traversing the input channels for each data frame, performing independent switching and gain coefficient control of the input channels and accumulating the results to obtain the output channel data.

9. An electronic device comprising: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the steps of the method of any one of claims 7 or 8.

10. A storage medium having a computer program stored thereon, characterized in that, When the program is executed by a processor, it implements the steps of the method according to any one of claims 7 or 8.

Images