Embedded firmware audio data modification method, device, medium and computer equipment

Abstract

The present invention discloses an embedded firmware audio data modification method, device, medium and computer equipment, which relate to the technical field of audio processing, including: performing correlation analysis and encoding processing on adjacent audio frames in a secondary cache, and storing the encoded audio frames in a third-level cache; dividing the audio frames in the third-level cache into M frames and N frames according to the computing power and bandwidth status of the embedded device, the M frames are locally encoded by the embedded device, and the N frames are transmitted to a back-end device; after the back-end device receives and processes the N frames of audio signals, the processing result is returned to the embedded device; the embedded device merges the locally encoded M frames of audio signals with the N frames of audio signals processed by the back-end in time sequence, and outputs complete audio signal data; through timestamp synchronization, interpolation frame filling and frame skipping processing, the problems of audio frame loss and time sequence discontinuity are solved, the integrity and continuity of the audio signal are guaranteed, and the efficiency and quality of audio processing are significantly improved.

Description

Technical Field

[0001] The present invention relates to the technical field of audio processing, and in particular to an embedded firmware audio data modification method, device, medium and computer equipment. Background Art

[0002] With the development of audio processing technology, embedded devices are increasingly used in audio acquisition, processing and transmission. In embedded devices, the acquisition, processing and transmission of audio signals are usually limited by computing power and bandwidth. Existing audio processing technologies mostly use fixed encoding schemes, which cannot dynamically adjust the processing method according to the computing power of the device and the network bandwidth. Therefore, when the bandwidth is limited or the computing power is insufficient, the audio quality is often affected. In addition, the noise processing and encoding methods of audio signals are also relatively simple. Especially when facing complex audio environments, existing technologies are difficult to effectively distinguish between noise and valid signals, resulting in a decrease in transmission efficiency and audio quality.

[0003] However, in the process of audio signal processing, the existing technology has shortcomings in handling the transmission and merging of audio frames, especially the inflexible handling when frames are lost or delayed. Most current technologies use simple timestamp alignment or fixed buffer processing methods, which cannot effectively deal with network jitter or bandwidth instability, resulting in possible breakage or distortion of audio signals when spliced. In addition, the existing audio frame priority division and cache management are relatively rough, and fail to fully utilize the resources of embedded devices for real-time optimization. Therefore, how to dynamically prioritize, encode and transmit audio frames under limited bandwidth and computing power has become an urgent problem to be solved. Summary of the invention

[0004] In view of the above existing problems, the present invention is proposed.

[0005] Therefore, the present invention provides an embedded firmware audio data modification method to solve the problems of audio quality degradation, frame loss and timing discontinuity caused by bandwidth and computing power limitations of audio signals in embedded devices.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0007] In a first aspect, the present invention provides an embedded firmware audio data modification method, which comprises:

[0008] Get the real-time audio signal, split it into fixed-length audio frames, and store them in the first-level cache;

[0009] Perform noise detection on the audio frames in the first-level cache, assign priorities based on the noise detection results, and store the audio frames with assigned priorities in the second-level cache;

[0010] Perform correlation analysis on adjacent audio frames in the secondary cache and perform encoding processing, and store the encoded audio frames in the tertiary cache;

[0011] According to the computing power and bandwidth of the embedded device, the audio frames in the third-level cache are divided into M frames and N frames. The M frames are encoded locally by the embedded device, and the N frames are transmitted to the back-end device.

[0012] After receiving and processing N frames of audio signals, the backend device returns the processing results to the embedded device;

[0013] The embedded device combines the locally encoded M frames of audio signals with the back-end processed N frames of audio signals in a time sequence and outputs complete audio signal data.

[0014] As a preferred solution of the embedded firmware audio data modification method of the present invention, the real-time audio signal is obtained, divided into audio frames of fixed length, and stored in the first-level cache, including the following steps:

[0015] Acquire analog audio signals in the external environment through a microphone array;

[0016] Using an analog-to-digital converter, convert the acquired analog audio signal into a digital audio signal;

[0017] Divide the digital audio signal into audio frames of fixed length A to form a frame sequence;

[0018] A ring buffer structure is used to write newly acquired audio frames in the frame sequence into the buffer through a write pointer, and the audio frames written into the buffer are read through a read pointer;

[0019] Assign a frame number to each audio frame read and store it in the first-level cache.

[0020] As a preferred solution of the embedded firmware audio data modification method of the present invention, wherein: performing noise detection on the audio frames in the first-level cache, assigning priorities based on the noise detection results, and storing the audio frames after the priority assignment in the second-level cache includes the following steps:

[0021] Extract each frame of audio signal from the first-level cache according to the assigned frame sequence number;

[0022] For each frame of audio signal, fast Fourier transform is used to perform frequency domain conversion to extract the frequency features of each frame;

[0023] Calculate signal power and noise power through the extracted frequency features to distinguish effective signals from noise;

[0024] Calculate the signal-to-noise ratio of each frame based on the signal power and the noise power;

[0025] According to the signal-to-noise ratio of each frame, each frame of audio signal is divided into high priority and low priority;

[0026] Each frame of the audio signal after priority division is stored in a secondary cache.

[0027] As a preferred solution of the embedded firmware audio data modification method of the present invention, wherein: performing correlation analysis on adjacent audio frames in the secondary cache and performing encoding processing, and storing the encoded audio frames in the tertiary cache includes the following steps:

[0028] Extract two adjacent audio frames from the secondary cache in time series order;

[0029] The mean square error is used to calculate the similarity between two adjacent audio frames and determine the correlation between adjacent frames;

[0030] For frame pairs with low similarity, only full encoding is performed, and for frame pairs with high similarity, full encoding and differential encoding are performed respectively;

[0031] The fully encoded and differentially encoded frames are stored in the L3 cache.

[0032] As a preferred solution of the embedded firmware audio data modification method of the present invention, the audio frames in the three-level cache are divided into M frames and N frames according to the computing power and bandwidth status of the embedded device, the M frames are locally encoded by the embedded device, and the N frames are transmitted to the back-end device for processing, including the following steps:

[0033] Extracting a continuous sequence of audio frames from the L3 cache;

[0034] Based on the current computational load and available bandwidth, the continuous sequence of audio frames is divided into M frames and N frames;

[0035] Adopt AAC encoding method and distribute M frame signals to local for encoding processing;

[0036] The encoded M frame signals are stored in a high priority area of ​​the three-level cache;

[0037] Adaptive differential pulse code modulation is used to encode the differential signals of adjacent sampling points of N frames of data;

[0038] quantizing the differential signal through a quantizer, and packing the quantized differential signal into a bit stream to generate an ADPCM compressed audio signal, which is stored in a low priority area;

[0039] The compressed audio stored in the low priority area is transmitted to the backend device through the embedded device.

[0040] As a preferred solution of the embedded firmware audio data modification method of the present invention, wherein: after the back-end device receives and processes N frames of audio signals, returning the processing results to the embedded device includes the following steps:

[0041] The back-end device receives the compressed audio signal through the network interface, and restores the actual value of each sampling point through ADPCM decoding, and restores the quantized differential value to the value of the original differential signal;

[0042] Reconstructing the original audio signal sequence of each audio frame by accumulating the differential signals;

[0043] Using a digital filter, frequency equalization is performed on the adjusted and reconstructed audio signal sequence;

[0044] Performing noise reduction and dynamic range control processing on the audio signal after frequency equalization processing;

[0045] The processed audio signal is divided into frames of fixed length, and each frame of audio signal data is subjected to modified discrete cosine transformation and quantization;

[0046] The quantized frequency domain data is further compressed through entropy coding to generate the final high-quality N-frame audio signal in AAC encoding format and return it to the embedded device.

[0047] As a preferred solution of the embedded firmware audio data modification method of the present invention, wherein: the embedded device combines the locally encoded M frames with the back-end processed N frames in time sequence, and outputs the complete audio signal data, including the following steps:

[0048] Extract the M-frame audio signal that has completed AAC encoding from the high priority area of ​​the third-level cache;

[0049] The embedded device receives N frames of high-quality audio signals from the backend device;

[0050] The embedded device scans the timestamp of each audio frame of the M-frame and N-frame audio signals to check the continuity of the timestamp;

[0051] For short-term frame loss, interpolation frame filling is performed; for long-term frame loss, frame skipping is performed;

[0052] Rearrange the M-frame and N-frame audio signals after time synchronization processing in time sequence;

[0053] Seamlessly splice the sorted audio signals to generate complete audio signal data.

[0054] In a second aspect, the present invention provides an embedded firmware audio data modification device, comprising:

[0055] The frame segmentation module obtains the real-time audio signal, segments it into audio frames of fixed length, and stores them in the first-level cache;

[0056] A noise detection module performs noise detection on the audio frames in the first-level cache, assigns priorities based on the noise detection results, and stores the audio frames with assigned priorities in the second-level cache;

[0057] A correlation encoding module performs correlation analysis on adjacent audio frames in the secondary cache and performs encoding processing, and stores the encoded audio frames in the third-level cache;

[0058] The frame allocation module divides the audio frames in the third-level cache into M frames and N frames according to the computing power and bandwidth status of the embedded device. The M frames are locally encoded by the embedded device, and the N frames are transmitted to the back-end device.

[0059] Backend processing module: after receiving and processing N frames of audio signals, the backend device returns the processing results to the embedded device;

[0060] The frame merging module combines the M frames of locally encoded audio signals with the N frames of audio signals processed by the backend in a time sequence and outputs complete audio signal data.

[0061] In a third aspect, the present invention provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: when the computer program is executed by the processor, any step of the embedded firmware audio data modification method as described in the first aspect of the present invention is implemented.

[0062] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, any step of the embedded firmware audio data modification method as described in the first aspect of the present invention is implemented.

[0063] The beneficial effects of the present invention are as follows: by performing noise detection and priority allocation on audio frames in the first-level cache, it is possible to ensure that high-quality audio signals are processed first, thereby improving transmission efficiency; the correlation analysis and differential coding processing of adjacent audio frames in the second-level cache further reduce the transmission of redundant data, and ensure efficient audio transmission under bandwidth limitations; by handing over M frames and N frames to embedded devices and back-end devices for processing respectively, a reasonable distribution of computing load is achieved, ensuring that the embedded device can still effectively process audio signals under high load conditions; through timestamp synchronization, interpolation frame filling and frame skipping processing, the problems of audio frame loss and timing discontinuity are solved, the integrity and continuity of audio signals are ensured, thereby significantly improving the efficiency and quality of audio processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0065] Figure 1 This is a flowchart of the embedded firmware audio data modification method in Example 1.

[0066] Figure 2 Schematic diagram of the embedded firmware audio data modification device in Example 1. DETAILED DESCRIPTION

[0067] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings.

[0068] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0069] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0070] Example 1, reference Figure 1 and Figure 2 , which is the first embodiment of the present invention, provides an embedded firmware audio data modification method, comprising the following steps:

[0071] S1. Obtain real-time audio signals, divide them into audio frames of fixed length, and store them in a first-level cache.

[0072] The analog audio signal in the external environment is acquired through a microphone array; the analog audio signal is converted into a digital audio signal using an analog-to-digital converter; the digital audio signal is divided into audio frames of a fixed length A to form a frame sequence; a circular buffer structure is adopted to write the newly collected audio frames in the frame sequence into the cache through a write pointer, and the audio frames written into the cache are read through a read pointer; a frame number is assigned to each audio frame read and stored in a first-level cache.

[0073] Specifically, the write pointer in the cache is responsible for writing the newly acquired frames into the cache. If the write pointer reaches the end of the cache, it will re-write from the beginning of the cache in a loop, overwriting the earliest data; the read pointer in the cache is responsible for sending the audio frames in the cache to the next processing stage. A certain distance Δ is maintained between the read pointer and the write pointer to prevent data from overwriting unprocessed frames. The minimum value of this distance should meet the real-time processing requirements, that is, , where Δ represents the minimum distance between the read and write pointers, d represents the processing delay coefficient of the embedded device, which indicates the number of frames that the write pointer needs to lead the read pointer in the cache. A larger d value means that the device needs more time to process each frame of data, so a larger distance needs to be maintained between the read and write pointers. For example, if d=2, the read pointer needs to lag behind the write pointer by at least two frame lengths to avoid data overwriting unprocessed data, and A represents the number of sampling points contained in each frame, that is, the fixed length of each frame of audio.

[0074] S2. Perform noise detection on the audio frames in the first-level cache, assign priorities based on the noise detection results, and store the audio frames after the priority assignment in the second-level cache.

[0075] S2.1. Extract each frame of audio signal from the first-level cache according to the assigned frame number; for each frame of audio signal, perform frequency domain conversion using fast Fourier transform to extract the frequency characteristics of each frame.

[0076] The expression of fast Fourier transform is:

[0077] ;

[0078] in, Indicates Frame The value of the frequency component, n represents the sampling point index, Represents the frame index, Indicates The value of the nth sampling point in the frame, k represents the index of the frequency component, and j represents the imaginary unit.

[0079] S2.2. Calculate the signal power and noise power through the extracted frequency features to distinguish the valid signal and noise in the frame.

[0080] The expression of signal power is:

[0081] ;

[0082] Where P represents the power of the effective signal in the audio frame, Indicates the frequency threshold;

[0083] when When the frequency corresponding to the frequency component is lower than the frequency threshold , so it is considered a valid signal; when , the frequency corresponding to the frequency component is higher than the frequency threshold and is therefore considered to be noise.

[0084] The expression for noise power is:

[0085] ;

[0086] Where B represents the noise power.

[0087] S2.3. Calculate the signal-to-noise ratio of each frame based on the signal power and the noise power; divide each frame of audio signal into high priority and low priority according to the signal-to-noise ratio of each frame; and store each frame of audio signal after priority division in a secondary cache.

[0088] The expression of signal-to-noise ratio is:

[0089] ;

[0090] in, Indicates The signal-to-noise ratio of the frame.

[0091] Set the signal-to-noise ratio threshold. When the signal-to-noise ratio is greater than the signal-to-noise ratio threshold, the frame is marked as high priority, indicating that it has less noise and higher audio signal quality; when the signal-to-noise ratio is less than the signal-to-noise ratio threshold, the frame is marked as low priority, indicating that there is more noise in the frame and the sound quality is poor.

[0092] S3. Perform correlation analysis on adjacent audio frames in the secondary cache and perform encoding processing, and store the encoded audio frames in the tertiary cache.

[0093] S3.1. Extract two adjacent audio frames from the secondary cache in time series order; and calculate the similarity between the two adjacent audio frames using mean square error.

[0094] The expression of mean square error is:

[0095] ;

[0096] in, Indicates adjacent frames and The mean square error of two adjacent frames is used to measure and The similarity between The smaller the value, the more similar the waveforms of the two frames are, which is used to judge the difference between frames. Indicates Frame audio signal, Indicates Frame audio signal, Indicates The value of the nth sample point in the frame.

[0097] In addition to the mean square error calculation based on the time domain, embedded devices can also choose to further analyze the correlation between adjacent frames through spectral similarity. Spectral similarity determines the similarity of adjacent frames by comparing their frequency domain distribution. The commonly used similarity measurement method is cosine similarity. The formula for spectral similarity is as follows:

[0098] ;

[0099] in, Indicates adjacent frames and The spectral cosine similarity of .

[0100] S3.2. Based on the similarity calculation results, the correlation between adjacent frames is determined; only full encoding is performed for frame pairs with low similarity, and full encoding and differential encoding are performed for frame pairs with high similarity; and frames that have been fully encoded and differentially encoded are stored in the third-level cache.

[0101] Setting similarity threshold ,when When , it means that the adjacent frames have a high temporal correlation. At this time, the device only frame Complete encoding is performed, and frame Differential encoding is performed, that is, only the difference data between frames is stored; when , it means that the correlation between adjacent frames is low and the two frames need to be fully encoded separately.

[0102] S4. According to the computing power and bandwidth status of the embedded device, the audio frames in the third-level cache are divided into M frames and N frames. The M frames are locally encoded by the embedded device, and the N frames are transmitted to the back-end device for processing.

[0103] Extract a continuous sequence of audio frames from the three-level cache; divide the continuous sequence of audio frames into M frames and N frames based on the current computing load and available bandwidth; use AAC encoding (advanced audio coding) to allocate M frame signals to local encoding processing; store the encoded M frame signals in the high-priority area of ​​the three-level cache; use adaptive differential pulse code modulation to encode the differential signals of adjacent sampling points for N frame data; quantize the differential signals through a quantizer, and pack the quantized differential signals into a bit stream to generate an ADPCM (adaptive differential pulse code modulation) compressed audio signal, which is stored in a low-priority area; transmit the compressed audio stored in the low-priority area to the back-end device through the embedded device.

[0104] Specifically, M frames refer to the number of audio frames processed locally on the embedded device. These frames are encoded with high-quality AAC and processed locally on the device. After processing, these frames are stored in the high-priority area of ​​the three-level cache to ensure that high-priority audio frames processed locally can be quickly output or further processed when needed.

[0105] N frames refer to the number of audio frames that are compressed by the embedded device and transmitted to the back-end device. These frames will be compressed by simple ADPCM encoding to reduce the transmission bandwidth occupation and transmitted to the back-end device by wireless or wired means. The back-end device usually has stronger computing resources, so it can perform more complex processing on these N frames of data.

[0106] S5. After receiving and processing N frames of audio signals, the backend device returns the processing results to the embedded device.

[0107] The back-end device receives the compressed audio signal through the network interface, and restores the actual value of each sampling point through ADPCM decoding, and restores the quantized differential value to the value of the original differential signal; reconstructs the original audio signal sequence of each audio frame by accumulating the differential signal; uses a digital filter to perform frequency equalization on the adjusted and reconstructed audio signal sequence; performs noise reduction and dynamic range control on the audio signal after frequency equalization; divides the processed audio signal into frames of fixed length, and performs modified discrete cosine transform and quantization on each frame of audio signal data; the quantized frequency domain data is further compressed through entropy coding to generate the final high-quality N-frame audio signal in the AAC encoding format and return it to the embedded device.

[0108] It should be noted that in the prior art, ADPCM (Adaptive Differential Pulse Code Modulation) decoding is usually used in simple audio compression and decompression applications, mainly in low-bandwidth transmission scenarios; this solution not only decodes ADPCM data, but also uses digital filters to perform frequency equalization on the reconstructed audio signal after decoding, which is a significant expansion on the basis of existing ADPCM decoding. Frequency equalization optimizes the overall auditory experience of audio by adjusting the gain of different frequency bands, which can effectively improve the sound quality, especially in bandwidth-limited scenarios.

[0109] In the prior art, the audio processing of embedded devices and back-end devices is usually separated. The embedded device only performs simple audio acquisition and compression, while the back-end device is responsible for complex audio processing, and there is a lack of collaborative solutions. This solution proposes an architecture for collaborative processing between embedded devices and back-end devices, which divides the audio data into two parts for processing: M frames are processed locally on the embedded device, and N frames are transmitted to the back-end device for more complex processing. After ADPCM decoding, frequency equalization, noise reduction, dynamic range control and other processing, the back-end device returns high-quality audio data to the embedded device for merging. Through division of labor and collaboration, the embedded device can process a part of the audio frames, which reduces the burden on the back-end device and reduces the network bandwidth usage; the powerful computing power of the back-end device can perform more complex audio processing on N frames to ensure the improvement of overall sound quality.

[0110] S6. The embedded device combines the locally encoded M frames with the back-end processed N frames in time sequence and outputs complete audio signal data.

[0111] Extract M frames of audio signals that have completed AAC encoding from the high priority area of ​​the three-level cache; the embedded device receives N frames of high-quality audio signals from the back-end device; the embedded device scans the timestamp of each audio frame of the M and N audio frames to check the continuity of the timestamp; for short-term frame loss, perform interpolation frame filling processing, and for long-term frame loss, perform frame skipping processing; rearrange the M and N audio frames that have undergone time synchronization processing in chronological order; seamlessly splice the sorted audio signals to generate complete audio signal data.

[0112] It should be noted that in the prior art, the merging of audio data usually relies on a simple timestamp alignment mechanism, or is spliced ​​based on a fixed buffer. However, timestamp alignment usually does not have a mechanism to handle frame loss, which can easily lead to discontinuity or misalignment during audio playback. This solution uses timestamp scanning and time synchronization mechanisms before the embedded device merges M frames and N frames, and performs interpolation and frame skipping for different frame loss phenomena, ensuring smooth transition during short-term frame loss and seamless skipping during long-term frame loss, avoiding the disadvantages of audio interruption or discontinuity caused by frame loss in traditional solutions.

[0113] This embodiment also provides an embedded firmware audio data modification device, including:

[0114] The frame segmentation module obtains the real-time audio signal, divides it into audio frames of fixed length, and stores it in the first-level cache; the noise detection module performs noise detection on the audio frames in the first-level cache, assigns priorities based on the noise detection results, and stores the audio frames with assigned priorities in the second-level cache; the correlation encoding module performs correlation analysis and encoding processing on adjacent audio frames in the second-level cache, and stores the encoded audio frames in the third-level cache; the frame allocation module divides the audio frames in the third-level cache into M frames and N frames according to the computing power and bandwidth conditions of the embedded device, the M frames are locally encoded by the embedded device, and the N frames are transmitted to the back-end device; the back-end processing module, after the back-end device receives and processes the N frames of audio signals, returns the processing results to the embedded device; the frame merging module, the embedded device merges the locally encoded M frames of audio signals with the N frames of audio signals processed by the back-end in time sequence, and outputs complete audio signal data.

[0115] This embodiment also provides a computer device, which is suitable for the embedded firmware audio data modification method, including: a memory and a processor; the memory is used to store computer executable instructions, and the processor is used to execute computer executable instructions to implement the embedded firmware audio data modification method proposed in the above embodiment.

[0116] The computer device may be a terminal, and the computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. The display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device may be a touch layer covered on the display screen, or a key, trackball or touchpad provided on the housing of the computer device, or an external keyboard, touchpad or mouse, etc.

[0117] This embodiment also provides a storage medium on which a computer program is stored. When the program is executed by a processor, the embedded firmware audio data modification method proposed in the above embodiment is implemented; the storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (Static Random Access Memory, referred to as SRAM), electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, referred to as EEPROM), erasable programmable read-only memory (Erasable Programmable Read Only Memory, referred to as EPROM), programmable read-only memory (Programmable Red-Only Memory, referred to as PROM), read-only memory (Read-Only Memory, referred to as ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0118] In summary, the present invention can ensure that high-quality audio signals are processed preferentially by performing noise detection and priority assignment on audio frames in the first-level cache, thereby improving transmission efficiency; the correlation analysis and differential coding processing of adjacent audio frames in the second-level cache further reduce the transmission of redundant data and ensure efficient audio transmission under bandwidth limitations; by handing over M frames and N frames to embedded devices and back-end devices for processing respectively, a reasonable distribution of computing load is achieved, ensuring that the embedded device can still effectively process audio signals under high load conditions; through timestamp synchronization, interpolation frame filling and frame skipping processing, the problems of audio frame loss and timing discontinuity are solved, the integrity and continuity of audio signals are ensured, thereby significantly improving the efficiency and quality of audio processing.

[0119] Example 2, referring to Table 1, is the second example of the present invention. To further verify the technical solution of the present invention, experimental simulation data of the embedded firmware audio data modification method are provided.

[0120] Two audio processing methods are selected for comparison: one is the existing traditional audio processing method, and the other is the embedded audio data processing method based on the present invention. The goal of the experiment is to test the processing efficiency, audio quality, and frame loss recovery capabilities of audio signals under different bandwidth and computing resource constraints. The audio signal used in the experiment is a natural sound with high noise in a simulated environment, which is collected using a microphone array.

[0121] The details are shown in Table 1 below:

[0122] Table 1 Experimental data record table

[0123]

[0124] The comparative advantages of the present invention over the prior art in multiple dimensions can be clearly seen from the experimental data table. First, in terms of audio signal delay, the present invention has a delay of only 300ms under low bandwidth (64kbps) conditions, while the delay of the prior art is as high as 500ms, indicating that the present invention can process and transmit audio signals more quickly. This is because the present invention can effectively reduce the processing time of low-priority frames through hierarchical caching and priority allocation mechanisms, thereby improving the real-time performance of the system.

[0125] Secondly, in terms of frame loss rate, the frame loss rate of the present invention under low, medium and high bandwidth conditions is 0.5%, 0.2% and 0.1% respectively, which is significantly lower than 5.0%, 3.0% and 2.0% of the prior art. This is due to the interpolation frame filling and frame skipping processing mechanism of the present invention, which can effectively repair lost frames under network jitter or bandwidth limitation conditions to ensure the continuity of audio signals.

[0126] In terms of signal-to-noise ratio (SNR), the SNR of the present invention under different bandwidth conditions is higher than that of the prior art. For example, under medium bandwidth conditions, the SNR of the present invention reaches 35dB, while the SNR of the prior art is only 25dB. This shows that the present invention can more effectively distinguish between noise and valid signals during noise detection and priority allocation, thereby improving the quality of audio signals.

[0127] In terms of coding efficiency, the coding efficiency of the present invention under low bandwidth conditions is 120 frames / second, while the prior art is only 80 frames / second; under high bandwidth conditions, the coding efficiency of the present invention reaches 220 frames / second, which is much higher than the 150 frames / second of the prior art. This result shows that the present invention reduces the amount of redundant data processing through correlation analysis and differential coding of adjacent audio frames, thereby improving coding efficiency.

[0128] Under low bandwidth conditions, the bandwidth utilization rate of the present invention reaches 95%, while that of the prior art is only 70%, indicating that the present invention is more efficient in utilizing bandwidth and can transmit more effective audio data under limited bandwidth conditions. At the same time, the sound quality score of the present invention under all bandwidth conditions is higher than that of the prior art. Under high bandwidth conditions, the sound quality score reaches 10 points, while that of the prior art is 8 points. This shows that the present invention can reduce the degradation of sound quality caused by bandwidth limitations while ensuring high-quality audio transmission.

[0129] In summary, the present invention significantly improves the processing efficiency and transmission quality of audio signals through technologies such as multi-level cache management, noise detection, priority allocation and dynamic encoding processing, especially under conditions of limited bandwidth and limited computing power, and shows better performance than the existing technology.

[0130] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. An embedded firmware audio data modification method, characterized in that: include, Get the real-time audio signal, split it into fixed-length audio frames, and store them in the first-level cache; Perform noise detection on the audio frames in the first-level cache, assign priorities based on the noise detection results, and store the audio frames with assigned priorities in the second-level cache; Perform correlation analysis on adjacent audio frames in the secondary cache and perform encoding processing, and store the encoded audio frames in the tertiary cache; According to the computing power and bandwidth of the embedded device, the audio frames in the third-level cache are divided into M frames and N frames. The M frames are encoded locally by the embedded device, and the N frames are transmitted to the back-end device. After receiving and processing N frames of audio signals, the backend device returns the processing results to the embedded device; The embedded device combines the locally encoded M frames of audio signals with the back-end processed N frames of audio signals in a time sequence and outputs complete audio signal data.

2. The embedded firmware audio data modification method according to claim 1, characterized in that: Acquiring real-time audio signals, dividing them into audio frames of fixed length, and storing them in the first-level cache includes the following steps: Acquire analog audio signals in the external environment through a microphone array; Using an analog-to-digital converter, convert the acquired analog audio signal into a digital audio signal; Divide the digital audio signal into audio frames of fixed length A to form a frame sequence; A ring buffer structure is used to write newly acquired audio frames in the frame sequence into the buffer through a write pointer, and the audio frames written into the buffer are read through a read pointer; Assign a frame number to each audio frame read and store it in the first-level cache.

3. The embedded firmware audio data modification method as claimed in claim 2, characterized in that: Performing noise detection on the audio frames in the first-level cache, assigning priorities based on the noise detection results, and storing the audio frames after the priority assignment in the second-level cache includes the following steps: Extract each frame of audio signal from the first-level cache according to the assigned frame sequence number; For each frame of audio signal, fast Fourier transform is used to perform frequency domain conversion to extract the frequency features of each frame; Calculate signal power and noise power through the extracted frequency features to distinguish effective signals from noise; Calculate the signal-to-noise ratio of each frame based on the signal power and the noise power; According to the signal-to-noise ratio of each frame, each frame of audio signal is divided into high priority and low priority; Each frame of the audio signal after priority division is stored in a secondary cache.

4. The embedded firmware audio data modification method as claimed in claim 3, characterized in that: Perform correlation analysis on adjacent audio frames in the L2 cache and encode them, and store the encoded audio frames in the L3 cache The following steps are included: Extract two adjacent audio frames from the secondary cache in time series order; The mean square error is used to calculate the similarity between two adjacent audio frames and determine the correlation between adjacent frames; For frame pairs with low similarity, only full encoding is performed, and for frame pairs with high similarity, full encoding and differential encoding are performed respectively; Store fully encoded and differentially encoded frames in the L3 cache.

5. The embedded firmware audio data modification method as claimed in claim 4, characterized in that: According to the computing power and bandwidth of the embedded device, the audio frames in the three-level cache are divided into M frames and N frames. The M frames are locally encoded by the embedded device, and the N frames are transmitted to the back-end device for processing. The following steps are included: Extracting a continuous sequence of audio frames from the L3 cache; Based on the current computational load and available bandwidth, the continuous sequence of audio frames is divided into M frames and N frames; Adopt AAC encoding method and distribute M frame signals to local for encoding processing; The encoded M frame signals are stored in a high priority area of ​​the three-level cache; Adaptive differential pulse code modulation is used to encode the differential signals of adjacent sampling points of N frames of data; quantizing the differential signal through a quantizer, and packing the quantized differential signal into a bit stream to generate an ADPCM compressed audio signal, which is stored in a low priority area; The compressed audio stored in the low priority area is transmitted to the backend device through the embedded device.

6. The embedded firmware audio data modification method according to claim 5, characterized in that: After the backend device receives and processes N frames of audio signals, returning the processing results to the embedded device includes the following steps: The back-end device receives the compressed audio signal through the network interface, and restores the actual value of each sampling point through ADPCM decoding, and restores the quantized differential value to the value of the original differential signal; Reconstructing the original audio signal sequence of each audio frame by accumulating the differential signals; Using a digital filter, frequency equalization is performed on the adjusted and reconstructed audio signal sequence; Performing noise reduction and dynamic range control processing on the audio signal after frequency equalization processing; The processed audio signal is divided into frames of fixed length, and each frame of audio signal data is subjected to modified discrete cosine transformation and quantization; The quantized frequency domain data is further compressed through entropy coding to generate the final high-quality N-frame audio signal in AAC encoding format and return it to the embedded device.

7. The embedded firmware audio data modification method according to claim 6, characterized in that: The embedded device combines the locally encoded M frames with the backend processed N frames in time sequence, and outputs the complete audio signal data including the following steps: Extract the M-frame audio signal that has completed AAC encoding from the high priority area of ​​the third-level cache; The embedded device receives N frames of high-quality audio signals from the backend device; The embedded device scans the timestamp of each audio frame of the M-frame and N-frame audio signals to check the continuity of the timestamp; For short-term frame loss, interpolation frame filling is performed; for long-term frame loss, frame skipping is performed; Rearrange the M-frame and N-frame audio signals after time synchronization processing in time sequence; Seamlessly splice the sorted audio signals to generate complete audio signal data.

8. An embedded firmware audio data modification device, based on the embedded firmware audio data modification method according to any one of claims 1 to 7, characterized in that: include, The frame segmentation module obtains the real-time audio signal, segments it into audio frames of fixed length, and stores them in the first-level cache; A noise detection module performs noise detection on the audio frames in the first-level cache, assigns priorities based on the noise detection results, and stores the audio frames with assigned priorities in the second-level cache; A correlation encoding module performs correlation analysis on adjacent audio frames in the secondary cache and performs encoding processing, and stores the encoded audio frames in the third-level cache; The frame allocation module divides the audio frames in the third-level cache into M frames and N frames according to the computing power and bandwidth status of the embedded device. The M frames are locally encoded by the embedded device, and the N frames are transmitted to the back-end device. Backend processing module: after receiving and processing N frames of audio signals, the backend device returns the processing results to the embedded device; The frame merging module combines the M frames of locally encoded audio signals with the N frames of audio signals processed by the backend in a time sequence and outputs complete audio signal data.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the embedded firmware audio data modification method described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the embedded firmware audio data modification method described in any one of claims 1 to 7 are implemented.

Images