Audio data enhancement method, device and medium based on sound pickup environment factors

Abstract

The present invention discloses an audio data enhancement method, device, and medium based on the collection of sound pickup environmental factors. The method comprises obtaining a sample training set of the original audio data to be enhanced; sequentially batching, verifying, labeling, and merging the sample training set; scheduling a microphone and a speaker to mix the entire audio data of each sample batch in a real environment with the sound pickup environmental factors; sequentially segmenting and labeling the entire recording data based on the batching and labeling to obtain an enhanced sample training set for the current sample batch; obtaining the enhanced sample training set for each sample batch and splicing them together to obtain the final enhanced sample training set for the original audio data. Advantages: The present invention simultaneously takes into account environmental factors such as environmental background noise, the distance between the microphone and the sound source, and interference generated within the microphone, thereby more effectively introducing environmental information, thereby improving the accuracy of the sound event detection model in a real environment and reducing performance degradation.

Description

Technical Field

[0001] The present invention relates to a method, device and medium for enhancing audio data collected based on sound pickup environment factors, and belongs to the technical field of digital media. Background Art

[0002] Sound event detection is the task of identifying sound events occurring in a given audio dataset, enabling various AI-based systems to distinguish sounds and understand the acoustic environment. Sound event detection is widely used in real-world scenarios such as smart devices, smart security, and smart homes, and has become an active research topic in the field of acoustic signal processing.

[0003] State-of-the-art audio classification algorithms employ several data augmentation methods. These methods do not generate additional training data but instead directly modify input features. Sample mixing (mixup) superimposes two audio samples in a specific ratio to produce a mixed sample. Spectrogram masking masks a continuous range of frames or frequency bands with a specified value. Band augmentation involves strategically selecting frequency bands and applying amplification or subtraction to the signals in those bands.

[0004] However, in real-world environments, the various differences in sound pickup environmental factors can lead to reduced classification model performance. Traditional audio data augmentation techniques focus on transforming audio signals or features, but lack attention to target environmental factors. Furthermore, data augmentation methods that do focus on target environmental factors often fail to fully incorporate these factors due to efficiency or cost constraints, resulting in limited effectiveness. Consequently, existing methods are poorly adaptable to real-world application environments.

[0005] The original method combines clean audio samples with real-world background noise samples in a certain ratio to create mixed samples, and then assigns the same label to the mixed samples as the original ones. However, this simple background noise mixing method only focuses on the impact of background noise. In reality, due to the variability of sound source distances and internal interference within the pickup device, the sound collected by the pickup device is not always clear and differs from the distribution of the clean training set. Furthermore, this simple method also fails to adequately mix the background noise, causing these noise signals to resemble random perturbations, which does not significantly improve the final model performance. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to overcome the defects of the prior art and provide an audio data enhancement method, device and medium based on the collection of sound pickup environment factors.

[0007] In order to solve the above technical problems, the present invention is implemented by adopting the following technical solutions.

[0008] In one aspect, the present invention discloses a method for enhancing audio data based on sound pickup environment factors, comprising:

[0009] Obtain a sample training set of original audio data to be enhanced;

[0010] The sample training set is sequentially batched, verified, label extracted, and merged to obtain several batches of complete audio data;

[0011] The microphone and speaker are scheduled to mix the audio data of each sample batch in a real environment to collect environmental factors and obtain the entire recording data of the current sample batch;

[0012] The entire recording data is sequentially segmented and labeled according to the batches and the corresponding data extracted by the labels to obtain an enhanced sample training set of the current sample batch;

[0013] Obtain the enhanced sample training set of each sample batch and concatenate them to obtain the final enhanced sample training set of the original audio data.

[0014] Furthermore, the batching includes:

[0015] Pre-set the number of samples batchsize for each batch of samples to be processed, read the data index idx, and divide the samples with data index [idx, idx + batchsize - 1] into one batch; if the current batch is the first batch, the data index idx is the initial index of the sample in the sample training set; if the current batch is not the first batch, the data index idx is the next data index after the last data index of the previous batch.

[0016] Furthermore, the verification includes:

[0017] Check whether each sample in the current batch is of the preset time length. If not, discard the samples that are not of the preset time length and record them in the log. Recursively select the next sample that meets the requirements and add it to the batch.

[0018] Furthermore, the tag extraction includes:

[0019] The labels of the current batch of samples are obtained from the label file of the sample training set of the original audio data.

[0020] Furthermore, the merging includes:

[0021] Concatenate all samples in the current batch into a complete audio data in data index order.

[0022] Furthermore, the scheduling of the microphone and the speaker to mix the audio data of each sample batch in a real environment to collect environmental factors to obtain the entire recording data of the current sample batch, including:

[0023] Call the speaker to play the entire audio data of the current sample batch, and call the pickup to record the audio data played by the speaker;

[0024] Get the recording time of the pickup when the speaker starts playing the entire audio data of the current sample batch - t1 mic , and the recording time t2 of the microphone when the speaker finishes playing the entire audio data of the current sample batch mic ;

[0025] Get the pickup record [t1 mic , t2 mic ] range is the entire recording data of the current sample batch.

[0026] Furthermore, the segmentation includes:

[0027] Determine the start time of the first sample of the entire recording data of the current sample batch, and split it into batchsize new samples according to the preset time length.

[0028] Furthermore, the marking includes:

[0029] Using the labels of the current batch of samples extracted during the label extraction process, each sample in this batch is sequentially labeled to form the final enhanced sample-label pair. Based on the final enhanced sample-label pair, the enhanced sample training set of the current sample batch is obtained.

[0030] In a second aspect, the present invention discloses an audio data enhancement device based on sound pickup environment factors, comprising:

[0031] An acquisition module, used to obtain a sample training set of original audio data to be enhanced;

[0032] A preprocessing module is used to sequentially perform batching, verification, label extraction, and merging operations on the sample training set to obtain several batches of complete audio data;

[0033] The acquisition module is used to schedule the microphone and speaker to mix the audio data of each sample batch in the real environment and collect the environmental factors to obtain the entire recording data of the current sample batch;

[0034] The data post-processing module is used to segment and label the entire recording data in sequence according to the corresponding data extracted by the batches and the labels to obtain the enhanced sample training set of the current sample batch; obtain the enhanced sample training set of each sample batch, and splice them to obtain the final enhanced sample training set of the original audio data.

[0035] The third invention discloses a computer-readable storage medium storing one or more programs, characterized in that the one or more programs include instructions, which, when executed by a computing device, enable the computing device to perform the method.

[0036] The beneficial effects achieved by the present invention are:

[0037] The method proposed in the present invention uses a collection method to add real-world sound pickup environmental factors to the original training data while retaining the original labels. This method simultaneously takes into account environmental factors such as environmental background noise, the distance between the microphone and the sound source, and the interference generated inside the microphone. It can more effectively introduce environmental information, thereby improving the accuracy of the sound event detection model in a real environment and reducing performance degradation. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is a schematic diagram of an audio data enhancement method based on sound pickup environment factors;

[0039] Figure 2 It is a data preprocessing flow chart;

[0040] Figure 3 It is a flow chart for collecting sound pickup environment factors;

[0041] Figure 4 It is a schematic diagram of time synchronization;

[0042] Figure 5 This is the data post-processing flow chart. DETAILED DESCRIPTION

[0043] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention and are not intended to limit the scope of protection of the present invention.

[0044] Example 1: This example introduces a method for enhancing audio data based on sound pickup environment factors, including:

[0045] Obtain a sample training set of original audio data to be enhanced;

[0046] The sample training set is sequentially batched, verified, label extracted, and merged to obtain several batches of complete audio data;

[0047] The microphone and speaker are scheduled to mix the audio data of each sample batch in a real environment to collect environmental factors and obtain the entire recording data of the current sample batch;

[0048] The entire recording data is sequentially segmented and labeled according to the batches and the corresponding data extracted by the labels to obtain an enhanced sample training set of the current sample batch;

[0049] Obtain the enhanced sample training set of each sample batch and concatenate them to obtain the final enhanced sample training set of the original audio data.

[0050] The batches include:

[0051] Pre-set the number of samples batchsize for each batch of samples to be processed, read the data index idx, and divide the samples with data index [idx, idx + batchsize - 1] into one batch; if the current batch is the first batch, the data index idx is the initial index of the sample in the sample training set; if the current batch is not the first batch, the data index idx is the next data index after the last data index of the previous batch.

[0052] The verification includes:

[0053] Check whether each sample in the current batch is of the preset time length. If not, discard the samples that are not of the preset time length and record them in the log. Recursively select the next sample that meets the requirements and add it to the batch.

[0054] The tag extraction includes:

[0055] The labels of the current batch of samples are obtained from the label file of the sample training set of the original audio data.

[0056] The merger includes:

[0057] Concatenate all samples in the current batch into a complete audio data in data index order.

[0058] The scheduling of the microphone and the speaker to mix the audio data of each sample batch in a real environment and pick up the environmental factors to obtain the entire recording data of the current sample batch, including:

[0059] Call the speaker to play the entire audio data of the current sample batch, and call the pickup to record the audio data played by the speaker;

[0060] Get the recording time of the pickup when the speaker starts playing the entire audio data of the current sample batch - t1 mic , and the recording time t2 of the microphone when the speaker finishes playing the entire audio data of the current sample batch mic;

[0061] Get the pickup record [t1 mic , t2 mic ] range is the entire recording data of the current sample batch.

[0062] The segmentation includes:

[0063] Determine the start time of the first sample of the entire recording data of the current sample batch, and split it into batchsize new samples according to the preset time length.

[0064] The marking includes:

[0065] Using the labels of the current batch of samples extracted during the label extraction process, each sample in this batch is sequentially labeled to form the final enhanced sample-label pair. Based on the final enhanced sample-label pair, the enhanced sample training set of the current sample batch is obtained.

[0066] Example 2, based on the same inventive concept as Example 1, introduces a method for enhancing audio data based on sound pickup environment factors, including:

[0067] Unlike traditional methods, the method proposed in the present invention uses a collection method to add the real-world sound pickup environmental factors to the original training data, and retains the original labels. This method also takes into account environmental factors such as environmental background noise, the distance between the microphone and the sound source, and the interference generated inside the microphone. It can more effectively introduce environmental information than traditional methods such as background noise mixing, thereby improving the accuracy of the sound event detection model in the real environment and reducing performance degradation. In order to efficiently realize the collection of environmental factors, an automated enhanced audio annotation data collection system based on sound pickup environmental factors (hereinafter referred to as the system) is proposed. The schematic diagram of this method is shown in the figure. Figure 1 Using the original training set as input, we can efficiently obtain enhanced samples containing target environmental factors.

[0068] The following introduces the four operating steps of the system proposed by this method.

[0069] (1) Data preprocessing: In the data preprocessing step, the samples must be batched, verified, label extracted, merged, etc. The specific steps are as follows: Figure 2 As shown, specifically:

[0070] Batching: Preset the batch size of samples to be processed. Read the last processed data index idx and group the samples with the data index [idx, idx + batchsize - 1] into a batch.

[0071] Verification: Check whether each sample in the batch is of the standard length of 10 seconds. If not, discard it directly and record it in the log. Recursively select the next sample that meets the requirements and add it to the batch.

[0072] Label extraction: Obtain the labels of this batch of samples from the label file of the dataset and temporarily record them in memory.

[0073] Merge: All samples in this batch are spliced ​​into a complete audio data. This is to prevent the slight delay between the system playing each data from causing time offset errors in the enhanced samples.

[0074] (2) Collection of sound pickup environment factors: When performing data enhancement on each sample batch, the system needs to schedule the pickup and speaker to implement the mixing of environmental factors for the raw data in the real environment, such as Figure 3 As shown, the working steps are as follows:

[0075] 1. First time synchronization: Request time synchronization information from the time synchronization module to obtain the system machine time and the pickup machine time pair (t1 sys , t1 mic ).

[0076] 2. Play samples: At the same time as the first time synchronization, use the wired speakers to play a batch of samples.

[0077] 3. Second time synchronization: When a batch of samples is played, the second time synchronization information is requested from the time synchronization module (t2 sys , t2 mic ).

[0078] 4. Download enhanced samples: To ensure that the data is processed completely inside the surveillance camera, 10 seconds after a batch of data is played, the system is wirelessly connected to the microphone to obtain [t1 mic , t2 mic ] range of audio playback data, if there are two inconsistent time synchronization information, the previous batch of data will be discarded and resent (the inconsistent time synchronization information here refers to [t1 sys , t2 sys ] and [t1 mic , t2 mic These two time ranges have a time error of more than 0.2 seconds in real-world timescales. This can lead to fine-grained mismatches between the frame-level labels of the unprocessed and processed data samples. The downloaded data is then sent to the data post-processing submodule.

[0079] The time synchronization is to ensure the validity of the enhanced samples and prevent the enhanced samples and the original samples from having label differences on a fine-grained time scale due to the hardware time asynchrony between the system host and the pickup. Time synchronization is mainly achieved through the Network Time Protocol. The Network Time Protocol (NTP) is a protocol used for time synchronization between computers. It can reduce the time error between two computers to less than 1ms, providing accurate and robust time services. In order to achieve time synchronization between the system host computer and the pickup host, an NTP server is set as a recognized time provider, and the NTP client is started on both the system host computer and the pickup host and a socket connection is established with the NTP server. In this way, in the same local area network, the two clients will receive data packets with the same time information at the same time and maintain millisecond-level time synchronization with each other. The system's time synchronization module is responsible for processing timestamp information and verifying the time information of the pickup host during downloading. The working mode of the time synchronization module is as follows: Figure 4 shown.

[0080] (3) Data post-processing: The data post-processing step involves segmenting the samples, label binding, and other operations, such as Figure 5 The specific steps are as follows:

[0081] 1. Segmentation: After playing a batch of samples, obtain the entire recording of the batch of samples from the device control module (the length of this recording may be greater than 10×batchsize), determine the start time of the first sample in the batch, and segment it according to the fixed sample length (10s) to split it into batchsize new samples.

[0082] 2. Automatic labeling: Using the label information stored in the memory of the preprocessing submodule, each sample in this batch is labeled sequentially to achieve automatic labeling, forming the final enhanced sample-label pair.

[0083] (4) Evaluation: In the evaluation step, the system automatically determines whether there is a time offset in sample generation. The judgment method is based on calculating the similarity between the original sample x1 and the collected sample x2. Due to the high interference in the experimental environment, even if the original sample and the collected sample are completely aligned, the frequency domain characteristics between them may be very different, while the time domain characteristics are not much different. Therefore, the evaluation module uses the sound waveform comparison to determine the similarity between the original sample and the new sample. Since the two samples have the same length, the Euclidean distance d between samples x1 and x2 is directly calculated:

[0084] ;

[0085] In the experiment, the empirical value T is taken as the threshold. When d<= T, it is considered that there is no time domain offset between samples. Otherwise, an error prompt is given in the system.

[0086] This method operates in the sample augmentation phase of the sound event classification process. It uses three data augmentation methods, namely audio transformation (pitch and time shift) and background noise mixing, to train two mainstream sound event classification models, and compares the average classification accuracy of the trained models.

[0087] Table 1

[0088] ;

[0089] Table 1 shows the average precision of two mainstream models on a real-world test set when trained using three different data augmentation methods. This method significantly improves performance on audio classification models of various structures. Table 1 shows PANN (Large-scale pretrained audio neural networks for audio pattern recognition) and PSLA (Improving audiotagging with pretraining, sampling, labeling, and aggregation).

[0090] Example 3, based on the same inventive concept as the other examples, introduces an audio data enhancement device based on sound pickup environment factors, including:

[0091] An acquisition module, used to obtain a sample training set of original audio data to be enhanced;

[0092] A preprocessing module is used to sequentially perform batching, verification, label extraction, and merging operations on the sample training set to obtain several batches of complete audio data;

[0093] The acquisition module is used to schedule the microphone and speaker to mix the audio data of each sample batch in the real environment and collect the environmental factors to obtain the entire recording data of the current sample batch;

[0094] The data post-processing module is used to segment and label the entire recording data in sequence according to the corresponding data extracted by the batches and the labels to obtain the enhanced sample training set of the current sample batch; obtain the enhanced sample training set of each sample batch, and splice them to obtain the final enhanced sample training set of the original audio data.

[0095] Example 4 is based on the same inventive concept as other examples. This example introduces a computer-readable storage medium that stores one or more programs, characterized in that the one or more programs include instructions that, when executed by a computing device, enable the computing device to execute the method in Example 1.

[0096] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0097] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0098] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0099] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0100] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for enhancing audio data based on sound pickup environment factors, characterized in that: include: Obtain a sample training set of original audio data to be enhanced; The sample training set is sequentially batched, verified, label extracted, and merged to obtain several batches of complete audio data; The microphone and speaker are scheduled to mix the audio data of each sample batch in a real environment to collect environmental factors and obtain the entire recording data of the current sample batch; The entire recording data is sequentially segmented and labeled according to the batches and the corresponding data extracted by the labels to obtain an enhanced sample training set of the current sample batch; Obtain the enhanced sample training set of each sample batch and concatenate them to obtain the final enhanced sample training set of the original audio data.

2. The audio data enhancement method based on sound pickup environment factors according to claim 1, characterized in that: The batches include: Pre-set the number of samples batchsize for each batch of samples to be processed, read the data index idx, and divide the samples with data index [idx, idx + batchsize - 1] into one batch; if the current batch is the first batch, the data index idx is the initial index of the sample in the sample training set; if the current batch is not the first batch, the data index idx is the next data index after the last data index of the previous batch.

3. The audio data enhancement method based on sound pickup environment factors according to claim 2, characterized in that: The verification includes: Check whether each sample in the current batch is of the preset time length. If not, discard the samples that are not of the preset time length and record them in the log. Recursively select the next sample that meets the requirements and add it to the batch.

4. The audio data enhancement method based on sound pickup environment factors according to claim 3 is characterized in that: The tag extraction includes: Obtain labels for the current batch of samples from a label file of a sample training set of the original audio data.

5. The audio data enhancement method based on sound pickup environment factors according to claim 4 is characterized in that: The merger includes: Concatenate all samples in the current batch into a complete audio data in data index order.

6. The audio data enhancement method based on sound pickup environment factors according to claim 1, characterized in that: The scheduling of the microphone and the speaker to mix the audio data of each sample batch in a real environment and pick up the environmental factors to obtain the entire recording data of the current sample batch, including: Call the speaker to play the entire audio data of the current sample batch, and call the pickup to record the audio data played by the speaker; Get the recording time of the pickup when the speaker starts playing the entire audio data of the current sample batch - t1 mic , and the recording time t2 of the microphone when the speaker finishes playing the entire audio data of the current sample batch mic ; Get the pickup record [t1 mic , t2 mic ] range is the entire recording data of the current sample batch.

7. The audio data enhancement method based on sound pickup environment factors according to claim 2, characterized in that: The segmentation includes: Determine the start time of the first sample of the entire recording data of the current sample batch, and split it into batchsize new samples according to the preset time length.

8. The audio data enhancement method based on sound pickup environment factors according to claim 7, characterized in that: The marking includes: Using the labels of the current batch of samples extracted during the label extraction process, each sample in this batch is sequentially labeled to form the final enhanced sample-label pair. Based on the final enhanced sample-label pair, the enhanced sample training set of the current sample batch is obtained.

9. An audio data enhancement device based on sound pickup environment factors, characterized in that: include: An acquisition module, used to obtain a sample training set of original audio data to be enhanced; A preprocessing module is used to sequentially perform batching, verification, label extraction, and merging operations on the sample training set to obtain several batches of complete audio data; The acquisition module is used to schedule the microphone and speaker to mix the audio data of each sample batch in the real environment and collect the environmental factors to obtain the entire recording data of the current sample batch; A data post-processing module is used to sequentially segment and label the entire recording data according to the batches and the corresponding data extracted by the labels to obtain an enhanced sample training set for the current sample batch; Obtain the enhanced sample training set of each sample batch and concatenate them to obtain the final enhanced sample training set of the original audio data.

10. A computer-readable storage medium storing one or more programs, characterized in that: The one or more programs include instructions that, when executed by a computing device, cause the computing device to perform the method of any one of claims 1 to 8 .

Images