An audio simulation system based on deep learning algorithm

Abstract

The present invention relates to the field of audio simulation, and discloses an audio simulation system based on a deep learning algorithm. The system comprises an audio signal data acquisition module, a data storage and marking module, a feature extraction module, a feature analysis module, a feature recognition module, a recognition comparison module, and a speech post-processing and output module. The collected audio signal data is segmented, labeled, and classified for storage, the features of the audio signal data are extracted through features, and then feature analysis is performed. Feature recognition is performed on the entire audio signal through feature recognition, and through recognition comparison, it is determined whether the quality of the entire audio signal meets the quality standard of speech output, and finally post-processing and output are performed, thereby improving the audio signal recognition efficiency, and effectively controlling the audio signal recognition result through recognition comparison, which is conducive to ensuring the accuracy of recognition, thereby saving the output time of the audio signal.

Description

Technical Field

[0001] The present invention relates to the technical field of audio simulation, and more specifically to an audio simulation system based on a deep learning algorithm. Background Art

[0002] Audio simulation system refers to a technical system that uses deep learning algorithms to simulate, synthesize or enhance sound. Traditional audio processing technologies, such as digital signal processing (DSP) and signal processing algorithms, can already achieve audio processing and enhancement to a certain extent. With the development of deep learning technology, especially the emergence of models such as recurrent neural networks (RNN) and convolutional neural networks (CNN), significant progress has been made in speech recognition, speech synthesis, emotion recognition and other fields. The development of these technologies has provided a solid foundation for the realization of audio simulation systems, but the emergence of deep learning algorithms has brought new breakthroughs in the field of audio processing, thereby enhancing the rise of reality and virtual reality technology and the advancement of audio processing technology. Deep learning can better capture the complex characteristics of audio.

[0003] However, the above process still has the following disadvantages:

[0004] First, the existing storage burden of a large amount of collected audio signal data is heavy, and the lack of segmentation annotation, storage and feature extraction of audio signal data not only causes data storage pressure, but also leads to inaccurate results of data feature extraction;

[0005] Secondly, the lack of simulation of complex audio feature analysis and automation and feature recognition cannot improve the efficiency of audio signal recognition. At the same time, there is a lack of recognition comparison to effectively control the audio signal recognition results, and the accuracy of recognition cannot be guaranteed, thereby increasing the output time of the audio signal. Summary of the invention

[0006] In order to overcome the above-mentioned defects of the prior art, the present invention provides an audio simulation system based on a deep learning algorithm to solve the problems existing in the above-mentioned background technology.

[0007] The present invention provides the following technical solution: an audio simulation system based on a deep learning algorithm, comprising:

[0008] Audio signal data acquisition module: by using a microphone to record the audio in the real environment to obtain the original audio signal, thereby collecting audio signal data and preprocessing it, and at the same time transmitting the preprocessed audio signal data to the data storage and marking module;

[0009] Data storage and labeling module: used to cut the audio signal data into several smaller segments, label the cut audio segments, and classify and store the labeled audio signal data in the database;

[0010] Feature extraction module: extracts features from the audio signal data in the cut audio segments by using a deep learning model to obtain time domain features and frequency domain features, and transmits the extracted audio signal features to the feature analysis module;

[0011] Feature analysis module: used to perform feature analysis on the audio signal features of each audio clip respectively, obtain feature evaluation index, evaluate different acoustic features in each audio clip by the feature evaluation index, and transmit the feature analysis results to the feature recognition module;

[0012] Feature recognition module: calculates the feature evaluation index and the first threshold to obtain a feature recognition coefficient, identifies the audio quality of the entire audio signal through the feature recognition coefficient, and transmits the feature recognition result to the recognition comparison module;

[0013] The calculation formula of the feature recognition coefficient is: u i represents the feature evaluation index of the i-th audio segment, μ represents the first threshold, and the features of the entire audio signal are identified by the comprehensive feature recognition coefficient, so as to judge the collected audio signal and serve as the basis for judging the speech output, and n represents the n audio segments cut in total;

[0014] The first threshold is set with reference to an existing audio quality assessment standard.

[0015] Identification and comparison module: used for comparing the feature identification coefficient with the second threshold value, judging whether the entire audio signal meets the quality standard of speech output, and transmitting the judgment result of meeting the quality standard of speech output to the speech post-processing and output module;

[0016] The recognition and comparison module compares the feature recognition coefficient with the second threshold value. If the feature recognition coefficient z is less than or equal to the preset threshold value Z, it is determined that the quality of the audio signal meets the output quality standard, and subsequent speech post-processing and output can be performed; if the feature recognition coefficient z is greater than the second threshold value Z, it is determined that the quality of the audio signal does not meet the output quality standard, and an information instruction for re-sound collection is required for the audio signal that does not meet the output quality standard;

[0017] The second threshold is set by relevant testing personnel based on historical data and experience.

[0018] Speech post-processing and output module: used to receive the judgment results and perform corresponding post-processing, and output corresponding information according to the results of post-processing.

[0019] Preferably, the sound collection method of the audio signal data collection module is to use a microphone array composed of multiple microphones, arrange the positions of the microphones to be suitable for the audio scene, use multiple microphones to collect audio signal data at the same time, and pre-process the collected audio signal data to adjust the quality of the audio signal data, including audio editing, format conversion, noise reduction and filtering.

[0020] Preferably, the specific operation steps of the data storage and marking module are:

[0021] S21: Determine the cutting segments of the audio signal data, and cut the audio signal data according to the speech segment as a cutting standard;

[0022] S22: according to the cutting standard, using the script to realize automatic cutting of the audio signal data into 1, 2, 3, ..., i, ..., n segments;

[0023] S23: annotate 1, 2, 3, ..., i, ..., n audio clips, and use a text file to record the label of each audio clip;

[0024] S24: Create a database table in the database, and then classify and store the annotated audio signal data into the database table.

[0025] Preferably, the time domain feature is used to analyze the timing characteristics of the signal, so as to understand the waveform shape and periodicity of the signal. The specific calculation formula is: x(j) represents the jth audio sample in the i-th audio segment, M represents the frame size, m represents the starting position of the audio frame, and i represents the i-th audio segment.

[0026] Preferably, the frequency domain feature is to analyze the frequency feature of the signal, so as to understand the frequency distribution of the signal. The specific calculation formula is: f ik represents the frequency of the kth frequency point in the ith audio segment, P i (k) represents the power of the frequency point in the i-th audio segment, and N represents the total number of frequency points.

[0027] Preferably, the feature evaluation index is an index used to quantify and evaluate the ability to distinguish different acoustic features in an audio signal. The specific calculation formula for calculating the feature evaluation index of the i-th audio segment is: R i represents the time domain feature, D i represents frequency domain features, a1 and a2 represent feature weights, and 0<a1<1, 0<a2<1.

[0028] Preferably, the post-processing of the speech post-processing and output module is to make corresponding adjustments to the output of the speech according to the detailed information annotated by the audio signal segment, including adjusting the speech speed and intonation, denoising and increasing the volume, and then outputting the processed audio signal through a speaker or headphones for the user to hear.

[0029] Technical effects and advantages of the present invention:

[0030] The present invention is provided with an audio signal data acquisition module to collect sound by using a signal acquisition device, convert the sound into an electrical signal, and then perform digital processing to convert it into a digital signal, thereby collecting audio signal data; the collected audio signal data is cut into a plurality of smaller segments by a data storage and marking module; the cut audio segments are marked and classified and stored, which is beneficial to reducing the heavy storage burden of a large amount of audio signal data, making the result of feature extraction more accurate and reliable; the feature extraction module extracts time domain features and frequency domain features; the feature analysis module performs feature analysis on the audio signal features of each audio segment to obtain a feature evaluation index; different acoustic features in each audio segment are evaluated by the feature evaluation index; the feature recognition module performs feature recognition on the audio signal by using a feature recognition coefficient; the recognition comparison module determines whether the quality of the audio signal meets the quality standard of the voice quality output; the voice post-processing and output module receives the judgment result and performs corresponding post-processing; according to the result of the post-processing, the corresponding information is output; the automatic feature analysis and feature recognition are performed, the audio signal recognition efficiency is improved; the audio signal recognition result is effectively controlled by the recognition comparison, which is beneficial to ensure the accuracy of recognition, thereby saving the output time of the audio signal. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 This is a structural diagram of an audio simulation system based on a deep learning algorithm of the present invention.

[0032] Figure 2 This is a flow chart of the operation of an audio simulation system based on a deep learning algorithm of the present invention. DETAILED DESCRIPTION

[0033] The technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the present invention. In addition, the forms of the various structures recorded in the following embodiments are merely illustrative. The audio simulation system based on a deep learning algorithm involved in the present invention is not limited to the various structures recorded in the following embodiments. All other implementations obtained by ordinary technicians in this field without making creative work belong to the scope of protection of the present invention.

[0034] The present invention provides an audio simulation system based on a deep learning algorithm, comprising:

[0035] Audio signal data acquisition module: By using a microphone to record the audio in the real environment to obtain the original audio signal, the audio signal data is collected and preprocessed, and the preprocessed audio signal data is transmitted to the data storage and marking module.

[0036] In this embodiment, the sound collection method of the audio signal data collection module is to use a microphone array composed of multiple microphones, arrange the positions of the microphones into a state suitable for the audio scene, use multiple microphones to collect audio signal data at the same time, and pre-process the collected audio signal data to adjust the quality of the audio signal data, including audio editing, format conversion, noise reduction and filtering.

[0037] It should be specifically noted that the audio signal data acquisition module realizes the automatic acquisition of audio signal data, better obtains audio signal data, improves the efficiency and accuracy of data acquisition, and ensures that the data format and quality meet the requirements.

[0038] Data storage and labeling module: used to cut the audio signal data into several smaller segments, label the cut audio segments, and classify and store the labeled audio signal data in the database.

[0039] In this embodiment, the specific operation steps of the data storage and marking module are:

[0040] S21: Determine the cutting segments of the audio signal data, and cut the audio signal data according to the speech segment as a cutting standard;

[0041] S22: according to the cutting standard, using the script to realize automatic cutting of the audio signal data into 1, 2, 3, ..., i, ..., n segments;

[0042] S23: annotate 1, 2, 3, ..., i, ..., n audio clips, and use a text file to record the label of each audio clip;

[0043] S24: Create a database table in the database, and then classify and store the annotated audio signal data into the database table.

[0044] It should be specifically explained that data annotation is performed on each cut audio segment, and a label is assigned to each segment, including voice commands and emotional annotations; a table suitable for storing audio signal data is created in the database; when storing, the data is classified and stored in different extensions according to the labels of the audio segments; the data storage and labeling module makes it easier for the system to identify and process the features of each segment, which is conducive to reducing the heavy storage burden of large amounts of audio signal data and making the feature extraction results more accurate and reliable.

[0045] Feature extraction module: By using the deep learning model to extract features from the audio signal data in the cut audio segment, the time domain features and frequency domain features are obtained, and the audio signal features after feature extraction are transmitted to the feature analysis module.

[0046] In this embodiment, the time domain feature is used to analyze the timing characteristics of the signal, so as to understand the waveform shape and periodicity of the signal. The specific calculation formula is: x(j) represents the jth audio sample in the i-th audio segment, M represents the frame size, m represents the starting position of the audio frame, and i represents the i-th audio segment.

[0047] The frequency domain feature is to analyze the frequency characteristics of the signal, so as to understand the frequency distribution of the signal. The specific calculation formula is: f ik represents the frequency of the kth frequency point in the ith audio segment, P i (k) represents the power of the frequency point in the i-th audio segment, and N represents the total number of frequency points.

[0048] It should be specifically stated that the specific steps of using the deep learning model to extract features from the audio signal data in the cut audio segment are:

[0049] S31: preprocessing the audio signal data, including converting into a suitable format, standardizing, and normalizing the audio signal;

[0050] S32: Use audio convolutional neural networks to represent audio signals as images, and use convolutional layers and pooling layers to learn audio features, thereby automatically extracting useful audio feature information from audio signals;

[0051] S33: Calculate the features that help identify the audio signal.

[0052] The frequency f of the kth frequency point in the i-th audio segment ik The calculation formula is k represents the index of the frequency point, F represents the sampling frequency of the signal, and Q represents the number of samples of the signal in the time domain;

[0053] The power P of the frequency point in the i-th audio segment i The calculation formula of (k) is X(k) represents the amplitude of the frequency point;

[0054] The feature extraction module can better describe the essential characteristics of the audio signal, thus facilitating subsequent audio analysis and improving the recognition efficiency of the audio signal.

[0055] Feature analysis module: used to perform feature analysis on the audio signal features of each audio clip respectively, obtain a feature evaluation index, evaluate different acoustic features in each audio clip through the feature evaluation index, and transmit the feature analysis results to the feature recognition module.

[0056] In this embodiment, the feature evaluation index is an indicator used to quantify and evaluate the ability to distinguish different acoustic features in an audio signal. The specific calculation formula for calculating the feature evaluation index of the i-th audio segment is: R i represents the time domain feature, D i represents frequency domain features, a1 and a2 represent feature weights, and 0<a1<1, 0<a2<1.

[0057] It should be specifically noted that the feature analysis module can automatically learn useful features from the data by analyzing the performance of different acoustic features in each audio clip and performing feature evaluation based on the specific acoustic feature type, thereby reducing human intervention, improving efficiency, reducing unnecessary calculations, saving computing resources, and realizing intelligent analysis of the audio, thereby improving the quality and clarity of the audio.

[0058] Feature recognition module: calculates the feature evaluation index and the first threshold to obtain a feature recognition coefficient, identifies the audio quality of the entire audio signal through the feature recognition coefficient, and transmits the feature recognition result to the recognition comparison module;

[0059] The calculation formula of the feature recognition coefficient is: u i represents the feature evaluation index of the i-th audio segment, μ represents the first threshold, and the features of the entire audio signal are identified by the comprehensive feature recognition coefficient, so as to judge the collected audio signal and serve as the basis for judging the speech output, and n represents the n audio segments cut in total;

[0060] The first threshold is set with reference to an existing audio quality assessment standard.

[0061] It should be specifically noted that the entire audio signal is judged for its overall quality by using the feature recognition coefficient, and is compared with the second threshold value, so as to identify whether the audio signal meets the quality output standard, thereby achieving the determination and output of the audio signal;

[0062] First, it is necessary to determine the first threshold μ: predict the audio signal of the validation set or the test set to obtain the predicted value of the feature recognition coefficient z; select a suitable threshold to convert the continuous feature recognition coefficient into a discrete audio quality label, and calculate the precision, recall and F1 score of the model according to different thresholds, and select the best threshold; the calculation formula for determining the first threshold can be expressed as μ = argmaxF1Score(μ), μ represents the selected best threshold, and F1Score(μ) represents the F1 score corresponding to the threshold μ.

[0063] Identification and comparison module: used for comparing the feature identification coefficient with the second threshold value, judging whether the entire audio signal meets the quality standard of speech output, and transmitting the judgment result of meeting the quality standard of speech output to the speech post-processing and output module;

[0064] The recognition and comparison module compares the feature recognition coefficient with the second threshold value. If the feature recognition coefficient z is less than or equal to the preset threshold value Z, it is determined that the quality of the audio signal meets the output quality standard, and subsequent speech post-processing and output can be performed; if the feature recognition coefficient z is greater than the second threshold value Z, it is determined that the quality of the audio signal does not meet the output quality standard, and an information instruction for re-sound collection is required for the audio signal that does not meet the output quality standard;

[0065] The second threshold is set by relevant testing personnel based on historical data and experience.

[0066] It should be specifically noted that effectively controlling the audio signal recognition result through recognition comparison is conducive to ensuring the accuracy of recognition, thereby saving the output time of the audio signal.

[0067] Speech post-processing and output module: used to receive the judgment results and perform corresponding post-processing, and output corresponding information according to the results of post-processing.

[0068] In this embodiment, the post-processing of the speech post-processing and output module is to adjust the output of the speech accordingly according to the detailed information annotated by the audio signal segment, including adjusting the speech speed and intonation, denoising and increasing the volume, and then outputting the processed audio signal through a speaker or headphones for the user to hear.

[0069] It should be noted that the specific application steps of audio simulation are:

[0070] S71: First, ensure that the collection environment is quiet, then use a professional microphone to collect a real sound that needs to be simulated, including but not limited to: a violin performance in a music classroom, and convert the collected violin performance sound into audio signal data through the pulse code modulation method and transmit it to the computer;

[0071] S72: using a script written in Python to segment the audio signal data into speech segments, and classifying and storing the segmented speech segments;

[0072] S73: performing feature screening through a deep learning model, and extracting parameters for respectively identifying audio signal characteristics of different cut segments;

[0073] S74: analyzing the audio signal features of the segments respectively, thereby evaluating the acoustic features in each audio signal segment;

[0074] S75: Based on the result of the feature analysis, the acoustic feature is input into the calculation model to obtain a feature recognition coefficient to judge the quality of the audio to be recognized;

[0075] S76: Compare the feature recognition coefficient with the second threshold value to determine whether the audio quality meets the output standard, perform post-processing on the audio signal that meets the quality output standard, and issue an information instruction to re-collect the audio signal that does not meet the quality output standard;

[0076] S77: The audio signal meeting the output standard is post-processed by audio processing software, and the processed audio signal data is then output as sound.

[0077] Finally: The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

[0078] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An audio simulation system based on a deep learning algorithm, characterized in that: include: Audio signal data acquisition module: by using a microphone to record the audio in the real environment to obtain the original audio signal, thereby collecting audio signal data and preprocessing it, and at the same time transmitting the preprocessed audio signal data to the data storage and marking module; Data storage and labeling module: used to cut the audio signal data into several smaller segments, label the cut audio segments, and classify and store the labeled audio signal data in the database; Feature extraction module: extracts features from the audio signal data in the cut audio segments by using a deep learning model to obtain time domain features and frequency domain features, and transmits the extracted audio signal features to the feature analysis module; Feature analysis module: used to perform feature analysis on the audio signal features of each audio clip respectively, obtain feature evaluation index, evaluate different acoustic features in each audio clip by the feature evaluation index, and transmit the feature analysis results to the feature recognition module; The feature evaluation index is an indicator used to quantify and evaluate the ability to distinguish different acoustic features in an audio signal. The specific calculation formula for calculating the feature evaluation index of the i-th audio segment is: , represents the time domain characteristics, represents the frequency domain characteristics, represents the feature weight, and , ; Feature recognition module: calculates the feature evaluation index and the first threshold to obtain a feature recognition coefficient, identifies the audio quality of the entire audio signal through the feature recognition coefficient, and transmits the feature recognition result to the recognition comparison module; The calculation formula of the feature recognition coefficient is: , represents the feature evaluation index of the i-th audio segment, represents the first threshold value, and the features of the entire audio signal are identified by the comprehensive feature recognition coefficient, so as to judge the collected audio signal and serve as the basis for judging the speech output, and n represents the n audio segments cut in total; The first threshold is set with reference to an existing audio quality assessment standard; Identification and comparison module: used for comparing the feature identification coefficient with the second threshold value, judging whether the entire audio signal meets the quality standard of speech output, and transmitting the judgment result of meeting the quality standard of speech output to the speech post-processing and output module; The recognition and comparison module compares the feature recognition coefficient with the second threshold value. If the feature recognition coefficient z is less than or equal to the preset threshold value Z, it is determined that the audio signal quality meets the output quality standard, and subsequent speech post-processing and output can be performed; If the feature recognition coefficient z is greater than the second threshold value Z, it is determined that the quality of the audio signal does not meet the output quality standard, and an information instruction to re-collect sound is required for the audio signal that does not meet the output quality standard; The second threshold is set by relevant testing personnel based on historical data and experience; Speech post-processing and output module: used to receive the judgment results and perform corresponding post-processing, and output corresponding information according to the results of post-processing.

2. The audio simulation system based on deep learning algorithm according to claim 1, characterized in that: The audio signal data acquisition module collects sound in a manner that uses a microphone array composed of multiple microphones, arranges the positions of the microphones to be suitable for the audio scene, uses multiple microphones to collect audio signal data at the same time, and pre-processes the collected audio signal data to adjust the quality of the audio signal data, including audio editing, format conversion, noise reduction, and filtering.

3. The audio simulation system based on deep learning algorithm according to claim 1, characterized in that: The specific operation steps of the data storage and marking module are: S21: Determine the cutting segments of the audio signal data, and cut the audio signal data according to the speech segment as a cutting standard; S22: according to the cutting standard, using the script to realize automatic cutting of the audio signal data into 1, 2, 3, ..., i, ..., n segments; S23: annotate 1, 2, 3, ..., i, ..., n audio clips, and use a text file to record the label of each audio clip; S24: Create a database table in the database, and then classify and store the annotated audio signal data into the database table.

4. The audio simulation system based on deep learning algorithm according to claim 1, characterized in that: The time domain feature is used to analyze the timing characteristics of the signal, so as to understand the waveform shape and periodicity of the signal. The specific calculation formula is: , represents the jth audio sample in the ith audio segment, M represents the frame size, m represents the starting position of the audio frame, and i represents the ith audio segment.

5. The audio simulation system based on deep learning algorithm according to claim 1, characterized in that: The frequency domain feature is to analyze the frequency characteristics of the signal, so as to understand the frequency distribution of the signal. The specific calculation formula is: , represents the frequency of the kth frequency point in the i-th audio segment, represents the power of the frequency point in the i-th audio segment, and N represents the total number of frequency points.

6. The audio simulation system based on deep learning algorithm according to claim 1, characterized in that: The post-processing of the speech post-processing and output module is to adjust the output of the speech accordingly according to the detailed information annotated by the audio signal segment, including adjusting the speech speed and intonation, denoising and increasing the volume, and then output the processed audio signal through a speaker or headphones for the user to hear.

Images