A method and system for enhancing audio data annotation accuracy based on time-frequency enhancement
Through the audio data labeling method based on time-frequency enhancement, using the Mel spectrogram and masking technology, the problem of low audio data labeling efficiency in the existing technology is solved, and higher labeling accuracy and efficiency are achieved, which improves teaching emotions. Automatic labeling accuracy.
Patent Information
- Application Number
- CN202210748518.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-06-29
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2042-06-29
AI Technical Summary
It is difficult for the existing technology to effectively and automatically label the audio data generated during the education and teaching process, resulting in low data analysis efficiency and affecting the application of artificial intelligence in the field of education.
Using the audio data labeling method based on time-frequency enhancement, the spectrum and logarithmic Mel spectrogram are obtained by converting the audio data into a mel spectrogram and using the audio attribute labeling model, combining frequency channel masking and time frame masking technology. Enhance the data processing to improve the generalization ability and labeling accuracy of the data.
It improves the accuracy and efficiency of teacher teaching audio data labeling, reduces the problem of incorrect labeling, enhances the accuracy of automatic labeling of teaching emotions, and improves classroom education evaluation.
Smart Images

Figure CN115132223B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of audio pattern recognition, and more specifically, to a method and system for enhancing audio data annotation accuracy based on time-frequency enhancement. Background Art
[0002] With the rapid development of artificial intelligence technology, more and more artificial intelligence technologies have appeared in the education industry, and it has become a trend to apply artificial intelligence technology in the field of education and teaching. In the education industry, the application of artificial intelligence technology plays a vital role in the various data in the education and teaching process, such as the audio data of teachers teaching. These data are crucial to the development of artificial intelligence in the field of education. After all, the development of artificial intelligence cannot be separated from a large amount of data.
[0003] In recent years, most of the education and teaching process has been presented online, resulting in a huge amount of daily data and complex data, which is not conducive to the use of artificial intelligence technology for analysis. In order to better use artificial intelligence technology to make more targeted adjustments and improvements to teachers' teaching strategies in the future, it is currently urgent to use an automatic annotation technology to automatically annotate the audio data generated in the education and teaching process. Summary of the invention
[0004] In response to at least one defect or improvement need in the prior art, the present invention provides a method for automatically labeling teaching emotions by performing data enhancement on audio data of teacher teaching, which can enhance the audio data of teacher teaching and make full use of audio information. Its purpose is to improve the effectiveness of audio data of teacher teaching collected in the education process, the speed of labeling audio data of teacher teaching, and the accuracy of labeling audio data of teacher teaching.
[0005] To achieve the above object, in a first aspect, the present invention provides a method for enhancing the accuracy of audio data annotation based on time-frequency enhancement, comprising the steps of:
[0006] Collecting first audio data, and converting the first audio data into a mel spectrogram through a transition including a linear spectrogram;
[0007] Input the Mel-spectrogram into the audio attribute annotation model to obtain a spectrogram and a logarithmic Mel-spectrogram respectively;
[0008] Using an audio data enhancement method to process the spectrogram and the logarithmic Mel-spectrogram to obtain enhanced first audio data;
[0009] The enhanced first audio data is input into the audio attribute labeling model to obtain the attribute label classification result of the first audio data.
[0010] Furthermore, the audio data enhancement method is specifically as follows:
[0011] By adopting mini-batch-based hybrid masking, the masked area of the hidden state of the first audio data and the masked area of the hidden state of the second audio data are mean-mixed to obtain the enhanced masked area of the hidden state of the first audio data; the masked area includes a frequency channel masked area and a time frame masked area.
[0012] Further, converting the first audio data into a mel spectrogram by a transition amount including a linear spectrogram specifically includes:
[0013] Performing frame windowing and fast Fourier transform operations on the first audio data to obtain the linear spectrogram;
[0014] Applying a Mel filter bank to the linear spectrogram to obtain a corresponding filter output;
[0015] The mel spectrogram with attribute labels is obtained by applying the filter output to the energy spectrum.
[0016] Furthermore, the Mel-spectrogram is input into the audio attribute annotation model to obtain a spectrogram as follows:
[0017] Inputting the Mel-spectrogram into the first network structure of the audio attribute annotation model to obtain a corresponding time-frequency invariant spectrogram feature vector;
[0018] The main architecture of the first network structure is CNN14, which applies a one-dimensional convolutional neural network to a waveform in the time domain; the first network structure includes a one-dimensional convolution input layer, several one-dimensional convolution blocks and a one-dimensional convolution output layer; each one-dimensional convolution block is composed of two convolution layers, and the layer behind each of the two convolution layers is a downsampling layer with a preset step size.
[0019] Furthermore, the Mel-spectrogram is input into the audio attribute annotation model to obtain a logarithmic Mel-spectrogram:
[0020] The mel-spectrogram is input into a second network structure of the audio attribute labeling model to obtain a corresponding time-frequency invariant logarithmic mel-spectrogram feature vector; the second network structure includes a two-dimensional convolution block.
[0021] Furthermore, the method further comprises the steps of:
[0022] The spectrogram feature vector and the logarithmic Mel-spectrogram feature vector are input into the audio attribute labeling model, and the attribute label classification result of the first audio data is obtained through feature connection operation and classification operation.
[0023] Furthermore, the range of the degree of masking is [10%, 25%].
[0024] In a second aspect, the present invention provides an audio data annotation accuracy enhancement system based on time-frequency enhancement, comprising:
[0025] A collection module, configured to collect first audio data, and convert the first audio data into a mel spectrogram through a transition amount including a linear spectrogram;
[0026] A conversion module, used for inputting the Mel-spectrogram into an audio attribute annotation model to obtain a spectrogram and a logarithmic Mel-spectrogram respectively;
[0027] An enhancement module, used for processing the spectrogram and the logarithmic Mel-spectrogram using an audio data enhancement method to obtain enhanced first audio data;
[0028] The classification module is used to input the enhanced first audio data into the audio attribute labeling model to obtain the attribute label classification result of the first audio data.
[0029] In a third aspect, the present invention provides an electronic device comprising at least one processing unit and at least one storage unit, wherein the storage unit stores a computer program, and when the computer program is executed by the processing unit, the processing unit is enabled to perform the steps of any one of the above-mentioned methods.
[0030] In a fourth aspect, the present invention provides a storage medium storing a computer program executable by an access authentication device, wherein when the computer program runs on the access authentication device, the access authentication device is enabled to perform the steps of any one of the above-mentioned methods.
[0031] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects compared with the prior art:
[0032] (1) The audio data annotation accuracy enhancement method based on time-frequency enhancement provided by the present invention uses waveform information and Mel-spectrogram information simultaneously for automatic annotation of teaching emotions. A novel data enhancement method is first used to process the teaching audio data, and then a model is used to accurately extract spectrogram features and logarithmic Mel-spectrogram features. The spectrogram features and logarithmic Mel-spectrogram features are connected to reduce the problem of incorrect annotation, which is of great significance to the improvement of classroom education evaluation.
[0033] (2) The present invention adopts a method of data enhancement for the input space and the hidden space of the deep neural network, and enhances the input and intermediate features using frequency channel masking blocks and time frame masking block technology. This technology can ensure that the model not only focuses on the most discriminative part of the feature, but also focuses on the entire part of the feature, thereby improving the generalization ability of the data. In addition, the spectrogram and the logarithmic Mel spectrogram are combined into a new representation in the model, and the spectrogram in the time domain and the logarithmic Mel spectrogram in the frequency domain can be fused into a representation in the time-frequency domain, providing more reference information for the final automatic labeling of the teaching emotion label, thereby improving the accuracy of the model's automatic labeling of teaching emotions. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0035] Figure 1 A schematic diagram of a flow chart of a method for enhancing the accuracy of audio data annotation based on time-frequency enhancement provided by an embodiment of the present invention;
[0036] Figure 2 A schematic diagram of a scenario for obtaining audio data in a teacher teaching environment provided by an embodiment of the present invention;
[0037] Figure 3 A schematic diagram of the network structure of the teaching emotion automatic labeling model provided by an embodiment of the present invention;
[0038] Figure 4 A block diagram of an electronic device suitable for implementing the method described above provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0039] In order to make the purpose, technical scheme and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.
[0040] The terms "first", "second" or "third" etc. in the specification, claims or the above-mentioned drawings of this application are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units inherent to these processes, methods, products or devices.
[0041] Automatic annotation of teaching emotions requires the work of annotating and classifying the audio of teachers' teaching. Classifying and processing the audio of teachers' teaching according to emotions is conducive to the efficient development of subsequent work. For the provided audio data of teachers' teaching, appropriate data enhancement can improve the balance of the audio data of teachers' teaching, thereby improving the accuracy of automatic annotation of teaching emotions. Therefore, after data enhancement of the provided audio data of teachers' teaching, the waveform and spectrum of the enhanced audio data of teachers' teaching are used to automatically annotate the audio data of teachers' teaching, which can improve the utilization rate of data in the teaching process. At the same time, compared with the manual annotation of audio data of teachers' teaching, the realization of automatic annotation of teaching emotions after data enhancement greatly speeds up the speed of data sorting and labeling, which has the advantages of saving time, effort and money.
[0042] like Figure 1 As shown, in one embodiment, a method for enhancing audio data annotation accuracy based on time-frequency enhancement mainly includes the following process steps.
[0043] S1. Data collection part.
[0044] The audio data may be any piece of speech data that needs to be annotated. In this embodiment, the audio data is the audio data of the teacher's teaching in the teaching process, that is, the recording data of the teaching site. Figure 2 A schematic diagram of a scene of acquiring audio data in a teacher teaching environment. In an optional implementation, according to the needs, the collected audio data of the teacher teaching is first preprocessed, specifically:
[0045] The input audio data of the teacher's teaching is preprocessed by pre-emphasis, framing and windowing operations. In a specific example, the window function of the windowing operation uses a Hamming window, and the window size of the Hamming window is 1024. The audio data of the teacher's teaching after framing is 100 frames per second. Then, short-time Fourier transform is used for processing, which is equivalent to performing fast Fourier transform on a series of audio data of the teacher's teaching to obtain a linear spectrogram. The fast Fourier transform has a parameter N, which indicates how many points are fast Fourier transformed. If the number of points in a frame is less than N, a zero-filling operation will be performed to fill it. Each point corresponds to a frequency point. The frequency represented by a certain point n is: f n =(n-1)*f s / N, where s is the step size, f s It is the frame frequency of the sound frame, that is, the frequency of several sound frames.
[0046] Then apply 64 Mel filter banks to the linear spectrogram, and the expression of the filter output is:
[0047]
[0048] Where m ranges from 1 to M, M represents the number of filter banks (here M is 64), k represents the number of points, H m (k) represents the filter output value. Then the output of the filter is applied to the energy spectrum to obtain the Mel spectrogram. The expression of the Mel spectrogram is:
[0049]
[0050] where |X(k)| 2 It represents the energy of the kth point in the energy spectrum. The output of each filter within the frequency range is used as the weight to multiply the energy of the corresponding frequency in the energy spectrum. Finally, an accumulation operation is performed, that is, the weighted sum process. MelSpec(m) represents the Mel spectrogram.
[0051] Clean up the abnormal audio files of teachers teaching (for example, audio clips of teachers teaching with empty content).
[0052] S2, data enhancement part.
[0053] In this embodiment, the VoiceAugment audio data enhancement method is used to enhance the input audio data of the teacher's teaching.
[0054] Among them, the VoiceAugment audio data enhancement method includes frequency channel masking blocks and time frame masking blocks, and adopts small-batch-based hybrid masking for processing.
[0055] Assumptions Represents the intermediate hidden state of the input data, where T and F represent the number of time frames and frequency channels, respectively. Time frame masking is used for continuous time frames [t0, t0+t], which means replacing the specified elements with zero or other values, where t is a value selected from a uniform distribution from 0 to the time masking parameter t′, and t0 is a value selected from [0, Tt]. Similarly, frequency channel masking is used for continuous frequency channels [f0, f0+f], where f is first selected from a uniform distribution from 0 to the frequency masking parameter f′, and f0 is a value selected from [0, Ff]. In order to simplify the process, for an audio sample of a teacher's teaching, the same time masking and frequency masking are performed on each intermediate hidden state of the same layer in the sample, that is:
[0056] t0=t=f0=f
[0057] In a specific example, there are C feature maps in the lth layer of the CNN (Convolutional Neural Network) model. Then these C feature maps share the same time region and frequency region in one training iteration and mask the audio samples of a single teacher's teaching.
[0058] For example, to enhance the hidden state of the lth layer of the selected teacher teaching audio sample (i.e., the first audio data mentioned in the claims), it is necessary to randomly select another teacher teaching audio sample (i.e., the second audio data mentioned in the claims) in the same small batch as the selected teacher teaching audio sample, and mask it with the hidden state of the lth layer of the randomly selected teacher teaching audio sample. The selected data enhancement method based on small batch hybrid masking processes the masked areas of the hidden states of the two selected teacher teaching audio samples in a mean mixing manner.
[0059] The specific content of the data enhancement algorithm based on small batch hybrid masking is:
[0060] Input: Hidden states of audio samples taught by the selected teacher Hidden states of audio samples from another teacher in the same mini-batch The number of consecutive time frames t, the index of the starting time t0, the number of consecutive frequency channels f, the index of the starting frequency f0.
[0061] Output: Hidden state after augmentation of audio samples of selected teacher teaching
[0062] The more specific process is:
[0063] Initialize the value of x' to x;
[0064] When i∈[t0,t0+t],j∈[0,F], calculate x′[i,j]:
[0065]
[0066] When i∈[0,T],j∈[f0,f0+f], calculate x′[i,j]:
[0067]
[0068] Return the value of x′ at this time, which is the final result.
[0069] In a specific example, the audio data of the teacher's teaching is substituted as a selected sample into the data enhancement algorithm based on small batch hybrid masking, so that the input space and hidden space of the audio data of the teacher's teaching can be enhanced to obtain the audio data of the teacher's teaching after data enhancement. The specific process is:
[0070] Downsample the input audio data of the teacher's teaching to 22.05kHz;
[0071] The audio data of the teacher's teaching is subjected to short-time Fourier transform with a window size w of 2048 and an overlap rate o of 25%;
[0072] Then use the Mel-scale filter bank on the audio data of the teacher's teaching;
[0073] Produces 256 filter bins and 43 frames per second of data;
[0074] Normalize the input audio data of the teacher's teaching to zero mean;
[0075] The Adam optimizer uses a total of 350 batches and the initial learning rate i is 1×10 -4 ;
[0076] The learning rate i starts to decay linearly from the 50th stage until it reaches 5×10 at the 250th stage. -6 , with the minimum learning rate of i being 5×10 -6 Train for another 100 batches;
[0077] The hyperparameters in the experiment are t0=43, f0=26, and about 10% of the time frame and frequency channels are masked.
[0078] Since the time frame and frequency channel are masked by the corresponding parts of the audio data of other teachers, it is equivalent to introducing interference and training the network's resolution. Mini-batch-based mixed masking is a natural algorithm that retains the entire sample information of the audio of the selected teacher, while the interference of the time frame and frequency channels of the audio samples of other teachers can enhance robustness. The degree of masking has a significant impact on the effect of data enhancement. Too much masking may cause the original teacher audio data to lose a lot of information, which will seriously confuse the training of the model. The performance of data enhancement is best when the degree of masking is [10%, 25%. The degree of masking used in the example is 10%, which is enough to guide the data enhancement algorithm to focus on low-resolution time frames and frequency channels and improve robustness.
[0079] S3, automatic labeling part.
[0080] Figure 3 : is a schematic diagram of the network structure of the teaching emotion automatic annotation model (ANNA model) provided in this embodiment. Figure 3 As shown, the teaching emotion automatic labeling model (that is, the audio attribute labeling model mentioned in the claims, such as happiness, sadness and other emotions are actually certain emotion-type attributes of the audio) includes a first network structure and a second network structure; the first network structure uses the input Mel spectrogram to obtain a spectrogram, and the second network structure uses the input Mel spectrogram to obtain a logarithmic Mel spectrogram.
[0081] The model connects and predicts the spectrogram features output by the first network structure and the log-mel spectrogram features output by the second network structure, and finally obtains the emotional label classification of the audio of the teacher's teaching.
[0082] In fact, the above-mentioned audio data enhancement method can also be used to process the spectrogram features and logarithmic Mel-spectrogram features to obtain enhanced first audio data; then, the enhanced first audio data is input into the teaching emotion automatic labeling model to obtain the attribute label classification result (emotion label classification situation) of the first audio data.
[0083] In this embodiment, the first network structure is a one-dimensional convolutional neural network, which is used to extract the spectrogram of time-frequency information.
[0084] In a specific example, CNN14 is used as the main architecture of the first network structure.
[0085] Apply a 1D CNN to the time-domain waveform.
[0086] The one-dimensional CNN starts with a convolutional layer with a filter w of length 11 and a step size s of 5 to reduce the size of the input. This operation reduces the length of the input by 5 times, reduces memory usage, and improves overall efficiency.
[0087] This is followed by three convolutional blocks, each of which consists of two convolutional layers designed to expand the receptive field of the convolutional layer.
[0088] The next layer after each convolutional block is a downsampling layer with a stride s of 4.
[0089] By using a step size of 3 and downsampling, a 32KHz teacher teaching audio can be downsampled to 100 frames per second.
[0090] Among them, the output size of the one-dimensional CNN is T×C, T is the number of frames, and C is the number of channels;
[0091] By dividing the number of channels C into C / f groups, each group after division has a frequency of f, which allows the final spectrogram to learn frequency information. The output can be reshaped into a tensor of size T, which is the final output spectrogram.
[0092] The first network structure uses the input Mel spectrogram to obtain the spectrogram, and its process includes:
[0093] Obtain the original audio data of the teacher's teaching, perform frame division, windowing and fast Fourier transformation on the audio data to obtain a linear spectrogram, then apply a Mel filter to the linear spectrogram and take the logarithm to obtain a Mel spectrogram with an emotion label;
[0094] The Mel-spectrogram is input into a first network structure, and a corresponding time-frequency invariant spectrogram feature vector is obtained through the network structure.
[0095] In this embodiment, a neural network is used to extract the time-frequency representation of the waveform, and the frequency information that the network lacks is learned in a one-dimensional convolutional neural network, thereby learning a new time-frequency transform, namely, a spectrogram feature vector.
[0096] In this embodiment, the second network model is a two-dimensional convolutional neural network, which is used to extract the logarithmic Mel spectrogram.
[0097] The second network structure uses the input Mel spectrogram to obtain the logarithmic Mel spectrogram. The process includes:
[0098] The audio data of the teacher's teaching is obtained, and the audio data is framed, windowed and fast Fourier transformed to obtain a linear spectrogram. A Mel filter is then applied to the linear spectrogram and the logarithm is taken to obtain a Mel spectrogram with an emotion label.
[0099] The Mel-spectrogram is input into a second network structure, and a corresponding time-frequency invariant logarithmic Mel-spectrogram feature vector is obtained through the network structure.
[0100] Use a two-dimensional convolution block to operate on the input data;
[0101] After being processed by the second network structure, a corresponding feature map is generated;
[0102] That is, the Mel-spectrogram is input into a second network structure, and the corresponding time-frequency invariant logarithmic Mel-spectrogram feature vector is obtained through the network structure.
[0103] The teaching emotion automatic labeling model also includes a connection network and a classification network.
[0104] The connection network is respectively connected to the output sides of the first network structure and the second network structure, and is used to connect the spectrogram features output by the first network structure and the logarithmic Mel spectrogram features output by the second network structure to obtain connection features.
[0105] In a specific example, the classification network may successively use a ReLU classifier and a Sigmoid classifier, which generates a sentiment label classification of the teacher's teaching audio according to the connection features.
[0106] Among them, the final network structure of the model is as follows Figure 3 As shown, including:
[0107] 12 convolutional layers with kernel size 3×3 and 2 fully connected layers.
[0108] Every two convolutional layers are followed by an average pooling layer of size 2×2 for downsampling.
[0109] Global pooling is used after the last convolutional layer to aggregate the feature maps into a fixed-length vector.
[0110] An additional fully connected layer is added after the global pooling layer to extract the embedded features and further improve the representation ability of the features.
[0111] Embed linear classifiers (ReLU and Sigmoid) in the features to achieve classification labeling tasks.
[0112] After obtaining the automatically annotated emotions of the teacher's teaching audio, the teacher's emotions can be further analyzed to improve the teacher's teaching ability.
[0113] In one embodiment, a system for enhancing the accuracy of audio data annotation based on time-frequency enhancement mainly includes:
[0114] A collection module, configured to collect first audio data, and convert the first audio data into a mel spectrogram through a transition amount including a linear spectrogram;
[0115] A conversion module, used for inputting the Mel-spectrogram into an audio attribute annotation model to obtain a spectrogram and a logarithmic Mel-spectrogram respectively;
[0116] An enhancement module, used for processing the spectrogram and the logarithmic Mel-spectrogram using an audio data enhancement method to obtain enhanced first audio data;
[0117] The classification module is used to input the enhanced first audio data into the audio attribute labeling model to obtain the attribute label classification result of the first audio data.
[0118] Figure 4 A block diagram of an electronic device suitable for implementing the method described above according to an embodiment of the present invention is schematically shown. Figure 4 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.
[0119] like Figure 4 As shown, the electronic device 1000 described in this embodiment includes: a processor 1001, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage part 1008 to a random access memory (RAM) 1003. The processor 1001 may include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (for example, an application-specific integrated circuit (ASIC)), etc. The processor 1001 may also include an onboard memory for caching purposes. The processor 1001 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.
[0120] In RAM 1003, various programs and data required for the operation of system 1000 are stored. Processor 1001, ROM 1002 and RAM 1003 are connected to each other through bus 1004. Processor 1001 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM 1002 and / or RAM 1003. It should be noted that the program can also be stored in one or more memories other than ROM 1002 and RAM 1003. Processor 1001 can also perform various operations of the method flow according to the embodiment of the present disclosure by executing the program stored in the one or more memories.
[0121] According to an embodiment of the present disclosure, the electronic device 1000 may further include an input / output (I / O) interface 1005, which is also connected to the bus 1004. The system 1000 may further include one or more of the following components connected to the I / O interface 1005: an input section 1006 including a keyboard, a mouse, etc.; an output section 1007 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN card, a modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the I / O interface 1005 as needed. A removable medium 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1010 as needed, so that a computer program read therefrom is installed into the storage section 1008 as needed.
[0122] According to the method flow of the embodiment of the present disclosure, it can be implemented as a computer software program. For example, the embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program contains a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 1009, and / or installed from the removable medium 1011. When the computer program is executed by the processor 1001, the above-mentioned functions defined in the system of the embodiment of the present disclosure are executed. According to the embodiment of the present disclosure, the system, equipment, device, module or unit described above can be implemented by a computer program module.
[0123] The embodiments of the present invention further provide a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiments of the present disclosure is implemented.
[0124] According to an embodiment of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In an embodiment of the present disclosure, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present disclosure, the computer-readable storage medium may include one or more memories other than the ROM 1002 and / or RAM 1003 described above.
[0125] It should be noted that the functional modules in the various embodiments of the present invention can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above-mentioned integrated modules can be implemented in the form of hardware or in the form of software functional modules. If the integrated modules are implemented in the form of software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product.
[0126] The flowchart or block diagram in the accompanying drawings illustrates the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the above-mentioned module, program segment or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0127] Those skilled in the art will appreciate that the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways, even if such combinations and / or combinations are not explicitly described in the present disclosure. In particular, without departing from the spirit and teachings of the present disclosure, the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways, and all of these combinations and / or combinations fall within the scope of the present disclosure.
[0128] Although the present disclosure has been shown and described with reference to specific exemplary embodiments of the present disclosure, it should be understood by those skilled in the art that various changes in form and details may be made to the present disclosure without departing from the spirit and scope of the present disclosure as defined by the appended claims and their equivalents. Therefore, the scope of the present disclosure should not be limited to the above-mentioned embodiments, but should be determined not only by the appended claims, but also by the equivalents of the appended claims.
Claims
1. A method for enhancing the accuracy of audio data annotation based on time-frequency enhancement, characterized in that: Includes steps: Collecting first audio data, and converting the first audio data into a mel spectrogram through a transition including a linear spectrogram; Input the Mel-spectrogram into the audio attribute annotation model to obtain a spectrogram and a logarithmic Mel-spectrogram respectively; The spectrogram and the logarithmic Mel-spectrogram are processed using an audio data enhancement method to obtain enhanced first audio data; the audio data enhancement method is specifically: Using mini-batch-based hybrid masking, the masked region of the hidden state of the first audio data and the masked region of the hidden state of the second audio data are mean-mixed to obtain an enhanced masked region of the hidden state of the first audio data; the masked region includes a frequency channel masked region and a time frame masked region; The enhanced first audio data is input into the audio attribute labeling model to obtain the attribute label classification result of the first audio data.
2. The method for enhancing the audio data annotation accuracy according to claim 1, characterized in that: Converting the first audio data into a Mel spectrogram by a transition amount including a linear spectrogram specifically comprises: Performing frame windowing and fast Fourier transform operations on the first audio data to obtain the linear spectrogram; Applying a Mel filter bank to the linear spectrogram to obtain a corresponding filter output; The mel spectrogram with attribute labels is obtained by applying the filter output to the energy spectrum.
3. The method for enhancing the audio data annotation accuracy according to claim 1, characterized in that: Inputting the Mel spectrogram into the audio attribute annotation model to obtain the spectrogram is specifically as follows: Inputting the Mel-spectrogram into the first network structure of the audio attribute annotation model to obtain a corresponding time-frequency invariant spectrogram feature vector; The main architecture of the first network structure is CNN14, which applies a one-dimensional convolutional neural network to a waveform in the time domain; the first network structure includes a one-dimensional convolution input layer, several one-dimensional convolution blocks and a one-dimensional convolution output layer; each one-dimensional convolution block is composed of two convolution layers, and the layer behind each of the two convolution layers is a downsampling layer with a preset step size.
4. The method for enhancing the audio data annotation accuracy according to claim 3, characterized in that: Inputting the Mel spectrogram into the audio attribute annotation model to obtain the logarithmic Mel spectrogram is specifically as follows: The mel-spectrogram is input into a second network structure of the audio attribute labeling model to obtain a corresponding time-frequency invariant logarithmic mel-spectrogram feature vector; the second network structure includes a two-dimensional convolution block.
5. The method for enhancing the audio data annotation accuracy according to claim 4, characterized in that: Also includes the steps: The spectrogram feature vector and the logarithmic Mel-spectrogram feature vector are input into the audio attribute labeling model, and the attribute label classification result of the first audio data is obtained through feature connection operation and classification operation.
6. The method for enhancing the audio data annotation accuracy according to claim 1, characterized in that: The range of masking degree is .
7. A system for enhancing the accuracy of audio data annotation based on time-frequency enhancement, characterized in that: include: A collection module, configured to collect first audio data, and convert the first audio data into a mel spectrogram through a transition amount including a linear spectrogram; A conversion module, used for inputting the Mel-spectrogram into an audio attribute annotation model to obtain a spectrogram and a logarithmic Mel-spectrogram respectively; The enhancement module is used to process the spectrogram and the logarithmic Mel-spectrogram using an audio data enhancement method to obtain enhanced first audio data; the audio data enhancement method is specifically: Using mini-batch-based hybrid masking, the masked region of the hidden state of the first audio data and the masked region of the hidden state of the second audio data are mean-mixed to obtain an enhanced masked region of the hidden state of the first audio data; the masked region includes a frequency channel masked region and a time frame masked region; The classification module is used to input the enhanced first audio data into the audio attribute labeling model to obtain the attribute label classification result of the first audio data.
8. An electronic device, characterized in that: The method comprises at least one processing unit and at least one storage unit, wherein the storage unit stores a computer program, and when the computer program is executed by the processing unit, the processing unit executes the steps of the method according to any one of claims 1 to 6.
9. A storage medium, characterized in that: It stores a computer program executable by an access authentication device. When the computer program runs on the access authentication device, the access authentication device executes the steps of the method according to any one of claims 1 to 6.
Citation Information
Patent Citations
Music audio classification method based on convolutional recurrent neural network
CN112199548A
Comprehensive teaching video voice text extraction method
CN114639377A
Sound signal processing method and apparatus, and electronic device
WO2022121799A1
