Audio processing method and device, electronic equipment and readable storage medium

Abstract

The application discloses an audio processing method and device, electronic equipment and a readable storage medium, and belongs to the technical field of audio processing. The method comprises the following steps: dividing a first audio file into N1 segments according to characteristic information of the first audio file, wherein N1 is a positive integer, and N1>1; performing first dimension reduction on characteristic information of a target segment, the target segment being one of the N1 segments; fusing the characteristic information after the first dimension reduction; performing second dimension reduction on the fused characteristic information; obtaining characteristic information of the N1 segments after processing respectively, and determining N2 pieces of labeling information of the first audio file according to the characteristic information of the N1 segments after processing, wherein N2 is a positive integer.

Description

Technical Field

[0001] This application belongs to the field of audio processing technology, specifically relating to an audio processing method, apparatus, electronic device, and readable storage medium. Background Technology

[0002] Currently, as the internet and big data become increasingly integrated into people's lives, the use of artificial intelligence for information recommendation is becoming more mature and sophisticated. For example, when people use music playback software, they select their favorite songs and click to play them. Therefore, based on a user's playback history within the software, similar songs, such as classical music, will be recommended to the user on the recommendation page.

[0003] In existing technologies, song categorization is achieved manually, meaning that relevant personnel label songs by genre. Furthermore, music playback software can recommend similar songs to users based on these labels.

[0004] It is evident that, in existing technologies, song genre labeling relies on subjective human interpretation, leading to inaccurate labeling. Summary of the Invention

[0005] The purpose of this application is to provide an audio processing method that can solve the problem that in the prior art, the labeling of song types relies on human subjective ideas, resulting in inaccurate labeling.

[0006] In a first aspect, embodiments of this application provide an audio processing method, the method comprising: dividing the first audio file into N1 segments based on feature information of the first audio file, where N1 is a positive integer and N1 > 1; performing a first dimensionality reduction on the feature information of a target segment, wherein the target segment is one of the N1 segments; fusing the feature information after the first dimensionality reduction; performing a second dimensionality reduction on the fused feature information; obtaining the processed feature information of the N1 segments respectively, and determining N2 annotation information of the first audio file based on the processed feature information of the N1 segments, where N2 is a positive integer.

[0007] Secondly, embodiments of this application provide an audio processing apparatus, comprising: a segmentation module, configured to divide a first audio file into N1 segments based on feature information of the first audio file, where N1 is a positive integer and N1 > 1; a first dimensionality reduction module, configured to perform a first dimensionality reduction on the feature information of a target segment, wherein the target segment is one of the N1 segments; a fusion module, configured to fuse the feature information after the first dimensionality reduction; a second dimensionality reduction module, configured to perform a second dimensionality reduction on the fused feature information; and a determination module, configured to obtain the processed feature information of the N1 segments respectively, and determine N2 annotation information of the first audio file based on the processed feature information of the N1 segments, where N2 is a positive integer.

[0008] Thirdly, embodiments of this application provide an electronic device including a processor and a memory, wherein the memory stores programs or instructions executable on the processor, and the programs or instructions, when executed by the processor, implement the steps of the method described in the first aspect.

[0009] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0010] Fifthly, embodiments of this application provide a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the method as described in the first aspect.

[0011] In a sixth aspect, embodiments of this application provide a computer program product stored in a storage medium, which is executed by at least one processor to implement the method described in the first aspect.

[0012] Thus, in the embodiments of this application, feature information of a first audio file is obtained, and the first audio file is divided into multiple segments based on the obtained feature information, thereby transforming the first audio file into a segmented structured file for subsequent data processing. In the subsequent processing, each segment is processed separately, including three steps: dimensionality reduction, fusion, and dimensionality reduction again. Finally, based on each processed segment, an aggregation matrix representing the features of the first audio file is output. Further, based on the output aggregation matrix, at least one annotation information of the first audio file is determined. It is evident that the feature processing of the first audio file results in a high information density in the final aggregation matrix, capable of expressing both the features of individual parts of the first audio file and the overall features of the first audio file, thereby achieving a high accuracy rate in intelligently determining the annotation information of the first audio file based on the aggregation matrix. Attached Figure Description

[0013] Figure 1 This is a flowchart of an audio processing method according to an embodiment of this application;

[0014] Figure 2 This is a block diagram of an audio processing apparatus according to an embodiment of this application;

[0015] Figure 3 This is one of the hardware structure diagrams of the electronic device according to an embodiment of this application;

[0016] Figure 4 This is the second schematic diagram of the hardware structure of the electronic device according to an embodiment of this application. Detailed Implementation

[0017] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0018] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0019] The audio processing method provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.

[0020] Figure 1 A flowchart of an audio processing method according to an embodiment of this application is shown. The method is applied to an electronic device and includes:

[0021] Step 110: Based on the feature information of the first audio file, divide the first audio file into N1 segments, where N1 is a positive integer and N1 > 1.

[0022] Optionally, the first audio file may be a song, an instrumental piece, or the like.

[0023] In this embodiment, based on the feature information of the first audio file, the audio file that cannot be directly processed can be converted into a segmented structured file. Further data processing and algorithm utilization are performed on each segment to finally obtain the annotation information of the first audio file.

[0024] For example, the entire song can be divided into four sub-segments by cutting it into three parts, and each sub-segment can be processed independently in subsequent processing.

[0025] Step 120: Perform the first dimensionality reduction on the feature information of the target segment. The target segment is one of N1 segments.

[0026] Step 130: Fuse the feature information after the first dimensionality reduction.

[0027] Step 140: Perform a second dimensionality reduction on the fused feature information.

[0028] In this embodiment, the processing procedure is illustrated using the target segment as an example.

[0029] Because the acquired feature information includes multiple types, overall data fusion and dimensionality reduction processing are required. Therefore, the data in each bin segment is aggregated and dimensionality reduced. Overall, through a three-step approach of dimensionality reduction, fusion, and dimensionality reduction, the main representations of the features are not erased due to dimensionality reduction and fusion, thereby increasing the information density that the final feature set can contain.

[0030] An exemplary processing procedure is as follows: First, for the feature information in the target segment, a feature matrix can be formed; the first step is to perform linear dimensionality reduction on the feature matrix using Principal Component Analysis (PCA) or Linear Discriminant Analysis (LDA), compressing the n1 features into n1 / 2 features to obtain a new feature matrix; the second step is to perform weighted fusion on all the new features after linear dimensionality reduction to obtain the weight value of each feature; the third step is to perform nonlinear dimensionality reduction on the weighted values ​​of all features again using Kernel Principal Component Analysis (KPCA) or Kernel Discriminant Analysis (KDA) to obtain n1 / 4 features.

[0031] Step 150: Obtain the feature information of N1 segments after processing, and determine the N2 annotation information of the first audio file based on the feature information of N1 segments after processing, where N2 is a positive integer.

[0032] In this step, after obtaining the dimensionality-reduced feature matrix of each bin segment, they are simply concatenated in the order of their occurrence to form a large feature matrix, which serves as the final output aggregation matrix.

[0033] Optionally, the output aggregation matrix is ​​expanded into a vector with the existing matrix in the sample music library, and the similarity is calculated. The most similar audio files (such as three) are selected as similar audio files, and the N2 annotation information of the first audio file is determined according to the annotation information corresponding to the similar audio files.

[0034] The sample music library includes a large number of audio files, and these audio files have been annotated with manually created information.

[0035] Optionally, in the process of selecting similar audio files, the audio files in the sample music library can be processed using the method provided in this application to obtain the corresponding aggregation matrix, thereby performing a comparison between the aggregation matrices.

[0036] Thus, in the embodiments of this application, feature information of a first audio file is obtained, and the first audio file is divided into multiple segments based on the obtained feature information, thereby transforming the first audio file into a segmented structured file for subsequent data processing. In the subsequent processing, each segment is processed separately, including three steps: dimensionality reduction, fusion, and dimensionality reduction again. Finally, based on each processed segment, an aggregation matrix representing the features of the first audio file is output. Further, based on the output aggregation matrix, at least one annotation information of the first audio file is determined. It is evident that the feature processing of the first audio file results in a high information density in the final aggregation matrix, capable of expressing both the features of individual parts of the first audio file and the overall features of the first audio file, thereby achieving a high accuracy rate in intelligently determining the annotation information of the first audio file based on the aggregation matrix.

[0037] Furthermore, in the embodiments of this application, the method for intelligently determining the annotation information of the first audio file does not require manual annotation, thus saving manpower and reducing costs.

[0038] In the audio processing method of another embodiment of this application, step 110 includes:

[0039] Sub-step A1: Based on the feature information of the first audio file, obtain N3 sections, where N3 is a positive integer and N3 > 1.

[0040] In this step, the first audio file is divided into multiple sections.

[0041] Optionally, the changes in the relevant parameters of the feature information within a section fall within a certain range.

[0042] Sub-step A2: Combine the feature information of N4 consecutive subsections into a sliding window feature matrix, where N4 is a positive integer and N4 < N3. The feature information of one subsection is used to form at least one sliding window feature matrix.

[0043] In this step, it is necessary to construct a sliding window feature matrix for the first audio file, transforming N3 sections into (N3-4) sliding window feature matrices to prepare for subsequent data processing.

[0044] First, construct the first sliding window feature matrix for the first 5 sections (N4 is 5 here). In the matrix, rows represent feature information, and columns represent the a-th section (a∈(0-5)). After constructing the first sliding window feature matrix, delete the first column of the matrix and add the features of the next section after the fifth column; and so on. After each of the N3 sections has been constructed at least once by the sliding window feature matrix, the construction of the sliding window feature matrix of the first audio file is complete.

[0045] Sub-step A3: Compare the similarity between the feature matrices of two adjacent sliding windows.

[0046] Sub-step A4: Based on the matrix boundaries corresponding to the first N5 minimum similarities, divide the first audio file into N1 segments, where N5 is a positive integer.

[0047] In the above two steps, similarity calculation is performed on the constructed sliding window feature matrix to complete the segmentation of the first audio file, such as dividing the first audio file into 3 to 4 intervals (segments).

[0048] For reference, starting from the second sliding window feature matrix, the similarity between adjacent matrices is calculated with the feature matrices of the previous and next sliding windows, respectively, in order to determine whether there is a significant change in the information contained in the two matrices.

[0049] For example, in the first step, each feature matrix is ​​flattened into a one-dimensional vector, and the similarity θ between the original vector and the vectors formed by the two preceding and following features is calculated by calculating the cosine similarity. ax θ bx Based on this, two similarities are calculated for all non-first and last feature matrices. The second step is to perform a deduplication and summation of the calculated similarities, that is, to calculate the later similarity θ of the n2-th matrix. an2 The front similarity θ with the n2+1th matrix b(n2+1) The summation forms a new deduplication similarity θ n2The third step is to sort all the deduplicated similarity scores and select the three smallest values, which represent the positions where the changes between the two sections are most significant. At these positions, the first audio file is segmented into four segments.

[0050] Based on the aforementioned embodiments, after processing the target segment, the segments are sorted from front to back according to their order of appearance in the first audio file, and the feature matrix of the target segment is finally obtained.

[0051] In this embodiment, a method for segmenting a first audio file is provided. This method involves creating a sliding window feature matrix and further combining the similarity ranking of adjacent sliding window feature matrices. The minimum similarity is used as the dividing line, so that the feature information in each segment has a high similarity after segmentation, which can be used for independent processing.

[0052] In the audio processing method of another embodiment of this application, step A1 includes:

[0053] Sub-step B1: Obtain the first waveform signal corresponding to the first audio file.

[0054] Sub-step B2: Based on the information about the change in the peak interval in the first waveform signal, divide the first audio file into N3 sections.

[0055] In this embodiment, a first waveform signal corresponding to the first audio file is obtained, and the first audio file is divided into sections based on the recognition of strong audio accents in the first waveform signal.

[0056] For example, the audio strong accent recognition method is based on instantaneous peak sampling. Therefore, sections can be divided according to the typical peak sampling intervals and commonly used beat times in the first waveform signal. Specifically, the first audio file is divided into sections according to the length corresponding to the commonly used beat times. If there is a significant change in the peak intervals in the first waveform signal, the degree of change indicates that the sound source rhythm or the audio beat has sped up (from 1 / 4 beat to 1 / 8 beat). The section division is then completed by combining the increased rhythm and beat.

[0057] Each segment that is ultimately divided is a feature vector of structured data.

[0058] In this embodiment, a segmentation method is provided to achieve segmentation based on the characteristics in the waveform signal, so that the segmentation is based on the waveform changes, in order to prepare for the subsequent construction of segmented structured data.

[0059] In the audio processing method of another embodiment of this application, step 130 includes:

[0060] Sub-step C1: In the target segment, determine the target feature matrix, which is the sliding window feature matrix that appears most frequently in the target segment.

[0061] Sub-step C2: Delete the first feature matrix in the target segment, where the correlation between the first feature matrix and the target feature matrix is ​​less than a preset value.

[0062] Sub-step C3: Obtain the weighted values ​​of the feature information in the feature matrices other than the first feature matrix in the target segment.

[0063] Optionally, the high-frequency dimensionality-reduced matrix (i.e., the target feature matrix) obtained by linearly reducing the overall feature matrix of the binned segment (i.e., the target segment) is first used as a positive sample. Each feature matrix is ​​then used to predict its relevance with the positive samples using a Support Vector Machine (SVM). The median of the positive relevance of all feature matrices within the binned segment is used as a threshold (i.e., a preset value). The feature matrix with the lowest relevance of 40% (i.e., the first feature matrix) is discarded. The remaining feature matrices are then linearly weighted using the formula ((x-min) / (max-min)) to obtain the weighted value of each feature.

[0064] In this embodiment, feature matrices with low frequency of occurrence are deleted, and only feature matrices with high frequency of occurrence are retained as the output of the aggregation matrix of the first audio file. This makes the annotation information obtained based on the aggregation matrix of the first audio file more reflective of the type of the first audio file, and makes the annotation of the audio file more accurate.

[0065] In another embodiment of the audio processing method of this application, before step 120, the method further includes:

[0066] Step D1: Based on the feature information of the target segment, construct N6 feature groups. The feature information contained in a feature group belongs to the same category, and N6 is a positive integer.

[0067] In this embodiment, all features are first grouped by manually screening features with similar meanings. For example, rhythm and tone can be classified as similar features, thus constructing 15 to 20 feature groups.

[0068] Furthermore, when performing dimensionality reduction, the data can be reduced in units of feature groups.

[0069] In this embodiment, similar features are grouped to obtain a feature group matrix, and then further processing is performed to optimize the processing.

[0070] In another embodiment of the audio processing method of this application, step 150 includes:

[0071] Sub-step E1: If the feature information after processing N1 segments matches the feature information after processing the preset audio file, obtain the target feature group in the preset audio file. The frequency of the target feature group appearing in the preset audio file meets the preset condition.

[0072] Sub-step E2: Determine the N2 annotation information of the first audio file based on the preset label information corresponding to the target feature group.

[0073] In this embodiment, the aggregation matrix of the output first audio file can be matched with the sample music library based on a pre-created sample music library, so that multiple matched audio files can be used as preset audio files.

[0074] For reference, for a preset audio file, the same data processing is performed according to the steps of the foregoing embodiments of this application, thereby statistically analyzing the frequency of occurrence of features of the preset audio file. Combined with the constructed feature groups, two features with the highest frequency of occurrence are selected as typical features for each feature group. Finally, 30 to 40 features can be selected as reference features for determining the annotation information of the first audio file, thereby realizing intelligent annotation of the first audio file.

[0075] In this embodiment, a large number of audio files recorded in the pre-created sample music library are processed using the feature processing method provided in this application. Based on this feature processing method, annotation information corresponding to the features is predefined so that when the audio files in the music library are matched with the features of the audio files to be processed, the annotation information of the audio files in the music library is referenced to annotate the audio files to be processed.

[0076] In another embodiment of the audio processing method of this application, before the first dimensionality reduction, a global feature matrix is ​​extracted from the target segment. For each feature in the segment, the three most frequent feature values ​​are statistically selected to form a result matrix: [feature: value 1, value 2, value 3]. This matrix serves as the overall label for the features of the segment, thus creating a matrix where the column coordinates represent the highest, second-highest, and third-highest frequencies, and the row coordinates represent the occurrence of each feature. This matrix is ​​defined as the global feature matrix of the target segment. The global feature matrix of the target segment can be used to label the segment.

[0077] In another embodiment of the audio processing method of this application, a method for creating a sample music library is provided.

[0078] The first step involves constructing a standard instrument library for each instrument using standard instrument sound sources in music media production software, or through sound sampling for very specific instruments. This library comprises approximately a thousand instruments. A standard five-octave, 40-tone scale is then created for each instrument, establishing a comprehensive standard sound source library for multiple tones of a single instrument. The second step, after completing the comprehensive sound source library, involves constructing a sample music library using 100 songs of different styles. For each song featuring 10 instruments, sample data is generated by combining sound sources of the same scale or tone from the standard sound source library (e.g., combining the 1 (dao), 3 (mi), 4 (fa), 5 (sao), and 6 (la) of the selected 10 instruments to cover possible sound combinations in the audio file). This process, with 10% of the samples combined, generates 300,000 audio combination samples. These samples are then combined with the segmented sound source files from the song samples, standardized using Fourier transform, and then subjected to fully connected cross-learning using a neural network (NN). This process ultimately yields a training result of over a thousand positive samples. The training sample processing was repeated for 100 songs of different styles using the steps described above. Finally, supplementary learning was performed on the sound features of the trailing instruments: approximately 100 instruments not appearing in the 100 songs were grouped into sets of five and mixed with five randomly selected instruments from the 20 most frequently occurring instruments in the 100 songs to form supplementary trailing sample groups. Trailing features were constructed using the same method as the song training samples. After training all samples, a total of 100,000 to 200,000 training samples were obtained. Each sample was recorded as an audio file.

[0079] Correspondingly, based on the sample music library created above, when the first audio file is received and processed, the first audio file is first standardized and divided into standard data by Fourier transform. Each audio segment and 10% of the training samples randomly selected from the sample feature results are inversely transformed to select the top 50 similar audio segment features. The similarity results of the two matrices are then added together after the inner product of the instruments appearing in the feature. Finally, the probability values ​​of the top 10 instruments are selected and subjected to softmax normalization to obtain the probability values ​​of the top 10 instruments in the segment. These 10 instruments are then added as features to the instrument features after the segment (format: [instrument name 1: probability value 1, instrument name 2: probability value 2, ... instrument 10: probability value 10]).

[0080] In the above process, the background sound source is structured based on machine learning: audio features are obtained by combining standard sound sources of various musical instruments with machine learning, and the 10 instruments with the highest audio probability and their feature probability values ​​are predicted.

[0081] Furthermore, the output annotation information can be related to the musical style of the instruments with the highest probability values, etc.

[0082] In another embodiment of the audio processing method of this application, based on the feature processing method provided in the embodiments of this application, the language type of the human voice in the song and the word recognition of the song can be identified by speech recognition technology based on the feature information of the audio file of the song, thereby completing the relevant annotation.

[0083] For example, it can identify whether a segment contains human voices and identify the human voices (tenor, soprano, etc.), including pitch and main timbre processing, in order to achieve annotation.

[0084] In another embodiment of the audio processing method of this application, based on the feature processing method provided in the embodiments of this application, rhythm and intensity recognition can also be performed for the annotation of audio files.

[0085] For example, based on the duration (ms) of the divided measures, the overall tempo of the audio file (articulate, slow, andante, moderate, fast, etc.) can be determined for labeling; similarly, the intensity of the waveform in a measure can be identified (gentle, smooth, intense, etc.) for labeling; furthermore, the number of different rhythmic peaks in a measure can be labeled (main rhythm 3 / 4 time, 6 / 8 time, or 4 / 4 time) and the relative intensity of the peaks can be labeled (strong-weak-weak in 3 / 4 time, strong-weak-second strong-weak in 4 / 4 time, strong-weak-weak-second strong-weak-weak in 6 / 8 time, etc.) for labeling.

[0086] In another embodiment of the audio processing method of this application, based on the feature processing method provided in the embodiments of this application, processing can also be performed according to the sound wave frequency of each tonic to identify each note (Fa sharp, Do flat, etc.) for annotation; the musical timbre of the tonic at each peak can also be structured to identify the tonic instrument for annotation.

[0087] In another embodiment of the audio processing method of this application, the first audio file can also be annotated based on hidden feature information in the first audio file.

[0088] Optionally, the hidden feature information can be: feature information obtained by mining more dimensions based on the changing trend of a certain waveform feature information.

[0089] For reference, the following steps may be included:

[0090] By performing long feature processing on fragmented structured data, the acquired features can be sorted by category. For example, if the feature related to pitch and rhythm is denoted as feature m, then features m+1, ..., m+s can be grouped together. In this way, similar features can be classified and sorted to form feature groups.

[0091] Add the section number before the feature to get a result like [Section 1: Feature 1, Feature 2, ... Feature n]; further, perform data transformation: transform from using section as the classification condition to using feature as the classification condition, to get a result like [Feature 1: Section a, Section b, ... Section m].

[0092] The frequency of statistical features is counted and sorted. The parameter range with the most frequent occurrence of a certain feature in the first audio file is selected as the important feature, and the second and third most frequent parameter ranges are selected as secondary supplementary features. Optionally, five parameter ranges are selected as high-frequency features of the entire audio file, and ten parameter ranges are selected as high-frequency supplementary features.

[0093] To determine the distribution of statistical features, the first audio file can be roughly divided into sections based on the frequency of feature occurrences. After dividing the high-frequency features and high-frequency supplementary features, if the distribution is relatively uniform or normal, it is considered a normal distribution. If the distribution shows obvious peaks or segments, the high-frequency supplementary features under that feature are upgraded to high-frequency features, indicating a significant change in the audio file, thus allowing for dual high-frequency annotation of that feature.

[0094] The features are made uniform in length, and each segment is arranged in the form of [feature 1, feature 2, feature 3, ..., feature n] to ensure that the features are of equal length, which facilitates subsequent feature classification.

[0095] Based on a combination of certain features or multiple features, a latent vector is obtained as a supplementary vector through a fully connected deep learning algorithm.

[0096] For the supplementary vector, user basic features (such as age, gender, geographic location information, occupation, consumption level, social attributes, etc.) can be used as inputs for basic classification features and other supplementary features. Recommendation algorithms are then used to classify the features to obtain the user group features with high interest corresponding to the supplementary vector.

[0097] By combining data analysts' and algorithm engineers' matching techniques with keywords from natural language recognition, the features expressed by each latent vector are defined and described. Examples include dance music popular among young people in smaller cities, music genres favored by young people in third- and fourth-tier cities, and dance music frequently used as background music in short videos, often accompanied by catchy dance moves.

[0098] Finally, the high-frequency features selected through statistical features and the relevant hidden features are combined to classify the first audio file in order to determine the annotation information of the first audio file.

[0099] In this embodiment, structured audio data is used for feature aggregation. Furthermore, the frequency, number of occurrences, and stages of occurrence are analyzed to understand the patterns of feature changes, which are then used as a reference to supplement the annotation information of the first audio file. It is evident that the in-depth feature learning and classification method proposed in this embodiment can learn and classify some hidden or difficult-to-quantify features.

[0100] In summary, this application obtains the classification features of audio files by using statistical principles on the basic features of structured data; in addition, it uses deep learning methods to obtain latent features that cannot be obtained directly, and combines experience or natural language processing to summarize some implicit style feature information contained in the audio files. By combining these two types of features, the final classification of audio files is obtained for audio file annotation.

[0101] Compared to existing technologies, the annotation of audio files in this application is not dependent on the knowledge and reserves of the annotation engineer. Therefore, there will be no omissions, errors, or multiple annotations due to the annotation engineer's own misunderstanding or mistakes, resulting in a higher annotation accuracy. In addition, the annotation of audio files in this application combines machine learning and rule matching and other methods to realize machine recognition of audio voiceprint features, thereby making the classification and annotation of audio materials faster.

[0102] The audio processing method provided in this application can be executed by an audio processing device. This application uses an audio processing device executing the audio processing method as an example to illustrate the audio processing device provided in this application.

[0103] Figure 2 A block diagram of an audio processing apparatus according to another embodiment of this application is shown, the apparatus comprising:

[0104] The segmentation module 10 is used to divide the first audio file into N1 segments based on the feature information of the first audio file, where N1 is a positive integer and N1 > 1;

[0105] The first dimensionality reduction module 20 is used to perform the first dimensionality reduction on the feature information of the target segment, which is one of N1 segments;

[0106] The fusion module 30 is used to fuse the feature information after the first dimensionality reduction.

[0107] The second dimensionality reduction module 40 is used to perform a second dimensionality reduction on the fused feature information;

[0108] The determination module 50 is used to obtain the feature information of N1 segments after processing, and determine the N2 annotation information of the first audio file based on the feature information of N1 segments after processing, where N2 is a positive integer.

[0109] Thus, in the embodiments of this application, feature information of a first audio file is obtained, and the first audio file is divided into multiple segments based on the obtained feature information, thereby transforming the first audio file into a segmented structured file for subsequent data processing. In the subsequent processing, each segment is processed separately, including three steps: dimensionality reduction, fusion, and dimensionality reduction again. Finally, based on each processed segment, an aggregation matrix representing the features of the first audio file is output. Further, based on the output aggregation matrix, at least one annotation information of the first audio file is determined. It is evident that the feature processing of the first audio file results in a high information density in the final aggregation matrix, capable of expressing both the features of individual parts of the first audio file and the overall features of the first audio file, thereby achieving a high accuracy rate in intelligently determining the annotation information of the first audio file based on the aggregation matrix.

[0110] Optionally, module 10 is divided into:

[0111] The first acquisition unit is used to acquire N3 sections based on the feature information of the first audio file, where N3 is a positive integer and N3 > 1;

[0112] The component unit is used to form a sliding window feature matrix from the feature information of N4 consecutive subsections, where N4 is a positive integer and N4 < N3. The feature information of one subsection is used to form at least one sliding window feature matrix.

[0113] The comparison unit is used to compare the similarity between the feature matrices of two adjacent sliding windows;

[0114] The segmentation unit is used to divide the first audio file into N1 segments based on the matrix boundaries corresponding to the first N5 minimum similarity scores, where N5 is a positive integer.

[0115] Optionally, the first acquisition unit includes:

[0116] The acquisition subunit is used to acquire the first waveform signal corresponding to the first audio file;

[0117] The sub-unit is used to divide the first audio file into N3 sections based on the peak interval change information in the first waveform signal.

[0118] Optionally, the fusion module 30 includes:

[0119] The first determining unit is used to determine the target feature matrix in the target segment, wherein the target feature matrix is ​​the sliding window feature matrix that appears most frequently in the target segment;

[0120] The deletion unit is used to delete the first feature matrix in the target segment, where the correlation between the first feature matrix and the target feature matrix is ​​less than a preset value;

[0121] The second acquisition unit is used to acquire the weighted values ​​of feature information in feature matrices other than the first feature matrix in the target segment.

[0122] Optionally, the device further includes:

[0123] The component module is used to build N6 feature groups based on the feature information of the target segment. The feature information contained in a feature group belongs to the same category, and N6 is a positive integer.

[0124] Optionally, module 50 is defined, including:

[0125] The third acquisition unit is used to acquire the target feature group in the preset audio file when the feature information after processing N1 segments matches the feature information after processing the preset audio file, and the frequency of the target feature group in the preset audio file meets the preset condition.

[0126] The second determining unit is used to determine N2 annotation information of the first audio file based on the preset label information corresponding to the target feature group.

[0127] The audio processing device in this application embodiment can be an electronic device or a component within an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television set (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the scope of the device.

[0128] The audio processing device in this application embodiment can be a device with a motion system. The motion system can be an Android motion system, an iOS motion system, or other possible motion systems; this application embodiment does not specifically limit it.

[0129] The audio processing apparatus provided in this application embodiment can implement the various processes implemented in the above method embodiments, and will not be described again here to avoid repetition.

[0130] Optionally, such as Figure 3 As shown, this application embodiment also provides an electronic device 100, including a processor 101, a memory 102, and a program or instructions stored in the memory 102 and executable on the processor 101. When the program or instructions are executed by the processor 101, they implement the various steps of any of the above-described audio processing method embodiments and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0131] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0132] Figure 4 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of this application.

[0133] The electronic device 1000 includes, but is not limited to, components such as: radio frequency unit 1001, network module 1002, audio output unit 1003, input unit 1004, sensor 1005, display unit 1006, user input unit 1007, interface unit 1008, memory 1009, and processor 1010.

[0134] Those skilled in the art will understand that the electronic device 1000 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 1010 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 4 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0135] The processor 1010 is configured to: divide the first audio file into N1 segments (where N1 is a positive integer and N1 > 1) based on the feature information of the first audio file; perform a first dimensionality reduction on the feature information of a target segment (where the target segment is one of the N1 segments); fuse the feature information after the first dimensionality reduction; perform a second dimensionality reduction on the fused feature information; obtain the processed feature information of the N1 segments respectively; and determine N2 annotation information of the first audio file (where N2 is a positive integer) based on the processed feature information of the N1 segments.

[0136] Thus, in the embodiments of this application, feature information of a first audio file is obtained, and the first audio file is divided into multiple segments based on the obtained feature information, thereby transforming the first audio file into a segmented structured file for subsequent data processing. In the subsequent processing, each segment is processed separately, including three steps: dimensionality reduction, fusion, and dimensionality reduction again. Finally, based on each processed segment, an aggregation matrix representing the features of the first audio file is output. Further, based on the output aggregation matrix, at least one annotation information of the first audio file is determined. It is evident that the feature processing of the first audio file results in a high information density in the final aggregation matrix, capable of expressing both the features of individual parts of the first audio file and the overall features of the first audio file, thereby achieving a high accuracy rate in intelligently determining the annotation information of the first audio file based on the aggregation matrix.

[0137] Optionally, the processor 1010 is further configured to: obtain N3 sections based on the feature information of the first audio file, where N3 is a positive integer and N3 > 1; form a sliding window feature matrix from the feature information of N4 consecutive sections, where N4 is a positive integer and N4 < N3, and the feature information of one section is used to form at least one sliding window feature matrix; compare the similarity between two adjacent sliding window feature matrices; and divide the first audio file into N1 segments based on the matrix boundaries corresponding to the first N5 minimum similarities, where N5 is a positive integer.

[0138] Optionally, the processor 1010 is further configured to acquire a first waveform signal corresponding to the first audio file; and divide the first audio file into N3 sections according to the peak interval change information in the first waveform signal.

[0139] Optionally, the processor 1010 is further configured to: determine a target feature matrix in the target segment, wherein the target feature matrix is ​​the sliding window feature matrix that appears most frequently in the target segment; delete a first feature matrix in the target segment, wherein the correlation between the first feature matrix and the target feature matrix is ​​less than a preset value; and obtain the weighted values ​​of the feature information in the other feature matrices in the target segment besides the first feature matrix.

[0140] Optionally, the processor 1010 is further configured to form N6 feature groups based on the feature information of the target fragment, wherein the feature information contained in a feature group belongs to the same category, and N6 is a positive integer.

[0141] Optionally, the processor 1010 is further configured to, when the feature information after processing the N1 segments matches the feature information after processing the preset audio file, obtain a target feature group in the preset audio file, wherein the frequency of the target feature group appearing in the preset audio file satisfies a preset condition; and determine N2 annotation information of the first audio file according to the preset tag information corresponding to the target feature group.

[0142] In summary, this application obtains the classification features of audio files by using statistical principles on the basic features of structured data; in addition, it uses deep learning methods to obtain latent features that cannot be obtained directly, and combines experience or natural language processing to summarize some implicit style feature information contained in the audio files. By combining these two types of features, the final classification of audio files is obtained for audio file annotation.

[0143] Compared to existing technologies, the annotation of audio files in this application is not dependent on the knowledge and reserves of the annotation engineer. Therefore, there will be no omissions, errors, or multiple annotations due to the annotation engineer's own misunderstanding or mistakes, resulting in a higher annotation accuracy. In addition, the annotation of audio files in this application combines machine learning and rule matching and other methods to realize machine recognition of audio voiceprint features, thereby making the classification and annotation of audio materials faster.

[0144] It should be understood that, in this embodiment, the input unit 1004 may include a graphics processing unit (GPU) 10041 and a microphone 10042. The GPU 10041 processes image data of still images or video images obtained by an image capture device (such as a camera) in video image capture mode or image capture mode. The display unit 1006 may include a display panel 10061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, etc. The user input unit 1007 includes at least one of a touch panel 10071 and other input devices 10072. The touch panel 10071 is also called a touch screen. The touch panel 10071 may include a touch detection device and a touch controller. Other input devices 10072 may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, power buttons, etc.), trackballs, mice, and joysticks, which will not be described in detail here. The memory 1009 can be used to store software programs and various data, including but not limited to applications and motion systems. Processor 1010 may integrate an application processor and a modem processor. The application processor mainly handles the action system, user page, and applications, while the modem processor mainly handles wireless communication. It is understood that the modem processor may also not be integrated into processor 1010.

[0145] The memory 1009 can be used to store software programs and various data. The memory 1009 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 1009 may include volatile memory or non-volatile memory, or it may include both volatile and non-volatile memory. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 1009 in this embodiment includes, but is not limited to, these and any other suitable types of memory.

[0146] The processor 1010 may include one or more processing units; optionally, the processor 1010 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into the processor 1010.

[0147] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described audio processing method embodiments and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0148] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0149] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described audio processing method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0150] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0151] This application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of the audio processing method embodiments described above, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0152] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0153] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0154] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. An audio processing method, characterized by, The method comprises: According to the feature information of the first audio file, N3 sections are obtained, N3 is a positive integer, and N3>1; The feature information of the continuous N4 sections is used to form a sliding window feature matrix, N4 is a positive integer, and N4 The similarity between two adjacent sliding window feature matrices is compared; According to the matrix boundary corresponding to the N5 smallest similarities, the first audio file is divided into N1 segments, N5 is a positive integer, and N1>1; The feature information of the target segment is subjected to first dimension reduction, and the target segment is one of the N1 segments; The feature information after the first dimension reduction is fused; The fused feature information is subjected to second dimension reduction; The feature information of the N1 segments after processing is obtained, and an aggregated matrix representing the features of the first audio file is output according to the feature information of the N1 segments after processing, and N2 label information of the first audio file is determined according to the aggregated matrix, N2 is a positive integer; The method further comprises:

2. The method of claim 1, wherein, The feature information of the first audio file corresponding to the first waveform signal is obtained; According to the wave peak interval change information in the first waveform signal, the first audio file is divided into N3 sections. Before the feature information of the target segment is subjected to first dimension reduction, the method further comprises:

3. The method of claim 1, wherein, According to the feature information of the target segment, N6 feature groups are established, and the feature information included in one feature group belongs to the same category, N6 is a positive integer. According to the feature information of the N1 segments after processing, the N2 label information of the first audio file is determined, comprising:

4. The method of claim 3, wherein, In the case that the feature information of the N1 segments after processing matches the feature information of a preset audio file after processing, a target feature group in the preset audio file is obtained, and the frequency of the target feature group in the preset audio file meets a preset condition; According to the preset label information corresponding to the target feature group, the N2 label information of the first audio file is determined. The device comprises:

5. An audio processing apparatus, characterized by comprising: A first obtaining unit is configured to obtain N3 sections according to the feature information of a first audio file, N3 is a positive integer, and N3>1; A forming unit is configured to form a sliding window feature matrix from the feature information of the continuous N4 sections, N4 is a positive integer, and N4 A comparison unit is configured to compare the similarity between two adjacent sliding window feature matrices. ​ The dividing unit is configured to divide the first audio file into N1 segments according to a matrix boundary corresponding to the N5 smallest similarities, N5 being a positive integer and N1>1; The first dimension reduction module is configured to perform first dimension reduction on the feature information of the target segment, the target segment being one of the N1 segments; The fusion module is configured to fuse the feature information after the first dimension reduction; The second dimension reduction module is configured to perform second dimension reduction on the fused feature information; The determining module is configured to obtain the feature information processed by the N1 segments respectively, and output an aggregated matrix representing the feature of the first audio file according to the feature information processed by the N1 segments, and determine N2 pieces of annotation information of the first audio file according to the aggregated matrix, N2 being a positive integer; The fusion module comprises: The first determining unit is configured to determine a target feature matrix in the target segment, the target feature matrix being a sliding window feature matrix with the highest frequency of occurrence in the target segment; The deleting unit is configured to delete a first feature matrix in the target segment, the correlation between the first feature matrix and the target feature matrix being less than a preset value; The second obtaining unit is configured to obtain a weighted value of the feature information in other feature matrices in the target segment except the first feature matrix.

6. The apparatus of claim 5, wherein, The first obtaining unit comprises: The obtaining subunit is configured to obtain a first waveform signal corresponding to the first audio file; The dividing subunit is configured to divide the first audio file into N3 measures according to the wave peak interval change information in the first waveform signal.

7. The apparatus of claim 5, wherein, The device further comprises: The assembling module is configured to assemble N6 feature groups according to the feature information of the target segment, one feature group containing feature information belonging to the same category, N6 being a positive integer.

8. The apparatus of claim 7, wherein, The determining module comprises: The third obtaining unit is configured to obtain a target feature group in a preset audio file in a case where the feature information processed by the N1 segments matches the feature information processed by the preset audio file, the frequency of occurrence of the target feature group in the preset audio file satisfying a preset condition; The second determining unit is configured to determine N2 pieces of annotation information of the first audio file according to preset label information corresponding to the target feature group.

9. An electronic device, comprising: The device comprises a processor and a memory, the memory storing programs or instructions executable on the processor, the programs or instructions being executed by the processor to implement the steps of the audio processing method according to any one of claims 1-4.

10. A readable storage medium, characterized by, The readable storage medium stores programs or instructions, the programs or instructions being executed by the processor to implement the steps of the audio processing method according to any one of claims 1-4.

Images