Audio processing method

Abstract

The embodiment of the invention provides an audio processing method, and the method comprises the steps: carrying out the feature extraction of a target audio, obtaining the target audio global feature of the target audio, carrying out the feature extraction of a standard audio corresponding to the target audio, and obtaining the standard audio global feature of the standard audio, the target audio global feature is a feature representing global audio change of the target audio, and the standard audio global feature is a feature representing global audio change of the standard audio; performing at least two types of audio evaluation processing by using the target audio global feature and the standard audio global feature to obtain at least two types of audio evaluation indexes; and based on the at least two types of audio evaluation indexes, determining an audio processing result of the target audio.

Description

Technical Field

[0001] The embodiments of this specification relate to the field of computer technology, and more particularly to an audio processing method. One or more embodiments of this specification also relate to a computing device, a computer-readable storage medium, and a computer program product. Background Art

[0002] With the continuous development of computer technology, audio data has also been widely used in various service scenarios; in this process, the evaluation and processing of audio data is often involved.

[0003] In the current process of evaluating audio data, manual evaluation is used to evaluate and process the audio data. However, manual evaluation can lead to inaccurate evaluation results and low evaluation efficiency. Therefore, how to evaluate audio data efficiently and accurately becomes a technical problem that needs to be solved. Summary of the invention

[0004] In view of this, an embodiment of the present specification provides an audio processing method. One or more embodiments of the present specification also relate to an audio processing device, a computing device, a computer-readable storage medium and a computer program product to solve the technical defects existing in the prior art.

[0005] According to a first aspect of an embodiment of this specification, there is provided an audio processing method, including: Performing feature extraction on the target audio to obtain a target audio global feature of the target audio, and performing feature extraction on the standard audio corresponding to the target audio to obtain a standard audio global feature of the standard audio, wherein the target audio global feature is a feature representing a global audio change of the target audio, and the standard audio global feature is a feature representing a global audio change of the standard audio; Perform at least two types of audio evaluation processing using the target audio global feature and the standard audio global feature to obtain at least two types of audio evaluation indicators; Based on the at least two types of audio evaluation indicators, an audio processing result of the target audio is determined.

[0006] According to a second aspect of an embodiment of this specification, there is provided an audio processing device, including: A feature extraction module is configured to perform feature extraction on a target audio to obtain a target audio global feature of the target audio, and perform feature extraction on a standard audio corresponding to the target audio to obtain a standard audio global feature of the standard audio, wherein the target audio global feature is a feature representing a global audio change of the target audio, and the standard audio global feature is a feature representing a global audio change of the standard audio; An audio evaluation module, configured to perform at least two types of audio evaluation processing using the target audio global feature and the standard audio global feature to obtain at least two types of audio evaluation indicators; The result determination module is configured to determine the audio processing result of the target audio based on the at least two types of audio evaluation indicators.

[0007] According to a third aspect of an embodiment of this specification, a computing device is provided, including: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer programs / instructions are executed by the processor, the steps of the above-mentioned audio processing method are implemented.

[0008] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores a computer program / instruction, and when the computer program / instruction is executed by a processor, the steps of the above-mentioned audio processing method are implemented.

[0009] According to a fifth aspect of the embodiments of this specification, a computer program product is provided, including a computer program / instruction, which implements the steps of the above-mentioned audio processing method when executed by a processor.

[0010] The audio processing method provided by one or more embodiments of the present specification extracts the target audio global features of the target audio and the standard audio global features of the standard audio during the process of evaluating the audio; the target audio global features can accurately represent the global audio changes of the target audio, and the standard audio global features can also accurately represent the global audio changes of the standard audio; and, the target audio global features and the standard audio global features are used to perform a variety of audio evaluation processes, and the audio data is quantifiably evaluated from multiple dimensions, thereby obtaining multiple accurate and comprehensive audio evaluation indicators; finally, the audio processing results are accurately determined based on the multiple audio evaluation indicators; the problem that manual evaluation methods may lead to inaccurate evaluation results is avoided, and at the same time, multiple audio evaluation indicators are determined by a quantifiably audio evaluation method, thereby avoiding the problem that manual evaluation methods may lead to low evaluation efficiency, thereby achieving efficient and accurate evaluation of audio data. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 is an application diagram of an audio processing method provided by an embodiment of this specification; Figure 2 is a flow chart of an audio processing method provided by an embodiment of this specification; Figure 3is a schematic diagram of a music score in an audio processing method provided by an embodiment of the present specification; Figure 4 is a schematic diagram of an onset index in an audio processing method provided by an embodiment of the present specification; Figure 5 is a schematic diagram of a mapping relationship in an audio processing method provided by an embodiment of this specification; Figure 6 is a processing flow chart of an audio processing method provided by an embodiment of this specification; Figure 7 is a structural schematic diagram of an audio processing device provided by an embodiment of this specification; Figure 8 It is a structural block diagram of a computing device provided by an embodiment of this specification. DETAILED DESCRIPTION

[0012] Many specific details are described in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of this specification, so this specification is not limited to the specific implementation disclosed below.

[0013] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0014] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0015] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0016] In one or more embodiments of this specification, a large model refers to a deep learning model with large-scale model parameters, which usually contains hundreds of millions, tens of billions, hundreds of billions, trillions, or even more than 10 trillion model parameters. A large model can also be called a foundation model / foundation model. The large model is pre-trained with large-scale unlabeled corpus to produce a pre-trained model with more than 100 million parameters. This model can adapt to a wide range of downstream tasks, and the model has good generalization ability, such as a large-scale language model (LLM), a multi-modal pre-training model, etc.

[0017] When the big model is used in practice, only a small number of samples are needed to fine-tune the pre-trained model and it can be applied to different tasks. The big model can be widely used in natural language processing (NLP), computer vision and other fields. Specifically, it can be applied to computer vision tasks such as visual question answering (VQA), image caption (IC), image generation, as well as natural language processing tasks such as text-based sentiment classification, text summary generation, and machine translation. The main application scenarios of the big model include digital assistants, intelligent robots, search, online education, office software, e-commerce, intelligent design, etc.

[0018] First, the terms and definitions used in one or more embodiments of this specification are explained.

[0019] Standard music: refers to music that is performed with high quality and accuracy approved by professional musicians or authoritative institutions. This standard music can be used to evaluate the music played by students.

[0020] Tempo: refers to the speed or rhythm of a piece of music, usually expressed in beats per minute (BPM). For example, a song with a Tempo of 120 BPM means there are 120 beats per minute.

[0021] Beats: refers to the basic time unit that constitutes music. It is the regular pulsation in music. You can imagine the beat as the "heartbeat" of music.

[0022] Chroma: is an audio feature that describes the tonal content of music, based on the idea of ​​representing musical notes as colors on a two-dimensional plane, where the vertical axis represents pitch and the horizontal axis represents time.

[0023] chroma_cens: is an audio feature mainly used to represent pitch changes in audio.

[0024] librosa: is a library dedicated to audio and music signal analysis.

[0025] dtaidistance: is a Python library for time series distance calculation, which provides a variety of time series distance calculation methods.

[0026] With the continuous development of computer technology, audio data has also been widely used in various service scenarios; in this process, audio data is often involved in the evaluation and processing. In the current process of evaluating audio data, manual evaluation is used to evaluate and process audio data, which can lead to inaccurate evaluation results and low evaluation efficiency. For example, in the field of music teaching, it is necessary to rely on the experience of teachers or teaching assistants to evaluate and score the music performance recordings submitted by students; in this process, for similar student assignments, it is completely dependent on the teacher's subjective scoring. Due to the different order of student assignments or the teacher's mental state, inconsistent evaluations may occur; and it takes a lot of teacher time. For example, a 10-minute piece of music and 30 student assignments take 5 hours to listen to.

[0027] Based on this, in this specification, an audio processing method is provided. One or more embodiments of this specification also involve an audio processing device, a computing device, a computer-readable storage medium and a computer program product, which are described in detail one by one in the following embodiments.

[0028] See also Figure 1 , Figure 1 A schematic diagram of an application of an audio processing method provided according to an embodiment of the present specification is shown. Figure 1In the application scenario shown, the audio processing method in this specification can be applied in a server 10, and the server 10 can be connected to one or more client devices 20 through a local area network connection, a wide area network connection, an Internet connection, or other types of data networks. The client device 20 here can include but is not limited to: a smart phone, a tablet computer, a laptop computer, a PDA, a personal computer, a smart home device, a car device, etc. The client device 20 can interact with the user through a graphical user interface to implement the method provided in the embodiment of this specification.

[0029] In the embodiment of the present specification, the system composed of the client device 20 and the server 10 can execute the following steps: the client device 20 executes the step of sending the student performance version mp3 to the server 10; the server 10 executes: feature extraction, multi-dimensional calculation, and multi-dimensional evaluation steps; wherein, feature extraction refers to: using the feature extraction method of extracting chroma_cens, feature extraction is performed on the standard mp3 and the student performance version mp3, and the chroma_cens of the standard mp3 and the student performance version mp3 is obtained; multi-dimensional calculation refers to: multi-dimensional calculation is performed on the audio data (student performance version mp3, standard mp3) from multiple dimensions to calculate different evaluation index values; multi-dimensional evaluation refers to: based on multiple evaluation index values, multi-dimensional evaluation is performed on the student performance version mp3 to obtain equal parts of the student performance version mp3; and the equal parts are sent to the client device 20.

[0030] See also Figure 2 , Figure 2 A flowchart of an audio processing method provided according to an embodiment of the present specification is shown, which specifically includes the following steps.

[0031] Step 202: perform feature extraction on the target audio to obtain the target audio global feature of the target audio, and perform feature extraction on the standard audio corresponding to the target audio to obtain the standard audio global feature of the standard audio, wherein the target audio global feature is a feature representing the global audio change of the target audio, and the standard audio global feature is a feature representing the global audio change of the standard audio.

[0032] Among them, the target audio can be understood as audio data that needs to be evaluated and processed, and the standard audio can be understood as audio data for evaluating the target audio; when the audio processing method of this specification is applied to different scenarios, the target audio and the standard audio are also different; for example, when the audio processing method is applied to a music scene, the target audio can be understood as the target music obtained by playing or singing, and the standard audio can be understood as the standard music used to evaluate the target audio for musical performance; for another example, when the audio processing method is applied to a pronunciation teaching scene, the target audio can be understood as the human voice audio (for example, speech audio, foreign language voice, Chinese voice, etc.) emitted by the student, and the standard audio can be understood as the standard pronunciation audio used for pronunciation evaluation of the human voice audio (for example, standard speech audio, standard foreign language voice, standard Chinese voice, etc. that meet the pronunciation standards); for another example, when the audio processing method is applied to a medical scene, the target audio can be understood as medical audio collected based on the patient (for example, auscultation, etc.); the standard audio can be understood as standard medical audio used for medical diagnosis and evaluation of medical audio (for example, auscultation that meets medical standards, etc.). For another example, when the audio processing method is applied to a detection scenario, the target audio can be understood as the detection sonar in the detection process; the standard audio can be understood as the standard sonar used to determine whether the detection sonar is abnormal; it should be noted that the detection scenarios in which sonar can be applied include but are not limited to: 1. Fisheries: used to detect the location and number of fish schools and improve fishing efficiency (such as the fishery sonar in the link you provided). 2. Navigation: used for underwater navigation, obstacle avoidance and depth measurement. 3. Underwater engineering: used for the laying and maintenance of underwater pipelines and cables. 4. Marine resource exploration: used for seabed geological exploration and oil and gas exploration. 5. Marine biological research: used to observe and study the behavior and distribution of marine organisms. 6. Marine geological research: used for the investigation and research of seabed topography.

[0033] The audio global feature can be understood as a feature used to represent the overall audio change or global audio change of audio data. The target audio global feature is the audio global feature of the target audio, and the standard audio global feature is the audio global feature of the standard audio.

[0034] The global audio feature may be Chroma-CENS, which is a stable feature based on Chroma (Chroma Energy Normalized Statistics, CENS), which improves robustness to noise and speed changes through time smoothing and normalization. The advantages of Chroma-CENS are: it is very robust to noise and speed changes in audio; it is suitable for the overall description of melodies and chords, rather than local changes. The applicable scenarios of Chroma-CENS include: comparative analysis of speed-varying audio (such as comparison between live and studio versions), music matching and retrieval, etc.; that is, the audio processing method of this specification can perform comparative analysis tasks, music matching and retrieval tasks of speed-varying audio based on Chroma-CENS.

[0035] The global audio feature may also be: an auditory feature extracted based on the human auditory system (MelSpectrogram), a compressed feature obtained by performing audio compression on the auditory feature (mfcc), a pitch feature (Chroma-STFT), or a high frequency resolution (Chroma-CQT).

[0036] Among them, the Mel Spectrogram can generate a spectrogram through Fourier transform of the audio signal, and then convert the frequency axis into the Mel scale (simulating the auditory characteristics of the human ear). The advantages of the Mel Spectrogram are: providing comprehensive spectrum information, especially the low-frequency part; it is friendly to human hearing perception and is widely used in speech and music processing. The applicable scenarios of the Mel Spectrogram include: speech recognition and music classification, as the basis for other feature extraction (such as MFCC); that is, the audio processing method of this specification can perform speech recognition and music classification tasks based on the Mel Spectrogram.

[0037] Among them, MFCC calculates Mel-Frequency Cepstral Coefficients (MFCC) from Mel Spectrogram, which further compresses the Mel spectrum information. The advantages of MFCC are: extracting compact features, facilitating modeling, imitating human auditory perception, being effective for speech recognition and classification, and low-dimensional representation is more suitable for calculation and storage. The applicable scenarios of this MFCC include: speech processing tasks (such as speech recognition, sentiment analysis), and music classification tasks. In other words, the audio processing method of this specification can perform speech processing tasks and music classification tasks based on MFCC.

[0038] Among them, Chroma-STFT can be understood as a pitch feature. The Chroma-STFT is a pitch feature extracted based on short-time Fourier transform (STFT), and calculates the energy distribution of audio in 12 semitones (C, C#, D, etc.) per frame. The advantages of the Chroma-STFT are: it is sensitive to pitch and tonality, can recognize musical characteristics such as scales and chords, and is easy to capture melody and chord changes in music. It can be applied to scenarios such as music tonality analysis, chord recognition, and music classification tasks; that is, the audio processing method of this specification can perform music tonality analysis, chord recognition, and music classification tasks based on Chroma-STFT.

[0039] Among them, Chroma-CQT is a Chroma feature calculated using Continuous Wavelet Transform (Constant-Q Transform, CQT). CQT provides higher frequency resolution. The advantages of Chroma-CQT are: higher frequency resolution, especially suitable for the analysis of long-lasting sounds in music; more suitable for processing music with stable tuning. It can be applied to: scenarios for accurate analysis of melody and tonality, and also to processing tasks of audio with long note duration, such as classical music. In other words, the audio processing method of this specification can perform tasks of analyzing melody and tonality and audio processing tasks of long note duration based on Chroma-CQT.

[0040] Based on the above content, it can be seen that this method can select appropriate features according to specific task requirements, and can also combine multiple features to obtain a more comprehensive audio representation.

[0041] Taking the application of the audio processing method provided in this specification in the music performance evaluation scenario as an example, the audio processing method is explained, wherein the target audio can be a student performance version of the mp3 (that is, the work that needs to be evaluated and scored), the standard audio is a standard mp3 (the standard mp3 can be produced without relying on midi files, but directly obtains and uses a certain mp3 file recognized as a standard as the standard mp3), and the standard audio global features and the target audio global features are chroma_cens.

[0042] Based on this, this method can obtain the student version of the mp3 and the standard mp3, and use the feature extraction method of extracting chroma_cens to extract features of the student version of the mp3 and the standard mp3, and obtain the chroma_cens of the student version of the mp3 and the chroma_cens of the standard mp3. Among them, the extraction operation of the audio global feature is implemented using the librosa library.

[0043] In one or more embodiments provided in this specification, before extracting features from the standard audio corresponding to the target audio to obtain the standard audio global features of the standard audio, the process further includes: Using the music score processing model, extracting audio information from the standard music score to obtain standard audio information in the standard music score; A standard audio information file is generated using the standard audio information, and the standard audio is generated based on the standard audio information file.

[0044] The standard musical score can be understood as the musical score corresponding to the standard audio. The standard audio information can be understood as the information describing the content of the standard audio, for example, the standard audio information can be information such as the key, beat, and notes. The standard audio information file can be understood as a file used to make the standard audio, for example, the standard audio information file can be a MIDI type file.

[0045] The music score processing model can be understood as a music score recognition model that recognizes standard audio information in standard music scores. The music score processing model can be a large model, a deep learning model, etc.

[0046] Using the above example, this method needs to determine the standard audio before extracting features from the standard audio. The specific method can also be: 1. Determine the music score.

[0047] This step is the raw data preparation stage, and music score files in PDF and image formats can be used. Figure 3 As shown, Figure 3 It is a schematic diagram of a music score in an audio processing method provided by an embodiment of this specification.

[0048] 2. Use a music score recognition model based on deep learning (i.e., music score processing model) to extract information such as key, beat, and notes (i.e., standard audio information) from the music score; 3. Use this information to generate a MIDI type file, and use the MIDI file (i.e. MIDI type file) to produce a standard MP3 file (i.e. standard audio).

[0049] Based on the above steps, it can be seen that the method uses standard music scores to generate standard audio, thereby ensuring the accuracy of the standard audio and further improving the accuracy of audio evaluation.

[0050] In one or more embodiments provided in this specification, before extracting features from the standard audio corresponding to the target audio to obtain the standard audio global features of the standard audio, the process further includes: Acquire at least two audios to be processed, and use an audio processing model to extract audio information from the at least two audios to be processed to obtain at least two audio information files; The at least two audio information files are merged to obtain a standard audio information file, and the standard audio is generated based on the standard audio information file.

[0051] Among them, the audio processing model can be understood as a model for identifying audio information in the audio to be processed. For example, the audio processing model can be a large model, a deep learning model, etc.

[0052] Using the above example, this method needs to determine the standard audio before extracting features from the standard audio. The specific method can also be: 1. Get mp3 files of performances by different musicians.

[0053] 2. Use a deep learning-based model to convert multiple mp3 files into midi files (i.e. at least two audio information files).

[0054] 3. Merge multiple MIDI files into a standard MIDI file (i.e., standard audio information file) through manual editing or application.

[0055] 4. Use standard midi files to create standard mp3 files (i.e. standard audio).

[0056] It should be noted that if a musician's performance version mp3 file is obtained, a deep learning-based model is used to directly use the performance version mp3 file to generate a standard midi type file (i.e., a standard audio information file), and use the standard midi file to produce a standard mp3 file (i.e., standard audio).

[0057] In addition, in one or more embodiments provided in this specification, it is also possible to directly search for publicly authorized MIDI files (i.e., standard audio information files) on the Internet, and use the publicly authorized MIDI files to produce standard MP3 files; or, manually edit MIDI files (i.e., standard audio information files), and use the manually produced MIDI files to produce standard MP3 files.

[0058] Based on the above content, it can be seen that the method provides multiple ways to generate standard audio, which improves the applicability of the method.

[0059] Step 204: Perform at least two types of audio evaluation processing using the target audio global features and the standard audio global features to obtain at least two types of audio evaluation indicators.

[0060] The audio evaluation process may be understood as a processing operation for evaluating or assessing the audio data, and the audio evaluation index may be understood as an index value obtained after the audio evaluation process is performed on the audio data.

[0061] In one or more embodiments provided in this specification, the at least two types of audio evaluation processing include audio rhythm analysis processing, note position recognition processing, similarity analysis processing, and empty sound position recognition processing; The step of performing at least two types of audio evaluation processing using the target audio global feature and the standard audio global feature to obtain at least two types of audio evaluation indicators includes steps 1 and 2: Step 1: Based on the global features of the target audio, perform the audio rhythm analysis processing, the note position recognition processing, the similarity analysis processing, and the empty sound position recognition processing to obtain at least two types of target audio evaluation indicators of the target audio.

[0062] Specifically, based on the target audio global features, the audio rhythm analysis process, the note position recognition process, the similarity analysis process, and the empty sound position recognition process are performed to obtain at least two types of target audio evaluation indicators of the target audio, including: Based on the target audio global feature, performing the audio rhythm analysis process on the target audio to obtain audio rhythm information of the target audio; Based on the target audio global feature, perform the note position recognition process on the target audio to obtain the note position information of the target audio, wherein the note position information includes the start time position and the end time position of at least one note in the target audio; Performing the similarity analysis process based on the target audio global feature and the standard audio global feature to obtain audio similarity information between the target audio and the standard audio, wherein the audio similarity information includes an association relationship between audio similarity and audio segments; Based on the target audio global feature, perform the empty sound position identification process on the target audio to obtain empty sound position information of the target audio, wherein the empty sound position information includes a start time position and an end time position of at least one empty sound in the target audio; The audio rhythm information, the note position information, the audio similarity information and the empty sound position information are determined as the at least two types of target audio evaluation indicators.

[0063] The audio rhythm analysis process may be understood as an operation of analyzing the tempo value and beat value of the audio data (target audio or standard audio); the audio rhythm information may be understood as the tempo value and the beat value.

[0064] The note position recognition process can be understood as the operation of identifying the note position of audio data (target audio or standard audio); the note position information can be the onset index of the MP3 file; the onset index can refer to the starting time point of the note; the onset index in this method includes a forward onset index (i.e., the starting time position) and a reverse onset index (the ending time position). Figure 4 is a schematic diagram of an onset index in an audio processing method provided in an embodiment of the present specification; based on Figure 4 It can be seen that Figure 4 The black dotted line in the middle represents the forward onset index, which is the starting position (starting time position) of each note heard when the audio is played forward; the black solid line represents the reverse onset index, which is to play the audio in reverse and obtain the starting position of each note, which can be understood as the ending position (ending time position) of each note when the audio is played normally. The onset index (starting point index) of the MP3 file is very important for understanding the temporal structure and dynamic changes of music. Onset refers to the moment when a sound event begins, such as a note, a drum hit, or any significant sound change. In audio processing and analysis, onset is the key to identifying musical rhythm patterns, beat tracking, and automatically marking musical passages.

[0065] The similarity analysis process can be understood as an operation of analyzing the similarity and association relationship between the target audio or the standard audio, and the audio similarity can be understood as a number indicating the degree of similarity between the target audio or the standard audio, for example, the audio similarity can be distance similarity, cosine similarity, etc. The audio segment association relationship can be understood as a mapping relationship between multiple target audio segments of the target audio and multiple standard audio segments of the standard audio.

[0066] The blank sound position information can be understood as an operation of identifying the note positions of audio data (target audio or standard audio); the blank sound position information can be the time position of the blank sound in the MP3 file, for example, the starting time position and the ending time position of the blank sound can be determined based on the starting time position and the ending time position of the note; after determining the starting time position and the ending time position of the note, the time position before the starting time position of the note can be determined as the ending time position of the blank sound, and the time position after the ending time position of the note can be determined as the starting time position of the blank sound.

[0067] It should be noted that the time position (starting time position, ending time position) in one or more embodiments of the present specification may be a time point.

[0068] Using the above example, based on the feature vector extracted in the previous step, this method can perform multi-dimensional calculations on the audio data from multiple dimensions and calculate different evaluation index values, thereby realizing the evaluation of the audio data; the evaluation of the student performance version of the mp3 specifically includes the following four steps: 1. Based on the chroma_cens of the student performance version of the MP3, calculate the tempo and beats values ​​of the student performance version of the MP3.

[0069] 2. Based on the chroma_cens of the student performance version of the mp3, calculate the onset index of the student performance version of the mp3, that is, the starting time point and the ending time point of the note.

[0070] 3. Use the Dynamic Time Warping (DTW) algorithm to calculate the distance similarity and mapping relationship between the student performance version of the mp3 and the standard mp3 based on the chroma_cens of the student performance version of the mp3 and the standard mp3.

[0071] Specifically, this method can use the dynamic time warping algorithm to calculate the similarity of the chroma_cens of two mp3s, so as to obtain the distance similarity and mapping relationship between the two mp3s; Figure 5 is a schematic diagram of a mapping relationship in an audio processing method provided by an embodiment of this specification, Figure 5 The solid line in the figure is the visualization of the correspondence (i.e., mapping relationship) between the two mp3 audio segments during the DTW calculation process. The DTW calculation result also returns a distance score (i.e., distance similarity) to indicate the similarity between the two audio segments. The smaller the distance score, the more similar the two audio segments are.

[0072] 4. Based on the set threshold, extract the time points of the empty sound intervals in the student performance mp3.

[0073] Specifically, the method extracts the "empty sound interval" (i.e., the period of silence or near silence) in the student performance MP3 file by setting a threshold to determine whether the amplitude of the audio signal is lower than a certain level. If the amplitude of the audio signal continues to be lower than the threshold for a period of time, the audio signal during this period can be considered as a "empty sound interval", and the start time point and end time point of the empty sound interval are determined.

[0074] Based on the above content, it can be seen that this method can perform multi-dimensional calculations on audio data from multiple dimensions and calculate different evaluation index values, thereby ensuring the comprehensiveness and accuracy of the evaluation.

[0075] Step 2: Based on the global features of the standard audio, perform the audio rhythm analysis processing, the note position recognition processing, the similarity analysis processing, and the empty sound position recognition processing to obtain at least two types of standard audio evaluation indicators for the standard audio.

[0076] Specifically, the audio rhythm analysis process, the note position recognition process, the similarity analysis process, and the empty sound position recognition process are performed based on the standard audio global features to obtain at least two types of standard audio evaluation indicators of the standard audio, including: Based on the standard audio global feature, performing the audio rhythm analysis process on the standard audio to obtain audio rhythm information of the standard audio; Based on the global features of the standard audio, the note position recognition process is performed on the standard audio to obtain the note position information of the standard audio, wherein the note position information includes the start time position and the end time position of at least one note in the standard audio; Performing the similarity analysis process based on the target audio global feature and the standard audio global feature to obtain audio similarity information between the target audio and the standard audio, wherein the audio similarity information includes an association relationship between audio similarity and audio segments; Based on the global features of the standard audio, perform the empty sound position identification process on the standard audio to obtain empty sound position information of the standard audio, wherein the empty sound position information includes a start time position and an end time position of at least one empty sound in the standard audio; The audio rhythm information, the note position information, the audio similarity information and the empty sound position information are determined as the at least two types of standard audio evaluation indicators.

[0077] Using the above example, based on the feature vector extracted in the previous step, this method can perform multi-dimensional calculations on the audio data from multiple dimensions and calculate different evaluation index values, thereby realizing the evaluation of the audio data; the evaluation of the standard mp3 specifically includes the following four steps: 1. Based on the chroma_cens of the standard mp3, calculate the tempo and beats values ​​of the standard mp3.

[0078] 2. Based on the chroma_cens of the standard MP3, calculate the onset index of the standard MP3, that is, the start and end time points of the note.

[0079] 3. Use the Dynamic Time Warping algorithm to calculate the distance similarity and mapping relationship between the student performance version of the mp3 and the standard mp3 based on the chroma_cens of the student performance version of the mp3 and the standard mp3.

[0080] 4. Based on the set threshold, extract the time points of the empty sound interval in the standard mp3.

[0081] Specifically, this method extracts the "silent interval" (i.e., the period of silence or near silence) in the standard MP3 file by setting a threshold to determine whether the amplitude of the audio signal is lower than a certain level. If the amplitude of the audio signal continues to be lower than the threshold for a period of time, the audio signal during this period can be considered as a "silent interval", and the start time point and end time point of the silent interval are determined.

[0082] Based on the above content, it can be seen that this method can perform multi-dimensional calculations on audio data from multiple dimensions and calculate different evaluation index values, thereby ensuring the comprehensiveness and accuracy of the evaluation.

[0083] Step 206: Determine an audio processing result of the target audio based on the at least two types of audio evaluation indicators.

[0084] Among them, the audio processing result can be understood as the result of processing the target audio. For example, the audio processing result can be the audio score of the target audio, the abnormal audio segment in the target audio, the audio type obtained by classifying the target audio, etc., and no specific restrictions are made here.

[0085] In one or more embodiments provided in this specification, the target audio is target music, the standard audio is standard music corresponding to the target music, and the audio processing result is a music score; The step of determining the audio processing result of the target audio based on the at least two types of audio evaluation indicators includes: Determining indicator weights corresponding to the at least two types of audio evaluation indicators; The target music is scored using the indicator weights and the at least two types of audio evaluation indicators to obtain the music score of the target music.

[0086] Continuing with the above example, after performing a multi-dimensional evaluation on the student performance version mp3, a multi-dimensional evaluation can be performed on the student performance version mp3 to obtain the score (i.e., the music score) of the student performance version mp3; specifically, this method is based on the 4 indicator values ​​obtained from the above multi-dimensional evaluation and combines the weight ratio to give a score for the student performance version mp3.

[0087] For example, if the teacher has a standard version of a piece of music played by a musician, then the teacher can compare and calculate all the students' homework mp3s (i.e., the student performance mp3s) with the standard version mp3s one by one (including tempo, onset, DTW), and then use the relative scoring method to score the students; for example, the student's homework with the smallest DTW distance score (i.e., audio similarity) is selected as 95 points, and the largest DTW distance score is selected as 50 points, and then other students' homework is scored in proportion. You can also use the absolute scoring method to score students, for example, the student's homework with a DTW distance score of less than 20 is considered to have a score higher than 90 points, etc.

[0088] For another example, if the teacher has versions of the same piece of music played by multiple musicians, it is similar to the above example, except that each student's assignment will have multiple DTW distance scores. In this case, the teacher can choose to take the maximum or average method to calculate a DTW score as the student's final DTW score. Alternatively, the teacher can compare the student assignments separately according to each musician's version, and when grading the student assignments, the teacher will not give a specific score, but rather which musician the student's performance is more like.

[0089] It should be noted that the above example only uses the DTW distance score. The tempo score and onset position can also be used together with DTW to participate in the assignment scoring. The principle is similar to using DTW as the basis for scoring.

[0090] Moreover, the weight ratio (i.e., indicator weight) in this method can be designed according to the actual application scenario. For example, the distance similarity and mapping relationship in the multidimensional evaluation can be set so that the weight of the evaluation indicator is higher.

[0091] Subsequently, the final scores of the students can be ranked in the class after the ratings (i.e., music scores) of all student assignments (i.e., mp3s of student performances) are obtained, and then the final scores of the students are given.

[0092] Based on the above embodiments, it can be seen that in the scenario of music teaching, this method can give clear quantifiable indicators from multiple angles. By giving quantifiable indicators, it can assist teachers in evaluating student works and achieve objective scoring; and, on the basis of the quantifiable indicators, only the inconsistencies in the music can be listened to, or even the scores can be given directly based on the indicators without listening to the music, thereby saving the teacher's time.

[0093] In one or more embodiments provided in this specification, the audio processing result is an abnormal audio segment; The step of determining the audio processing result of the target audio based on the at least two types of audio evaluation indicators includes: Determine a plurality of target audio segments in the target audio and a plurality of standard audio segments in the standard audio; Determining an associated audio segment of each target audio segment from the plurality of standard audio segments based on the segment position information of the plurality of target audio segments and the segment position information of the plurality of standard audio segments; Determining, based on the at least two types of audio evaluation indicators, audio segment similarities between each target audio segment and the associated audio segment of each target audio segment; A target audio segment whose audio segment similarity is less than or equal to a preset similarity threshold is determined as an abnormal audio segment.

[0094] The audio segment can be understood as part of the audio in the audio data (target audio, standard audio); the target audio segment is the audio segment of the target audio; and the standard audio segment is the audio segment of the standard audio.

[0095] The segment position information may be understood as the position of the audio segment in the audio data. For example, the segment position information may be the time position or time point of the audio segment.

[0096] The associated audio segment may be understood as a standard audio segment having a corresponding relationship or a mapping relationship with the target audio segment.

[0097] The audio segment similarity can be understood as data representing the similarity between the target audio segment and the associated audio segment. For example, the audio segment similarity can be distance similarity, cosine similarity, DTW distance score, etc.

[0098] The preset similarity threshold can be set according to the actual application scenario and is not specifically limited here.

[0099] The abnormal audio segment can be understood as a target audio segment that is not similar to the associated audio segment.

[0100] Continuing with the above example, this method is not only applicable to the evaluation of music performances and song singing; it can also be applied to the evaluation of medical audio such as auscultation in medical treatment, or to the evaluation of sonar in detection scenarios. By quickly and accurately finding out where the differences (i.e., abnormal audio segments) are between two audio segments, it is possible to assist doctors in making diagnoses during the medical diagnosis process. Specifically, this method can determine multiple medical audio segments (i.e., target audio segments) in medical audio, and determine multiple standard medical audio segments (i.e., standard audio segments) in standard medical audio; determine a mapping relationship based on the time position (i.e., segment position information) of the medical audio segment and the standard medical audio segment, and based on the mapping relationship, determine an associated standard medical audio segment for each medical audio segment from multiple standard medical audio segments.

[0101] The audio segment similarity between each medical audio segment and the associated standard medical audio segment is calculated, and the medical audio segment whose audio segment similarity is less than or equal to the preset similarity threshold is determined as an abnormal medical audio segment (i.e., an abnormal audio segment), so as to quickly and accurately find the differences between the two audio segments, thereby assisting doctors in making a diagnosis.

[0102] The audio processing method provided by one or more embodiments of the present specification extracts the target audio global features of the target audio and the standard audio global features of the standard audio during the process of evaluating the audio; the target audio global features can accurately represent the global audio changes of the target audio, and the standard audio global features can also accurately represent the global audio changes of the standard audio; and, the target audio global features and the standard audio global features are used to perform a variety of audio evaluation processes, and the audio data is quantifiably evaluated from multiple dimensions, thereby obtaining multiple accurate and comprehensive audio evaluation indicators; finally, the audio processing results are accurately determined based on the multiple audio evaluation indicators; the problem that manual evaluation methods may lead to inaccurate evaluation results is avoided, and at the same time, multiple audio evaluation indicators are determined by a quantifiably audio evaluation method, thereby avoiding the problem that manual evaluation methods may lead to low evaluation efficiency, thereby achieving efficient and accurate evaluation of audio data.

[0103] The following combination Figure 6 , taking the application of the audio processing method provided in this specification in the music performance evaluation scenario as an example, the audio processing method is further described. Figure 6 A processing flow chart of an audio processing method provided by an embodiment of the present specification is shown, which specifically includes the following steps.

[0104] Step 602: Get the standard version mp3.

[0105] Before this method extracts features from the standard version of MP3, it is necessary to determine the standard version of MP3. The specific method can be: 1. Determine the music score.

[0106] This step is the raw data preparation stage, and music score files in PDF or image format can be used.

[0107] 2. Use a music score recognition model based on deep learning to extract information such as key, beat, and notes from the music score; 3. Use this information to generate a MIDI type file, and use the MIDI file to create a standard MP3 file.

[0108] In addition, you can also obtain standard audio by: 1. Get mp3 files of performances by different musicians.

[0109] 2. Use a deep learning based model to convert multiple mp3 files into midi files.

[0110] 3. Merge multiple midi files into a standard midi file through manual editing or application.

[0111] 4. Use standard midi files to create standard mp3 files.

[0112] It should be noted that if a musician's performance version mp3 file is obtained, a deep learning-based model is used to directly use the performance version mp3 file to generate a standard midi type file, and the standard midi file is used to produce a standard mp3 file.

[0113] In addition, the method can also directly search for publicly authorized midi files on the Internet, and use the publicly authorized midi files to produce standard MP3 files; or, use manual editing to produce midi files, and use the manually produced midi files to produce standard MP3 files.

[0114] Step 604: Obtain the student performance version mp3.

[0115] Specifically, obtain the student performance version mp3, which can be understood as a work that needs to be evaluated and scored.

[0116] Step 606: Feature extraction.

[0117] Specifically, a feature extraction method for extracting chroma_cens is used to extract features of the standard mp3 to obtain the chroma_cens of the standard mp3. The feature extraction operation is implemented using the librosa library.

[0118] Step 608: Feature extraction.

[0119] Specifically, a feature extraction method for extracting chroma_cens is used to extract features from the student performance version of the MP3 to obtain the chroma_cens of the standard MP3. The feature extraction operation is implemented using the librosa library.

[0120] Step 610: Multi-dimensional calculation.

[0121] Specifically, based on the feature vector extracted in the previous step, this method can perform multi-dimensional calculations on the audio data (student performance version mp3, standard mp3) from multiple dimensions, calculate different evaluation index values, and thus realize the evaluation of the audio data; the evaluation of the two mp3s specifically includes the following four steps: 1. Based on the chroma_cens of the student performance version of the mp3, calculate the tempo and beats values ​​of the student performance version of the mp3, as well as the tempo and beats values ​​of the standard mp3.

[0122] 2. Based on the chroma_cens of the student version of the mp3, calculate the onset index of the student version of the mp3 and the onset index of the standard mp3; The onset index refers to the start and end time points of a note.

[0123] 3. Use the Dynamic Time Warping (DTW) algorithm to calculate the distance similarity and mapping relationship between the student performance version of the mp3 and the standard mp3 based on the chroma_cens of the student performance version of the mp3 and the standard mp3.

[0124] Specifically, this method can use the dynamic time warping algorithm to calculate the similarity of the chroma_cens of two mp3s, so as to obtain the distance similarity and mapping relationship between the two mp3s; Figure 5 is a schematic diagram of a mapping relationship in an audio processing method provided by an embodiment of this specification, Figure 5 The solid line in the figure is the visualization of the correspondence (i.e., mapping relationship) between the two mp3 audio segments during the DTW calculation process. The DTW calculation result also returns a distance score (i.e., distance similarity) to indicate the similarity between the two audio segments. The smaller the distance score, the more similar the two audio segments are.

[0125] 4. Based on the set threshold, the time points of the empty sound intervals in the student performance version mp3 and the standard mp3 are extracted respectively.

[0126] Specifically, the method extracts the "silent interval" (i.e., the period of silence or near silence) in two MP3 files by setting a threshold to determine whether the amplitude of the audio signal is lower than a certain level. If the amplitude of the audio signal is continuously lower than the threshold for a period of time, the audio signal during this period can be considered as a "silent interval", and the start time point and end time point of the silent interval are determined.

[0127] Step 612: Multi-dimensional evaluation.

[0128] Specifically, after multi-dimensional evaluation of the student performance version mp3 and the standard version mp3, the student performance version mp3 can be multi-dimensionally evaluated to obtain equal scores of the student performance version mp3; specifically, this method gives a score for the student performance version mp3 based on the 4 indicator values ​​obtained from the above multi-dimensional evaluation and combined with the weight ratio.

[0129] For example, if the teacher has a standard version of a piece of music played by a musician, then the teacher can compare and calculate all the students' homework mp3s with the standard version mp3 one by one (including tempo, onset, DTW), and then use the relative scoring method to score the students; for example, the student with the smallest DTW distance score in the homework is selected as 95 points, and the largest DTW distance score is selected as 50 points, and then the other students' homework is scored in proportion. You can also use the absolute scoring method to score students, for example, the student's homework with a DTW distance score less than 20 is considered to have a score higher than 90 points, etc.

[0130] For another example, if the teacher has versions of the same piece of music played by multiple musicians, it is similar to the above example, except that each student's assignment will have multiple DTW distance scores. In this case, the teacher can choose to take the maximum or average method to calculate a DTW score as the student's final DTW score. Alternatively, the teacher can compare the student assignments separately according to each musician's version, and when grading the student assignments, the teacher will not give a specific score, but rather which musician the student's performance is more like.

[0131] It should be noted that the above example only uses the DTW distance score. The tempo score and onset position can also be used together with DTW to participate in the assignment scoring. The principle is similar to using DTW as the basis for scoring.

[0132] Moreover, the weight ratio in this method can be designed according to the actual application scenario. For example, the weight of evaluation indicators such as distance similarity and mapping relationship in multidimensional evaluation can be set higher.

[0133] Based on the above steps, it can be seen that this method provides a quantifiable music performance evaluation method; in the context of music teaching, this method can give clear quantifiable indicators from multiple angles, and by giving quantifiable indicators, it can assist teachers in evaluating student works and achieve objective scoring; and, based on the quantifiable indicators, only the inconsistencies in the music can be listened to, or even the scores can be given directly based on the indicators without listening to the music, thereby saving the teacher's time.

[0134] Corresponding to the above method embodiment, this specification also provides an audio processing device embodiment, Figure 7 FIG. 2 shows a schematic diagram of the structure of an audio processing device provided by an embodiment of the present specification. Figure 7 As shown, the device comprises: The feature extraction module 702 is configured to perform feature extraction on the target audio to obtain the target audio global feature of the target audio, and perform feature extraction on the standard audio corresponding to the target audio to obtain the standard audio global feature of the standard audio, wherein the target audio global feature is a feature representing a global audio change of the target audio, and the standard audio global feature is a feature representing a global audio change of the standard audio; The audio evaluation module 704 is configured to perform at least two types of audio evaluation processing using the target audio global feature and the standard audio global feature to obtain at least two types of audio evaluation indicators; The result determination module 706 is configured to determine the audio processing result of the target audio based on the at least two types of audio evaluation indicators.

[0135] Optionally, the at least two types of audio evaluation processing include audio rhythm analysis processing, note position recognition processing, similarity analysis processing, and empty sound position recognition processing; The audio evaluation module 704 is further configured to: Based on the target audio global features, the audio rhythm analysis process, the note position recognition process, the similarity analysis process, and the empty sound position recognition process are performed to obtain at least two types of target audio evaluation indicators of the target audio; Based on the global features of the standard audio, the audio rhythm analysis process, the note position recognition process, the similarity analysis process, and the empty sound position recognition process are performed to obtain at least two types of standard audio evaluation indicators of the standard audio.

[0136] Optionally, the audio evaluation module 704 is further configured to: Based on the target audio global feature, performing the audio rhythm analysis process on the target audio to obtain audio rhythm information of the target audio; Based on the target audio global feature, perform the note position recognition process on the target audio to obtain the note position information of the target audio, wherein the note position information includes the start time position and the end time position of at least one note in the target audio; Performing the similarity analysis process based on the target audio global feature and the standard audio global feature to obtain audio similarity information between the target audio and the standard audio, wherein the audio similarity information includes an association relationship between audio similarity and audio segments; Based on the target audio global feature, perform the empty sound position identification process on the target audio to obtain the empty sound position information of the target audio, wherein the empty sound position information includes the start time position and the end time position of at least one empty sound in the target audio; The audio rhythm information, the note position information, the audio similarity information and the empty sound position information are determined as the at least two types of target audio evaluation indicators.

[0137] Optionally, the audio evaluation module 704 is further configured to: Based on the standard audio global feature, performing the audio rhythm analysis process on the standard audio to obtain audio rhythm information of the standard audio; Based on the global features of the standard audio, the note position recognition process is performed on the standard audio to obtain the note position information of the standard audio, wherein the note position information includes the start time position and the end time position of at least one note in the standard audio; Performing the similarity analysis process based on the target audio global feature and the standard audio global feature to obtain audio similarity information between the target audio and the standard audio, wherein the audio similarity information includes an association relationship between audio similarity and audio segments; Based on the global features of the standard audio, perform the empty sound position identification process on the standard audio to obtain empty sound position information of the standard audio, wherein the empty sound position information includes a start time position and an end time position of at least one empty sound in the standard audio; The audio rhythm information, the note position information, the audio similarity information and the empty sound position information are determined as the at least two types of standard audio evaluation indicators.

[0138] Optionally, the target audio is target music, the standard audio is standard music corresponding to the target music, and the audio processing result is a music score; The result determination module 706 is configured to: Determining indicator weights corresponding to the at least two types of audio evaluation indicators; The target music is scored using the indicator weights and the at least two types of audio evaluation indicators to obtain the music score of the target music.

[0139] Optionally, the audio processing result is an abnormal audio segment; The result determination module 706 is configured to: Determine a plurality of target audio segments in the target audio and a plurality of standard audio segments in the standard audio; Determining an associated audio segment of each target audio segment from the plurality of standard audio segments based on the segment position information of the plurality of target audio segments and the segment position information of the plurality of standard audio segments; Determining, based on the at least two types of audio evaluation indicators, audio segment similarities between each target audio segment and the associated audio segment of each target audio segment; A target audio segment whose audio segment similarity is less than or equal to a preset similarity threshold is determined as an abnormal audio segment.

[0140] Optionally, the audio processing device further includes a first standard audio determination module, configured to: Using the music score processing model, extracting audio information from the standard music score to obtain standard audio information in the standard music score; A standard audio information file is generated using the standard audio information, and the standard audio is generated based on the standard audio information file.

[0141] Optionally, the audio processing device further includes a second standard audio determination module configured to: Acquire at least two audios to be processed, and use an audio processing model to extract audio information from the at least two audios to be processed to obtain at least two audio information files; The at least two audio information files are merged to obtain a standard audio information file, and the standard audio is generated based on the standard audio information file.

[0142] The audio processing device provided by one or more embodiments of the present specification extracts the target audio global features of the target audio and the standard audio global features of the standard audio during the process of evaluating the audio; the target audio global features can accurately represent the global audio changes of the target audio, and the standard audio global features can also accurately represent the global audio changes of the standard audio; and, utilizes the target audio global features and the standard audio global features to perform a variety of audio evaluation processes, and performs quantifiable evaluations on the audio data from multiple dimensions, thereby obtaining multiple accurate and comprehensive audio evaluation indicators; finally, accurately determines the audio processing results based on the multiple audio evaluation indicators; avoids the problem that manual evaluation methods may lead to inaccurate evaluation results, and at the same time, determines multiple audio evaluation indicators through a quantifiable audio evaluation method, avoids the problem that manual evaluation methods may lead to low evaluation efficiency, thereby achieving efficient and accurate evaluation of audio data.

[0143] The above is a schematic scheme of an audio processing device of this embodiment. It should be noted that the technical scheme of the audio processing device and the technical scheme of the above audio processing method belong to the same concept, and the details not described in detail in the technical scheme of the audio processing device can be referred to the description of the technical scheme of the above audio processing method.

[0144] Figure 8The block diagram of a computing device 800 according to an embodiment of the present specification is shown. The components of the computing device 800 include but are not limited to a memory 810 and a processor 820. The processor 820 is connected to the memory 810 via a bus 830, and a database 850 is used to store data.

[0145] The computing device 800 also includes an access device 840 that enables the computing device 800 to communicate via one or more networks 860. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 840 may include one or more of any type of network interface (e.g., a network interface card (NIC)) of wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a world-wide interoperability for microwave access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, and a near field communication (NFC).

[0146] In one embodiment of the present specification, the above components of the computing device 800 and Figure 8 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Figure 8 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0147] The computing device 800 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 800 may also be a mobile or stationary server.

[0148] The processor 820 is used to execute the following computer executable instructions, which implement the steps of the above audio processing method when executed by the processor.

[0149] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the computing device embodiment, since it is basically similar to the audio processing method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the audio processing method embodiment.

[0150] An embodiment of the present specification further provides a computer-readable storage medium storing a computer program / instruction, which implements the steps of the above-mentioned audio processing method when executed by a processor.

[0151] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the computer-readable storage medium embodiment, since it is basically similar to the audio processing method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the audio processing method embodiment.

[0152] An embodiment of the present specification further provides a computer program product, including a computer program / instruction, which implements the steps of the above audio processing method when executed by a processor.

[0153] The above is a schematic scheme of a computer program product of this embodiment. It should be noted that the technical scheme of the computer program product and the technical scheme of the above audio processing method belong to the same concept, and the details not described in detail in the technical scheme of the computer program product can be referred to the description of the technical scheme of the above audio processing method.

[0154] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0155] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0156] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0157] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0158] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to only the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that technicians in the relevant technical field can well understand and use this specification. This specification is only limited by the claims and their full scope and equivalents.

Claims

1. An audio processing method, characterized in that: include: Performing feature extraction on the target audio to obtain a target audio global feature of the target audio, and performing feature extraction on the standard audio corresponding to the target audio to obtain a standard audio global feature of the standard audio, wherein the target audio global feature is a feature representing a global audio change of the target audio, and the standard audio global feature is a feature representing a global audio change of the standard audio; Perform at least two types of audio evaluation processing using the target audio global feature and the standard audio global feature to obtain at least two types of audio evaluation indicators; Based on the at least two types of audio evaluation indicators, an audio processing result of the target audio is determined.

2. The audio processing method according to claim 1, characterized in that: The at least two types of audio evaluation processing include audio rhythm analysis processing, note position recognition processing, similarity analysis processing, and empty sound position recognition processing; The step of performing at least two types of audio evaluation processing using the target audio global feature and the standard audio global feature to obtain at least two types of audio evaluation indicators includes: Based on the target audio global features, the audio rhythm analysis process, the note position recognition process, the similarity analysis process, and the empty sound position recognition process are performed to obtain at least two types of target audio evaluation indicators of the target audio; Based on the global features of the standard audio, the audio rhythm analysis process, the note position recognition process, the similarity analysis process, and the empty sound position recognition process are performed to obtain at least two types of standard audio evaluation indicators of the standard audio.

3. The audio processing method according to claim 2, characterized in that: The step of performing the audio rhythm analysis process, the note position recognition process, the similarity analysis process, and the empty sound position recognition process based on the target audio global feature to obtain at least two types of target audio evaluation indicators of the target audio includes: Based on the target audio global feature, performing the audio rhythm analysis process on the target audio to obtain audio rhythm information of the target audio; Based on the target audio global feature, perform the note position recognition process on the target audio to obtain the note position information of the target audio, wherein the note position information includes the start time position and the end time position of at least one note in the target audio; Performing the similarity analysis process based on the target audio global feature and the standard audio global feature to obtain audio similarity information between the target audio and the standard audio, wherein the audio similarity information includes an association relationship between audio similarity and audio segments; Based on the target audio global feature, perform the empty sound position identification process on the target audio to obtain the empty sound position information of the target audio, wherein the empty sound position information includes the start time position and the end time position of at least one empty sound in the target audio; The audio rhythm information, the note position information, the audio similarity information and the empty sound position information are determined as the at least two types of target audio evaluation indicators.

4. The audio processing method according to claim 2, characterized in that: The audio rhythm analysis process, the note position recognition process, the similarity analysis process, and the empty sound position recognition process are performed based on the standard audio global features to obtain at least two types of standard audio evaluation indicators of the standard audio, including: Based on the standard audio global feature, performing the audio rhythm analysis process on the standard audio to obtain audio rhythm information of the standard audio; Based on the global features of the standard audio, the note position recognition process is performed on the standard audio to obtain the note position information of the standard audio, wherein the note position information includes the start time position and the end time position of at least one note in the standard audio; Performing the similarity analysis process based on the target audio global feature and the standard audio global feature to obtain audio similarity information between the target audio and the standard audio, wherein the audio similarity information includes an association relationship between audio similarity and audio segments; Based on the global features of the standard audio, perform the empty sound position identification process on the standard audio to obtain empty sound position information of the standard audio, wherein the empty sound position information includes a start time position and an end time position of at least one empty sound in the standard audio; The audio rhythm information, the note position information, the audio similarity information and the empty sound position information are determined as the at least two types of standard audio evaluation indicators.

5. The audio processing method according to any one of claims 1 to 4, characterized in that: The target audio is the target music, the standard audio is the standard music corresponding to the target music, and the audio processing result is the music score; The step of determining the audio processing result of the target audio based on the at least two types of audio evaluation indicators includes: Determining indicator weights corresponding to the at least two types of audio evaluation indicators; The target music is scored using the indicator weights and the at least two types of audio evaluation indicators to obtain the music score of the target music.

6. The audio processing method according to any one of claims 1 to 4, characterized in that: The audio processing result is an abnormal audio segment; The step of determining the audio processing result of the target audio based on the at least two types of audio evaluation indicators includes: Determine a plurality of target audio segments in the target audio and a plurality of standard audio segments in the standard audio; Determining an associated audio segment of each target audio segment from the plurality of standard audio segments based on the segment position information of the plurality of target audio segments and the segment position information of the plurality of standard audio segments; Determining, based on the at least two types of audio evaluation indicators, audio segment similarities between each target audio segment and the associated audio segment of each target audio segment; A target audio segment whose audio segment similarity is less than or equal to a preset similarity threshold is determined as an abnormal audio segment.

7. The audio processing method according to any one of claims 1 to 4, characterized in that: Before extracting features from the standard audio corresponding to the target audio to obtain the standard audio global features of the standard audio, the method further includes: Using the music score processing model, extracting audio information from the standard music score to obtain standard audio information in the standard music score; A standard audio information file is generated using the standard audio information, and the standard audio is generated based on the standard audio information file.

8. A computing device, characterized in that include: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer program / instructions are executed by the processor, the steps of the audio processing method according to any one of claims 1 to 7 are implemented.

9. A computer-readable storage medium, characterized in that: It stores a computer program / instruction, which, when executed by a processor, implements the steps of the audio processing method described in any one of claims 1 to 7.

10. A computer program product, characterized in that The method comprises a computer program / instruction, which, when executed by a processor, implements the steps of the audio processing method according to any one of claims 1 to 7.

Images