Audio processing method and apparatus, storage medium and electronic device

Abstract

Provided in the embodiments of the present disclosure are an audio processing method and apparatus, a storage medium and an electronic device. The audio processing method comprises: acquiring an audio to be processed, and segmenting said audio to obtain a plurality of audio segments; and inputting the plurality of audio segments into a pre-trained audio classification model, so as to obtain global classification information of said audio and / or local classification information of the audio segments, the global classification information and the local classification information respectively being multi-label classification information. By means of the pre-trained audio classification model, the present disclosure performs classification processing with different granularities on an audio to be processed, so as to obtain global classification information of said audio and / or local classification information of each audio segment, thus satisfying requirements for classifying audio data with different granularities. Additionally, both the global classification information and the local classification information are multi-label classification information, which can label overlap audio events within the audio data, thus improving the classification accuracy and recall rate.

Description

Audio processing method, device, storage medium and electronic device

[0001] This application claims priority to Chinese Patent Application No. 202410245215.3 filed on March 4, 2024, and the contents of the above-mentioned Chinese patent application disclosure are hereby incorporated by reference in their entirety as a part of this application. Technical Field

[0002] The present disclosure relates to audio processing technology, and more particularly to an audio processing method, apparatus, storage medium, and electronic device. Background Art

[0003] Audio data may include multiple different types of audio signals. By classifying and processing the audio data, different business processes can be performed on the audio data according to the types of audio signals contained in the audio data.

[0004] Currently, audio event detection (AED) models can be used to score each event in audio data and output the main events in the audio data. However, the recall rate of the audio data classification and recognition results of AED models is low. Summary of the Invention

[0005] The present disclosure provides an audio processing method, apparatus, storage medium, and electronic device to improve the accuracy and recall rate of audio classification results.

[0006] In a first aspect, an embodiment of the present disclosure provides an audio processing method, including:

[0007] Acquire audio to be processed, and segment the audio to be processed to obtain multiple audio segments;

[0008] The multiple audio segments are input into a pre-trained audio classification model to obtain global classification information of the audio to be processed and / or local classification information of the audio segments, wherein the global classification information and the local classification information are multi-label classification information.

[0009] In a second aspect, an embodiment of the present disclosure further provides an audio processing device, comprising:

[0010] An audio segmentation module is used to obtain audio to be processed, and segment the audio to be processed to obtain multiple audio segments;

[0011] An audio classification module is used to input the multiple audio segments into a pre-trained audio classification model to obtain global classification information of the audio to be processed and / or local classification information of the audio segments, wherein the global classification information and the local classification information are multi-label classification information respectively.

[0012] In a third aspect, an embodiment of the present disclosure further provides an electronic device, the electronic device comprising:

[0013] one or more processors;

[0014] a storage device for storing one or more programs,

[0015] When the one or more programs are executed by the one or more processors, the one or more processors implement the audio processing method provided in the embodiment of the present disclosure.

[0016] In a fourth aspect, an embodiment of the present disclosure further provides a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to execute the audio processing method provided in the embodiment of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The above and other features, advantages, and aspects of the various embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that the originals and elements are not necessarily drawn to scale.

[0018] FIG1 is a flow chart of an audio processing method provided by an embodiment of the present disclosure;

[0019] FIG2 is a schematic diagram of the processing process of the feature vectors corresponding to multiple audio segments by the classification module;

[0020] FIG3 is a schematic structural diagram of an audio processing device provided by an embodiment of the present disclosure; and

[0021] FIG4 is a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0022] The following describes embodiments of the present disclosure in more detail with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0023] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0024] As used herein, the term "including" and its variations are open-ended, i.e., "including but not limited to." The term "based on" means "based, at least in part, on." The term "one embodiment" means "at least one embodiment," the term "another embodiment" means "at least one additional embodiment," and the term "some embodiments" means "at least some embodiments." Other terms are defined in the following description.

[0025] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0026] It should be noted that the modifications of "one" and "multiple" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, they should be understood as "one or more".

[0027] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0028] It is understandable that before using the technical solutions disclosed in the various embodiments of this disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved in this disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0029] For example, in response to a user's active request, a prompt message is sent to the user to clearly inform the user that the operation requested will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the electronic device, application, server, storage medium, or other software or hardware that performs the operations of the disclosed technical solution based on the prompt message.

[0030] As an optional but non-limiting implementation, in response to receiving a user's active request, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. Furthermore, the pop-up window may also contain a selection control for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0031] It is understandable that the above notification and user authorization process are merely illustrative and do not limit the implementation of the present disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0032] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) must comply with the requirements of relevant laws, regulations and relevant provisions.

[0033] When the Audio Event Classification (AED) model classifies audio data, if multiple audio events overlap in time, it can only output the event result with the highest confidence. For example, when the audio data contains both human speech and background music, the AED model can only identify the human speech event and not the music event, resulting in a low recall rate for the audio recognition results. Furthermore, the AED's recognition results are too fine-grained, making them difficult to quickly use in business applications.

[0034] In response to the above technical problems, the embodiments of the present disclosure provide an audio processing method, which classifies audio data based on an end-to-end audio classification model, implements multi-label classification processing of audio data, and improves the accuracy and recall rate of audio data classification.

[0035] Figure 1 is a flow chart of an audio processing method provided by an embodiment of the present disclosure. The embodiment of the present disclosure is applicable to the situation of classifying audio. The method can be performed by an audio processing device, which can be implemented in the form of software and / or hardware. Optionally, it can be implemented by an electronic device, which can be a mobile terminal, PC or server, etc.

[0036] As shown in FIG1 , the method includes:

[0037] S110: Acquire audio to be processed, and segment the audio to be processed to obtain multiple audio segments.

[0038] S120: Input the multiple audio segments into a pre-trained audio classification model to obtain global classification information of the audio to be processed and / or local classification information of the audio segments, wherein the global classification information and the local classification information are multi-label classification information.

[0039] The audio to be processed can be UGC (User-generated-Content) audio or PGC (Professionally-generated Content) audio. For example, the audio to be processed can be audio data from a live broadcast, audio data uploaded by users on an audio and video playback platform, or audio data from user-uploaded videos.

[0040] Optionally, the audio to be processed may be segmented into audio segments of preset time intervals based on a preset time interval; optionally, the audio to be processed may be segmented into audio segments based on the duration of the audio to be processed and the number of audio segments; optionally, the audio to be processed may be segmented by receiving user-configured segmentation parameters and then dividing the audio to be processed into audio segments based on the segmentation parameters, wherein the segmentation parameters include but are not limited to one or more of the number of audio segments, the duration of the audio segments, and the positions of the audio segment nodes. For example, an audio track of audio data is displayed in an interactive interface, and the audio segment node positions selected on the audio track are received.

[0041] The audio classification model is used to classify multiple audio segments to obtain global classification information of the audio data and one or more items of local classification information of each audio segment. The global classification information of the audio data is the classification information obtained by classifying the audio data as a whole, and the local classification information of the audio segment is the classification information obtained by classifying each audio segment as a processing object. In this embodiment, the audio data is classified at different calibers by the audio classification model to obtain wide-caliber classification information (i.e., global classification information) and narrow-caliber classification information (i.e., local classification information) of the audio data, thereby improving the comprehensiveness of the audio data classification.

[0042] The global classification information and the local classification information are multi-label classification information, that is, the global classification information includes the confidence of the audio data corresponding to multiple classification labels, and the local classification information includes the confidence of the audio segment corresponding to multiple classification labels. For the global classification information, a first threshold corresponding to each classification label is set, and the confidence corresponding to the classification label in the global classification information is compared. If the confidence corresponding to the classification label in the global classification information is greater than the first threshold corresponding to the classification label, it is determined that the classification result of the audio data includes the classification label. For the local classification information, a second threshold corresponding to each classification label is set. If the confidence corresponding to the classification label in the local classification information is greater than the second threshold corresponding to the classification label, it is determined that the classification result of the audio segment includes the classification label. Among them, the first thresholds corresponding to different classification labels can be the same or different, the second thresholds corresponding to different classification labels can be the same or different, and the first threshold and second threshold corresponding to the same classification label can also be the same or different.

[0043] Optionally, the classification labels include one or more of the following: noise, music, and human voice. It is understandable that the above classification labels are only examples, and different classification labels can be set according to classification requirements.

[0044] Exemplarily, the global classification information for audio data includes a 30% confidence level for noise, a 90% confidence level for music, and a 78% confidence level for vocals, wherein the first thresholds for each of the aforementioned classification labels are 70%, and accordingly, the classification result for the audio data is determined to include music and vocals. Exemplarily, the local classification information for a particular audio segment includes an 80% confidence level for noise, a 90% confidence level for music, and a 78% confidence level for vocals, wherein the second thresholds for each of the aforementioned classification labels are 70%, and accordingly, the classification result for the audio segment includes noise, music, and vocals.

[0045] By performing multi-label classification processing on audio data or audio segments respectively, multi-label classification information is obtained, overlapping music events can be accurately detected, and at least one classification label for audio data or audio segments can be obtained, thereby improving the classification accuracy and recall rate of audio data and audio segments.

[0046] On the basis of the above embodiment, the audio classification model can be a neural network model, which is pre-trained and has the ability to perform classification processing on audio data of different calibers. Optionally, the audio classification model includes a spectrum conversion module, a feature vector extraction module and a classification module, wherein the spectrum conversion module is used to convert the audio segment into a spectrum block; the feature vector extraction module is used to extract the feature vector of the spectrum block; the classification module includes multiple fully connected layers, the first fully connected layer in the classification module outputs the local classification information of the audio segment, and the second fully connected layer in the classification module outputs the global classification information of the audio to be processed.

[0047] The spectrum conversion module may be a CQT (Constant-Q transform) module, which refers to a filter bank with exponentially distributed center frequencies and varying filter bandwidths, but with a center frequency to bandwidth ratio of constant Q. Each audio segment is converted by the spectrum conversion module into a spectrum block for that audio segment, i.e., a spectrum transform graph that changes over time.

[0048] The feature vector extraction module is connected to the spectrum conversion module to extract feature vectors from the spectrum blocks corresponding to each audio segment to obtain an embedding vector corresponding to each audio segment. The feature vector extraction module may include an IBN network module and a GEM network module connected in sequence. The IBN network module may be a ResNet-IBN network module, which may include a convolution block, at least one IBN network block, and a residual network block. The GEM network module may be a GEM pool network layer.

[0049] The classification module is connected to the feature vector extraction module and includes multiple fully connected layers connected in sequence. The classification module processes the embedding vectors corresponding to the audio segments in sequence to obtain global classification information for the processed audio and / or local classification information for the audio segments. Optionally, the second fully connected layer can also be a softmax layer.

[0050] Optionally, the classification module performs a max mean operation on the feature vectors corresponding to the multiple audio segments. For example, see Figure 2, which is a schematic diagram of the processing process of the classification module on the feature vectors corresponding to the multiple audio segments. Among them, the input information of the classification module is the feature vector L*dim corresponding to the multiple audio segments, L is the number of audio segments, and the feature vectors corresponding to the multiple audio segments are processed based on the fully connected layer. The processing process can be to calculate the feature vectors corresponding to the multiple audio segments and the weights of the fully connected layer (i.e., perform a max operation) to obtain the confidence of the multiple classification labels corresponding to the multiple audio segments, i.e., logit n corresponding to the L audio segments, where n is the number of classification labels, and the output is the local classification information of the audio segment. The confidence of the multiple classification labels corresponding to the multiple audio segments is processed by the softmax layer (i.e., perform a mean operation) to obtain the global classification information of the audio data, i.e., the confidence of the multiple classification labels logit 1…logit n.

[0051] The processing process of the feature vectors corresponding to multiple audio segments by the classification module can be expressed by the following formula: Logit k = MaxMean(X,W k ) Where X is the feature vector corresponding to multiple audio segments, W k is the weight of the fully connected layer in the classification module, and Logit k is the confidence of the k-th classification label.

[0052] Optionally, the confidence level corresponding to each classification label in the global classification information is determined based on the confidence levels of the classification labels corresponding to the multiple audio segments. For example, the confidence level of the audio data corresponding to the classification label may be the average of the confidence levels of the multiple audio segments for the same classification label. Exemplarily, each audio segment may correspond to a confidence level of a noise label, and the average of the confidence levels of the multiple audio segments corresponding to the noise labels may be used as the confidence level of the audio data corresponding to the noise label. For example, the confidence level of the audio data corresponding to the classification label may be the maximum value of the confidence levels of the multiple audio segments for the same classification label. Exemplarily, the confidence levels of the multiple audio segments corresponding to the noise labels are compared, and the maximum confidence level of the noise label is used as the confidence level of the audio data corresponding to the noise label.

[0053] The technical solution of the disclosed embodiments utilizes a pre-trained audio classification model to perform classification processing on the audio data at different scales, obtaining global classification information for the audio data at different scales and / or local classification information for each audio segment. This meets the requirements for classifying audio data at different scales. Furthermore, both global and local classification information are multi-label, marking overlapping audio events within the audio data and improving classification accuracy and recall.

[0054] On the basis of the above embodiment, after obtaining the local classification information and / or the global classification information, service processing may be performed on the audio to be processed based on the local classification information and / or the global classification information.

[0055] Optionally, based on the local classification information corresponding to the audio segments, the audio segments corresponding to the first label in the audio to be processed are intercepted and concatenated to obtain audio data corresponding to the first label. Based on the local classification information corresponding to the audio segments and the second threshold corresponding to each classification label, the classification results of the audio segments can be determined, for example, the classification results of the first audio segment as noise and music, the classification results of the second audio segment as music, and the classification results of the third audio segment as human voice and noise, etc. The first label can be any of the aforementioned classification labels. Taking the first label of music as an example, the audio segments corresponding to the music label are intercepted from the audio to be processed, such as the first audio segment and the second audio segment, to form audio data corresponding to the music label. Optionally, business processing is performed on the audio data corresponding to the first label, such as cover song recognition, fingerprint recognition, etc., on the audio data corresponding to the music label. By intercepting the audio data corresponding to the music label from the completed audio data, other audio segments outside the music label are removed, reducing the processing of audio segments that do not contain music.

[0056] Optionally, when the confidence of the second label in the global classification information of the audio to be processed meets a preset condition, the business processing model corresponding to the second label is called, and the audio to be processed is processed based on the business processing model to obtain the business processing result of the audio to be processed.

[0057] Exemplarily, the confidence of the second label in the global classification information meets the preset conditions, indicating that the audio to be processed includes an audio event corresponding to the second label. Taking the second label as a noise label as an example, the confidence of the noise label meets the preset conditions. For example, the confidence of the noise label may be greater than the first threshold corresponding to the noise label, indicating that the audio to be processed includes a noisy audio signal, and the business processing corresponding to the second label can be performed on the audio to be processed. Exemplarily, when the second label is a noise label, the business processing corresponding to the second label includes but is not limited to noise reduction processing. Accordingly, the business processing model corresponding to the second label includes a noise reduction model. Exemplarily, when the second label is a music label as an example, the business processing corresponding to the second label includes but is not limited to cover song recognition and fingerprint recognition, etc. Accordingly, the business processing model corresponding to the second label includes but is not limited to cover song recognition model and fingerprint recognition model, etc. Exemplarily, when the second label is a voice label as an example, the business processing corresponding to the second label includes but is not limited to semantic recognition, translation and text conversion, etc. Accordingly, the business processing model corresponding to the second label includes but is not limited to semantic recognition model, translation model and text conversion model.

[0058] In this embodiment, the global classification information of the audio to be processed is identified through the audio classification model, which provides a basis for subsequent business processing of the audio to be processed, improves the pertinence of the subsequent business processing, and reduces invalid business processing.

[0059] Based on the above embodiment, the training method of the audio classification model includes: obtaining a sample data set, which includes unlabeled sample audio; classifying the unlabeled sample audio based on a set audio event classification model to obtain event labels of multiple audio segments in the unlabeled sample audio; de-duplicating and merging the event labels of multiple audio segments in the unlabeled sample audio to obtain classification labels of the unlabeled sample audio; training the audio classification model to be trained based on the unlabeled sample audio and the classification labels of the unlabeled sample audio to obtain a trained audio classification model.

[0060] Among them, the unlabeled sample audio is a sample audio for which no classification label is set. The audio event classification model can be an AED model, which is a pre-trained audio event classification model. The unlabeled sample audio is classified and processed by the AED model to obtain event labels for multiple audio segments in the unlabeled sample audio. The event labels may include music, noise, and human voice. Exemplarily, the event label of the first audio segment in the unlabeled sample audio may be music, the event label of the second audio segment may be music, the event label of the third audio segment may be noise, the event label of the fourth audio segment may be human voice, etc. The AED model identifies the main events of each audio segment and obtains event labels.

[0061] Optionally, generate a classification label set for the unlabeled audio sample. The initial state of the classification label set is empty. Traverse multiple audio segments of the unlabeled audio sample and compare the event label of any audio segment with the classification label set. If the classification label set does not include the event label of the audio segment, then add the event label of the audio segment to the classification label set. Repeat this process to obtain the classification label of the unlabeled audio sample.

[0062] Optionally, the event labels corresponding to multiple audio segments of the unlabeled sample audio are merged, for example, the event labels of all audio segments are added to a classification label set, and the labels in the classification label set are deduplicated to obtain the classification labels of the unlabeled sample audio.

[0063] By determining the classification labels of unlabeled sample audio in the above manner, the manual label setting process is reduced and the label determination process is simplified. It is understandable that the sample dataset may also include labeled sample audio, and the classification labels of the labeled sample audio are manually set.

[0064] The sample audio in the sample data set is adjusted based on the classification label of the sample audio so that the sample audio corresponding to different types of classification labels meets a quantity balance condition, wherein the quantity balance condition can be that the data difference of the sample audio corresponding to different types of classification labels is less than a preset value, or that the quantity ratio of the sample audio corresponding to different types of classification labels meets a preset ratio range.

[0065] The adjustment of the sample audio may include adding new sample audio or removing sample audio corresponding to a large number of classification labels.

[0066] The audio classification model to be trained is iteratively trained based on the sample audio and the corresponding classification labels of the sample audio until the training termination conditions are met, thereby obtaining a trained audio classification model. During each iterative training process, a loss function is generated based on the corresponding classification labels of the sample audio and the predicted classification information of the audio classification model to be trained. The parameters of the audio classification model to be trained are adjusted based on the loss function.

[0067] Optionally, the loss function is determined based on the cross entropy loss term corresponding to each of the classification labels, and the average of the cross entropy loss terms corresponding to multiple classification labels is determined as the loss function. Exemplarily, the loss function can be expressed by the following formula:

[0068] Among them, N is the number of classification labels, y i is the i-th classification label, p(x i ) is the input x i Make a prediction and get the prediction result corresponding to the i-th classification label.

[0069] FIG3 is a schematic structural diagram of an audio processing device provided by an embodiment of the present disclosure. As shown in FIG3 , the device includes: an audio segmentation module 210 and an audio classification module 220 .

[0070] The audio segmentation module 210 is used to obtain the audio to be processed and segment the audio to be processed to obtain multiple audio segments;

[0071] The audio classification module 220 is used to input the multiple audio segments into a pre-trained audio classification model to obtain global classification information of the audio to be processed and / or local classification information of the audio segments, wherein the global classification information and the local classification information are multi-label classification information respectively.

[0072] The technical solution provided by the disclosed embodiments utilizes a pre-trained audio classification model to perform classification processing on the audio data at different scales, obtaining global classification information for the audio data at different scales and / or local classification information for each audio segment, thereby meeting the requirements for different classification scales of audio data. Furthermore, both global and local classification information are multi-label classification information, which can identify overlapping audio events in the audio data and improve classification accuracy and recall.

[0073] Based on the above embodiment, optionally, the audio classification model includes a spectrum conversion module, a feature vector extraction module and a classification module, wherein the spectrum conversion module is used to convert the audio segment into a spectrum block; the feature vector extraction module is used to extract the feature vector of the spectrum block; the classification module includes multiple fully connected layers, the first fully connected layer in the classification module outputs local classification information of the audio segment, and the second fully connected layer in the classification module outputs global classification information of the audio to be processed.

[0074] Based on the above embodiment, optionally, the local classification information includes confidences corresponding to multiple classification labels;

[0075] The global classification information includes confidences corresponding to a plurality of classification labels, respectively. The confidence corresponding to each classification label in the global classification information is determined based on confidences corresponding to the classification labels of the plurality of audio segments.

[0076] Based on the above embodiment, optionally, the device further includes:

[0077] an audio processing module, configured to perform one or more of the following: intercepting and concatenating audio segments corresponding to the first label in the audio to be processed based on the local classification information corresponding to the audio segments, to obtain audio data corresponding to the first label;

[0078] When the confidence of the second label in the global classification information of the audio to be processed meets a preset condition, the business processing model corresponding to the second label is called, and the audio to be processed is processed based on the business processing model to obtain a business processing result of the audio to be processed.

[0079] Based on the above embodiment, optionally, the device further includes:

[0080] The model training module is used to obtain a sample data set, which includes unlabeled sample audio; classify the unlabeled sample audio based on a set audio event classification model to obtain event labels of multiple audio segments in the unlabeled sample audio; de-duplicate and merge the event labels of multiple audio segments in the unlabeled sample audio to obtain classification labels of the unlabeled sample audio; and train an audio classification model to be trained based on the unlabeled sample audio and the classification labels of the unlabeled sample audio to obtain a trained audio classification model.

[0081] Optionally, the sample audios corresponding to different types of classification labels meet a quantity balance condition.

[0082] Optionally, the classification label includes one or more of the following: noise, music, and human voice;

[0083] During the training process of the audio classification model, the loss function is determined based on the cross entropy loss term corresponding to each of the classification labels.

[0084] The audio processing device provided by the embodiments of the present disclosure can execute the audio processing method provided by any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0085] It is worth noting that the various units and modules included in the above-mentioned device are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the embodiments of the present disclosure.

[0086] FIG4 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure. Referring to FIG4 , a schematic diagram of the structure of an electronic device (such as a terminal device or server in FIG4 ) 500 suitable for implementing an embodiment of the present disclosure is shown below. The terminal device in the embodiment of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. The electronic device shown in FIG4 is merely an example and should not impose any limitations on the functions and scope of use of the embodiments of the present disclosure.

[0087] As shown in FIG4 , the electronic device 500 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 into a random access memory (RAM) 503. Various programs and data required for the operation of the electronic device 500 are also stored in the RAM 503. The processing device 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An edit / output (I / O) interface 505 is also connected to the bus 504.

[0088] Typically, the following devices may be connected to the I / O interface 505: an input device 506 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 507 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 508 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 509. The communication device 509 may allow the electronic device 500 to communicate with other devices wirelessly or by wire to exchange data. Although FIG4 shows the electronic device 500 with various devices, it should be understood that not all of the devices shown are required to be implemented or present. More or fewer devices may alternatively be implemented or present.

[0089] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 509, or installed from the storage device 508, or installed from the ROM 502. When the computer program is executed by the processing device 501, the above-mentioned functions defined in the method of the embodiment of the present disclosure are performed.

[0090] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0091] The electronic device provided by the embodiment of the present disclosure and the audio processing method provided by the above embodiment belong to the same inventive concept. For technical details not fully described in this embodiment, please refer to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0092] An embodiment of the present disclosure provides a computer storage medium on which a computer program is stored. When the program is executed by a processor, the audio processing method provided by the above embodiment is implemented.

[0093] It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device, or component. In the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0094] In some embodiments, the client and server can communicate using any currently known or later developed network protocol, such as HTTP (HyperText Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or later developed network.

[0095] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0096] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device:

[0097] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: obtains the audio to be processed, segments the audio to be processed, and obtains multiple audio segments; inputs the multiple audio segments into a pre-trained audio classification model to obtain global classification information of the audio to be processed and / or local classification information of the audio segments, wherein the global classification information and the local classification information are multi-label classification information respectively.

[0098] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including, but not limited to, object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0099] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0100] The units involved in the embodiments described in this disclosure may be implemented in software or hardware. In some cases, the name of a unit does not limit the unit itself. For example, the first acquisition unit may also be described as a "unit for acquiring at least two Internet Protocol addresses."

[0101] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0102] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0103] According to one or more embodiments of the present disclosure, Example 1 provides an audio processing method, including:

[0104] Acquire audio to be processed, and segment the audio to be processed to obtain multiple audio segments;

[0105] The multiple audio segments are input into a pre-trained audio classification model to obtain global classification information of the audio to be processed and / or local classification information of the audio segments, wherein the global classification information and the local classification information are multi-label classification information.

[0106] According to one or more embodiments of the present disclosure, Example 2 provides the audio processing method of Example 1, further comprising:

[0107] The audio classification model includes a spectrum conversion module, a feature vector extraction module and a classification module, wherein the spectrum conversion module is used to convert the audio segment into a spectrum block; the feature vector extraction module is used to extract the feature vector of the spectrum block; the classification module includes multiple fully connected layers, the first fully connected layer in the classification module outputs local classification information of the audio segment, and the second fully connected layer in the classification module outputs global classification information of the audio to be processed.

[0108] According to one or more embodiments of the present disclosure, Example 3 provides the audio processing method of Example 1, further comprising:

[0109] The local classification information includes confidence levels corresponding to multiple classification labels;

[0110] The global classification information includes confidences corresponding to a plurality of classification labels, respectively. The confidence corresponding to each classification label in the global classification information is determined based on confidences corresponding to the classification labels of the plurality of audio segments.

[0111] According to one or more embodiments of the present disclosure, Example 4 provides the audio processing method of Example 1, further comprising:

[0112] The method further includes one or more of the following: based on the local classification information corresponding to the audio segment, intercepting and splicing the audio segment corresponding to the first label in the audio to be processed to obtain audio data corresponding to the first label;

[0113] When the confidence of the second label in the global classification information of the audio to be processed meets a preset condition, the business processing model corresponding to the second label is called, and the audio to be processed is processed based on the business processing model to obtain a business processing result of the audio to be processed.

[0114] According to one or more embodiments of the present disclosure, Example 5 provides the audio processing method of Example 1, further comprising:

[0115] The training method of the audio classification model includes: obtaining a sample data set, wherein the sample data set includes unlabeled sample audio; classifying the unlabeled sample audio based on a set audio event classification model to obtain event labels of multiple audio segments in the unlabeled sample audio; de-duplicating and merging the event labels of the multiple audio segments in the unlabeled sample audio to obtain classification labels of the unlabeled sample audio; and training the audio classification model to be trained based on the unlabeled sample audio and the classification labels of the unlabeled sample audio to obtain a trained audio classification model.

[0116] According to one or more embodiments of the present disclosure, Example 6 provides the audio processing method of Example 1, further comprising:

[0117] The sample audio corresponding to different types of classification labels meets the quantity balance condition.

[0118] According to one or more embodiments of the present disclosure, Example 7 provides the audio processing method of Example 1, further comprising:

[0119] The classification labels include one or more of the following: noise, music, and human voice;

[0120] During the training process of the audio classification model, the loss function is determined based on the cross entropy loss term corresponding to each of the classification labels.

[0121] According to one or more embodiments of the present disclosure, Example 8 provides an audio processing device, including:

[0122] An audio segmentation module is used to obtain audio to be processed, and segment the audio to be processed to obtain multiple audio segments;

[0123] An audio classification module is used to input the multiple audio segments into a pre-trained audio classification model to obtain global classification information of the audio to be processed and / or local classification information of the audio segments, wherein the global classification information and the local classification information are multi-label classification information respectively.

[0124] The above description is merely a preferred embodiment of the present disclosure and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but also includes other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned disclosed concepts. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this disclosure.

[0125] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details have been included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0126] Although the subject matter has been described in language specific to structural features and / or methodological logical acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims.

Claims

1. An audio processing method, comprising: Acquire audio to be processed, and segment the audio to be processed to obtain multiple audio segments; The multiple audio segments are input into a pre-trained audio classification model to obtain global classification information of the audio to be processed and / or local classification information of the audio segments, wherein the global classification information and the local classification information are multi-label classification information.

2. The method according to claim 1, wherein The audio classification model includes a spectrum conversion module, a feature vector extraction module and a classification module, wherein the spectrum conversion module is used to convert the audio segment into a spectrum block; the feature vector extraction module is used to extract the feature vector of the spectrum block; the classification module includes multiple fully connected layers, the first fully connected layer in the classification module outputs local classification information of the audio segment, and the second fully connected layer in the classification module outputs global classification information of the audio to be processed.

3. The method according to claim 1 or 2, wherein: The local classification information includes confidence levels corresponding to multiple classification labels; The global classification information includes confidences corresponding to a plurality of classification labels, respectively. The confidence corresponding to each classification label in the global classification information is determined based on confidences corresponding to the classification labels of the plurality of audio segments.

4. The method according to any one of claims 1 to 3, further comprising one or more of the following: Based on the local classification information corresponding to the audio segment, the audio segment corresponding to the first label in the audio to be processed is intercepted and spliced ​​to obtain audio data corresponding to the first label; When the confidence of the second label in the global classification information of the audio to be processed meets a preset condition, the business processing model corresponding to the second label is called, and the audio to be processed is processed based on the business processing model to obtain a business processing result of the audio to be processed.

5. The method according to any one of claims 1 to 4, wherein: The training method of the audio classification model includes: Acquire a sample data set, wherein the sample data set includes unlabeled sample audio; Classify the unlabeled sample audio based on a set audio event classification model to obtain event labels for multiple audio segments in the unlabeled sample audio; De-duplication and merging of event labels of multiple audio segments in the unlabeled sample audio to obtain a classification label of the unlabeled sample audio; The audio classification model to be trained is trained based on the unlabeled sample audio and the classification label of the unlabeled sample audio to obtain a trained audio classification model.

6. The method according to claim 5, wherein: The sample audio corresponding to different types of classification labels meets the quantity balance condition.

7. The method according to claim 5 or 6, wherein: The classification labels include one or more of the following: noise, music, and human voice; During the training process of the audio classification model, the loss function is determined based on the cross entropy loss term corresponding to each of the classification labels.

8. An audio processing device, comprising: An audio segmentation module is configured to obtain audio to be processed, and segment the audio to be processed to obtain multiple audio segments; The audio classification module is configured to input the multiple audio segments into a pre-trained audio classification model to obtain global classification information of the audio to be processed and / or local classification information of the audio segments, wherein the global classification information and the local classification information are multi-label classification information respectively.

9. An electronic device comprising: one or more processors; A storage device configured to store one or more programs, wherein When the one or more programs are executed by the one or more processors, the one or more processors are enabled to implement the audio processing method according to any one of claims 1 to 7.

10. A storage medium containing computer-executable instructions, wherein: When the computer executable instructions are executed by a computer processor, they are used to perform the audio processing method according to any one of claims 1 to 7.

Images