A dubbing audio analysis method, device, electronic device and storage medium

Abstract

The present application relates to a dubbing audio analysis method, device, electronic device and storage medium, the method comprising: obtaining the dubbing audio of a dubbing actor; determining the spectrogram image corresponding to the dubbing audio according to the dubbing audio; judging whether the dubbing audio has pronunciation abnormalities according to the spectrogram image and the spectrogram image of the pre-classified problematic audio; if the judgment result is that the dubbing audio has pronunciation abnormalities, determining the type of pronunciation abnormality according to the spectrogram image and the spectrogram image of the pre-classified problematic audio. The present application determines the spectrogram image corresponding to the dubbing audio for the dubbing audio, first determines whether there is a pronunciation abnormality according to the spectrogram image and the spectrogram image of the pre-classified problematic audio, and in the case of pronunciation abnormalities, determines the type of pronunciation abnormality according to the spectrogram image and the spectrogram image of the pre-classified problematic audio, and by introducing the spectrogram image, the dubbing problem of voice workers can be quickly and comprehensively solved.

Description

Technical Field

[0001] The present application relates to the field of dubbing technology, and in particular to a dubbing audio analysis method, device, electronic device and storage medium. Background Art

[0002] What is saliva sound: When voice workers (voice actors, announcers, dubbing artists, singers, etc.) record or dub with high precision, the tongue in the mouth collides and rubs with other parts, forming bubbles. The sound produced by the bubble bursting or the dubbing artist swallowing saliva is saliva sound. Saliva sound is an eternal topic that troubles voice workers, and most related practitioners have such a situation. Saliva sound has a great impact on voice workers. From the perspective of effect, it will affect the effect of recording training and fail to meet the recording requirements; from the perspective of work results, a large amount of saliva sound will lead to rework of recording work, deducting costs, and have a negative impact on the ability of actors and studios. In artificial intelligence-related work, saliva sound will affect the training effect of TTS voice. When training a certain TTS voice, due to the presence of a large amount of saliva sound in the training audio, the effect was not good, and it was forced to re-record, wasting a lot of time. Therefore, for recording studios, how to detect saliva sound and re-record it as soon as possible during recording; for AI companies, how to quickly screen audio files after getting the audio to see if there are too many saliva sounds that affect model training will become particularly important. At the same time, in addition to saliva sounds, for professional speech workers, there are also various sound problems such as dental sounds, bubble sounds, and breath sounds.

[0003] The current method to solve this type of problem is to have an experienced sound engineer or voice analysis technician screen the recording files, mainly by listening to the recording and checking the sound wave information in combination with possible abnormalities in the voice to find the corresponding problem points. Then arrange for subsequent re-recording, and the person who screens the recording must have strong professional knowledge to distinguish. However, for a large amount of audio, listening to it over and over again will waste a lot of time and manpower for manual screening, and it is impossible to accurately find all the problems. In actual dubbing recording work, it is fastest to find the sound problem at the time of recording / dubbing and re-record it on the spot. It is time-consuming and laborious to find it later. Summary of the invention

[0004] Based on the above problems, the present application provides a dubbing audio analysis method, device, electronic device and storage medium.

[0005] In a first aspect, an embodiment of the present application provides a dubbing audio analysis method, comprising:

[0006] Get the dubbing audio of the voice actor;

[0007] Determine a spectrogram image corresponding to the dubbing audio according to the dubbing audio;

[0008] determining whether the dubbing audio has pronunciation abnormalities based on the spectrogram image and the pre-classified spectrogram image of the problematic audio;

[0009] If the judgment result is that the dubbing audio has pronunciation abnormality, the type of pronunciation abnormality is determined based on the spectrogram image and the pre-classified spectrogram image of the problematic audio.

[0010] Furthermore, in the above-mentioned dubbing audio analysis method, the types of abnormal pronunciation include at least: saliva sounds, dental sounds, bubble sounds and breathing sounds.

[0011] Furthermore, in the above-mentioned dubbing audio analysis method, the classification steps of the spectrogram image of the pre-classified problematic audio are as follows:

[0012] Get problematic audio for multiple voice actors;

[0013] Classifying the problematic audio to obtain multiple problematic audios;

[0014] Extract spectrogram information of various problematic audios;

[0015] Features of spectrogram information that annotate various problematic audio;

[0016] The problematic audio is classified according to the features of the spectrogram information of multiple problematic audios to obtain the spectrogram images of the problematic audios.

[0017] Furthermore, in the above-mentioned dubbing audio analysis method, the problematic audio is classified to obtain a plurality of problematic audios, including:

[0018] The audios of saliva sound, dental sound, bubble sound and breath sound are classified to obtain saliva sound audio, dental sound audio, bubble sound audio and breath sound audio.

[0019] Furthermore, in the above-mentioned dubbing audio analysis method, after determining the type of abnormal pronunciation, it also includes:

[0020] Correct the dubbing audio according to the type of pronunciation abnormality.

[0021] Furthermore, in the above-mentioned dubbing audio analysis method, the dubbing audio is corrected, including: correcting the dubbing audio by correcting the frequency, amplitude, timbre, pitch, and loudness of the dubbing sound.

[0022] Furthermore, in the above-mentioned dubbing audio analysis method, the problematic audio is classified according to the characteristics of the spectrogram information of multiple problematic audios to obtain the spectrogram image of the problematic audio using image recognition technology and algorithm clustering analysis technology.

[0023] In a second aspect, an embodiment of the present application further provides a dubbing audio analysis device, comprising:

[0024] Acquisition module: used to obtain the dubbing audio of the dubbing actor;

[0025] A first determination module: used to determine the spectrogram image corresponding to the dubbing audio according to the dubbing audio;

[0026] A judgment module: used for judging whether the dubbing audio has abnormal pronunciation according to the spectrogram image and the pre-classified spectrogram image of the problematic audio;

[0027] The second determination module is used to determine the type of abnormal pronunciation based on the spectrogram image and the pre-classified spectrogram image of the problematic audio if the judgment result is that the dubbing audio has abnormal pronunciation.

[0028] In a third aspect, an embodiment of the present application further provides an electronic device, including: a processor and a memory;

[0029] The processor is used to execute any one of the above-mentioned dubbing audio analysis methods by calling the program or instruction stored in the memory.

[0030] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, which stores a program or instruction, and the program or instruction enables a computer to execute any one of the above-mentioned dubbing audio analysis methods.

[0031] The advantages of the embodiments of the present application are: the present application relates to a dubbing audio analysis method, device, electronic device and storage medium, the method comprising: obtaining the dubbing audio of the dubbing actor; determining the spectrogram image corresponding to the dubbing audio according to the dubbing audio; judging whether the dubbing audio has pronunciation abnormalities according to the spectrogram image and the spectrogram image of the pre-classified problematic audio; if the judgment result is that the dubbing audio has pronunciation abnormalities, determining the type of pronunciation abnormality according to the spectrogram image and the spectrogram image of the pre-classified problematic audio. The present application determines the spectrogram image corresponding to the dubbing audio for the dubbing audio, first determines whether there is a pronunciation abnormality according to the spectrogram image and the spectrogram image of the pre-classified problematic audio, and in the case of pronunciation abnormalities, determines the type of pronunciation abnormality according to the spectrogram image and the spectrogram image of the pre-classified problematic audio, and by introducing the spectrogram image, the dubbing problem of voice workers can be quickly and comprehensively solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the conventional technology, the drawings required for use in the embodiments or the conventional technology descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0033] Figure 1 A method for analyzing dubbing audio provided in an embodiment of the present application is shown in FIG. Figure 1 ;

[0034] Figure 2 A method for analyzing dubbing audio provided in an embodiment of the present application is shown in FIG. Figure 2 ;

[0035] Figure 3 A schematic diagram of a dubbing audio analysis device provided in an embodiment of the present application;

[0036] Figure 4 It is a schematic block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0037] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present application, so the present application is not limited by the specific implementation disclosed below.

[0038] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application. The term "and / or" used herein includes any and all combinations of one or more of the related listed items.

[0039] Figure 1 A method for analyzing dubbing audio provided in an embodiment of the present application is shown in FIG. Figure 1 .

[0040] In the first aspect, the present application embodiment provides a dubbing audio analysis method, combining Figure 1 , including four steps from S101 to S104:

[0041] S101: Obtain the dubbing audio of the dubbing actor.

[0042] Specifically, in the embodiment of the present application, the dubbing audio of the voice actor can be the dubbing audio of any voice actor in the recording studio, or the dubbing audio of any voice actor that needs to be accepted by a third party, and so on.

[0043] S102: Determine a spectrogram image corresponding to the dubbing audio according to the dubbing audio.

[0044] Specifically, in an embodiment of the present application, determining the spectrogram image corresponding to the dubbing audio based on the dubbing audio is to extract the dubbing audio through image recognition technology and algorithm clustering analysis technology to obtain the spectrogram image corresponding to the dubbing audio.

[0045] S103: Determine whether the dubbing audio has pronunciation abnormality based on the spectrogram image and the pre-classified spectrogram image of the problematic audio.

[0046] Specifically, in the embodiment of the present application, based on the spectrogram image and the pre-classified spectrogram images of problematic audio, it is possible to intuitively determine where the abnormalities are in the spectrogram image of the dubbing audio, where they are located in the audio, and which audios they are.

[0047] S104: If the judgment result is that the dubbing audio has pronunciation abnormality, the type of the pronunciation abnormality is determined according to the spectrogram image and the pre-classified spectrogram image of the problematic audio.

[0048] Specifically, in the embodiment of the present application, when it is determined that the spectrogram image of the dubbing audio has an abnormality, the spectrogram image of the dubbing audio is further determined to determine the type of abnormal pronunciation, that is, one or more of saliva sound, dental sound, bubbling sound and breathing sound, based on the spectrogram image and the spectrogram image of the pre-classified problematic audio. Therefore, the dubbing problem of voice workers can be quickly and comprehensively solved by introducing the spectrogram image.

[0049] Furthermore, in the above-mentioned dubbing audio analysis method, the types of abnormal pronunciation include at least: saliva sounds, dental sounds, bubble sounds and breathing sounds.

[0050] Figure 2 A method for analyzing dubbing audio provided in an embodiment of the present application is shown in FIG. Figure 2 .

[0051] Furthermore, in the above-mentioned dubbing audio analysis method, Figure 2 The classification step of the spectrogram image of the pre-classified problematic audio includes five steps from S201 to S205:

[0052] S201: Acquire problematic audios of multiple voice actors.

[0053] Specifically, in the embodiment of the present application, the audio with problems of multiple voice actors obtained may be the saliva sound audio, dental sound audio, bubble sound audio and breathing sound audio of multiple actors.

[0054] S202: Classify the problematic audio to obtain multiple problematic audios.

[0055] Specifically, in an embodiment of the present application, the saliva sound audio, dental sound audio, bubble sound audio and breath sound audio of multiple actors are classified, such as the saliva sound audio of the voice actor is classified into one category, the dental sound audio of the voice actor is classified into one category, the bubble sound audio of the voice actor is classified into one category, and the breath sound audio of the voice actor is classified into one category.

[0056] S203: Extracting spectrogram information of multiple problematic audios.

[0057] Specifically, in an embodiment of the present application, extracting spectrogram information of multiple problematic audios may be extracting spectrogram information of each type of audio after classification, such as extracting spectrogram information of bubble sound audio, extracting spectrogram information of breath sound audio, extracting spectrogram information of dental sound audio, and extracting spectrogram information of saliva sound audio.

[0058] S204: Marking features of spectrogram information of multiple problematic audios.

[0059] Specifically, in the embodiment of the present application, after extracting the spectrogram information of bubble sound audio, extracting the spectrogram information of breath sound audio, extracting the spectrogram information of dental sound audio, and extracting the spectrogram information of saliva sound audio, the features of each problematic spectrogram information are marked. It should be understood that the features of the spectrogram information of each problematic audio are different.

[0060] S205: Classify the problematic audio according to the features of the spectrogram information of the multiple problematic audios to obtain the spectrogram images of the problematic audios.

[0061] Specifically, in the embodiment of the present application, the spectrogram information features of each problematic audio are different, and the saliva sound audio, dental sound audio, bubble sound audio and breath sound audio are classified according to the spectrogram information features to obtain different spectrogram images.

[0062] Furthermore, in the above-mentioned dubbing audio analysis method, the problematic audio is classified to obtain a plurality of problematic audios, including:

[0063] The audios of saliva sound, dental sound, bubble sound and breath sound are classified to obtain saliva sound audio, dental sound audio, bubble sound audio and breath sound audio.

[0064] Furthermore, in the above-mentioned dubbing audio analysis method, after determining the type of abnormal pronunciation, it also includes:

[0065] Correct the dubbing audio according to the type of pronunciation abnormality.

[0066] Specifically, in the embodiment of the present application, the abnormal part of the dubbing audio can be directly corrected so that the dubbing audio meets the requirements, saving a lot of workload.

[0067] Furthermore, in the above-mentioned dubbing audio analysis method, the dubbing audio is corrected, including: correcting the dubbing audio by correcting the frequency, amplitude, timbre, pitch, and loudness of the dubbing sound.

[0068] Specifically, in the embodiment of the present application, the abnormal part of the dubbing audio is corrected by correcting the frequency, amplitude, timbre, pitch, and loudness of the dubbing sound, so that the spectrogram image of the dubbing audio meets the requirements of the dubbing audio and the sound is smoother.

[0069] Furthermore, in the above-mentioned dubbing audio analysis method, the problematic audio is classified according to the characteristics of the spectrogram information of multiple problematic audios to obtain the spectrogram image of the problematic audio using image recognition technology and algorithm clustering analysis technology.

[0070] Specifically, in the embodiment of the present application, the waveforms of the spectrogram information of various problematic audios are different, and the problematic audios are classified by image recognition technology and algorithm clustering analysis technology to obtain spectrogram images of the problematic audios.

[0071] Figure 3 A schematic diagram of a dubbing audio analysis device provided in an embodiment of the present application.

[0072] In a second aspect, the present application also provides a dubbing audio analysis device, combined with Figure 3 ,include:

[0073] Acquisition module 301: used to acquire the dubbing audio of the dubbing actor.

[0074] Specifically, in the embodiment of the present application, the acquisition module 301 acquires the dubbing audio of the dubbing actor. The dubbing audio of the dubbing actor can be the dubbing audio of any dubbing actor in the recording studio, or the dubbing audio of any dubbing actor that needs to be accepted by a third party, and so on.

[0075] The first determination module 302 is used to determine the spectrogram image corresponding to the dubbing audio according to the dubbing audio.

[0076] Specifically, in the embodiment of the present application, the first determination module 302 determines the spectrogram image corresponding to the dubbing audio based on the dubbing audio by extracting the dubbing audio through image recognition technology and algorithm clustering analysis technology to obtain the spectrogram image corresponding to the dubbing audio.

[0077] The judgment module 303 is used to judge whether there is any pronunciation abnormality in the dubbing audio according to the spectrogram image and the pre-classified spectrogram image of the problematic audio.

[0078] Specifically, in the embodiment of the present application, the judgment module 303 can intuitively judge where the abnormalities are in the spectrogram image of the dubbing audio, where they are located in the audio, and which audios they are based on the spectrogram image and the pre-classified spectrogram images of problematic audio.

[0079] The second determination module 304 is used to determine the type of pronunciation abnormality according to the spectrogram image and the pre-classified spectrogram image of the problematic audio if the judgment result is that the dubbing audio has pronunciation abnormality.

[0080] Specifically, in the embodiment of the present application, when the second determination module 304 determines that the spectrogram image of the dubbing audio has an abnormality, it further determines the type of abnormal pronunciation of the spectrogram image of the dubbing audio according to the spectrogram image and the spectrogram image of the pre-classified problematic audio, that is, one or more of saliva sound, dental sound, bubbling sound and breathing sound. Therefore, the introduction of the spectrogram image can quickly and comprehensively solve the dubbing problem of voice workers.

[0081] In a third aspect, an embodiment of the present application further provides an electronic device, including: a processor and a memory;

[0082] The processor is used to execute any one of the above-mentioned dubbing audio analysis methods by calling the program or instruction stored in the memory.

[0083] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, which stores a program or instruction, and the program or instruction enables a computer to execute any one of the above-mentioned dubbing audio analysis methods.

[0084] Figure 4 It is a schematic block diagram of an electronic device provided by an embodiment of the present disclosure.

[0085] like Figure 4As shown, the electronic device includes: at least one processor 401, at least one memory 402 and at least one communication interface 403. The various components in the electronic device are coupled together through a bus system 404. The communication interface 403 is used to transmit information between external devices. It can be understood that the bus system 404 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 404 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, Figure 4 Various buses are labeled as bus system 404 .

[0086] It can be understood that the memory 402 in this embodiment can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories.

[0087] In some implementations, the memory 402 stores the following elements, executable units or data structures, or a subset thereof, or an extended set thereof: an operating system and application programs.

[0088] Among them, the operating system includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various basic services and process hardware-based tasks. The application program includes various application programs, such as a media player (Media Player), a browser (Browser), etc., which are used to implement various application services. A program that implements any method in a dubbing audio analysis method provided in an embodiment of the present application can be included in the application program.

[0089] In an embodiment of the present application, the processor 401 calls a program or instruction stored in the memory 402, specifically, a program or instruction stored in an application, and the processor 401 is used to execute the steps of each embodiment of a dubbing audio analysis method provided in an embodiment of the present application.

[0090] Get the dubbing audio of the voice actor;

[0091] Determine a spectrogram image corresponding to the dubbing audio according to the dubbing audio;

[0092] determining whether the dubbing audio has pronunciation abnormalities based on the spectrogram image and the pre-classified spectrogram image of the problematic audio;

[0093] If the judgment result is that the dubbing audio has pronunciation abnormality, the type of pronunciation abnormality is determined based on the spectrogram image and the pre-classified spectrogram image of the problematic audio.

[0094] Any method in a dubbing audio analysis method provided in an embodiment of the present application can be applied to a processor 401, or implemented by a processor 401. The processor 401 can be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor 401 or an instruction in the form of software. The above-mentioned processor 401 can be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0095] The steps of any method in a dubbing audio analysis method provided in an embodiment of the present application can be directly embodied as being executed by a hardware decoding processor, or being executed by a combination of hardware and software units in a decoding processor. The software unit can be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 402, and the processor 401 reads the information in the memory 402, and completes the steps of a dubbing audio analysis method in combination with its hardware.

[0096] Those skilled in the art will appreciate that although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features from different embodiments is meant to be within the scope of the present application and to form different embodiments.

[0097] Those skilled in the art will appreciate that the description of each embodiment has its own emphasis, and for parts not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0098] The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A dubbing audio analysis method, characterized in that: include: Get the dubbing audio of the voice actor; Determine a spectrogram image corresponding to the dubbing audio according to the dubbing audio; Determining whether the dubbing audio has pronunciation abnormality according to the spectrogram image and the pre-classified spectrogram image of the problematic audio; The abnormal pronunciation types include at least: saliva sound, dental sound, bubble sound and breath sound; If the judgment result is that the dubbing audio has pronunciation abnormality, the type of the pronunciation abnormality is determined according to the spectrogram image and the spectrogram image of the pre-classified problematic audio.

2. A dubbing audio analysis method according to claim 1, characterized in that: The classification steps of the pre-classified spectrogram image of the problematic audio are as follows: Get problematic audio for multiple voice actors; Classifying the problematic audio to obtain multiple problematic audios; Extracting spectrogram information of the multiple problematic audios; marking features of the spectrogram information of the multiple problematic audios; The problematic audio is classified according to the features of the spectrogram information of the multiple problematic audios to obtain the spectrogram images of the problematic audios.

3. A dubbing audio analysis method according to claim 2, characterized in that: The problematic audio is classified to obtain a plurality of problematic audios, including: The audios of saliva sound, dental sound, bubble sound and breath sound are classified to obtain saliva sound audio, dental sound audio, bubble sound audio and breath sound audio.

4. A dubbing audio analysis method according to claim 1, characterized in that: After determining the type of abnormal pronunciation, the method further includes: The dubbing audio is corrected according to the type of abnormal pronunciation.

5. A dubbing audio analysis method according to claim 4, characterized in that: The step of correcting the dubbing audio includes: Correct the dubbing audio by modifying the frequency, amplitude, timbre, pitch, and loudness of the dubbing sound.

6. A dubbing audio analysis method according to claim 2, characterized in that: The method of classifying the problematic audios according to the features of the spectrogram information of the multiple problematic audios to obtain the spectrogram images of the problematic audios adopts image recognition technology and algorithm clustering analysis technology.

7. A dubbing audio analysis device, characterized in that: include: Acquisition module: used to obtain the dubbing audio of the dubbing actor; A first determining module: used to determine a spectrogram image corresponding to the dubbing audio according to the dubbing audio; A judgment module: used for judging whether the dubbing audio has abnormal pronunciation according to the spectrogram image and the pre-classified spectrogram image of the problematic audio; The abnormal pronunciation types include at least: saliva sound, dental sound, bubble sound and breath sound; The second determination module is used to determine the type of abnormal pronunciation according to the spectrogram image and the pre-classified spectrogram image of the problematic audio if the judgment result is that the dubbing audio has abnormal pronunciation.

8. An electronic device, characterized in that: include: Processor and memory; The processor is used to execute the dubbing audio analysis method as described in any one of claims 1 to 6 by calling the program or instruction stored in the memory.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program or instruction, and the program or instruction enables a computer to execute a dubbing audio analysis method as claimed in any one of claims 1 to 6.

Images