A method, device, computer equipment and medium for processing audio data

Abstract

The embodiments of the present application provide an audio data processing method, device, computer equipment and medium, which can be applied to various scenarios such as cloud technology, artificial intelligence, smart transportation, audio, etc. The method includes: displaying original text information in an application interface; obtaining target audio data corresponding to target timbre information and target text information; target text information refers to text information selected in the original text information; obtaining spliced ​​audio data for the original text information; spliced ​​audio data is obtained by splicing fused audio data and standard audio data corresponding to the remaining text information; the remaining text information is the text information in the original text information except the target text information; the fused audio data is obtained by fusing the target audio data and the target timbre information. The present application can improve the richness of audio creation and improve the quality of audio data.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an audio data processing method, apparatus, computer equipment and medium. Background Art

[0002] At present, the main method of dubbing production is to call the speech synthesis algorithm, that is, to directly input the text information that needs to be voiced into the speech synthesis algorithm, and directly generate the audio data corresponding to the text information through the speech synthesis algorithm. For the voice synthesis algorithm to achieve dubbing production, audio data with standard pronunciation can be quickly generated. However, the generated audio data is relatively simple in audio information such as tone, speaking speed and emotion, thereby reducing the richness of audio creation. In addition, because different scenes require audio data with different audio information, the single audio data generated by the speech synthesis algorithm is difficult to adapt to the dubbing needs in a variety of scenes, thereby significantly reducing the quality of the audio data. Summary of the invention

[0003] The embodiments of the present application provide an audio data processing method, apparatus, computer equipment, and medium, which can improve the richness of audio creation and improve the quality of audio data.

[0004] An embodiment of the present application provides an audio data processing method, including:

[0005] Display original text information in the application interface;

[0006] Obtaining target audio data corresponding to target timbre information and target text information; the target text information refers to text information selected from the original text information; the audio content in the target audio data matches the target text information;

[0007] Acquire spliced ​​audio data for the original text information; the spliced ​​audio data is obtained by splicing the fused audio data and the standard audio data corresponding to the remaining text information; the remaining text information is the text information in the original text information except the target text information; the fused audio data is obtained by fusing the target audio data and the target timbre information; the audio content in the standard audio data matches the remaining text information.

[0008] An embodiment of the present application provides an audio data processing device, including:

[0009] A text display module, used to display original text information in the application interface;

[0010] The audio acquisition module is used to acquire target audio data corresponding to target timbre information and target text information; the target text information refers to text information selected from the original text information; the audio content in the target audio data matches the target text information;

[0011] The audio splicing module is used to obtain spliced ​​audio data for the original text information; the spliced ​​audio data is obtained by splicing the fused audio data and the standard audio data corresponding to the remaining text information; the remaining text information is the text information in the original text information except the target text information; the fused audio data is obtained by fusing the target audio data and the target timbre information; the audio content in the standard audio data matches the remaining text information.

[0012] Among them, the text display module is specifically used to display the application interface; the application interface includes a text entry area;

[0013] The text display module is specifically used to respond to the input operation on the text entry area and display the input original text information in the text entry area.

[0014] Among them, the text display module is specifically used to display the application interface; the application interface includes a text upload control;

[0015] The text display module is specifically used to respond to the trigger operation on the text upload control and display a text selection interface for selecting a text file;

[0016] The text display module is specifically used to respond to the text file selection operation on the text selection interface and use the text file selected based on the text file selection operation as the target text file;

[0017] The text display module is specifically used to respond to the text file confirmation operation on the text selection interface, use the text information in the target text file as the original text information, and display the original text information in the application interface.

[0018] Among them, the audio acquisition module includes:

[0019] A text selection unit, for responding to a text selection operation on the original text information, and taking the text information selected based on the text selection operation as the target text information;

[0020] A voice conversion unit, used to respond to the voice conversion operation for the target text information and display an audio conversion interface;

[0021] A timbre acquisition unit, used to acquire target timbre information in the audio conversion interface;

[0022] The audio acquisition unit is used to respond to the audio upload operation on the audio conversion interface and acquire the target audio data corresponding to the target text information.

[0023] Wherein, the application interface includes a first voice conversion control;

[0024] The voice conversion unit is specifically configured to respond to a trigger operation on the first voice conversion control and display an audio conversion interface.

[0025] The speech conversion unit is specifically used to respond to the trigger operation on the target text information and display the text control list; the text control list includes the second speech conversion control;

[0026] The voice conversion unit is specifically configured to respond to a trigger operation on the second voice conversion control and display an audio conversion interface.

[0027] Among them, the audio conversion interface includes a recording start control;

[0028] The audio acquisition unit is specifically used to respond to the trigger operation of the recording start control and display the recording stop control in the audio conversion interface;

[0029] The audio acquisition unit is specifically used to respond to the trigger operation on the recording stop control, acquire the audio data recorded by the target object in the time interval between responding to the trigger operation on the recording start control and responding to the trigger operation on the recording stop control, and use the audio data recorded by the target object as the target audio data corresponding to the target text information;

[0030] The audio acquisition unit is also specifically used to display the audio file identifier corresponding to the target audio data in the audio conversion interface.

[0031] Among them, the audio conversion interface includes an audio upload control;

[0032] The audio acquisition unit is specifically used to respond to the trigger operation on the audio upload control and display an audio selection interface for selecting an audio file;

[0033] The audio acquisition unit is specifically used to respond to the audio file selection operation on the audio selection interface and use the audio file selected based on the audio file selection operation as the target audio file;

[0034] The audio acquisition unit is specifically used to respond to the audio file confirmation operation on the audio selection interface and use the audio data in the target audio file as the target audio data corresponding to the target text information;

[0035] The audio acquisition unit is also specifically used to display the audio file identifier corresponding to the target audio data in the audio conversion interface.

[0036] Wherein, the audio conversion interface includes one or more candidate timbre information;

[0037] A timbre acquisition unit, specifically configured to respond to a timbre selection operation for one or more candidate timbre information and use the candidate timbre information selected based on the timbre selection operation as target timbre information;

[0038] The timbre acquisition unit is also specifically used to highlight the target timbre information in the audio conversion interface.

[0039] Among them, the audio splicing module includes:

[0040] A fusion unit, for responding to a confirmation operation on the target audio data and the target timbre information, fusing the target audio data and the target timbre information to obtain fused audio data;

[0041] A splicing unit, used to obtain standard audio data corresponding to the remaining text information, and splice the fused audio data and the standard audio data to obtain spliced ​​audio data for the original text information;

[0042] The audio splicing module is also specifically used to display an audio conversion mark in a target area associated with the target text information when generating spliced ​​audio data; the audio conversion mark is used to represent the target timbre information corresponding to the target text information in one or more candidate timbre information.

[0043] Among them, the application interface includes audio download controls and audio sharing controls;

[0044] The device is also specifically used to respond to the trigger operation of the audio download control and download the spliced ​​audio data to the terminal disk;

[0045] The device is also specifically used to respond to a trigger operation on an audio sharing control to share the spliced ​​audio data.

[0046] The audio acquisition unit includes:

[0047] The audio upload subunit is used to respond to the audio upload operation on the audio conversion interface and obtain the auxiliary audio data associated with the target text information;

[0048] A denoising processing subunit, used to perform denoising on the auxiliary audio data to obtain denoised auxiliary audio data;

[0049] A content detection subunit is used to perform content detection on the auxiliary audio data after denoising to obtain a content matching degree between the audio content in the auxiliary audio data after denoising and the target text information;

[0050] The audio determination subunit is used to determine the denoised auxiliary audio data as the target audio data corresponding to the target text information if the content matching degree is greater than the matching degree threshold.

[0051] The denoising processing subunit is specifically used to input the auxiliary frequency signal of the auxiliary audio data into the denoising network model, and perform denoising processing on the auxiliary frequency signal through the denoising network model to obtain the target frequency signal;

[0052] The denoising processing subunit is specifically used to obtain the audio properties of the auxiliary audio data, restore the target frequency signal through the audio properties, and obtain the denoised auxiliary audio data.

[0053] The content detection subunit is specifically used to determine the audio pronunciation sequence of the auxiliary audio data after denoising and the text pronunciation sequence of the target text information;

[0054] The content detection subunit is specifically used to divide the audio pronunciation sequence into at least two audio pronunciation subsequences, determine the subsequence matching degree of each audio pronunciation subsequence relative to the text pronunciation sequence, and take the audio pronunciation subsequence whose subsequence matching degree is greater than the subsequence threshold as the matching pronunciation subsequence;

[0055] The content detection subunit is specifically used to determine the proportion of matching pronunciation subsequences in at least two audio pronunciation subsequences, and use the proportion of matching pronunciation subsequences in at least two audio pronunciation subsequences as the content matching degree between the audio content in the denoised auxiliary audio data and the target text information.

[0056] The content detection subunit is specifically used to perform speech recognition on the denoised auxiliary audio data to obtain audio text information of the denoised auxiliary audio data;

[0057] The content detection subunit is specifically used to determine the text similarity between the audio text information and the target text information, and use the text similarity as the content matching degree between the audio content in the denoised auxiliary audio data and the target text information.

[0058] The audio acquisition unit is further specifically configured to display an error prompt message in the audio conversion interface if the content matching degree is less than or equal to the matching degree threshold; or,

[0059] The audio acquisition unit is also specifically used to display an error prompt interface if the content matching degree is less than or equal to the matching degree threshold, and display error prompt information in the error prompt interface.

[0060] The fusion unit is specifically used to input the target audio data into the fusion network model, and extract the audio text information and audio features in the target audio data through the fusion network model; the audio features include at least one of emotional features, tone features or rhythmic features;

[0061] The fusion unit is specifically used to fuse the audio text information, audio features and target timbre information in the fusion network model to obtain fused audio data.

[0062] The splicing unit includes:

[0063] The speech conversion subunit is used to input the original text information into the speech conversion network model, perform speech conversion on the original text information through the speech conversion network model, and obtain the original audio data corresponding to the original text information;

[0064] a position determination subunit, for determining an audio start position and an audio end position for the target text information in the original audio data according to the position information of the target text information in the original text information, and using the audio data before the audio start position and the audio data after the audio end position of the original audio data as standard audio data corresponding to the remaining text information;

[0065] The audio splicing subunit is used to extract candidate audio data from the fused audio data, and splice the candidate audio data to the audio start position and the audio end position in the standard audio data to obtain the spliced ​​audio data for the original text information.

[0066] The position determination subunit is specifically used to obtain the original duration of the original audio data and the unit duration of each unit text in the original text data;

[0067] The position determination subunit is specifically used to determine the audio start position and audio end position for the target text information in the original audio data according to the original duration, the unit duration and the position information of the target text information in the original text information.

[0068] The audio splicing subunit is specifically used to obtain fused waveform data of the fused audio data, and determine the silent time period in the fused audio data according to the fused waveform data;

[0069] The audio splicing sub-unit is specifically used to cut the fused audio data according to the silent time period, and use the fused audio data with the audio data corresponding to the cut silent time period as the candidate audio data.

[0070] An embodiment of the present application provides a computer device, including: a processor and a memory;

[0071] The processor is connected to the memory, wherein the memory is used to store a computer program. When the computer program is executed by the processor, the computer device executes the method provided in the embodiment of the present application.

[0072] On the one hand, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. The computer program is suitable for being loaded and executed by a processor so that a computer device having the processor executes the method provided by the embodiment of the present application.

[0073] In one aspect, the present application provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the method provided in the present application.

[0074] The embodiment of the present application can fuse the target audio data and the target timbre information through the voice change fusion technology, obtain the fused audio data with the audio information indicated by the target audio data and the standard timbre pronunciation indicated by the target timbre information, and then splice the fused audio data with the standard audio data of the standard timbre pronunciation generated by the speech synthesis technology to obtain the spliced ​​audio data for the original text information. Among them, the fused audio data is generated based on the target text information selected in the original text information, and the standard audio data is generated based on the remaining text information in the original text information except the target text information. Therefore, the embodiment of the present application can merge the fused audio data with audio information and the standard audio data with the standard timbre pronunciation, so that the generated spliced ​​audio data simultaneously meets the dubbing demands of the standard timbre pronunciation without audio information and the standard timbre pronunciation with audio information, so that audio data with different audio information can be generated for different scenes in the original text information, and then the richness of audio creation can be improved while ensuring the efficiency of dubbing, and the quality of audio data can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0076] Figure 1 It is a structural diagram of a network architecture provided by an embodiment of the present application;

[0077] Figure 2 This is a schematic diagram of a scenario for data interaction provided by an embodiment of the present application;

[0078] Figure 3It is a flowchart of an audio data processing method provided in an embodiment of the present application;

[0079] Figure 4a This is a schematic diagram of a scene for displaying original text information provided by an embodiment of the present application;

[0080] Figure 4b This is a schematic diagram of a scene for displaying original text information provided by an embodiment of the present application;

[0081] Figure 4c This is a schematic diagram of a scene for displaying original text information provided by an embodiment of the present application;

[0082] Figure 5a It is a scene schematic diagram of a display audio conversion interface provided by an embodiment of the present application;

[0083] Figure 5b It is a scene schematic diagram of a display audio conversion interface provided by an embodiment of the present application;

[0084] Figure 5c It is a scene schematic diagram of a display audio conversion interface provided by an embodiment of the present application;

[0085] Figure 6a is a schematic diagram of a scenario for obtaining target audio data provided by an embodiment of the present application;

[0086] Figure 6b is a schematic diagram of a scenario for obtaining target audio data provided by an embodiment of the present application;

[0087] Figure 7a It is a schematic diagram of a scene for displaying an audio conversion mark provided in an embodiment of the present application;

[0088] Figure 7b It is a schematic diagram of a scene for displaying an audio conversion mark provided in an embodiment of the present application;

[0089] Figure 8 It is a flowchart of an audio data processing method provided in an embodiment of the present application;

[0090] Fig. 9 It is a flowchart of an audio data processing method provided in an embodiment of the present application;

[0091] Fig.10 It is a flowchart of an audio synthesis solution provided in an embodiment of the present application;

[0092] Fig.11 It is a flowchart of an audio voice changing solution provided in an embodiment of the present application;

[0093] Fig.12is a structural schematic diagram of an audio data processing device provided in an embodiment of the present application;

[0094] Fig.13 It is a structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0095] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0096] It should be understood that artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0097] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. Basic artificial intelligence technologies generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operating / interactive systems, mechatronics and other technologies. Artificial intelligence software technologies mainly include computer vision technology, speech processing technology, natural language processing technology, as well as machine learning / deep learning, autonomous driving, smart transportation and other major directions.

[0098] For details, see Figure 1 , Figure 1 Schematic diagram of a network architecture provided by an embodiment of the present application. Figure 1 As shown, the network architecture may include a service server 2000 and a terminal device cluster. The terminal device cluster may specifically include one or more terminal devices, and the number of terminal devices in the terminal device cluster is not limited. Figure 1As shown, the multiple terminal devices may specifically include terminal device 3000a, terminal device 3000b, terminal device 3000c, ..., terminal device 3000n; terminal device 3000a, terminal device 3000b, terminal device 3000c, ..., terminal device 3000n may respectively be directly or indirectly connected to the business server 2000 via wired or wireless communication, so that each terminal device may exchange data with the business server 2000 via the network connection.

[0099] Each terminal device in the terminal device cluster may include: smart phones, tablet computers, laptop computers, desktop computers, smart homes, smart TVs, wearable devices, vehicle terminals and other smart terminals with data processing functions. Figure 1 Each terminal device in the terminal device cluster shown in FIG. 1 may be installed with an application client. When the application client runs in each terminal device, it may be respectively connected to the above-mentioned Figure 1 Data is exchanged between the business servers 2000 shown. The application client may be an independent client or an embedded sub-client integrated in a client, which is not limited in the present application.

[0100] The application client may specifically include a browser, a vehicle client, a smart home client, an entertainment client, a multimedia client (e.g., a video client), a social client, an information client, and other clients with data processing functions. The vehicle terminal may be a smart terminal in a smart transportation scenario, and the application client on the vehicle terminal may be the above-mentioned vehicle client.

[0101] Among them, the business server 2000 can be a server corresponding to the application client. The business server 2000 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, as well as big data and artificial intelligence platforms.

[0102] For ease of understanding, the embodiments of the present application can be Figure 1 Select one terminal device from the multiple terminal devices shown as the target terminal device. Figure 1 The terminal device 3000b shown is used as a target terminal device, and an application client with data processing function can be installed in the target terminal device. At this time, the target terminal device can exchange data with the service server 2000 through the application client.

[0103] For ease of understanding, in the embodiments of the present application, users who make audio data through the application client may be collectively referred to as target objects, that is, the target object is the object that logs in to the application client. For ease of understanding, in the embodiments of the present application, text information uploaded by the target object in the application client for making audio data may be collectively referred to as original text information. Among them, the audio data made by the target object through the application client may be the original audio data corresponding to the original text information, or it may be spliced ​​audio data generated for the original text information, and the spliced ​​audio data is generated based on the original audio data and the target audio data provided by the target object.

[0104] Wherein, the original audio data may be audio data generated by a TTS (TextToSpeech, text-to-speech, speech synthesis) algorithm; the spliced ​​audio data may be audio data obtained by splicing the standard audio data and the fused audio data in the original audio data; the fused audio data may be audio data obtained by fusing the target audio data and the target timbre information; the target audio data may be audio data uploaded based on the target text information; the target text information may be text information selected in the original text information; and the standard audio data may be audio data of the remaining text information in the original text information except the target text information. Wherein, the standard audio data is audio data synthesized based on the remaining text information, and therefore, the embodiment of the present application may also refer to the standard audio data as synthesized audio data.

[0105] It should be understood that the above network framework can be applied to the fields of audio content production and dubbing. The business scenarios applicable to the above network framework can specifically include: movie dubbing scenarios, game dubbing scenarios, novel dubbing scenarios, etc. The network framework can realize the production of audio data in business scenarios such as movie dubbing, game dubbing, and novel dubbing. The business scenarios applicable to the network framework will not be listed one by one here. For example, in the movie dubbing scenario, the original text information here can be the movie script uploaded by the target object, and the audio data generated by the movie script can realize dubbing for the movie. For another example, in the game dubbing scenario, the original text information here can be the game script uploaded by the target object, and the audio data generated by the game script can be a virtual game (for example, a narrative game (Adventure Game, referred to as AVG game), a narrative game refers to a game in which the player controls the character to conduct a "virtual adventure" as the main line, and an interactive game is carried out in the form of completing a task or solving a puzzle) to realize dubbing. For another example, in the novel dubbing scenario, the original text information here is a novel text that can be uploaded by the target object, and the audio data generated by the novel text can realize the production of an audio novel.

[0106] For ease of understanding, the embodiment of the present application is explained by taking the original text information as a movie script as an example. Therefore, the embodiment of the present application can realize dubbing for the target text information in the movie script (i.e., one or more movie lines in the movie script) on the basis of generating original audio data for the movie script, and then fuse the target audio data generated for one or more movie lines with the target timbre information to obtain target audio data that is compatible with the audio information in the target audio data and fused audio data of the target timbre information, and then obtain the audio data of the movie to which the movie script belongs (i.e., spliced ​​audio data) based on the fused audio data and the original audio data.

[0107] For further understanding, please refer to Figure 2 , Figure 2 Schematic diagram of a scenario for data interaction provided by an embodiment of the present application. Figure 2 The server 20a shown may be the above Figure 1 The service server 2000 in the corresponding embodiment is as follows: Figure 2 The terminal device 20b shown may be the above Figure 1 The target terminal device in the corresponding embodiment. Among them, the terminal device 20b is installed with an application client, and the target object corresponding to the terminal device 20b may be the object 20c, and the application client may be used to display the original text information uploaded by the object 20c through the application client.

[0108] like Figure 2 The terminal device 20b shown can display the original text information uploaded by the object 20c in the application interface of the application client, and send the original text information to the server 20a, so that the server 20a converts the original text information into original audio data through the speech synthesis algorithm. When the object 20c needs to use the target text information of the real audio information, a trigger operation can be performed for the speech conversion function in the application interface. In this way, the terminal device 20b can respond to the trigger operation for the speech conversion function in the application interface, and use the text information selected in the original text information based on the trigger operation as the target text information, and then send the target text information to the server 20a. Among them, the audio information may include but is not limited to tone information, emotional information and rhythm information.

[0109] like Figure 2 As shown, the server 20a can use the text information other than the target text information in the original text information as the remaining text information, and divide the original audio data corresponding to the original text information into two parts according to the target text information and the remaining text information, namely the audio data corresponding to the target text information and the audio data corresponding to the remaining text information. For ease of understanding, the embodiment of the present application can use the audio data corresponding to the remaining text information in the original audio data as the standard audio data.

[0110] Furthermore, if Figure 2 As shown, the terminal device 20b can obtain the target audio data corresponding to the target timbre information and the target text information, and then send the target timbre information and the target audio data to the server 20a in response to the confirmation operation on the target audio data and the target timbre information. The target audio data can be the audio data uploaded by the object 20c through the application client. The object 20c can upload the audio data by reading a recording or uploading an audio file. The audio content in the target audio data matches the target text information.

[0111] The target timbre information may be the timbre information selected by the object 20c from one or more candidate timbre information displayed by the terminal device 20b, and the one or more candidate timbre information may be the timbre information configured by the server 20a. The timbre information may represent the characteristics of a sound, and refers to the sensory characteristics of a sound. Different sound-emitting bodies (for example, musical instruments) have different timbres due to their different materials and structures. In other words, the terminal device 20b may obtain one or more candidate timbre information from the server 20a in response to a trigger operation for the voice conversion function in the application interface, and then display the obtained one or more candidate timbre information, so that the object 20c may select the target timbre information from the one or more candidate timbre information.

[0112] In this way, after receiving the target timbre information and the target audio data, the server 20a can fuse the target timbre information and the target audio data to obtain fused audio data, and then splice the fused audio data and the standard audio data to obtain spliced ​​audio data for the original text information. Among them, the audio content in the standard audio data matches the remaining text information. In other words, the server 20a can replace the audio data corresponding to the target text information in the original audio data through the fused audio data generated based on the target text information to obtain spliced ​​audio data for the original text information.

[0113] The fused audio data includes real audio information (e.g., tone, emotion, rhythm), the timbre of the fused audio data is the selected target timbre information, and the fused audio data can be spliced ​​to the target text information selected in the original text information. The spliced ​​audio data includes both audio data with pure synthetic timbre (i.e., standard audio data) and audio data with real tone, emotion, rhythm and target timbre information (i.e., fused audio data).

[0114] It is understandable that when the object 20c needs to obtain the spliced ​​audio data, it can send an audio acquisition request to the server 20a through the terminal device 20b, so that the server 20a can respond to the audio acquisition request and return the spliced ​​audio data to the terminal device 20b, so that the object 20c can audition or download the spliced ​​audio data in the application client of the terminal device 20b. The audio acquisition request can be an audio audition request or an audio download request, which is not limited in this application.

[0115] It can be seen that the embodiment of the present application can select the target text information whose audio information needs to be modified in the original text information, and then obtain the target audio data with audio information generated for the target text information, and combine the target audio data with the selected target timbre information to obtain fused audio data with audio information and standard timbre. Furthermore, by splicing the fused audio data with the standard audio data corresponding to the remaining text information in the original text information except the target text information, spliced ​​audio data with both audio information and standard timbre can be generated, thereby enriching the audio information in the audio data, improving the richness of the audio creation, and improving the quality of the audio data.

[0116] For further information, see Figure 3 , Figure 3 The method can be executed by a server, or by an application client, or by both the server and the application client. The server can be the above-mentioned Figure 2 The corresponding server 20a in the embodiment, the application client can be the above Figure 2 The application client in the corresponding embodiment. For ease of understanding, the embodiment of the present application takes the method executed by the application client as an example for explanation. Among them, the audio data processing method may include the following steps S101-S103:

[0117] Step S101, displaying original text information in the application interface;

[0118] It should be understood that the application client can display an application interface, wherein the application interface includes a text entry area. Further, the application client can respond to an input operation on the text entry area and display the inputted original text information in the text entry area.

[0119] For easier understanding, see Figure 4a , Figure 4a Schematic diagram of a scene for displaying original text information provided by an embodiment of the present application. Figure 4aThe application interface 40a and the application interface 40b shown may be application interfaces of the application client at different times, and the application interface 40a and the application interface 40b may include a text entry area 41a, wherein the target object corresponding to the application client may be the object 41b.

[0120] like Figure 4a As shown, object 41b can perform input operations on text entry area 41a. In this way, the application client can respond to the input operations performed by object 41b on text entry area 41a, and use the text information input based on the input operations as original text information. The original text information here can be original text information 41c. The original text information 41c here can be "When the sun is at its highest point in the sky, I am hoeing the field, and my sweat is dripping onto the soil. Who knows that every grain of rice on the plate is the result of hard work?".

[0121] At this time, the object 41b can directly enter the original text information 41c in the text entry area 41a. Figure 4a As shown, the application client can switch the application interface from the application interface 40a to the application interface 40b, and display the original text information 41c in the text entry area 41a of the application interface 40b.

[0122] Optionally, it should be understood that the application client can display an application interface. The application interface includes a text upload control. Further, the application client can respond to a trigger operation on the text upload control and display a text selection interface for selecting a text file. Further, the application client can respond to a text file selection operation on the text selection interface and use the text file selected based on the text file selection operation as the target text file. Further, the application client can respond to a text file confirmation operation on the text selection interface and use the text information in the target text file as the original text information and display the original text information in the application interface.

[0123] Among them, the format of the target text file can be txt (Text, text format), doc (Document, document), docx (Office Open Xml Document), pdf (Portable Document Format, portable document format), etc. The formats of the target text files are not listed one by one here.

[0124] For easier understanding, see Figure 4b and Figure 4c , Figure 4b and Figure 4c Schematic diagram of a scene for displaying original text information provided by an embodiment of the present application. Figure 4b and Figure 4cThe application interface 42a, the application interface 42b, the application interface 42c and the application interface 42d shown may be application interfaces of the application client at different times. The application interface 42a may be Figure 4a The application interface 40a and the application interface 42d in the corresponding embodiment can be Figure 4a The application interface 40b, the application interface 42a and the application interface 42d in the corresponding embodiment may include a text entry area 43c. Among them, the target object corresponding to the application client may be object 43b, and object 43b may be Figure 4a The object 41b in the corresponding embodiment.

[0125] like Figure 4b The application interface 42a shown may include a text upload control 43a, and an object 43b may perform a trigger operation on the text upload control 43a. In this way, the application client may respond to the trigger operation performed by the object 43b on the text upload control 43a and display a text selection interface 44a for selecting a text file.

[0126] like Figure 4b As shown, the text selection interface 44a may include folders or text files, the number of folders may be one or more, and the number of text files may be one or more. For ease of understanding, the example of the number of folders being 1 and the number of text files being 1 is used for illustration. Specifically, the one or more folders may include folder J, and the one or more text files may include file 43d.

[0127] It can be understood that object 43b can perform a text file selection operation on the text selection interface 44a (for example, object 43b can perform a text file selection operation on file 43d in the text selection interface 44a). In this way, the application client can respond to the text file selection operation performed by object 43b on file 43d and use the file 43d selected based on the text file selection operation as the target text file.

[0128] Optionally, object 43b may also perform a folder selection operation on folder J, so that the application client may respond to the folder selection operation performed by object 43b on folder J and display one or more text files (e.g., file A1 and file A2, not shown in the figure) under folder J in text selection interface 44a. Further, object 43b may perform a text file selection operation on text files (e.g., file A1) under folder J, so that the application client may respond to the text file selection operation performed by object 43b on file A1 and use file A1 selected based on the text file selection operation as the target text file.

[0129] like Figure 4b and Figure 4c As shown, when the application client responds to the text file selection operation, the text selection interface can be changed from Figure 4b The text selection interface 44a shown switches to Figure 4c The text selection interface 44b highlights the file 43d in the text selection interface 44b, that is, sets the display state of the file 43d to be selected. The text selection interface 44a and the text selection interface 44b may be text selection interfaces of the application client at different times.

[0130] like Figure 4c As shown, the text selection interface 44b may include a text confirmation control 43e, and the object 43b may perform a text file confirmation operation on the text selection interface 44b (for example, the object 43b may perform a text file confirmation operation on the text confirmation control 43e in the text selection interface 44b). In this way, the application client may respond to the text file confirmation operation performed by the object 43b on the text confirmation control 43e, and use the text information in the target text file as the original text information. The original text information here may be the original text information 43f, and the original text information 43f here may be "When the sun is at its highest point in the sky, I am hoeing the field, and my sweat is dripping onto the soil. Who knows that every grain of rice on the plate is the result of hard work?".

[0131] At this time, the object 43b can indirectly enter the original text information 43f in the text entry area 43c. Figure 4b and Figure 4c The application client may switch the application interface from the application interface 42a to the application interface 42d, and display the original text information 43f in the text input area 43c of the application interface 42d.

[0132] Step S102, obtaining target audio data corresponding to target timbre information and target text information;

[0133] Specifically, the application client can respond to the trigger operation for the voice conversion function and obtain the target audio data corresponding to the target timbre information and the target text information. The specific process of the application client responding to the trigger operation for the voice conversion function can be described as follows: the application client can respond to the text selection operation for the original text information and use the text information selected based on the text selection operation as the target text information. Among them, the target text information refers to the text information selected in the original text information, that is, the target text information is the text information selected in the original text information based on the text selection operation, that is, the target text information is the text information selected in the original text information based on the trigger operation for the voice conversion function. Further, the application client can respond to the voice conversion operation for the target text information and display the audio conversion interface. Further, the application client can obtain the target timbre information in the audio conversion interface. Further, the application client can respond to the audio upload operation for the audio conversion interface and obtain the target audio data corresponding to the target text information. Among them, the audio content in the target audio data matches the target text information.

[0134] For easier understanding, see Figure 5a , Figure 5a is a schematic diagram of a scene showing an audio conversion interface provided by an embodiment of the present application. Figure 5a The application interface 50a and the application interface 50b shown in the figure may be the application interfaces of the application client at different times. The application interface 50a and the application interface 50b may include a text entry area 51a. The application interface 50a may be Figure 4a The application interface 40b or Figure 4c The corresponding application interface 42d in the embodiment. The target object corresponding to the application client may be the object 51c.

[0135] like Figure 5a The text entry area 51a shown can display the original text information 51b, and the object 51c can perform a text selection operation on the original text information 51b. In this way, the application client can respond to the text selection operation performed by the object 51c on the original text information 51b, and use the text information 51d selected based on the text selection operation as the target text information. The text information 51d here can be "Who knows what's on the plate".

[0136] like Figure 5a As shown, when responding to the text selection operation, the application client can switch the application interface from application interface 50a to application interface 50b, and highlight the target text information in application interface 50b, that is, set the display state of the target text information to the selected state.

[0137] The application interface includes a first voice conversion control. It should be understood that the application client can respond to the trigger operation for the first voice conversion control and display the audio conversion interface. For ease of understanding, the specific process of the application client responding to the trigger operation for the first voice conversion control can be found in Figure 5b , Figure 5b is a schematic diagram of a scene showing an audio conversion interface provided by an embodiment of the present application. Figure 5b The application interface 52a shown may be Figure 5a The application interface 50b in the corresponding embodiment, Figure 5b The object 53c shown may be Figure 5a The object 51c in the corresponding embodiment.

[0138] like Figure 5b As shown, the application interface 52a may include a first voice conversion control 53b, and the object 53c may perform a trigger operation on the first voice conversion control 53b. In this way, the application client may respond to the trigger operation performed by the object 53c on the first voice conversion control 53b, and display the audio conversion interface 52b for the target text information 53a in the selected state.

[0139] Optionally, it should be understood that the application client can respond to the trigger operation for the target text information and display a text control list. The text control list includes the second voice conversion control. Further, the application client can respond to the trigger operation for the second voice conversion control and display an audio conversion interface. For ease of understanding, the specific process of the application client responding to the trigger operation for the second voice conversion control is shown in Figure 5c , Figure 5c is a schematic diagram of a scene showing an audio conversion interface provided by an embodiment of the present application. Figure 5c The application interface 54a shown may be Figure 5a The application interface 50b, the application interface 54a and the application interface 54b in the corresponding embodiment may be the application interfaces of the application client at different times. Figure 5c The object 55c shown may be Figure 5a The object 51c in the corresponding embodiment.

[0140] like Figure 5c As shown, object 55c can perform a trigger operation on the target text information 55a in the selected state, so that the application client can respond to the trigger operation performed by object 55c on the target text information 55a in the selected state, switch the application interface from application interface 54a to application interface 54b, and display the text control list in application interface 54b. The text control list may include the second voice conversion control 55b.

[0141] like Figure 5cAs shown, the object 55c can perform a trigger operation on the second voice conversion control 55b, so that the application client can respond to the trigger operation performed by the object 55c on the second voice conversion control 55b and display the audio conversion interface 54c for the selected target text information 55a.

[0142] Among them, the audio conversion interface includes a recording start control. It should be understood that the application client can respond to the trigger operation for the recording start control and display the recording stop control in the audio conversion interface. Further, the application client can respond to the trigger operation for the recording stop control, obtain the audio data recorded by the target object in the time interval between responding to the trigger operation for the recording start control and responding to the trigger operation for the recording stop control, and use the audio data recorded by the target object as the target audio data corresponding to the target text information. Further, the application client can display the audio file identifier corresponding to the target audio data in the audio conversion interface.

[0143] For easier understanding, see Figure 6a and Figure 6b , Figure 6a and Figure 6b Schematic diagram of a scenario for obtaining target audio data provided by an embodiment of the present application. Figure 6a and Figure 6b The audio conversion interface 60a, the audio conversion interface 60b and the audio conversion interface 60c shown in the figure can be the audio conversion interfaces of the application client at different times. The audio conversion interface 60c can be Figure 5b The audio conversion interface 52b or Figure 5c The target object corresponding to the application client may be object 61a, and object 61a may be Figure 4a The object 41b in the corresponding embodiment.

[0144] like Figure 6a The audio conversion interface 60a shown may include a recording start control 61b, and the object 61a may perform a trigger operation on the recording start control 61b. In this way, the application client may respond to the trigger operation performed by the object 61a on the recording start control 61b, switch the audio conversion interface from the audio conversion interface 60a to the audio conversion interface 60b, and display the recording stop control 61c in the audio conversion interface 60b.

[0145] like Figure 6aAs shown, object 61a can perform a trigger operation on the recording stop control 61c. In this way, the application client can respond to the trigger operation performed by object 61a on the recording stop control 61c, obtain the audio data recorded by object 61a within the time interval (for example, 5 seconds) between responding to the trigger operation on the recording start control 61b and responding to the trigger operation on the recording stop control 61c, and use the audio data recorded by object 61a as the target audio data associated with the target text information (i.e., "Who knows what's on the plate").

[0146] like Figure 6b As shown, when the application client responds to the trigger operation of the recording stop control 61c, the audio conversion interface can be switched from the audio conversion interface 60b to the audio conversion interface 60c, and the audio file identifier 61d corresponding to the target audio data is displayed in the audio conversion interface 60c. The audio file identifier 61d can be used to audition the target audio data, and the time bar in the audio file identifier 61d can be used to adjust the playback progress of the target audio data.

[0147] Optionally, when responding to a trigger operation on a recording start control, the application client may also display a recording pause control in the audio conversion interface. Further, the application client may respond to a trigger operation on a recording pause control, use the audio data recorded by the target object in the time interval between responding to the trigger operation on the recording start control and responding to the trigger operation on the recording pause control as the first audio data, and display a recording continuation control in the audio conversion interface. Further, the application client may respond to a trigger operation on a recording continuation control, and continue to obtain the audio data recorded by the target object. Further, the application client may respond to a trigger operation on a recording stop control, use the audio data recorded by the target object in the time interval between responding to the trigger operation on the recording continuation control and responding to the trigger operation on the recording stop control as the second audio data. Further, the application client may splice the first audio data and the second audio data to obtain the target audio data corresponding to the target text information.

[0148] Optionally, the audio conversion interface includes an audio upload control. It should be understood that the application client can respond to a trigger operation for the audio upload control to display an audio selection interface for selecting an audio file. Further, the application client can respond to an audio file selection operation for the audio selection interface, using the audio file selected based on the audio file selection operation as the target audio file. Further, the application client can respond to an audio file confirmation operation for the audio selection interface, using the audio data in the target audio file as the target audio data corresponding to the target text information. Further, the application client can display the audio file identifier corresponding to the target audio data in the audio conversion interface.

[0149] It should be understood that Figure 6a The audio conversion interface 60a shown may include an audio upload control 62a. The specific process of the application client obtaining the target audio data based on the audio upload control 62a can be referred to above. Figure 4b and Figure 4c The description of obtaining the target text information based on the text upload control 43a in the corresponding embodiment will not be repeated here.

[0150] Please see again Figure 6a and Figure 6b When the application client obtains the audio conversion interface 60c based on the recording start control 61b or the audio upload control 62a, the audio conversion interface 60c may include an update recording start control 62b and an update audio upload control 62c. The update recording start control 62b may re-upload the updated audio data in the same way as the recording start control 61b, and the update audio upload control 62c may re-upload the updated audio data in the same way as the audio upload control 62a. The updated audio data may be used to update the target audio data.

[0151] The audio conversion interface includes one or more candidate timbre information. It should be understood that the application client can respond to the timbre selection operation for one or more candidate timbre information and use the candidate timbre information selected based on the timbre selection operation as the target timbre information. Further, the application client can highlight the target timbre information in the audio conversion interface.

[0152] For easier understanding, see Figure 7a , Figure 7a is a schematic diagram of a scene showing an audio conversion mark provided by an embodiment of the present application. Figure 7a The audio conversion interface 70a and the audio conversion interface 70b shown in the figure can be audio conversion interfaces of the application client at different times. The audio conversion interface 70a can be Figure 6b The corresponding audio conversion interface 60c in the embodiment. The target object corresponding to the application client may be the object 72a.

[0153] like Figure 7a As shown, the audio conversion interface 70a may include one or more candidate timbre information, and the one or more candidate timbre information may specifically include candidate timbre information 71a, candidate timbre information 71b, candidate timbre information 71c, and candidate timbre information 71d. Object 72a may perform a timbre selection operation on one or more candidate timbre information (e.g., candidate timbre information 71a), so that the application client may respond to the timbre selection operation performed by object 72a on candidate timbre information 71a, and use the candidate timbre information 71a selected based on the timbre selection operation as the target timbre information.

[0154] like Figure 7a As shown, when responding to a timbre selection operation, the application client can switch the audio conversion interface from the audio conversion interface 70a to the audio conversion interface 70b, and highlight the target timbre information 71a in the audio conversion interface 70b, that is, set the display status of the target timbre information 71a to the selected state.

[0155] Step S103, obtaining the spliced ​​audio data for the original text information.

[0156] Specifically, the application client can respond to the confirmation operation for the target audio data and the target timbre information, and obtain the spliced ​​audio data for the original text information. The spliced ​​audio data is obtained by splicing the fused audio data and the standard audio data corresponding to the remaining text information; the remaining text information is the text information in the original text information except the target text information; the audio content in the standard audio data matches the remaining text information. The fused audio data is obtained by fusing the target audio data and the target timbre information.

[0157] It should be understood that when generating the spliced ​​audio data, the application client can display an audio conversion identifier in the target area associated with the target text information, wherein the audio conversion identifier is used to represent the target timbre information corresponding to the target text information in one or more candidate timbre information.

[0158] For easier understanding, see Figure 7b , Figure 7b is a schematic diagram of a scene showing an audio conversion mark provided by an embodiment of the present application. Figure 7b The audio conversion interface 70b shown is Figure 7a The audio conversion interface 70b, the application interface 70c and the audio conversion interface 70b in the corresponding embodiment are different interfaces of the application client.

[0159] like Figure 7b The audio conversion interface 70b shown may include a fusion confirmation control 72b, and the object 72a may perform a confirmation operation on the target audio data and the target timbre information 71a (for example, the object 72a may perform a confirmation operation on the fusion confirmation control 72b). In this way, the application client may respond to the confirmation operation performed by the object 72a on the fusion confirmation control 72b, and may display an audio conversion identifier 72d (i.e., "imitation piglet") in the target area 72c of the application interface 70c associated with the target text information. The audio conversion identifier 72d is used to represent the target timbre information 71a corresponding to the target text information (i.e., "Who knows what's on the plate") in one or more candidate timbre information.

[0160] Among them, the target area 72c can be located at any position in the application interface 70c to reduce the impact on the display effect of the original text information. Generally, the target area 72c can be located at the edge of the target text information. For ease of understanding, the embodiment of the present application takes the example of the target area 72c being located on the right side of the target text information for explanation.

[0161] Optionally, the application client may respond to a trigger operation for an audio conversion identifier and display a timbre selection interface. The timbre selection interface may include one or more candidate timbre information. Further, the application client may respond to a trigger operation for one or more candidate timbre information and use the candidate timbre information selected based on the trigger operation for one or more candidate timbre information as updated timbre information. Further, the application client may respond to a confirmation operation for the updated timbre information and obtain updated concatenated audio data for the original text information. The updated concatenated audio data is obtained by concatenating the updated fused audio data and the standard audio data corresponding to the remaining text information; the updated fused audio data is obtained by fusing the target audio data and the updated timbre information.

[0162] Optionally, it should be understood that when generating the updated spliced ​​audio data, the application client can display an updated audio conversion identifier in the target area associated with the target text information, wherein the updated audio conversion identifier is used to represent the updated timbre information corresponding to the target text information in one or more candidate timbre information.

[0163] It can be seen that the embodiment of the present application can fuse the target audio data and the target timbre information through the voice change fusion technology, and obtain the fused audio data with the audio information indicated by the target audio data and the standard timbre pronunciation indicated by the target timbre information, and then splice the fused audio data with the standard audio data of the standard timbre pronunciation generated by the speech synthesis technology to obtain the spliced ​​audio data for the original text information. Among them, the fused audio data is generated based on the target text information selected in the original text information, and the standard audio data is generated based on the remaining text information in the original text information except the target text information. Therefore, the embodiment of the present application can merge the fused audio data with audio information and the standard audio data with the standard timbre pronunciation, so that the generated spliced ​​audio data simultaneously meets the dubbing demands of the standard timbre pronunciation without audio information and the standard timbre pronunciation with audio information, so that audio data with different audio information can be generated for different scenes in the original text information, and then the richness of audio creation can be improved while ensuring the efficiency of dubbing, and the quality of audio data can be improved.

[0164] For further information, see Figure 8 , Figure 8The method can be executed by a server, or by an application client, or by both the server and the application client. The server can be the above-mentioned Figure 2 The corresponding server 20a in the embodiment, the application client can be the above Figure 2 The application client in the corresponding embodiment. For ease of understanding, the embodiment of the present application takes the method executed by the application client as an example for explanation. Among them, the audio data processing method may include the following steps S201-S203:

[0165] Step S201, displaying an audio conversion mark in a target area associated with target text information;

[0166] Among them, the application interface includes audio download controls and audio sharing controls.

[0167] Please see again Figure 7b , Figure 7b The standard timbre information 80a shown can represent the timbre information corresponding to the standard audio data, the standard timbre information 80a can be "elegant", and the speaking speed of the standard audio data is 1.0x (i.e., 1 times the speed). The application client can respond to the trigger operation for the standard timbre information 80a, obtain the sample timbre information, and update the standard timbre information 80a through the sample timbre information. At this time, the application client can update the standard timbre information 80a corresponding to the standard audio data to the sample timbre information. Among them, the sample timbre information and the standard timbre information 80a are both obtained from the timbre library.

[0168] Optionally, the same timbre information in the timbre library may correspond to different dialects. For example, the timbre information "beautiful girl" in the timbre library may correspond to "Sichuan dialect", and the timbre information "beautiful girl" may correspond to "Hunan dialect".

[0169] like Figure 7b The audio download control 80b shown can be used to download the spliced ​​audio data, such as Figure 7b The audio sharing control 80c shown can be used to share the spliced ​​audio data. The specific process of the application client responding to the trigger operation of the audio download control 80b can be seen in step S202, and the specific process of the application client responding to the audio sharing control 80c can be seen in step S203. In addition, the application interface 70c can also include an audio audition control 80d, which can be used to audition the spliced ​​audio data. When the application client responds to the trigger operation of the audio audition control 80d, the spliced ​​audio data can be played on the application interface 70c.

[0170] Step S202, in response to a trigger operation on the audio download control, downloading the spliced ​​audio data to a terminal disk;

[0171] Specifically, the application client can respond to the trigger operation for the audio download control and display the audio format list. The audio format list includes one or more audio format information. Further, the application client can respond to the trigger operation for the one or more audio format information, use the audio format selected based on the trigger operation for the one or more audio format information as the target audio format information, and download the spliced ​​audio data with the target audio format information to the terminal disk.

[0172] The audio format list may be used to display audio format information supported by the spliced ​​audio data, for example, MP3 (Moving Picture Experts Group Audio Layer III) format and WAV (Windows Media Audio) format.

[0173] Among them, when the application client responds to the trigger operation on the audio format list, it can automatically download the spliced ​​audio data to the default directory, or it can display the directory selection interface, and then when responding to the directory selection operation on the target selection interface, it can load the spliced ​​audio data to the target directory selected based on the directory selection operation.

[0174] Step S203, responding to the triggering operation on the audio sharing control, sharing the spliced ​​audio data.

[0175] Specifically, the application client can respond to the trigger operation for the audio sharing control and display the sharing platform list. The sharing platform list includes one or more sharing platform information. Further, the application client can respond to the trigger operation for one or more sharing platform information, use the sharing platform information selected based on the trigger operation for one or more sharing platform information as the target sharing platform information, and share the spliced ​​audio data to the social sharing platform to which the target sharing platform information belongs.

[0176] The social sharing platform may be a sharing platform within the application client or a sharing platform outside the application client (i.e., a sharing platform in other application clients). In other words, the application client may share the spliced ​​audio data to the application client itself or to other application clients.

[0177] Optionally, the application client can respond to the trigger operation on the audio sharing control to obtain the audio link of the spliced ​​audio data. The audio link is used to open the spliced ​​audio data, and the terminal device to which the application client belongs can display the audio link when responding to the paste operation on the audio link, and further display the spliced ​​audio data pointed to by the audio link when responding to the open operation on the audio link.

[0178] It can be seen that the target object in the embodiment of the present application can automatically produce original audio data through the platform tool of productization, and the original audio data is corrected by the target audio data uploaded, that is, the pronunciation, tone, emotion, etc. are adjusted, so that the audio data produced is more standard, emotional, and rich in tone. At the same time, the target object can also review the generated spliced ​​audio data, thereby achieving a significant reduction in the cost of dubbing expenses and improving the user experience of making dubbing audio. In addition, after generating the spliced ​​audio data, the target object can also download, audition or share the spliced ​​audio data, improving the user experience of the target object using the platform tool.

[0179] For further information, see Fig. 9 , Fig. 9 The method can be executed by a server, or by an application client, or by both the server and the application client. The server can be the above-mentioned Figure 2 The corresponding server 20a in the embodiment, the application client can be the above Figure 2 The application client in the corresponding embodiment. For ease of understanding, the embodiment of the present application takes the method executed by the application client as an example for explanation. Among them, the audio data processing method may include the following steps S301-S307:

[0180] Step S301, displaying original text information in the application interface;

[0181] The specific process of the application client displaying the original text information in the application interface can be found in the above Figure 4a , Figure 4b and Figure 4c The description of the corresponding embodiments will not be repeated here.

[0182] Step S302, in response to a text selection operation on the original text information, taking the text information selected based on the text selection operation as the target text information;

[0183] The specific process of the application client determining the target text information based on the text selection operation can be found in the above Figure 5a The description of the corresponding embodiments will not be repeated here.

[0184] Step S303, in response to the voice conversion operation for the target text information, displaying an audio conversion interface;

[0185] The specific process of the application client displaying the audio conversion interface based on the voice conversion operation can be found in the above Figure 5b and Figure 5c The description of the corresponding embodiments will not be repeated here.

[0186] Step S304, obtaining target timbre information in the audio conversion interface;

[0187] The specific process of the application client obtaining the target timbre information in the audio conversion interface can be found in the above Figure 7a The description of the corresponding embodiments will not be repeated here.

[0188] Step S305, in response to the audio upload operation on the audio conversion interface, obtaining target audio data corresponding to the target text information;

[0189] Specifically, the application client can respond to the audio upload operation for the audio conversion interface and obtain the auxiliary audio data associated with the target text information. Furthermore, the application client can perform denoising on the auxiliary audio data to obtain denoised auxiliary audio data. Furthermore, the application client can perform content detection on the denoised auxiliary audio data to obtain the content match between the audio content in the denoised auxiliary audio data and the target text information. Furthermore, if the content match is greater than the match threshold, the application client can determine the denoised auxiliary audio data as the target audio data corresponding to the target text information.

[0190] Among them, the auxiliary audio data can be the audio data uploaded through the recording start control and the recording stop control, that is, the auxiliary audio data can be the audio data recorded by the target object in the time interval between responding to the trigger operation of the recording start control and responding to the trigger operation of the recording stop control. Optionally, the auxiliary audio data can be the audio data uploaded through the audio upload control, that is, the auxiliary audio data can be the audio data in the target audio file selected by the target object in the audio selection interface. Among them, the audio data in the target audio file can be audio data recorded in advance, or it can be audio data with audio information generated by a model.

[0191] It should be understood that the specific process of the application client performing denoising on the auxiliary audio data can be described as follows: the application client can input the auxiliary frequency signal of the auxiliary audio data into the denoising network model, and perform denoising on the auxiliary frequency signal through the denoising network model to obtain the target frequency signal. Furthermore, the application client can obtain the audio properties of the auxiliary audio data, and restore the target frequency signal through the audio properties to obtain the denoised auxiliary audio data.

[0192] The audio attributes may include but are not limited to sampling rate, bit depth, duration, and number of channels. Restoring the target frequency information through the audio attributes can ensure that the sampling rate, bit depth, duration, and number of channels of the generated denoised auxiliary audio data remain unchanged after denoising.

[0193] Among them, the denoising process is carried out in the frequency domain. The frequency domain signal is concerned with frequency distribution and amplitude. Therefore, the key to denoising is to extract the spectrum of the noise, and then perform an inverse compensation operation on the noisy speech (i.e., the auxiliary audio data) according to the spectrum of the noise, so as to obtain the denoised speech (i.e., the denoised auxiliary audio data). In practical applications, after Fourier transforming the digital sampling signal, the spectrum of the signal (i.e., the auxiliary frequency signal) can be obtained. After the processing in the frequency domain is completed, the inverse Fourier transform can be used to convert the signal from the frequency domain to the time domain.

[0194] Among them, the denoising network model can be used to denoise the auxiliary audio data (i.e., reduce noise). The denoising process can remove the noise in the auxiliary audio data whose frequency signal is too high or too low. There are many types of noise here, including white noise with stable spectrum, unstable impulse noise and fluctuating noise. It should be understood that the embodiment of the present application does not limit the model type of the denoising network model. At the same time, the embodiment of the present application does not limit the specific algorithm used for denoising.

[0195] It should be understood that the specific process of the application client performing content detection on the auxiliary audio data after denoising can be described as: the application client can determine the audio pronunciation sequence of the auxiliary audio data after denoising and the text pronunciation sequence of the target text information. Further, the application client can divide the audio pronunciation sequence into at least two audio pronunciation subsequences, determine the subsequence matching degree of each audio pronunciation subsequence relative to the text pronunciation sequence, and use the audio pronunciation subsequence whose subsequence matching degree is greater than the subsequence threshold as the matching pronunciation subsequence. Further, the application client can determine the proportion of matching pronunciation subsequences in at least two audio pronunciation subsequences, and use the proportion of matching pronunciation subsequences in at least two audio pronunciation subsequences as the content matching degree (i.e., pronunciation completeness) between the audio content in the auxiliary audio data after denoising and the target text information.

[0196] Among them, the audio pronunciation sequence and the text pronunciation sequence are comparisons at the pronunciation phoneme level. The audio pronunciation sequence can be the phoneme sequence corresponding to the auxiliary audio data after denoising, and the text pronunciation sequence can be the phoneme sequence corresponding to the target text information. In other words, the phoneme sequence corresponding to the auxiliary audio data after denoising can be called the audio content in the auxiliary audio data after denoising. For example, when the target text information is "Who knows the food on the plate", the text pronunciation sequence can be "sheizhipanzhongcan"; when the audio pronunciation in the auxiliary audio data after denoising is "Who knows the food on the plate", the audio pronunciation sequence can be "sheizhipanzhongcanya".

[0197] Among them, the subsequence matching degree can be used to represent the matching degree of each audio pronunciation subsequence, and the content matching degree can be used to represent the matching degree of the audio pronunciation sequence. If the subsequence matching degree is greater than the subsequence threshold, it can be determined that the audio pronunciation subsequence matches the text pronunciation sequence; if the content matching degree is greater than the matching degree threshold, it can be determined that the audio pronunciation sequence matches the text pronunciation sequence. It should be understood that the embodiment of the present application does not limit the specific values ​​of the subsequence threshold and the matching degree threshold. For example, the subsequence threshold can be 90%, and the matching degree threshold can be 60%.

[0198] The application client may compare each audio pronunciation subsequence with the text pronunciation sequence to determine the subsequence matching degree corresponding to each audio pronunciation subsequence. Optionally, the application client may also divide the text pronunciation sequence into at least two text pronunciation subsequences, and compare each audio pronunciation subsequence with each text pronunciation subsequence, thereby determining the subsequence matching degree corresponding to each audio pronunciation subsequence. The number of at least two text pronunciation subsequences and the number of at least two audio pronunciation subsequences may be the same or different.

[0199] The application client may divide the text pronunciation sequence into at least two text pronunciation subsequences according to the sequence length of the audio pronunciation subsequence; optionally, the application client may also divide the text pronunciation sequence into at least two text pronunciation subsequences according to the duration of the audio pronunciation subsequence in the auxiliary audio data after denoising. The sequence length of each audio pronunciation subsequence may be the same or different, and the sequence length of each text pronunciation subsequence may be the same or different.

[0200] Wherein, it is understandable that the application client can use the N-Gram model to obtain the audio candidate sequence associated with each audio pronunciation subsequence, and obtain the text candidate sequence associated with each text pronunciation subsequence. For example, the embodiment of the present application can set the maximum N-Gram retained to be 4-Gram. At this time, the number of phonemes in the audio candidate sequence and the text candidate sequence can be 1, 2, 3 or 4. In this way, the application client can determine the candidate sequence similarity between the audio candidate sequence and the text candidate sequence under 1-Gram, 2-Gram, 3-Gram and 4-Gram respectively, and then perform weighted summation on the candidate sequence similarity under 1-Gram, 2-Gram, 3-Gram and 4-Gram, and obtain the subsequence matching degree corresponding to the audio pronunciation subsequence.

[0201] Optionally, the application client may also directly determine the sequence similarity between the audio pronunciation sequence and the text pronunciation sequence without dividing the audio pronunciation sequence and the text pronunciation sequence, and use the sequence similarity as the content match between the audio content in the auxiliary audio data after denoising and the target text information. It should be understood that the application embodiment does not limit the specific method for determining the sequence similarity. For example, the embodiment of the application can determine the sequence similarity between the audio pronunciation sequence and the text pronunciation sequence by cosine similarity.

[0202] Optionally, it should be understood that the specific process of the application client performing content detection on the auxiliary audio data after denoising can be described as follows: the application client can perform speech recognition on the auxiliary audio data after denoising to obtain audio text information of the auxiliary audio data after denoising. Further, the application client can determine the text similarity between the audio text information and the target text information, and use the text similarity as the content matching degree between the audio content in the auxiliary audio data after denoising and the target text information.

[0203] Among them, the application client can obtain the audio text information of the auxiliary audio data after denoising through ASR (Automatic Speech Recognition). In other words, the audio text information of the auxiliary audio data after denoising can be called the audio content in the auxiliary audio data after denoising. For example, the target text information can be "Who knows what's on the plate", and the audio text information of the auxiliary audio data after denoising can be "Who knows what's on the plate". It should be understood that the embodiments of the present application do not limit the specific method for determining text similarity. For example, the embodiments of the present application can determine the text similarity between the audio text information and the target text information through cosine similarity.

[0204] Optionally, if the content matching degree is less than or equal to the matching degree threshold, the application client may display an error prompt message in the audio conversion interface. Optionally, if the content matching degree is less than or equal to the matching degree threshold, the application client may display an error prompt interface and display an error prompt message in the error prompt interface. For example, the error prompt message may be "The uploaded audio does not match the text, please re-upload the audio!". If the content matching degree is less than or equal to the matching degree threshold, the target audio data needs to be re-uploaded.

[0205] Step S306, in response to the confirmation operation on the target audio data and the target timbre information, the target audio data and the target timbre information are merged to obtain merged audio data;

[0206] Specifically, the application client can input the target audio data into the fusion network model, and extract the audio text information and audio features in the target audio data through the fusion network model. The audio features include at least one of emotional features, tone features, or rhythmic features. Furthermore, the application client can fuse the audio text information, audio features, and target timbre information in the fusion network model to obtain fused audio data.

[0207] Wherein, when the audio features include emotional features, tone features and prosodic features, the application client may fuse the audio text information, emotional features, tone features, prosodic features and target timbre information to obtain fused audio data having tone information, emotional information and prosodic information. Optionally, when the audio features include tone features and prosodic features, the application client may fuse the audio text information, tone features, prosodic features and target timbre information to obtain fused audio data having tone information and prosodic information.

[0208] It should be understood that the embodiment of the present application does not limit the model type of the fusion network model. At the same time, the embodiment of the present application does not limit the specific algorithm used to fuse the target audio data and the target timbre information.

[0209] Step S307, obtaining the standard audio data corresponding to the remaining text information, splicing the fused audio data and the standard audio data, and obtaining the spliced ​​audio data for the original text information.

[0210] Specifically, the application client can input the original text information into the speech conversion network model, perform speech conversion on the original text information through the speech conversion network model, and obtain the original audio data corresponding to the original text information. Furthermore, the application client can determine the audio start position and audio end position for the target text information in the original audio data based on the position information of the target text information in the original text information, and use the audio data before the audio start position and the audio data after the audio end position of the original audio data as the standard audio data corresponding to the remaining text information. Furthermore, the application client can extract candidate audio data from the fused audio data, splice the candidate audio data to the audio start position and audio end position in the standard audio data, and obtain spliced ​​audio data for the original text information.

[0211] It should be understood that the embodiment of the present application does not limit the model type of the voice conversion network model. Among them, the application client can use the audio data of the original audio data before the audio start position as the first standard audio data, and the audio data of the original audio data after the audio end position as the second standard audio data. Among them, the first standard audio data and the second standard audio data can be collectively referred to as standard audio data. In this way, the application client can splice the header of the candidate audio data to the tail of the first standard audio data (i.e. the audio start position), and splice the tail of the candidate audio data to the header of the second standard audio data (i.e. the audio end position).

[0212] It is understandable that the original audio data in the embodiment of the present application may have the same audio information, and the audio information here may include but is not limited to tone information, emotional information and rhythmic information. Optionally, the original audio data in the embodiment of the present application may also have different audio information, and the different audio information is generated when the original text information is converted into the original audio data through the speech conversion network model, that is, the speech conversion network model can convert the original text information into the original audio data while predicting different audio information corresponding to different text information in the original text information. At this time, since the audio information predicted by the speech conversion network model may be inaccurate, the embodiment of the present application can be used to modify the different audio information in the original audio data.

[0213] Optionally, if the position information of the target text information in the original text information is the header position (i.e., the target text information is located at the header of the original text information), the application client can determine the audio end position for the target text information in the original audio data, and use the audio data after the audio end position of the original audio data as the standard audio data corresponding to the remaining text information. Furthermore, the application client can extract candidate audio data from the fused audio data, and splice the candidate audio data to the audio end position in the standard audio data, that is, splice the candidate audio data before the standard audio data, to obtain the spliced ​​audio data for the original text information.

[0214] Optionally, similarly, if the position information of the target text information in the original text information is the tail position (i.e., the target text information is located at the tail of the original text information), the application client can determine the audio start position for the target text information in the original audio data, and use the audio data before the audio start position of the original audio data as the standard audio data corresponding to the remaining text information. Furthermore, the application client can extract candidate audio data from the fused audio data, and splice the candidate audio data to the audio start position in the standard audio data, that is, splice the candidate audio data to the standard audio data to obtain the spliced ​​audio data for the original text information.

[0215] It should be understood that the application client can obtain the original duration of the original audio data and the unit duration of each unit text in the original text data. Further, the application client can determine the audio start position and audio end position for the target text information in the original audio data according to the original duration, the unit duration and the position information of the target text information in the original text information.

[0216] Among them, the candidate audio data can be the voiced segment in the fused audio data (that is, the candidate audio data can be the audio data with sound in the fused audio data). According to the original duration of the original audio data predicted by the speech conversion network model and the unit duration of each unit text, the corresponding position of the candidate audio data in the original audio data can be calculated, and the starting time (that is, the time of the audio starting position in the original audio data) and the ending time (that is, the time of the audio ending position in the original audio data) of the corresponding position are given. Finally, the audio data in the time period is replaced by the candidate audio data to complete the sound conversion process and output the final audio data after sound conversion (that is, the spliced ​​audio data).

[0217] It should be understood that the application client obtains the fused waveform data of the fused audio data, and determines the silent time period in the fused audio data according to the fused waveform data. Further, the application client can cut the fused audio data according to the silent time period, and use the fused audio data with the audio data corresponding to the silent time period cut out as the candidate audio data.

[0218] It is understandable that in order to ensure a natural transition between the front and back audio data of the fused audio data in the original audio data, the application client can cut the front and back silent parts of the voice-changed audio data (i.e., the fused audio data) and only retain the voiced segment part (i.e., the candidate audio data). Optionally, the application client can also cut the silent part in the fused audio data to obtain the candidate audio data.

[0219] The fused waveform data can be used to generate an audio waveform graph of the fused audio data. In the audio waveform graph, the amplitude of the sound waves in the silent parts at the beginning and the end is very small, while the amplitude of the effective speech part is relatively large. Therefore, the silent part in the fused audio data can be removed by fusing the waveform data to obtain the candidate audio data with sound.

[0220] Optionally, if the target text information selected by the target object in the original text information is the original text information itself, that is, the text information selected based on the text selection operation is the original text information, the application client can generate fused audio data for the original text information, and then directly use the fused audio data as the spliced ​​audio data for the original text information.

[0221] Optionally, the application client may also directly input the remaining text information into the speech conversion network model, perform speech conversion on the remaining text information through the speech conversion network model, and obtain standard audio data corresponding to the remaining text information.

[0222] For easier understanding, see Fig.10 , Fig.10 FIG. 1 is a flow chart of an audio synthesis solution provided in an embodiment of the present application. Fig.10 The flowchart shown mainly includes the following main processing steps: text import (i.e., step S11), selecting the text segment to be voice-changed (i.e., step S12), recording / uploading audio (i.e., step S13 and step S14), selecting the voice-changed tone (i.e., step S15), voice-changed processing and replacing the audio segment (i.e., step S16), synthesizing the full-text audio and downloading the audio (i.e., step S17), etc.

[0223] like Fig.10As shown, the application client can execute step S11 to display the original text information in the application interface, and then execute step S12 to respond to the text selection operation for the original text information, and use the text information selected based on the text selection operation as the target text information (i.e., the text segment that needs to be voice-changed), and then display the audio conversion interface for the target text information. Further, the application client can execute step S13 or step S14 to respond to the audio upload operation for the audio conversion interface, and obtain the target audio data corresponding to the target text information. The target audio data here can be the audio data recorded by step S13, or the audio data in the file uploaded by step S14.

[0224] like Fig.10 As shown, the application client can execute step S15, select the target timbre information from one or more candidate timbre information in the audio conversion interface, and then execute step S16, respond to the confirmation operation on the target audio data and the target timbre information, merge the target audio data and the target timbre information to obtain merged audio data. The merged audio data can be used to replace the audio data corresponding to the target text information in the original audio data.

[0225] like Fig.10 As shown, the application client can execute step S17, obtain the standard audio data corresponding to the remaining text information, splice the fused audio data and the standard audio data, obtain the spliced ​​audio data (i.e., full-text audio) for the original text information, and then download the full-text audio.

[0226] For easier understanding, see Fig.11 , Fig.11 Schematic diagram of a flow chart of an audio voice changing solution provided in an embodiment of the present application. Fig.11 The flowchart shown mainly describes the specific execution functions of the sound conversion module and the implementation methods of each function after the target object uploads the target audio data and before the sound conversion function is completed. Among them, the sound conversion module is mainly composed of four parts: audio denoising module (i.e. step S23), content detection module (i.e. step S24), voice changing module (i.e. step S26), and audio splicing module (i.e. step S27).

[0227] like Fig.11As shown, the application client can execute step S21, which can represent opening the application client in the terminal device. Further, the application client can execute step S22, and receive the auxiliary audio data uploaded by the target object through step S22. The auxiliary audio data here can be the audio data recorded by the target object or the audio data in the uploaded file. Further, the application client can execute step S23, and denoise the auxiliary audio data through the denoising module in step S23. The denoising module can be used to remove the noise inside the auxiliary audio data to obtain the denoised auxiliary audio data.

[0228] Further, the application client can execute step S24, and perform content detection on the denoised auxiliary audio data through the content detection module in step S24. The content detection module can take the denoised auxiliary audio data and the selected text of the target object (i.e., the target text information) as input, and is used to compare the internal text content of the denoised auxiliary audio data (i.e., the audio content in the denoised auxiliary audio data, i.e., the audio pronunciation sequence or audio text information) with the target text information to see whether it is consistent, and obtain a content detection result.

[0229] Among them, the application client can execute step S25, and judge the content detection result through step S25. If the content detection result indicates that the audio content in the auxiliary audio data after denoising is inconsistent with the target text information, the application client can execute step S29, and return an error mark through step S29, prompting the target object to upload inconsistent audio content, and prompting the target object to re-upload the target audio data. Further, the application client can execute step S30, and determine whether the target object re-uploads the target audio data in step S30. If the target object re-uploads the target audio data, execute step S22; if the target object does not re-upload the target audio data, execute step S28 and end the process.

[0230] Optionally, if the content detection result indicates that the audio content in the auxiliary audio data after denoising is consistent with the target text information, the application client can execute step S26, and the auxiliary audio data (i.e., target audio data) after denoising and the target timbre information are merged by the voice changing module in step S26, that is, the target audio data is changed in voice by the target timbre information. The voice changing module can be input with the target audio data and the target timbre, and is responsible for converting the timbre information in the target audio data uploaded with emotion, tone and rhythm into the target timbre information needed by the target object, while retaining the emotion, tone and rhythm information in the target audio data, completing the voice changing function, and outputting the audio after voice changing (i.e., fusion audio data). In the internal flow of the voice changing module, it is necessary to extract the features related to content (i.e., audio text information), emotion, tone, and rhythm information from the target audio data, and the extraction of these information will be disturbed by noise, and then affect the sound conversion (i.e., voice changing) effect, so it is necessary to add a denoising module before the voice changing module. Wherein, the emotion related information will be characterized by the fundamental frequency feature.

[0231] Further, the application client can execute step S27, and the fused audio data output by the voice changing module is spliced ​​into the original audio data by the audio splicing module in step S27 according to the position of the text content (i.e., target text information) selected by the target object, that is, the corresponding part of the original audio data is replaced by the fused audio data, and finally the content audio (i.e., spliced ​​audio data) required by the target object is formed. It is understandable that when the target object does not need to modify the spliced ​​audio data, the application client executes step S28 and the process ends.

[0232] It can be seen that the embodiment of the present application provides a technology that combines machine dubbing with artificial dubbing. First, the original text information is imported into a text dubbing tool (i.e., an application client); secondly, the original text information is synthesized into a standard timbre voice audio (i.e., original audio data) based on a speech synthesis algorithm; again, the text (i.e., the target text information) that needs to be read with rich emotions in the original text information is selected, and the target text information is read artificially with emotions and intonation; then, the emotion, tone and rhythm of the artificial reading segment are extracted by voice changing technology, and this part of emotion, tone and rhythm is combined with the standard timbre model (i.e., the target timbre information) to generate a synthetic timbre voice audio (i.e., fused audio data) with artificial reading emotion, tone and rhythm; finally, the generated synthetic timbre voice audio with artificial reading emotion, tone and rhythm is spliced ​​with other unselected standard timbre voice audio (i.e., standard audio data) in the order of the original text to form an audio file having both synthetic timbre audio (fused audio data) and standard audio data with artificial reading emotion, tone and rhythm.

[0233] For further information, see Fig.12 , Fig.12 1 is a structural diagram of an audio data processing device provided in an embodiment of the present application. The audio data processing device 1 may include: a text display module 11, an audio acquisition module 12, and an audio splicing module 13;

[0234] The text display module 11 is used to display the original text information in the application interface;

[0235] The text display module 11 is specifically used to display the application interface; the application interface includes a text entry area;

[0236] The text display module 11 is specifically used to respond to the input operation on the text entry area and display the input original text information in the text entry area.

[0237] Among them, the text display module 11 is specifically used to display the application interface; the application interface includes a text upload control;

[0238] The text display module 11 is specifically used to respond to the trigger operation on the text upload control and display a text selection interface for selecting a text file;

[0239] The text display module 11 is specifically used to respond to the text file selection operation on the text selection interface and use the text file selected based on the text file selection operation as the target text file;

[0240] The text display module 11 is specifically used to respond to the text file confirmation operation on the text selection interface, use the text information in the target text file as the original text information, and display the original text information in the application interface.

[0241] The audio acquisition module 12 is used to acquire target audio data corresponding to target timbre information and target text information; the target text information refers to text information selected from the original text information; the audio content in the target audio data matches the target text information;

[0242] The audio acquisition module 12 includes: a text selection unit 121, a voice conversion unit 122, a timbre acquisition unit 123, and an audio acquisition unit 124;

[0243] A text selection unit 121, configured to respond to a text selection operation on the original text information and use the text information selected based on the text selection operation as the target text information;

[0244] The voice conversion unit 122 is used to respond to the voice conversion operation on the target text information and display the audio conversion interface;

[0245] Wherein, the application interface includes a first voice conversion control;

[0246] The voice conversion unit 122 is specifically configured to respond to a trigger operation on the first voice conversion control and display an audio conversion interface.

[0247] The speech conversion unit 122 is specifically used to respond to the trigger operation on the target text information and display the text control list; the text control list includes the second speech conversion control;

[0248] The voice conversion unit 122 is specifically configured to respond to a trigger operation on the second voice conversion control and display an audio conversion interface.

[0249] The timbre acquisition unit 123 is used to acquire target timbre information in the audio conversion interface;

[0250] Wherein, the audio conversion interface includes one or more candidate timbre information;

[0251] The timbre acquisition unit 123 is specifically configured to respond to a timbre selection operation for one or more candidate timbre information and use the candidate timbre information selected based on the timbre selection operation as the target timbre information;

[0252] The timbre acquisition unit 123 is also specifically used to highlight the target timbre information in the audio conversion interface.

[0253] The audio acquisition unit 124 is used to respond to the audio upload operation on the audio conversion interface and acquire the target audio data corresponding to the target text information.

[0254] Among them, the audio conversion interface includes a recording start control;

[0255] The audio acquisition unit 124 is specifically used to respond to the trigger operation of the recording start control and display the recording stop control in the audio conversion interface;

[0256] The audio acquisition unit 124 is specifically used to respond to the trigger operation on the recording stop control, acquire the audio data recorded by the target object in the time interval between responding to the trigger operation on the recording start control and responding to the trigger operation on the recording stop control, and use the audio data recorded by the target object as the target audio data corresponding to the target text information;

[0257] The audio acquisition unit 124 is further specifically configured to display the audio file identifier corresponding to the target audio data in the audio conversion interface.

[0258] Among them, the audio conversion interface includes an audio upload control;

[0259] The audio acquisition unit 124 is specifically used to respond to the trigger operation on the audio upload control and display an audio selection interface for selecting an audio file;

[0260] The audio acquisition unit 124 is specifically configured to respond to an audio file selection operation on the audio selection interface and use the audio file selected based on the audio file selection operation as a target audio file;

[0261] The audio acquisition unit 124 is specifically used to respond to the audio file confirmation operation on the audio selection interface and use the audio data in the target audio file as the target audio data corresponding to the target text information;

[0262] The audio acquisition unit 124 is further specifically configured to display the audio file identifier corresponding to the target audio data in the audio conversion interface.

[0263] The audio acquisition unit 124 includes: an audio uploading subunit 1241, a denoising processing subunit 1242, a content detection subunit 1243, and an audio determination subunit 1244;

[0264] The audio upload subunit 1241 is used to respond to the audio upload operation on the audio conversion interface and obtain the auxiliary audio data associated with the target text information;

[0265] A denoising processing subunit 1242 is used to perform denoising on the auxiliary audio data to obtain denoised auxiliary audio data;

[0266] The denoising processing subunit 1242 is specifically used to input the auxiliary frequency signal of the auxiliary audio data into the denoising network model, and perform denoising processing on the auxiliary frequency signal through the denoising network model to obtain a target frequency signal;

[0267] The denoising processing subunit 1242 is specifically used to obtain audio properties of the auxiliary audio data, restore the target frequency signal according to the audio properties, and obtain the denoised auxiliary audio data.

[0268] The content detection subunit 1243 is used to perform content detection on the auxiliary audio data after denoising to obtain a content matching degree between the audio content in the auxiliary audio data after denoising and the target text information;

[0269] The content detection subunit 1243 is specifically used to determine the audio pronunciation sequence of the auxiliary audio data after denoising and the text pronunciation sequence of the target text information;

[0270] The content detection subunit 1243 is specifically used to divide the audio pronunciation sequence into at least two audio pronunciation subsequences, determine the subsequence matching degree of each audio pronunciation subsequence relative to the text pronunciation sequence, and take the audio pronunciation subsequence whose subsequence matching degree is greater than the subsequence threshold as the matching pronunciation subsequence;

[0271] The content detection subunit 1243 is specifically used to determine the proportion of matching pronunciation subsequences in at least two audio pronunciation subsequences, and use the proportion of matching pronunciation subsequences in at least two audio pronunciation subsequences as the content matching degree between the audio content in the denoised auxiliary audio data and the target text information.

[0272] The content detection subunit 1243 is specifically used to perform speech recognition on the denoised auxiliary audio data to obtain audio text information of the denoised auxiliary audio data;

[0273] The content detection subunit 1243 is specifically used to determine the text similarity between the audio text information and the target text information, and use the text similarity as the content matching degree between the audio content in the denoised auxiliary audio data and the target text information.

[0274] The audio determination subunit 1244 is used to determine the denoised auxiliary audio data as the target audio data corresponding to the target text information if the content matching degree is greater than the matching degree threshold.

[0275] The specific implementation of the audio uploading subunit 1241, the denoising processing subunit 1242, the content detecting subunit 1243 and the audio determining subunit 1244 can be found in the above Fig. 9 The description of step S305 in the corresponding embodiment will not be repeated here.

[0276] The audio acquisition unit 124 is further specifically configured to display an error message in the audio conversion interface if the content matching degree is less than or equal to the matching degree threshold; or,

[0277] The audio acquisition unit 124 is further specifically configured to display an error prompt interface if the content matching degree is less than or equal to the matching degree threshold, and to display error prompt information in the error prompt interface.

[0278] The specific implementation of the text selection unit 121, the voice conversion unit 122, the timbre acquisition unit 123 and the audio acquisition unit 124 can be found in the above Figure 3 The description of step S102 in the corresponding embodiment will not be repeated here.

[0279] The audio splicing module 13 is used to obtain spliced ​​audio data for the original text information; the spliced ​​audio data is obtained by splicing the fused audio data and the standard audio data corresponding to the remaining text information; the remaining text information is the text information in the original text information except the target text information; the fused audio data is obtained by fusing the target audio data and the target timbre information; the audio content in the standard audio data matches the remaining text information.

[0280] The audio splicing module 13 includes: a fusion unit 131 and a splicing unit 132;

[0281] A fusion unit 131 is used to respond to a confirmation operation on the target audio data and the target timbre information, fuse the target audio data and the target timbre information to obtain fused audio data;

[0282] The fusion unit 131 is specifically used to input the target audio data into the fusion network model, and extract the audio text information and audio features in the target audio data through the fusion network model; the audio features include at least one of emotional features, tone features or rhythm features;

[0283] The fusion unit 131 is specifically used to fuse the audio text information, the audio features and the target timbre information in the fusion network model to obtain fused audio data.

[0284] A splicing unit 132 is used to obtain standard audio data corresponding to the remaining text information, and splice the fused audio data and the standard audio data to obtain spliced ​​audio data for the original text information;

[0285] The splicing unit 132 includes: a voice conversion subunit 1321, a position determination subunit 1322, and an audio splicing subunit 1323;

[0286] The speech conversion subunit 1321 is used to input the original text information into the speech conversion network model, perform speech conversion on the original text information through the speech conversion network model, and obtain the original audio data corresponding to the original text information;

[0287] The position determination subunit 1322 is used to determine the audio start position and the audio end position of the target text information in the original audio data according to the position information of the target text information in the original text information, and use the audio data before the audio start position and the audio data after the audio end position of the original audio data as the standard audio data corresponding to the remaining text information;

[0288] The position determination subunit 1322 is specifically used to obtain the original duration of the original audio data and the unit duration of each unit text in the original text data;

[0289] The position determination subunit 1322 is specifically used to determine the audio start position and audio end position of the target text information in the original audio data according to the original duration, the unit duration and the position information of the target text information in the original text information.

[0290] The audio splicing subunit 1323 is used to extract candidate audio data from the fused audio data, and splice the candidate audio data to the audio start position and the audio end position in the standard audio data to obtain spliced ​​audio data for the original text information.

[0291] The audio splicing subunit 1323 is specifically used to obtain fused waveform data of the fused audio data, and determine the silent time period in the fused audio data according to the fused waveform data;

[0292] The audio splicing sub-unit 1323 is specifically used to cut the fused audio data according to the silent time period, and use the fused audio data with the audio data corresponding to the silent time period cut off as the candidate audio data.

[0293] The specific implementation of the voice conversion subunit 1321, the position determination subunit 1322 and the audio splicing subunit 1323 can be found in the above Fig. 9 The description of step S307 in the corresponding embodiment will not be repeated here.

[0294] The audio splicing module 13 is also specifically used to display an audio conversion mark in a target area associated with the target text information when generating spliced ​​audio data; the audio conversion mark is used to represent the target timbre information corresponding to the target text information in one or more candidate timbre information.

[0295] The specific implementation of the fusion unit 131 and the splicing unit 132 can be found in the above Figure 3 The description of step S103 in the corresponding embodiment will not be repeated here.

[0296] Among them, the application interface includes audio download controls and audio sharing controls;

[0297] The audio data processing device 1 is also specifically used to respond to a trigger operation on an audio download control and download the spliced ​​audio data to a terminal disk;

[0298] The audio data processing device 1 is also specifically used to respond to a trigger operation on the audio sharing control and share the spliced ​​audio data.

[0299] The specific implementation of the text display module 11, the audio acquisition module 12 and the audio splicing module 13 can be found in the above Figure 3 In the corresponding embodiment, steps S101 to S103, Figure 8 In the corresponding embodiment, steps S201 to S203, and Fig. 9 The description of steps S301 to S307 in the corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of the same method will not be repeated here either.

[0300] For further information, see Fig.13 , Fig.13 Schematic diagram of the structure of a computer device provided in an embodiment of the present application. Fig.13 As shown, the computer device 1000 may include: a processor 1001, a network interface 1004 and a memory 1005. In addition, the above-mentioned computer device 1000 may also include: a user interface 1003, and at least one communication bus 1002. Among them, the communication bus 1002 is used to realize the connection and communication between these components. Among them, in some embodiments, the user interface 1003 may include a display screen (Display), a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. Optionally, the network interface 1004 may include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory, or it may be a non-volatile memory (non-volatile memory), such as at least one disk storage. Optionally, the memory 1005 may also be at least one storage device located away from the aforementioned processor 1001. As Fig.13 As shown, the memory 1005 as a computer-readable storage medium may include an operating system, a network communication module, a user interface module, and a device control application program.

[0301] In such Fig.13 In the computer device 1000 shown, the network interface 1004 can provide a network communication function; the user interface 1003 is mainly used to provide an input interface for the user; and the processor 1001 can be used to call the device control application stored in the memory 1005 to achieve:

[0302] Display original text information in the application interface;

[0303] Obtaining target audio data corresponding to target timbre information and target text information; the target text information refers to text information selected from the original text information; the audio content in the target audio data matches the target text information;

[0304] Acquire spliced ​​audio data for the original text information; the spliced ​​audio data is obtained by splicing the fused audio data and the standard audio data corresponding to the remaining text information; the remaining text information is the text information in the original text information except the target text information; the fused audio data is obtained by fusing the target audio data and the target timbre information; the audio content in the standard audio data matches the remaining text information.

[0305] It should be understood that the computer device 1000 described in the embodiment of the present application can execute the above Figure 3 , Figure 8 and Fig. 9 The description of the audio data processing method in the corresponding embodiment can also be performed as described above. Fig.12 The description of the audio data processing device 1 in the corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of the same method will not be repeated here either.

[0306] In addition, it should be pointed out here that: the embodiment of the present application also provides a computer-readable storage medium, and the computer-readable storage medium stores a computer program executed by the audio data processing device 1 mentioned above, and the computer program includes program instructions. When the processor executes the program instructions, it can execute the above-mentioned Figure 3 , Figure 8 and Fig. 9 The description of the audio data processing method in the corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of the same method will not be repeated. For technical details not disclosed in the computer-readable storage medium embodiment involved in the present application, please refer to the description of the method embodiment of the present application.

[0307] In addition, it should be noted that: the embodiment of the present application also provides a computer program product or a computer program, which may include computer instructions, and the computer instructions may be stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor may execute the computer instructions, so that the computer device executes the above Figure 3 , Figure 8 and Fig. 9 The description of the audio data processing method in the corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of the same method will not be repeated. For technical details not disclosed in the computer program product or computer program embodiment involved in this application, please refer to the description of the method embodiment of this application.

[0308] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0309] The above disclosure is only the preferred embodiment of the present application, which certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. An audio data processing method, characterized in that: include: Display original text information in the application interface; In response to the speech conversion operation for the target text information, an audio conversion interface is displayed, and target timbre information is acquired in the audio conversion interface; the target text information refers to the text information selected from the original text information; In response to an audio upload operation on the audio conversion interface, target audio data corresponding to the target text information is acquired, wherein the audio content in the target audio data matches the target text information, the target audio data includes real-person audio information corresponding to the target text information, and the real-person audio information includes tone information, emotion information, and rhythm information; In response to a confirmation operation on the target audio data and the target timbre information, the target audio data is input into a fusion network model, and audio text information and audio features in the target audio data are extracted through the fusion network model, wherein the audio features include tone features, emotional features, and rhythm features corresponding to the real person audio information; In the fusion network model, the audio text information, the audio features and the target timbre information are fused to obtain fused audio data; Acquire standard audio data corresponding to the remaining text information, and splice the fused audio data and the standard audio data to obtain spliced ​​audio data for the original text information; the remaining text information is text information in the original text information except the target text information; the audio content in the standard audio data matches the remaining text information; Wherein, the step of obtaining the target audio data corresponding to the target text information includes: Acquiring auxiliary audio data associated with the target text information; Inputting the auxiliary frequency signal of the auxiliary audio data into a denoising network model, and performing denoising processing on the auxiliary frequency signal through the denoising network model to obtain a target frequency signal; Acquire audio properties of the auxiliary audio data, and restore the target frequency signal according to the audio properties to obtain denoised auxiliary audio data; Performing content detection on the denoised auxiliary audio data to obtain a content matching degree between the audio content in the denoised auxiliary audio data and the target text information; If the content matching degree is less than or equal to the matching degree threshold, an error prompt message is displayed; If the content matching degree is greater than the matching degree threshold, the denoised auxiliary audio data is determined as the target audio data corresponding to the target text information.

2. The method according to claim 1, characterized in that The displaying of the original text information in the application interface includes: Displaying an application interface; the application interface includes a text entry area; In response to an input operation on the text entry area, the input original text information is displayed in the text entry area.

3. The method according to claim 1, characterized in that The displaying of the original text information in the application interface includes: Display an application interface; the application interface includes a text upload control; In response to a trigger operation on the text upload control, a text selection interface for selecting a text file is displayed; In response to a text file selection operation on the text selection interface, taking a text file selected based on the text file selection operation as a target text file; In response to a text file confirmation operation on the text selection interface, the text information in the target text file is used as original text information, and the original text information is displayed in the application interface.

4. The method according to claim 1, characterized in that In response to the voice conversion operation for the target text information, before displaying the audio conversion interface, the method further includes: In response to a text selection operation on the original text information, the text information selected based on the text selection operation is used as the target text information.

5. The method according to claim 1, characterized in that The application interface includes a first voice conversion control; The step of responding to the voice conversion operation of the target text information and displaying the audio conversion interface includes: In response to a trigger operation on the first voice conversion control, an audio conversion interface is displayed.

6. The method according to claim 1, characterized in that The step of responding to the voice conversion operation of the target text information and displaying the audio conversion interface includes: In response to a trigger operation on the target text information, displaying a text control list; the text control list includes a second voice conversion control; In response to a trigger operation on the second voice conversion control, an audio conversion interface is displayed.

7. The method according to claim 1, characterized in that The audio conversion interface includes a recording start control; The step of responding to the audio upload operation on the audio conversion interface and obtaining target audio data corresponding to the target text information includes: In response to a trigger operation on the recording start control, displaying a recording stop control in the audio conversion interface; In response to a trigger operation on the recording stop control, audio data recorded by the target object in a time interval between responding to a trigger operation on the recording start control and responding to a trigger operation on the recording stop control is obtained, and the audio data recorded by the target object is used as target audio data corresponding to the target text information; The method further comprises: The audio file identifier corresponding to the target audio data is displayed in the audio conversion interface.

8. The method according to claim 1, characterized in that The audio conversion interface includes an audio upload control; The step of responding to the audio upload operation on the audio conversion interface and obtaining target audio data corresponding to the target text information includes: In response to a trigger operation on the audio upload control, displaying an audio selection interface for selecting an audio file; In response to an audio file selection operation on the audio selection interface, taking an audio file selected based on the audio file selection operation as a target audio file; In response to an audio file confirmation operation on the audio selection interface, using audio data in the target audio file as target audio data corresponding to the target text information; The method further comprises: The audio file identifier corresponding to the target audio data is displayed in the audio conversion interface.

9. The method according to claim 1, characterized in that: The audio conversion interface includes one or more candidate timbre information; The step of obtaining target timbre information in the audio conversion interface includes: In response to a timbre selection operation for the one or more candidate timbre information, taking the candidate timbre information selected based on the timbre selection operation as target timbre information; The method further comprises: The target timbre information in the audio conversion interface is highlighted.

10. The method according to claim 1, characterized in that The method further comprises: When the spliced ​​audio data is generated, an audio conversion mark is displayed in a target area associated with the target text information; the audio conversion mark is used to represent the target timbre information corresponding to the target text information in one or more candidate timbre information.

11. The method according to claim 1, characterized in that: The application interface includes an audio download control and an audio sharing control; The method further comprises: In response to a trigger operation on the audio download control, the spliced ​​audio data is downloaded to a terminal disk; In response to a triggering operation on the audio sharing control, the spliced ​​audio data is shared.

12. The method according to claim 1, characterized in that The performing content detection on the auxiliary audio data after denoising to obtain a content matching degree between the audio content in the auxiliary audio data after denoising and the target text information includes: Determine an audio pronunciation sequence of the denoised auxiliary audio data and a text pronunciation sequence of the target text information; Divide the audio pronunciation sequence into at least two audio pronunciation subsequences, determine the subsequence matching degree of each audio pronunciation subsequence relative to the text pronunciation sequence, and take the audio pronunciation subsequence whose subsequence matching degree is greater than the subsequence threshold as the matching pronunciation subsequence; Determine the proportion of the matching pronunciation subsequence in the at least two audio pronunciation subsequences, and use the proportion of the matching pronunciation subsequence in the at least two audio pronunciation subsequences as the content matching degree between the audio content in the denoised auxiliary audio data and the target text information.

13. The method according to claim 1, characterized in that The performing content detection on the auxiliary audio data after denoising to obtain a content matching degree between the audio content in the auxiliary audio data after denoising and the target text information includes: Performing speech recognition on the auxiliary audio data after denoising to obtain audio text information of the auxiliary audio data after denoising; The text similarity between the audio text information and the target text information is determined, and the text similarity is used as the content matching degree between the audio content in the denoised auxiliary audio data and the target text information.

14. The method according to claim 1, characterized in that The error message displayed includes: Displaying an error message in the audio conversion interface; or, An error prompt interface is displayed, in which error prompt information is displayed.

15. The method according to claim 1, characterized in that The step of acquiring the standard audio data corresponding to the remaining text information and splicing the fused audio data with the standard audio data to obtain the spliced ​​audio data for the original text information includes: Inputting the original text information into a speech conversion network model, performing speech conversion on the original text information through the speech conversion network model, and obtaining original audio data corresponding to the original text information; According to the position information of the target text information in the original text information, determine the audio start position and the audio end position for the target text information in the original audio data, and use the audio data before the audio start position and the audio data after the audio end position of the original audio data as the standard audio data corresponding to the remaining text information; Extract candidate audio data from the fused audio data, and splice the candidate audio data to the audio start position and the audio end position in the standard audio data to obtain spliced ​​audio data for the original text information.

16. The method according to claim 15, characterized in that The step of determining the audio start position and the audio end position of the target text information in the original audio data according to the position information of the target text information in the original text information includes: Acquire the original duration of the original audio data and the unit duration of each unit text in the original text data; According to the original duration, the unit duration and the position information of the target text information in the original text information, an audio start position and an audio end position for the target text information are determined in the original audio data.

17. The method according to claim 15, characterized in that The extracting candidate audio data from the fused audio data comprises: Acquire fused waveform data of the fused audio data, and determine a silent time period in the fused audio data according to the fused waveform data; The fused audio data is cut according to the silent time period, and the fused audio data from which the audio data corresponding to the silent time period is cut is used as candidate audio data.

18. An audio data processing device, characterized in that: include: A text display module, used to display original text information in the application interface; An audio acquisition module, used to acquire target audio data corresponding to target timbre information and target text information; The target text information refers to the text information selected from the original text information; the audio content in the target audio data matches the target text information, the target audio data includes real-person audio information corresponding to the target text information, and the real-person audio information includes tone information, emotion information and rhythm information; An audio splicing module, used to obtain spliced ​​audio data for the original text information; The audio acquisition module comprises: A voice conversion unit, used to respond to the voice conversion operation for the target text information and display an audio conversion interface; A timbre acquisition unit, used to acquire target timbre information in the audio conversion interface; An audio acquisition unit, configured to respond to an audio upload operation on the audio conversion interface and acquire target audio data corresponding to the target text information; The audio acquisition unit comprises: An audio uploading subunit, configured to respond to an audio uploading operation on the audio conversion interface and obtain auxiliary audio data associated with the target text information; A denoising processing subunit is used to input the auxiliary frequency signal of the auxiliary audio data into a denoising network model, perform denoising on the auxiliary frequency signal through the denoising network model, and obtain a target frequency signal; obtain audio properties of the auxiliary audio data, and restore the target frequency signal through the audio properties to obtain denoised auxiliary audio data; A content detection subunit, configured to perform content detection on the auxiliary audio data after denoising to obtain a content matching degree between the audio content in the auxiliary audio data after denoising and the target text information; an audio determination subunit, configured to determine the denoised auxiliary audio data as target audio data corresponding to the target text information if the content matching degree is greater than a matching degree threshold; The audio acquisition unit is further configured to display an error prompt message if the content matching degree is less than or equal to a matching degree threshold; The audio splicing module includes: A fusion unit, for responding to a confirmation operation on the target audio data and the target timbre information, inputting the target audio data into a fusion network model, extracting audio text information and audio features in the target audio data through the fusion network model, wherein the audio features include tone features, emotional features, and rhythmic features corresponding to the real person audio information; in the fusion network model, fusing the audio text information, the audio features, and the target timbre information to obtain fused audio data; A splicing unit is used to obtain standard audio data corresponding to the remaining text information, splice the fused audio data and the standard audio data, and obtain spliced ​​audio data for the original text information; the remaining text information is the text information in the original text information except the target text information, and the audio content in the standard audio data matches the remaining text information.

19. A computer device, characterized in that: include: Processor and memory; The processor is connected to the memory, wherein the memory is used to store a computer program, and the processor is used to call the computer program so that the computer device executes the method according to any one of claims 1 to 17.

20. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which is suitable for being loaded and executed by a processor, so that a computer device having the processor executes the method according to any one of claims 1 to 17.

21. A computer program product, characterized in that The computer program product includes computer instructions, which are stored in a computer-readable storage medium and are suitable for being read and executed by a processor, so that a computer device having the processor executes the method according to any one of claims 1 to 17.

Images