Voice editing method and device, storage medium, electronic equipment and product

Abstract

The invention relates to the technical field of voice processing, and particularly provides a voice editing method and device, a storage medium, electronic equipment and a product, and the voice editing method can comprise the steps: obtaining an original audio text similar to an edited audio text; calculating the edited audio text and the original audio text to obtain at least two editing intervals; and carrying out editing processing on the at least two editing intervals at the same time to obtain a speech synthesis signal. Some embodiments of the invention can improve the efficiency of voice editing.

Description

Technical Field

[0001] The present application relates to the field of speech processing technology, and in particular to a speech editing method, device, storage medium, electronic device and product. Background Art

[0002] With the continuous development of speech processing technology, speech editing systems have realized the function of partially modifying audio. Traditional speech editing systems can only modify one interval at a time when editing speech. In other words, if the original text to the edited text contains at least two intervals that need to be modified, it needs to be processed twice, and the editing efficiency is low.

[0003] Therefore, how to provide an efficient technical solution for voice editing becomes a technical problem that needs to be solved urgently. Summary of the invention

[0004] The purpose of some embodiments of the present application is to provide a method, device, storage medium, electronic device and product for voice editing. The technical solutions of the embodiments of the present application can improve the efficiency and accuracy of voice editing.

[0005] In a first aspect, some embodiments of the present application provide a method for speech editing, comprising: obtaining an original audio text similar to an edited audio text; performing calculations on the edited audio text and the original audio text to obtain at least two edit intervals; and performing editing processing on the at least two edit intervals simultaneously to obtain a speech synthesis signal.

[0006] Some embodiments of the present application automatically obtain and edit the original audio text similar to the audio text, calculate the two to obtain at least two editing intervals, and finally edit the at least two editing intervals at the same time to obtain a speech synthesis signal. The embodiments of the present application can automatically search for the original audio text and edit multiple editing intervals at one time, which can improve the efficiency of speech editing and has high practicality.

[0007] In some embodiments, obtaining the original audio text similar to the edited audio text includes: confirming that the edited audio text exists in an original text in the original text set, and then using the one original text as the original audio text; or calculating the edit distance between the edited audio text and each original text in the original text set; using the original text with the smallest edit distance as the original audio text, wherein the edited audio text does not exist in the original text with the smallest edit distance.

[0008] Some embodiments of the present application determine the optimal original audio text by calculating the edit distance between the edited audio text and the original text in the original text set or confirming whether the edited audio text exists in the original text, without the need for manual specification, thereby reducing labor costs.

[0009] In some embodiments, the edit distance characterizes the number of inserted, deleted, or replaced characters required to transform each original text into the edited audio text.

[0010] Some embodiments of the present application can quickly select the original audio text through the edit distance.

[0011] In some embodiments, the calculating the edited audio text and the original audio text to obtain at least two edit intervals includes: identifying an area to be edited from the original audio text to the edited audio text, wherein the area to be edited includes at least two areas; determining the boundary of the area to be edited to obtain the at least two edit intervals.

[0012] Some embodiments of the present application identify the area to be edited in the original audio text and then determine the corresponding boundary to obtain the editing area to ensure the accuracy of voice editing.

[0013] In some embodiments, determining the boundary of the area to be edited to obtain the at least two editing intervals includes: obtaining the start and end time nodes of each area in the area to be edited in the original audio text; using the start and end time nodes as the boundary of each area to determine the at least two editing intervals.

[0014] Some embodiments of the present application determine the boundary of each area in the area to be edited by determining the start and end time nodes of the original audio text, so as to ensure accurate processing of the edited speech.

[0015] In some embodiments, the simultaneously editing the at least two editing intervals to obtain a speech synthesis signal includes: creating a masking matrix equal to the length of the original audio text, wherein the elements of each editing interval in the masking matrix have different values ​​from the elements of the non-editing interval; after masking the edited audio text with the masking matrix, performing speech synthesis processing on each editing interval to obtain a speech editing result; and integrating the speech editing result with the audio of the non-editing interval to obtain the speech synthesis signal.

[0016] Some embodiments of the present application create a masking matrix to mask and distinguish the editing interval from the non-editing interval, thereby achieving simultaneous and accurate editing of the editing interval, obtaining an accurate speech synthesis signal, and having high efficiency.

[0017] In a second aspect, some embodiments of the present application provide a speech editing device, comprising: an acquisition module for acquiring an original audio text similar to an edited audio text; a calculation module for calculating the edited audio text and the original audio text to obtain at least two edit intervals; an editing module for simultaneously editing the at least two edit intervals to obtain a speech synthesis signal.

[0018] In a third aspect, some embodiments of the present application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can implement the method described in any embodiment of the first aspect.

[0019] In a fourth aspect, some embodiments of the present application provide an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor can implement a method as described in any embodiment of the first aspect when executing the program.

[0020] In a fifth aspect, some embodiments of the present application provide a computer program product, wherein the computer program product comprises a computer program, wherein the computer program, when executed by a processor, can implement the method described in any embodiment of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of some embodiments of the present application, the drawings required for use in some embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0022] Figure 1 A system diagram of voice editing provided for some embodiments of the present application;

[0023] Figure 2 One of the flow charts of the voice editing method provided in some embodiments of the present application;

[0024] Figure 3 A second flow chart of a method for voice editing provided in some embodiments of the present application;

[0025] Figure 4 A block diagram of a voice editing device provided in some embodiments of the present application;

[0026] Figure 5 A schematic diagram of an electronic device is provided for some embodiments of the present application. DETAILED DESCRIPTION

[0027] The technical solutions in some embodiments of the present application will be described below in conjunction with the drawings in some embodiments of the present application.

[0028] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0029] In the related art, the voice editing system is used to partially modify the audio. Its input is audio A, text B corresponding to the audio, and text C corresponding to the audio to be synthesized, and the output is the audio corresponding to text C. For example, if text B is "take a train to Chengdu" and text C is "take a train to Kunming", then the audio corresponding to "take a train to Kunming" will be output. Compared with the speech synthesis system, the speech editing system only changes part of the text and then performs speech synthesis. If only a few words are changed, the synthesis effect of the speech editing system will be better and the synthesis time will be shorter.

[0030] At present, the speech editing system has the following two disadvantages: first, only one interval can be changed at a time; second, the original text corresponding to the text to be edited must be manually specified. For example, the original text is "take a train from Beijing to Shanghai", and the text to be edited is "take a train from Tianjin to Chengdu". There are two editing intervals from the original text to the edited text, one is "Beijing" to "Tianjin", and the other is "Shanghai" to "Chengdu". For traditional speech editing systems, there are two ways to achieve this: one is to perform two synthesises, each time only changing one part, which is less efficient; the other is to merge the intervals to be edited, that is, to change "Beijing to Shanghai" to "Tianjin to Chengdu". However, if this method is used, if the number of words between the edited intervals is relatively large, the speech synthesis effect will deteriorate. Moreover, the original text corresponding to the text to be edited must be manually specified during current speech editing; for example, the text corresponding to the audio to be synthesized is "take a train from Tianjin to Chengdu", and it must be manually selected to determine that the original text is "take a train from Beijing to Shanghai". This will consume a lot of manpower costs in actual applications and bring a lot of trouble.

[0031] It can be seen from the above-mentioned related technologies that the existing voice editing method is inefficient and has high labor costs.

[0032] In view of this, some embodiments of the present application provide a method for voice editing, which can automatically obtain and edit the original audio text corresponding to the audio text; then determine at least two editing intervals based on the edited audio text and the original audio text; finally, edit the at least two voice editing intervals at the same time to obtain a speech synthesis signal. Some embodiments of the present application can automatically search for the original audio text without manual intervention, and can edit multiple intervals at the same time, thereby improving the efficiency of voice editing.

[0033] The following is combined with Figure 1 The overall composition structure of the voice editing system provided by some embodiments of the present application is exemplified.

[0034] like Figure 1 As shown, some embodiments of the present application provide a voice editing system, which may include: a terminal 100 and a voice editing server 200. The terminal 100 may send the edited audio or the edited audio text to the voice editing server 200. If the voice editing server 200 receives the edited audio, it may convert it into the corresponding edited audio text. Afterwards, the voice editing server 200 may automatically find the original audio text similar to the edited audio text; then, based on the edited audio text and the original audio text, at least two edit intervals that need to be edited are determined, and finally, the editing process is performed to obtain a speech synthesis signal, and the signal is fed back to the terminal 100.

[0035] In some embodiments of the present application, the terminal 100 may be a mobile terminal or a non-portable computer terminal, which is not specifically limited in the embodiments of the present application.

[0036] In other embodiments of the present application, if the terminal 100 has the function of the voice editing server 200 to search for the original audio text, determine the editing interval and the editing process, the voice editing server 200 may not be set. The specific setting can be made according to the actual application scenario, and the embodiment of the present application is not specifically limited here.

[0037] The following is combined with Figure 2 The implementation process of voice editing performed by the voice editing server 200 provided in some embodiments of the present application is exemplified.

[0038] Please see attached Figure 2 , Figure 2 A flow chart of a method for voice editing provided in some embodiments of the present application, the method for voice editing may include:

[0039] S210, obtaining an original audio text similar to the edited audio text.

[0040] For example, in some embodiments of the present application, a character string similar to the edited audio text character string is first found as the original audio text.

[0041] In some embodiments of the present application, S210 may include: confirming that the edited audio text exists in an original text in the original text set, and then using the original text as the original audio text.

[0042] For example, in some embodiments of the present application, if the edited audio text can be directly obtained by deleting a certain original text in the original text set, then the certain original text can be directly used as the original audio text. That is, the edited audio text and the original text are in a containment relationship.

[0043] In some other embodiments of the present application, S210 may include: calculating the edit distance between the edited audio text and each original text in the original text set; taking the original text with the smallest edit distance as the original audio text, wherein the edited audio text does not exist in the original text with the smallest edit distance.

[0044] For example, in other embodiments of the present application, the BK-Tree technology can be used to determine the original audio text. The BK-Tree technology can efficiently search for a string that is relatively similar to a string, where the similarity is measured by the edit distance, and the smaller the edit distance, the more similar it is. The BK-Tree technology can be used to automatically search for the original audio text that is most similar to the edited audio text from the original text set. When using the voice editing system, only the edited audio text needs to be input, and the original audio text does not need to be specified, and the audio corresponding to the edited audio text can be obtained, which is more efficient. Specifically, s1 is the edited audio text, and s2 is an original text in the original text set. BK-Tree uses the edit distance as an index to measure the similarity of two strings, and calculates the minimum number (i.e., the edit distance is the minimum) of the insertion, deletion and replacement operations of the characters required to change a string (i.e., s2) to another string (i.e., s1), so that the original audio text corresponding to the edited audio text s1 can be found quickly.

[0045] For example, an audio clip is intercepted from an audio file, and its timbre and rhythm are basically consistent with the original audio. Edit audio text A: "Take a train from Tianjin to Chengdu", the original text set has text B: "Take a train from Beijing to Shanghai", and text C: "How long does it take to take a train from Tianjin to Chengdu". Among them, the edit distance between text A and text B is 4, and the edit distance with text C is 6. Since text A exists in text C, text C is selected as the original audio text for voice editing. In another case, if the edit distances between A and the original text set are 4 and 5 respectively, the original text corresponding to the edit distance of 4 is selected as the original audio text.

[0046] S220, calculating the edited audio text and the original audio text to obtain at least two edit intervals.

[0047] For example, in some embodiments of the present application, the interval that needs to be edited from the original audio text to the edited audio text is calculated using the original audio text and the edited audio text to obtain at least two edit intervals.

[0048] In some embodiments of the present application, S220 may include: identifying an area to be edited from the original audio text to the edited audio text, wherein the area to be edited includes at least two areas; determining the boundary of the area to be edited to obtain the at least two editing intervals.

[0049] For example, in some embodiments of the present application, the original audio text is "take a train from Beijing to Shanghai", and the edited audio text to be edited is "take a train from Tianjin to Chengdu". There are two areas to be edited from the original text to the edited text, one is "Beijing" changes to "Tianjin", and the other is "Shanghai" changes to "Chengdu". Then the boundary between the two is determined to obtain the edit interval.

[0050] In some embodiments of the present application, S220 may include: obtaining the start and end time nodes of each area in the to-be-edited area in the original audio text; using the start and end time nodes as the boundaries of each area, and determining the at least two editing intervals.

[0051] For example, in some embodiments of the present application, for each identified area to be edited, the time points to the start and end in the original audio text recording are clearly defined (as a specific example of the start and end time nodes), and these time points are used to define the boundaries of each editing interval. The boundaries of each editing interval can be determined at the same time. For example, the starting point of the editing interval "Beijing" is the recording time node of the word "North", and the end point is the recording time node of the word "Beijing". The starting point and the end point constitute the boundary of the editing interval "Beijing".

[0052] S230: Perform editing processing on the at least two editing intervals simultaneously to obtain a speech synthesis signal.

[0053] For example, in some embodiments of the present application, a speech synthesis signal can be output after multiple editing intervals are edited at the same time. In practical applications, the edited speech text can be obtained first, and then the edited speech text is converted into audio to obtain a speech synthesis signal. Among them, the text conversion to speech can use a related speech conversion model, which is not specifically limited in the embodiments of the present application.

[0054] In some embodiments of the present application, S230 may include: creating a masking matrix equal to the length of the original audio text, wherein the elements of each editing interval in the masking matrix have different values ​​from the elements of the non-editing interval; after masking the edited audio text using the masking matrix, performing speech synthesis processing on each editing interval to obtain a speech editing result; integrating the speech editing result with the audio of the non-editing interval to obtain the speech synthesis signal.

[0055] For example, in some embodiments of the present application, a vector matrix (as a specific example of a masking matrix, referred to as a matrix) equal to the length of the original audio text is established, and this matrix is ​​used to mask a specific interval in the speech corresponding to the original audio text. In this matrix, the elements corresponding to the editing intervals are set to 0, and the rest (that is, the non-editing intervals) are set to 1. After the original audio text is masked using the masking matrix, all editing intervals marked as 0 are subjected to speech synthesis processing, while the non-editing intervals marked as 1 remain as they are. The processed multiple intervals (as a specific example of the speech editing result) are reintegrated with the unprocessed intervals (that is, the non-editing intervals) to obtain the final edited speech synthesis signal.

[0056] Through the masking operation of the above-mentioned embodiment of the present application, the corresponding part of the editing interval is set to 0, and the other parts are set to 1. This technology can support the simultaneous calculation of the editing of multiple intervals, so that the masking positions can be dispersed, thereby improving the efficiency of voice editing.

[0057] The following is combined with Figure 3 The specific process of voice editing provided by some embodiments of the present application is exemplified.

[0058] Please see attached Figure 3 , Figure 3 A flow chart of a voice editing method provided for some embodiments of the present application.

[0059] The above process is explained below as an example.

[0060] S310, searching the original audio text most similar to the edited audio text from the original text collection.

[0061] S320, identifying a region to be edited in the original audio text and the edited audio text.

[0062] S330: Determine the boundary of the area to be edited, and obtain at least two editing intervals.

[0063] S340, creating a masking matrix having a length equal to that of the original audio text.

[0064] S350, after masking the edited audio text using the masking matrix, perform speech synthesis processing on each edited interval to obtain a speech editing result.

[0065] S360: Integrate the speech editing result with the audio of the non-editing interval to obtain a speech synthesis signal.

[0066] It can be understood that the specific implementation process of S310 to S360 can refer to the method embodiment provided above, and in order to avoid repetition, the detailed description is appropriately omitted here.

[0067] Through some of the above-mentioned embodiments of the present application, it can be seen that the present application only needs to perform one synthesis operation for an edited audio text no matter how many editing intervals it contains, and the synthesis speed is very fast; the user only needs to input the edited audio text to be synthesized, and does not need to specify the original audio text, which brings great convenience to the user.

[0068] Please refer to Figure 4 , Figure 4 The block diagram of the voice editing device provided by some embodiments of the present application is shown. It should be understood that the voice editing device corresponds to the above method embodiment and can execute each step involved in the above method embodiment. The specific functions of the voice editing device can be found in the above description. To avoid repetition, the detailed description is appropriately omitted here.

[0069] Figure 4 The speech editing device includes at least one software function module that can be stored in a memory in the form of software or firmware or solidified in the speech editing device, and the speech editing device includes: an acquisition module 410, used to obtain an original audio text similar to the edited audio text; a calculation module 420, used to calculate the edited audio text and the original audio text to obtain at least two edit intervals; an editing module 430, used to simultaneously edit the at least two edit intervals to obtain a speech synthesis signal.

[0070] In some embodiments of the present application, the acquisition module 410 is used to confirm that the edited audio text exists in an original text in the original text set, and then use the one original text as the original audio text; or, calculate the edit distance between the edited audio text and each original text in the original text set; and use the original text with the smallest edit distance as the original audio text, wherein the edited audio text does not exist in the original text with the smallest edit distance.

[0071] In some embodiments of the present application, the computing module 420 is used to identify the area to be edited from the original audio text to the edited audio text, wherein the area to be edited includes at least two areas; determine the boundary of the area to be edited to obtain the at least two editing intervals.

[0072] In some embodiments of the present application, the calculation module 420 is used to obtain the start and end time nodes of each area in the to-be-edited area in the original audio text; and determine the at least two editing intervals using the start and end time nodes as the boundaries of each area.

[0073] In some embodiments of the present application, the editing module 430 is used to create a masking matrix that is equal to the length of the original audio text, wherein the elements of each editing interval in the masking matrix have different values ​​from the elements of the non-editing interval; after masking the edited audio text using the masking matrix, each editing interval is subjected to speech synthesis processing to obtain a speech editing result; and the speech editing result is integrated with the audio of the non-editing interval to obtain the speech synthesis signal.

[0074] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the device described above can refer to the corresponding process in the aforementioned method, and will not be described in detail here.

[0075] Some embodiments of the present application further provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can implement the operations of the method corresponding to any of the above methods provided in the above embodiments.

[0076] Some embodiments of the present application further provide a computer program product, which includes a computer program, wherein when the computer program is executed by a processor, it can implement the operations corresponding to any of the above methods provided in the above embodiments.

[0077] like Figure 5As shown, some embodiments of the present application provide an electronic device 500, which includes: a memory 510, a processor 520, and a computer program stored in the memory 510 and executable on the processor 520, wherein the processor 520 can implement a method as described in any of the above embodiments when reading the program from the memory 510 through a bus 530 and executing the program.

[0078] Processor 520 can process digital signals and can include various computing structures, such as complex instruction set computer structure, reduced instruction set computer structure, or a structure that implements a combination of multiple instruction sets. In some examples, processor 520 can be a microprocessor.

[0079] The memory 510 may be used to store instructions executed by the processor 520 or data related to the execution of instructions. These instructions and / or data may include codes for implementing some or all functions of one or more modules described in the embodiments of the present application. The processor 520 of the disclosed embodiment may be used to execute instructions in the memory 510 to implement the method shown above. The memory 510 includes a dynamic random access memory, a static random access memory, a flash memory, an optical memory, or other memory known to those skilled in the art.

[0080] The above description is only an embodiment of the present application and is not intended to limit the scope of protection of the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application. It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.

[0081] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0082] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

Claims

1. A method for voice editing, characterized in that: include: Obtaining an original audio text similar to the edited audio text; Calculating the edited audio text and the original audio text to obtain at least two edit intervals; The at least two editing intervals are edited simultaneously to obtain a speech synthesis signal.

2. The method according to claim 1, characterized in that The step of obtaining an original audio text similar to the edited audio text comprises: confirming that the edited audio text exists in an original text in the original text set, and taking the original text as the original audio text; or, Calculating the edit distance between the edited audio text and each original text in the original text set; The original text with the smallest edit distance is used as the original audio text, wherein the edited audio text does not exist in the original text with the smallest edit distance.

3. The method according to claim 2, characterized in that The edit distance characterizes the number of inserted, deleted or replaced characters required from each original text to the edited audio text.

4. The method according to any one of claims 1 to 3, characterized in that The step of calculating the edited audio text and the original audio text to obtain at least two edit intervals includes: Identifying a region to be edited from the original audio text to the edited audio text, wherein the region to be edited includes at least two regions; Determine the boundary of the area to be edited to obtain the at least two editing intervals.

5. The method according to claim 4, characterized in that The step of determining the boundary of the area to be edited to obtain the at least two editing intervals includes: Obtain the start and end time nodes of each area in the to-be-edited area in the original audio text; The at least two editing intervals are determined using the start and end time nodes as the boundaries of each region.

6. The method according to any one of claims 1 to 3, characterized in that The step of simultaneously performing editing processing on the at least two editing intervals to obtain a speech synthesis signal includes: Creating a masking matrix having a length equal to that of the original audio text, wherein the value of an element of each edit interval in the masking matrix is ​​different from that of an element of a non-edit interval; After masking the edited audio text with the masking matrix, performing speech synthesis processing on each edited interval to obtain a speech editing result; The speech editing result is integrated with the audio of the non-editing interval to obtain the speech synthesis signal.

7. A device for voice editing, characterized in that: include: An acquisition module, used for acquiring an original audio text similar to the edited audio text; A calculation module, used to calculate the edited audio text and the original audio text to obtain at least two edit intervals; The editing module is used to simultaneously perform editing processing on the at least two editing intervals to obtain a speech synthesis signal.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program executes the method according to any one of claims 1 to 6 when executed by a processor.

9. An electronic device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the computer program executes the method according to any one of claims 1 to 6 when being run by the processor.

10. A computer program product, characterized in that The computer program product comprises a computer program, wherein the computer program executes the method according to any one of claims 1 to 6 when executed by a processor.

Images