Method, device, electronic device and storage medium for generating sentence-reading text

Abstract

The present application relates to a method, device, electronic device, and storage medium for generating punctuation text. The method comprises: obtaining pre-processed recording data corresponding to recording data, wherein the pre-processed recording data includes initial audio features, text data, and the word length corresponding to the text data; performing audio reconstruction processing based on the initial audio features, text data, and the word length corresponding to the text data to generate reconstructed audio features; inputting the reconstructed audio features and the initial audio features into a preset error calculation module to generate an audio difference; and outputting the punctuation text corresponding to the recording data when it is detected that the audio difference maintains the preset difference. Compared to the existing technology, manual annotation is not required, but the annotated text is automatically transcribed from the recording data, which can quickly generate punctuation annotated text. By automatically mining and classifying implicit information, compared to manual annotation and retraining models, no personal subjective experience or habits are introduced, thereby improving the accuracy of punctuation prediction.

Description

Technical Field

[0001] The present application relates to the field of speech recognition, and in particular to a method, device, electronic device and storage medium for generating sentence-reading text. Background Art

[0002] Sentence prediction uses machine learning algorithms to predict pause locations in text. It is generally used to guide pronunciation pauses in speech synthesis systems. Currently, automatic sentence prediction is typically performed using machine learning models. This involves inputting a sentence and predicting a pause type for each character within it, for example, 0 for no pause, 1 for a short pause, and 2 for a long pause.

[0003] However, existing machine learning models require pre-training with a large amount of manually annotated data. Specifically for sentence reading prediction, the position and type of pauses are determined by the annotator's comprehensive understanding of the meaning of each sentence and his or her own imagined reading. The annotation results are highly subjective and arbitrary, resulting in a reduced accuracy in sentence reading prediction. Summary of the Invention

[0004] In view of the above problems, the embodiments of the present application are proposed to provide a method, device, electronic device and storage medium for generating punctuation text that overcome the above problems or at least partially solve the above problems.

[0005] According to a first aspect of the present application, a method for generating a sentence-reading text is provided, the method comprising:

[0006] Obtaining pre-processed recording data corresponding to the recording data, wherein the pre-processed recording data includes initial audio features, text data, and a word length corresponding to the text data;

[0007] Performing audio reconstruction processing based on the initial audio features, the text data, and the word length corresponding to the text data to generate a reconstructed audio feature;

[0008] Inputting the reconstructed audio feature and the initial audio feature into a preset error calculation module to generate an audio difference;

[0009] When it is detected that the audio difference value maintains the preset difference value, the punctuation text corresponding to the recording data is output. Optionally, the obtaining of the pre-processed recording data corresponding to the recording data includes:

[0010] Inputting the recorded data into a preset speech recognition system to generate initial audio features;

[0011] The initial audio features are recognized and processed to generate text data and a word length corresponding to the text data.

[0012] Optionally, inputting the recorded data into a preset speech recognition system to generate initial audio features includes:

[0013] Segmenting the recorded data to generate recorded segment data;

[0014] The recording segment data is subjected to feature extraction to generate initial audio features.

[0015] Optionally, performing audio reconstruction processing based on the initial audio features, text data, and a word length corresponding to the text data to generate the reconstructed audio features includes:

[0016] Obtaining a word embedding subvector corresponding to the word data according to the word data, and obtaining mean super information corresponding to each word data in the recording data according to the initial audio feature and the word length;

[0017] Inputting the word embedding sub-vector and the mean super information into the preset attention structure module to generate a scene weight value corresponding to the text data;

[0018] Expanding the scene weight value according to the word length to generate an expanded scene weight matrix;

[0019] Inputting the extended scene weight matrix into a first preset acoustic model to generate a first matrix;

[0020] The mean super information and the first matrix are summed, and the summation result is input into a second preset acoustic model to generate reconstructed audio features corresponding to the recorded data.

[0021] Optionally, obtaining a character embedding sub-vector corresponding to the text data according to the text data includes:

[0022] Perform table lookup processing on the character data in a preset Chinese character list to obtain a sequence number corresponding to the character data;

[0023] Generate a word embedding vector corresponding to the text data according to the sequence number;

[0024] The word embedding vector is equally divided according to a preset sentence type to generate at least one word embedding sub-vector.

[0025] Optionally, obtaining the mean super information corresponding to each text data in the recording data according to the initial audio feature and the word length includes:

[0026] Inputting the initial audio features into a preset neural network to obtain super information corresponding to the recording data;

[0027] The super information is segmented and averaged according to the word length to obtain the average super information corresponding to each text data in the recording data.

[0028] According to a second aspect of the present application, a device for generating a sentence-reading text is provided, the device comprising:

[0029] A first module is configured to obtain pre-processed recording data corresponding to the recording data, wherein the pre-processed recording data includes initial audio features, text data, and a word length corresponding to the text data;

[0030] The second module is configured to perform audio reconstruction processing based on the initial audio features, the text data, and the word length corresponding to the text data to generate a reconstructed audio feature;

[0031] A third module is configured to input the reconstructed audio feature and the initial audio feature into a preset error calculation module to generate an audio difference;

[0032] The fourth module is used to output the punctuation text corresponding to the recording data when it is detected that the audio difference maintains a preset difference.

[0033] Optionally, the determining module includes:

[0034] The first submodule is configured to input the recorded data into a preset speech recognition system to generate initial audio features;

[0035] The second submodule is used to recognize and process the initial audio features to generate text data and a word length corresponding to the text data.

[0036] According to a third aspect of the present application, an electronic device is provided, comprising a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus;

[0037] Memory for storing computer programs;

[0038] The processor is used to execute the program stored in the memory.

[0039] According to a fourth aspect of the present application, a computer-readable storage medium is provided, on which a computer program is stored.

[0040] The embodiment of the present application obtains pre-processed recording data corresponding to the recording data, wherein the pre-processed recording data includes initial audio features, text data, and the word length corresponding to the text data; performs audio reconstruction processing based on the initial audio features, text data, and the word length corresponding to the text data to generate a reconstructed audio feature; inputs the reconstructed audio feature and the initial audio feature into a preset error calculation module to generate an audio difference; and outputs the punctuation text corresponding to the recording data when it is detected that the audio difference maintains the preset difference. That is, by extracting features from the pre-processed recording data, initial audio features, text data and word length are obtained, and then audio reconstruction processing is performed. The error between the reconstructed audio features and the initial audio features is calculated. When the error value remains at a certain value and no longer changes, the punctuation text corresponding to the recording data is output. Compared with the existing technology, there is no need for manual annotation, but the annotated text is automatically transcribed from the recording data, including text content, pause position and pause type information. The annotated text can be generated quickly, at low cost, cold-started and in large quantities. By automatically mining and classifying implicit information, compared with the manual annotation and retraining model, no personal subjective experience and habits are introduced, thereby improving the accuracy of punctuation prediction.

[0041] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Various other advantages and benefits will become apparent to those skilled in the art by reading the detailed description of the preferred embodiment below.The accompanying drawings are only for the purpose of illustrating the preferred embodiment and are not to be considered as limiting the present application.

[0043] Figure 1 is a flow chart of a method for generating a sentence-reading text according to an exemplary embodiment;

[0044] Figure 2 is a flow chart of another method for generating a sentence-reading text according to an exemplary embodiment;

[0045] Figure 3 is a flow chart of another method for generating a sentence-reading text according to an exemplary embodiment;

[0046] Figure 4 is a block diagram of a device for generating sentence-reading text according to an exemplary embodiment;

[0047] Figure 5 The figure is a schematic diagram of a sentence-reading text generation system according to an exemplary embodiment. DETAILED DESCRIPTION

[0048] The following describes exemplary embodiments of the present application in more detail with reference to the accompanying drawings. Although exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0049] The terms "first," "second," and the like in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of this application can be implemented in an order other than that illustrated or described herein, and that the objects distinguished by "first," "second," and the like are generally of the same type, and do not limit the number of objects; for example, the first object can be one or more. In addition, the term "and / or" in the specification and claims refers to at least one of the connected objects, and the character " / " generally indicates that the objects connected are in an "or" relationship.

[0050] The following, in conjunction with the accompanying drawings, describes in detail the method, device, electronic device and storage medium for generating sentence-reading text provided by the embodiments of the present application through specific embodiments and their application scenarios.

[0051] It should be noted that, in the embodiments of the present application, Figure 5 As shown, the left side shows an arbitrary third-party open-source speech recognition system, which preprocesses the input recording to identify the text content and the duration of each word. The right side shows our own USAPT model, which is dedicated to reconstructing the audio features of the original recording based on the text and word length. During the reconstruction process, the attention weights of the model's internal module, Classification Attention, are the final output, representing the weight distribution of each word's punctuation type. Model training continuously optimizes the accuracy of the reconstruction results, making them closer to the actual details of the original recording, and ensuring that the attention weight output also closely approximates the actual punctuation of the original recording.

[0052] The first embodiment of the present application relates to a method for generating sentence-reading text. Figure 1 is a flowchart of a method for generating a sentence-reading text according to an exemplary embodiment. Figure 1 Shown, including:

[0053] Step 101: obtaining pre-processed recording data corresponding to the recording data, wherein the pre-processed recording data includes initial audio features, text data, and word length corresponding to the text data;

[0054] It should be noted that in the embodiment of the present application, it is first necessary to obtain pre-processed recording data. In order to facilitate understanding by those skilled in the art, an input recording with a length of 2 seconds and a text content of "Hello, world" is used as an example for explanation.

[0055] Specifically, obtaining pre-processed recording data is to input the recording data into a preset speech recognition system. Specifically, the speech recognition system can be selected according to actual conditions, and this application does not make specific limitations. The speech recognition system first uses a speech signal analysis algorithm to extract audio features from the recording. For example, the first step is to divide the frame and divide the input audio into short segments with a granularity of 10 milliseconds. Then, the audio features of each frame are calculated, such as an 80-dimensional fbank. After processing, a matrix of size 200x80 corresponding to the recording can be obtained. Then, the preset recognition algorithm is executed on the 200x80 matrix. The text can be Hello, World, and how many frames each text corresponds to in 200 lines. For example, if the word length result is [20,30,50,20,80], it means that the word "you" corresponds to 1-20 frames in 200 lines, and the word "good" corresponds to 21-50 frames in 200 lines, etc., which means that in the original recording, the word "you" occupies the first 200 milliseconds and the word "good" occupies the next 300 milliseconds.

[0056] Furthermore, the audio features, text data, and the word length corresponding to the text data are used as preprocessing results of a sentence of recording. The above operation is performed on each sentence of the recording set, and the preprocessing results of each sentence are saved as a group.

[0057] Step 102: Perform audio reconstruction processing based on the initial audio features, the text data, and the word length corresponding to the text data to generate a reconstructed audio feature;

[0058] It should be noted that in the embodiment of the present application, reconstructed audio features can be obtained by modeling the initial audio features, text data, and the word length corresponding to the text data. The reconstructed audio features obtained through the above data, through word embedding vectors and acoustic features, respectively reflect what was said and how it was said. The combination of the two lays a sufficient information source for sound reconstruction, which is conducive to more accurate sound reconstruction.

[0059] Step 103: input the reconstructed audio features and the initial audio features into a preset error calculation module to generate an audio difference;

[0060] It should be noted that in the embodiment of the present application, the reconstructed audio features and the original true initial audio features in the data group are sent to the error calculation module to calculate the mean square error between the two, which represents the gap between the audio features reconstructed by the USAPT model and the true audio features. The goal of model training is to continuously narrow the gap. Specifically, the step of reconstructing the audio features can be repeated for each recording data in the training set. After one round, the order is shuffled and started again until the MSE Loss no longer decreases.

[0061] Step 104: When it is detected that the audio difference value maintains the preset difference value, output the punctuation text corresponding to the recording data.

[0062] It should be noted that, in the embodiment of the present application, when it is detected that the audio difference maintains the preset difference, that is, the MSE Loss output by the USAPT model no longer decreases, it means that the reconstructed audio features are basically close to the initial audio features. At this time, the sentence reading text corresponding to the recording data is output. For example, a sentence with a recording length of 2 seconds and the text content of hello world is input. After the training is completed, a labeled text is obtained: you#0good#0ah#1world#0world#2.

[0063] The embodiment of the present application obtains pre-processed recording data corresponding to the recording data, wherein the pre-processed recording data includes initial audio features, text data, and the word length corresponding to the text data; performs audio reconstruction processing based on the initial audio features, text data, and the word length corresponding to the text data to generate a reconstructed audio feature; inputs the reconstructed audio feature and the initial audio feature into a preset error calculation module to generate an audio difference; and outputs the punctuation text corresponding to the recording data when it is detected that the audio difference maintains the preset difference. That is, by extracting features from the pre-processed recording data, initial audio features, text data and word length are obtained, and then audio reconstruction processing is performed. The error between the reconstructed audio features and the initial audio features is calculated. When the error value remains at a certain value and no longer changes, the punctuation text corresponding to the recording data is output. Compared with the existing technology, there is no need for manual annotation, but the annotated text is automatically transcribed from the recording data, including text content, pause position and pause type information. The annotated text can be generated quickly, at low cost, cold-started and in large quantities. By automatically mining and classifying implicit information, compared with the manual annotation and retraining model, no personal subjective experience and habits are introduced, thereby improving the accuracy of punctuation prediction.

[0064] The second embodiment of the present application relates to another method for generating sentence-reading texts. Figure 2 is a flowchart of another method for generating a sentence-reading text according to an exemplary embodiment. Figure 2 Shown, including:

[0065] Step 1011: input the recorded data into a preset speech recognition system to generate initial audio features;

[0066] Step 1012: Recognize and process the initial audio features to generate text data and the word length corresponding to the text data.

[0067] It should be noted that, in the embodiment of the present application, in the above steps 1011-1012, an input recording length of 2 seconds and a text content of "Hello, world" are used as an example for explanation. Specifically, obtaining pre-processed recording data is to input the recording data into a preset speech recognition system. Specifically, the speech recognition system can be selected according to actual conditions, and this application does not make specific restrictions. The speech recognition system first uses a speech signal analysis algorithm to extract audio features from the recording. For example, the first step is to frame the input audio into short segments with a granularity of 10 milliseconds, and then calculate the audio for each frame. Frequency features, such as the 80-dimensional fbank, can be processed to obtain a 200x80 matrix corresponding to the recording. Then, the preset recognition algorithm is executed on the 200x80 matrix, and the text "hello, world" and the number of frames corresponding to each word in the 200 lines can be obtained. For example, if the word length result is [20,30,50,20,80], it means that the word "you" corresponds to frames 1-20 in the 200 lines, and the word "good" corresponds to frames 21-50 in the 200 lines, etc., which means that in the original recording, the word "you" occupies the first 200 milliseconds and the word "good" occupies the next 300 milliseconds.

[0068] Furthermore, the audio features, text data, and the word length corresponding to the text data are used as preprocessing results of a sentence of recording. The above operation is performed on each sentence of the recording set, and the preprocessing results of each sentence are saved as a group.

[0069] Step 102: Perform audio reconstruction processing based on the initial audio features, the text data, and the word length corresponding to the text data to generate a reconstructed audio feature;

[0070] Step 103: input the reconstructed audio features and the initial audio features into a preset error calculation module to generate an audio difference;

[0071] Step 104: When it is detected that the audio difference value maintains the preset difference value, output the punctuation text corresponding to the recording data.

[0072] It should be noted that the above steps 102-104 refer to the previous discussion and will not be repeated here.

[0073] The embodiment of the present application obtains pre-processed recording data corresponding to the recording data, wherein the pre-processed recording data includes initial audio features, text data, and the word length corresponding to the text data; performs audio reconstruction processing based on the initial audio features, text data, and the word length corresponding to the text data to generate a reconstructed audio feature; inputs the reconstructed audio feature and the initial audio feature into a preset error calculation module to generate an audio difference; and outputs the punctuation text corresponding to the recording data when it is detected that the audio difference maintains the preset difference. That is, by extracting features from the pre-processed recording data, initial audio features, text data and word length are obtained, and then audio reconstruction processing is performed. The error between the reconstructed audio features and the initial audio features is calculated. When the error value remains at a certain value and no longer changes, the punctuation text corresponding to the recording data is output. Compared with the existing technology, there is no need for manual annotation, but the annotated text is automatically transcribed from the recording data, including text content, pause position and pause type information. The annotated text can be generated quickly, at low cost, cold-started and in large quantities. By automatically mining and classifying implicit information, compared with the manual annotation and retraining model, no personal subjective experience and habits are introduced, thereby improving the accuracy of punctuation prediction.

[0074] The third embodiment of the present application relates to another method for generating sentence-reading text. Figure 3 is a flowchart of another method for generating a sentence-reading text according to an exemplary embodiment. Figure 3 Shown, including:

[0075] Step 101: obtaining pre-processed recording data corresponding to the recording data, wherein the pre-processed recording data includes initial audio features, text data, and word length corresponding to the text data;

[0076] It should be noted that the above step 101 is described with reference to the previous discussion and will not be repeated here.

[0077] Step 1021: obtaining a word embedding subvector corresponding to the word data according to the word data, and obtaining mean super information corresponding to each word data in the recording data according to the initial audio feature and the word length;

[0078] It should be noted that in the embodiment of the present application, after pre-processing the recorded data, the initial audio features, text data and word length are obtained. At this time, the above data needs to be processed.

[0079] Further, obtaining the word embedding sub-vector corresponding to the text data according to the text data includes: performing a look-up table process on the text data in a preset Chinese character list to obtain the serial number corresponding to the text data; generating a word embedding vector corresponding to the text data according to the serial number; and equally dividing the word embedding vector according to a preset punctuation type to generate at least one word embedding sub-vector.

[0080] It should be noted that in the embodiments of the present application, a list of common Chinese characters is preset in advance. For example, there are 6,763 GB characters. The serial number of each character is obtained by looking up the table for the text in each group of data. Those that cannot be found are uniformly numbered 0. For example, for "ni hao a shi jie", the serial numbers obtained by looking up the table are [10, 100, 1, 1000, 2000]. A matrix with a size of 6764xD is set up by a neural network, where D is a dimension that can be evenly divided by the number of punctuation types. The reason for using a neural network to construct it is that during the model training (gradient descent) process, the values inside the matrix will change in a direction that makes the final MSE LOSS smaller. For the time, an example of the matrix dimension is 6764x168. According to the serial number results obtained by looking up the table, the row vectors corresponding to the respective row numbers are taken for each character, that is, the 10th row, 100th row, 1st row, 1000th row, and 2000th row are taken from 6764x168 to form a first matrix of 5x168. The 168-dimensional row vector taken for each character is called a word embedding vector, which is used as the numerical representation of each character and participates in the subsequent model operations.

[0081] Since the same character is regarded as a different character under different punctuation conditions. For example, "a" in this example is a short pause, and in another sentence, it may be a non-pause or a long pause. So, the "a" with a short pause, the "a" with a long pause, and the "a" with a non-pause are regarded as three different characters, and different word embedding vectors are taken correspondingly. However, the differences among these three are not as extreme as those of completely different characters. Therefore, the word embedding vector obtained for each text data is split into three sub-vectors, which respectively implicitly represent the three punctuation conditions of this text data. The shape of each word embedding vector changes from 168 to 3x56, and finally, the 5 rows become a result with a size of 5x3x56, denoted as EC.

[0082] Therefore, the dimension of the first matrix is equally divided according to the number of punctuation types, and the first matrix 5x168 is transformed into 5x3x56, that is, the word embedding vector is equally divided according to the preset punctuation type to generate at least one word embedding sub-vector. The specific equal division is determined according to the punctuation type of each text data.

[0083] Furthermore, the method of obtaining the mean super-information corresponding to each text data in the recorded data based on the initial audio features and the word length includes: inputting the initial audio features into a preset neural network to obtain the super-information corresponding to the recorded data; and performing segmented mean processing on the super-information according to the word length to obtain the mean super-information corresponding to each text data in the recorded data.

[0084] It should be noted that in the embodiment of the present application, the audio features in each set of pre-processed recording data can be input into a neural network module called Para-Info Extractor, so that the information volume of the audio features can be supplemented through this module. The USAPT model reconstructs the original recording based on text and word length, but the text only reflects what was said, not how it was said. Therefore, by processing the initial audio features, the implicit information of the original recording data can be obtained, for example, the ups and downs, lightness and heaviness, and breath sense of the human voice in the recording. These invisible information are collectively referred to as super information Para-Info.

[0085] In addition, the internal structure of the neural network module of Para-Info Extractor can be a multi-layer stacked two-dimensional convolutional neural network, and the size of the output result is set to 200x256, where 256 is a preset dimension.

[0086] After obtaining the super information, we need to determine the overall super information in the pronunciation of each word based on the word length. For example, for a result of size 200x256, based on the word length [20, 30, 50, 20, 80], we take the average of the first 20 rows to obtain a mean vector, and take the average of the next 30 rows to obtain the second mean vector, and so on. A total of 5 mean vectors are obtained to form a result of size 5x256, which is recorded as VM.

[0087] Step 1022: input the word embedding sub-vector and the mean super information into the preset attention structure module to generate a scene weight value corresponding to the text data;

[0088] It should be noted that in an embodiment of the present application, the word embedding sub-vector and mean super-information obtained in step 1021 are input into the preset attention structure module, so that the weights of various sentence reading scenarios of each word are determined according to the super-information of each word. The super-information comes from the acoustic information of the recording, and the word embedding amount reflects the text information. Therefore, the weights of different types of text information can be actually determined according to the acoustic information.

[0089] Specifically, for the first-dimensional loop of 5x256 VM and 5x3x56 EC, the following two steps are performed for the 256-dimensional VMi and 3x56-dimensional ECi of the i-th round (i=1-5):

[0090] a) First, a dedicated neural network module is used to calculate the attention weights Weight(VMi, ECi) = [w1, w2, w3], where each w is a 1-dimensional number and the sum of all w is equal to 1;

[0091] b) Calculate the weighted sum Attention(VMi, ECi) = ECi1·w1+ECi2·w2+ECi3·w3, where ECi1ECi2ECi3 are the rows of ECi.

[0092] It should be noted that the weighted sum Attention is the scene weight value.

[0093] Step 1023: Expand the scene weight value according to the word length to generate an expanded scene weight matrix;

[0094] It should be noted that in the embodiment of the present application, after obtaining the scene weight value, it is necessary to copy and expand it according to the word length. Specifically, for example, for a 5x56 scene weight value, according to the word length [20,30,50,20,80], the first row of Attention is copied 20 copies, the second row is copied 30 copies, the third row is copied 50 copies, the fourth row is copied 20 copies, and the fifth row is copied 80 copies to form an extended scene weight matrix to obtain a 200x56 result.

[0095] Step 1024: input the extended scene weight matrix into a first preset acoustic model to generate a first matrix;

[0096] It should be noted that, in the embodiment of the present application, the extended scene weight matrix is ​​input into the first preset acoustic model, and progress is made towards acoustic reconstruction.

[0097] Among them, the first preset acoustic model can be represented as a Reconstruct Encoder module, the internal structure of which can be a multi-layer one-dimensional convolutional network. If the network width is 256, a result with a size of 200x256 is obtained.

[0098] Step 1025 , summing the mean super information and the first matrix, and inputting the summation result into a second preset acoustic model to generate reconstructed audio features corresponding to the recorded data.

[0099] It should be noted that in the embodiment of the present application, the mean super-information and the first matrix are summed, and the summation result is input into the second preset acoustic model to reconstruct the audio features of the original sound. The first matrix is ​​calculated all the way from the word embedding vector, and the mean super-information is directly calculated from the acoustic features, which respectively reflect what was said and how it was said. The combination of the two provides a sufficient information source for sound reconstruction, which is conducive to more accurate sound reconstruction.

[0100] The second preset acoustic model can be represented as a Merge Encoder module. Its internal structure can be a bidirectional LSTM followed by a fully connected layer. The width of the last fully connected layer should be equal to the dimensions of the audio features. Here, the result is a size of 200x80.

[0101] Step 103: input the reconstructed audio features and the initial audio features into a preset error calculation module to generate an audio difference;

[0102] It should be noted that in the embodiment of the present application, the reconstructed audio features and the original true audio features in the recording data are sent to the error calculation module to calculate the mean square error between the two, which can represent the gap between the audio features reconstructed by the model USAPT and the true audio features.

[0103] It should be noted that the goal of model training is to continuously narrow this gap. Specifically, this can be achieved by reversely calculating the gradient of each variable in the neural network, and then each variable updates its own gradient.

[0104] Step 104: When it is detected that the audio difference value maintains the preset difference value, output the punctuation text corresponding to the recording data.

[0105] It should be noted that the above step 104 is described with reference to the previous discussion and will not be described again here.

[0106] The embodiment of the present application obtains pre-processed recording data corresponding to the recording data, wherein the pre-processed recording data includes initial audio features, text data, and the word length corresponding to the text data; performs audio reconstruction processing based on the initial audio features, text data, and the word length corresponding to the text data to generate a reconstructed audio feature; inputs the reconstructed audio feature and the initial audio feature into a preset error calculation module to generate an audio difference; and outputs the punctuation text corresponding to the recording data when it is detected that the audio difference maintains the preset difference. That is, by extracting features from the pre-processed recording data, initial audio features, text data and word length are obtained, and then audio reconstruction processing is performed. The error between the reconstructed audio features and the initial audio features is calculated. When the error value remains at a certain value and no longer changes, the punctuation text corresponding to the recording data is output. Compared with the existing technology, there is no need for manual annotation, but the annotated text is automatically transcribed from the recording data, including text content, pause position and pause type information. The annotated text can be generated quickly, at low cost, cold-started and in large quantities. By automatically mining and classifying implicit information, compared with the manual annotation and retraining model, no personal subjective experience and habits are introduced, thereby improving the accuracy of punctuation prediction.

[0107] In addition, in the embodiment of the present application, the machine learning model used adopts an unsupervised training method. There is no need to annotate any data for it. The recording is directly input and automatically trained. At the end of the training, the annotated text is directly obtained. The machine learning model used is oriented to the inverse process of speech recognition, aiming to reconstruct the original sound from the result of speech recognition. The annotated text is obtained by the way during the reconstruction process. In addition to the results (text and word length) obtained by speech recognition, the input information for speech reconstruction also adds a super information Para-Info to supplement the information source. The two inputs respectively reflect what was said and how it was said. The two inputs are merged to reconstruct the speech, which is conducive to reconstructing more accurate and more fidelity details. For the input word embedding vector, the different sentence reading types of the same word are represented by dimensional deformation, and then the weighted fusion of each subdivision type is performed through the attention mechanism. The subdivision of the word embedding vector is conducive to the accuracy of the final reconstruction. After the subdivision, it is dynamically fused through attention, allowing the model itself to dynamically find a compromise balance between the two extremes of absolute indiscriminate and absolute subdivision.

[0108] The fourth embodiment of the present application relates to a sentence-reading text generation device, Figure 4 is a block diagram of a sentence-reading text generation device according to an exemplary embodiment. Figure 4 Shown, including:

[0109] The first module 401 is configured to obtain pre-processed recording data corresponding to the recording data, wherein the pre-processed recording data includes initial audio features, text data, and a word length corresponding to the text data;

[0110] The second module 402 is configured to perform audio reconstruction processing based on the initial audio features, the text data, and the word length corresponding to the text data to generate a reconstructed audio feature;

[0111] The third module 403 is configured to input the reconstructed audio feature and the initial audio feature into a preset error calculation module to generate an audio difference;

[0112] The fourth module 404 is configured to output the punctuation text corresponding to the recording data when it is detected that the audio difference value maintains a preset difference value.

[0113] Furthermore, the first module includes:

[0114] The first submodule is configured to input the recorded data into a preset speech recognition system to generate initial audio features;

[0115] The second submodule is used to recognize and process the initial audio features to generate text data and a word length corresponding to the text data.

[0116] The embodiment of the present application obtains pre-processed recording data corresponding to the recording data, wherein the pre-processed recording data includes initial audio features, text data, and the word length corresponding to the text data; performs audio reconstruction processing based on the initial audio features, text data, and the word length corresponding to the text data to generate a reconstructed audio feature; inputs the reconstructed audio feature and the initial audio feature into a preset error calculation module to generate an audio difference; and outputs the punctuation text corresponding to the recording data when it is detected that the audio difference maintains the preset difference. That is, by extracting features from the pre-processed recording data, initial audio features, text data and word length are obtained, and then audio reconstruction processing is performed. The error between the reconstructed audio features and the initial audio features is calculated. When the error value remains at a certain value and no longer changes, the punctuation text corresponding to the recording data is output. Compared with the existing technology, there is no need for manual annotation, but the annotated text is automatically transcribed from the recording data, including text content, pause position and pause type information. The annotated text can be generated quickly, at low cost, cold-started and in large quantities. By automatically mining and classifying implicit information, compared with the manual annotation and retraining model, no personal subjective experience and habits are introduced, thereby improving the accuracy of punctuation prediction.

[0117] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0118] Furthermore, based on the same inventive concept, a specific embodiment of the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and runnable on the processor, wherein the processor implements the method steps in any of the above embodiments when executing the computer program.

[0119] Based on the same inventive concept, in a specific embodiment of the present application, the processor can implement any implementation of the method in the embodiment of the present application when executing the computer program.

[0120] Since the electronic devices described in the specific embodiments of this application are devices used to implement the methods of the embodiments of this application, those skilled in the art will be able to understand the specific structure and variations of the devices based on the methods described in the embodiments of this application, and therefore, they will not be described in detail here. All devices used in the methods of the embodiments of this application fall within the scope of protection to be provided by this application.

[0121] Based on the same inventive concept, the specific embodiments of the present application also provide a storage medium corresponding to the method in the embodiment: this embodiment provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the method steps in any of the above embodiments are implemented.

[0122] During the specific implementation process, when the computer program is executed by a processor, any implementation method of the specific embodiment of the present application can be implemented.

[0123] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0124] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, devices, storable media, and processors. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0125] In a typical configuration, the computer device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory. Memory may include non-permanent storage in computer-readable media, random access memory (RAM), and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium. Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can be implemented using any method or technology to store information. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include non-transitory computer-readable media such as modulated data signals and carrier waves.

[0126] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0127] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0128] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0129] Although preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they become aware of the basic inventive concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0130] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or terminal device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or terminal device that includes the element.

[0131] The above is a detailed introduction to the method, device, electronic device and storage medium for generating sentence-reading text provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core idea of ​​the present application. At the same time, for those skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A method for generating sentence-reading text, characterized in that: The method comprises: Obtaining pre-processed recording data corresponding to the recording data, wherein the pre-processed recording data includes initial audio features, text data, and a word length corresponding to the text data; Performing audio reconstruction processing based on the initial audio features, the text data, and the word length corresponding to the text data to generate a reconstructed audio feature; Inputting the reconstructed audio feature and the initial audio feature into a preset error calculation module to generate an audio difference; When it is detected that the audio difference value maintains a preset difference value, outputting the sentence reading text corresponding to the recording data; The performing audio reconstruction processing based on the initial audio features, the text data, and the word length corresponding to the text data to generate the reconstructed audio features includes: Obtaining a word embedding subvector corresponding to the word data according to the word data, and obtaining mean super information corresponding to each word data in the recording data according to the initial audio feature and the word length; Inputting the word embedding sub-vector and the mean super information into a preset attention structure module to generate a scene weight value corresponding to the text data; Expanding the scene weight value according to the word length to generate an expanded scene weight matrix; Inputting the extended scene weight matrix into a first preset acoustic model to generate a first matrix; The mean super information and the first matrix are summed, and the summation result is input into a second preset acoustic model to generate reconstructed audio features corresponding to the recorded data.

2. The method according to claim 1, characterized in that The obtaining of pre-processed recorded data corresponding to the recorded data includes: Inputting the recorded data into a preset speech recognition system to generate initial audio features; The initial audio features are recognized and processed to generate text data and a word length corresponding to the text data.

3. The method according to claim 2, characterized in that Inputting the recorded data into a preset speech recognition system to generate initial audio features includes: Segmenting the recorded data to generate recorded segment data; The recording segment data is subjected to feature extraction to generate initial audio features.

4. The method according to claim 1, wherein The obtaining, according to the text data, a character embedding sub-vector corresponding to the text data comprises: Perform table lookup processing on the character data in a preset Chinese character list to obtain a sequence number corresponding to the character data; Generate a word embedding vector corresponding to the text data according to the sequence number; The word embedding vector is equally divided according to a preset sentence type to generate at least one word embedding sub-vector.

5. The method according to claim 1, characterized in that The obtaining of the mean super information corresponding to each text data in the recording data according to the initial audio feature and the word length includes: Inputting the initial audio features into a preset neural network to obtain super information corresponding to the recording data; The super information is segmented and averaged according to the word length to obtain the average super information corresponding to each text data in the recording data.

6. A device for generating sentence-reading text, characterized in that: The device comprises: A first module is configured to obtain pre-processed recording data corresponding to the recording data, wherein the pre-processed recording data includes initial audio features, text data, and a word length corresponding to the text data; The second module is configured to perform audio reconstruction processing based on the initial audio features, the text data, and the word length corresponding to the text data to generate a reconstructed audio feature; A third module is configured to input the reconstructed audio feature and the initial audio feature into a preset error calculation module to generate an audio difference; A fourth module is configured to output a punctuation text corresponding to the recording data when it is detected that the audio difference value maintains a preset difference value; The second module includes: an acquisition submodule, configured to acquire a word embedding subvector corresponding to the text data according to the text data, and to acquire a mean super information corresponding to each text data in the recording data according to the initial audio feature and the word length; A first generating submodule is configured to input the word embedding subvector and the mean super information into a preset attention structure module to generate a scene weight value corresponding to the text data; A second generating submodule is configured to perform an expansion process on the scene weight value according to the word length to generate an expanded scene weight matrix; a third generating submodule, configured to input the extended scene weight matrix into a first preset acoustic model to generate a first matrix; The fourth generating submodule is used to sum the mean super information and the first matrix, and input the summation result into a second preset acoustic model to generate a reconstructed audio feature corresponding to the recording data.

7. The device according to claim 6, characterized in that The first module includes: The first submodule is configured to input the recorded data into a preset speech recognition system to generate initial audio features; The second submodule is used to recognize and process the initial audio features to generate text data and a word length corresponding to the text data.

8. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; Memory for storing computer programs; A processor, configured to implement the method according to any one of claims 1 to 5 when executing a program stored in a memory.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images