Text enhancement method and device, electronic equipment and storage medium

Abstract

The invention provides a text enhancement method and device, electronic equipment and a storage medium, and the method comprises the steps: carrying out the feature extraction processing of voice data, and obtaining the audio features of the voice data; determining the type of the audio feature according to the audio feature of the voice data; determining an enhancement mode of a second text matched with the first text of the voice data according to the type of the audio feature; and performing text enhancement processing on the first text according to the audio features and the enhancement mode of the second text to obtain the second text. Therefore, the text enhancement effect can be optimized.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a text enhancement method, device, electronic device and storage medium. Background Art

[0002] With the development of electronic technology, the application of NLP (Natural Language Processing) technology is becoming more and more widespread. Text augmentation is a technology used in the field of NLP to improve the generalization ability of models and data diversity. It generates new training samples by performing a series of transformations on the original text data, thereby expanding the data set without adding additional data.

[0003] By using text enhancement technology, the labeled data required for model training can be quickly expanded, but if the quality of text enhancement is poor, it may affect the accuracy of the labeled data. Summary of the invention

[0004] The embodiments of the present application provide a text enhancement method, device, electronic device and storage medium, which can optimize the text enhancement effect.

[0005] In a first aspect, an embodiment of the present application provides a text enhancement method, comprising:

[0006] Performing feature extraction processing on the speech data to obtain audio features of the speech data;

[0007] Determining the type of the audio feature according to the audio feature of the speech data;

[0008] Determining, according to the type of the audio feature, a manner of enhancing a second text that matches the first text of the speech data;

[0009] The first text is enhanced according to the audio feature and the enhancement method of the second text to obtain a second text.

[0010] In a second aspect, an embodiment of the present application provides a text enhancement device, including:

[0011] An extraction unit, used to perform feature extraction processing on the speech data to obtain audio features of the speech data;

[0012] a determining unit, configured to determine a type of the audio feature according to the audio feature of the speech data;

[0013] The determining unit is further configured to determine, according to the type of the audio feature, an enhancement method for a second text that matches the first text of the speech data;

[0014] The enhancement unit is used to perform text enhancement processing on the first text according to the audio feature and the enhancement method of the second text to obtain the second text.

[0015] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor; and a memory configured to store computer-executable instructions, wherein the computer-executable instructions, when executed, cause the processor to perform the text enhancement method as described in the first aspect.

[0016] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium for storing computer-executable instructions, which, when executed by a processor, implement the text enhancement method as described in the first aspect.

[0017] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the text enhancement method as described in the first aspect.

[0018] It can be seen that in the embodiment of the present application, first, feature extraction processing is performed on the speech data to obtain the audio features of the speech data; then, the type of audio features is determined according to the audio features of the speech data; then, the enhancement method of the second text matching the first text of the speech data is determined according to the type of audio features; finally, text enhancement processing is performed on the first text according to the audio features and the enhancement method of the second text to obtain the second text. In this way, considering that speech data collected in different scenes, or speech data from different characters in the same scene, may have their own characteristics, this characteristic leads to differences in the extraction effects of different types of audio features in the speech data. In actual applications, it may happen that some types of audio features can be successfully extracted from the speech data, but the extraction effects of other types of audio features are poor. This may result in different types of audio features extracted from each speech data after feature extraction processing is performed on different speech data. Therefore, by first extracting the audio features, then determining the type of the audio features, and then determining the enhancement method of the second text that matches the first text of the voice data according to the type of the audio features, compared with using a fixed text enhancement method, a personalized text enhancement method suitable for the voice data can be selected more flexibly, which is conducive to optimizing the text enhancement effect when performing text enhancement processing in subsequent steps. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative labor.

[0020] Figure 1 A schematic diagram of an implementation environment of a text enhancement method provided in an embodiment of the present application;

[0021] Figure 2 A processing flow chart of a text enhancement method provided in an embodiment of the present application;

[0022] Figure 3 A processing flow chart of another text enhancement method provided in an embodiment of the present application;

[0023] Figure 4 A processing flow chart of another text enhancement method provided in an embodiment of the present application;

[0024] Figure 5 A schematic diagram of a text enhancement device provided in an embodiment of the present application;

[0025] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments of the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0027] This specification provides a text enhancement method embodiment:

[0028] In the complaint scenario, part of the robot agent's job is to appease emotionally agitated complaining customers. As a result, the voice data from the robot agent may have a calm tone and a stable volume, while the voice data from the customer may have an intense tone and an unstable volume. This difference in characteristics may lead to significant differences in the audio features that can be extracted when using voice data for text enhancement processing.

[0029] If the difference in characteristics is ignored and the same pre-configured text enhancement method is used for text enhancement processing, it may result in that the text enhancement effect of a part of speech data is very good, but the text enhancement effect of another part of speech data is poor.

[0030] Therefore, in order to solve the above problems, an embodiment of the present application provides a text enhancement method.

[0031] The text enhancement method provided in one or more embodiments of this specification may be applicable to an implementation environment of the text enhancement method, which implementation environment at least includes a server 101 for generating a second text.

[0032] The server 101 may be a single server, or a server cluster consisting of a plurality of servers, or one or more cloud servers in a cloud computing platform, for generating the second text.

[0033] In this implementation environment, when the server 101 generates the second text, first, it performs feature extraction processing on the voice data to obtain the audio features of the voice data; then, it determines the type of audio features according to the audio features of the voice data; then, it determines the enhancement method of the second text that matches the first text of the voice data according to the type of audio features; finally, it performs text enhancement processing on the first text according to the audio features and the enhancement method of the second text to obtain the second text. In this way, considering that the voice data collected in different scenes, or the voice data from different roles in the same scene, may have their own characteristics, this characteristic leads to differences in the extraction effects of different types of audio features in the voice data. In actual applications, it may happen that some types of audio features can be successfully extracted from the voice data, but the extraction effects of other types of audio features are poor, which may result in different types of audio features extracted from each voice data after feature extraction processing is performed on different voice data. Therefore, by first extracting the audio features, then determining the type of the audio features, and then determining the enhancement method of the second text that matches the first text of the voice data according to the type of the audio features, compared with using a fixed text enhancement method, a personalized text enhancement method suitable for the voice data can be selected more flexibly, which is conducive to optimizing the text enhancement effect when performing text enhancement processing in subsequent steps.

[0034] Figure 2 A processing flow chart of a text enhancement method provided in an embodiment of the present application. Figure 2 The text enhancement method provided in this embodiment specifically includes steps S202 to S206:

[0035] Step S202: performing feature extraction processing on the speech data to obtain audio features of the speech data.

[0036] Speech data can be any data in the form of audio collected when communicating or talking to oneself in natural language.

[0037] For example, in a financial scenario, the voice data may be voice data sent by an agent or voice data sent by a customer when the agent communicates with the customer during a call.

[0038] Before executing step S202, voice data and a first text may be acquired first; the first text is obtained by performing voice recognition processing on the voice data.

[0039] Speech recognition processing can be achieved through ASR (Automatic Speech Recognition) technology. The goal of ASR is to convert the vocabulary content in human speech into computer-readable input, such as keystrokes, binary codes or character sequences.

[0040] The voice data may be obtained by recording, receiving uploaded voice data, or reading voice data from a preset storage area.

[0041] Acquiring the first text may be performing speech recognition processing on the acquired speech data to obtain the first text, or may be reading the first text from a preset storage area, where the first text is obtained in advance by performing speech recognition processing on the speech data.

[0042] Audio features refer to quantitative parameters extracted from audio signals that can represent the audio content or properties. These features can be used for a variety of purposes, such as audio recognition, music information retrieval, speech recognition, audio classification, audio signal processing, etc. Audio features are usually divided into time domain features, frequency domain features, and time-frequency domain features. Among them, time domain features refer to features extracted directly from the time waveform of the audio signal, which reflect the characteristics of the signal changing over time. Frequency domain features are features extracted by converting the audio signal to the frequency domain (such as through Fourier transform), which reflect the frequency components of the signal. Time-frequency domain features combine information in the time domain and frequency domain, and they are usually extracted through time-frequency analysis methods (such as short-time Fourier transform, wavelet transform, etc.).

[0043] The extraction of audio features is a key step in audio processing and analysis. They provide basic data for subsequent audio analysis and processing. By selecting appropriate features, audio signals can be effectively classified, identified and understood.

[0044] The audio feature of the speech data obtained by executing step S202 may be one audio feature, or may include multiple different audio features.

[0045] The voice data is subjected to feature extraction processing to obtain the audio features of the voice data, and the feature extraction processing may be performed on the voice data according to a preset feature extraction method to obtain the audio features of the voice data.

[0046] For example, the types of audio features to be extracted are pre-configured to include a first type and a second type, and the preset feature extraction method includes a first type of feature extraction method and a second type of feature extraction method. In the process of performing feature extraction processing on speech data to obtain audio features of the speech data, the following steps can be performed: performing feature extraction processing on the speech data according to the first type of feature extraction method to obtain the first type of audio features of the speech data; performing feature extraction processing on the speech data according to the second type of feature extraction method to obtain the second type of audio features of the speech data.

[0047] Exemplarily, the audio features of the speech data may include one or more of the following: pause feature information, stress feature information, and tone recognition results.

[0048] The following are the definitions of "pause", "stress" and "tone":

[0049] (a1) In speech signal processing and audio analysis, a pause can be a silent segment in a sound signal, that is, a part without speech activity. These silent segments can be intervals deliberately created by the speaker while he is thinking, organizing his speech, or for emphasis.

[0050] Specifically in one or more embodiments of the present application, pause feature information is used to describe silent segments in voice data.

[0051] (a2) In phonetics and audio signal processing, stress is the emphasis on certain syllables or words during pronunciation. This emphasis can be achieved by increasing the volume of the syllable, raising the pitch, prolonging the duration of the syllable, or changing the timbre of the syllable.

[0052] Specifically in one or more embodiments of the present application, the accent feature information is used to describe the accent segments in the speech data.

[0053] (a3) In phonetics and audio signal processing, tone of voice can be the changes in pitch, volume, and rhythm of the voice when speaking, which can convey the speaker's emotional state, attitude, and intention.

[0054] Specifically in one or more embodiments of the present application, the tone recognition result is used to reflect the emotional state, attitude and intention of the speaker who sends the voice data.

[0055] In a specific implementation, the audio features of the speech data include a tone recognition result of the speech data; performing feature extraction processing on the speech data to obtain the audio features of the speech data, including: performing conversion processing on the speech data to obtain a spectrogram of the speech data; the spectrogram is used to represent the pitch information of the speech data; performing segmentation processing on the spectrogram to obtain a plurality of sub-spectrograms arranged in chronological order; determining, based on the plurality of sub-spectrograms, pitch change information of the speech data in two adjacent sub-spectrograms; if the pitch change information indicates that the pitch information of the speech data rises, determining the pitch comparison results of the two adjacent sub-spectrograms to be a first preset result; if the pitch change information indicates that the pitch information of the speech data remains unchanged, determining the pitch comparison results of the two adjacent sub-spectrograms to be a second preset result; if the pitch change information indicates that the pitch information of the speech data decreases, determining the pitch comparison results of the two adjacent sub-spectrograms to be a third preset result; and obtaining the tone recognition result of the speech data by querying in a pre-configured correspondence between the pitch comparison results and the tone according to the pitch comparison results of the two adjacent sub-spectrograms.

[0056] A spectrogram is a graphical representation of the frequency components of a sound signal that change over time. It forms a two-dimensional image by decomposing the sound signal into components of different frequencies and plotting the changes of these components over time on a graph. In a spectrogram, the horizontal axis usually represents time, the vertical axis represents frequency, and the color or grayscale represents the signal strength (i.e., amplitude) at the corresponding time and frequency.

[0057] Specifically in one or more embodiments of the present specification, the spectrogram is used to represent the pitch information of speech data.

[0058] In the process of converting the speech data to obtain the spectrogram of the speech data, the speech data can be preprocessed to obtain preprocessed feature data of the speech data; the preprocessed feature data can be converted to obtain the spectrogram of the speech data, and the spectrogram is used to describe the preprocessed feature data.

[0059] Exemplarily, the preprocessing feature data includes but is not limited to at least one of the following: tone, pitch, volume, rhythm, sound quality, etc. The following briefly introduces each preprocessing feature data:

[0060] (b1) Pitch refers to the pitch of a voice when speaking, which can express different emotions and attitudes.

[0061] (b2) Pitch (Intonation) can be the rising and falling pattern of the tone of voice when speaking, which has an important impact on the tone and meaning of the sentence.

[0062] (b3) Loudness can be the strength of a person’s voice when speaking, and can convey emotions such as emphasis, excitement, or calmness.

[0063] (b4) Rhythm refers to the speed and pause pattern of speaking, which affects the fluency of language and the clarity of expression.

[0064] (b5) Timbre can be the texture of the voice, including changes in resonance and articulation, which can reflect the speaker's personality and emotional state.

[0065] Exemplarily, the abscissa of the spectrogram may be time, and the ordinate of the spectrogram may be pitch.

[0066] The preprocessed feature data is converted to obtain a spectrogram, and the preprocessed feature data may be input into a preset audio processing library for conversion to obtain a spectrogram.

[0067] The preset audio processing library may include one or more audio processing functions, each audio processing function having a corresponding function. One of the audio processing functions may be used to convert the received pre-processed feature data into a spectrogram.

[0068] Exemplarily, the preset audio processing library may be a librosa library, which is a very powerful third-party library for python speech signal processing.

[0069] The spectrogram is segmented to obtain a plurality of sub-spectrograms arranged in time sequence. The spectrogram may be segmented according to the time length of the speech data to obtain a plurality of sub-spectrograms arranged in time sequence.

[0070] For example, if the time length of the speech data is 3 minutes, the spectrogram is divided into three sub-spectrograms, namely, front, middle and back, according to the time length. The front sub-spectrogram describes the pitch information of the first minute of the speech data, the middle sub-spectrogram describes the pitch information of the second minute of the speech data, and the back sub-spectrogram describes the pitch information of the third minute of the speech data.

[0071] According to the plurality of sub-spectroggrams, pitch change information of speech data in two adjacent sub-spectroggrams is determined.

[0072] For example, the spectrogram is segmented to obtain three sub-spectrograms arranged in chronological order: Figure 1 , sub-sound spectrum Figure 2 And sub-spectrum Figure 3 The two adjacent sub-spectrograms include: Figure 1 With sub-sound spectrum Figure 2 , and the adjacent sub-sound spectra Figure 2 Harmony sound spectrum Figure 3 According to the sub-sound spectrum Figure 1 Harmony sound spectrum Figure 2 Sub-sound spectrum can be determined Figure 1 Harmony sound spectrum Figure 2 The pitch change information of the speech data, and the sub-voice spectrum Figure 2 Harmony sound spectrum Figure 3 Sub-sound spectrum can be determined Figure 2 Harmony sound spectrum Figure 3 Pitch change information of voice data.

[0073] If the pitch change information indicates that the pitch information of the speech data increases, the pitch comparison result of the two adjacent sub-spectrograms is determined to be the first preset result; if the pitch change information indicates that the pitch information of the speech data remains unchanged, the pitch comparison result of the two adjacent sub-spectrograms is determined to be the second preset result; if the pitch change information indicates that the pitch information of the speech data decreases, the pitch comparison result of the two adjacent sub-spectrograms is determined to be the third preset result.

[0074] Exemplarily, the pitch change information of speech data in two adjacent sub-spectrograms may include the following three situations:

[0075] (1) The pitch change information indicates that the pitch information of the voice data rises. In this case, the pitch comparison result is the first preset result S1.

[0076] (2) The pitch change information indicates that the pitch information of the speech data remains unchanged. In this case, the pitch comparison result is the second preset result S2.

[0077] (3) The pitch change information indicates that the pitch information of the voice data decreases. In this case, the pitch comparison result is the third preset result S3.

[0078] In addition, the pitch comparison results can be classified more finely, for example: a small increase, a large increase, unchanged, a small decrease, a large decrease, and so on.

[0079] For example, the pitch change information indicates that the pitch information of the voice data rises, and the rising amplitude of the pitch information is less than the preset pitch threshold. In this case, the pitch comparison result is s1; the pitch change information indicates that the pitch information of the voice data rises, and the rising amplitude of the pitch information is greater than or equal to the preset pitch threshold. In this case, the pitch comparison result is s2; the pitch change information indicates that the pitch information of the voice data remains unchanged. In this case, the pitch comparison result is s3; the pitch change information indicates that the pitch information of the voice data decreases, and the decreasing amplitude of the pitch information is less than the preset pitch threshold. In this case, the pitch comparison result is s4; the pitch change information indicates that the pitch information of the voice data decreases, and the decreasing amplitude of the pitch information is greater than or equal to the preset pitch threshold. In this case, the pitch comparison result is s5.

[0080] According to the pitch comparison results of two adjacent sub-spectrograms, a query is performed in the pre-configured correspondence between the pitch comparison results and the tone to obtain the tone recognition result of the speech data.

[0081] For example, the adjacent sub-spectra Figure 1 Harmony sound spectrum Figure 2 The pitch change information 1 can be determined by the adjacent sub-sound spectra Figure 2 Harmony sound spectrum Figure 3 The pitch change information 2 can be determined. The pitch change information 1 and the pitch change information 2 may be the same or different. There is no necessary connection between the two and they need to be determined independently. A corresponding pitch comparison result 1 can be determined from the pitch change information 1, and a corresponding pitch comparison result 2 can be determined from the pitch change information 2.

[0082] In this way, when the spectrogram includes three sub-spectrograms, two pitch change information can be obtained, and then two pitch comparison results can be determined.

[0083] A correspondence between a pitch comparison result and a tone is preconfigured, and the tone recognition result of the speech data can be obtained by querying the two determined pitch comparison results in the correspondence.

[0084] For example, the correspondence between the configured pitch comparison result and the tone includes:

[0085] (S1,S1) corresponds to mood 1

[0086] (S2,S2) corresponds to mood 2

[0087] (S3,S3) corresponds to mood 3

[0088] (S1, S2) corresponds to tone 4

[0089] …

[0090] Assume that the spectrogram includes sub-spectrum Figure 1 , sub-sound spectrum Figure 2 And sub-spectrum Figure 3 . It consists of two adjacent sub-sound spectra Figure 1 With sub-sound spectrum Figure 2 The pitch change information 1 can be determined, and the pitch comparison result 1 can be determined to be S3 based on the pitch change information 1; Figure 2 With sub-sound spectrum Figure 3The pitch change information 2 can be determined, and the pitch comparison result 2 can be determined as S3 from the pitch change information 2. The two pitch comparison results of the speech data can be expressed as (S3, S3). According to the two pitch comparison results, query processing is performed in a pre-configured correspondence to obtain the tone 3, and the tone 3 is used as the tone recognition result of the speech data.

[0091] In a specific implementation, pitch change information of speech data in two adjacent sub-spectrograms is determined based on a plurality of sub-spectrograms, including: determining an average pitch of the sub-spectrograms based on the pitch information included in each sub-spectrogram; calculating a first difference in the average pitches of two adjacent sub-spectrograms in a plurality of sub-spectrograms; if the first difference is less than a preset value, determining that the pitch information of the speech data represented by the pitch change information has risen; if the first difference is equal to the preset value, determining that the pitch information of the speech data represented by the pitch change information remains unchanged; if the first difference is greater than the preset value, determining that the pitch information of the speech data represented by the pitch change information has fallen.

[0092] The horizontal axis of each sub-spectrogram may be time, and the vertical axis may be pitch.

[0093] Each sub-spectrogram can describe the pitch information of a period of time.

[0094] For example, the spectrogram is divided into: sub-spectrum Figure 1 , sub-sound spectrum Figure 2 And sub-spectrum Figure 3 , etc. Among them, the sub-sound spectrum Figure 1 Describes the pitch information of the time period [T0, T1), sub-sound spectrum Figure 2 Describes the pitch information of the time period [T1, T2), sub-sound spectrum Figure 3 Describes the pitch information of the time period [T2, T3), and so on.

[0095] According to the pitch information included in each sub-spectrogram, the average pitch of the sub-spectrogram can be determined.

[0096] Taking into account that the pitch information included in each sub-spectrogram can be regarded as continuous data, the average value of the continuous data can be calculated by integrating the data and then dividing it by the length of the data range. Specifically, in one or more embodiments of the present application, in the process of determining the average pitch of the sub-spectrogram based on the pitch information included in each sub-spectrogram, an integral calculation can be performed based on the pitch information included in the sub-spectrogram to obtain a first calculation result, and then the average pitch of the sub-spectrogram can be determined based on the first calculation result and the length of the time range of the sub-spectrogram.

[0097] In the plurality of sub-spectrograms, the first difference of the average pitch of two adjacent sub-spectrograms is calculated. For example, the spectrogram is divided into: Figure 1 , sub-sound spectrum Figure 2 And sub-spectrum Figure 3 , and so on. About the adjacent sub-sound spectra Figure 1 With sub-sound spectrum Figure 2 , the sub-spectrum Figure 1 The average pitch of the sub-sound spectrum is x1. Figure 2 The average pitch of is x2, and the first difference between x2 and x1 is calculated to get (x2-x1). Figure 2 With sub-sound spectrum Figure 3 , the sub-spectrum Figure 2 The average pitch of the sub-sound spectrum is x2. Figure 3 The average pitch is x3, and the first difference between x3 and x2 is calculated to obtain (x3-x2).

[0098] The preset values ​​can be customized.

[0099] Exemplarily, the preset value may be 0. If the first difference is less than 0, it is determined that the pitch information of the voice data represented by the pitch change information rises; if the first difference is equal to 0, it is determined that the pitch information of the voice data represented by the pitch change information remains unchanged; if the first difference is greater than 0, it is determined that the pitch information of the voice data represented by the pitch change information falls.

[0100] The above implementation method exemplifies the implementation process of how to extract the tone recognition result from the voice data. In addition, the tone recognition result can also be extracted from the voice data by other methods:

[0101] The speech data is input into the tone classification model for classification processing to obtain the tone classification result, which is used as the tone recognition result.

[0102] The tone classification model can be obtained by training the audio classification model through labeled data.

[0103] Exemplarily, the audio classification model may be a HTS-AT (A HIERARCHICAL TOKEN-SEMANTIC AUDIOTRANSFORMER FOR SOUND CLASSIFICATION AND DETECTION) model.

[0104] In the text enhancement method provided in one or more embodiments of the present application, any one or more feature extraction methods can be used to extract the tone recognition result from the speech data.

[0105] In addition, when the audio features include pause feature information, feature extraction processing is performed on the speech data to obtain the audio features of the speech data. The preprocessed feature data can be converted with the first text input preset audio processing library to obtain time information representing the pause, and the time information is used as the pause feature information.

[0106] For example, in the speech data, before the x1th second, user 1 is speaking, between the x1th second and the x2th second, there is no speaking sound, and after the x2th second, user 1 is speaking. The time period [x1, x2] formed by x1 and x2 can be used as time information representing the pause.

[0107] The preset audio processing library may include one or more audio processing functions, each of which has a corresponding function. One of the audio processing functions may be used to convert the received pre-processed feature data and the first text into time information representing a pause.

[0108] Exemplarily, the preset audio processing library may be a librosa library.

[0109] In the case where the audio features include pause feature information, feature extraction processing is performed on the voice data to obtain the audio features of the voice data, or volume detection processing is performed on the voice data to obtain a volume detection result, and time information characterizing the pause is determined based on a comparison result between the volume detection result and a first preset volume threshold, and the time information is used as the pause feature information.

[0110] For example, in the speech data, before the x1th second, the volume of the speech data is greater than or equal to the first preset volume threshold, starting from the x1th second, the volume of the speech data is less than the first preset volume threshold, and ending at the x2th second, after the x2th second, the volume of the speech data is greater than or equal to the first preset volume threshold. The time period [x1, x2] composed of x1 and x2 can be used as time information representing the pause.

[0111] The first preset volume threshold may be a custom configured volume value. In a time period, when the volume of the voice data is less than the first preset volume threshold, it can be approximately regarded that the voice data in the time period has no speaking sound.

[0112] In the case where the audio features include stress feature information, feature extraction processing is performed on the speech data to obtain the audio features of the speech data. The preprocessed feature data and the first text input are converted into a preset audio processing library to obtain time information representing the stress and a sub-text corresponding to the time information in the first text, and the time information representing the stress and the corresponding sub-text are used as stress feature information.

[0113] For example, in the voice data, user 1 is speaking between seconds x1 and x2, but his pronunciation is light between seconds x1 and x3, and he obviously emphasizes his pronunciation between seconds x3 and x2. The sub-text read out with emphasized pronunciation is "XXXX". The time period [x3, x2] formed by x3 and x2 can be used as time information representing the stress, and the sub-text "XXXX" is the sub-text corresponding to the time period [x3, x2] in the first text.

[0114] In the case where the audio features include stress feature information, feature extraction processing is performed on the speech data to obtain the audio features of the speech data, or volume detection processing is performed on the speech data to obtain a volume detection result, and based on the comparison result between the volume detection result and a second preset volume threshold, time information representing the stress and a sub-text corresponding to the time information in the first text are determined, and the time information representing the stress and the corresponding sub-text are used as stress feature information.

[0115] For example, in the speech data, before the x1th second, the volume of the speech data is less than the second preset volume threshold, starting from the x1th second, the volume of the speech data is greater than or equal to the second preset volume threshold, and ending at the x2th second, after the x2th second, the volume of the speech data is less than the second preset volume threshold. The time period [x1, x2] composed of x1 and x2 can be used as time information representing the stress, and the corresponding subtext is determined in the first text according to the time period [x1, x2].

[0116] The second preset volume threshold may be a custom configured volume value. In a time period, if the volume of the voice data is greater than or equal to the second preset volume threshold, it can be considered that the voice data in the time period are all accents.

[0117] Step S204: determining the type of audio features according to the audio features of the speech data.

[0118] It should be noted that, during the execution of the aforementioned step S202, the type of audio features to be extracted is predetermined, that is, when processing different speech data, the type of audio features to be extracted from each speech data is the same, and the preset feature extraction method adopted in step S202 is also the same. However, considering that different speech data may have different characteristics, after performing the feature extraction process, some types of audio features may not be extracted from the speech data, or some types of audio features may have poor extraction effects and are not suitable for subsequent text enhancement processing. Therefore, in the process of processing different speech data, the types of audio features of the speech data may be the same or different, and it is necessary to determine the type of audio features by executing step S204.

[0119] For example, the types of audio features to be extracted are pre-configured to include a first type and a second type, and the preset feature extraction methods include a feature extraction method of the first type and a feature extraction method of the second type.

[0120] The voice data 1 is subjected to feature extraction processing according to the first type of feature extraction method to obtain the first type of audio feature X1 of the voice data; the voice data is subjected to feature extraction processing according to the second type of feature extraction method to obtain a null value.

[0121] The voice data 2 is subjected to feature extraction processing according to the first type of feature extraction method to obtain the first type of audio feature X2 of the voice data; the voice data is subjected to feature extraction processing according to the second type of feature extraction method to obtain the second type of audio feature X3 of the voice data.

[0122] The voice data 3 is processed for feature extraction according to the first type of feature extraction method to obtain a null value; the voice data is processed for feature extraction according to the second type of feature extraction method to obtain a second type of audio feature X4 of the voice data.

[0123] In summary, according to the audio features of speech data 1, it can be determined that the types of audio features include the first type; according to the audio features of speech data 2, it can be determined that the types of audio features include the first type and the second type; according to the audio features of speech data 3, it can be determined that the types of audio features include the second type.

[0124] For another example, the type of the audio feature to be extracted includes the second type, that is, the audio feature to be extracted includes stress feature information.

[0125] Audio feature extraction is performed on voice data 1. The emotion of voice data 1 fluctuates violently, and part of the voice is stressed. Therefore, stress feature information is extracted from voice data 1, and the extraction effect is very good.

[0126] Audio feature extraction processing is performed on voice data 2. The volume of voice data 2 remains unchanged from the beginning to the end, and there is no obvious accent segment. Therefore, accent feature information cannot be extracted from voice data 2.

[0127] Audio feature extraction processing is performed on the voice data 3. The volume of the voice data 3 fluctuates to a certain extent, but the fluctuation is small and there is no obvious accent segment. Therefore, the accent feature information cannot be extracted from the voice data 3.

[0128] Therefore, even if the same preset feature extraction method is used to perform feature extraction processing on the speech data, the types of audio features of different speech data may be the same or may be different.

[0129] In the process of determining the type of audio features based on the audio features of speech data, the following steps can be performed: for each preset type, determine whether the audio features of the speech data include the audio features of the preset type; if the audio features of the speech data include the audio features of the preset type, add the preset type to the type of audio features.

[0130] The preset types can be custom configured, for example, the preset types include type 1, type 2, and type 3.

[0131] For type 1, determine whether the audio features of the speech data include the audio features of type 1. If the audio features of the speech data include the audio features of type 1, add type 1 to the type of audio features. Otherwise, skip type 1 and continue to process the next type.

[0132] For type 2, determine whether the audio features of the speech data include the audio features of type 2. If the audio features of the speech data include the audio features of type 2, add type 2 to the type of audio features. Otherwise, skip type 2 and continue to process the next type.

[0133] For type 3, determine whether the audio features of the voice data include the audio features of type 3. If the audio features of the voice data include the audio features of type 3, add type 1 to the type of audio features. Otherwise, end the determination process.

[0134] In this way, in the process of processing different voice data, after executing step S204, the type of audio features may include only type 1, may include only type 2, may include only type 3, may include both type 1 and type 2, may include both type 2 and type 3, may include both type 1 and type 3, may include both type 1, type 2 and type 3.

[0135] The type of the audio feature can be determined based on the audio feature of the speech data, or the audio feature of the speech data can be screened and processed, and the type of the screened audio feature can be determined based on the screened audio feature.

[0136] Specifically, the audio features of the speech data are screened and processed, and each preset type of audio features in the audio features of the speech data is evaluated to obtain an evaluation score; if the evaluation score is greater than or equal to a preset score threshold, the type is added to the type of audio features.

[0137] The evaluation score of each preset type of audio feature can reflect the feature extraction effect of the audio feature of this preset type. The higher the evaluation score, the better the feature extraction effect is, and the more suitable the audio feature of this preset type is for text enhancement processing. Conversely, the lower the evaluation score, the worse the feature extraction effect is, and the audio feature of this preset type is not suitable for text enhancement processing.

[0138] The preset type can be custom configured. For example, the preset types include type 1 and type 2.

[0139] For type 1, the audio features of type 1 in the audio features of the speech data are evaluated and processed to obtain an evaluation score. If the evaluation score is greater than or equal to the preset score threshold, type 1 is added to the type of audio features. If the evaluation score is less than the preset score threshold, the audio features of type 1 are deleted from the audio features of the speech data.

[0140] For type 2, the audio features of type 2 in the audio features of the speech data are evaluated and processed to obtain an evaluation score. If the evaluation score is greater than or equal to the preset score threshold, type 2 is added to the type of audio features. If the evaluation score is less than the preset score threshold, the audio features of type 2 are deleted from the audio features of the speech data.

[0141] It should be noted that if the audio features of the speech data do not include audio features of a certain preset type, the evaluation score of the audio features of this type can be set to a preset value, for example, 0, so that the evaluation score is less than the preset score threshold.

[0142] The audio features of speech data are screened based on the consideration that in practical applications, speech data in different scenarios, or speech data from different characters in the same scenario, may have different characteristics, which may lead to huge differences in the extraction effects when extracting different types of audio features from speech data.

[0143] For example, in a complaint scenario, part of the voice data comes from the recording data of a robot agent, whose volume is very stable and there is almost no obvious accent. In this case, extracting accent feature information is not very meaningful. Another part of the voice data comes from the recording data of customers. Customers who have complaints are often emotional, and thus accents appear frequently in the voice data. The effect of extracting accent feature information is very obvious.

[0144] By screening, the types of audio features with poor extraction effects can be eliminated in the process of determining the types of audio features, which is conducive to determining a better text enhancement method in the subsequent steps.

[0145] Step S206: Determine, according to the type of the audio feature, a method for enhancing the second text that matches the first text of the speech data.

[0146] The number of types of audio features may be one or more. The number of enhancement methods of the second text may be one or more.

[0147] The second text matching the first text refers to an enhanced text that is semantically consistent with the first text. For a first text, the number of second texts matching the first text may be greater than or equal to one.

[0148] Text augmentation is a technique used in the field of NLP to improve the generalization ability of models and data diversity. It generates new training samples by performing a series of transformations on the original text data, thereby expanding the data set without adding additional data. The purpose of text augmentation is to help the model better learn the diversity and complexity of language, reduce overfitting, and improve the performance of the model on unknown data.

[0149] There are many ways to enhance text, including but not limited to the following:

[0150] Synonym Replacement: Replace certain words in a sentence with their synonyms.

[0151] Random Insertion: Randomly select a word and insert its synonyms at random positions in the sentence.

[0152] Random Swap: Randomly select two words in a sentence and swap their positions.

[0153] Random Deletion: Randomly delete words in a sentence with a certain probability.

[0154] Back Translation: Translating a text into another language and then translating it back into the original language.

[0155] Text Rewriting: Use a language model to generate new sentences that are semantically similar to the original sentence but expressed differently.

[0156] Rule-based Transformation: Transform text according to a predefined set of rules, such as changing active voice to passive voice.

[0157] The second text may be enhanced in one of the above-mentioned text enhancement methods, or in a punctuation enhancement method, or in a function word enhancement method, and so on.

[0158] Punctuation enhancement refers to inserting new punctuation marks into the text, or deleting the original punctuation marks, or updating the original punctuation marks, and so on.

[0159] Function words can be enhanced by inserting new function words into the text, or deleting the original function words, or updating the original function words, and so on.

[0160] Function words refer to words that do not have complete meanings but have grammatical meanings or functions. They must be attached to content words or sentences, express grammatical meanings, cannot form sentences alone, cannot be used as grammatical components alone, and cannot be repeated.

[0161] During specific implementation, a third correspondence table may be preconfigured, and the third correspondence table includes at least one set of correspondences between enhancement modes and types of audio features.

[0162] In some implementations, determining the enhancement method of the second text that matches the first text of the voice data according to the type of audio features can be performed by querying the aforementioned third correspondence table according to the type of audio features to obtain the enhancement method of the second text that matches the first text of the voice data.

[0163] For example, the third correspondence table includes:

[0164] Enhancement mode 1<——>Type 1, Type 2;

[0165] Enhancement mode 2<——>Type 1, Type 3;

[0166] Enhancement type 3 <——> type 4, and so on.

[0167] The types of audio features of speech data 1 include: type 1 and type 2.

[0168] A query process is performed in the third correspondence table according to type 1 and type 2 to obtain an enhancement method of the second text matching the first text of voice data 1 , where the enhancement method of the second text is enhancement method 1 .

[0169] The types of audio features of speech data 2 include: type 1 and type 3.

[0170] According to type 1 and type 3, query processing is performed in the third correspondence table to obtain the enhancement method of the second text matching the first text of the voice data 2, and the enhancement method of the second text is enhancement method 2.

[0171] For another example, the third correspondence table includes:

[0172] Enhancement mode 1<——>Type 1, Type 2;

[0173] Enhancement mode 2<——>Type 1, Type 3;

[0174] Enhancement type 3 <——> type 4, and so on.

[0175] The types of audio features of voice data 3 include: Type 1.

[0176] According to type 1, query processing is performed in the third correspondence table to obtain the enhancement method of the second text matching the first text of the voice data 3. The enhancement methods of the second text include: enhancement method 1 and enhancement method 2.

[0177] The types of audio features of speech data 4 include: type 2 and type 4.

[0178] According to type 2, query processing is performed in the third correspondence table to obtain the enhancement method of the second text matching the first text of the voice data 4, and the enhancement method of the second text includes enhancement method 1.

[0179] According to type 4, query processing is performed in the third correspondence table to obtain the enhancement method of the second text matching the first text of voice data 4, and the enhancement method of the second text includes enhancement method 3.

[0180] In summary, the enhancement methods of the second text corresponding to the voice data 4 include: enhancement method 1 and enhancement method 3.

[0181] In addition, in the text enhancement method provided in the embodiment of the present application, the reason why audio features are extracted first and then the enhancement method of the second text matching the first text of the voice data is determined according to the type of audio features, rather than adopting a fixed enhancement method, is that in actual applications, voice data in different scenarios, or voice data from different characters in the same scenario, may have different characteristics, which may lead to huge differences in the extraction effect when extracting audio features from voice data.

[0182] For example, in a complaint scenario, part of the voice data comes from the recording data of a robot agent, whose volume is very stable and there is almost no obvious accent. In this case, extracting accent feature information is not very meaningful. Another part of the voice data comes from the recording data of customers. Customers who have complaints are often emotional, and thus accents appear frequently in the voice data. The effect of extracting accent feature information is very obvious.

[0183] If we ignore the differences in the characteristics of speech data in different scenarios, treat all speech data equally, extract the same type of audio features, and predetermine a fixed enhancement method, it may result in that the text enhancement effect of speech data is very good in some scenarios, but the text enhancement effect of speech data is poor in other scenarios.

[0184] In a specific implementation, the type of audio features includes at least one of a first type, a second type and a third type, the first type is pause feature information describing silent segments in speech data, the second type is stress feature information describing stress segments in speech data, and the third type is a tone recognition result of speech data; determining an enhancement method for a second text matching a first text of speech data according to the type of audio features includes: if the type of audio features includes the first type, determining that the enhancement method for the second text matching the first text includes the first enhancement method; if the type of audio features includes the second type, determining that the enhancement method for the second text matching the first text includes the second enhancement method; if the type of audio features includes the third type, determining that the enhancement method for the second text matching the first text includes the first enhancement method and the second enhancement method.

[0185] The first enhancement method can refer to the aforementioned punctuation enhancement method. The second enhancement method can refer to the aforementioned function word enhancement method.

[0186] The first type is the pause feature information describing the silent segments in the speech data. The second type is the stress feature information describing the stress segments in the speech data. The third type is the tone recognition result of the speech data.

[0187] In the specific implementation process of determining the enhancement method of the second text matching the first text of the speech data according to the type of the audio feature, there may be the following situations:

[0188] (c1) The types of audio features include only the first type, that is, the audio features include only pause feature information.

[0189] In this case, it may be determined that the enhancement mode of the second text matching the first text includes the first enhancement mode.

[0190] (c2) The type of the audio feature includes only the second type, that is, the audio feature includes only accent feature information.

[0191] In this case, it can be determined that the enhancement method of the second text matching the first text includes the second enhancement method.

[0192] (c3) The types of audio features only include the third type, that is, the audio features only include the tone recognition results.

[0193] In this case, it can be determined that the enhancement mode of the second text matching the first text includes a first enhancement mode and a second enhancement mode.

[0194] (c4) The type of the audio feature includes both the first type and the second type, that is, the audio feature includes both pause feature information and stress feature information.

[0195] In this case, it can be determined that the enhancement mode of the second text matching the first text includes a first enhancement mode and a second enhancement mode.

[0196] (c5) The type of audio features includes both the first type and the third type, that is, the audio features include both pause feature information and tone recognition results.

[0197] In this case, it can be determined that the enhancement mode of the second text matching the first text includes a first enhancement mode and a second enhancement mode.

[0198] (c6) The type of the audio feature includes both the second type and the third type, that is, the audio feature includes both the stress feature information and the tone recognition result.

[0199] In this case, it can be determined that the enhancement mode of the second text matching the first text includes a first enhancement mode and a second enhancement mode.

[0200] (c7) The types of audio features include the first type, the second type and the third type, that is, the audio features include pause feature information, stress feature information and tone recognition results.

[0201] In this case, it can be determined that the enhancement mode of the second text matching the first text includes a first enhancement mode and a second enhancement mode.

[0202] In another implementation, the type of audio features includes at least one of a first type, a second type, and a third type, the first type is pause feature information describing silent segments in speech data, the second type is stress feature information describing stress segments in speech data, and the third type is a tone recognition result of speech data; determining an enhancement method for a second text matching a first text of speech data according to the type of audio features includes: if the type of audio features includes the first type and the third type, determining that the enhancement method for the second text matching the first text includes the first enhancement method; if the type of audio features includes the second type and the third type, determining that the enhancement method for the second text matching the first text includes the second enhancement method.

[0203] In addition, if the type of audio features does not meet the above two conditions, a custom configured preset enhancement method can be used.

[0204] In summary, in the specific implementation process of determining the enhancement method of the second text matching the first text of the speech data according to the type of the audio feature, there may be the following situations:

[0205] (d1) The types of audio features include only the first type, that is, the audio features include only pause feature information.

[0206] In this case, it may be determined that the enhancement mode of the second text matching the first text includes a preset enhancement mode.

[0207] (d2) The type of the audio feature includes only the second type, that is, the audio feature includes only accent feature information.

[0208] In this case, it may be determined that the enhancement mode of the second text matching the first text includes a preset enhancement mode.

[0209] (d3) The types of audio features only include the third type, that is, the audio features only include tone recognition results.

[0210] In this case, it may be determined that the enhancement mode of the second text matching the first text includes a preset enhancement mode.

[0211] (d4) The type of the audio feature includes both the first type and the second type, that is, the audio feature includes both pause feature information and stress feature information.

[0212] In this case, it may be determined that the enhancement mode of the second text matching the first text includes a preset enhancement mode.

[0213] (d5) The type of audio features includes both the first type and the third type, that is, the audio features include both pause feature information and tone recognition results.

[0214] In this case, it may be determined that the enhancement mode of the second text matching the first text includes the first enhancement mode.

[0215] (d6) The type of the audio feature includes both the second type and the third type, that is, the audio feature includes both the stress feature information and the tone recognition result.

[0216] In this case, it may be determined that the enhancement mode of the second text matching the first text includes the second enhancement mode.

[0217] (d7) The types of audio features include the first type, the second type and the third type, that is, the audio features include pause feature information, stress feature information and tone recognition results.

[0218] In this case, it can be determined that the enhancement mode of the second text matching the first text includes a first enhancement mode and a second enhancement mode.

[0219] Step S208: Perform text enhancement processing on the first text according to the audio feature and the enhancement method of the second text to obtain the second text.

[0220] The number of enhancement modes for the second text may be one or more.

[0221] For each enhancement method, the first text can be enhanced once according to the audio features and the enhancement method to obtain a second text, or the first text can be enhanced multiple times according to the audio features and the enhancement method to obtain multiple different second texts.

[0222] In addition, a first text may be enhanced using a plurality of different enhancement methods to obtain a second text.

[0223] For example, a punctuation enhancement process is performed on the first text to obtain a second text 1;

[0224] Perform function word enhancement processing on the first text to obtain a second text 2;

[0225] Perform function word enhancement processing on the first text to obtain a second text 3;

[0226] Perform function word enhancement processing on the first text to obtain a second text 4;

[0227] The first text is subjected to a punctuation enhancement process and a function word enhancement process to obtain a second text 5.

[0228] The punctuation enhancement processing refers to performing text enhancement processing according to a first enhancement method, and the function word enhancement processing refers to performing text enhancement processing according to a second enhancement method.

[0229] The number of the second texts may be one or more.

[0230] During specific implementation, multiple enhancement methods and a text enhancement model corresponding to each enhancement method may be pre-configured, and the text enhancement model is used to perform text enhancement processing on the input text through the enhancement method.

[0231] The first text is enhanced according to the audio features and the enhancement method of the second text to obtain the second text. The corresponding target text enhancement model is determined according to the enhancement method of the second text, and the audio features and the first text are input into the target text enhancement model for text enhancement to obtain the second text.

[0232] After the second text is generated, the second text may be included as enhanced text of the first text.

[0233] The semantic similarity between the first text and the second text may be less than a preset similarity threshold.

[0234] For example, the first text is: Not his sister.

[0235] The first text is enhanced according to the audio features and the enhancement method of the second text to obtain the second text:

[0236] Second Text 1: No, his sister.

[0237] Second Text 2: No, his sister.

[0238] Second Text 3: No, his sister.

[0239] In a specific implementation, the audio features include pause feature information and tone recognition results, and the enhancement method of the second text includes the first enhancement method; the first text is enhanced according to the audio features and the enhancement method of the second text to obtain the second text, including: determining the punctuation type of the first punctuation to be inserted in the first text and the insertion position of the first punctuation in the first text according to the pause feature information and the tone recognition result; inserting the first punctuation in the first text according to the punctuation type and the insertion position to obtain the second text.

[0240] The first punctuation mark can be any punctuation mark, for example, an exclamation mark, a question mark, a comma, etc.

[0241] The audio feature may include pause feature information and tone recognition results. In this case, the enhancement method of the second text may include the first enhancement method. Then, the first text is enhanced according to the audio feature and the enhancement method of the second text to obtain the second text. The first text may be enhanced according to the pause feature information, the tone recognition result, and the first enhancement method to obtain the second text. In the process of performing text enhancement on the first text according to the pause feature information, the tone recognition result, and the first enhancement method to obtain the second text, the following steps may be performed: according to the pause feature information and the tone recognition result, the punctuation type of the first punctuation to be inserted in the first text and the insertion position of the first punctuation in the first text are determined; according to the punctuation type and the insertion position, the first punctuation is inserted into the first text to obtain the second text.

[0242] The punctuation type of the first punctuation mark may be one of a plurality of preset punctuation mark types. Exemplarily, the plurality of preset punctuation mark types include: an exclamation mark, a question mark, a comma, and the like.

[0243] Taking the example of a first text including K characters, where K is an integer greater than 1, the insertion position of the first punctuation mark can be represented by the arrangement sequence number of a character adjacent to and before the first punctuation mark after the first punctuation mark is inserted into the first text among the K characters, or can be represented by the arrangement sequence number of a character adjacent to and after the first punctuation mark after the first punctuation mark is inserted into the first text among the K characters.

[0244] For example, the first text is: ABCDEFG. The first punctuation mark "," needs to be inserted between C and D. The arrangement number of C in the 7 characters included in the first text is "3", and the insertion position of the first punctuation mark can be represented by "3".

[0245] For another example, the first text is: ABCDEFG. The first punctuation mark "?" needs to be inserted between F and G. The arrangement number of G in the 7 characters included in the first text is "7", so the insertion position of the first punctuation mark can be represented by "7".

[0246] According to the pause feature information and the tone recognition result, the punctuation type of the first punctuation to be inserted in the first text and the insertion position of the first punctuation in the first text are determined. The punctuation type of the first punctuation to be inserted in the first text can be determined according to the tone recognition result, and the insertion position of the first punctuation to be inserted in the first text can be determined according to the pause feature information. In specific implementation, the correspondence between the tone and the punctuation type can be pre-configured. According to the tone recognition result, the punctuation type of the first punctuation to be inserted in the first text can be determined. According to the tone recognition result, the punctuation type of the first punctuation to be inserted in the first text can be obtained by querying the correspondence between the tone and the punctuation type according to the tone recognition result.

[0247] For example, the pause feature information includes a time information T1 representing the pause, and the tone recognition result is "exclamation". According to the tone recognition result "exclamation", it can be determined that the punctuation type of the first punctuation is "exclamation mark", and according to the time information T1 representing the pause, it is determined in the first text that the first punctuation needs to be inserted between the Nth character and the N+1th character in the first text, where N is an integer greater than 0, and the insertion position of the first punctuation can be represented by N.

[0248] According to the punctuation type and the insertion position, a first punctuation mark is inserted into the first text to obtain a second text.

[0249] For example, if the punctuation type is "exclamation mark", the insertion position of the first punctuation mark is represented by "N", where N is an integer greater than 0. An exclamation mark is added after the Nth character in the first text, and the original N+1th character and subsequent characters are all moved back one position.

[0250] In a specific implementation manner, the audio features include stress feature information and a tone recognition result, and the enhancement manner of the second text includes a second enhancement manner; text enhancement processing is performed on the first text according to the audio features, the second text's enhancement manner to obtain the second text, including: determining the type of the first function word to be inserted in the first text and the insertion position of the first function word in the first text according to the stress feature information and the tone recognition result; inserting the first function word into the first text according to the type of the function word and the insertion position to obtain the second text.

[0251] The first function word can be any function word, for example, "ah", "de", "heng", and so on.

[0252] The audio features may include stress feature information and a tone recognition result. In this case, the enhancement manner of the second text may include a second enhancement manner. Then, text enhancement processing is performed on the first text according to the audio features and the second text's enhancement manner to obtain the second text, which may be to perform text enhancement processing on the first text according to the stress feature information, the tone recognition result, and the second enhancement manner to obtain the second text.

[0253] In the process of performing text enhancement processing on the first text according to the stress feature information, the tone recognition result, and the second enhancement manner to obtain the second text, the following steps may be executed: determining the type of the first function word to be inserted in the first text and the insertion position of the first function word in the first text according to the stress feature information and the tone recognition result; inserting the first function word into the first text according to the type of the function word and the insertion position to obtain the second text.

[0254] The type of the first function word may be one of multiple preset function word types. Exemplarily, the multiple preset function word types include: "ah", "de", "heng", and so on.

[0255] Taking the first text including K characters as an example, where K is an integer greater than 1, the insertion position of the first function word can be represented by the arrangement serial number of a character adjacent to and before the first function word in the K characters after inserting the first function word into the first text, or can also be represented by the arrangement serial number of a character adjacent to and after the first function word in the K characters after inserting the first function word into the first text.

[0256] For example, the first text is: ABCDEFG. The first function word "de" needs to be inserted between C and D, and the arrangement serial number of C in the 7 characters included in the first text is "3", then the insertion position of the first function word can be represented by "3".

[0257] For another example, the first text is: ABCDEFG. The first punctuation mark "ah" needs to be inserted between F and G. The arrangement serial number of G among the 7 characters included in the first text is "7", so the insertion position of the first function word can be represented by "7".

[0258] According to the stress feature information and the mood recognition result, to determine the function word type of the first function word to be inserted in the first text and the insertion position of the first function word in the first text, it can be to determine the function word type of the first function word to be inserted in the first text according to the mood recognition result, and, according to the stress feature information, determine the insertion position of the first function word to be inserted in the first text in the first text.

[0259] In specific implementation, the corresponding relationship between the mood and the function word type can be pre-configured. To determine the function word type of the first function word to be inserted in the first text according to the mood recognition result, it can be to query and process in the corresponding relationship between the mood and the function word type according to the mood recognition result to obtain the function word type of the first function word.

[0260] For example, the stress feature information includes a time information T2 representing stress, and the mood recognition result is "doubt". According to the mood recognition result of "doubt", it can be determined that the function word type of the first function word is "ah", and, according to the time information T2 representing stress, it is determined in the first text that the first function word needs to be inserted between the Nth character and the (N + 1)th character in the first text, where N is an integer greater than 0, then the insertion position of the first function word can be represented by N. According to the function word type and the insertion position, insert the first function word in the first text to obtain the second text.

[0261] For example, the function word type is "ah", and the insertion position of the first function word is represented by "N", where N is an integer greater than 0. Add an "ah" after the Nth character in the first text, and move the original (N + 1)th character and the subsequent characters one position backward.

[0262] In a specific implementation manner, after performing text enhancement processing on the first text according to the audio feature and the enhancement method of the second text to obtain the second text, the text enhancement method further includes: if the number of the second texts is greater than the preset number threshold, perform an evaluation process on each second text to obtain the evaluation score of each second text; according to the evaluation score of each second text, select a preset number of second texts as the target texts.

[0263] Performing an evaluation process on each second text can be to input each second text into a preset evaluation model for evaluation processing to obtain the evaluation score.

[0264] The preset evaluation model can be obtained by training a classification model with labeled data.

[0265] For any second text, the higher the evaluation score of the second text, the better the text enhancement effect of the second text. Conversely, the lower the evaluation score of the second text, the worse the text enhancement effect of the second text.

[0266] According to the ranking result of the evaluation score of each second text, a preset number of second texts are selected as target texts. The target texts may be high-quality enhanced texts with expected text enhancement effects.

[0267] In addition, the text enhancement method may further include: constructing training data or a prompt text set of a preset language model based on the first text and the second text.

[0268] The preset language model can be a text classification model or a text generation model, etc.

[0269] Constructing training data of a preset language model based on a first text and a second text may be performed by annotating the second text based on the label information of the first text to obtain the training data when the annotated data of the preset language model includes the first text and the label information of the first text.

[0270] Since the second text is an enhanced text of the first text, the semantics of the first text are consistent with those of the second text. Therefore, when the annotation data used for model training of the preset language model includes the first text, the label information of the first text can be copied and added to the second text, thereby quickly expanding the annotation data and increasing the number of samples.

[0271] Constructing a prompt text set of a preset language model according to the first text and the second text may be to assemble the first text and the second text to obtain the prompt text set of the preset language model.

[0272] In such Figure 2In the embodiment shown, first, feature extraction processing is performed on the speech data to obtain the audio features of the speech data; then, the type of the audio features is determined according to the audio features of the speech data; then, the enhancement method of the second text matching the first text of the speech data is determined according to the type of the audio features; finally, the first text is subjected to text enhancement processing according to the audio features and the enhancement method of the second text to obtain the second text. In this way, considering that the speech data collected in different scenes, or the speech data of different roles from the same scene, may have their own characteristics, which leads to differences in the extraction effects of different types of audio features in the speech data, in practical applications, it may occur that some types of audio features can be successfully extracted from the speech data, but the extraction effect of other types of audio features is poor, which may result in different types of audio features extracted from each speech data after feature extraction processing is performed on different speech data. Therefore, by first extracting the audio features, then determining the type of the audio features, and then determining the enhancement method of the second text matching the first text of the speech data according to the type of the audio features, compared with the use of a fixed text enhancement method, a personalized text enhancement method suitable for the speech data can be selected more flexibly, which is conducive to optimizing the text enhancement effect when performing text enhancement processing in subsequent steps.

[0273] Figure 3 A processing flow chart of another text enhancement method provided in an embodiment of the present application.

[0274] Step S302, inputting the speech data into the speech feature extraction model.

[0275] The speech feature extraction model may include: a first sub-model for extracting a first audio feature from the speech data, a second sub-model for extracting a second audio feature from the speech data, and a third sub-model for extracting a third audio feature from the speech data.

[0276] The first audio feature, the second audio feature, and the third audio feature are three different types of audio features. Figure 2 Corresponding description part of the audio features in the embodiment.

[0277] Step S304: The speech feature extraction model discriminates the tone, pause, etc. of each character and outputs the speech features in a conversation.

[0278] For step S302 and step S304 , reference may be made to the corresponding description of the aforementioned step S202 .

[0279] Step S306, performing various corrections on the first text according to punctuation marks and function words representing different tones and pauses in the language.

[0280] For step S306 , reference may be made to the corresponding description parts of the aforementioned steps S204 - S208 .

[0281] Step S308: All the multiple correction results are collected as enhanced text of the first text.

[0282] The enhanced text of the first text may refer to the corresponding description part of the aforementioned second text.

[0283] Since the technical concept is the same, the description in this embodiment is relatively simple, and the relevant parts may refer to the corresponding description of the method embodiment provided above.

[0284] The following takes the application of a text enhancement method provided in this embodiment in a consulting scenario as an example. Figure 4 The text enhancement method provided in this embodiment is further described. Figure 4 A processing flow chart of another text enhancement method provided in an embodiment of the present application.

[0285] like Figure 4 As shown, in step S402, the voice data of the robot agent and the customer are input into the voice feature extraction model.

[0286] The speech feature extraction model may include: a first sub-model for extracting a first audio feature from the speech data, a second sub-model for extracting a second audio feature from the speech data, and a third sub-model for extracting a third audio feature from the speech data.

[0287] The first audio feature, the second audio feature, and the third audio feature are three different types of audio features. Figure 2 Corresponding description part of the audio features in the embodiment.

[0288] Step S404: The speech feature extraction model determines the tone, pauses, etc. of each sentence of each character during the conversation, and the model outputs the speech features of each sentence in the conversation.

[0289] For step S402 and step S404 , reference may be made to the corresponding description of the aforementioned step S202 .

[0290] Step S406, performing various corrections on the first text according to punctuation marks and function words representing different tones and pauses in the language.

[0291] For step S406 , reference may be made to the corresponding description of the aforementioned steps S204 to S208 .

[0292] For example, the first text is: I can't hear it clearly again.

[0293] Correction result 1 is: Say it again, can’t hear clearly.

[0294] Correction result 2 is: Say it again, I can’t hear it clearly.

[0295] Correction result 3 is: Please say it again, I can’t hear it clearly.

[0296] Step S408: All the multiple correction results are collected as enhanced text of the first text.

[0297] The enhanced text of the first text may refer to the corresponding description part of the aforementioned second text.

[0298] Step S410: Using the enhanced customer data for iterative training of a customer intent recognition model.

[0299] The enhanced customer data is the enhanced text of the aforementioned first text.

[0300] For example, the first text is: repeat, I can't hear clearly. The intention corresponding to the first text is "I didn't hear clearly".

[0301] Correction result 1 is: Say it again, can’t hear clearly.

[0302] Correction result 2 is: Say it again, I can’t hear it clearly.

[0303] Correction result 3 is: Please say it again, I can’t hear it clearly.

[0304] Regarding the intention of "didn't hear clearly", correction result 1, correction result 2 and correction result 3 can all be included as enhanced text of the first text, and the enhanced text is stored together with the first text in the "didn't hear clearly" intention data of the customer intent recognition model, which enhances the diversity of training data and will have a positive impact on the training of the customer intent recognition model.

[0305] Step S412, using the enhanced agent data as training data or prompt text for a text generation model in a consultation scenario.

[0306] In the consultation scenario, the robot agent needs to input the customer's consultation questions into the text generation model for text generation processing in order to respond to the customer.

[0307] For example, after the customer expresses the intention of “didn’t hear clearly”, the agent’s corresponding standard speech is “Sorry, I’ll repeat it, XXXXXXX”, and the intention of this speech is recorded as “repeated statement”.

[0308] The text enhancement processing for the "repeated statement" text is as follows:

[0309] Enhancement 1: Sorry, I'll say it again.

[0310] Enhancement 2: Sorry, I'll repeat myself.

[0311] Enhancement 3: Sorry, let me repeat it again.

[0312] While inputting the customer's consultation questions into the text generation model, the robot agent can also give the text generation model some answer-related prompts. Specifically, the enhanced text of the first text can be given to the text generation model as a prompt to improve the accuracy of the text generation model's response.

[0313] Since the technical concept is the same, the description in this embodiment is relatively simple, and the relevant parts may refer to the corresponding description of the method embodiment provided above.

[0314] An embodiment of a text enhancement device provided in this specification is as follows:

[0315] In the above-mentioned embodiment, a text enhancement method is provided. Based on the same technical concept, the embodiment of the present application also provides a text enhancement device, which is described below in conjunction with the accompanying drawings.

[0316] Figure 5 A schematic diagram of a text enhancement device provided in an embodiment of the present application. This embodiment provides a text enhancement device 500, including:

[0317] The extraction unit 502 is used to perform feature extraction processing on the speech data to obtain audio features of the speech data;

[0318] A determining unit 504, configured to determine a type of the audio feature according to the audio feature of the speech data;

[0319] The determining unit 504 is further configured to determine, according to the type of the audio feature, an enhancement method for a second text matching the first text of the speech data;

[0320] The enhancement unit 506 is used to perform text enhancement processing on the first text according to the audio feature and the enhancement method of the second text to obtain a second text.

[0321] Optionally, the type of the audio feature includes at least one of a first type, a second type, and a third type, the first type is pause feature information describing a silent segment in the voice data, the second type is stress feature information describing a stress segment in the voice data, and the third type is a tone recognition result of the voice data; when the determination unit 506 determines the enhancement method of the second text matching the first text of the voice data according to the type of the audio feature, the determination unit 506 performs the following steps:

[0322] If the type of the audio feature includes the first type, determining that the enhancement method of the second text matching the first text includes the first enhancement method;

[0323] If the type of the audio feature includes the second type, determining that the enhancement method of the second text matching the first text includes the second enhancement method;

[0324] If the type of the audio feature includes the third type, determining that the enhancement method of the second text matching the first text includes the first enhancement method and the second enhancement method.

[0325] Optionally, the audio feature includes the pause feature information and the tone recognition result, and the enhancement method of the second text includes the first enhancement method; when the enhancement unit 506 performs text enhancement processing on the first text according to the audio feature and the enhancement method of the second text to obtain the second text, the following steps are performed:

[0326] Determining, according to the pause feature information and the tone recognition result, a punctuation type of a first punctuation mark to be inserted into the first text and an insertion position of the first punctuation mark in the first text;

[0327] The first punctuation mark is inserted into the first text according to the punctuation mark type and the insertion position to obtain the second text.

[0328] Optionally, the audio feature includes the stress feature information and the tone recognition result, and the enhancement method of the second text includes the second enhancement method; when the enhancement unit 506 performs text enhancement processing on the first text according to the audio feature, the audio feature, and the enhancement method of the second text to obtain the second text, the following steps are performed:

[0329] Determining, according to the stress feature information and the tone recognition result, a function word type of a first function word to be inserted into the first text and an insertion position of the first function word in the first text;

[0330] The first function word is inserted into the first text according to the function word type and the insertion position to obtain the second text.

[0331] Optionally, the audio feature of the speech data includes a tone recognition result of the speech data; when the extraction unit 502 performs feature extraction processing on the speech data to obtain the audio feature of the speech data, the extraction unit 502 performs the following steps:

[0332] Converting the speech data to obtain a spectrogram of the speech data; the spectrogram is used to represent pitch information of the speech data;

[0333] Segmenting the spectrogram to obtain a plurality of sub-spectrograms arranged in chronological order;

[0334] Determining pitch change information of the speech data in two adjacent sub-spectrograms according to the multiple sub-spectrograms;

[0335] If the pitch change information indicates that the pitch information of the speech data increases, determining that the pitch comparison results of two adjacent sub-spectrogens are the first preset results;

[0336] If the pitch change information indicates that the pitch information of the speech data remains unchanged, determining that the pitch comparison results of two adjacent sub-spectrogens are the second preset results;

[0337] If the pitch change information indicates that the pitch information of the speech data decreases, determining that the pitch comparison results of two adjacent sub-spectrogens are the third preset results;

[0338] According to the pitch comparison results of two adjacent sub-spectrograms, a query is performed in the pre-configured correspondence between the pitch comparison results and the tone to obtain the tone recognition result of the speech data.

[0339] Optionally, when determining the pitch change information of the speech data in two adjacent sub-spectroglas according to the multiple sub-spectroglas, the extraction unit 502 performs the following steps:

[0340] Determining the average pitch of each sub-spectrogram according to the pitch information included in the sub-spectrogram;

[0341] Calculating a first difference in average pitches of two adjacent sub-spectrograms among the plurality of sub-spectrograms;

[0342] If the first difference is less than a preset value, determining that the pitch change information represents an increase in pitch information of the voice data;

[0343] If the first difference is equal to a preset value, determining that the pitch information representing the pitch information of the speech data remains unchanged;

[0344] If the first difference is greater than a preset value, it is determined that the pitch change information represents a decrease in pitch information of the speech data.

[0345] Optionally, the text enhancement device 500 further includes:

[0346] an evaluation unit, configured to, if the number of the second texts is greater than a preset number threshold, evaluate each second text to obtain an evaluation score for each second text;

[0347] A selection unit is used to select a preset number of second texts as target texts according to the evaluation score of each second text.

[0348] In an embodiment of the present application, a text enhancement device includes: an extraction unit for performing feature extraction processing on speech data to obtain audio features of the speech data; a determination unit for determining the type of audio features based on the audio features of the speech data; the determination unit is also used to determine the enhancement method of the second text matching the first text of the speech data based on the type of audio features; an enhancement unit is used to perform text enhancement processing on the first text based on the audio features and the enhancement method of the second text to obtain the second text. In this way, considering that speech data collected in different scenes, or speech data from different characters in the same scene, may have their own characteristics, which leads to differences in the extraction effects of different types of audio features in the speech data, in actual applications, it may happen that some types of audio features can be successfully extracted from the speech data, but the extraction effects of other types of audio features are poor, which may result in different types of audio features extracted from each speech data after feature extraction processing is performed on different speech data. Therefore, by first extracting the audio features, then determining the type of the audio features, and then determining the enhancement method of the second text that matches the first text of the voice data according to the type of the audio features, compared with using a fixed text enhancement method, a personalized text enhancement method suitable for the voice data can be selected more flexibly, which is conducive to optimizing the text enhancement effect when performing text enhancement processing in subsequent steps.

[0349] Corresponding to the above-described text enhancement method, based on the same technical concept, an embodiment of the present application further provides an electronic device, which is used to execute the above-provided text enhancement method. Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0350] like Figure 6As shown, the electronic device may have relatively large differences due to different configurations or performances, and may include one or more processors 601 and memory 602, and the memory 602 may store one or more storage applications or data. Among them, the memory 602 can be a temporary storage or a permanent storage. The application stored in the memory 602 may include one or more modules (not shown in the figure), and each module may include a series of computer executable instructions in the electronic device. Furthermore, the processor 601 can be configured to communicate with the memory 602 to execute a series of computer executable instructions in the memory 602 on the electronic device. The electronic device may also include one or more power supplies 603, one or more wired or wireless network interfaces 604, one or more input / output interfaces 605, one or more keyboards 606, etc.

[0351] In a specific embodiment, the electronic device includes a memory and one or more programs, wherein the one or more programs are stored in the memory, and the one or more programs may include one or more modules, and each module may include a series of computer executable instructions for the electronic device, and the one or more programs configured to be executed by one or more processors include the following computer executable instructions:

[0352] Performing feature extraction processing on the speech data to obtain audio features of the speech data;

[0353] Determining the type of the audio feature according to the audio feature of the speech data;

[0354] Determining, according to the type of the audio feature, a manner of enhancing a second text that matches the first text of the speech data;

[0355] The first text is enhanced according to the audio feature and the enhancement method of the second text to obtain a second text.

[0356] The computer-readable storage medium provided in this embodiment is used to store computer-executable instructions. When the computer-executable instructions are executed by a processor, the following process is implemented:

[0357] Performing feature extraction processing on the speech data to obtain audio features of the speech data;

[0358] Determining the type of the audio feature according to the audio feature of the speech data;

[0359] Determining, according to the type of the audio feature, a manner of enhancing a second text that matches the first text of the speech data;

[0360] The first text is enhanced according to the audio features and the enhancement method of the second text to obtain the second text. It should be noted that the embodiment of the computer-readable storage medium in this specification and the embodiment of the text enhancement method in this specification are based on the same inventive concept, so the specific implementation of this embodiment can refer to the implementation of the corresponding method above, and the repeated parts will not be repeated.

[0361] Another embodiment of the present disclosure further provides a computer program product, the computer program product comprising a computer program, and when the computer program is executed by a processor, the following process is implemented:

[0362] Performing feature extraction processing on the speech data to obtain audio features of the speech data;

[0363] Determining the type of the audio feature according to the audio feature of the speech data;

[0364] Determining, according to the type of the audio feature, a manner of enhancing a second text that matches the first text of the speech data;

[0365] The first text is enhanced according to the audio feature and the enhancement method of the second text to obtain a second text.

[0366] The computer program product in the embodiment of the present disclosure can implement each process of the above-mentioned text enhancement method embodiment and achieve the same effect and function, which will not be repeated here.

[0367] In an embodiment of the present application, first, feature extraction processing is performed on the speech data to obtain the audio features of the speech data; then, the type of audio features is determined based on the audio features of the speech data; then, the enhancement method of the second text matching the first text of the speech data is determined based on the type of audio features; finally, text enhancement processing is performed on the first text based on the audio features and the enhancement method of the second text to obtain the second text. In this way, considering that speech data collected in different scenes, or speech data from different characters in the same scene, may have their own characteristics, this characteristic leads to differences in the extraction effects of different types of audio features in the speech data. In actual applications, it may happen that some types of audio features can be successfully extracted from the speech data, but the extraction effects of other types of audio features are poor. This may result in different types of audio features extracted from each speech data after feature extraction processing is performed on different speech data. Therefore, by first extracting the audio features, then determining the type of the audio features, and then determining the enhancement method of the second text that matches the first text of the voice data according to the type of the audio features, compared with using a fixed text enhancement method, a personalized text enhancement method suitable for the voice data can be selected more flexibly, which is conducive to optimizing the text enhancement effect when performing text enhancement processing in subsequent steps.

[0368] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0369] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the embodiments of the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, this specification may adopt the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0370] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable electronic device to produce a machine, so that the instructions executed by the processor of the computer or other programmable electronic device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0371] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable electronic device to work in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0372] These computer program instructions may also be loaded onto a computer or other programmable electronic device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0373] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0374] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0375] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0376] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0377] Embodiments of the present application may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. One or more embodiments of the present specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communications network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0378] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0379] The above description is only an embodiment of this document and is not intended to limit this document. For those skilled in the art, this document may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this document should be included in the scope of the claims of this document.

Claims

1. A text enhancement method, characterized in that: include: Performing feature extraction processing on the speech data to obtain audio features of the speech data; Determining the type of the audio feature according to the audio feature of the speech data; Determining, according to the type of the audio feature, a manner of enhancing a second text that matches the first text of the speech data; The first text is enhanced according to the audio feature and the enhancement method of the second text to obtain a second text.

2. The method according to claim 1, characterized in that The type of the audio feature includes at least one of a first type, a second type, and a third type, the first type is pause feature information describing a silent segment in the voice data, the second type is stress feature information describing a stress segment in the voice data, and the third type is a tone recognition result of the voice data; the method of determining, according to the type of the audio feature, an enhancement method of a second text matching the first text of the voice data includes: If the type of the audio feature includes the first type, determining that the enhancement method of the second text matching the first text includes the first enhancement method; If the type of the audio feature includes the second type, determining that the enhancement method of the second text matching the first text includes the second enhancement method; If the type of the audio feature includes the third type, determining that the enhancement method of the second text matching the first text includes the first enhancement method and the second enhancement method.

3. The method according to claim 2, characterized in that The audio feature includes the pause feature information and the tone recognition result, and the enhancement method of the second text includes the first enhancement method; performing text enhancement processing on the first text according to the audio feature and the enhancement method of the second text to obtain the second text includes: Determining, according to the pause feature information and the tone recognition result, a punctuation type of a first punctuation mark to be inserted into the first text and an insertion position of the first punctuation mark in the first text; The first punctuation mark is inserted into the first text according to the punctuation mark type and the insertion position to obtain the second text.

4. The method according to claim 2, characterized in that: The audio feature includes the stress feature information and the tone recognition result, the enhancement method of the second text includes the second enhancement method; performing text enhancement processing on the first text according to the audio feature, the audio feature, and the enhancement method of the second text to obtain the second text includes: Determining, according to the stress feature information and the tone recognition result, a function word type of a first function word to be inserted into the first text and an insertion position of the first function word in the first text; The first function word is inserted into the first text according to the function word type and the insertion position to obtain the second text.

5. The method according to claim 1, characterized in that The audio feature of the speech data includes a tone recognition result of the speech data; the feature extraction process on the speech data to obtain the audio feature of the speech data includes: Converting the speech data to obtain a spectrogram of the speech data; the spectrogram is used to represent pitch information of the speech data; Segmenting the spectrogram to obtain a plurality of sub-spectrograms arranged in chronological order; Determining pitch change information of the speech data in two adjacent sub-spectrograms according to the multiple sub-spectrograms; If the pitch change information indicates that the pitch information of the speech data increases, determining that the pitch comparison results of two adjacent sub-spectrogens are the first preset results; If the pitch change information indicates that the pitch information of the speech data remains unchanged, determining that the pitch comparison results of two adjacent sub-spectrogens are the second preset results; If the pitch change information indicates that the pitch information of the speech data decreases, determining that the pitch comparison results of two adjacent sub-spectrogens are the third preset results; According to the pitch comparison results of two adjacent sub-spectrograms, a query is performed in the pre-configured correspondence between the pitch comparison results and the tone to obtain the tone recognition result of the speech data.

6. The method according to claim 5, characterized in that The step of determining pitch change information of the speech data in two adjacent sub-spectroglas according to the plurality of sub-spectroglas comprises: Determining the average pitch of each sub-spectrogram according to the pitch information included in the sub-spectrogram; Among the plurality of sub-spectroglas, calculating a first difference in average pitch between two adjacent sub-spectroglas; If the first difference is less than a preset value, determining that the pitch change information represents an increase in pitch information of the voice data; If the first difference is equal to a preset value, determining that the pitch information representing the pitch information of the speech data remains unchanged; If the first difference is greater than a preset value, it is determined that the pitch change information represents a decrease in pitch information of the speech data.

7. The method according to claim 1, characterized in that After performing text enhancement processing on the first text according to the audio feature and the enhancement mode of the second text to obtain the second text, the method further includes: If the number of the second texts is greater than a preset number threshold, each second text is evaluated to obtain an evaluation score of each second text; According to the evaluation score of each second text, a preset number of second texts are selected as target texts.

8. A text enhancement device, characterized in that: include: An extraction unit, used to perform feature extraction processing on the speech data to obtain audio features of the speech data; a determining unit, configured to determine a type of the audio feature according to the audio feature of the speech data; The determining unit is further used to determine a text enhancement method of a second text matching the first text of the speech data according to the type of the audio feature; The enhancement unit is used to perform text enhancement processing on the first text according to the audio feature and the enhancement method of the second text to obtain the second text.

9. An electronic device, characterized in that: include: processor; And, a memory configured to store computer executable instructions, which, when executed, cause the processor to perform the text enhancement method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store computer-executable instructions, and when the computer-executable instructions are executed by a processor, the text enhancement method according to any one of claims 1 to 7 is implemented.

11. A computer program product, characterized in that The invention comprises a computer program, which, when executed by a processor, implements the text enhancement method according to any one of claims 1 to 7.

Images