Paralanguage annotation method and device, equipment, storage medium and program product

Abstract

The invention discloses a sublingual annotation method and device, equipment, a storage medium and a program product, and relates to the technical field of voice processing, and the method comprises the steps: carrying out the sublingual detection of target voice data, and obtaining an initial detection result, the initial detection result comprises whether each voice frame of the target voice data belongs to a sublingual or not, the sub-language category belongs to the sub-language; correcting the initial detection result according to the length of a sublanguage fragment formed by continuous voice frames belonging to the same sublanguage category and the interval between the sublanguage fragments to obtain a target detection result; and performing sublingual labeling on a text corresponding to the target voice data based on the target detection result. On the basis of the method and the device, the purpose of automatically performing sublingual annotation on the text is achieved, the sublingual annotation efficiency is improved, and the sublingual annotation cost is reduced.

Description

Technical Field

[0001] The present application relates to the technical field of audio processing, and in particular to a paralanguage annotation method, apparatus, device, storage medium and program product. Background Art

[0002] Paralanguage refers to non-verbal elements other than spoken language in language communication, such as inhalation and laughter, which help convey information or enhance the effectiveness of communication. In the field of speech synthesis, paralanguage, as a non-verbal feature, provides rich context and emotional information. This information plays an important role in the comprehensive understanding and processing of speech signals, and can improve the naturalness and fluency of synthesized speech.

[0003] At present, the speech synthesis model is trained with text as training samples and the speech corresponding to the text as labels. In order to improve the speech synthesis effect, the text used as training samples will be annotated with paralinguistic tags, that is, paralinguistic tags are added to certain positions in the text. However, the current paralinguistic tags are added to the text manually after listening to the speech corresponding to the text, and the annotation efficiency is low.

[0004] Therefore, how to improve the efficiency of paralanguage annotation has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] In view of the above problems, the present application provides a paralanguage annotation method, device, equipment, storage medium and program product to improve the efficiency of paralanguage annotation. The specific scheme is as follows:

[0006] A first aspect of the present application provides a paralanguage annotation method, the method comprising:

[0007] Performing paralanguage detection on the target speech data to obtain an initial detection result; the initial detection result includes: whether each speech frame of the target speech data belongs to a paralanguage, and the paralanguage category when it belongs to a paralanguage;

[0008] According to the length of the paralanguage segment and the interval between the paralanguage segments, the initial detection result is corrected to obtain a target detection result; wherein continuous speech frames belonging to the same paralanguage category constitute a paralanguage segment;

[0009] The text corresponding to the target speech data is annotated with a paralinguistic tag based on the target detection result.

[0010] In a possible implementation, performing paralanguage detection on the target speech data includes:

[0011] Encoding each speech frame of the target speech data to obtain a coding feature of each speech frame;

[0012] Each speech frame is classified based on the coding features of each speech frame to obtain the initial detection result.

[0013] In a possible implementation, classifying each speech frame based on the coding feature of each speech frame to obtain the initial detection result includes:

[0014] Based on the coding features of each speech frame, each speech frame is classified and predicted to obtain a probability distribution of each speech frame; the probability distribution of each speech frame is the probability of the speech frame corresponding to each category; the categories include: non-paralinguistic category and multiple paralinguistic categories;

[0015] The initial category of each speech frame is determined in turn according to the probability distribution of each speech frame; wherein:

[0016] If the maximum probability in the probability distribution of the first speech frame is greater than a first threshold, determining the initial category of the first speech frame to be the category corresponding to the maximum probability, otherwise determining the initial category of the first speech frame to be a non-paralanguage category;

[0017] For each non-first speech frame, the initial category of the non-first speech frame is determined according to the initial category of the previous speech frame of the non-first speech frame and the probability distribution of the non-first speech frame.

[0018] In a possible implementation, the process of determining the initial category of a non-first speech frame includes:

[0019] If the initial category of the previous speech frame of the non-first speech frame is the non-paralanguage category or the previous speech frame is the last frame of any paralanguage category, and the maximum probability in the probability distribution of the non-first speech frame is greater than a first threshold, the initial category of the non-first speech frame is determined to be the category corresponding to the maximum probability, otherwise, the initial category of the non-first speech frame is determined to be the non-paralanguage category;

[0020] If the initial category of the previous speech frame of the non-first speech frame is any paralinguistic category, the previous speech frame is not the last frame of any paralinguistic category, and the probability corresponding to any paralinguistic category in the probability distribution of the non-first speech frame is greater than a second threshold, it is determined that the initial category of the non-first speech frame is any paralinguistic category and is not the last frame of any paralinguistic category; if the probability corresponding to any paralinguistic category in the probability distribution of the non-first speech frame is less than or equal to the second threshold, it is determined that the initial category of the non-first speech frame is any paralinguistic category and is the last frame of any paralinguistic category; the second threshold is greater than the first threshold.

[0021] In a possible implementation, the modifying of the initial detection result according to the length of the paralanguage segment and the interval between the paralanguage segments includes:

[0022] If the speech frames in two paralanguage segments belong to the same paralanguage category, and the number of speech frames between the two paralanguage segments is less than a first preset number, the category to which the speech frames between the two paralanguage segments belong is updated to the paralanguage category to which the speech frames in the two paralanguage segments belong, and the speech frames between the two paralanguage segments and the two paralanguage segments are merged into one paralanguage segment.

[0023] In a possible implementation, the paralanguage category includes a plurality of determined paralanguage categories and uncertain paralanguage categories; and the modifying the initial detection result according to the length of the paralanguage segment and the interval between the paralanguage segments includes at least one of the following:

[0024] If the speech frames in two paralanguage segments belong to the same paralanguage category, and the number of speech frames between the two paralanguage segments is less than a first preset number, updating the category to which the speech frames between the two paralanguage segments belong to be the paralanguage category to which the speech frames in the two paralanguage segments belong, and merging the speech frames between the two paralanguage segments and the two paralanguage segments into one paralanguage segment;

[0025] If the speech frames in the first paralanguage segment of two paralanguage segments belong to a determined first paralanguage category, and the speech frames in the second paralanguage segment belong to an uncertain paralanguage category, the number of speech frames included in the second paralanguage segment is less than a second preset number, and the number of speech frames between the two paralanguage segments is less than the first preset number, then the categories to which the speech frames in the second paralanguage segment and the speech frames between the two paralanguage segments belong are updated to the first paralanguage category, and the speech frames between the two paralanguage segments and the two paralanguage segments are merged into one paralanguage segment.

[0026] In a possible implementation, the method further includes:

[0027] The paralanguage segment including a number of speech frames less than a third preset number is modified into a non-paralanguage segment.

[0028] In a possible implementation, the performing paralinguistic tagging on the text corresponding to the target speech data based on the target detection result includes:

[0029] Corresponding to any para-language segment, according to the target temporal relationship between each speech frame in the any para-language segment and the speech frames corresponding to each word in the text, a para-language tag corresponding to the any para-language segment is added to the text;

[0030] The paralanguage tag represents the paralanguage category to which any paralanguage segment belongs and its start time and end time; the temporal relationship between the paralanguage tag and each word in the text is the target temporal relationship.

[0031] A second aspect of the present application provides a paralanguage annotation device, comprising:

[0032] A detection module is used to perform paralanguage detection on the target speech data to obtain an initial detection result; the initial detection result includes: whether each speech frame of the target speech data belongs to a paralanguage, and the paralanguage category when it belongs to a paralanguage;

[0033] A correction module is used to correct the initial detection result according to the length of the paralanguage segment and the interval between the paralanguage segments to obtain a target detection result; wherein continuous speech frames belonging to the same paralanguage category constitute a paralanguage segment;

[0034] The tagging module is used to perform paralinguistic tagging on the text corresponding to the target speech data based on the target detection result.

[0035] A third aspect of the present application provides a computer program product, including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements the paralanguage annotation method of the first aspect or any implementation of the first aspect.

[0036] A fourth aspect of the present application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:

[0037] The memory is used to store computer programs;

[0038] The processor is used to execute the computer program, so that the electronic device can implement the paralanguage annotation method of the first aspect or any implementation manner of the first aspect.

[0039] A fifth aspect of the present application provides a computer storage medium, wherein the storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can use the paralanguage annotation method of the first aspect or any implementation of the first aspect.

[0040] By means of the above technical solution, the paralanguage annotation method, device, equipment, storage medium and program product provided by the present application perform paralanguage detection on the target speech data to obtain an initial detection result, which includes: whether each speech frame of the target speech data belongs to a paralanguage, and the paralanguage category when it belongs to a paralanguage; according to the length of the paralanguage segment composed of continuous speech frames belonging to the same paralanguage category, and the interval between the paralanguage segments, the initial detection result is corrected to obtain the target detection result; based on the target detection result, the text corresponding to the target speech data is annotated with the paralanguage. Based on the present application, the purpose of automatically annotating the text with the paralanguage is achieved, the efficiency of the paralanguage annotation is improved, and the cost of the paralanguage annotation is reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0042] Figure 1 A flowchart of an implementation of the paralanguage annotation method provided in this application;

[0043] Figure 2 An example diagram of the occurrence of paralanguage between utterances provided for this application;

[0044] Figure 3 A flowchart for implementing paralanguage detection of target speech data provided by the present application;

[0045] Figure 4 A structural diagram of a paralanguage recognition model provided in this application;

[0046] Figure 5 A flowchart for implementing the method of classifying each speech frame based on the coding features of each speech frame to obtain an initial detection result provided by the present application;

[0047] Figure 6 A structural schematic diagram of a paralanguage annotation device provided in this application;

[0048] Figure 7 A schematic diagram of the structure of an electronic device provided in this application. DETAILED DESCRIPTION

[0049] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0050] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0051] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0052] In order to improve the efficiency of paralanguage annotation, this application scheme is proposed.

[0053] like Figure 1 As shown, a flowchart of an implementation of the paralanguage annotation method provided in an embodiment of the present application may include:

[0054] Step S101: perform paralanguage detection on target speech data to obtain an initial detection result, which includes whether each speech frame of the target speech data belongs to a paralanguage and the paralanguage category when it belongs to a paralanguage.

[0055] The present application performs frame-level paralanguage detection on target speech data, that is, detects whether each speech frame of the target speech data is a paralanguage, and if so, which paralanguage it is.

[0056] In natural speech conversations, common paralinguistic information includes inhalation, lip smacking, laughter, dragging, pauses, stuttering, repetition, etc. An example of the appearance of paralinguistic information between discourses is shown in the figure below: Figure 2 As shown, it is often shorter information, such as the inhalation sound, which usually occurs tens of milliseconds before the speech.

[0057] Step S102: According to the length of the paralanguage segment and the interval between the paralanguage segments, the initial detection result is corrected to obtain the target detection result. Among them, continuous speech frames belonging to the same paralanguage category constitute a paralanguage segment.

[0058] In order to avoid the negative impact of false detection on paralanguage labeling, the present application regards continuous speech frames belonging to the same paralanguage category as a paralanguage segment, and then corrects the initial detection result according to the length of each paralanguage segment and the interval between the paralanguage segments to obtain the target detection result.

[0059] The method of correcting the initial detection result may include but is not limited to at least one of the following: merging paralanguage segments with smaller intervals (for example, intervals smaller than a certain threshold) and containing speech frames belonging to the same paralanguage category, correcting paralanguage segments with smaller lengths, etc.

[0060] This application study found that speech frames are usually 20ms, but in reality, at least 0.2 second segments can be perceived by humans, so the frame-level results need to be merged into segments. By merging paralanguage segments with smaller intervals and containing speech frames belonging to the same paralanguage category, correcting paralanguage segments with smaller lengths, etc., it is ensured that the detected paralanguage segments are all paralanguage segments that can be perceived by humans.

[0061] Step S103: performing paralinguistic annotation on the text corresponding to the target speech data based on the target detection result.

[0062] The text corresponding to the target voice data may be text obtained by performing voice recognition on the target voice data.

[0063] Alternatively, the target voice data is voice data obtained by a reader reading the text.

[0064] Optionally, the correspondence between each word in the text and the speech frames in the target speech data may be preset, that is, each word in the text may be marked in advance to which speech frames in the target speech data each word corresponds.

[0065] As an example, the target speech data and the corresponding text can be any text sample in a data set used to train a speech synthesis model, and the speech tag of the any text, where the speech tag is a speech frame of the target speech data annotated with a single word in the text. Based on this, the correspondence between the single word and the speech frame in the present application can be easily obtained.

[0066] Corresponding to any para-language segment, the para-language label corresponding to the any para-language segment can be added to the text according to the temporal relationship between each speech frame in the any para-language segment and the speech frames corresponding to each single word in the text, thereby realizing a para-language annotation of the text.

[0067] The paralanguage annotation method provided in the embodiment of the present application performs paralanguage detection on the target speech data to obtain an initial detection result, which includes: whether each speech frame of the target speech data belongs to a paralanguage, and the paralanguage category when it belongs to a paralanguage; the initial detection result is corrected according to the length of the paralanguage segment composed of consecutive speech frames belonging to the same paralanguage category, and the interval between the paralanguage segments to obtain the target detection result; based on the target detection result, the text corresponding to the target speech data is annotated with a paralanguage. Based on this application, the purpose of automatically annotating text with a paralanguage is achieved, the efficiency of paralanguage annotation is improved, and the cost of paralanguage annotation is reduced.

[0068] In an optional embodiment, a flowchart of implementing the above-mentioned paralanguage detection on the target speech data is as follows: Figure 3 As shown, it may include:

[0069] Step S301: Encode each speech frame of the target speech data to obtain encoding features of each speech frame.

[0070] Optionally, the audio features of each speech frame can be extracted first, and the audio features of each speech frame can be input into a trained encoding module (referred to as the first encoding module for ease of description and distinction). The first encoding module uses the audio features of each speech frame to encode each speech frame to obtain the encoding features of each speech frame. Common audio features include spectral features, cepstral features, Mel-frequency cepstral coefficients (MFCC), and FilterBank features, which can reflect the spectral characteristics, energy distribution, and pitch information of speech signals. As an example, the first encoding module can be trained based on the encoding module in the pre-trained Whisper model.

[0071] Optionally, the target speech data can be directly input into a trained encoding module (referred to as the second encoding module for ease of description and distinction), and features are extracted from each speech frame and encoded by the second encoding module to obtain encoding features of each speech frame. As an example, the second encoding module can be trained based on an encoding module in a pre-trained Hubert_Base_CN model, or the second encoding module can be trained based on a pre-trained Wav2vec.

[0072] Step S302: Classify each speech frame based on the coding features of each speech frame to obtain an initial detection result.

[0073] Optionally, each speech frame can be classified and predicted based on the coding features of each speech frame to obtain the probability distribution of each speech frame, and the initial detection result can be determined based on the probability distribution of each speech frame. The probability distribution of each speech frame is the probability of the speech frame corresponding to each category. The categories here include: non-paralanguage category and multiple paralanguage categories.

[0074] For example, each category includes 5 categories: inhalation sound (denoted as NXQ), lip smacking sound (denoted as NZZ), laughter (denoted as NXS, where laughter refers to laughter between words and continuous laughter), dragging sound (denoted as TY, referring to the elongation of the sound) and non-paralinguistic information (denoted as OTHER, such as silence, human voice, etc.). Based on this, the probability distribution of each speech frame is the probability that the speech frame corresponds to each of the above 5 categories. It should be noted that the 5 categories here are only examples and do not constitute a limitation of this application. In actual applications, there may be more categories, for example, other paralinguistic information (denoted as NN, which refers to other paralinguistic sounds that cannot be classified, used to absorb probability), jamming (the phenomenon that the speaker's brain goes blank and falls into silence or repeatedly corrects grammatical errors), repetition, etc.

[0075] In an optional embodiment, encoding each speech frame of the target speech data and classifying and predicting each speech frame based on the encoding features of each speech frame can be implemented by a paralanguage recognition model. Figure 4 As shown, a structural diagram of a paralanguage recognition model provided in an embodiment of the present application may include:

[0076] Coding module 401 and classification module 402;

[0077] The encoding module 401 is used to encode the input target speech data or the audio features of each speech frame of the target speech data to obtain the encoding features of each speech frame of the target speech data.

[0078] That is, the input of the encoding module 401 is the target speech data, or the input of the encoding module 401 is the audio features of each speech frame of the target speech data.

[0079] The classification module 402 is used to perform classification prediction on each speech frame based on the coding features of each speech frame to obtain the probability distribution of each speech frame.

[0080] The paralanguage recognition model can be trained using speech data as training samples and using the categories annotated for each speech frame as labels. The category annotated for any speech frame indicates which category the speech frame belongs to. Taking six categories (inhalation, lip smacking, laughter, dragging sound, other paralanguage information and non-paralanguage information) as an example, the category label of a certain speech frame is 100000, indicating that the certain speech frame belongs to paralanguage and is an inhalation sound; the category label of a certain speech frame is 010000, indicating that the certain speech frame belongs to paralanguage and is a lip smacking sound; the category label of a certain speech frame is 001000, indicating that the certain speech frame belongs to paralanguage and is a laughter; the category label of a certain speech frame is 000100, indicating that the certain speech frame belongs to paralanguage and is a dragging sound; the category label of a certain speech frame is 000010, indicating that the certain speech frame belongs to paralanguage and is other paralanguage information; the category label of a certain speech frame is (000001), indicating that the certain speech frame does not belong to paralanguage, that is, the certain speech frame belongs to non-paralanguage information.

[0081] The initial parameters of the encoding module 401 are the parameters of the pre-trained encoding module. The pre-trained encoding module may be the encoding module of a pre-trained speech recognition model (eg, HuBERT, Whisper, etc.). By comparison, the encoding module 401 has the best effect when it is trained based on the encoding module in the pre-trained Hubert_Base_CN model.

[0082] The classification module 402 can be a linear classification layer or a non-linear classification layer. The linear classification layer can perform a linear transformation on the coded features of each received speech frame to map to the category space, and then convert it into a probability distribution through softmax. The non-linear classification layer can perform a non-linear transformation on the coded features of each received speech frame to map to the category space, and then convert it into a probability distribution through softmax.

[0083] The classification module 402 may adopt a simple fully connected network or a more complex structure, such as a convolutional network CNN, a long short-term memory network LSTM, a network based on an attention mechanism Transformer, etc. If the amount of training data is relatively small, a simple fully connected network may be adopted, and if the amount of training data is relatively large, a more complex structure may be adopted.

[0084] The training process may include two stages of training. In the first stage of training, the parameters of the encoding module 401 are frozen, and only the parameters of the classification module 402 are updated. After the classification module 402 training is stable, the second stage of training begins. In the second stage of training, the parameters of the encoding module 401 and the classification module 402 are updated simultaneously.

[0085] As an example, in the first stage of training, the audio features of the training sample or each speech frame of the training sample can be input into the encoding module 401 to obtain the encoding features of each speech frame of the training sample; the classification module 402 classifies and predicts the encoding features of each speech frame of the training sample to obtain the probability distribution of each speech frame of the training sample. The parameters of the classification module 402 are updated with the goal that the category represented by the probability distribution of each speech frame is close to the label corresponding to the speech frame. Optionally, the probability distribution of the speech frame can be aligned to the corresponding label using the sample equalization loss (Focal Loss), that is, the Focal Loss loss is calculated according to the label corresponding to the speech frame, and the parameters of the classification module 402 are updated with the goal of reducing the loss. The change of each round of loss relative to the previous round of loss can be monitored. When the change of the loss is less than the threshold for a certain number of consecutive times, it is determined that the first stage of training is over and the second stage of training can be entered.

[0086] In the second stage of training, the audio features of the training sample or each speech frame of the training sample can be input into the encoding module 401 to obtain the encoding features of each speech frame of the training sample; the classification module 402 based on the first stage of training performs classification prediction on the encoding features of each speech frame of the training sample to obtain the probability distribution of each speech frame of the training sample. With the goal of the category represented by the probability distribution of each speech frame approaching the label corresponding to the speech frame, the parameters of the encoding module 401 and the classification module 402 are updated. Optionally, the probability distribution of the speech frame can be aligned to the corresponding label using the sample equalization loss (Focal Loss), that is, the Focal Loss loss is calculated according to the label corresponding to the speech frame, and the parameters of the encoding module 401 and the classification module 402 are updated with the goal of reducing the loss. The change of each round of loss relative to the previous round of loss can be monitored, and when the change of the loss is less than the threshold for a certain number of consecutive times, it is determined that the second stage of training is over.

[0087] The first stage of training and the second stage of training use the same training dataset.

[0088] When the classification module 402 adopts a simple fully connected network, the classification module 402 can achieve the effect of a complex network through the second stage of training.

[0089] When the classification module 402 adopts a relatively complex network structure, the classification module 402 can achieve the effect of a more complex network through the second stage of training.

[0090] In an optional embodiment, the above-mentioned classification of each speech frame based on the coding features of each speech frame to obtain an implementation flow chart of the initial detection result is as follows: Figure 5 As shown, it may include:

[0091] Step S501: Classify and predict each speech frame based on the coding features of each speech frame to obtain the probability distribution of each speech frame. The probability distribution of each speech frame is the probability of the speech frame corresponding to each category. Each category includes: non-paralinguistic category and multiple paralinguistic categories.

[0092] The specific implementation method can be found in the above-mentioned embodiment, which will not be described again here.

[0093] Step S502: determining the initial category of each speech frame in turn according to the probability distribution of each speech frame.

[0094] If the maximum probability in the probability distribution of the first speech frame is greater than a first threshold, the initial category of the first speech frame is determined to be the category corresponding to the maximum probability; otherwise, the initial category of the first speech frame is determined to be a non-paralanguage category.

[0095] Optionally, the first threshold may be 0.5, or other values, such as 0.48, 0.51, etc. The effect when the first threshold is 0.5 is better than the effect when the first threshold is other values.

[0096] The research of this application found that paralanguage information is all information between the speaker's words, and there will not be two paralanguage triggers at the same time. The first threshold is set to 0.5 to ensure that only one category can be activated at the same time in each frame. Moreover, in the absence of paralanguage, if the probability of a certain category rises above 0.5, it can be considered that the paralanguage is triggered, and the falsely triggered part can be deleted through the subsequent correction process, so the first threshold can be set lower, such as 0.5.

[0097] For each non-first speech frame, the initial category of the non-first speech frame is determined according to the initial category of the previous speech frame of the non-first speech frame and the probability distribution of the non-first speech frame.

[0098] When determining the initial category of each speech frame, the present application not only considers the probability that the speech frame belongs to each category, but also considers the initial category to which the previous speech frame of the speech frame belongs.

[0099] The manner of determining the initial category of each non-first speech frame is the same. In an optional embodiment, one implementation manner of determining the initial category of the non-first speech frame may be:

[0100] If the initial category of the previous speech frame of the non-first speech frame is the non-paralanguage category or the previous speech frame is the last frame of any paralanguage category, and the maximum probability in the probability distribution of the non-first speech frame is greater than the first threshold, then the initial category of the non-first speech frame is determined to be the category corresponding to the maximum probability; otherwise, the initial category of the non-first speech frame is determined to be the non-paralanguage category.

[0101] If the initial category of the previous speech frame of the non-first speech frame is any sub-language category, the previous speech frame is not the last frame of the any sub-language category, and the probability corresponding to the any sub-language category in the probability distribution of the non-first speech frame is greater than the second threshold, then it is determined that the initial category of the non-first speech frame is the any sub-language category, but not the last frame of the any sub-language category; if the probability corresponding to the any sub-language category in the probability distribution of the non-first speech frame is less than or equal to the second threshold, it is determined that the initial category of the non-first speech frame is the any sub-language category and is the last frame of the any sub-language category.

[0102] The second threshold is greater than the first threshold. Optionally, the second threshold may be 0.95, or other values, such as 0.9, 0.97, etc. The present application has found that in the middle of the paralanguage segment, the probability will be at a maximum value of 0.98 to 0.99, and the probability will be exited when it is less than 0.95, which can make the exit of the paralanguage segment faster, and the misjudged frames in the middle can be absorbed in the subsequent correction, making the result more accurate.

[0103] For example, assuming that the initial category of the n-1th (n is an integer greater than 1) speech frame is an inhalation sound, and it is not the last frame of the inhalation sound, then when determining the initial category of the nth speech frame, the probability corresponding to the inhalation sound in the probability distribution of the nth speech frame (the maximum probability may or may not correspond to the inhalation sound) is looked at. If the probability corresponding to the inhalation sound is greater than 0.95, it is considered that the initial category of the nth speech frame is an inhalation sound and it has not ended yet, that is, the nth speech frame is not the last frame of the inhalation sound. If the probability corresponding to the inhalation sound is less than or equal to 0.95, it is considered that the initial category of the nth speech frame is an inhalation sound, and the nth speech frame is the last frame of the inhalation sound.

[0104] When determining the initial category of the n+1th speech frame, if the initial category of the nth speech frame is an inhalation sound and it is not the last frame of the inhalation sound, then the probability corresponding to the inhalation sound in the probability distribution of the n+1th speech frame (the maximum probability may correspond to the inhalation sound or may not correspond to the inhalation sound) is looked at. If the probability corresponding to the inhalation sound is greater than 0.95, then it is considered that the initial category of the n+1th speech frame is an inhalation sound and it is not the last frame of the inhalation sound. If the probability corresponding to the inhalation sound is less than or equal to 0.95, then it is considered that the initial category of the n+1th speech frame is an inhalation sound and the n+1th speech frame is the last frame of the inhalation sound.

[0105] When judging the initial category of the n+1th speech frame, if the initial category of the nth speech frame is an inhalation sound and it is the last frame of the inhalation sound, then look at the maximum probability in the probability distribution of the n+1th speech frame. If the maximum probability is greater than 0.5, the n+1th frame is considered to be the category corresponding to the maximum probability, otherwise the n+1th frame is considered not to be paraspeech.

[0106] In an optional embodiment, an implementation method of correcting the initial detection result according to the length of the paralanguage segment and the interval between the paralanguage segments may be:

[0107] If the speech frames in the two paralanguage segments belong to the same paralanguage category, and the number of speech frames between the two paralanguage segments is less than a first preset number, the category to which the speech frames between the two paralanguage segments belong is updated to the paralanguage category to which the speech frames in the two paralanguage segments belong, and the speech frames between the two paralanguage segments and the two paralanguage segments are merged into one paralanguage segment.

[0108] The two paralanguage segments here can be two adjacent paralanguage segments (i.e., there is no paralanguage segment between the two paralanguage segments), or they can be two non-adjacent paralanguage segments (i.e., there is at least one paralanguage segment between the two paralanguage segments).

[0109] The speech frames may be traversed in the order of the speech frames, that is, from the front to the back, and the similar paralanguage segments may be judged in turn. If the speech frames in the two paralanguage segments belong to the same paralanguage category, and the number of speech frames between the two paralanguage segments is less than a first preset number, the category to which the speech frames between the two paralanguage segments belong is updated to the paralanguage category to which the speech frames in the two paralanguage segments belong, and the speech frames between the two paralanguage segments and the two paralanguage segments are merged into one paralanguage segment. Otherwise, no merging process is performed, that is, the relative position relationship of the two paralanguage segments is kept unchanged.

[0110] Optionally, the first preset number may be 5, or other values, such as 4, 6 or 7.

[0111] As an example, assume that the first to sixth speech frames do not belong to paralanguage, the 10 speech frames from the seventh to the 16th speech frames (recorded as the first paralanguage segment) all belong to inhalation sounds, the 17th speech frame does not belong to inhalation sounds, and the 7 speech frames from the 18th to the 24th speech frames (recorded as the second paralanguage segment) all belong to inhalation sounds. Since the first and second paralanguage segments are both inhalation sound segments, there is only one speech frame between the first and second paralanguage segments, which is less than 5. Therefore, the first paralanguage segment, the 17th speech frame and the second paralanguage segment are merged into a new inhalation sound segment, which is recorded as the third paralanguage segment. That is, the third paralanguage segment includes 18 speech frames from the seventh to the 24th speech frame, and these 18 speech frames all belong to inhalation sounds. In other words, the category of the 17th speech frame is corrected to inhalation sound.

[0112] Further, assuming that none of the 25th to 27th speech frames are inhalation sounds, and the 28th to 30th speech frames (recorded as the fourth paralanguage segment) are inhalation sounds, then since the third and fourth paralanguage segments are both inhalation sound segments, and there are only three speech frames between the two paralanguage segments, which is less than 5, the third paralanguage segment, the 25th to 27th speech frames, and the fourth paralanguage segment are merged into a new inhalation sound segment, recorded as the fifth paralanguage segment, that is, the fifth paralanguage segment includes 24 speech frames from the 7th to the 30th speech frame, and these 24 speech frames are all inhalation sounds.

[0113] Assuming that none of the 25th to 27th speech frames belong to paralanguage, and all of the 28th to 30th speech frames (recorded as the fourth paralanguage segment) belong to lip smacking sounds, since the fourth paralanguage segment and the third paralanguage segment belong to different categories of paralanguage segments, it is not determined whether the third paralanguage segment needs to be merged with the fourth paralanguage segment and the speech frames in between. It is directly determined whether there is a paralanguage segment containing lip smacking sounds after the fourth paralanguage segment that can be merged with the fourth paralanguage segment, and so on, until all paralanguage segments are traversed.

[0114] Furthermore, after the merging process is completed, the paralanguage segment including a number of speech frames less than a third preset number may be corrected to a non-paralanguage segment.

[0115] As an example, the third preset number may be 10, that is, if the number of speech frames in a paralanguage segment is less than 10, the paralanguage segment is considered to be a misjudgment, and the category of each speech frame in the paralanguage segment is modified to non-paralanguage. The third preset number may also be other values, such as 8, 9, 11, 12, etc.

[0116] In an optional embodiment, the paralanguage categories include multiple determined paralanguage categories (e.g., inhalation, lip smacking, laughter, and dragging sounds) and uncertain paralanguage categories (e.g., other paralanguage information; that is, paralanguage, but not any of inhalation, lip smacking, laughter, and dragging sounds). Based on this, another implementation method of correcting the initial detection result according to the length of the paralanguage segment and the interval of the paralanguage segment can be:

[0117] If the speech frames in the two paralanguage segments belong to the same paralanguage category, and the number of speech frames between the two paralanguage segments is less than a first preset number, the category to which the speech frames between the two paralanguage segments belong is updated to the paralanguage category to which the speech frames in the two paralanguage segments belong, and the speech frames between the two paralanguage segments and the two paralanguage segments are merged into one paralanguage segment.

[0118] The specific implementation process can be found in the aforementioned embodiment, which will not be described again here.

[0119] If the speech frames in the first paralanguage segment of the two paralanguage segments belong to the determined first paralanguage category, the speech frames in the second paralanguage segment belong to the uncertain paralanguage category, the number of speech frames included in the second paralanguage segment is less than the second preset number, and the number of speech frames between the two paralanguage segments is less than the first preset number, then the categories to which the speech frames in the second paralanguage segment and the speech frames between the two paralanguage segments belong are updated to the first paralanguage category, and the speech frames between the two paralanguage segments and the two paralanguage segments are merged into one paralanguage segment.

[0120] Optionally, the second preset number may be 10, or other values, such as 8, 9, 11, 12, etc.

[0121] The two paralanguage segments here can be two adjacent paralanguage segments (i.e., there is no paralanguage segment between the two paralanguage segments), or they can be two non-adjacent paralanguage segments (i.e., there is at least one paralanguage segment between the two paralanguage segments). As an example, it is assumed that none of the first to sixth speech frames belong to paralanguage, and the ten speech frames from the seventh to the sixteenth speech frames (recorded as the first paralanguage segment) belong to inhalation sounds, the seventeenth speech frame does not belong to inhalation sounds nor to other paralanguage information, and the seven speech frames from the eighteenth to the twenty-fourth speech frames (recorded as the second paralanguage segment) belong to other paralanguage information. Since the first paralanguage segment and the second paralanguage segment are both paralanguage segments, there is only one speech frame between the first paralanguage segment and the second paralanguage segment, which is less than 5, and the number of speech frames included in the second paralanguage segment is less than 10. Therefore, the first paralanguage segment, the seventeenth speech frame, and the second paralanguage segment are merged into a new inhalation sound segment, which is recorded as the third paralanguage segment. That is, the third paralanguage segment includes 18 speech frames from the seventh to the twenty-fourth speech frame, and these 18 speech frames all belong to inhalation sounds. That is, the categories of the 17th speech frame and the speech frames in the second paralanguage segment are corrected to inhalation sounds.

[0122] Each speech frame can be traversed in the order of the speech frames, that is, from the front to the back, and similar paralanguage segments can be judged and processed in turn based on the above method until all paralanguage segments are traversed.

[0123] Furthermore, after the merging process is completed, the paralanguage segment including a number of speech frames less than a third preset number may be corrected to a non-paralanguage segment. The specific implementation method may refer to the above embodiment, which will not be described in detail here.

[0124] In an optional embodiment, an implementation method of performing paralanguage annotation on the text corresponding to the target speech data based on the target detection result may be:

[0125] Corresponding to any para-language segment, according to the temporal relationship between each speech frame in the para-language segment and the speech frames corresponding to each word in the text (for the convenience of description and distinction, recorded as the target temporal relationship), the para-language tag corresponding to the para-language segment is added to the text;

[0126] The paralanguage tag represents the paralanguage category to which any paralanguage segment belongs and its start time and end time; the temporal relationship between the paralanguage tag and each word in the text is the target temporal relationship.

[0127] The start time refers to the start time of the first speech frame in any one of the language segments, and the end time refers to the end time of the last speech frame in any one of the language segments.

[0128] As an example, suppose a paralanguage segment includes speech frames from the 20th to the 29th, and the 10 speech frames are all lip smacking sounds. The speech frames in the text that are adjacent to the 10 speech frames in time sequence correspond to two adjacent characters, wherein the first character corresponds to the 10th to the 19th speech frames, and the second character corresponds to the 30th to the 42nd speech frames. Obviously, the speech frames in the paralanguage segment are located between the speech frames corresponding to the two adjacent characters, so the paralanguage label corresponding to the paralanguage segment is added between the two adjacent characters.

[0129] The following is an evaluation of the method of this application. Generally speaking, the accuracy of paralanguage is judged at the frame level, that is, the judgment of each frame in the audio is correct. In fact, we found that such judgment is not appropriate, because for example, there is a paralanguage from 0.15 seconds to 0.45 seconds, and our output result is 0.14 seconds to 0.50 seconds, which is completely correct in use, but there will be an error of 0.06 seconds in the evaluation index; and we are more concerned about which paralanguages ​​are correctly / incorrectly recognized, rather than which frames are correctly / incorrectly recognized. Therefore, the evaluation criteria we use are at the interval level, among which there are two situations where the interval is correct:

[0130] Case 1: The difference between the start time and the end time of the two intervals (i.e., the predicted paralanguage interval and the marked paralanguage interval) are both less than the threshold of 0.05 seconds.

[0131] Case 2: The difference between the start time or the end time of the two intervals (i.e. the predicted paralanguage interval and the annotated paralanguage interval) is less than the threshold of 0.05 seconds, and the length of the result interval outside the annotated interval is less than 30% of the annotated length. That is, the difference between the start time of the predicted paralanguage interval and the start time of the annotated paralanguage interval is less than the threshold of 0.05 seconds, and the end time of the predicted paralanguage interval exceeds the end time of the annotated paralanguage interval by no more than 30%, or, the difference between the end time of the predicted paralanguage interval and the end time of the annotated paralanguage interval is less than the threshold of 0.05 seconds, and the start time of the predicted paralanguage interval exceeds the start time of the annotated paralanguage interval by no more than 30%.

[0132] The evaluation indicators are precision and recall, where:

[0133] The accuracy of each category is: Precision = the number of correct intervals in the current class result / the number of intervals in the current class result

[0134] The recall rate of each category is: Recall = the number of intervals hit in the current class annotation / the number of intervals in the current class annotation

[0135] The overall accuracy is: Precision = the number of correct intervals in all results / the number of intervals in all results

[0136] The overall recall rate is: Recall = the number of intervals hit in all annotations / the number of all annotated intervals

[0137] According to the above evaluation criteria, the final evaluation results are:

[0138] Overall: precision = 80.044%, recall = 61.08%;

[0139] Inspiratory NXQ: precision = 85.22%, recall = 69.04%;

[0140] Laughter NXS: precision = 64.71%, recall = 52.89%;

[0141] Lip smacking sound NZZ: precision = 41.74%, recall = 57.83%;

[0142] Drag sound TY: precision=22.14%, recall=51.79%.

[0143] The current interval judgment standard is stricter than the actual situation. In the WenetSpeech test data, 100 audio samples were manually selected for listening. The results showed that all the inhalation NXQ and laughter NXS that can be perceived by the human ear were hit by this application, but there were 2 false positives. The dragging sound TY and the lip smacking sound NZZ were less present in the extracted audio, but the precision was above 80%, which basically met the usage requirements.

[0144] Corresponding to the method embodiment, the present application also provides a paralanguage annotation device. A structural schematic diagram of the paralanguage annotation device provided in the embodiment of the present application is as follows: Figure 6 As shown, it may include:

[0145] Detection module 601, correction module 602 and labeling module 603;

[0146] The detection module 601 is used to perform paralanguage detection on the target speech data to obtain an initial detection result; the initial detection result includes: whether each speech frame of the target speech data belongs to a paralanguage, and the paralanguage category when it belongs to a paralanguage;

[0147] The correction module 602 is used to correct the initial detection result according to the length of the paralanguage segment and the interval between the paralanguage segments to obtain the target detection result; wherein, continuous speech frames belonging to the same paralanguage category constitute a paralanguage segment;

[0148] The annotation module 603 is used to perform paralinguistic annotation on the text corresponding to the target speech data based on the target detection result.

[0149] The paralanguage annotation device provided in the embodiment of the present application performs paralanguage detection on the target speech data to obtain an initial detection result, which includes: whether each speech frame of the target speech data belongs to a paralanguage, and the paralanguage category when it belongs to a paralanguage; the initial detection result is corrected according to the length of the paralanguage segment composed of consecutive speech frames belonging to the same paralanguage category, and the interval between the paralanguage segments to obtain the target detection result; and the text corresponding to the target speech data is annotated with the paralanguage based on the target detection result. Based on the present application, the purpose of automatically annotating text with the paralanguage is achieved, the efficiency of paralanguage annotation is improved, and the cost of paralanguage annotation is reduced.

[0150] In an optional embodiment, when the detection module 601 performs paralanguage detection on the target speech data, it is used to:

[0151] Encoding each speech frame of the target speech data to obtain a coding feature of each speech frame;

[0152] Classifying each speech frame based on the coding features of each speech frame to obtain the initial detection result;

[0153] In an optional embodiment, the detection module 601 classifies each speech frame based on the coding features of each speech frame, and when the initial detection result is obtained, is used to:

[0154] Based on the coding features of each speech frame, each speech frame is classified and predicted to obtain a probability distribution of each speech frame; the probability distribution of each speech frame is the probability of the speech frame corresponding to each category; the categories include: non-paralinguistic category and multiple paralinguistic categories;

[0155] The initial category of each speech frame is determined in turn according to the probability distribution of each speech frame; wherein:

[0156] If the maximum probability in the probability distribution of the first speech frame is greater than a first threshold, determining the initial category of the first speech frame to be the category corresponding to the maximum probability, otherwise determining the initial category of the first speech frame to be a non-paralanguage category;

[0157] For each non-first speech frame, the initial category of the non-first speech frame is determined according to the initial category of the previous speech frame of the non-first speech frame and the probability distribution of the non-first speech frame.

[0158] In an optional embodiment, when the detection module 601 determines the initial category of the non-first speech frame, it is used to:

[0159] If the initial category of the previous speech frame of the non-first speech frame is the non-paralanguage category or the previous speech frame is the last frame of any paralanguage category, and the maximum probability in the probability distribution of the non-first speech frame is greater than a first threshold, the initial category of the non-first speech frame is determined to be the category corresponding to the maximum probability, otherwise, the initial category of the non-first speech frame is determined to be the non-paralanguage category;

[0160] If the initial category of the previous speech frame of the non-first speech frame is any paralinguistic category, the previous speech frame is not the last frame of any paralinguistic category, and the probability corresponding to any paralinguistic category in the probability distribution of the non-first speech frame is greater than a second threshold, it is determined that the initial category of the non-first speech frame is any paralinguistic category and is not the last frame of any paralinguistic category; if the probability corresponding to any paralinguistic category in the probability distribution of the non-first speech frame is less than or equal to the second threshold, it is determined that the initial category of the non-first speech frame is any paralinguistic category and is the last frame of any paralinguistic category; the second threshold is greater than the first threshold.

[0161] In an optional embodiment, when the correction module 602 corrects the initial detection result according to the length of the paralanguage segment and the interval between the paralanguage segments, it is used to:

[0162] If the speech frames in two paralanguage segments belong to the same paralanguage category, and the number of speech frames between the two paralanguage segments is less than a first preset number, the category to which the speech frames between the two paralanguage segments belong is updated to the paralanguage category to which the speech frames in the two paralanguage segments belong, and the speech frames between the two paralanguage segments and the two paralanguage segments are merged into one paralanguage segment.

[0163] In an optional embodiment, the paralanguage category includes a plurality of determined paralanguage categories and uncertain paralanguage categories; the correction module 602 corrects the initial detection result according to the length of the paralanguage segment and the interval between the paralanguage segments, including at least one of the following:

[0164] If the speech frames in two paralanguage segments belong to the same paralanguage category, and the number of speech frames between the two paralanguage segments is less than a first preset number, updating the category to which the speech frames between the two paralanguage segments belong to be the paralanguage category to which the speech frames in the two paralanguage segments belong, and merging the speech frames between the two paralanguage segments and the two paralanguage segments into one paralanguage segment;

[0165] If the speech frames in the first paralanguage segment of two paralanguage segments belong to a determined first paralanguage category, and the speech frames in the second paralanguage segment belong to an uncertain paralanguage category, the number of speech frames included in the second paralanguage segment is less than a second preset number, and the number of speech frames between the two paralanguage segments is less than the first preset number, then the categories to which the speech frames in the second paralanguage segment and the speech frames between the two paralanguage segments belong are updated to the first paralanguage category, and the speech frames between the two paralanguage segments and the two paralanguage segments are merged into one paralanguage segment.

[0166] In an optional embodiment, the correction module 602 is further configured to:

[0167] The paralanguage segment including a number of speech frames less than a third preset number is modified into a non-paralanguage segment.

[0168] In an optional embodiment, when the annotation module 603 performs paralinguistic annotation on the text corresponding to the target speech data based on the target detection result, it is used to:

[0169] Corresponding to any para-language segment, according to the target temporal relationship between each speech frame in the any para-language segment and the speech frames corresponding to each word in the text, a para-language tag corresponding to the any para-language segment is added to the text;

[0170] The paralanguage tag represents the paralanguage category to which any paralanguage segment belongs and its start time and end time; the temporal relationship between the paralanguage tag and each word in the text is the target temporal relationship.

[0171] The present application also provides an electronic device in an embodiment. Figure 7 As shown, it shows a structural schematic diagram of an electronic device suitable for implementing the embodiment of the present application. The electronic device in the embodiment of the present application can be a terminal device (such as a mobile phone, a tablet computer, a laptop computer, a desktop computer, etc.), or a server (can be a single server, can be a server cluster, or can be a cloud server, etc.). Figure 7 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0172] like Figure 7 As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 to a random access memory (RAM) 703. When the electronic device is powered on, various programs and data required for the operation of the electronic device are also stored in the RAM 703. The processing device 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0173] Typically, the following devices may be connected to the I / O interface 705: an input device 706 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 707 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 708 including, for example, a memory card, a hard disk, etc.; and a communication device 709. The communication device 709 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Figure 7 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0174] The embodiment of the present application also provides a computer program product including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any of the paralanguage annotation methods provided in the embodiment of the present application.

[0175] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any of the paralanguage annotation methods provided in the embodiments of the present application.

[0176] It should be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0177] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0178] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product. Professionals and technicians may use different methods to implement the described functions for each specific solution, but such implementation should not be considered beyond the scope of this application.

[0179] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

[0180] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0181] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A paralanguage annotation method, characterized in that: The method comprises: Performing paralanguage detection on the target speech data to obtain an initial detection result; the initial detection result includes: whether each speech frame of the target speech data belongs to a paralanguage, and the paralanguage category when it belongs to a paralanguage; According to the length of the paralanguage segment and the interval between the paralanguage segments, the initial detection result is corrected to obtain a target detection result; wherein continuous speech frames belonging to the same paralanguage category constitute a paralanguage segment; The text corresponding to the target speech data is annotated with a paralinguistic tag based on the target detection result.

2. The method according to claim 1, characterized in that The performing paralanguage detection on the target speech data includes: Encoding each speech frame of the target speech data to obtain a coding feature of each speech frame; Each speech frame is classified based on the coding features of each speech frame to obtain the initial detection result.

3. The method according to claim 1, characterized in that The classifying each speech frame based on the coding features of each speech frame to obtain the initial detection result includes: Based on the coding features of each speech frame, each speech frame is classified and predicted to obtain a probability distribution of each speech frame; the probability distribution of each speech frame is the probability of the speech frame corresponding to each category; the categories include: non-paralinguistic category and multiple paralinguistic categories; The initial category of each speech frame is determined in turn according to the probability distribution of each speech frame; wherein: If the maximum probability in the probability distribution of the first speech frame is greater than a first threshold, determining the initial category of the first speech frame to be the category corresponding to the maximum probability, otherwise determining the initial category of the first speech frame to be a non-paralanguage category; For each non-first speech frame, the initial category of the non-first speech frame is determined according to the initial category of the previous speech frame of the non-first speech frame and the probability distribution of the non-first speech frame.

4. The method according to claim 3, characterized in that The process of determining the initial category of a non-first speech frame includes: If the initial category of the previous speech frame of the non-first speech frame is the non-paralanguage category or the previous speech frame is the last frame of any paralanguage category, and the maximum probability in the probability distribution of the non-first speech frame is greater than a first threshold, the initial category of the non-first speech frame is determined to be the category corresponding to the maximum probability, otherwise, the initial category of the non-first speech frame is determined to be the non-paralanguage category; If the initial category of the previous speech frame of the non-first speech frame is any paralinguistic category, the previous speech frame is not the last frame of any paralinguistic category, and the probability corresponding to any paralinguistic category in the probability distribution of the non-first speech frame is greater than a second threshold, it is determined that the initial category of the non-first speech frame is any paralinguistic category and is not the last frame of any paralinguistic category; if the probability corresponding to any paralinguistic category in the probability distribution of the non-first speech frame is less than or equal to the second threshold, it is determined that the initial category of the non-first speech frame is any paralinguistic category and is the last frame of any paralinguistic category; the second threshold is greater than the first threshold.

5. The method according to claim 1, characterized in that The modifying of the initial detection result according to the length of the paralanguage segment and the interval between the paralanguage segments includes: If the speech frames in two paralanguage segments belong to the same paralanguage category, and the number of speech frames between the two paralanguage segments is less than a first preset number, the category to which the speech frames between the two paralanguage segments belong is updated to the paralanguage category to which the speech frames in the two paralanguage segments belong, and the speech frames between the two paralanguage segments and the two paralanguage segments are merged into one paralanguage segment.

6. The method according to claim 1, characterized in that The paralanguage categories include a plurality of determined paralanguage categories and uncertain paralanguage categories; the initial detection result is corrected according to the length of the paralanguage segment and the interval between the paralanguage segments, including at least one of the following: If the speech frames in two paralanguage segments belong to the same paralanguage category, and the number of speech frames between the two paralanguage segments is less than a first preset number, updating the category to which the speech frames between the two paralanguage segments belong to be the paralanguage category to which the speech frames in the two paralanguage segments belong, and merging the speech frames between the two paralanguage segments and the two paralanguage segments into one paralanguage segment; If the speech frames in the first paralanguage segment of two paralanguage segments belong to a determined first paralanguage category, and the speech frames in the second paralanguage segment belong to an uncertain paralanguage category, the number of speech frames included in the second paralanguage segment is less than a second preset number, and the number of speech frames between the two paralanguage segments is less than the first preset number, then the categories to which the speech frames in the second paralanguage segment and the speech frames between the two paralanguage segments belong are updated to the first paralanguage category, and the speech frames between the two paralanguage segments and the two paralanguage segments are merged into one paralanguage segment.

7. The method according to claim 5 or 6, characterized in that: Also includes: The paralanguage segment including a number of speech frames less than a third preset number is modified into a non-paralanguage segment.

8. The method according to claim 1, characterized in that The performing paralinguistic tagging on the text corresponding to the target speech data based on the target detection result includes: Corresponding to any para-language segment, according to the target temporal relationship between each speech frame in the any para-language segment and the speech frames corresponding to each word in the text, a para-language tag corresponding to the any para-language segment is added to the text; The paralanguage tag represents the paralanguage category to which any paralanguage segment belongs and its start time and end time; the temporal relationship between the paralanguage tag and each word in the text is the target temporal relationship.

9. A paralanguage annotation device, characterized in that: include: A detection module, used to perform paralanguage detection on the target speech data to obtain an initial detection result; The initial detection result includes: whether each speech frame of the target speech data belongs to a paralanguage, and the paralanguage category if it belongs to a paralanguage; A correction module is used to correct the initial detection result according to the length of the paralanguage segment and the interval between the paralanguage segments to obtain a target detection result; wherein continuous speech frames belonging to the same paralanguage category constitute a paralanguage segment; The tagging module is used to perform paralinguistic tagging on the text corresponding to the target speech data based on the target detection result.

10. A computer program product, characterized in that The method comprises computer-readable instructions, and when the computer-readable instructions are executed on an electronic device, the electronic device implements the paralanguage annotation method according to any one of claims 1 to 8.

11. An electronic device, characterized in that: The electronic device comprises at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is configured to execute the computer program so that the electronic device can implement the paralanguage annotation method according to any one of claims 1 to 8.

12. A computer storage medium, characterized in that: The storage medium carries one or more computer programs, and when the one or more computer programs are executed by an electronic device, the electronic device can implement the paralanguage annotation method according to any one of claims 1 to 8.

Images