Paralanguage annotation method, device, equipment, storage medium and program product

Abstract

This application discloses a paralanguage annotation method, apparatus, device, storage medium, and program product, relating to the field of speech processing technology. The method comprises: performing paralanguage detection on target speech data to obtain an initial detection result, the initial detection result including: whether each speech frame of the target speech data belongs to a paralanguage, and the paralanguage category if it does; correcting the initial detection result based on the length of a paralanguage segment consisting of consecutive speech frames belonging to the same paralanguage category, and the interval between paralanguage segments, to obtain a target detection result; and performing paralanguage annotation on the text corresponding to the target speech data based on the target detection result. This application achieves the goal of automatically annotating text with paralanguage, improves the efficiency of paralanguage annotation, and reduces the cost of paralanguage annotation.

Description

Technical Field

[0001] The present application relates to the field of audio processing technology, and in particular to a paralanguage annotation method, apparatus, device, storage medium, and program product. Background Art

[0002] Paralanguage refers to non-verbal elements in communication beyond spoken language, such as inhalations and laughter. These elements help convey information or enhance communication. In speech synthesis, paralanguage, as a non-verbal feature, provides rich contextual and emotional information. This information plays a crucial role in the comprehensive understanding and processing of speech signals, enhancing the naturalness and fluency of synthesized speech.

[0003] Currently, speech synthesis models are trained using text as training samples and the corresponding speech as labels. To improve speech synthesis, the training text is annotated with paralinguistic tags, which adds paralinguistic tags to certain locations within the text. However, current paralinguistic tags are manually added to the text after listening to the corresponding speech, resulting in low annotation efficiency.

[0004] Therefore, how to improve the efficiency of paralanguage annotation has become a technical problem that needs to be solved urgently. Summary of the Invention

[0005] In view of the above problems, this application provides a paralanguage annotation method, apparatus, device, storage medium, and program product to improve the efficiency of paralanguage annotation. The specific solution is as follows:

[0006] A first aspect of the present application provides a paralanguage annotation method, the method comprising:

[0007] Performing paralanguage detection on the target speech data to obtain an initial detection result; the initial detection result includes: whether each speech frame of the target speech data belongs to a paralanguage, and the paralanguage category if it belongs to a paralanguage;

[0008] The initial detection result is modified according to the length of the paralanguage segment and the interval between the paralanguage segments to obtain a target detection result; wherein consecutive speech frames belonging to the same paralanguage category constitute a paralanguage segment;

[0009] Perform paralinguistic annotation on text corresponding to the target speech data based on the target detection result.

[0010] In a possible implementation, performing paralanguage detection on the target speech data includes:

[0011] Encoding each speech frame of the target speech data to obtain encoding features of each speech frame;

[0012] Each speech frame is classified based on the coding features of each speech frame to obtain the initial detection result.

[0013] In a possible implementation, classifying each speech frame based on the coding features of each speech frame to obtain the initial detection result includes:

[0014] Classifying and predicting each speech frame based on its coding features to obtain a probability distribution for each speech frame; the probability distribution for each speech frame is the probability of the speech frame corresponding to each category; the categories include: a non-paralanguage category and multiple paralanguage categories;

[0015] The initial category of each speech frame is determined in turn according to the probability distribution of each speech frame; wherein:

[0016] If the maximum probability in the probability distribution of the first speech frame is greater than a first threshold, determining the initial category of the first speech frame to be the category corresponding to the maximum probability; otherwise, determining the initial category of the first speech frame to be a non-paralanguage category;

[0017] For each non-first speech frame, the initial category of the non-first speech frame is determined according to the initial category of the previous speech frame of the non-first speech frame and the probability distribution of the non-first speech frame.

[0018] In one possible implementation, the process of determining the initial category of a non-first speech frame includes:

[0019] If the initial category of the previous speech frame of the non-first speech frame is the non-paralanguage category or the previous speech frame is the last frame of any paralanguage category, and the maximum probability in the probability distribution of the non-first speech frame is greater than a first threshold, determining the initial category of the non-first speech frame to be the category corresponding to the maximum probability; otherwise, determining the initial category of the non-first speech frame to be the non-paralanguage category;

[0020] If the initial category of the previous speech frame of the non-first speech frame is any paralanguage category, the previous speech frame is not the last frame of the any paralanguage category, and the probability corresponding to the any paralanguage category in the probability distribution of the non-first speech frame is greater than a second threshold, it is determined that the initial category of the non-first speech frame is the any paralanguage category and is not the last frame of the any paralanguage category; if the probability corresponding to the any paralanguage category in the probability distribution of the non-first speech frame is less than or equal to the second threshold, it is determined that the initial category of the non-first speech frame is the any paralanguage category and is the last frame of the any paralanguage category; the second threshold is greater than the first threshold.

[0021] In a possible implementation, the modifying of the initial detection result according to the length of the paralanguage segment and the interval between the paralanguage segments includes:

[0022] If the speech frames in two paralanguage segments belong to the same paralanguage category, and the number of speech frames between the two paralanguage segments is less than a first preset number, the categories to which the speech frames between the two paralanguage segments belong are updated to the paralanguage categories to which the speech frames in the two paralanguage segments belong, and the speech frames between the two paralanguage segments and the two paralanguage segments are merged into one paralanguage segment.

[0023] In one possible implementation, the paralanguage categories include a plurality of determined paralanguage categories and indeterminate paralanguage categories; and the modifying of the initial detection result based on the length of the paralanguage segments and the intervals between the paralanguage segments includes at least one of the following:

[0024] If the speech frames in two paralanguage segments belong to the same paralanguage category, and the number of speech frames between the two paralanguage segments is less than a first preset number, updating the category of the speech frames between the two paralanguage segments to the paralanguage category of the speech frames in the two paralanguage segments, and merging the speech frames between the two paralanguage segments and the two paralanguage segments into one paralanguage segment;

[0025] If the speech frames in a first paralanguage segment of two paralanguage segments belong to a determined first paralanguage category, and the speech frames in a second paralanguage segment belong to an uncertain paralanguage category, the number of speech frames contained in the second paralanguage segment is less than a second preset number, and the number of speech frames located between the two paralanguage segments is less than the first preset number, then the categories to which the speech frames in the second paralanguage segment and the speech frames located between the two paralanguage segments belong are updated to the first paralanguage category, and the speech frames located between the two paralanguage segments and the two paralanguage segments are merged into one paralanguage segment.

[0026] In a possible implementation, the method further includes:

[0027] The paralanguage segment including a number of speech frames less than a third preset number is modified into a non-paralanguage segment.

[0028] In a possible implementation, performing paralinguistic tagging on the text corresponding to the target speech data based on the target detection result includes:

[0029] For each paralanguage segment, based on the target temporal relationship between each speech frame in the paralanguage segment and the speech frames corresponding to each word in the text, a paralanguage tag corresponding to the paralanguage segment is added to the text;

[0030] The paralanguage tag represents the paralanguage category to which any paralanguage segment belongs and its start time and end time; and the temporal relationship between the paralanguage tag and each word in the text is the target temporal relationship.

[0031] A second aspect of the present application provides a paralanguage annotation device, comprising:

[0032] a detection module configured to perform paralanguage detection on the target speech data to obtain an initial detection result; the initial detection result includes: whether each speech frame of the target speech data belongs to a paralanguage, and the paralanguage category if it belongs to a paralanguage;

[0033] a correction module, configured to correct the initial detection result according to the length of the paralanguage segment and the interval between the paralanguage segments to obtain a target detection result; wherein consecutive speech frames belonging to the same paralanguage category constitute a paralanguage segment;

[0034] The tagging module is configured to perform paralinguistic tagging on the text corresponding to the target speech data based on the target detection result.

[0035] A third aspect of the present application provides a computer program product, comprising computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements the paralanguage annotation method of the first aspect or any implementation of the first aspect.

[0036] A fourth aspect of the present application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:

[0037] The memory is used to store computer programs;

[0038] The processor is configured to execute the computer program so that the electronic device can implement the paralanguage annotation method of the first aspect or any implementation of the first aspect.

[0039] In a fifth aspect, the present application provides a computer storage medium carrying one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the paralanguage annotation method according to the first aspect or any implementation of the first aspect.

[0040] By leveraging the aforementioned technical solution, the paralanguage annotation method, apparatus, device, storage medium, and program product provided in this application perform paralanguage detection on target speech data to obtain an initial detection result. The initial detection result includes: whether each speech frame of the target speech data belongs to a paralanguage, and if so, the paralanguage category. The initial detection result is corrected based on the length of a paralanguage segment consisting of consecutive speech frames belonging to the same paralanguage category, as well as the interval between paralanguage segments, to obtain a target detection result. Based on the target detection result, the text corresponding to the target speech data is annotated with the paralanguage. This application achieves the goal of automatically annotating text with paralanguage, improves the efficiency of paralanguage annotation, and reduces the cost of paralanguage annotation. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The above and other features, advantages, and aspects of the various embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that the originals and elements are not necessarily drawn to scale.

[0042] Figure 1 A flowchart of an implementation of the paralanguage annotation method provided in this application;

[0043] Figure 2 An example diagram of the occurrence of paralanguage across discourses provided for this application;

[0044] Figure 3 A flowchart for implementing paralanguage detection on target speech data provided by this application;

[0045] Figure 4 A schematic diagram of the structure of the paralanguage recognition model provided in this application;

[0046] Figure 5 A flowchart for classifying each speech frame based on the coding features of each speech frame to obtain an initial detection result provided by the present application;

[0047] Figure 6 A schematic diagram of the structure of the paralanguage annotation device provided in this application;

[0048] Figure 7 A schematic diagram of the structure of the electronic device provided in this application. DETAILED DESCRIPTION

[0049] The following describes the embodiments of the present application in conjunction with the accompanying drawings. The terms used in the implementation methods of the present application are only used to explain the specific embodiments of the present application and are not intended to limit the present application.

[0050] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0051] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, and this is merely a way of distinguishing the objects of the same attributes when describing them in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0052] In order to improve the efficiency of paralanguage annotation, this application scheme is proposed.

[0053] like Figure 1 As shown, a flowchart of an implementation of the paralanguage annotation method provided in an embodiment of the present application may include:

[0054] Step S101: Perform paralanguage detection on target speech data to obtain an initial detection result, which includes whether each speech frame of the target speech data belongs to a paralanguage and the paralanguage category if it belongs to a paralanguage.

[0055] The present application performs frame-level paralanguage detection on target speech data, that is, detecting whether each speech frame of the target speech data is a paralanguage, and if so, which paralanguage it is.

[0056] In natural speech conversations, common paralinguistic information includes inhalation, lip smacking, laughter, dragging, pauses, stuttering, repetition, etc. An example of the appearance of paralinguistic information between words is shown in the figure below. Figure 2 As shown, it is often a short message, for example, the inhalation sound, which usually occurs tens of milliseconds before the speech.

[0057] Step S102: Based on the length of the paralanguage segments and the intervals between the segments, the initial detection result is modified to obtain the target detection result. Continuous speech frames belonging to the same paralanguage category constitute a paralanguage segment.

[0058] To avoid the negative impact of false detection on paralanguage annotation, this application treats consecutive speech frames belonging to the same paralanguage category as a paralanguage segment. The initial detection result is then corrected based on the length of each paralanguage segment and the interval between the segments to obtain the target detection result.

[0059] Methods for correcting the initial detection results may include, but are not limited to, at least one of the following: merging paralanguage segments with smaller intervals (e.g., intervals smaller than a certain threshold) and containing speech frames belonging to the same paralanguage category, correcting paralanguage segments with smaller lengths, etc.

[0060] This application's research found that speech frames are typically 20ms long, but in reality, humans only perceive segments of at least 0.2 seconds. Therefore, it is necessary to merge frame-level results into segments. By merging paralanguage segments that contain speech frames of the same paralanguage category and correcting for shorter paralanguage segments, we ensure that the detected paralanguage segments are human-perceivable.

[0061] Step S103: performing paralinguistic annotation on the text corresponding to the target speech data based on the target detection result.

[0062] The text corresponding to the target voice data may be text obtained by performing voice recognition on the target voice data.

[0063] Alternatively, the target voice data is voice data obtained by a reader reading the text.

[0064] Optionally, the correspondence between each word in the text and the speech frames in the target speech data may be preset, that is, each word in the text may be marked in advance to which speech frames in the target speech data it corresponds to.

[0065] As an example, the target speech data and the corresponding text can be any text sample in a dataset used to train a speech synthesis model, as well as the speech tags of the text, where the speech tags are speech frames of the target speech data annotated with individual words in the text. Based on this, the correspondence between individual words and speech frames in this application can be easily obtained.

[0066] For any paralanguage segment, the paralanguage tag corresponding to the segment can be added to the text based on the temporal relationship between the speech frames in the segment and the speech frames corresponding to the individual words in the text, thereby achieving a paralanguage annotation of the text.

[0067] The paralanguage annotation method provided in an embodiment of the present application performs paralanguage detection on target speech data to obtain an initial detection result. The initial detection result includes: whether each speech frame of the target speech data belongs to a paralanguage, and the paralanguage category if it does belong to a paralanguage; the initial detection result is corrected based on the length of paralanguage segments consisting of consecutive speech frames belonging to the same paralanguage category, and the interval between paralanguage segments, to obtain a target detection result; and based on the target detection result, the text corresponding to the target speech data is annotated with the paralanguage. This application achieves the goal of automatically annotating text with paralanguage, improves the efficiency of paralanguage annotation, and reduces the cost of paralanguage annotation.

[0068] In an optional embodiment, a flowchart of implementing the above-mentioned paralanguage detection of target speech data is as follows: Figure 3 As shown, this may include:

[0069] Step S301: Encode each speech frame of the target speech data to obtain encoding features of each speech frame.

[0070] Optionally, the audio features of each speech frame can be extracted first, and then input into a trained encoding module (referred to as the first encoding module for ease of description and distinction). The first encoding module then uses the audio features of each speech frame to encode each speech frame, thereby obtaining encoding features for each speech frame. Common audio features include spectral features, cepstral features, Mel-frequency cepstral coefficients (MFCCs), and FilterBank features. These features can reflect the spectral characteristics, energy distribution, and pitch information of the speech signal. As an example, the first encoding module can be trained based on the encoding module in the pre-trained Whisper model.

[0071] Alternatively, the target speech data can be directly input into a pre-trained encoding module (referred to as the second encoding module for ease of description and distinction). The second encoding module extracts features from each speech frame and encodes them to obtain encoded features for each speech frame. For example, the second encoding module can be trained based on the encoding module in a pre-trained Hubert_Base_CN model, or the second encoding module can be trained based on a pre-trained Wav2vec model.

[0072] Step S302: Classify each speech frame based on its coding features to obtain an initial detection result.

[0073] Optionally, each speech frame can be classified and predicted based on its coding features to obtain a probability distribution for each speech frame. The initial detection result is determined based on the probability distribution for each speech frame. The probability distribution for each speech frame represents the probability of the speech frame corresponding to each category. The categories here include non-paralanguage categories and multiple paralanguage categories.

[0074] For example, the categories include five: inhalation (denoted as NXQ), lip smacking (denoted as NZZ), laughter (denoted as NXS, where laughter refers to laughter between sentences and continuous laughter), drawl (denoted as TY, referring to prolonged sound), and non-paralinguistic information (denoted as OTHER, such as silence, human voice, etc.). Based on this, the probability distribution of each speech frame is the probability that the speech frame corresponds to each of the above five categories. It should be noted that the five categories here are only examples and do not constitute a limitation of this application. In actual applications, more categories can be used, for example, including other paralinguistic information (denoted as NN, which refers to other paralinguistic sounds that cannot be categorized and is used to absorb probability), freezes (the phenomenon of the speaker's mind going blank and falling into silence or repeatedly correcting grammatical errors), repetitions, etc.

[0075] In an optional embodiment, encoding each speech frame of the target speech data and classifying and predicting each speech frame based on the encoding features of each speech frame can be achieved through a paralanguage recognition model. Figure 4 FIG. 1 is a schematic diagram of a structure of a paralanguage recognition model provided in an embodiment of the present application, which may include:

[0076] Encoding module 401 and classification module 402;

[0077] The encoding module 401 is used to encode the input target speech data or the audio features of each speech frame of the target speech data to obtain the encoding features of each speech frame of the target speech data.

[0078] That is, the input of the encoding module 401 is the target speech data, or the input of the encoding module 401 is the audio features of each speech frame of the target speech data.

[0079] The classification module 402 is used to perform classification prediction on each speech frame based on the coding features of each speech frame to obtain the probability distribution of each speech frame.

[0080] The paralanguage recognition model can be trained using speech data as training samples and using the categories annotated for each speech frame as labels. The category annotated for any speech frame indicates which category the speech frame belongs to. Taking six categories (inhalation, lip smacking, laughter, drawl, other paralanguage information, and non-paralanguage information) as an example, the category label of a speech frame is 100000, indicating that the speech frame belongs to paralanguage and is an inhalation sound; the category label of a speech frame is 010000, indicating that the speech frame belongs to paralanguage and is a lip smacking sound; the category label of a speech frame is 001000, indicating that the speech frame belongs to paralanguage and is a laughter; the category label of a speech frame is 000100, indicating that the speech frame belongs to paralanguage and is a drawl; the category label of a speech frame is 000010, indicating that the speech frame belongs to paralanguage and is other paralanguage information; the category label of a speech frame is (000001), indicating that the speech frame does not belong to paralanguage, that is, the speech frame belongs to non-paralanguage information.

[0081] The initial parameters of encoding module 401 are the parameters of a pre-trained encoding module. This pre-trained encoding module can be the encoding module of a pre-trained speech recognition model (e.g., HuBERT, Whisper, etc.). By comparison, encoding module 401 trained based on the encoding module in the pre-trained Hubert_Base_CN model performs best.

[0082] The classification module 402 can be a linear classification layer or a nonlinear classification layer. The linear classification layer can perform a linear transformation on the coded features of each received speech frame to map them to a category space, and then convert them into a probability distribution through softmax. The nonlinear classification layer can perform a nonlinear transformation on the coded features of each received speech frame to map them to a category space, and then convert them into a probability distribution through softmax.

[0083] The classification module 402 can adopt a simple fully connected network or a more complex structure, such as a convolutional network (CNN), a long short-term memory (LSTM) network, or an attention-based network (Transformer). If the amount of training data is relatively small, a simple fully connected network can be used; if the amount of training data is relatively large, a more complex structure can be used.

[0084] The training process may include two stages of training. In the first stage of training, the parameters of the encoding module 401 are frozen, and only the parameters of the classification module 402 are updated. After the classification module 402 training is stable, the second stage of training begins. In the second stage of training, the parameters of the encoding module 401 and the classification module 402 are updated simultaneously.

[0085] As an example, in the first phase of training, the audio features of a training sample or each speech frame of the training sample can be input into encoding module 401 to obtain encoding features for each speech frame of the training sample. Classification module 402 then performs classification prediction on the encoding features of each speech frame of the training sample to obtain a probability distribution for each speech frame of the training sample. The parameters of classification module 402 are updated with the goal of ensuring that the category represented by the probability distribution of each speech frame approximates the label corresponding to the speech frame. Optionally, a sample-balanced loss (Focal Loss) can be used to align the probability distribution of speech frames with the corresponding label. Specifically, a Focal Loss loss is calculated based on the label corresponding to the speech frame, and the parameters of classification module 402 are updated with the goal of minimizing this loss. The change in the loss in each round relative to the previous round can be monitored. When the change in the loss is less than a threshold for a certain number of consecutive times, the first phase of training is concluded, and the second phase of training can begin.

[0086] In the second phase of training, the audio features of the training sample or each speech frame of the training sample can be input into the encoding module 401 to obtain the encoding features of each speech frame of the training sample. Based on the encoding features obtained in the first phase of training, the classification module 402 performs classification prediction on each speech frame of the training sample to obtain the probability distribution of each speech frame of the training sample. The parameters of the encoding module 401 and the classification module 402 are updated with the goal of ensuring that the category represented by the probability distribution of each speech frame approaches the corresponding label of the speech frame. Optionally, a sample-balanced loss (Focal Loss) can be used to align the probability distribution of the speech frames with the corresponding labels. Specifically, a Focal Loss loss is calculated based on the corresponding labels of the speech frames. The parameters of the encoding module 401 and the classification module 402 are updated with the goal of minimizing this loss. The change in the loss in each round relative to the previous round can be monitored. The second phase of training is determined to be complete when the change in the loss is less than a threshold for a certain number of consecutive times.

[0087] The first stage training and the second stage training use the same training dataset.

[0088] When the classification module 402 adopts a simple fully connected network, the classification module 402 can achieve the effect of a complex network through the second stage of training.

[0089] When the classification module 402 adopts a relatively complex network structure, the second stage of training can enable the classification module 402 to achieve the effect of a more complex network.

[0090] In an optional embodiment, the above-mentioned classification of each speech frame based on the coding features of each speech frame to obtain an implementation flow chart of the initial detection result is as follows: Figure 5 As shown, this may include:

[0091] Step S501: Classify and predict each speech frame based on its coding features to obtain a probability distribution for each speech frame. The probability distribution for each speech frame represents the probability of the speech frame corresponding to each category. The categories include a non-paralanguage category and multiple paralanguage categories.

[0092] The specific implementation method can be found in the above embodiments and will not be described in detail here.

[0093] Step S502: determining the initial category of each speech frame according to the probability distribution of each speech frame in turn.

[0094] If the maximum probability in the probability distribution of the first speech frame is greater than a first threshold, the initial category of the first speech frame is determined to be the category corresponding to the maximum probability; otherwise, the initial category of the first speech frame is determined to be the non-paralanguage category.

[0095] Optionally, the first threshold may be 0.5, or other values, such as 0.48, 0.51, etc. The effect when the first threshold is 0.5 is better than the effect when the first threshold is other values.

[0096] This study found that paralanguage information is the information between speakers' utterances, and two paralanguages ​​cannot trigger each other simultaneously. Setting the first threshold to 0.5 ensures that only one category can be activated simultaneously in each frame. Furthermore, in the absence of paralanguage, a probability of a particular category rising above 0.5 can be considered a trigger for paralanguage. False triggers can be eliminated through subsequent correction processes, so the first threshold can be set lower, such as 0.5.

[0097] For each non-first speech frame, the initial category of the non-first speech frame is determined according to the initial category of the previous speech frame of the non-first speech frame and the probability distribution of the non-first speech frame.

[0098] When determining the initial category of each speech frame, the present application not only considers the probability that the speech frame belongs to each category, but also considers the initial category to which the previous speech frame of the speech frame belongs.

[0099] The manner of determining the initial category of each non-first speech frame is the same. In an optional embodiment, one implementation manner of determining the initial category of the non-first speech frame may be:

[0100] If the initial category of the previous speech frame of the non-first speech frame is the non-paralanguage category or the previous speech frame is the last frame of any paralanguage category, and the maximum probability in the probability distribution of the non-first speech frame is greater than a first threshold, then the initial category of the non-first speech frame is determined to be the category corresponding to the maximum probability; otherwise, the initial category of the non-first speech frame is determined to be the non-paralanguage category.

[0101] If the initial category of the previous speech frame of the non-first speech frame is any sub-language category, the previous speech frame is not the last frame of the any sub-language category, and the probability corresponding to the any sub-language category in the probability distribution of the non-first speech frame is greater than the second threshold, then it is determined that the initial category of the non-first speech frame is the any sub-language category, but it is not the last frame of the any sub-language category; if the probability corresponding to the any sub-language category in the probability distribution of the non-first speech frame is less than or equal to the second threshold, it is determined that the initial category of the non-first speech frame is the any sub-language category and it is the last frame of the any sub-language category.

[0102] The second threshold is greater than the first threshold. Optionally, the second threshold can be 0.95, or other values, such as 0.9, 0.97, etc. This study found that in the middle of a paralanguage segment, the probability reaches a maximum value between 0.98 and 0.99. Exiting the segment with a probability less than 0.95 can accelerate the exit of the paralanguage segment, allowing misjudged frames in the middle to be absorbed in subsequent corrections, resulting in more accurate results.

[0103] For example, assuming that the initial category of the n-1th (n is an integer greater than 1) speech frame is an inhalation sound, and it is not the last frame of the inhalation sound, then when determining the initial category of the nth speech frame, the probability distribution of the nth speech frame (the maximum probability may or may not correspond to an inhalation sound) is considered to correspond to the inhalation sound. If the probability corresponding to the inhalation sound is greater than 0.95, then the initial category of the nth speech frame is considered to be an inhalation sound, and it has not ended yet, that is, the nth speech frame is not the last frame of the inhalation sound. If the probability corresponding to the inhalation sound is less than or equal to 0.95, then the initial category of the nth speech frame is considered to be an inhalation sound, and the nth speech frame is considered to be the last frame of the inhalation sound.

[0104] When determining the initial category of the n+1th speech frame, if the initial category of the nth speech frame is an inhalation and it is not the last frame with an inhalation, then the probability corresponding to the inhalation in the probability distribution of the n+1th speech frame (the maximum probability may or may not correspond to an inhalation) is considered. If the probability corresponding to the inhalation is greater than 0.95, then the initial category of the n+1th speech frame is considered to be an inhalation and it is not the last frame with an inhalation. If the probability corresponding to the inhalation is less than or equal to 0.95, then the initial category of the n+1th speech frame is considered to be an inhalation and the n+1th speech frame is considered to be the last frame with an inhalation.

[0105] When determining the initial category of the n+1th speech frame, if the initial category of the nth speech frame is an inhalation sound and it is the last frame of the inhalation sound, then the maximum probability in the probability distribution of the n+1th speech frame is considered. If the maximum probability is greater than 0.5, the n+1th frame is considered to be of the category corresponding to the maximum probability. Otherwise, the n+1th frame is considered not to be paraspeech.

[0106] In an optional embodiment, an implementation method for correcting the initial detection result based on the length of the paralanguage segment and the interval between the paralanguage segments may be:

[0107] If the speech frames in the two paralanguage segments belong to the same paralanguage category, and the number of speech frames between the two paralanguage segments is less than a first preset number, the category to which the speech frames between the two paralanguage segments belong is updated to the paralanguage category to which the speech frames in the two paralanguage segments belong, and the speech frames between the two paralanguage segments and the two paralanguage segments are merged into one paralanguage segment.

[0108] The two paralanguage segments here can be two adjacent paralanguage segments (i.e., there is no paralanguage segment between the two paralanguage segments), or they can be two non-adjacent paralanguage segments (i.e., there is at least one paralanguage segment between the two paralanguage segments).

[0109] The speech frames can be traversed in a sequential order, that is, from front to back, to sequentially determine similar paralanguage segments. If the speech frames in two paralanguage segments belong to the same paralanguage category, and the number of speech frames between the two paralanguage segments is less than a first preset number, the category of the speech frames between the two paralanguage segments is updated to the paralanguage category of the speech frames in the two paralanguage segments, and the speech frames between the two paralanguage segments are merged with the two paralanguage segments into one paralanguage segment. Otherwise, no merging is performed, i.e., the relative positional relationship between the two paralanguage segments remains unchanged.

[0110] Optionally, the first preset number may be 5, or other values, such as 4, 6 or 7.

[0111] As an example, assume that none of the first through sixth speech frames are paralanguage, and all ten speech frames from the seventh through the sixteenth (referred to as the first paralanguage segment) are inhalations. The seventeenth speech frame is not an inhalation, and all seven speech frames from the eighteenth through the twenty-fourth (referred to as the second paralanguage segment) are inhalations. Since both the first and second paralanguage segments are inhalations, and there is only one speech frame between them (a total of less than five), the first, seventeenth, and second paralanguage segments are merged into a new inhalation segment, referred to as the third paralanguage segment. This third paralanguage segment includes the eighteen speech frames from the seventh through the twenty-fourth, all of which are inhalations. In other words, the classification of the seventeenth speech frame is modified to inhalation.

[0112] Furthermore, assuming that none of the speech frames from the 25th to the 27th are inhalation sounds, and the speech frames from the 28th to the 30th (recorded as the fourth paralanguage segment) are inhalation sounds, then since the third and fourth paralanguage segments are both inhalation sound segments, and there are only three speech frames between these two paralanguage segments, which is less than 5, the third paralanguage segment, the speech frames from the 25th to the 27th, and the fourth paralanguage segment are merged into a new inhalation sound segment, recorded as the fifth paralanguage segment. That is, the fifth paralanguage segment includes the 24 speech frames from the 7th to the 30th speech frame, and these 24 speech frames are all inhalation sounds.

[0113] Assume that none of the speech frames from the 25th to the 27th belong to paralanguage, and all of the speech frames from the 28th to the 30th (recorded as the fourth paralanguage segment) belong to lip smacking sounds. Since the fourth and third paralanguage segments belong to different categories of paralanguage segments, it is not determined whether the third and fourth paralanguage segments and the speech frames in between need to be merged. Instead, it is directly determined whether there is a paralanguage segment containing lip smacking sounds after the fourth paralanguage segment that can be merged with the fourth paralanguage segment. This process is repeated until all paralanguage segments are traversed.

[0114] Furthermore, after the merging process is completed, the paralanguage segment containing a number of speech frames less than a third preset number may be modified into a non-paralanguage segment.

[0115] As an example, the third preset number may be 10. That is, if the number of speech frames in a paralanguage segment is less than 10, the paralanguage segment is considered a misclassification, and the classification of each speech frame in the paralanguage segment is changed to non-paralanguage. The third preset number may also be other values, such as 8, 9, 11, or 12.

[0116] In an optional embodiment, the paralanguage categories include multiple defined paralanguage categories (e.g., inhales, lip smacking, laughter, and drawls) and indeterminate paralanguage categories (e.g., other paralanguage information; i.e., paralanguage information that is not inhales, lip smacking, laughter, or drawls). Based on this, another implementation method for modifying the initial detection results based on the length and spacing of the paralanguage segments can be:

[0117] If the speech frames in the two paralanguage segments belong to the same paralanguage category, and the number of speech frames between the two paralanguage segments is less than a first preset number, the category to which the speech frames between the two paralanguage segments belong is updated to the paralanguage category to which the speech frames in the two paralanguage segments belong, and the speech frames between the two paralanguage segments and the two paralanguage segments are merged into one paralanguage segment.

[0118] The specific implementation process can be found in the aforementioned embodiment and will not be described again here.

[0119] If the speech frames in the first of the two paralanguage segments belong to the determined first paralanguage category, and the speech frames in the second paralanguage segment belong to the uncertain paralanguage category, the number of speech frames contained in the second paralanguage segment is less than the second preset number, and the number of speech frames located between the two paralanguage segments is less than the first preset number, then the categories to which the speech frames in the second paralanguage segment and the speech frames located between the two paralanguage segments belong are updated to the first paralanguage category, and the speech frames located between the two paralanguage segments and the two paralanguage segments are merged into one paralanguage segment.

[0120] Optionally, the second preset number may be 10, or other values, such as 8, 9, 11, 12, etc.

[0121] The two paralanguage segments here can be two adjacent paralanguage segments (i.e., there is no paralanguage segment between the two paralanguage segments), or they can be two non-adjacent paralanguage segments (i.e., there is at least one paralanguage segment between the two paralanguage segments). As an example, assume that none of the first to sixth speech frames belong to paralanguage, the ten speech frames from the seventh to the sixteenth speech frames (referred to as the first paralanguage segment) belong to inhalations, the seventeenth speech frame belongs to neither inhalation nor other paralanguage information, and the seven speech frames from the eighteenth to the twenty-fourth speech frames (referred to as the second paralanguage segment) belong to other paralanguage information. Since both the first and second paralanguage segments are paralanguage segments, there is only one speech frame between the first and second paralanguage segments, which is less than five, and the second paralanguage segment contains less than ten speech frames. Therefore, the first, seventeenth, and second paralanguage segments are merged into a new inhalation segment, referred to as the third paralanguage segment. That is, the third paralanguage segment includes the eighteen speech frames from the seventh to the twenty-fourth speech frames, and all of these eighteen speech frames belong to inhalations. That is, the categories of the 17th speech frame and the speech frames in the second paralanguage segment are corrected to inhalation sounds.

[0122] Each speech frame can be traversed in the order of the speech frames, that is, from the front to the back, and similar paralanguage segments can be judged and processed in turn based on the above method until all paralanguage segments are traversed.

[0123] Furthermore, after the merging process is completed, the paralanguage segment containing less than a third preset number of speech frames can be modified into a non-paralanguage segment. Specific implementation methods can be found in the above embodiments and will not be described in detail here.

[0124] In an optional embodiment, an implementation method of performing paralinguistic tagging on the text corresponding to the target speech data based on the target detection result may be:

[0125] For each paralanguage segment, based on the temporal relationship between each speech frame in the paralanguage segment and the speech frames corresponding to each word in the text (referred to as the target temporal relationship for ease of description and distinction), add the paralanguage tag corresponding to the paralanguage segment to the text;

[0126] The paralanguage tag represents the paralanguage category to which any paralanguage segment belongs and its start time and end time; the temporal relationship between the paralanguage tag and each word in the text is the target temporal relationship.

[0127] The start time refers to the start time of the first speech frame in any one of the speech segments, and the end time refers to the end time of the last speech frame in any one of the speech segments.

[0128] As an example, suppose a paralanguage segment includes speech frames 20 through 29, and all of these 10 frames are lip-smacking sounds. The speech frames temporally adjacent to these 10 frames in the text correspond to two adjacent characters, where the first character corresponds to the 10th through 19th frames, and the second character corresponds to the 30th through 42nd frames. Clearly, the speech frames in this paralanguage segment are located between the speech frames corresponding to these two adjacent characters. Therefore, the paralanguage label corresponding to this paralanguage segment is added between these two adjacent characters.

[0129] The following is an evaluation of the method used in this application. Generally, the accuracy of paraspeech recognition is judged at the frame level, that is, the accuracy of each frame in the audio is statistically analyzed. However, in practice, we found that this judgment is not appropriate. For example, if a paraspeech segment is from 0.15 seconds to 0.45 seconds, our output result is 0.14 seconds to 0.50 seconds. This is completely correct in use, but there will be an error of 0.06 seconds in the evaluation index. Moreover, we are more concerned with which segments of paraspeech are correctly / incorrectly recognized, rather than which frames are correctly / incorrectly recognized. Therefore, the evaluation standard used is the interval level, where there are two cases where the interval is correct:

[0130] Case 1: The difference between the start time and the end time of the two intervals (i.e., the predicted paralanguage interval and the marked paralanguage interval) are both smaller than the threshold of 0.05 seconds.

[0131] Case 2: The difference between the start or end times of the two intervals (i.e., the predicted paralanguage interval and the annotated paralanguage interval) is less than the threshold of 0.05 seconds, and the length of the resulting interval outside the annotated interval is less than 30% of the annotated length. This means that the difference between the start time of the predicted paralanguage interval and the start time of the annotated paralanguage interval is less than the threshold of 0.05 seconds, and the end time of the predicted paralanguage interval exceeds the end time of the annotated paralanguage interval by no more than 30%. Alternatively, the difference between the end time of the predicted paralanguage interval and the end time of the annotated paralanguage interval is less than the threshold of 0.05 seconds, and the start time of the predicted paralanguage interval exceeds the start time of the annotated paralanguage interval by no more than 30%.

[0132] The evaluation indicators used are precision and recall, where:

[0133] The accuracy of each category is: Precision = the number of correct intervals in the current class result / the number of intervals in the current class result

[0134] The recall rate of each category is: Recall = the number of intervals hit in the current class annotation / the number of intervals in the current class annotation

[0135] The overall accuracy is: Precision = the number of correct intervals in all results / the number of intervals in all results

[0136] The overall recall rate is: Recall = the number of intervals hit in all annotations / the number of all annotation intervals

[0137] According to the above evaluation criteria, the final evaluation results are:

[0138] Overall: precision = 80.044%, recall = 61.08%;

[0139] Inspiratory NXQ: precision = 85.22%, recall = 69.04%;

[0140] Laughter NXS: precision = 64.71%, recall = 52.89%;

[0141] Lip smacking sound NZZ: precision = 41.74%, recall = 57.83%;

[0142] Drag sound TY: precision=22.14%, recall=51.79%.

[0143] The current interval judgment standard is stricter than the actual situation. In the WenetSpeech test data, 100 audio samples were manually sampled for listening. The results showed that all the inhalation sounds NXQ and laughter sounds NXS that can be perceived by the human ear were hit by this application. However, there were two false positives. The dragged sound TY and the lip smacking sound NZZ were less common in the extracted audio, but the precision was above 80%, which basically met the usage requirements.

[0144] Corresponding to the method embodiment, the present application also provides a paralanguage annotation device. A structural diagram of the paralanguage annotation device provided in the embodiment of the present application is as follows: Figure 6 As shown, this may include:

[0145] Detection module 601, correction module 602 and labeling module 603;

[0146] The detection module 601 is configured to perform paralanguage detection on the target speech data to obtain an initial detection result; the initial detection result includes: whether each speech frame of the target speech data belongs to a paralanguage, and the paralanguage category if it belongs to a paralanguage;

[0147] The correction module 602 is used to correct the initial detection result according to the length of the paralanguage segment and the interval between the paralanguage segments to obtain a target detection result; wherein, consecutive speech frames belonging to the same paralanguage category constitute a paralanguage segment;

[0148] The annotation module 603 is configured to perform paralinguistic annotation on the text corresponding to the target speech data based on the target detection result.

[0149] The paralanguage annotation device provided in an embodiment of the present application performs paralanguage detection on target speech data to obtain an initial detection result. The initial detection result includes: whether each speech frame of the target speech data belongs to a paralanguage, and the paralanguage category if it does belong to a paralanguage; based on the length of a paralanguage segment consisting of consecutive speech frames belonging to the same paralanguage category and the interval between paralanguage segments, the initial detection result is corrected to obtain a target detection result; and based on the target detection result, the text corresponding to the target speech data is annotated with the paralanguage. Based on this application, the purpose of automatically annotating text with paralanguage is achieved, the efficiency of paralanguage annotation is improved, and the cost of paralanguage annotation is reduced.

[0150] In an optional embodiment, when performing paralanguage detection on the target speech data, the detection module 601 is configured to:

[0151] Encoding each speech frame of the target speech data to obtain encoding features of each speech frame;

[0152] Classifying each speech frame based on the coding features of each speech frame to obtain the initial detection result;

[0153] In an optional embodiment, the detection module 601 classifies each speech frame based on the coding features of each speech frame, and when obtaining the initial detection result, is used to:

[0154] Classifying and predicting each speech frame based on its coding features to obtain a probability distribution for each speech frame; the probability distribution for each speech frame is the probability of the speech frame corresponding to each category; the categories include: a non-paralanguage category and multiple paralanguage categories;

[0155] The initial category of each speech frame is determined in turn according to the probability distribution of each speech frame; wherein:

[0156] If the maximum probability in the probability distribution of the first speech frame is greater than a first threshold, determining the initial category of the first speech frame to be the category corresponding to the maximum probability; otherwise, determining the initial category of the first speech frame to be a non-paralanguage category;

[0157] For each non-first speech frame, the initial category of the non-first speech frame is determined according to the initial category of the previous speech frame of the non-first speech frame and the probability distribution of the non-first speech frame.

[0158] In an optional embodiment, when the detection module 601 determines the initial category of a non-first speech frame, it is configured to:

[0159] If the initial category of the previous speech frame of the non-first speech frame is the non-paralanguage category or the previous speech frame is the last frame of any paralanguage category, and the maximum probability in the probability distribution of the non-first speech frame is greater than a first threshold, determining the initial category of the non-first speech frame to be the category corresponding to the maximum probability; otherwise, determining the initial category of the non-first speech frame to be the non-paralanguage category;

[0160] If the initial category of the previous speech frame of the non-first speech frame is any paralanguage category, the previous speech frame is not the last frame of the any paralanguage category, and the probability corresponding to the any paralanguage category in the probability distribution of the non-first speech frame is greater than a second threshold, it is determined that the initial category of the non-first speech frame is the any paralanguage category and is not the last frame of the any paralanguage category; if the probability corresponding to the any paralanguage category in the probability distribution of the non-first speech frame is less than or equal to the second threshold, it is determined that the initial category of the non-first speech frame is the any paralanguage category and is the last frame of the any paralanguage category; the second threshold is greater than the first threshold.

[0161] In an optional embodiment, when the correction module 602 corrects the initial detection result based on the length of the paralanguage segment and the interval between the paralanguage segments, it is configured to:

[0162] If the speech frames in two paralanguage segments belong to the same paralanguage category, and the number of speech frames between the two paralanguage segments is less than a first preset number, the categories to which the speech frames between the two paralanguage segments belong are updated to the paralanguage categories to which the speech frames in the two paralanguage segments belong, and the speech frames between the two paralanguage segments and the two paralanguage segments are merged into one paralanguage segment.

[0163] In an optional embodiment, the paralanguage categories include a plurality of determined paralanguage categories and indeterminate paralanguage categories; and the correction module 602 corrects the initial detection result based on the length of the paralanguage segments and the intervals between the paralanguage segments, including at least one of the following:

[0164] If the speech frames in two paralanguage segments belong to the same paralanguage category, and the number of speech frames between the two paralanguage segments is less than a first preset number, updating the category of the speech frames between the two paralanguage segments to the paralanguage category of the speech frames in the two paralanguage segments, and merging the speech frames between the two paralanguage segments and the two paralanguage segments into one paralanguage segment;

[0165] If the speech frames in a first paralanguage segment of two paralanguage segments belong to a determined first paralanguage category, and the speech frames in a second paralanguage segment belong to an uncertain paralanguage category, the number of speech frames contained in the second paralanguage segment is less than a second preset number, and the number of speech frames located between the two paralanguage segments is less than the first preset number, then the categories to which the speech frames in the second paralanguage segment and the speech frames located between the two paralanguage segments belong are updated to the first paralanguage category, and the speech frames located between the two paralanguage segments and the two paralanguage segments are merged into one paralanguage segment.

[0166] In an optional embodiment, the correction module 602 is further configured to:

[0167] The paralanguage segment including a number of speech frames less than a third preset number is modified into a non-paralanguage segment.

[0168] In an optional embodiment, when the annotation module 603 performs paralinguistic annotation on the text corresponding to the target speech data based on the target detection result, it is configured to:

[0169] For each paralanguage segment, based on the target temporal relationship between each speech frame in the paralanguage segment and the speech frames corresponding to each word in the text, a paralanguage tag corresponding to the paralanguage segment is added to the text;

[0170] The paralanguage tag represents the paralanguage category to which any paralanguage segment belongs and its start time and end time; and the temporal relationship between the paralanguage tag and each word in the text is the target temporal relationship.

[0171] An electronic device is also provided in an embodiment of the present application. Figure 7 , which shows a schematic diagram of a structure of an electronic device suitable for implementing the embodiments of the present application. The electronic device in the embodiments of the present application can be a terminal device (such as a mobile phone, a tablet computer, a laptop computer, a desktop computer, etc.), or a server (which can be a single server, a server cluster, or a cloud server, etc.). Figure 7 The electronic device shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0172] like Figure 7 As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 701, which can perform various appropriate actions and processes based on programs stored in a read-only memory (ROM) 702 or programs loaded from a storage device 708 into a random access memory (RAM) 703. When the electronic device is powered on, the RAM 703 also stores various programs and data required for the operation of the electronic device. The processing device 701, ROM 702, and RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0173] Typically, the following devices may be connected to the I / O interface 705: an input device 706 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 707 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 708 including, for example, a memory card, a hard disk, etc.; and a communication device 709. The communication device 709 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Figure 7 The electronic device is shown with various devices, but it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed instead.

[0174] An embodiment of the present application further provides a computer program product including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any of the paralanguage annotation methods provided in the embodiments of the present application.

[0175] An embodiment of the present application further provides a computer-readable storage medium carrying one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any of the paralanguage annotation methods provided in the embodiments of the present application.

[0176] It should be noted that the device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. In addition, in the drawings of the device embodiments provided in this application, the connection relationship between the modules indicates that there is a communication connection between them, which can be specifically implemented as one or more communication buses or signal lines.

[0177] Through the description of the above embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software plus necessary general-purpose hardware, and of course can also be implemented by dedicated hardware including application-specific integrated circuits, dedicated CPUs, dedicated memories, dedicated components, etc. In general, all functions performed by computer programs can be easily implemented with corresponding hardware, and the specific hardware structures used to implement the same function can also be diverse, such as analog circuits, digital circuits, or dedicated circuits. However, for the present application, software program implementation is a better implementation method in most cases. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer's floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk or optical disk, etc., and includes a number of instructions to enable a computer device (which can be a personal computer, training equipment, or network equipment, etc.) to execute the methods described in each embodiment of the present application.

[0178] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments may be implemented in the form of a computer program product. Professionals and technicians may use different methods to implement the described functions for each specific solution, but such implementation should not be considered beyond the scope of this application.

[0179] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, a computer, a training device or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website, a computer, a training device or a data center. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium can be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0180] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0181] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present application. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A paralanguage annotation method, characterized in that: The method comprises: Encoding each speech frame of the target speech data to obtain encoding features of each speech frame; Classifying each speech frame based on the encoding features of each speech frame to obtain an initial detection result; the initial detection result includes: whether each speech frame of the target speech data belongs to a paralanguage, and the paralanguage category if it belongs to a paralanguage; classifying each speech frame based on the encoding features of each speech frame includes: classifying and predicting each speech frame based on the encoding features of each speech frame to obtain a probability distribution for each speech frame; the probability distribution of each speech frame is the probability of the speech frame corresponding to each category; the categories include: a non-paralanguage category and multiple paralanguage categories; determining the initial category of each speech frame in turn based on the probability distribution of each speech frame; wherein: if the maximum probability in the probability distribution of the first speech frame is greater than a first threshold, determining the initial category of the first speech frame to be the category corresponding to the maximum probability; otherwise, determining the initial category of the first speech frame to be the non-paralanguage category; for each non-first speech frame, determining the initial category of the non-first speech frame based on the initial category of the previous speech frame of the non-first speech frame and the probability distribution of the non-first speech frame; The initial detection result is modified according to the length of the paralanguage segment and the interval between the paralanguage segments to obtain a target detection result; wherein consecutive speech frames belonging to the same paralanguage category constitute a paralanguage segment; Perform paralinguistic annotation on text corresponding to the target speech data based on the target detection result.

2. The method according to claim 1, characterized in that The process of determining the initial category of non-first speech frames includes: If the initial category of the previous speech frame of the non-first speech frame is the non-paralanguage category or the previous speech frame is the last frame of any paralanguage category, and the maximum probability in the probability distribution of the non-first speech frame is greater than a first threshold, determining the initial category of the non-first speech frame to be the category corresponding to the maximum probability; otherwise, determining the initial category of the non-first speech frame to be the non-paralanguage category; If the initial category of the previous speech frame of the non-first speech frame is any paralanguage category, the previous speech frame is not the last frame of the any paralanguage category, and the probability corresponding to the any paralanguage category in the probability distribution of the non-first speech frame is greater than a second threshold, it is determined that the initial category of the non-first speech frame is the any paralanguage category and is not the last frame of the any paralanguage category; if the probability corresponding to the any paralanguage category in the probability distribution of the non-first speech frame is less than or equal to the second threshold, it is determined that the initial category of the non-first speech frame is the any paralanguage category and is the last frame of the any paralanguage category; the second threshold is greater than the first threshold.

3. The method according to claim 1, characterized in that The modifying of the initial detection result according to the length of the paralanguage segment and the interval between the paralanguage segments includes: If the speech frames in two paralanguage segments belong to the same paralanguage category, and the number of speech frames between the two paralanguage segments is less than a first preset number, the categories to which the speech frames between the two paralanguage segments belong are updated to the paralanguage categories to which the speech frames in the two paralanguage segments belong, and the speech frames between the two paralanguage segments and the two paralanguage segments are merged into one paralanguage segment.

4. The method according to claim 1, wherein The paralanguage categories include a plurality of determined paralanguage categories and uncertain paralanguage categories; and the modifying of the initial detection result according to the length of the paralanguage segments and the intervals between the paralanguage segments includes at least one of the following: If the speech frames in two paralanguage segments belong to the same paralanguage category, and the number of speech frames between the two paralanguage segments is less than a first preset number, updating the category of the speech frames between the two paralanguage segments to the paralanguage category of the speech frames in the two paralanguage segments, and merging the speech frames between the two paralanguage segments and the two paralanguage segments into one paralanguage segment; If the speech frames in a first paralanguage segment of two paralanguage segments belong to a determined first paralanguage category, and the speech frames in a second paralanguage segment belong to an uncertain paralanguage category, the number of speech frames contained in the second paralanguage segment is less than a second preset number, and the number of speech frames located between the two paralanguage segments is less than the first preset number, then the categories to which the speech frames in the second paralanguage segment and the speech frames located between the two paralanguage segments belong are updated to the first paralanguage category, and the speech frames located between the two paralanguage segments and the two paralanguage segments are merged into one paralanguage segment.

5. The method according to claim 3 or 4, characterized in that Also includes: The paralanguage segment including a number of speech frames less than a third preset number is modified into a non-paralanguage segment.

6. The method according to claim 1, characterized in that The performing paralinguistic tagging on the text corresponding to the target speech data based on the target detection result includes: For each paralanguage segment, based on the target temporal relationship between each speech frame in the paralanguage segment and the speech frames corresponding to each word in the text, a paralanguage tag corresponding to the paralanguage segment is added to the text; The paralanguage tag represents the paralanguage category to which any paralanguage segment belongs and its start time and end time; and the temporal relationship between the paralanguage tag and each word in the text is the target temporal relationship.

7. A paralanguage annotation device, characterized in that: include: A detection module, configured to encode each speech frame of the target speech data to obtain encoding features of each speech frame; Classify each speech frame based on its coding features to obtain an initial detection result; The initial detection result includes: whether each speech frame of the target speech data belongs to a paralanguage, and the paralanguage category if it belongs to a paralanguage; when the detection module classifies each speech frame based on the encoding features of each speech frame, it is used to: classify and predict each speech frame based on the encoding features of each speech frame to obtain a probability distribution for each speech frame; the probability distribution of each speech frame is the probability of the speech frame corresponding to each category; the categories include: non-paralanguage category and multiple paralanguage categories; the initial category of each speech frame is determined in turn based on the probability distribution of each speech frame; wherein: if the maximum probability in the probability distribution of the first speech frame is greater than a first threshold, the initial category of the first speech frame is determined to be the category corresponding to the maximum probability; otherwise, the initial category of the first speech frame is determined to be the non-paralanguage category; for each non-first speech frame, the initial category of the non-first speech frame is determined based on the initial category of the previous speech frame of the non-first speech frame and the probability distribution of the non-first speech frame; a correction module, configured to correct the initial detection result according to the length of the paralanguage segment and the interval between the paralanguage segments to obtain a target detection result; wherein consecutive speech frames belonging to the same paralanguage category constitute a paralanguage segment; The tagging module is configured to perform paralinguistic tagging on the text corresponding to the target speech data based on the target detection result.

8. A computer program product, characterized in that The method comprises computer-readable instructions, which, when executed on an electronic device, enable the electronic device to implement the paralanguage annotation method according to any one of claims 1 to 6.

9. An electronic device, characterized in that: The electronic device comprises at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is configured to execute the computer program so that the electronic device can implement the paralanguage annotation method according to any one of claims 1 to 6.

10. A computer storage medium, characterized in that The storage medium carries one or more computer programs, and when the one or more computer programs are executed by an electronic device, the electronic device can implement the paralanguage annotation method according to any one of claims 1 to 6.

Images