Method for constructing text data set based on dialogue recording

Abstract

The application discloses a dialogue recording-based text data set construction method, comprising the following steps: S1, obtaining a dialogue recording set, and sequentially taking out a dialogue recording to be processed from the dialogue recording set; S2, extracting a dialogue person feature vector set and a dialogue person number from the dialogue recording to be processed; S3, separating effective dialogue recordings to form single-person recordings and put the single-person recordings into a single-person recording set; and S4, converting the single-person recordings into recording texts to form a recording text set. In the case that the dialogue person number in the dialogue recording is unknown and clustering is not performed, a high-quality dialogue person feature vector set and a more accurate dialogue person number are simultaneously obtained, low-quality recording samples with a dialogue person number exceeding a maximum dialogue person number are removed, and speech with a large number of overlapping different dialogue persons is separated, so that noise is reduced when the data set is established, artificial efficiency is improved, and the quality of the dialogue text data set is improved.

Description

Technical Field

[0001] This invention belongs to the field of computer technology, and more specifically, this invention relates to a method for constructing a speech text dataset based on dialogue recordings. Background Technology

[0002] In the wave of the internet, more and more companies in both traditional and non-traditional sectors are beginning to prioritize the development of online platforms. Among the various types of online platforms, scenarios involving dialogues between two or more people, such as product consultations, after-sales service, and online meetings, are not uncommon. As the user base continues to expand into lower-tier cities and rural areas, text-based dialogues are becoming increasingly insufficient to meet business needs, leading more businesses to adopt voice-based dialogue modes.

[0003] As information intelligence continues to deepen, many enterprises are introducing intelligent robots to provide customers with more timely, standardized, and high-quality services. However, these intelligent robots have encountered some difficulties when it comes to voice dialogue. Due to the difficulty in processing speech, most traditional natural language processing research is based on text; and due to the sensitivity of speech data, there are relatively few multi-person dialogue speech datasets available on the market.

[0004] Due to the "cocktail party problem," the creation of multi-person dialogue text datasets with significant speech overlap often requires entirely human intervention. The lack of both suitable datasets and processing methods has hindered the development of methods directly applicable to dialogue speech processing. Summary of the Invention

[0005] This invention provides a method for constructing a speech text dataset based on dialogue recordings, aiming to improve the above-mentioned problems.

[0006] This invention is implemented as follows: a method for constructing a speech text dataset based on dialogue recordings, the method specifically including the following steps:

[0007] S1. Obtain the dialogue recording set, and retrieve the dialogue recordings to be processed from the dialogue recording set in turn;

[0008] S2. Extract the set of feature vectors of the speakers and the number of speakers from the dialogue recording to be processed;

[0009] S3. Separate the valid dialogue recordings into individual recordings and put them into the individual recording collection;

[0010] S4. Convert individual recordings into audio text to form a collection of audio texts.

[0011] Furthermore, step S2 specifically includes the following steps:

[0012] Obtain the dialogue recording frame matrix E of the dialogue recording to be processed. The dialogue recording frame matrix E is composed of dialogue recording frames.

[0013] Input the dialogue recording frame matrix E into the dialogue person encoding neural network to obtain the dialogue recording embedding vector matrix Z.

[0014] The dialogue recording is embedded into a vector matrix Z and input into the dialogue decoding neural network to obtain the dialogue feature vector matrix V and the number of dialogues S.

[0015] Furthermore, the interlocutor encoding neural network consists of A layers, and the calculation formula for layer a is as follows:

[0016]

[0017]

[0018]

[0019] Where Pos is the position code. For the parameter matrix, Let d be the parameter vector. H Here, LN is the scaling factor, GLU is the layer normalization operation, sigmoid is the sigmoid function, and transpose is the matrix transpose operation. The voice recording embedding vector matrix Z = transpose(Z0). A ).

[0020] Furthermore, the interlocutor decoding neural network consists of B layers, and the calculation formula for the b-th layer in the s-th decoding process is as follows:

[0021]

[0022]

[0023]

[0024]

[0025] Among them, Pos, Encoding the fragment, For the parameter matrix, Let d be the parameter vector. Z [SEP] is the scaling factor, and [SEP] is the delimiter encoding. Decoding ends when the network outputs the end-of-line character encoding [EOS], and at this point, S = s - 2.

[0026] Furthermore, the specific methods for obtaining effective conversation recordings are as follows:

[0027] The number of speakers in each recorded conversation, S, is compared with the preset maximum number of speakers, S0. max Compare, if S>S max If the condition is met, the recorded dialogue is marked as invalid; otherwise, it is marked as valid and added to the valid dialogue recording set.

[0028] Furthermore, the specific method for separating valid dialogue recordings is as follows:

[0029] The speech frame tensor U of the effective dialogue recording O and the dialogue recording embedding vector matrix Z are used to... O The feature vector of the s-th interlocutor Input a dual-path neural network and output the recording of the s-th person in the conversation. Add to the individual recording collection.

[0030] Furthermore, the dual-path neural network consists of an inner neural subnetwork and an outer neural subnetwork, where the output of the inner neural subnetwork serves as the input of the outer neural subnetwork. The calculation formula for the i-th layer of the inner neural subnetwork is as follows:

[0031]

[0032] The formula for calculating the j-th layer of the outer neural subnetwork is as follows:

[0033]

[0034]

[0035]

[0036] In the formula, N innerlayer N outerlayer d represents the number of layers in the inner and outer neural subnetworks, respectively. base It is the base number of voids. For the parameter matrix, For parameter vectors.

[0037] Furthermore, the process of forming the audio transcript set is as follows:

[0038] Take any single recording from the single recording set, convert the single recording into a corresponding audio text; put the audio text into the audio text set, and delete the single recording from the single recording set; repeat the above steps until the single recording set is empty.

[0039] Furthermore, the process of obtaining the dialogue recording frame matrix E is as follows:

[0040] The dialogue recording to be processed is X = [x1, x2, ..., x...].T ]∈R 1×T Input formula (1) to obtain the dialogue recording frame matrix. Where T is the recording length, C EMB Where N is the dimension of the recording frame and N is the number of recording frames, as shown in formula (1) below:

[0041] E = ReLU(Conv1d1(X)) (1)

[0042] Where ReLU is a linear rectified function, and Conv1d1 is a function with 1 input channel and C output channels. EMB One-dimensional convolution operation.

[0043] The method for constructing a dialogue text dataset based on dialogue recordings proposed in this invention obtains a high-quality set of dialogue feature vectors and a more accurate number of dialogue participants without knowing the number of participants in the dialogue recordings or performing clustering. Based on this, low-quality recording samples with more than the maximum number of participants are removed, and speech from different participants with a large amount of overlap is separated. This reduces noise, improves manual efficiency, and enhances the quality of the dialogue text dataset when building the dataset. Attached Figure Description

[0044] Figure 1 A flowchart illustrating a method for constructing a speech text dataset based on dialogue recordings, as provided in an embodiment of the present invention. Detailed Implementation

[0045] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings, so as to help those skilled in the art to have a more complete, accurate and in-depth understanding of the inventive concept and technical solution of the present invention.

[0046] The method for constructing a dialogue text dataset based on dialogue recordings proposed in this invention obtains a high-quality set of dialogue feature vectors and a more accurate number of dialogue participants without knowing the number of participants in the dialogue recordings or performing clustering. Based on this, low-quality recording samples with more than the maximum number of participants are removed, and speech from different participants with a large amount of overlap is separated. This reduces noise, improves manual efficiency, and enhances the quality of the dialogue text dataset when building the dataset.

[0047] Figure 1 A flowchart of a method for constructing a speech text dataset based on dialogue recordings provided in an embodiment of the present invention. The method specifically includes the following steps:

[0048] Step 1: Obtain the dialogue recording set, and then retrieve the dialogue recordings to be processed from the dialogue recording set one by one;

[0049] Acquire dialogue recordings from transducer devices, storage media, or transmission devices, place the acquired dialogue recordings into a dialogue recording set in sequence, and then retrieve one dialogue recording to be processed from the dialogue recording set in sequence.

[0050] Step 2: Extract the set of feature vectors of the speakers and the number of speakers from the dialogue recording to be processed;

[0051] The set of speaker feature vectors contains the feature vectors of all speakers in the dialogue recording to be processed. Each speaker's feature vector contains their speech features and is used to uniquely identify each speaker. The dialogue recording frame matrix is ​​obtained and input into the speaker feature extraction network to obtain the speaker feature vectors and the number of speakers.

[0052] Sub-step 1: Record the dialogue to be processed X = [x1, x2, ..., x T ]∈R 1×T Input formula (1) to obtain a dialogue recording frame matrix composed of dialogue recording frames. Where T is the recording length, C EMB Where N is the dimension of the recording frame, and N is the number of recording frames. Formula (1) is as follows:

[0053] E = ReLU(Conv1d1(X)) (1)

[0054] Where ReLU is a linear rectified function, and Conv1d1 is a function with 1 input channel and C output channels. EMB One-dimensional convolution operation;

[0055] Input the dialogue recording frame matrix E into the dialogue user encoding neural network to obtain the dialogue recording embedding vector matrix. The dialogue recording is embedded into a vector matrix and input into the dialogue decoder neural network to obtain the dialogue feature vector corresponding to the dialogue recording to be processed.

[0056] Sub-step 2: Input the dialogue recording frame matrix E into the dialogue user feature extraction neural network to obtain the dialogue user feature vector matrix. Where S represents the number of people in the conversation.

[0057] Each column vector in the interlocutor feature vector matrix V The dialogue feature vectors are the feature vectors of different speakers in the dialogue recording, and these feature vectors form the dialogue feature vector set V. set ={v s |s=1,2,...,S}.

[0058] The dialogue user feature extraction neural network consists of a dialogue user encoding neural network and a dialogue user decoding neural network. The dialogue recording frame matrix E is input into the dialogue user encoding neural network to obtain the dialogue recording embedding vector matrix Z, which is composed of dialogue recording embedding vectors. The calculation formula for the a-th layer (1≤a≤A) in the dialogue user encoding neural network, which consists of a total of A layers, is as follows:

[0059]

[0060]

[0061]

[0062] Where Pos is the position code. For the parameter matrix, Let d be the parameter vector. H Here, LN is the scaling factor, GLU is the layer normalization operation, sigmoid is the sigmoid function, and transpose is the matrix transpose operation. The voice recording embedding vector matrix Z = transpose(Z0). A ).

[0063] The dialogue recording is embedded into a vector matrix Z and input into a dialogue decoding neural network consisting of B layers. An autoregressive decoding method is then used to obtain the dialogue feature vector matrix. Where S represents the number of speakers obtained from decoding, and each column vector in the speaker feature vector matrix V represents a speaker. These are the feature vectors of different speakers, and the set of speaker feature vectors V is composed of these feature vectors. set ={v s |s=1,2,...,S}. Before the decoding ends, S is unknown, and the b-th layer (1≤b≤B) is obtained in the s-th decoding (1≤s≤S+2). The formula for calculating time is as follows:

[0064]

[0065]

[0066]

[0067]

[0068] Among them, Pos, Encoding the fragment, For the parameter matrix, Let d be the parameter vector. Z[SEP] is the scaling factor, and [SEP] is the delimiter encoding. Decoding ends when the network outputs the end-of-line character encoding [EOS], and at this point, S = s - 2.

[0069] Step 3: Mark the recorded conversation

[0070] For each dialogue recording in the collection, perform steps 1 to 3 above, and obtain the number of speakers S in each dialogue recording. Then, compare S with the preset maximum number of speakers S0. max Compare, if S>S max If the dialogue recording is invalid, it is marked as invalid; otherwise, it is marked as valid, invalid dialogue recordings are discarded, and valid dialogue recordings are added to the set of valid dialogue recordings.

[0071] Step 4: Separation of the interlocutors;

[0072] Dialogue separation is based on valid dialogue recordings, separating portions of the dialogues from different speakers into individual recordings, ultimately forming a set of individual recordings. This invention converts valid dialogue recordings into speech frame tensors. Based on the speech frame tensors, the dialogue recording embedding vector matrix Z, and the speaker feature vector set V, a dual-path neural network is used to process the data to obtain the set of individual recordings.

[0073] Extract valid dialogue recordings O sequentially from the set of valid dialogue recordings, and perform sub-steps 1 to 4 for each valid dialogue recording O;

[0074] Sub-step 1 records the valid dialogues O = [o1, o2, ..., o T ]∈R 1×T Input formula (2) to obtain the speech frame tensor. Formula (2) is as follows:

[0075] U=PReLU(Conv1d2(O)) (2)

[0076] Where PReLU is a parameterized linear rectifier unit, and Conv1d2 is a rectifier unit with 1 input channel and 1 output channel. CSPL For the one-dimensional convolution operation, Conv1d2 has the same window length and stride as Conv1d1, T is the time length, and T' is the dimension of the second dimension of the tensor, which is essentially the number of speech frames.

[0077] Sub-step 2 initializes the single-person recording set to an empty set;

[0078] Sub-step 3 involves embedding the speech frame tensor U of the effective dialogue recording O and the dialogue recording into the vector matrix Z. O The feature vector of the s-th interlocutor Input a dual-path neural network and output the recording of the s-th person in the conversation. Add to the individual recording collection.

[0079] A dual-path neural network consists of an inner neural subnetwork and an outer neural subnetwork. The output of the inner neural subnetwork serves as the input to the outer neural subnetwork. The calculation formula for the i-th layer of the inner neural subnetwork is as follows:

[0080]

[0081] The formula for calculating the j-th layer of the outer neural subnetwork is as follows:

[0082]

[0083]

[0084]

[0085] In the formula, N innerlayer N outerlayer d represents the number of layers in the inner and outer neural subnetworks, respectively. base It is the base number of voids. For the parameter matrix, Let be the parameter vector. The IPathLayer function is calculated as follows:

[0086]

[0087] In the formula, BN is the batch normalization function, and GLU is the Gaussian error linear unit. The window length is 3, the step size is 1, and the number of input channels is C. SPL The number of output channels is 2C. SPL One-dimensional convolution operation, The window length is 3, the step size is 1, and the number of input channels is C. EMB The number of output channels is 2C. SPL One-dimensional convolution operation, It is the base number of voids, d. base The i-1th power, It has a window length of 3, a step size of 1, and 2C input channels. SPL The number of output channels is 2C. SPL The void coefficient is Depth-separable convolution operations, It has a window length of 3, a step size of 1, and 2C input channels. SPL The number of output channels is C SPL One-dimensional convolution operation,

[0088] The OPathLayer function is as follows:

[0089]

[0090] In the formula, The window length is 3, the step size is 1, and the number of input channels is C. SPL The number of output channels is C SPL One-dimensional convolution operation, The window length is 3, the step size is 1, and the number of input channels is C. EMB The number of output channels is C SPL One-dimensional convolution operation, LConvld OP The window length is 201, the step size is 1, and the number of input channels is [missing information]. Number of output channels Depth-separable convolution operations, The window length is 3, the step size is 1, and the number of input channels is [missing information]. The number of output channels is C SPL One-dimensional convolution operation,

[0091] The Inv function is as follows:

[0092] Inv(x,y)=ConvTrld(x⊙y)

[0093] In the formula, ConvTr1d is the number of input channels C. SPL The one-dimensional transpose convolution operation with 1 output channel has the same window length and stride as Conv1d1 and Conv1d2, and its ⊙ is the Hadamard product.

[0094] Sub-step 4 repeats sub-step 3 starting from s=1 until all speakers in the valid dialogue recording O have been processed.

[0095] Step 5: Dialogue speech recognition;

[0096] The collection of individual audio recordings is converted into a collection of audio texts through speech recognition. All the audio texts constitute the dialogue text dataset. The process of constructing the dialogue text dataset is as follows:

[0097] If the set of recorded texts does not exist in sub-step 1, then create an empty set as the set of recorded texts.

[0098] Sub-step 2: Take any single recording from the single recording set and convert the single recording into the corresponding recording text;

[0099] Sub-step 3: Add the audio text to the audio text collection and delete the individual audio recording from the individual audio recording collection;

[0100] Sub-step 4 repeats the above steps until the single-person recording set is empty.

[0101] Step 6: Create a dialogue text dataset;

[0102] All audio recordings in the entire dataset undergo manual verification, text cleaning, and anonymization to obtain the dialogue text dataset. Further text cleaning includes removing interjections, segmenting long sentences based on punctuation, word segmentation, and named entity recognition. Anonymization includes removing names of people, places, companies, phone numbers, ID numbers, specific dates, and payment amounts from the text.

[0103] The present invention has been described by way of example. Obviously, the specific implementation of the present invention is not limited to the above-described manner. Any non-substantial improvements made using the inventive concept and technical solution of the present invention, or the direct application of the inventive concept and technical solution of the present invention to other situations without modification, are all within the protection scope of the present invention.

Claims

1. A method for constructing a single-person text dataset based on dialogue recordings, characterized in that, The method specifically includes the following steps: S1. Obtain the dialogue recording set, and retrieve the dialogue recordings to be processed from the dialogue recording set in turn; S2. Extract the set of feature vectors of the speakers and the number of speakers from the dialogue recording to be processed; S3. Separate the valid dialogue recordings into individual recordings and put them into the individual recording collection; S4. Convert individual recordings into audio transcripts to form a collection of audio transcripts; Step S2 specifically includes the following steps: Obtain the dialogue recording frame matrix of the dialogue recording to be processed. Dialogue recording frame matrix Composed of spoken recording frames; Dialogue recording frame matrix Input the dialogue speaker encoding neural network to obtain the dialogue recording embedding vector matrix. ; Embedding dialogue recordings into a vector matrix Input the interlocutor into the decoding neural network to obtain the interlocutor feature vector matrix. and the number of people in the conversation, S; The interlocutor encoding neural network consists of A layers. The calculation formula for the a-th layer is as follows: ； ； ； in, For position encoding, , , , , , For the parameter matrix, For parameter vectors, The scaling factor is LN, the layer normalization operation is GLU, the gated linear unit is sigmoid, the sigmoid function is transpose, and the voice recording embedding vector matrix is ​​used. ; The interlocutor decoding neural network consists of B layers. The calculation formula for the b-th layer in the s-th decoding process is as follows: ； ； ； ； in, , , Encoding the fragment, , , , , For the parameter matrix, , For parameter vectors, This is the scaling factor. For delimiter encoding, decoding ends when the network outputs the end-of-line character encoded as [EOS], and at this point, S = s - 2. ; The specific method for separating valid dialogue recordings is as follows: Record effective conversations speech frame tensor Dialogue recording embedding vector matrix The feature vector of the s-th interlocutor Input a dual-path neural network and output the recording of the s-th person in the conversation. Add it to the individual recording collection; A dual-path neural network consists of an inner neural subnetwork and an outer neural subnetwork. The output of the inner neural subnetwork serves as the input to the outer neural subnetwork. The calculation formula for the i-th layer of the inner neural subnetwork is as follows: ； The formula for calculating the j-th layer of the outer neural subnetwork is as follows: ； ； ； In the formula, , These represent the number of layers in the inner and outer neural subnetworks, respectively. It is the base number of voids. For the parameter matrix, For parameter vectors.

2. The method for constructing a single-person text dataset based on dialogue recordings as described in claim 1, characterized in that, The specific methods for obtaining valid conversation recordings are as follows: The number of speakers in each recorded conversation, S, is compared with the preset maximum number of speakers, S0. max If S > S max If the condition is met, the recorded dialogue is marked as invalid; otherwise, it is marked as valid and added to the valid dialogue recording set.

3. The method for constructing a single-person text dataset based on dialogue recordings as described in claim 1, characterized in that, The process of forming the audio transcript collection is as follows: Take any single recording from the single recording set, convert the single recording into a corresponding audio text; put the audio text into the audio text set, and delete the single recording from the single recording set; repeat the above steps until the single recording set is empty.

4. The method for constructing a single-person text dataset based on dialogue recordings as described in claim 2, characterized in that, Dialogue recording frame matrix The specific process for obtaining it is as follows: Record the conversation to be processed Input formula (1) to obtain the dialogue recording frame matrix. Where T is the recording length, Where N is the dimension of the recording frame and N is the number of recording frames, as shown in formula (1) below: （1） Where ReLU is a linear rectified function. For input channels with 1 and output channels with 1 One-dimensional convolution operation.

Images