Data processing method and device and electronic equipment

Abstract

The embodiment of the invention provides a data processing method and device and electronic equipment. The method comprises the following steps: receiving each piece of sample data in a first training set through an audio quantizer, and outputting a quantization sequence corresponding to each piece of sample data; the quantization sequence corresponding to one piece of sample data comprises a classification result corresponding to each voice segment obtained by segmenting the piece of sample data; selecting a quantization sequence corresponding to one piece of positive sample data and a quantization sequence corresponding to one piece of negative sample data to serve as existing sequences to be input into the large language model, and outputting a confused target quantization sequence; obtaining a first word vector sequence corresponding to the target quantization sequence by querying the quantization dictionary; and receiving the first word vector sequence through an audio inverse quantizer, and outputting target audio data corresponding to the first word vector sequence. According to the embodiment of the invention, the data processing time in the negative sample data acquisition process can be shortened, and the cost and period of collecting the negative sample data are reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of speech recognition, and in particular to a data processing method and device and electronic equipment. BACKGROUND

[0002] With the development of artificial intelligence, more and more electronic devices support voice interaction function. Voice wake-up, as a switch of voice interaction, occupies an important part.

[0003] The quality of the voice classification model determines the accuracy of voice wake-up. The difficulty of training a high-quality voice classification model lies in collecting negative sample data. To solve the problem of difficult collection of negative sample data, the following steps can be used to train the voice classification model:

[0004] 1. Collect positive sample data and negative sample data to construct a training set;

[0005] 2. Use the training set to train an initial version of the voice classification model;

[0006] 3. Put the voice classification model into use and collect false wake-up data output by the voice classification model as negative sample data;

[0007] 4. After a period of time, add the collected negative sample data to the training set to train the next version of the voice classification model;

[0008] Repeat steps 3 and 4 to eventually obtain a voice classification model with an accuracy that meets the preset requirements.

[0009] The cycle of steps 3 and 4 is long, which requires a lot of time, resulting in low efficiency of model training. In addition, in the case of a change in the wake-up word, the above steps 1-4 need to be re-executed to train a new voice classification model, resulting in high time cost of the entire development process and long product iteration cycle. SUMMARY

[0010] The embodiments of the present application provide a data processing method, device and electronic equipment, which can shorten the data processing time in the process of obtaining negative sample data and reduce the cost and cycle of collecting negative sample data.

[0011] In a first aspect, the embodiments of the present application disclose a data processing method, which comprises:

[0012] Each sample data in a first training set is received by an audio quantizer, and a quantization sequence corresponding to each sample data is output; wherein the quantization sequence corresponding to each sample data includes the classification result corresponding to each voice segment obtained by dividing the sample data; the first training set includes positive sample data and negative sample data, and the positive sample data and the negative sample data are audio data;

[0013] select a quantization sequence corresponding to a positive sample data and a quantization sequence corresponding to a negative sample data as an existing sequence to input a large language model, and output a target quantization sequence confused with the existing sequence through the large language model;

[0014] obtain a first word vector sequence corresponding to the target quantization sequence by querying a quantization dictionary; the quantization dictionary includes a mapping relationship between a word vector and a classification result of a speech segment;

[0015] receive the first word vector sequence through an audio dequantizer, and output target audio data corresponding to the first word vector sequence.

[0016] In a second aspect, an embodiment of the present application discloses a data processing device, and the device comprises:

[0017] an audio quantizer configured to receive each sample data in a first training set and output a quantization sequence corresponding to the sample data; wherein the quantization sequence corresponding to a sample data includes a classification result corresponding to each speech segment obtained by segmenting the sample data; the first training set includes positive sample data and negative sample data, and the positive sample data and the negative sample data are audio data;

[0018] a large language model configured to receive an existing sequence and output a target quantization sequence confused with the existing sequence; the existing sequence includes a quantization sequence corresponding to a positive sample data and a quantization sequence corresponding to a negative sample data;

[0019] a dictionary query module configured to obtain a first word vector sequence corresponding to the target quantization sequence by querying a quantization dictionary; the quantization dictionary includes a mapping relationship between a word vector and a classification result of a speech segment;

[0020] an audio dequantizer configured to receive the first word vector sequence and output target audio data corresponding to the first word vector sequence.

[0021] In a third aspect, an embodiment of the present application discloses an electronic device, which comprises a processor, a memory, a communication interface and a communication bus, the processor, the memory and the communication interface complete communication with each other through the communication bus; the memory is configured to store at least one executable instruction, and the executable instruction causes the processor to execute the steps of the data processing method as described above.

[0022] Embodiments of the present application have the following advantages:

[0023] This invention utilizes an audio quantizer to predict the quantized sequences of positive and negative sample data. A quantized sequence of one positive and one of one negative sample data are selected and input into a large language model as existing sequences. The target quantized sequence predicted by the large language model is similar to both the positive and negative sample data quantized sequences, thus creating confusion between the output target quantized sequence and the input existing sequences. A first word vector sequence corresponding to the target quantized sequence is obtained by querying a quantization dictionary. Finally, an audio dequantizer is used to convert the first word vector sequence into target audio data, which can be used as negative sample data for training a speech classification model. Obtaining negative sample data using the data processing method of this invention shortens data processing time, thereby reducing the cost and cycle of collecting negative sample data.

[0024] Furthermore, in the case of voice classification task migration, such as when the classification labels or preset wake words change, the time cost of the development process can be reduced and the product iteration cycle can be shortened because the cycle of steps 3 and 4 in the relevant technology is shortened.

[0025] Furthermore, in this embodiment of the invention, the audio quantizer and the audio dequantizer are jointly trained using the first training set used to train the audio classifier, and negative sample data is generated using the first training set, without having to collect training data again, which can reduce operating costs and improve training efficiency. Attached Figure Description

[0026] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is a flowchart illustrating the steps of an embodiment of the data processing method of the present invention;

[0028] Figure 2 This is a schematic diagram of the process of jointly training the audio quantizer and the audio dequantizer of the present invention;

[0029] Figure 3 This is a schematic diagram of the structure of an audio quantizer according to the present invention;

[0030] Figure 4 This is a schematic diagram of the structure of an audio dequantizer according to the present invention;

[0031] Figure 5 This is a schematic diagram illustrating the process by which the present invention obtains the final word vector of a speech segment;

[0032] Figure 6 This is a schematic diagram illustrating the process of constructing the first text sequence and the second text sequence according to the present invention;

[0033] Figure 7 This is a structural block diagram of an embodiment of a data processing device according to the present invention;

[0034] Figure 8 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0035] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0036] The terms "first," "second," etc., used in the specification and claims of this invention are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, the first object can be one or more. Furthermore, the term "and / or" in the specification and claims is used to describe the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. In embodiments of this invention, the term "multiple" refers to two or more, and other quantifiers are similar.

[0037] To facilitate understanding of the technical solutions of this invention, some technical terms involved in this invention will be introduced below.

[0038] Large language models are one application of deep learning, especially in the field of natural language processing (NLP). Large language models require training on massive amounts of text data to learn various patterns and structures of language. ChatGPT is an example of a large language model, trained to understand and generate human language for effective dialogue and answering various questions.

[0039] Large language models are trained to solve general (multi-domain) language problems, such as text classification, question answering, document summarization, and text generation.

[0040] (1) Text classification: Large language models can analyze and learn from input text to classify it into one or more predefined categories. For example, large language models can be used to classify emails as spam or tweets as positive, negative, or neutral.

[0041] (2) Question answering: Large language models can answer natural language questions posed by users. For example, large language models can be used to answer user queries in search engines or user questions in intelligent assistants.

[0042] (3) Document summarization: Large language models can automatically extract key information from text to generate document summaries or excerpts. For example, large language models can be used to generate summaries of news articles or extract key plot points and events from novels.

[0043] (4) Text generation: Large language models can use previously learned patterns and structures to generate new text. For example, large language models can be used to generate poems, short stories, or articles on specific topics.

[0044] In practice, a general-purpose large language model can be pre-trained first, and then fine-tuned for a specific target to obtain a single model applicable to a specific task or domain.

[0045] In the pre-training phase, the model is trained using large-scale, general-purpose text data to learn the basic structure of language and various common-sense concepts. Then, in the fine-tuning phase, further training is performed using smaller, more specific datasets. The datasets used in the fine-tuning phase are typically task- or domain-specific, such as medical text, legal text, or specific dialogue data. Fine-tuning allows the model to better understand and generate language within that specific domain, thereby performing the task more effectively.

[0046] While large language models require a large amount of general-purpose text data during the pre-training phase, only a relatively small amount of domain-specific data is needed during the fine-tuning phase. This is because the model has already learned a significant amount of language knowledge and common sense during pre-training, and the fine-tuning phase primarily adapts the model to specific tasks or domains. This allows large language models to perform well even in data-scarce domains, greatly reducing the complexity and cost of developing and maintaining different models.

[0047] Reference Figure 1 The diagram illustrates a flowchart of an embodiment of a data processing method according to the present invention, the method comprising the following steps:

[0048] Step 101: Receive each sample data from the first training set through an audio quantizer and output the quantization sequence corresponding to each sample data; wherein, the quantization sequence corresponding to a sample data includes the classification result corresponding to each speech segment obtained by segmenting the sample data; the first training set includes positive sample data and negative sample data, and the positive sample data and negative sample data are audio data;

[0049] Step 102: Select a quantization sequence corresponding to a positive sample data and a quantization sequence corresponding to a negative sample data as existing sequences and input them into the large language model. The large language model then outputs a target quantization sequence that is confused with the existing sequence.

[0050] Step 103: Obtain the first word vector sequence corresponding to the target quantization sequence by querying the quantization dictionary; the quantization dictionary contains the mapping relationship between word vectors and the classification results corresponding to speech segments;

[0051] Step 104: Receive the first word vector sequence through the audio dequantizer and output the target audio data corresponding to the first word vector sequence.

[0052] The data processing method provided by this invention can be applied to electronic devices. These electronic devices may include smartphones, desktop computers, tablets, laptops, digital assistants, augmented reality (AR) / virtual reality (VR) devices, intelligent voice interaction devices, smart home appliances, smart wearable devices, in-vehicle terminal devices, and other terminal devices. The electronic device can also be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0053] This invention allows for the pre-training of the audio quantizer, audio dequantizer, and large language model. The audio quantizer receives audio data and performs serialization processing on the received audio data to obtain a quantized sequence. Serialization is the process of converting the state information of an object into a form that can be stored or transmitted. In audio processing scenarios, serialization converts audio data (which may be analog signals) into digital form and may compress it to reduce the resources required for storage and transmission. For audio data, this typically involves sampling, quantizing, encoding, and possibly compressing analog signals for storage or transmission over a network. The audio quantizer can be a pre-trained neural network model.

[0054] In this embodiment of the invention, the audio quantizer can receive audio data of a preset length, classify the audio data of the preset length, and output a classification result. This classification result can be used to characterize the encoding features of the audio data of the preset length. For example, an audio data piece (assuming its length is greater than the preset length) can be divided into several speech segments according to the preset length, with each speech segment being an audio data piece of the preset length. The several speech segments divided from the audio data are sequentially input into the audio quantizer to obtain the quantization sequence corresponding to the audio data. This quantization sequence includes the classification result corresponding to each speech segment obtained from the division of the audio data.

[0055] An audio dequantizer is used to deserialize a quantized sequence to obtain audio data. Deserialization is the process of extracting data from a stored or transmitted form and reconstructing the original object. In audio processing scenarios, deserialization involves decoding the quantized sequence, possibly decompressing it, and converting it back to its original analog or digital form for playback or further processing. The audio dequantizer can be a pre-trained neural network model.

[0056] A Large Language Model (LLM) is a deep learning model trained on a large amount of text data that can generate natural language text or understand the meaning of language text. LLMs can handle various natural language tasks, such as text classification, question answering, and dialogue. In this embodiment of the invention, the LLM can be a Generative Pre-trained Transformer (GPT) LLM, which is a large language model based on the Transformer architecture. The LLM can be used to predict the next character based on an input character sequence. For example, given a character sequence consisting of N (N is a positive integer) characters, the LLM can predict the (N+1)th character.

[0057] This invention does not impose any restrictions on the model structures of the audio quantizer, audio dequantizer, and large language model. In one example, the audio quantizer can be implemented using a ResNet50 (Residual Network) structure, the audio dequantizer can be implemented using a HiFi-GAN structure, and the GPT large language model can be implemented using a Transformer structure.

[0058] In specific implementations, the audio quantizer, audio dequantizer, and large language model can be neural networks. This invention does not limit the network structures of the audio quantizer, audio dequantizer, speech classification model, and large language model; they can use the same network structure or different network structures. The neural network includes, but is not limited to, at least one or more combinations, superpositions, or nestings of the following: CNN (Convolutional Neural Network), LSTM (Long Short-Term Memory) network, RNN (Simple Recurrent Neural Network), Transformer network, attention neural network, etc.

[0059] In this invention, positive and negative sample data can be collected to construct a first training set. An initial version of the speech classification model can be trained using this first training set. The speech classification model (or audio classifier) ​​can be used to predict the matching probability value of audio data under preset category labels. For example, if the preset category labels include A and B, the speech classification model can predict the probability value of an audio segment belonging to category A and the probability value of belonging to category B. This invention does not limit the preset category labels; for example, the preset category labels can include human voices, cat meows, and dog barks, then the speech classification model can identify whether an audio segment contains human voices, cat meows, or dog barks. Similarly, the speech classification model can predict the probability value of an audio segment containing a preset wake word. Since negative sample data is difficult to collect, the speech classification model can be an initial version of the speech classification model trained using a certain number of collected positive and negative sample data.

[0060] This invention does not limit the model structure or training method of the speech classification model. In one example, the audio classifier can be implemented using a ResNet50 architecture.

[0061] Because negative sample data is difficult to collect, resulting in a limited number of negative samples, the accuracy of the initial version of the speech classification model is insufficient to meet expectations. To address this issue, a large amount of negative sample data can be generated by implementing the method of this invention. This generated negative sample data is then added to the first training set to further train the initial version of the speech classification model, thereby improving the accuracy of the speech classification model and ensuring that the accuracy of the trained speech classification model meets expectations.

[0062] The data processing method provided by this invention can quickly generate negative sample data for training a speech classification model, shortening the data processing time and thus reducing the cycle of steps 3 and 4 in related technologies, thereby improving the efficiency of training the speech classification model. Furthermore, in cases of speech classification task migration, such as changes in classification labels or preset wake words, the shortened cycles of steps 3 and 4 can reduce the time cost of the development process and shorten the product iteration cycle.

[0063] In this embodiment of the invention, the audio quantizer and the audio dequantizer can be trained directly using the first training set, and negative sample data can be generated using the first training set, without having to collect training data again, which can reduce operating costs and improve training efficiency.

[0064] After pre-training the audio quantizer, audio dequantizer, and large language model, each sample data from the first training set is input into the audio quantizer to obtain the quantization sequence corresponding to each sample data.

[0065] Specifically, each positive sample data in the first training set is input into the audio quantizer to obtain a quantized sequence of each positive sample data, and each negative sample data is input into the audio quantizer to obtain a quantized sequence of each negative sample data.

[0066] Furthermore, the quantization sequence can be a numerical sequence to facilitate subsequent reasoning processing of large language models.

[0067] For example, inputting audio A (a positive sample data) into an audio quantizer will yield the quantized sequence of audio A, denoted as sequence A. Let's assume sequence A is {1,3,2,2,3,3,2,1,8,4,2,1,1,1,2,3,4}. Similarly, inputting audio B (a negative sample data) into the audio quantizer will yield the quantized sequence of audio B, denoted as sequence B. Let's assume sequence B is {2,1,1,1,4,1,3,2,2,2,2,2,1,1,1,1}.

[0068] Next, a quantized sequence of positive sample data and a quantized sequence of negative sample data are selected as existing sequences and input into the trained large language model. The large language model can then predict the target quantized sequence that is confused with the existing sequence.

[0069] A trained large language model (such as GPT) possesses logical reasoning capabilities and can predict a target quantized sequence that is similar to (and confusingly similar to) an existing input sequence. In this embodiment of the invention, a quantized sequence of a positive sample data and a quantized sequence of a negative sample data are selected as existing sequences and input into the large language model. The target quantized sequence predicted by the large language model is similar to both the quantized sequences of the positive and negative samples, thus the output target quantized sequence has a confusing effect on the input existing sequence.

[0070] Furthermore, after the large language model (such as GPT) has been trained, pre-set template dialogues for GPT can be set. These pre-set template dialogues are used to assign a specific task to GPT.

[0071] For example, a preset template phrase is set as "Given {sequence A} and {sequence B}, predict x easily confused sequences C". The existing sequences in the preset template phrase include sequence A and sequence B, and the target quantized sequence is sequence C, which is similar to the existing sequences; where sequence A is the quantized sequence of positive sample data, and sequence B is the quantized sequence of negative sample data. Since the training process of the large language model uses a large amount of text data, it has developed a certain ability for logical reasoning and pattern finding. Therefore, sequence C is a similarity sequence inferred by the large language model from the key points extracted from the given existing sequences. Thus, based on the input existing sequence A and existing sequence B, the large language model can infer sequence C, which is similar to sequence A and sequence B, to achieve the purpose of confusion.

[0072] For example, the sequences A{1,3,2,2,3,3,2,1,8,4,2,1,1,1,2,3,4} and B{2,1,1,1,4,1,3,2,2,2,2,2,1,1,1,1} are input into the large language model as existing sequences in the preset template speech. The model then uses GPT to predict x (assuming x = 2 in this example) sequences C. The specific value of x can be set according to actual needs, and x is a positive integer.

[0073] For example, the GPT analysis inference is as follows:

[0074] 1. Observe the distribution of numbers: Sequence A contains more types of numbers, while sequence B is mainly composed of 1 and 2, with 1 being far more numerous than 2. The distribution of numbers in sequence A is more even, while sequence B is highly biased towards 1.

[0075] 2. Look for common features: Both sequences contain a large number of 1s and 2s. The beginning and end of sequence A may be visually similar to certain parts of sequence B (such as consecutive 1s or 2s).

[0076] 3. Consider the possibility of confusion: If we construct a sequence C, it should contain features similar to both A and B, but not exactly the same, in order to simulate the situation of confusion.

[0077] One possible strategy is to select the numbers that appear in both sequence A and sequence B (mainly 1 and 2) and try to combine them in some way so that C is visually or statistically similar to both sequence A and sequence B.

[0078] Based on the above analysis, the following two sequences C can be constructed as candidates for confusion:

[0079] Sequence C1:{1,2,1,2,1,2,1,2,1,2,1,2,1,2,1,2,1}

[0080] Sequence C1 consists entirely of alternating 1s and 2s and is visually very similar to sequence B (although sequence B has more 1s), but sequence C1 also shows the possibility of alternating 1s and 2s as in sequence A (although the alternation pattern in sequence A is less regular).

[0081] Sequence C2:{1,1,1,2,2,1,1,2,2,1,1,2,2,1,1,2,2}

[0082] Sequence C2 is characterized by the pairing of 1 and 2, which is also reflected to some extent in sequences A and B (although not the main feature), but statistically it is similar to sequences A and B, especially in the types and distribution of numbers.

[0083] In this example, the generated sequences C1 and C2 are the target quantization sequences. In a specific implementation, the number of target quantization sequences C generated based on existing sequences A and B is determined by x in the preset template script.

[0084] Next, the first word vector sequence corresponding to the target quantization sequence is obtained by querying the quantization dictionary. The target quantization sequence contains the classification results of the encoded features of each speech segment, and the first word vector sequence contains the encoded feature representation of each speech segment.

[0085] This invention can pre-establish a quantization dictionary, which contains a mapping relationship between preset word vectors and classification results corresponding to speech segments (according to encoded features). A word vector can be the encoded feature representation of a speech segment of preset length, corresponding to a classification result. For example, if the audio quantizer is set to have 1000 classification results, then the audio quantizer has 1000 trainable word vectors (or biases), and these 1000 trainable word vectors can constitute the "quantization dictionary"; or, if the audio quantizer is set to have 100 classification results, then the audio quantizer has 100 trainable word vectors, and these 100 trainable word vectors can constitute the "quantization dictionary".

[0086] Furthermore, this invention utilizes the indices of word vectors in the quantization dictionary to represent the classification results corresponding to those word vectors, resulting in a numerical sequence of word vectors. For example, the quantization dictionary includes five word vectors, denoted as V1, V2, V3, V4, and V5. These five word vectors have a one-to-one correspondence with the classification results corresponding to five speech segments. Assuming that the classification results corresponding to these five speech segments are represented by indices 1, 2, 3, 4, and 5, where word vector V1 corresponds to classification result 1, representing the encoding features of speech segment 1; word vector V2 corresponds to classification result 2, representing the encoding features of speech segment 2; and so on.

[0087] In one example, assuming the target quantization sequence is {1,3,2,1,4}, by querying the quantization dictionary, the word vector corresponding to each classification result in the target quantization sequence {1,3,2,1,4} can be obtained, thus yielding the first word vector sequence corresponding to the target quantization sequence {1,3,2,1,4}. For example, if the word vector corresponding to classification result 1 is V1, the word vector corresponding to classification result 3 is V3, the word vector corresponding to classification result 2 is V2, the word vector corresponding to classification result 1 is V1, and the word vector corresponding to classification result 4 is V4, then the first word vector sequence is {V1,V3,V2,V1,V4}.

[0088] Finally, the first word vector sequence is input into the audio dequantizer to convert it into target audio data. For example, inputting the first word vector sequence corresponding to sequence C1 into the audio dequantizer to convert it into target audio data results in one generated negative sample data. Inputting the first word vector sequence corresponding to sequence C2 into the audio dequantizer to convert it into target audio data results in another generated negative sample data.

[0089] Following the above method, a new quantized sequence of positive sample data and a new quantized sequence of negative sample data are selected and input into the large language model as existing sequences in the preset template speech. This predicts x new target quantized sequences. The first word vector sequences corresponding to the new x target quantized sequences are then input into the audio dequantizer to convert them into target audio data, resulting in x new negative sample data. Repeating steps 102 and 103 yields a large amount of negative sample data.

[0090] In an optional embodiment of the present invention, the method may further include:

[0091] The target audio data is added as negative sample data to the first training set, and the speech classification model is trained using the first training set.

[0092] After generating negative sample data using the above method, the negative sample data can be added to the first training set. The initial version of the speech classification model can then be trained using the first training set to improve the accuracy of the speech classification model.

[0093] In an optional embodiment of the present invention, the method may further include: jointly training the audio quantizer and the audio inverse quantizer using the first training set, specifically including the following steps:

[0094] Step S11: Initialize the audio quantizer and audio dequantizer;

[0095] Step S12: Take a sample data from the first training set as the input audio, divide it into several speech segments according to a preset length, and then input them into the initialized audio quantizer in sequence to predict the quantization sequence corresponding to the input audio.

[0096] Step S13: By querying the quantization dictionary, obtain the word vectors corresponding to each classification result in the quantization sequence corresponding to the input audio, and form a second word vector sequence in order;

[0097] Step S14: Input the second word vector sequence into the initialized audio dequantizer to predict the decoded audio;

[0098] Step S15: Calculate the first loss value based on the decoded audio and the input audio;

[0099] Step S16: Update the model parameters of the audio quantizer and the audio dequantizer according to the first loss value;

[0100] Step S17: Proceed to the next iteration. When the iteration stopping condition is met, the trained audio quantizer and audio dequantizer are obtained.

[0101] In this embodiment of the invention, the audio quantizer and audio dequantizer are trained using a joint training method.

[0102] First, a quantization dictionary is established, which contains a mapping relationship between preset word vectors and the classification results corresponding to speech segments. A word vector can be a feature representation of a speech segment of a preset length. Furthermore, the classification results corresponding to the word vectors in the quantization dictionary can correspond to the classification results that the audio quantizer can predict.

[0103] In this embodiment of the invention, the quantization dictionary can serve as a constraint for jointly training the audio quantizer and the audio dequantizer. Since the model's representations can be infinite, leading to prolonged training time, this invention constrains the model's representations by incorporating a quantization dictionary, limiting the representations of the audio quantizer and the audio dequantizer to word vectors within the quantization dictionary, thereby accelerating the model training process.

[0104] Reference Figure 2 The diagram illustrates the flowchart of the joint training of the audio quantizer and the audio dequantizer of this invention. Figure 2 As shown, assuming there are 4 pre-set word vectors in the quantization dictionary, denoted as V1, V2, V3, and V4, these 4 word vectors correspond to the classification results of 4 speech segments as 1, 2, 3, and 4, respectively.

[0105] First, the audio quantizer and audio dequantizer are initialized. A sample data point (positive or negative) is taken from the first training set as input audio. This input audio is segmented into several speech segments according to a preset length, and then sequentially input into the initialized audio quantizer. The audio quantizer predicts the classification result corresponding to each speech segment, thus obtaining the quantized sequence corresponding to the input audio. The quantized sequence corresponding to the input audio includes the classification result corresponding to each speech segment obtained from the input audio segmentation.

[0106] In one example, for the current input audio, assume it is divided into 5 speech segments of a preset length, which can be set according to actual needs. These 5 speech segments are sequentially input into an audio quantizer for serialization. Assume the output classification result for each speech segment is: 1, 3, 2, 1, 4. That is, the classification result for the first speech segment is 1, the classification result for the second speech segment is 3, and so on. The classification result vector for this input audio is {1,3,2,1,4}.

[0107] In this embodiment of the invention, the audio quantizer can predict the probability value of each classification result in the quantization dictionary for the received speech segment, and take the one with the highest probability value as the classification result of the speech segment.

[0108] For example, in the example above, after the first speech segment is input into the audio quantizer, the audio quantizer predicts the probability value of each classification result in the quantization dictionary corresponding to the first speech segment. Assuming that the probability value of the first speech segment belonging to classification result 1 is the highest, the predicted classification result for the first speech segment is 1. Similarly, the predicted classification result for the second speech segment is 3. And so on, thus obtaining the quantization sequence of the input audio as {1,3,2,1,4}.

[0109] Then, based on the classification result of each speech segment, the quantization dictionary is queried, and the word vectors corresponding to each classification result in the quantization sequence {1,3,2,1,4} are obtained from the quantization dictionary and arranged in order to form a second word vector sequence. For example, for the quantization sequence {1,3,2,1,4} of the input audio, the word vectors corresponding to each classification result can be extracted from the quantization dictionary and arranged in order to form a second word vector sequence, such as {V1,V3,V2,V1,V4}.

[0110] Next, the second word vector sequence is input into the initialized audio dequantizer to predict the decoded audio. Based on the difference between the decoded audio (predicted value) and the input audio (true value), a first loss value can be calculated. The model parameters of the audio quantizer and the audio dequantizer are updated based on the first loss value. In the next iteration, a new sample data (positive or negative) is taken from the first training set as the input audio, and the above steps are repeated. When the iteration stopping condition is met, the trained audio quantizer and audio dequantizer are obtained.

[0111] This invention does not limit the method for calculating the first loss value. For example, a general loss function can be used to calculate the model loss. General loss functions include cross-entropy loss function, cosine similarity loss function, etc.

[0112] The iteration stopping condition may include a first loss value being less than a preset value, or the number of iterations reaching a preset number.

[0113] In an optional embodiment of the present invention, the method may further include: updating the parameters of the quantization dictionary based on the first loss value.

[0114] In this embodiment of the invention, the quantization dictionary can be a trainable vector group composed of N vectors of dimension V. The quantization dictionary contains N×V word vectors, each corresponding to its own weight parameters. During the joint training of the audio quantizer and the audio dequantizer, the quantization dictionary can also be trained to continuously optimize the weight parameters of each word vector in the quantization dictionary.

[0115] Specifically, after calculating the first loss value, the model parameters of the audio quantizer, the model parameters of the audio dequantizer, and the weight parameters of each word vector in the quantization dictionary can be updated based on the first loss value, and the next iteration can be entered. When the iteration stopping condition is met, the trained audio quantizer, audio dequantizer, and quantization dictionary are obtained.

[0116] It should be noted that the embodiments of the present invention do not impose any restrictions on the structure of the audio quantizer and the structure of the audio dequantizer.

[0117] In an optional embodiment of the present invention, the audio quantizer may include a quantization encoder and a classifier. The quantization encoder is used to receive a speech segment of a preset length, encode the received speech segment into a coding feature, and input the coding feature into the classifier. The classifier is used to output the classification result corresponding to the coding feature.

[0118] Reference Figure 3 The diagram shows a structural schematic of an audio quantizer according to the present invention. Figure 3 As shown, the audio quantizer may include a quantization encoder and a classifier. The classifier has M categories, where M is a positive integer. The value of M can be set according to actual needs, and M can be the number of word vectors preset in the quantization dictionary. The quantization encoder receives speech segments, encodes the received speech segments into encoded features, and inputs the encoded features into the classifier. The classifier classifies the received encoded features and outputs the classification result corresponding to the encoded features.

[0119] In an optional embodiment of the present invention, the audio dequantizer may include a quantization decoder and a generator. The quantization decoder is used to receive a sequence of word vectors, decode the received sequence of word vectors into decoded features, and input the decoded features into the generator. The generator is used to convert the decoded features into audio data.

[0120] Reference Figure 4 The diagram shows a schematic of the structure of an audio inverse quantizer according to the present invention. Figure 4 As shown, the audio dequantizer includes a quantization decoder and a generator. The quantization decoder receives a sequence of word vectors, decodes the received word vector sequence into decoded features, and inputs the decoded features into the generator; the generator generates corresponding audio data (or decoded audio) based on the received decoded features.

[0121] In an optional embodiment of the present invention, after taking a sample data from the first training set as input audio, dividing it into several speech segments according to a preset length, and then sequentially inputting them into an initialized audio quantizer, the method may further include:

[0122] Step S21: Obtain the prediction result of the audio quantizer for the current speech segment; the prediction result includes the probability value of each classification result in the quantization dictionary corresponding to the current speech segment;

[0123] Step S22: Set a masking code for the prediction result; wherein the masking code corresponding to the highest probability value is 1, and the masking code corresponding to the other probability values ​​is 0;

[0124] Step S22: Multiply the prediction result by the corresponding masking code to obtain an intermediate result;

[0125] Step S23: Multiply the intermediate result with each word vector in the quantization dictionary and add them together to obtain the final word vector;

[0126] Step S24: Arrange the final word vectors into a second word vector sequence in order.

[0127] In practical implementation, the audio quantizer predicts the probability value of each classification result in the quantization dictionary for the received speech segment, and takes the one with the highest probability value as the classification result of the speech segment. Then, it obtains the corresponding word vectors according to the classification results, forming a second word vector sequence such as {V1,V3,V2,V1,V4}. However, the resulting second word vector sequence is not differentiable, which may affect the propagation of model loss and thus the accuracy of the model. To solve this problem, this invention makes the obtained second word vector sequence differentiable by executing steps S21 to S24.

[0128] Specifically, for the current input audio, it is divided into several speech segments according to a preset length and then sequentially input into an audio quantizer for serialization processing. For the current speech segment, the prediction result of the audio quantizer for the current speech segment is obtained; the prediction result includes the probability value of the current speech segment corresponding to each classification result in the quantization dictionary.

[0129] Reference Figure 5 This diagram illustrates the process by which the present invention obtains the final word vector of a speech segment, and this final word vector is differentiable. For example... Figure 5As shown, for the first speech segment, the audio quantizer predicts the probability values ​​of each classification result in its corresponding quantization dictionary as follows: 0.9, 0.1, 0.2, and 0.3. That is, the probability value of predicting the first speech segment as belonging to classification result 1 corresponding to the pre-set word vector V1 is 0.9, the probability value of belonging to classification result 2 corresponding to the pre-set word vector V2 is 0.1, the probability value of belonging to classification result 3 corresponding to the pre-set word vector V3 is 0.2, and the probability value of belonging to classification result 4 corresponding to the pre-set word vector V4 is 0.3.

[0130] In this embodiment of the invention, a masking code is set for the prediction result. Specifically, the masking code corresponding to the highest probability value is set to 1, and the masking codes corresponding to the other probability values ​​are set to 0.

[0131] For example, in this example, the probability value of 0.9 is the maximum for the first speech segment. Therefore, the masking code corresponding to the probability value of 0.9 is set to 1, and the masking codes corresponding to the other probability values ​​are all set to 0.

[0132] Next, the prediction result is multiplied by the corresponding masking code to obtain an intermediate result. For example, the probability value 0.9 is multiplied by the masking code 1, the probability value 0.1 is multiplied by the masking code 0, the probability value 0.2 is multiplied by the masking code 0, and the probability value 0.3 is multiplied by the masking code 0 to obtain an intermediate result, which can be denoted as {0.9,0,0,0}.

[0133] Next, the intermediate results are multiplied and summed with the corresponding word vectors in the quantization dictionary to obtain the final word vectors. For example, multiplying and summing the intermediate results {0.9,0,0,0} with the corresponding word vectors {V1,V2,V3,V4}, i.e., calculating:

[0134] 0.9×V1+0×V2+0×V3+0×V4

[0135] Therefore, the final word vector corresponding to the first speech segment is 0.9 × V1, not V1, and this final word vector is differentiable. Following the same method, the final word vectors corresponding to each of the remaining speech segments obtained from the input audio segmentation are calculated separately. The final word vectors corresponding to each speech segment of the input audio are then sequentially combined to form the second word vector sequence corresponding to that input audio. The resulting second word vector sequence achieves the same effect and is differentiable, allowing joint training to accurately pass on the loss value and improve the accuracy of model training.

[0136] This invention can pre-train a large language model, and this invention does not limit the model structure or training method of the large language model.

[0137] In an optional embodiment of the present invention, the method may further include: training a large language model using a second training set, specifically including the following steps:

[0138] Step S31: Construct a second training set, which includes text data;

[0139] Step S32: Initialize a large language model;

[0140] Step S33: Concatenate the text data in the second training set to obtain a text sequence containing T characters, where T is a positive integer;

[0141] Step S34: Input the character sequence composed of the first N characters in the text sequence into the initialized large language model to predict the (N + 1)-th character; where 0 < N < T;

[0142] Step S35: Calculate a second loss value according to the difference between the predicted (N + 1)-th character and the (N + 1)-th character in the text sequence;

[0143] Step S36: Update the model parameters of the large language model according to the second loss value;

[0144] Step S37: Enter the next round of iteration. When the iteration stop condition is met, obtain the trained large language model.

[0145] In a specific implementation, text data in various languages can be collected to construct a second training set for training the logical reasoning ability of the large language model. Concatenate all the text data in the second training set to obtain a text sequence containing T characters, where T is a positive integer.

[0146] First, initialize the large language model. In the first round of iteration, input the first N (0 < N < T) characters in the text sequence into the initialized large language model, predict the (N + 1)-th character through the initialized large language model, and calculate the second loss value according to the difference between the predicted (N + 1)-th character and the (N + 1)-th character in the text sequence; update the model parameters of the large language model according to the second loss value; enter the next round of iterative training.

[0147] In the next round of iteration (the second round of iteration), input the first N + 1 characters in the text sequence into the large language model, predict the (N + 2)-th character through the large language model, and calculate the second loss value according to the difference between the predicted (N + 2)-th character and the (N + 2)-th character in the text sequence; update the model parameters of the large language model according to the second loss value; enter the next round of iteration (the third round of iteration) training, and so on. When the iteration stop condition is met, obtain the trained large language model.

[0148] This invention does not limit the method for calculating the second loss value. For example, a general loss function can be used to calculate the model loss. General loss functions include cross-entropy loss function, cosine similarity loss function, etc.

[0149] The iteration stopping condition may include a second loss value being less than a preset value, or the number of iterations reaching a preset number, or all characters in the text sequence having been entered.

[0150] The large language model trained in this way has reasoning ability and can predict the next character based on an existing character sequence. This invention utilizes the reasoning ability of the large language model to infer potentially confusing audio quantization sequences, thereby generating negative sample data and reducing the cost and time required to collect negative sample data for training speech classification models.

[0151] After training a large language model with reasoning capabilities, this invention can further fine-tune the trained large language model to optimize its performance. Fine-tuning is a method that, based on a pre-trained model, further trains and adjusts some or all of the model's parameters for a specific task or data domain. By using limited training data, it allows the model to better adapt to a specific task, thereby improving its performance on that task. The specific task could be, for example, predicting confusing sequences based on existing sequences.

[0152] In an optional embodiment of the present invention, the method may further include:

[0153] Step S41: Extract a preset number of positive sample data and a preset number of negative sample data from the first training set;

[0154] Step S42: Input each extracted sample data into the audio quantizer to obtain the quantization sequence of each sample data; and input each extracted sample data into the trained speech classification model to obtain the category probability value of each sample data.

[0155] Step S43: Concatenate the quantized sequence and class probability value of each positive sample data in sequence to obtain the first text sequence; and concatenate the quantized sequence and class probability value of each negative sample data in sequence to obtain the second text sequence;

[0156] Step S44: Fine-tune the large language model using the first text sequence and the second text sequence. The fine-tuning process is similar to the training process of the large language model.

[0157] This invention embodiment can use the first training set to fine-tune a trained large language model. Specifically, a preset number of positive sample data and a preset number of negative sample data can be extracted from the first training set, and the extracted sample data can be used to fine-tune the trained large language model. This invention embodiment does not limit the preset number. For example, the preset number is S, where S is a positive integer.

[0158] Reference Figure 6 The diagram illustrates the process of constructing the first and second text sequences according to the present invention. Figure 6 As shown, S positive samples and S negative samples are extracted from the first training set, assuming S is 4. The extracted positive samples include positive sample 1 to positive sample 4. The extracted negative samples include negative sample 1 to negative sample 4.

[0159] Each extracted sample data is input into a trained audio quantizer to obtain a quantized sequence for each sample data; and each extracted sample data is input into a trained speech classification model to obtain a class probability value for each sample data.

[0160] Specifically, positive samples 1 through 4 are input into the trained audio quantizer to obtain the quantized sequences corresponding to positive samples 1 through 4, respectively. For example... Figure 6 As shown, assume that the quantization sequence of positive sample 1 is {1,3,2,2}, the quantization sequence of positive sample 2 is {3,3,2,1}, the quantization sequence of positive sample 3 is {4,2,1,1}, and the quantization sequence of positive sample 4 is {1,2,3,4}.

[0161] Input positive samples 1 through 4 into the trained speech classification model to obtain the class probability values ​​corresponding to positive samples 1 through 4, respectively. Figure 6 As shown, assume that the class probability value of positive sample 1 is 0.9, the class probability value of positive sample 2 is 0.88, the class probability value of positive sample 3 is 1.0, and the class probability value of positive sample 4 is 0.92.

[0162] Therefore, for a predetermined number of positive sample data points, the quantization sequence and category probability value corresponding to each positive sample data point can be obtained. In this embodiment of the invention, the set of these quantization sequences and category probability values ​​is referred to as the positive sample quantization set.

[0163] For the positive sample quantization set, the quantization sequence and class probability value of each positive sample data are concatenated sequentially to obtain the first text sequence. For example, the first text sequence is "quantization sequence of positive sample 1 + class probability value of positive sample 1 + quantization sequence of positive sample 2 + class probability value of positive sample 2 + ... + quantization sequence of positive sample S + class probability value of positive sample S".

[0164] Similarly, a negative sample quantization set can be constructed using the extracted S negative sample data. For the negative sample quantization set, the quantization sequence and class probability value of each negative sample data are concatenated sequentially to obtain a second text sequence. For example, the second text sequence is "quantization sequence of negative sample 1 + class probability value of negative sample 1 + quantization sequence of negative sample 2 + class probability value of negative sample 2 + ... + quantization sequence of negative sample S + class probability value of negative sample S".

[0165] The trained large language model can be fine-tuned using the first and second text sequences. The fine-tuning process is similar to the training process of the large language model.

[0166] Furthermore, in this embodiment of the invention, the first text sequence and the second text sequence are used to fine-tune the trained large language model. For example, the first text sequence is first used to fine-tune the trained large language model, and then the second text sequence is used to fine-tune the finely tuned large language model again. Alternatively, the second text sequence is first used to fine-tune the trained large language model, and then the first text sequence is used to fine-tune the finely tuned large language model again.

[0167] The process of fine-tuning using the first text sequence and the second text sequence is the same. This invention will describe the process using the first text sequence as an example.

[0168] First, the character sequence consisting of the first N characters from the first text sequence is input into the large language model, which then predicts the (N+1)th character. The character sequence consisting of the first N characters is the first quantized sequence in the positive sample quantization set. The initial input is the complete content of the first quantized sequence; the purpose is to allow the fine-tuned large language model to predict sequences of the same type based on the given sequence, thus achieving the purpose of confusion. Figure 6 For example, if the first text sequence is {1,3,2,2,0.9,3,3,2,1,0.88,4,2,1,1,1.0,1,2,3,4,0.92}, then the first character sequence input to the large language model is {1,3,2,2}. The large language model predicts the (N+1)th character based on the received character sequence {1,3,2,2}. The difference between the (N+1)th character predicted by the large language model (predicted value) and the (N+1)th character in the first text sequence (0.9, i.e., the true value) is used to calculate the second loss value. The model parameters of the large language model are updated based on the second loss value, and the next iteration begins.

[0169] In the next iteration (second iteration), N+1 characters are input to the large language model, i.e., the input character sequence {1,3,2,2,0.9}. The large language model predicts the N+2th character based on the received character sequence {1,3,2,2,0.9}. The difference between the N+2th character predicted by the large language model (predicted value) and the N+2th character in the first text sequence (3, i.e., the true value) is used to calculate the second loss value. The model parameters of the large language model are updated based on the second loss value; then the next iteration begins.

[0170] In the next iteration (the third iteration), N+2 characters are input to the large language model, i.e., the input character sequence {1,3,2,2,0.9,3}. The large language model predicts the (N+3)th character based on the received character sequence {1,3,2,2,0.9,3}. The difference between the (N+3)th character predicted by the large language model (the predicted value) and the (N+3)th character in the first text sequence (3, i.e., the true value) is used to calculate the second loss value. The model parameters of the large language model are updated based on the second loss value; then the next iteration begins. This process continues in this manner.

[0171] After fine-tuning the trained large language model using the first text sequence and the second text sequence respectively, a fine-tuned large language model can be obtained.

[0172] Furthermore, the audio length of each of the S extracted positive and S negative sample data is greater than a preset length to ensure that each sample data can obtain a quantized sequence after serialization processing by the audio quantizer. This invention does not limit the preset length. For example, the preset length can be 200ms to ensure that the audio of each sample data lasts for 4 to 5 seconds.

[0173] After the large language model is fine-tuned, step 104 can use the fine-tuned large language model. That is, a quantized sequence of one positive sample data and a quantized sequence of one negative sample data are selected as existing sequences in the preset template speech and input into the fine-tuned large language model to predict the target quantized sequence.

[0174] In summary, this invention utilizes an audio quantizer to predict the quantized sequences of positive and negative sample data. A quantized sequence of one positive and one of one negative sample data are selected and input into a large language model as existing sequences. The target quantized sequence predicted by the large language model is similar to both the positive and negative sample data quantized sequences, thus creating confusion between the output target quantized sequence and the input existing sequences. A first word vector sequence corresponding to the target quantized sequence is obtained by querying a quantization dictionary. Finally, an audio dequantizer is used to convert the first word vector sequence into target audio data, which can be used as negative sample data for training a speech classification model. Obtaining negative sample data using the data processing method of this invention shortens data processing time, thereby reducing the cost and cycle of collecting negative sample data.

[0175] Furthermore, in the case of voice classification task migration, such as when the classification labels or preset wake words change, the time cost of the development process can be reduced and the product iteration cycle can be shortened because the cycle of steps 3 and 4 in the relevant technology is shortened.

[0176] Furthermore, in this embodiment of the invention, the audio quantizer and the audio dequantizer are jointly trained using the first training set used to train the audio classifier, and negative sample data is generated using the first training set, without having to collect training data again, which can reduce operating costs and improve training efficiency.

[0177] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.

[0178] Reference Figure 7 The diagram illustrates a structural block diagram of an embodiment of a data processing apparatus according to the present invention. The apparatus may include:

[0179] The audio quantizer 701 is used to receive each sample data in the first training set and output the quantization sequence corresponding to each sample data; wherein, the quantization sequence corresponding to a sample data includes the classification result corresponding to each speech segment obtained by segmenting the sample data; the first training set includes positive sample data and negative sample data, wherein the positive sample data and negative sample data are audio data;

[0180] The large language model 702 is used to receive an existing sequence and output a target quantized sequence that is confused with the existing sequence; the existing sequence includes a quantized sequence corresponding to a positive sample data and a quantized sequence corresponding to a negative sample data.

[0181] The dictionary query module 703 is used to obtain the first word vector sequence corresponding to the target quantization sequence by querying the quantization dictionary; the quantization dictionary contains the mapping relationship between word vectors and the classification results corresponding to speech segments;

[0182] The audio dequantizer 704 is used to receive the first word vector sequence and output the target audio data corresponding to the first word vector sequence.

[0183] Optionally, the audio quantizer 701 includes a quantizer encoder and a classifier. The quantizer encoder receives a speech segment of a preset length, encodes the received speech segment into encoded features, and inputs the encoded features into the classifier. The classifier outputs a classification result corresponding to the encoded features. The audio dequantizer 704 includes a quantizer decoder and a generator. The quantizer decoder receives a word vector sequence, decodes the received word vector sequence into decoded features, and inputs the decoded features into the generator. The generator converts the decoded features into audio data.

[0184] Optionally, the apparatus further includes a first training module, which is used to jointly train the audio quantizer and the audio inverse quantizer using the first training set; the first training module includes:

[0185] The first initialization submodule is used to initialize the audio quantizer and audio dequantizer;

[0186] The first iteration training submodule includes a first acquisition unit, a loss calculation unit, and an update iteration unit;

[0187] The first acquisition unit is used to extract a sample data from the first training set as input audio, divide it into several speech segments according to a preset length, and then input them sequentially into an initialized audio quantizer to predict the quantization sequence corresponding to the input audio; by querying the quantization dictionary, it obtains the word vector corresponding to each classification result in the quantization sequence corresponding to the input audio, and arranges them into a second word vector sequence in order.

[0188] The loss calculation unit is used to input the second word vector sequence into an initialized audio dequantizer to predict the decoded audio; and to calculate a first loss value based on the decoded audio and the input audio.

[0189] The update and iteration unit is used to update the model parameters of the audio quantizer and the audio de - quantizer according to the first loss value; enter the next round of iteration, and when the iteration stop condition is met, obtain the trained audio quantizer and audio de - quantizer.

[0190] Optionally, the device further includes a second training module for training the large - language model using a second training set; the second training module includes:

[0191] A dataset construction sub - module for constructing a second training set, where the second training set includes text data;

[0192] A second initialization sub - module for initializing the large - language model;

[0193] A text splicing sub - module for splicing the text data in the second training set to obtain a text sequence containing T characters, where T is a positive integer;

[0194] A second iterative training sub - module for inputting the character sequence composed of the first N characters in the text sequence into the initialized large - language model to predict the (N + 1) - th character; where 0 < N < T; calculating a second loss value according to the difference between the predicted (N + 1) - th character and the (N + 1) - th character in the text sequence; updating the model parameters of the large - language model according to the second loss value; entering the next round of iteration, and when the iteration stop condition is met, obtaining the trained large - language model.

[0195] Optionally, the first iterative training sub - module further includes a second acquisition unit;

[0196] Among them, the second acquisition unit is used to obtain the prediction result of the audio quantizer for the current speech segment; the prediction result includes the probability values of each classification result in the quantization dictionary corresponding to the current speech segment; setting a masking code for the prediction result; where the masking code corresponding to the maximum probability value is 1, and the masking codes corresponding to the remaining probability values are 0; multiplying the prediction result by the corresponding masking code to obtain an intermediate result; multiplying the intermediate result by each word vector in the quantization dictionary and adding them up to obtain a final word vector; forming a second word vector sequence in order with the final word vectors.

[0197] Optionally, the device further includes a model fine - tuning module for:

[0198] A preset number of positive sample data and a preset number of negative sample data are extracted from the first training set; each extracted sample data is input into the audio quantizer to obtain a quantized sequence for each sample data; each extracted sample data is input into the trained speech classification model to obtain a class probability value for each sample data; the quantized sequence and class probability value of each positive sample data are concatenated sequentially to obtain a first text sequence; the quantized sequence and class probability value of each negative sample data are concatenated sequentially to obtain a second text sequence; the large language model is fine-tuned using the first text sequence and the second text sequence.

[0199] Optionally, the apparatus further includes a model retraining module for:

[0200] The target audio data is added as negative sample data to the first training set, and the speech classification model is trained using the first training set.

[0201] The data processing apparatus provided by this invention uses an audio quantizer to predict the quantization sequences of positive and negative sample data. It selects one quantization sequence from positive and one from negative sample data as existing sequences and inputs them into a large language model. The target quantization sequence predicted by the large language model is similar to both the positive and negative sample data quantization sequences, thus creating confusion between the output target quantization sequence and the input existing sequences. A first word vector sequence corresponding to the target quantization sequence is obtained by querying a quantization dictionary. Finally, an audio dequantizer converts the first word vector sequence into target audio data, which can be used as negative sample data for training a speech classification model. Obtaining negative sample data using the data processing method of this invention shortens data processing time, thereby reducing the cost and cycle of collecting negative sample data.

[0202] Furthermore, in the case of voice classification task migration, such as when the classification labels or preset wake words change, the time cost of the development process can be reduced and the product iteration cycle can be shortened because the cycle of steps 3 and 4 in the relevant technology is shortened.

[0203] Furthermore, in this embodiment of the invention, the audio quantizer and the audio dequantizer are jointly trained using the first training set used to train the audio classifier, and negative sample data is generated using the first training set, without having to collect training data again, which can reduce operating costs and improve training efficiency.

[0204] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0205] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0206] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0207] Reference Figure 8 This is a schematic diagram of the structure of the electronic device provided in an embodiment of the present invention. Figure 8 As shown, the electronic device includes: a processor, a memory, a communication interface, and a communication bus. The processor, the memory, and the communication interface communicate with each other through the communication bus. The memory is used to store at least one executable instruction, which causes the processor to execute the steps of the secure storage method described in the foregoing embodiments.

[0208] This invention provides a non-transitory computer-readable storage medium that, when the instructions in the storage medium are executed by a program or processor of a terminal, enables the terminal to perform the steps of the data processing method described in the foregoing embodiments.

[0209] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0210] Furthermore, the beneficial effects of using the same method will not be repeated here. For technical details not disclosed in the computer program products or computer program embodiments involved in this application, please refer to the description of the method embodiments of this application.

[0211] Other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of the invention are indicated by the following claims.

[0212] It should be understood that the present invention is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

[0213] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

[0214] The above provides a detailed description of the data processing method, apparatus, and electronic device provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A data processing method, characterized by, The method comprises: receiving each piece of sample data in the first training set by an audio quantizer, and outputting a quantization sequence corresponding to each piece of sample data; wherein the quantization sequence corresponding to one piece of sample data comprises the classification result corresponding to each speech segment obtained by segmenting the piece of sample data; the first training set comprises positive sample data and negative sample data, and the positive sample data and the negative sample data are audio data; selecting a quantization sequence corresponding to one piece of positive sample data and a quantization sequence corresponding to one piece of negative sample data as an existing sequence to input a large language model, and outputting a target quantization sequence confused with the existing sequence by the large language model; obtaining a first word vector sequence corresponding to the target quantization sequence by querying a quantization dictionary; the quantization dictionary comprises a mapping relationship between a word vector and a classification result corresponding to a speech segment; receiving the first word vector sequence by an audio dequantizer, and outputting target audio data corresponding to the first word vector sequence.

2. The method of claim 1, wherein, The method comprises: The audio quantizer comprises a quantization encoder and a classifier, the quantization encoder is used to receive a speech segment of a preset length, encode the received speech segment into an encoded feature, and input the encoded feature into the classifier; the classifier is used to output a classification result corresponding to the encoded feature; The audio dequantizer comprises a quantization decoder and a generator, the quantization decoder is used to receive a word vector sequence, decode the received word vector sequence into a decoded feature, and input the decoded feature into the generator; the generator is used to convert the decoded feature into audio data.

3. The method of claim 1 or 2, further comprising: initializing the audio quantizer and the audio dequantizer; taking one piece of sample data from the first training set as input audio, segmenting the input audio into a plurality of speech segments of a preset length, and inputting the speech segments into the initialized audio quantizer in sequence to obtain a quantization sequence corresponding to the input audio; obtaining a word vector corresponding to each classification result in the quantization sequence corresponding to the input audio by querying the quantization dictionary, and sequentially forming a second word vector sequence; inputting the second word vector sequence into the initialized audio dequantizer to obtain a decoded audio; calculating a first loss value according to the decoded audio and the input audio; updating model parameters of the audio quantizer and the audio dequantizer according to the first loss value; entering the next iteration, and obtaining a trained audio quantizer and a trained audio dequantizer when an iteration stop condition is met.

4. The method of claim 3, wherein, After the step of taking one piece of sample data from the first training set as input audio, segmenting the input audio into a plurality of speech segments of a preset length, and inputting the speech segments into the initialized audio quantizer in sequence, the method further comprises: obtaining a prediction result of the current speech segment by the audio quantizer; the prediction result comprises probability values of each classification result in the quantization dictionary corresponding to the current speech segment; setting a mask for the prediction result; wherein the mask corresponding to the maximum probability value is 1, and the masks corresponding to the remaining probability values are 0; multiplying the prediction result and the corresponding mask to obtain an intermediate result; The intermediate result is multiplied and added to each word vector in the quantization dictionary to obtain a final word vector; The final word vector is sequentially composed into a second word vector sequence.

5. The method according to claim 1 or 2, characterized in that, The method further comprises: constructing a second training set, wherein the second training set comprises text data; initializing a large language model; concatenating the text data in the second training set to obtain a text sequence containing T characters, where T is a positive integer; inputting a character sequence composed of the first N characters in the text sequence into the initialized large language model to predict the (N+1)th character; where 0 calculating a second loss value according to the difference between the predicted (N+1)th character and the (N+1)th character in the text sequence; updating the model parameters of the large language model according to the second loss value; entering the next round of iteration, and obtaining a trained large language model when the iteration stopping condition is met.

6. The method of claim 1 or 2, wherein, The method further comprises: extracting a preset number of positive sample data and a preset number of negative sample data from the first training set; inputting each extracted sample data into the audio quantizer to obtain a quantization sequence of each sample data; and inputting each extracted sample data into the trained speech classification model to obtain a category probability value of each sample data; concatenating the quantization sequence and the category probability value of each positive sample data in sequence to obtain a first text sequence; and concatenating the quantization sequence and the category probability value of each negative sample data in sequence to obtain a second text sequence; fine-tuning the large language model using the first text sequence and the second text sequence.

7. The method according to claim 1 or 2, characterized in that, The method further comprises: adding the target audio data as negative sample data to the first training set to train a speech classification model using the first training set.

8. A data processing apparatus, characterized by, The device comprises: an audio quantizer configured to receive each sample data in the first training set and output a quantization sequence corresponding to the sample data; wherein the quantization sequence corresponding to a sample data comprises a classification result corresponding to each speech segment obtained by segmenting the sample data; the first training set comprises positive sample data and negative sample data, and the positive sample data and the negative sample data are audio data; a large language model configured to receive an existing sequence and output a target quantization sequence that is confused with the existing sequence; the existing sequence comprises a quantization sequence corresponding to a positive sample data and a quantization sequence corresponding to a negative sample data; a dictionary query module configured to obtain a first word vector sequence corresponding to the target quantization sequence by querying a quantization dictionary; the quantization dictionary comprises a mapping relationship between a word vector and a classification result of a speech segment; an audio dequantizer configured to receive the first word vector sequence and output target audio data corresponding to the first word vector sequence.

9. The apparatus of claim 8, wherein, The device further comprises a first training module configured to jointly train the audio quantizer and the audio dequantizer using the first training set; the first training module comprises: a first initialization submodule configured to initialize the audio quantizer and the audio dequantizer; The first iteration training submodule comprises a first acquisition unit, a loss calculation unit and an update iteration unit. The first acquisition unit is configured to take a piece of sample data from the first training set as input audio, divide the input audio into a plurality of speech segments according to a preset length, input the speech segments into an initialized audio quantizer in sequence, and predict a quantization sequence corresponding to the input audio; query the quantization dictionary to obtain a word vector corresponding to each classification result in the quantization sequence corresponding to the input audio, and sequentially form a second word vector sequence. The loss calculation unit is configured to input the second word vector sequence into an initialized audio dequantizer, predict a decoded audio, and calculate a first loss value according to the decoded audio and the input audio. The update iteration unit is configured to update model parameters of the audio quantizer and the audio dequantizer according to the first loss value, enter a next round of iteration, and obtain a trained audio quantizer and audio dequantizer when an iteration stop condition is met.

10. An electronic device, comprising: Comprise: A processor, a memory, a communication interface and a communication bus, the processor, the memory and the communication interface complete mutual communication through the communication bus; The memory is used to store at least one executable instruction, and the executable instruction makes the processor execute the steps of the data processing method in any one of claims 1 to 7.

Images