Speech model processing method, device and computer equipment

Abstract

The present invention discloses a speech model processing method, apparatus, and computer equipment. The method comprises: obtaining a pre-trained language model, an initial speech model, and training set data; constructing an auxiliary pooling module on the initial speech model to obtain a process speech model, wherein the auxiliary pooling module includes a local pooling block, which is used to adjust the pooling range of the speech frame input into the initial speech model and obtain local speech features corresponding to the speech frame within the pooling range, wherein the pooling range includes the speech frame and multiple adjacent frames of the speech frame; and performing distillation training on the process speech model using the pre-trained language model based on the training set data to obtain a target speech model. The present invention solves the technical problem of low accuracy of speech models obtained through training in related technologies.

Description

Technical Field

[0001] The present invention relates to the field of machine intelligence technology, and in particular to a speech model processing method, apparatus, and computer equipment. Background Art

[0002] In related technologies, semantic understanding of speech is increasingly required in more and more scenarios. Therefore, machine learning-based speech models (or speech understanding models) are required to fully encode and infer the semantic information in speech features. To obtain an accurate speech model, machine training requires a large amount of triple-labeled data consisting of speech, text, and semantic tags. However, the labeling process is complex, with many differences, long time consumption, and high cost. As a result, speech training can only be based on a small amount of complex labeled data, resulting in low accuracy of the trained speech model.

[0003] To address the above-mentioned problems, no effective solutions have been proposed so far. Summary of the Invention

[0004] Embodiments of the present invention provide a speech model processing method, apparatus, and computer device to at least solve the technical problem of low accuracy of speech models obtained through training in related technologies.

[0005] According to one aspect of an embodiment of the present invention, a speech model processing method is provided, comprising: obtaining a pre-trained language model, an initial speech model and training set data; constructing an auxiliary pooling module on the initial speech model to obtain a process speech model, wherein the auxiliary pooling module comprises a local pooling block, the local pooling block is used to adjust the pooling range of the speech frame input into the initial speech model, and obtain local speech features corresponding to the speech frame within the pooling range, wherein the pooling range includes the speech frame and multiple adjacent frames of the speech frame; based on the training set data, the process speech model is distilled and trained using the pre-trained language model to obtain a target speech model.

[0006] Optionally, constructing an auxiliary pooling module on the initial speech model includes: adding a global aggregation block and the local pooling block to the initial speech model, wherein the global aggregation block is used to perform global speech features of the speech frames input into the initial speech model within the global speech frame sequence; constructing a pooling fusion block for fusing the global aggregation block and the local pooling block, wherein the auxiliary pooling module includes: the global aggregation block, the local pooling block and the pooling fusion block.

[0007] Optionally, the method further includes: constructing the local pooling block in the following manner: obtaining the weight of the speech frame in the global speech frame sequence, and the window size used to represent the pooling range; and constructing the local pooling block based on the weight and the window size.

[0008] Optionally, based on the training set data, the pre-trained language model is used to perform distillation training on the process speech model to obtain a target speech model, including: based on the training set data, the pre-trained language model is used to perform distillation training on the process speech model to obtain a trained process speech model; and the part corresponding to the initial speech model is extracted from the trained process speech model to obtain the target speech model.

[0009] Optionally, based on the training set data, the process speech model is distilled and trained using the pre-trained language model to obtain a target speech model, including: constructing a comparison loss function for distilling the process speech model using the pre-trained language model; based on the training set data and the comparison loss function, the process speech model is distilled and trained using the pre-trained language model to obtain a target speech model.

[0010] Optionally, based on the training set data and the comparison loss function, the pre-trained language model is used to perform distillation training on the process speech model to obtain a target speech model, including: based on the training set data, determining the loss value corresponding to the comparison loss function obtained by performing distillation training on the process speech model using the pre-trained language model; adjusting the module parameters of the auxiliary pooling module based on the loss value, and adjusting the model parameters of the initial speech model to obtain the target speech model.

[0011] Optionally, adjusting the module parameters of the auxiliary pooling module based on the loss value and adjusting the model parameters of the initial speech model to obtain the target speech model includes: determining a back propagation algorithm corresponding to the back propagation from the output of the auxiliary pooling module to the input of the initial speech model; adjusting the module parameters of the auxiliary pooling module and adjusting the model parameters of the initial speech model based on the loss value and the back propagation algorithm to obtain the target speech model.

[0012] Optionally, based on the training set data and the comparison loss function, the pre-trained language model is used to perform distillation training on the process speech model to obtain a target speech model, including: dividing part of the training set data from the training set data; freezing the model parameters of the initial speech model in the process speech model to obtain a semi-frozen process speech model; based on the part of the training set data, the pre-trained language model is used to perform distillation training on the semi-frozen process speech model to obtain an optimized process speech model, wherein the module parameters of the auxiliary pooling module in the optimized process speech model are preliminarily optimized; based on the remaining training set data after the division in the training set data, the pre-trained language model is used to perform distillation training on the optimized process speech model to obtain the target speech model.

[0013] Optionally, after the process speech model is distilled and trained based on the training set data using the pre-trained language model to obtain the target speech model, the method further includes: receiving the target speech; inputting the target speech into the target speech model to obtain the semantic encoding result corresponding to the target speech.

[0014] According to another aspect of the present invention, a speech model processing method is provided, comprising: receiving a target speech; inputting the target speech into a target speech model to obtain a semantic encoding result corresponding to the target speech, wherein the target speech model is obtained by distilling and training an initial speech model having an auxiliary pooling module based on training set data using a pre-trained language model, the auxiliary pooling module comprising a local pooling block, the local pooling block being used to adjust the pooling range of the speech frame input into the initial speech model, and obtaining local speech features corresponding to the speech frame within the pooling range, the pooling range comprising the speech frame and multiple adjacent frames of the speech frame.

[0015] According to another aspect of the present invention, a speech model processing method is provided, comprising: receiving a customer service speech, wherein the customer service speech carries a question requested to the customer service; inputting the customer service speech into a target speech model to obtain a semantic encoding result corresponding to the customer service speech, wherein the target speech model is obtained by distilling and training an initial speech model having an auxiliary pooling module based on training set data using a pre-trained language model, the auxiliary pooling module comprising a local pooling block, the local pooling block being used to adjust the pooling range of the speech frame input into the initial speech model, and obtaining local speech features corresponding to the speech frame within the pooling range, the pooling range comprising the speech frame and multiple adjacent frames of the speech frame; determining the question and the answer corresponding to the question based on the semantic encoding result; and feeding back an intelligent speech including the answer.

[0016] According to another aspect of the present invention, a speech model processing device is provided, comprising: an acquisition module for acquiring a pre-trained language model, an initial speech model and training set data; a construction module for constructing an auxiliary pooling module on the initial speech model to obtain a process speech model, wherein the auxiliary pooling module includes a local pooling block, the local pooling block is used to adjust the pooling range of the speech frame input into the initial speech model, and obtain local speech features corresponding to the speech frame within the pooling range, and the pooling range includes the speech frame and multiple adjacent frames of the speech frame; a training module for performing distillation training on the process speech model using the pre-trained language model based on the training set data to obtain a target speech model.

[0017] According to another aspect of the present invention, a computer-readable storage medium is provided, which includes a stored program, wherein when the program is run, the device where the computer-readable storage medium is located is controlled to execute any one of the above-mentioned speech model processing methods.

[0018] According to another aspect of the present invention, a computer device is provided, comprising: a memory and a processor, wherein the memory stores a computer program; the processor is configured to execute the computer program stored in the memory, wherein when the computer program is executed, the processor executes any one of the above-described speech model processing methods.

[0019] In an embodiment of the present invention, a pre-trained language model is used to perform distillation training on a process speech model including an initial speech model, thereby achieving the purpose of improving the speech model training efficiency, and constructing an auxiliary pooling module on the initial speech model that can adjust the pooling range of the speech frame accordingly, so that the speech frames input to the speech model can be accurately aligned with the text, that is, alignment between the speech features and the text features is achieved, thereby achieving the technical effect of ensuring the accuracy of the trained speech model on the basis of ensuring the speech model training efficiency, and thus solving the technical problem of low accuracy of the trained speech model in the related art. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0021] Figure 1 A hardware structure block diagram of a computer terminal for implementing a speech model processing method is shown;

[0022] Figure 2is a flow chart of a speech model processing method 1 according to embodiment 1 of the present invention;

[0023] Figure 3 is a flow chart of a second method for processing a speech model according to embodiment 1 of the present invention;

[0024] Figure 4 is a flow chart of a third method for processing a speech model according to embodiment 1 of the present invention;

[0025] Figure 5 is a schematic diagram of a text-speech teacher-student learning system provided according to an optional embodiment of the present invention;

[0026] Figure 6 1 is a structural block diagram of a voice module processing device according to an embodiment of the present invention;

[0027] Figure 7 This is a structural block diagram of a second voice module processing device according to an embodiment of the present invention;

[0028] Figure 8 This is a structural block diagram of a third voice module processing device according to an embodiment of the present invention;

[0029] Figure 9 It is a structural block diagram of a computer terminal according to an embodiment of the present invention. DETAILED DESCRIPTION

[0030] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0031] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0032] First, some nouns or terms that appear in the description of the embodiments of the present application are subject to the following interpretations:

[0033] Knowledge distillation generally involves first training a high-performing teacher model (the large model). This model's output is then used as soft labels, along with the data's true labels as hard labels. These two are then combined to train a student model (the small model). This method compresses the model to a very small size, resulting in a small model with performance similar to that of a more complex model, but with significantly less complexity and computational effort.

[0034] Transcripts: The original meaning of transcription is the process of faithfully converting the characters of one phonetic writing system into the characters of another phonetic writing system according to a character comparison table (including diacritical marks for basic characters and single phonemes represented by double characters). Transcription is actually the transformation of one form of content into another. When applied to speech, a transcript is the text obtained by converting speech, where the speech can be real-time or recorded.

[0035] Automatic Speech Recognition (ASR), in simple terms, converts speech signals into text and outputs it (displayed on a screen). During this process, the machine doesn't know the specific meaning of the speech; it simply converts the two signals. Furthermore, according to the concept of ASR, the initial state of the signal is speech, so ASR can also be understood as making the machine hear.

[0036] A kernel function, a statistical term, refers to a support vector machine (SVM) that maps the input space to a high-dimensional feature space through a nonlinear transformation. The dimensionality of this feature space can be very high. If the SVM solver relies solely on the inner product operation, and there exists a function in the low-dimensional input space that is exactly equal to this inner product in the high-dimensional space, the SVM can directly obtain the inner product of the nonlinear transformation from this function, significantly simplifying the computation. This function is called a kernel function.

[0037] In most cases, speech signals are non-stationary and time-varying. However, within a short timeframe, such as 10-30 ms, speech signals can be considered steady and time-invariant. Therefore, a Fourier transform is performed on short timeframes, and adjacent frames are connected to obtain a good approximation of the signal's frequency profile. During speech signal processing, to reduce the impact of the overall non-stationary and time-varying nature of the speech signal, the signal is segmented. Each segment is called a frame, or speech frame, and the frame length is generally 25 ms. Furthermore, to ensure smooth transitions between frames and maintain continuity, framing generally uses overlapping segments, ensuring that adjacent frames overlap to a certain extent, with any missing segments padded with zeros. The time difference between the starting positions of two adjacent frames is called the frame shift, which is typically 10 ms.

[0038] Windowing: After dividing the speech signal into speech frames, each frame can be multiplied by a window function, such as a Hamming window, to increase the continuity of the left and right ends of the frame, offset the Fourier transform assumption (data is infinite), and reduce spectral leakage.

[0039] Contrastive Loss is usually used in the model training of the siamese network to effectively process the relationship between data pairs in the siamese network. The contrast loss function is mainly used in dimensionality reduction, that is, the two samples that were originally similar are still similar in the feature space after dimensionality reduction (feature extraction); and the two samples that were originally dissimilar are still dissimilar in the feature space after dimensionality reduction. In the embodiment of the present application, the focus is on the alignment relationship between the speech frame and the text. Therefore, in the process of model training, the token instance contrast loss (Token-Instance Contrastive Loss, abbreviated as TICL) is used to characterize, that is, the alignment parameters based on the word-frame relationship and the sentence relationship are used for knowledge transfer between text and speech models.

[0040] Pre-training, in simple terms, involves using as much training data as possible to extract as many common features as possible, thereby reducing the model's learning burden for specific tasks. Pre-training has two key characteristics: 1. Model parameters are no longer randomly initialized, but are instead pre-trained through a series of tasks. 2. The training task is broken down into two steps: learning commonalities and learning specific features. Therefore, pre-training can be seen as a decomposition of the learning task. Pre-training typically involves pooling a large amount of inexpensively collected training data, using some pre-training method to learn the commonalities. These commonalities are then "transplanted" into a task-specific model, and then "fine-tuned" using a small amount of labeled data from the relevant domain. This allows the model to "learn" only the "commonalities" needed for the specific task.

[0041] The self-attention module, which implements the self-attention mechanism, focuses limited attention on key information. In the Transformer model, the input and output of the self-attention module are both vector sequences, with each vector representing an item. The module performs an attention operation on the input sequence itself, aggregating global information based on the complete input sequence to update each item in the sequence. This allows for modeling the global correlation between items. For example, in natural language processing, it can be used to assess which words are most likely to form a sentence.

[0042] Pooling functions are typically used to construct the pooling layer in convolutional neural networks. The input to the pooling layer generally comes from the previous convolutional layer. Common pooling functions include max pooling and average pooling. Their forward computation is very simple: max pooling simply finds the maximum value, while average pooling finds the average value. The main function of the pooling function is to provide strong robustness (for example, max pooling takes the maximum value in a small area. If other values ​​in this area change slightly, or the image is slightly shifted, the pooled result remains unchanged). It also reduces the number of parameters, preventing overfitting. This reduction in parameters also helps with computation. Because pooling layers generally have no parameters, backpropagation only requires the derivative of the input parameters, eliminating the need for weight updates.

[0043] Receptive Field: In the Convolutional Neural Networks (CNN) structure, the value of each output node of the fully connected layer (FC layer) depends on all the inputs of the FC layer, while the value of each output node of the convolutional layer (CONV layer) depends only on an area of ​​the CONV layer input. Other input values ​​outside this area will not affect the output value. This area is the receptive field. For example, in a convolutional neural network that recognizes objects from images, the size of the area mapped by the pixels on the feature map output by each layer on the input image. A more popular explanation is that a point on the feature map corresponds to an area on the input image.

[0044] The backpropagation algorithm (BP) is a learning algorithm suitable for multi-layer neural networks and is based on the gradient descent method. The input-output relationship of a BP network is essentially a mapping relationship: the function performed by a BP neural network with n inputs and m outputs is a continuous mapping from n-dimensional Euclidean space to a finite field in m-dimensional Euclidean space. This mapping is highly nonlinear. Its information processing capability comes from the multiple compounding of simple nonlinear functions, resulting in a strong function reproduction ability. This is the basis for the application of the BP algorithm. The backpropagation algorithm mainly consists of two steps (excitation propagation and weight update) that are repeatedly iterated until the network's response to the input reaches the predetermined target range. The learning process of the BP algorithm consists of a forward propagation process and a backward propagation process. During the forward propagation process, the input information passes through the input layer and the hidden layer, and is processed layer by layer and transmitted to the output layer. If the desired output value is not obtained at the output layer, the sum of the squares of the error between the output and the desired value is taken as the objective function. Backpropagation is then performed, and the partial derivatives of the objective function with respect to each neuron weight are calculated layer by layer. This gradient of the objective function with respect to the weight vector is then formed and used as the basis for modifying the weights. Network learning is completed during the weight modification process. When the error reaches the desired value, network learning ends.

[0045] Connectionist temporal classification (CTC) is an algorithm commonly used in fields such as speech recognition and text recognition. It is used to solve the problem of different input and output sequence lengths and misalignment. During model training, it can be used as a loss function to calculate the loss value.

[0046] Example 1

[0047] According to an embodiment of the present invention, a method embodiment of a speech model processing method is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0048] The method embodiment provided in Example 1 of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG1 shows a hardware structure block diagram of a computer terminal for implementing a speech model processing method. Figure 1 As shown, the computer terminal 10 (or mobile device) may include one or more processors (shown as 102a, 102b, ..., 102n in the figure, and the processor may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission device for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It will be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown.

[0049] It should be noted that the one or more processors and / or other data processing circuits described above may generally be referred to herein as "data processing circuitry". The data processing circuitry may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuitry may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10 (or mobile device). As described in the embodiments of the present application, the data processing circuitry serves as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0050] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the speech model processing method in the embodiment of the present invention. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, implementing the vulnerability detection method of the above-mentioned application. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely located relative to the processor, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0051] The transmission device is used to receive or send data via a network. Specific examples of the aforementioned network may include a wireless network provided by the communications provider of the computer terminal 10. In one embodiment, the transmission device includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, the transmission device may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0052] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 (or mobile device).

[0053] In the above operating environment, in response to the above problems mentioned in this application, this application provides the following Figure 2 The speech model processing method shown. Figure 2 : is a flow chart of a speech model processing method 1 according to embodiment 1 of the present invention, such as Figure 2 As shown, the process includes the following steps:

[0054] S202, obtaining a pre-trained language model, an initial speech model, and training set data;

[0055] As an optional embodiment, the method of this embodiment can be executed by a terminal or server used to process speech models. For example, when applied to a terminal used to perform speech model processing, it can easily implement a model for simple speech scenarios. For another example, when applied to a server, it can utilize the server's abundant computing resources or a relatively larger and more accurate pre-trained language model to more accurately train the target speech model.

[0056] It should be noted that the aforementioned terminals can be of various types, for example, mobile terminals with certain computing capabilities, or fixed computers with model training capabilities. The aforementioned servers can also be of various types, for example, local servers or virtual cloud servers. Depending on the computing power, the servers can be single computers or computer clusters integrating multiple computers.

[0057] As an optional embodiment, the pre-trained language model can be a language model trained on a large amount of text, which is used to semantically encode the text. Since text training is more easily accessible with a rich corpus than speech training, an accurate language model can be trained based on text, resulting in a pre-trained language model that meets the requirements.

[0058] As an optional embodiment, the above-mentioned initial speech model can also be a speech model obtained by training based on certain speech corpus, that is, the initial speech model is not completely a default initialized speech model. Therefore, it can also improve the efficiency of speech training to a certain extent.

[0059] As an optional embodiment, the above-mentioned training set data can be parallel training corpora in which speech and text are aligned, and these parallel training corpora can be annotated in advance. It should be noted that the annotation described here refers to sentences, that is, in a sentence, each word or character corresponds to which speech frames in the speech. In addition, the data in the training set data can be Chinese or non-Chinese. Relatively speaking, the method proposed in this application can train the speech model to adaptively align text and speech frames, and therefore, it is more suitable for some non-Chinese application scenarios where the alignment of speech and text is not very good.

[0060] S204: constructing an auxiliary pooling module on the initial speech model to obtain a process speech model, wherein the auxiliary pooling module includes a local pooling block, the local pooling block is used to adjust the pooling range of the speech frame input into the initial speech model, and obtain local speech features corresponding to the speech frame within the pooling range, wherein the pooling range includes the speech frame and multiple adjacent frames of the speech frame;

[0061] As an optional embodiment, the above-mentioned local pooling block adaptively adjusts the range of speech frames associated with the upper and lower parts of the speech frame in the input initial speech model to achieve the semantic influence of the speech frame above and below itself. For example, due to the difference between speech and text (for example, the length of the pronunciation syllables in the speech is different, so that a word in the text may need to correspond to multiple speech frames, or a syllable corresponds to multiple speech frames, or a pronunciation phoneme may correspond to multiple speech frames), it is not a one-to-one relationship between speech frames and texts, or a fixed many-to-one relationship, that is, not one speech frame corresponds to a word or character in a text, nor are multiple fixed speech frames corresponding to words, characters, syllables, or phonemes in a text. It is necessary to adaptively adjust the alignment relationship between text features and speech features. In other words, the above-mentioned adaptive adjustment is to align according to specific pronunciation characteristics.

[0062] As an optional embodiment, there are many ways to construct the above-mentioned local pooling block. For example, the local pooling block can be constructed in the following way: obtain the weight of the speech frame in the global speech frame sequence, and the window size used to represent the pooling range; based on the weight and window size, construct the local pooling block. Among them, the weight of the speech frame in the global speech frame sequence represents the influence of the speech frame on the semantics of the whole sentence, and the window size represents the adjacent speech frames that have an impact on the speech frame. Therefore, based on the influence of the speech frame itself and the influence of the context on itself, the speech features represented by the speech frame itself can be accurately represented. In the above way, by using the local mutual influence between speech frames, the speech features represented by the speech frame in its own local area can be obtained more comprehensively and accurately.

[0063] As an optional embodiment, when constructing an auxiliary pooling module on the initial speech model, in order to make the information expressed by the auxiliary pooling module more complete and enable the auxiliary pooling module to adapt to the needs of various speech recognition tasks, the auxiliary pooling module may include, in addition to the above-mentioned local pooling block, a global aggregation block. For example, constructing an auxiliary pooling module on the initial speech model may include: adding a global aggregation block and a local pooling block to the initial speech model, wherein the global aggregation block is used to calculate the global speech features of the speech frame input into the initial speech model within the global speech frame sequence; constructing a pooling fusion block for fusing the global aggregation block and the local pooling block, wherein the auxiliary pooling module includes: a global aggregation block, a local pooling block and a pooling fusion block. Therefore, by constructing a global aggregation block and a local pooling block in the auxiliary pooling module, since both global features and local features are taken into account, the features of the speech frame can be expressed more comprehensively. Since the speech object to be recognized is generally a sentence, taking a sentence as an example, the above-mentioned global speech frame sequence refers to the speech frame sequence included in the sentence. When the above-mentioned local pooling block corresponds to the adjusted pooling range of the speech frame input into the initial speech model, the speech frame involved can be any frame in the speech frame sequence. Therefore, for a sentence of speech, the global speech features of the global aggregation block represent sentence-level information, and the local speech features of the local pooling block represent context-related frame-level information. The global speech features corresponding to the above-mentioned global aggregation block and the local speech features corresponding to the local pooling block are then fused through the pooling fusion block. The influence of the speech frame on the semantics of the entire sentence and the influence of other adjacent speech frames on the semantics of the speech frame can be obtained, so that the speech features (vectors) output from the auxiliary pooling module can fully represent the semantics of the input speech.

[0064] S206 , based on the training set data, using the pre-trained language model to perform distillation training on the process speech model to obtain a target speech model.

[0065] As an optional embodiment, when a pre-trained language model is used to perform distillation training on the process speech model based on the training set data to obtain a target speech model, since the target object of the training is the initial speech model, the auxiliary pooling module is used to assist in the training of the initial speech model. Therefore, after the pre-trained language model is used to perform distillation training on the process speech model based on the training set data to obtain the trained process speech model, the portion corresponding to the initial speech model can be extracted from the trained process speech model to obtain the target speech model. Subsequently, testing or application can be performed directly based on the obtained target speech model. It should be noted that when testing or using the obtained target speech model, it is not necessary to discard the auxiliary pooling module. That is, when testing or using the target speech model, the trained process speech model can be used. Of course, since the initial speech model is relatively the most important part, after training, that is, when a portion corresponding to the initial speech model that meets the requirements is obtained, testing and application can be performed directly based on this portion. When using direct testing and application, if the accuracy is met, the model has fewer components and is simpler, more portable, and more efficient.

[0066] As an optional embodiment, when a pre-trained language model is used to perform distillation training on a process speech model based on training set data to obtain a target speech model, model training can be performed based on different constructed loss functions. This embodiment adopts distillation training based on a teacher-student model, so a comparison loss function with simple and clear representation can be used as a loss function for training. Therefore, based on the training set data, a pre-trained language model is used to perform distillation training on a process speech model to obtain a target speech model, which can include the following processing: constructing a comparison loss function for performing distillation training on a process speech model using a pre-trained language model; based on the training set data and the comparison loss function, the pre-trained language model is used to perform distillation training on the process speech model to obtain a target speech model. The comparison loss function is used to compare the difference between the output of the pre-trained language model as the teacher model and the output of the speech model as the student model. Through intuitive comparison, the parameters of the speech model as the student model are adjusted, which can improve the training efficiency of the model.

[0067] As an optional embodiment, based on the training set data and the comparison loss function, a pre-trained language model is used to perform distillation training on the process speech model to obtain a target speech model, including: determining the loss value corresponding to the comparison loss function obtained by distilling the process speech model using the pre-trained language model based on the training set data; adjusting the module parameters of the auxiliary pooling module based on the loss value, and adjusting the model parameters of the initial speech model to obtain the target speech model. Based on the loss value corresponding to the comparison loss function, we can directly focus on the difference between the encoding vector of the pre-trained language model and the encoding vector of the speech model, that is, the difference between the semantics recognized from the pre-trained language model and the semantics recognized from the speech model. According to the above difference, the module parameters of the auxiliary pooling module and the model parameters of the initial speech model are efficiently adjusted to obtain the target speech model. It should be noted that when adjusting the module parameters of the auxiliary pooling module and adjusting the model parameters of the initial speech model, it may be completed in multiple iterations.

[0068] As an optional embodiment, the module parameters of the auxiliary pooling module are adjusted based on the loss value, and the model parameters of the initial speech model are adjusted to obtain the target speech model, including: determining the back propagation algorithm corresponding to the back propagation from the output of the auxiliary pooling module to the input of the initial speech model; based on the loss value and the back propagation algorithm, the module parameters of the auxiliary pooling module are adjusted, and the model parameters of the initial speech model are adjusted to obtain the target speech model.

[0069] Back propagation algorithm is a kind of learning algorithm of multi-layer neural network, mainly by two links (incentive propagation, weight update) repeated cycle iteration, until the response to input of network reaches predetermined target range.The learning process of back propagation algorithm includes forward propagation process and back propagation process.In forward propagation process, input information passes through input layer through hidden layer, is processed layer by layer and is passed to output layer.If expected output value is not obtained in output layer, then the square sum of output and expected error is taken as objective function, proceed to back propagation, obtain the partial derivative of objective function to each neuron weight layer by layer, constitute the ladder amount of objective function to weight vector, as the basis of modifying weight, the learning of network is completed in weight modification process.When error reaches desired value, network learning terminates.In embodiment of the present application, forward propagation is namely from the data in training set data being input to speech model, from speech model output, the output of speech model is input to auxiliary pooling module, finally output through auxiliary pooling module, obtain the output of whole process speech model. Backpropagation is the reverse of the forward propagation described above. This involves reversing the model's parameters layer by layer, starting with the output of the speech model. This involves modifying the parameters of the auxiliary pooling module and then reversing the parameters of the initial speech model. Training ends after multiple rounds of forward and backward propagation based on the training data, until the loss between the speech encoding vector output by the pre-trained language model and the speech encoding vector output by the speech model meets the predetermined loss range, achieving the desired training goal.

[0070] As an optional embodiment, when a pre-trained language model is used to perform distillation training on the process speech model based on the training set data and the comparison loss function to obtain the target speech model, in order to make the training of the target speech model more efficient, the entire training process can be divided into stages. For example, since the auxiliary pooling module is a brand-new module, in order to avoid causing excessive interference to the initial speech model that already has a certain speech recognition ability in the early stage of training, that is, to avoid excessive adjustment of the model parameters of the speech module, a partial training method can be adopted. For example, the model parameters of the initial speech model can be frozen first. When the training of the auxiliary pooling module reaches a certain level, the freezing of the initial speech model is completely released to achieve an overall adjustment of the parameters of the entire process speech model, thereby achieving the training goal.

[0071] Based on the training set data and the comparison loss function, the process speech model is distilled and trained using a pre-trained language model to obtain a target speech model. The following processing methods can be used: divide part of the training set data from the training set data; freeze the model parameters of the initial speech model in the process speech model to obtain a semi-frozen process speech model; based on part of the training set data, use the pre-trained language model to distill and train the semi-frozen process speech model to obtain an optimized process speech model, wherein the module parameters of the auxiliary pooling module in the optimized process speech model are preliminarily optimized; based on the remaining training set data after the division of the training set data, use the pre-trained language model to distill and train the optimized process speech model to obtain the target speech model. The above-mentioned staged training method can be used to complete the training of each part of the entire process speech model in a targeted manner, avoiding interference with other parts. When each part reaches a certain level, it is adjusted accordingly based on the entire model, effectively achieving the goal of improving the training efficiency of the entire model while ensuring the accuracy of model training.

[0072] As an optional embodiment, after distilling and training the process speech model using a pre-trained language model based on training set data to obtain a target speech model, the method further includes: receiving a target speech; inputting the target speech into the target speech model, and obtaining a semantic encoding result corresponding to the target speech. After obtaining the target speech model, various semantic scenarios can be recognized based on the obtained target speech model to obtain corresponding semantic encoding results, and then various scenario tasks can be completed based on the obtained semantic encoding results. For example, offline question answering and online real-time consultation can be carried out based on the obtained semantic encoding results.

[0073] In an embodiment of the present invention, a pre-trained language model is used to perform distillation training on a process speech model including an initial speech model, thereby achieving the purpose of improving the speech model training efficiency, and constructing an auxiliary pooling module on the initial speech model that can adjust the pooling range of the speech frame accordingly, so that the speech frames input to the speech model can be accurately aligned with the text, that is, alignment between the speech features and the text features is achieved, thereby achieving the technical effect of ensuring the accuracy of the trained speech model on the basis of ensuring the speech model training efficiency, and thus solving the technical problem of low accuracy of the trained speech model in the related art.

[0074] Figure 3 : is a flow chart of the second speech model processing method according to embodiment 1 of the present invention, as shown in FIG. Figure 3 As shown, the process includes the following steps:

[0075] S302, receiving target speech;

[0076] S304: Input the target speech into the target speech model to obtain the semantic encoding result corresponding to the target speech, wherein the target speech model is obtained by distilling and training the initial speech model constructed with an auxiliary pooling module based on the training set data using a pre-trained language model. The auxiliary pooling module includes a local pooling block, which is used to adjust the pooling range of the speech frame input into the initial speech model and obtain the local speech features corresponding to the speech frame within the pooling range. The pooling range includes the speech frame and multiple adjacent frames of the speech frame.

[0077] Through the above steps, the target speech model obtained in the above speech model processing method 1 is used to perform semantic recognition on the target speech to obtain the corresponding semantic encoding result. Due to the adaptive alignment of the target speech model between text features and speech features, the obtained semantic encoding result can be made more accurate.

[0078] Figure 4 : is a flow chart of the third method for processing a speech model according to embodiment 1 of the present invention, as shown in FIG. Figure 4 As shown, the process includes the following steps:

[0079] S402, receiving a customer service voice message, wherein the customer service voice message carries a question for the customer service;

[0080] S404: Input the customer service speech into the target speech model to obtain a semantic encoding result corresponding to the customer service speech, wherein the target speech model is obtained by distilling and training an initial speech model with an auxiliary pooling module based on the training set data using a pre-trained language model. The auxiliary pooling module includes a local pooling block, which is used to adjust the pooling range of the speech frame input into the initial speech model and obtain local speech features corresponding to the speech frame within the pooling range, where the pooling range includes the speech frame and multiple adjacent frames of the speech frame.

[0081] S406, determining the question and the answer corresponding to the question based on the semantic coding result;

[0082] S408, feeding back the intelligent voice including the answer.

[0083] Through the above steps, the target speech model obtained in the above speech model processing method 1 is used to perform semantic recognition on the customer service speech containing questions to be consulted to the customer service, and the corresponding semantic encoding results are obtained. The answer corresponding to the question is fed back based on the obtained semantic encoding results. Due to the adaptive alignment between the text features and the speech features of the target speech model, the obtained semantic encoding is more accurate, and thus the answer based on the question feedback can also be more accurate.

[0084] As mentioned above, for numerous intelligent voice scenarios, such as audio and video conferencing, smart phone customer service quality inspection and pre-calling, etc., the results of speech recognition need to be intelligently parsed and organized. This requires that the speech model can fully encode and infer the semantic information in the speech features. However, training the speech model requires a large amount of triple-labeled data consisting of speech, text, and semantic labels. The labeling process is prone to many discrepancies, is time-consuming, and expensive. At the same time, the speech encoder in the speech understanding model often focuses on local features and cannot encode more fine-grained information, making it impossible to effectively derive high-level semantic content that focuses more on global information.

[0085] To address these issues, one possible solution is to introduce a pre-trained text-based language model as a teacher model and use techniques such as knowledge distillation to transfer knowledge from the text model to the speech model. However, due to the inherent differences between speech and text modalities (for example, speech sequences are often longer than text; speech features tend to focus more locally, while text features are more global), knowledge transfer is often not ideal.

[0086] One possible solution to this problem is to use a convolutional network with a fixed kernel size to further downsample speech features, reducing their length and thus minimizing the fine-grained differences with text features. However, because the correspondence between speech frame-level features and text words can be variable, this fixed-length kernel function cannot fully capture the speech-to-text alignment.

[0087] Given this, this optional embodiment proposes a multi-granularity pooling module to improve the performance of speech models in speech comprehension tasks. In this optional embodiment, a new pooling module (corresponding to the aforementioned local pooling block) can adaptively extract and align multiple fine-grained information from speech based on the features of a text-based pre-trained language model, thereby more effectively transferring knowledge. This text-to-speech knowledge transfer architecture effectively addresses the issue of fine-grained discrepancies when aligning textual knowledge to speech.

[0088] In this optional embodiment, a pre-trained language model is used as the teacher model, and knowledge distillation technology is used to transfer the knowledge in the text model (i.e., the pre-trained language model) to the speech model. A new pooling module is added to the speech model, which serves as the student model, so that the speech frames in the speech can be adaptively aligned with the text information. Based on this technical solution, the training method based on the teacher-student model can effectively improve training efficiency. Based on the newly added pooling module, the accuracy of training and the precision of alignment between text and speech can be guaranteed. Therefore, it is ultimately possible to efficiently train and obtain an accurate speech model.

[0089] Figure 5 Schematic diagram of a text-speech teacher-student learning system according to an optional embodiment of the present invention. Figure 5 As shown in the figure, the text-speech teacher-student learning system is a cross-modal knowledge distillation system, with a pre-trained language model as the teacher model and a speech model as the student model. These two models are pre-trained using a large amount of text and speech data, respectively. Then, using the contrastive learning method, partial speech-text parallel data is used for knowledge distillation training. For example, speech is input into the speech model and transcripts are input into the pre-trained language model. The purpose is to transfer the knowledge in the pre-trained language model to the speech model. To achieve this goal, during the training process, an auxiliary pooling network (corresponding to the above-mentioned auxiliary pooling module) is added on the basis of the speech model to better align it with the pre-trained language model.

[0090] The above-mentioned auxiliary pooling layer (APLY) is a multi-fine-grained pooling network based on adaptive pooling technology. The operation of APLY in the teacher-student learning system is explained below in conjunction with the attached figure of the above-mentioned text-to-speech teacher-student learning system. The basic student model is a pre-trained speech encoder (such as Wav2Vec2). During the training process, the Token-Instance Contrastive Loss (TICL) function is combined with the CTC loss function as the training loss function, with the aim of transferring knowledge from the pre-trained language model (such as BERT) as the teacher model to the student model. In order to better represent the sequence information of speech, it is necessary to model the local information (local pooling window) and global information (global aggregation) of the speech features at the same time. Among them, local information mainly refers to a certain number of adjacent frames before and after the current frame, and global information refers to the information contained in the current sentence. To this end, the proposed APLY can include two components: 1) a global aggregation block implemented as a self-attention module, where the query vector is the average pooled vector of the entire sequence; 2) an adaptive pooling block (Adaptive pooling) around the context pooling of each frame (i.e., the local pooling block referred to above) to encode local information. It is characterized by the main use of the pooling function as an agent for passing feedback to the speech encoder. Specifically, an adaptive context pooling block is used to adaptively parse the receptive field of the speech encoder, thereby reducing the difference between the two modalities. Then, pooling fusion is performed, which is a fusion module similar to gating (corresponding to the pooling fusion block (PoolingFusion) mentioned above) to fuse the outputs of the global and local encoders.

[0091] (1) We propose to use an auxiliary pooling network to transfer knowledge from text to speech features, that is, to complete the knowledge distillation from text to speech, thereby improving the ability of speech features to encode semantic information. The auxiliary pooling network can automatically learn the fine-grained local features for the target task, while also integrating global information to enhance the multi-granularity encoding capability of features. The auxiliary pooling network can play a role in better aligning features in the trans-membrane (text-speech) knowledge distillation framework.

[0092] (2) It is proposed to use an adaptive context pooling module as a receiver of speech features for textual knowledge, and adaptively change the pooling range according to the alignment relationship between speech features and textual features. Therefore, through this adaptive context pooling module, adaptive local pooling can be used to encode local information, that is, it can better encode multi-granularity speech information, allowing the speech model itself to learn from the training data what granularity to use to encode local information, and thus better align with textual knowledge. Therefore, it can be used to better inherit the semantic knowledge transferred from the pre-trained language model, thereby reducing the dependence on labeled data.

[0093] It should be noted that for the aforementioned method embodiments, for simplicity of description, they are all expressed as a series of action combinations. However, those skilled in the art should be aware that the present invention is not limited by the order of the actions described, because according to the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0094] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a computer-readable storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods of various embodiments of the present invention.

[0095] Example 2

[0096] According to an embodiment of the present invention, a device for implementing the above-mentioned speech model processing method 1 is also provided. Figure 6 : is a structural block diagram of a voice module processing device according to an embodiment of the present invention. Figure 6 As shown, the device includes: an acquisition module 60, a construction module 62 and a training module 64, and the device is described from the front.

[0097] An acquisition module 60 is used to acquire a pre-trained language model, an initial speech model and training set data; a construction module 62 is connected to the above-mentioned acquisition module 60, and is used to construct an auxiliary pooling module on the initial speech model to obtain a process speech model, wherein the auxiliary pooling module includes a local pooling block, and the local pooling block is used to adjust the pooling range of the speech frame input into the initial speech model, and obtain the local speech features corresponding to the speech frame within the pooling range, and the pooling range includes the speech frame and multiple adjacent frames of the speech frame; a training module 64 is connected to the above-mentioned construction module 62, and is used to perform distillation training on the process speech model based on the training set data using the pre-trained language model to obtain a target speech model.

[0098] It should be noted that the acquisition module 60, construction module 62, and training module 64 described above correspond to steps S202 to S206 in Example 1. The examples and application scenarios implemented by the modules and corresponding steps are the same, but are not limited to the contents disclosed in Example 1. It should be noted that the modules described above, as part of the apparatus, can be run in the computer terminal 10 provided in Example 1.

[0099] According to an embodiment of the present invention, a device for implementing the second speech model processing method is also provided. Figure 7 : is a structural block diagram of a second voice module processing device according to an embodiment of the present invention. Figure 7 As shown, the device includes: a first receiving module 70 and a first processing module 72. The device is described below.

[0100] The first receiving module 70 is used to receive the target speech; the first processing module 72 is connected to the above-mentioned first receiving module 70, inputs the target speech into the target speech model, and obtains the semantic encoding result corresponding to the target speech, wherein the target speech model is based on the training set data, and adopts the pre-trained language model to distill and train the initial speech model constructed with the auxiliary pooling module. The auxiliary pooling module includes a local pooling block, and the local pooling block is used to adjust the pooling range of the speech frame input into the initial speech model, and obtain the local speech features corresponding to the speech frame within the pooling range. The pooling range includes the speech frame and multiple adjacent frames of the speech frame.

[0101] It should be noted that the first receiving module 70 and the first processing module 72 correspond to steps S302 to S304 in Example 1. The examples and application scenarios implemented by the above modules and the corresponding steps are the same, but are not limited to the contents disclosed in Example 1. It should be noted that the above modules, as part of the device, can be run in the computer terminal 10 provided in Example 1.

[0102] According to an embodiment of the present invention, a device for implementing the third method for processing the speech model is also provided. Figure 8 : is a structural block diagram of a third voice module processing device according to an embodiment of the present invention. Figure 8 As shown, the device includes: a second receiving module 80, a second processing module 82, a determination module 84 and a feedback module 86. The device is described below.

[0103] The second receiving module 80 is used to receive customer service voice, wherein the customer service voice carries questions requested for consultation to the customer service; the second processing module 82 is connected to the above-mentioned second receiving module 80, and is used to input the customer service voice into the target voice model to obtain the semantic encoding result corresponding to the customer service voice, wherein the target voice model is based on the training set data, and the initial voice model constructed with the auxiliary pooling module is trained by distillation using a pre-trained language model, and the auxiliary pooling module includes a local pooling block, and the local pooling block is used to adjust the pooling range of the voice frame input into the initial voice model, and obtain the local voice features corresponding to the voice frame within the pooling range, and the pooling range includes the voice frame and multiple adjacent frames of the voice frame; the determination module 84 is connected to the above-mentioned second processing module 82, and determines the question and the answer corresponding to the question based on the semantic encoding result; the feedback module 86 is connected to the above-mentioned determination module 84, and is used to feedback the intelligent voice including the answer.

[0104] Example 3

[0105] The embodiment of the present invention can provide a computer terminal, which can be any computer terminal device in a computer terminal group. Optionally, in this embodiment, the computer terminal can also be replaced by a terminal device such as a mobile terminal.

[0106] Optionally, in this embodiment, the computer terminal may be located in at least one network device among a plurality of network devices of a computer network.

[0107] In this embodiment, the above-mentioned computer terminal can execute the program code of the following steps in the speech model processing method of the application: obtaining a pre-trained language model, an initial speech model and training set data; constructing an auxiliary pooling module on the initial speech model to obtain a process speech model, wherein the auxiliary pooling module includes a local pooling block, and the local pooling block is used to adjust the pooling range of the speech frame input into the initial speech model, and obtain the local speech features corresponding to the speech frame within the pooling range, and the pooling range includes the speech frame and multiple adjacent frames of the speech frame; based on the training set data, the pre-trained language model is used to perform distillation training on the process speech model to obtain the target speech model.

[0108] Optionally, Figure 9 1 is a block diagram of a computer terminal according to an embodiment of the present invention. Figure 9As shown, the computer terminal may include: one or more (only one is shown in the figure) processors 92, memory 94, etc.

[0109] Among them, the memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the speech model processing method and device in the embodiments of the present invention. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, realizing the above-mentioned speech model processing method. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely located relative to the processor, and these remote memories can be connected to the computer terminal via a network. Examples of the above-mentioned network include but are not limited to the Internet, corporate intranet, local area network, mobile communication network and combinations thereof.

[0110] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: obtain a pre-trained language model, an initial speech model and training set data; build an auxiliary pooling module on the initial speech model to obtain a process speech model, wherein the auxiliary pooling module includes a local pooling block, the local pooling block is used to adjust the pooling range of the speech frame input into the initial speech model, and obtain the local speech features corresponding to the speech frame within the pooling range, and the pooling range includes the speech frame and multiple adjacent frames of the speech frame; based on the training set data, the pre-trained language model is used to perform distillation training on the process speech model to obtain a target speech model.

[0111] Optionally, the above-mentioned processor can also execute the program code of the following steps: constructing an auxiliary pooling module on the initial speech model, including: adding a global aggregation block and a local pooling block to the initial speech model, wherein the global aggregation block is used to perform global speech features of the speech frames input into the initial speech model within the range of the global speech frame sequence; constructing a pooling fusion block for fusing the global aggregation block and the local pooling block, wherein the auxiliary pooling module includes: a global aggregation block, a local pooling block and a pooling fusion block.

[0112] Optionally, the processor may also execute the program code of the following steps: constructing a local pooling block by: obtaining the weight of the speech frame in the global speech frame sequence, and the window size used to represent the pooling range; and constructing a local pooling block based on the weight and window size.

[0113] Optionally, the processor may also execute the program code of the following steps: based on the training set data, using the pre-trained language model to perform distillation training on the process speech model to obtain a target speech model, including: based on the training set data, using the pre-trained language model to perform distillation training on the process speech model to obtain a trained process speech model; extracting the part corresponding to the initial speech model from the trained process speech model to obtain a target speech model.

[0114] Optionally, the processor may also execute the program code of the following steps: based on the training set data, using the pre-trained language model to perform distillation training on the process speech model to obtain a target speech model, including: constructing a comparison loss function for performing distillation training on the process speech model using the pre-trained language model; based on the training set data and the comparison loss function, using the pre-trained language model to perform distillation training on the process speech model to obtain a target speech model.

[0115] Optionally, the processor may also execute the program code of the following steps: based on the training set data and the comparison loss function, the process speech model is distilled and trained using a pre-trained language model to obtain a target speech model, including: based on the training set data, determining the loss value corresponding to the comparison loss function obtained by distilling the process speech model using the pre-trained language model; adjusting the module parameters of the auxiliary pooling module based on the loss value, and adjusting the model parameters of the initial speech model to obtain the target speech model.

[0116] Optionally, the above-mentioned processor can also execute the program code of the following steps: adjusting the module parameters of the auxiliary pooling module based on the loss value, and adjusting the model parameters of the initial speech model to obtain the target speech model, including: determining the back propagation algorithm corresponding to the back propagation from the output of the auxiliary pooling module to the input of the initial speech model; based on the loss value and the back propagation algorithm, adjusting the module parameters of the auxiliary pooling module, and adjusting the model parameters of the initial speech model to obtain the target speech model.

[0117] Optionally, the processor may also execute the program code of the following steps: based on the training set data and the comparison loss function, the process speech model is distilled and trained using a pre-trained language model to obtain a target speech model, including: dividing part of the training set data from the training set data; freezing the model parameters of the initial speech model in the process speech model to obtain a semi-frozen process speech model; based on part of the training set data, the semi-frozen process speech model is distilled and trained using a pre-trained language model to obtain an optimized process speech model, wherein the module parameters of the auxiliary pooling module in the optimized process speech model are preliminarily optimized; based on the remaining training set data after the division in the training set data, the optimized process speech model is distilled and trained using a pre-trained language model to obtain a target speech model.

[0118] Optionally, the above-mentioned processor can also execute the program code of the following steps: after distilling and training the process speech model based on the training set data using the pre-trained language model to obtain the target speech model, it also includes: receiving the target speech; inputting the target speech into the target speech model to obtain the semantic encoding result corresponding to the target speech.

[0119] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: receiving the target speech; inputting the target speech into the target speech model to obtain the semantic encoding result corresponding to the target speech, wherein the target speech model is based on the training set data and uses the pre-trained language model to distill and train the initial speech model constructed with the auxiliary pooling module. The auxiliary pooling module includes a local pooling block, which is used to adjust the pooling range of the speech frame input into the initial speech model, and obtain the local speech features corresponding to the speech frame within the pooling range. The pooling range includes the speech frame and multiple adjacent frames of the speech frame.

[0120] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: receiving customer service voice, wherein the customer service voice carries questions requested to the customer service; inputting the customer service voice into the target voice model to obtain the semantic encoding result corresponding to the customer service voice, wherein the target voice model is based on the training set data, and uses the pre-trained language model to distill and train the initial voice model constructed with the auxiliary pooling module, the auxiliary pooling module includes a local pooling block, the local pooling block is used to adjust the pooling range of the voice frame input into the initial voice model, and obtain the local voice features corresponding to the voice frame within the pooling range, and the pooling range includes the voice frame and multiple adjacent frames of the voice frame; based on the semantic encoding result, determine the question and the answer corresponding to the question; feedback includes the intelligent voice of the answer.

[0121] It can be understood by those skilled in the art that Figure 9The structure shown is for illustration only, and the computer terminal may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a handheld computer, a mobile Internet device (MID), a PAD, or other terminal devices. Figure 9 It does not limit the structure of the above electronic device. For example, the computer terminal 9 may also include Figure 9 More or fewer components (such as network interfaces, display devices, etc.) shown in, or with Figure 9 Different configurations shown.

[0122] A person skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, which may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0123] Example 4

[0124] The embodiment of the present invention further provides a computer-readable storage medium. Optionally, in this embodiment, the computer-readable storage medium can be used to store the program code executed by the speech model processing method provided in the first embodiment.

[0125] Optionally, in this embodiment, the computer-readable storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

[0126] Optionally, in this embodiment, a computer-readable storage medium is configured to store program code for performing the following steps: obtaining a pre-trained language model, an initial speech model, and training set data; constructing an auxiliary pooling module on the initial speech model to obtain a process speech model, wherein the auxiliary pooling module includes a local pooling block, the local pooling block is used to adjust the pooling range of the speech frame input into the initial speech model, and obtain local speech features corresponding to the speech frame within the pooling range, and the pooling range includes the speech frame and multiple adjacent frames of the speech frame; based on the training set data, the process speech model is distilled and trained using the pre-trained language model to obtain a target speech model.

[0127] Optionally, in this embodiment, the computer-readable storage medium is also configured to store program code for performing the following steps: constructing an auxiliary pooling module on the initial speech model, including: adding a global aggregation block and a local pooling block to the initial speech model, wherein the global aggregation block is used to perform global speech features of the speech frames input into the initial speech model within the global speech frame sequence; constructing a pooling fusion block for fusing the global aggregation block and the local pooling block, wherein the auxiliary pooling module includes: a global aggregation block, a local pooling block and a pooling fusion block.

[0128] Optionally, in this embodiment, the computer-readable storage medium is also configured to store program code for performing the following steps: constructing a local pooling block by: obtaining the weight of the speech frame in the global speech frame sequence, and the window size used to represent the pooling range; constructing a local pooling block based on the weight and window size.

[0129] Optionally, in this embodiment, the computer-readable storage medium is also configured to store program code for executing the following steps: based on the training set data, using a pre-trained language model to perform distillation training on the process speech model to obtain a target speech model, including: based on the training set data, using a pre-trained language model to perform distillation training on the process speech model to obtain a trained process speech model; extracting the part corresponding to the initial speech model from the trained process speech model to obtain a target speech model.

[0130] Optionally, in this embodiment, the computer-readable storage medium is also configured to store program code for executing the following steps: based on the training set data, using the pre-trained language model to perform distillation training on the process speech model to obtain a target speech model, including: constructing a comparison loss function for performing distillation training on the process speech model using the pre-trained language model; based on the training set data and the comparison loss function, using the pre-trained language model to perform distillation training on the process speech model to obtain a target speech model.

[0131] Optionally, in this embodiment, the computer-readable storage medium is also configured to store program code for executing the following steps: based on the training set data and the comparison loss function, the process speech model is distilled and trained using a pre-trained language model to obtain a target speech model, including: based on the training set data, determining the loss value corresponding to the comparison loss function obtained by distilling the process speech model using the pre-trained language model; adjusting the module parameters of the auxiliary pooling module based on the loss value, and adjusting the model parameters of the initial speech model to obtain the target speech model.

[0132] Optionally, in this embodiment, the computer-readable storage medium is also configured to store program code for performing the following steps: adjusting the module parameters of the auxiliary pooling module based on the loss value, and adjusting the model parameters of the initial speech model to obtain a target speech model, including: determining a backpropagation algorithm corresponding to the backpropagation from the output of the auxiliary pooling module to the input of the initial speech model; adjusting the module parameters of the auxiliary pooling module based on the loss value and the backpropagation algorithm, and adjusting the model parameters of the initial speech model to obtain the target speech model.

[0133] Optionally, in this embodiment, the computer-readable storage medium is further configured to store program code for executing the following steps: based on the training set data and the comparison loss function, using a pre-trained language model to perform distillation training on the process speech model to obtain a target speech model, including: dividing part of the training set data from the training set data; freezing the model parameters of the initial speech model in the process speech model to obtain a semi-frozen process speech model; based on part of the training set data, using a pre-trained language model to perform distillation training on the semi-frozen process speech model to obtain an optimized process speech model, wherein the module parameters of the auxiliary pooling module in the optimized process speech model are preliminarily optimized; based on the remaining training set data after the division in the training set data, using the pre-trained language model to perform distillation training on the optimized process speech model to obtain a target speech model.

[0134] Optionally, in this embodiment, the computer-readable storage medium is also configured to store program code for executing the following steps: after distilling and training the process speech model based on the training set data using a pre-trained language model to obtain a target speech model, it also includes: receiving the target speech; inputting the target speech into the target speech model to obtain a semantic encoding result corresponding to the target speech.

[0135] Optionally, in this embodiment, a computer-readable storage medium is configured to store program code for executing the following steps: receiving a target speech; inputting the target speech into a target speech model to obtain a semantic encoding result corresponding to the target speech, wherein the target speech model is based on training set data and is obtained by distilling and training an initial speech model constructed with an auxiliary pooling module using a pre-trained language model, the auxiliary pooling module includes a local pooling block, and the local pooling block is used to adjust the pooling range of the speech frame input into the initial speech model, and obtain local speech features corresponding to the speech frame within the pooling range, and the pooling range includes the speech frame and multiple adjacent frames of the speech frame.

[0136] Optionally, in this embodiment, a computer-readable storage medium is configured to store program code for executing the following steps: receiving a customer service voice, wherein the customer service voice carries a question requested to the customer service; inputting the customer service voice into a target voice model to obtain a semantic encoding result corresponding to the customer service voice, wherein the target voice model is based on training set data, and a pre-trained language model is used to perform distillation training on an initial voice model constructed with an auxiliary pooling module, the auxiliary pooling module includes a local pooling block, and the local pooling block is used to adjust the pooling range of the voice frame input into the initial voice model, and obtain local voice features corresponding to the voice frame within the pooling range, and the pooling range includes the voice frame and multiple adjacent frames of the voice frame; based on the semantic encoding result, determining the question and the answer corresponding to the question; and feeding back intelligent voice including the answer.

[0137] The serial numbers of the above embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments.

[0138] In the above embodiments of the present invention, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0139] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0140] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0141] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0142] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a computer-readable storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned computer-readable storage medium includes: various media that can store program codes, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0143] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A speech model processing method, characterized in that: include: Obtain pre-trained language model, initial speech model and training set data; Constructing an auxiliary pooling module on the initial speech model to obtain a process speech model, wherein the auxiliary pooling module includes a local pooling block, the local pooling block is used to adjust the pooling range of the speech frame input into the initial speech model, and obtain local speech features corresponding to the speech frame within the pooling range, wherein the pooling range includes the speech frame and multiple adjacent frames of the speech frame; Based on the training set data, the pre-trained language model is used to perform distillation training on the process speech model to obtain a target speech model.

2. The method according to claim 1, characterized in that The auxiliary pooling module is constructed on the initial speech model, comprising: Adding a global aggregation block and the local pooling block to the initial speech model, wherein the global aggregation block is used to perform global speech features of the speech frames input into the initial speech model within the scope of the global speech frame sequence; A pooling fusion block is constructed for fusing the global aggregation block and the local pooling block, wherein the auxiliary pooling module includes: the global aggregation block, the local pooling block and the pooling fusion block.

3. The method according to claim 1, characterized in that The method further includes: constructing the local pooling block by: Obtaining a weight of the speech frame in a global speech frame sequence and a window size for representing the pooling range; Based on the weight and the window size, the local pooling block is constructed.

4. The method according to claim 1, wherein The method of performing distillation training on the process speech model based on the training set data using the pre-trained language model to obtain a target speech model includes: Based on the training set data, using the pre-trained language model to perform distillation training on the process speech model to obtain a trained process speech model; The part corresponding to the initial speech model is extracted from the trained process speech model to obtain the target speech model.

5. The method according to claim 1, wherein The method of performing distillation training on the process speech model based on the training set data using the pre-trained language model to obtain a target speech model includes: Constructing a comparison loss function for performing distillation training on the process speech model using the pre-trained language model; Based on the training set data and the comparison loss function, the process speech model is distilled and trained using the pre-trained language model to obtain a target speech model.

6. The method according to claim 5, characterized in that The method of performing distillation training on the process speech model based on the training set data and the comparison loss function using the pre-trained language model to obtain a target speech model includes: Determining, based on the training set data, a loss value corresponding to the comparison loss function obtained by performing distillation training on the process speech model using the pre-trained language model; The module parameters of the auxiliary pooling module are adjusted based on the loss value, and the model parameters of the initial speech model are adjusted to obtain the target speech model.

7. The method according to claim 6, characterized in that The adjusting the module parameters of the auxiliary pooling module based on the loss value and the adjusting the model parameters of the initial speech model to obtain the target speech model includes: Determining a back propagation algorithm corresponding to back propagation from the output of the auxiliary pooling module to the input of the initial speech model; Based on the loss value and the back propagation algorithm, the module parameters of the auxiliary pooling module are adjusted, and the model parameters of the initial speech model are adjusted to obtain the target speech model.

8. The method according to claim 5, characterized in that The method of performing distillation training on the process speech model based on the training set data and the comparison loss function using the pre-trained language model to obtain a target speech model includes: Dividing part of the training set data from the training set data; Freezing the model parameters of the initial speech model in the process speech model to obtain a semi-frozen process speech model; Based on the partial training set data, the semi-frozen process speech model is subjected to distillation training using the pre-trained language model to obtain an optimized process speech model, wherein module parameters of the auxiliary pooling module in the optimized process speech model are preliminarily optimized; Based on the remaining training set data after the division in the training set data, the optimized process speech model is distilled and trained using the pre-trained language model to obtain the target speech model.

9. The method according to any one of claims 1 to 8, characterized in that After the process speech model is distilled and trained based on the training set data using the pre-trained language model to obtain a target speech model, the method further includes: Receive target speech; The target speech is input into the target speech model to obtain a semantic encoding result corresponding to the target speech.

10. A speech model processing method, characterized in that: include: Receive target speech; The target speech is input into a target speech model to obtain a semantic encoding result corresponding to the target speech, wherein the target speech model is obtained by distilling and training an initial speech model with an auxiliary pooling module based on training set data using a pre-trained language model, and the auxiliary pooling module includes a local pooling block, and the local pooling block is used to adjust the pooling range of the speech frame input into the initial speech model, and obtain the local speech features corresponding to the speech frame within the pooling range, and the pooling range includes the speech frame and multiple adjacent frames of the speech frame.

11. A speech model processing method, characterized in that: include: Receive a customer service voice message, wherein the customer service voice message carries a question for the customer service; Inputting the customer service speech into a target speech model to obtain a semantic encoding result corresponding to the customer service speech, wherein the target speech model is obtained by distilling and training an initial speech model constructed with an auxiliary pooling module based on training set data using a pre-trained language model, and the auxiliary pooling module includes a local pooling block, which is used to adjust the pooling range of the speech frame input into the initial speech model and obtain local speech features corresponding to the speech frame within the pooling range, wherein the pooling range includes the speech frame and multiple adjacent frames of the speech frame; Determining the question and the answer corresponding to the question based on the semantic encoding result; The feedback includes an intelligent voice of the answer.

12. A speech model processing device, characterized in that: include: The acquisition module is used to obtain the pre-trained language model, the initial speech model and the training set data; A construction module is used to construct an auxiliary pooling module on the initial speech model to obtain a process speech model, wherein the auxiliary pooling module includes a local pooling block, the local pooling block is used to adjust the pooling range of the speech frame input into the initial speech model, and obtain local speech features corresponding to the speech frame within the pooling range, wherein the pooling range includes the speech frame and multiple adjacent frames of the speech frame; A training module is used to perform distillation training on the process speech model using the pre-trained language model based on the training set data to obtain a target speech model.

13. A computer-readable storage medium, characterized in that The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the speech model processing method according to any one of claims 1 to 11.

14. A computer device, characterized in that: include: memory and processor, The memory stores a computer program; The processor is used to execute the computer program stored in the memory, and when the computer program is running, the processor executes the speech model processing method according to any one of claims 1 to 11.

Images