Model training method, speech recognition method and device, equipment and storage medium

Abstract

The application provides a model training method, a speech recognition method and device, equipment and a storage medium, and belongs to the field of financial technology. The method comprises the following steps: inputting sample voice data and sample text data into a neural network model comprising an encoding network and a decoding network; performing feature extraction on the sample voice data through a convolutional layer of the encoding network to obtain a sample voice feature vector; performing attention calculation on the sample voice feature vector through a sparse attention layer of the encoding network to obtain a sample voice representation vector; performing encoding processing on the sample text data through the decoding network to obtain a sample text encoding vector; performing reconstruction processing on the sample text encoding vector and the sample voice representation vector through an attention layer of the decoding network to obtain a sample relationship feature vector; and updating parameters of the neural network model through a loss function and the sample relationship feature vector to obtain a speech recognition model, which can improve the speech recognition accuracy of the model in a financial scenario.

Description

Technical Field

[0001] This application relates to the field of financial technology, and in particular to a model training method, a speech recognition method and apparatus, device and storage medium. Background Technology

[0002] With the development of network, communication, and computer technologies, enterprises are exhibiting characteristics of digitalization, remote operation, virtualization, and networking, leading to a surge in online businesses. Communication and dialogue between customers and enterprises have also evolved from face-to-face consultations and negotiations to remote exchanges and communication based on the internet, telephone, and other means. Against this backdrop, intelligent voice interaction is being widely applied in fields such as finance, logistics, and customer service.

[0003] Currently, voice-interactive financial trading platforms handle a large volume of telephone voice services daily, addressing diverse customer needs including pre-sales inquiries, purchases, after-sales support, and complaints. During these calls, intelligent customer service robots need to respond appropriately to different users. If the intelligent customer service robot fails to accurately identify the user's needs expressed in the voice data, it can lead to inconsistencies between the voice data-based service response and the user's requirements, impacting service quality and customer satisfaction.

[0004] Speech recognition primarily involves converting audio into text. Most current speech recognition methods rely on neural network models, but these models often fail to effectively capture useful speech information, resulting in poor training performance. Therefore, improving model training efficiency has become a pressing technical challenge. Summary of the Invention

[0005] The main objective of this application is to provide a model training method, a speech recognition method and apparatus, device and storage medium, which aim to improve the training effect of the model.

[0006] To achieve the above objectives, a first aspect of this application proposes a model training method, the training method comprising:

[0007] Obtain sample speech data and corresponding sample text data;

[0008] The sample speech data and the sample text data are input into a preset neural network model, wherein the neural network model includes an encoding network and a decoding network;

[0009] The sample speech data is feature extracted through the convolutional layers of the encoding network to obtain the sample speech feature vector;

[0010] Attention is calculated on the sample speech feature vector through the sparse attention layer of the coding network to obtain the sample speech representation vector;

[0011] The sample text data is encoded using the decoding network to obtain a sample text encoding vector.

[0012] The sample text encoding vector and the sample speech representation vector are reconstructed by the attention layer of the decoding network to obtain a sample relationship feature vector, which is used to characterize the degree of correlation between the sample text encoding vector and the sample speech representation vector.

[0013] The neural network model is trained by updating its parameters using a preset loss function and the sample relationship feature vector, thereby obtaining a speech recognition model.

[0014] In some embodiments, the step of extracting features from the sample speech data through the convolutional layers of the coding network to obtain a sample speech feature vector includes:

[0015] The sample speech data is processed by convolutional layer to obtain sample speech convolution vector;

[0016] The sample speech convolution vector is downsampled by the convolutional layer to obtain the sample speech sampling features.

[0017] The sampled speech features are positionally encoded to obtain the sampled speech feature vector.

[0018] In some embodiments, the step of performing attention calculation on the sample speech feature vector through the sparse attention layer of the encoding network to obtain the sample speech representation vector includes:

[0019] The sample speech feature vector is normalized by the preset function of the sparse attention layer to obtain sample probability features;

[0020] The sample speech feature vector is filtered according to the sample probability features and the preset probability threshold to obtain candidate speech feature vectors, wherein the candidate speech feature vectors are sample speech feature vectors whose sample probability features are greater than the probability threshold.

[0021] Attention is calculated on the candidate speech feature vectors according to preset weight parameters to obtain the sample speech representation vector.

[0022] In some embodiments, the process of reconstructing the sample text encoding vector and the sample speech representation vector through the attention layer of the decoding network to obtain the sample relation feature vector includes:

[0023] The sample text representation vector is obtained by performing self-attention calculation on the sample text encoding vector through the attention layer.

[0024] The attention layer performs attention calculations on the sample speech representation vector and the sample text representation vector to obtain an initial relation feature vector.

[0025] Based on the preset vector feature dimensions, the initial relation feature vector is mapped to the preset vector space to obtain the sample relation feature vector.

[0026] In some embodiments, updating the parameters of the neural network model using a preset loss function and the sample relationship feature vector to train the neural network model and obtain a speech recognition model includes:

[0027] The target loss value is obtained by calculating the loss of the sample relationship feature vector using the loss function.

[0028] The target loss value is backpropagated to update the model parameters of the neural network model, thereby obtaining the speech recognition model.

[0029] To achieve the above objectives, a second aspect of this application provides a speech recognition method, the method comprising:

[0030] Acquire the target speech data to be processed;

[0031] The target speech data is input into a speech recognition model for recognition processing to obtain target text data, wherein the target text data is used to represent the speech content of the target speech data; the speech recognition model is trained according to the training method described in the first aspect.

[0032] To achieve the above objectives, a third aspect of this application provides a model training apparatus, the training apparatus comprising:

[0033] The sample data acquisition module is used to acquire sample speech data and sample text data corresponding to the sample speech data.

[0034] A data input module is used to input the sample speech data and the sample text data into a preset neural network model, wherein the neural network model includes an encoding network and a decoding network;

[0035] The feature extraction module is used to extract features from the sample speech data through the convolutional layers of the encoding network to obtain sample speech feature vectors;

[0036] The calculation module is used to perform attention calculation on the sample speech feature vector through the sparse attention layer of the coding network to obtain the sample speech representation vector;

[0037] The encoding module is used to encode the sample text data through the decoding network to obtain the sample text encoding vector;

[0038] The reconstruction module is used to reconstruct the sample text encoding vector and the sample speech representation vector through the attention layer of the decoding network to obtain a sample relationship feature vector, which is used to characterize the degree of correlation between the sample text encoding vector and the sample speech representation vector.

[0039] The training module is used to update the parameters of the neural network model using a preset loss function and the sample relationship feature vector to train the neural network model and obtain a speech recognition model.

[0040] To achieve the above objectives, a fourth aspect of this application provides a speech recognition device, the device comprising:

[0041] The target speech data acquisition module is used to acquire the target speech data to be processed.

[0042] The speech recognition module is used to input the target speech data into the speech recognition model for recognition processing to obtain target text data, wherein the target text data is used to represent the speech content of the target speech data; the speech recognition model is trained according to the training device described in the third aspect.

[0043] To achieve the above objectives, a fifth aspect of the present application provides an electronic device, the electronic device including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the training method described in the first aspect or the method described in the second aspect.

[0044] To achieve the above objectives, a sixth aspect of the present application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the training method described in the first aspect or the method described in the second aspect.

[0045] The training method, speech recognition method, training device, electronic device, and computer-readable storage medium proposed in this application acquire sample speech data and corresponding sample text data. The sample speech data and sample text data are then input into a pre-defined neural network model, which includes an encoding network and a decoding network. Features are extracted from the sample speech data through the convolutional layers of the encoding network to obtain sample speech feature vectors. Attention is then calculated on these sample speech feature vectors through the sparse attention layers of the encoding network to obtain sample speech representation vectors. This method can conveniently filter redundant information irrelevant to speech recognition in the sample speech data, effectively reducing the total amount of information and improving the model's recognition speed. Furthermore, the sample text data is encoded through the decoding network to obtain sample text encoding vectors. The sample text encoding vectors and sample speech representation vectors are then reconstructed through the attention layers of the decoding network to obtain sample relationship feature vectors. These sample relationship feature vectors characterize the correlation between the sample text encoding vectors and sample speech representation vectors, reflecting the model's recognition performance and improving the model's training efficiency. Finally, the neural network model is trained by updating its parameters using a preset loss function and sample relationship feature vectors, resulting in a speech recognition model. The internal parameters of the neural network model can be continuously adjusted to ensure its recognition performance meets training requirements. This yields a speech recognition model capable of recognizing the text content of target speech data, improving its accuracy. Consequently, the intelligent customer service robot can more accurately identify the requests represented by the speech data of service recipients during dialogues, improving targeted responses and service feedback. This effectively enhances the quality and effectiveness of dialogue in financial transactions, enabling intelligent voice dialogue services, improving customer service quality and satisfaction, and ultimately increasing business conversion rates. Attached Figure Description

[0046] Figure 1 This is a flowchart of the training method for the model provided in the embodiments of this application;

[0047] Figure 2 yes Figure 1 The flowchart of step S103 in the process;

[0048] Figure 3 yes Figure 1 The flowchart of step S104 in the process;

[0049] Figure 4 yes Figure 1 The flowchart of step S106 in the process;

[0050] Figure 5 yesFigure 1 The flowchart of step S107 in the process;

[0051] Figure 6 This is a flowchart of the speech recognition method provided in the embodiments of this application;

[0052] Figure 7 This is a schematic diagram of the structure of the training device for the model provided in the embodiments of this application;

[0053] Figure 8 This is a schematic diagram of the structure of the speech recognition device provided in the embodiments of this application;

[0054] Figure 9 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0056] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0057] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0058] First, let's analyze some of the terms used in this application:

[0059] Artificial intelligence (AI) is a new branch of computer science that studies, develops, and applies theories, methods, technologies, and systems to simulate, extend, and expand human intelligence. It aims to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thought. Furthermore, AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results.

[0060] Natural Language Processing (NLP): NLP uses computers to process, understand, and utilize human language (such as Chinese and English). NLP is a branch of artificial intelligence and an interdisciplinary field of computer science and linguistics, often referred to as computational linguistics. NLP includes syntactic analysis, semantic analysis, and discourse understanding. It is commonly used in machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, intent recognition, information extraction and filtering, text classification and clustering, sentiment analysis, and opinion mining. It involves data mining, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computation.

[0061] Information extraction is a text processing technique that extracts factual information such as entities, relationships, and events from natural language text and outputs it as structured data. Information extraction is a technique for extracting specific information from text data. Text data is composed of specific units, such as sentences, paragraphs, and chapters. Text information is composed of smaller, specific units, such as characters, words, phrases, sentences, paragraphs, or combinations of these units. Extracting noun phrases, names of people, and place names from text data is an example of text information extraction. Of course, text information extraction techniques can extract information of various types.

[0062] Mel-Frequency Cipstal Coefficients (MFCCs) are a set of key coefficients used to construct a Mel-Frequency Cipstal spectrum. From a segment of a music signal, a set of cepstrum values ​​can be obtained that is sufficient to represent the music signal. The Mel-Frequency Cipstal Coefficients are the cepstrum values ​​derived from this cepstrum (i.e., the spectrum of the spectrum). Unlike a regular cepstrum, the most distinctive feature of the Mel-Frequency Cipstrum is that its frequency bands are uniformly distributed across the Mel scale. In other words, compared to the linear cepstrum representations commonly seen, this frequency band is closer to the non-linear human auditory system. For example, Mel-Frequency Cipstals are frequently used in audio compression techniques.

[0063] Fourier transform: Represents a function that satisfies certain conditions as a linear combination of trigonometric functions (sine and / or cosine functions) or their integrals. In different research fields, the Fourier transform has various variants, such as the continuous Fourier transform and the discrete Fourier transform.

[0064] Transformer Layer: The neural network consists of an embedding layer (which can be called an input embedding layer) and at least one transformer layer. There can be N transformer layers (N is a positive integer). The embedding layer includes an input embedding layer and a positional encoding layer. In the input embedding layer, word embeddings are performed on each word in the current input to obtain word embedding vectors. In the positional encoding layer, the position of each word in the current input is obtained, and a position vector is generated for each word. Each transformer layer includes sequentially adjacent attention layers, add and normalize layers, feedforward layers, and add and normalize layers. In the input embedding layer, the current input is embedded to obtain multiple feature vectors. In the attention layer, P input vectors are obtained from the layer above the transformer layer. Using any first input vector among these P vectors as the center, intermediate vectors are obtained based on the correlation between each input vector within a predefined attention window and that first input vector. This process determines P intermediate vectors corresponding to the P input vectors. In the pooling layer, the P intermediate vectors are merged into Q output vectors, where at least one of the output vectors obtained from the last transformer layer is used as the feature representation of the current input. In the embedding layer, the current input (which can be text input, such as a paragraph or a sentence; the text can be Chinese / English or other languages) is embedded to obtain multiple feature vectors. After obtaining the current input, the embedding layer can embed each word in the current input to obtain the feature vectors of each word.

[0065] Encoder: Transforms an input sequence into a fixed-length vector.

[0066] Decoding: This involves transforming a previously generated fixed vector into an output sequence; the input sequence can be text, speech, image, or video; the output sequence can be text or image.

[0067] Backpropagation: The general principle of backpropagation is as follows: The training data is input into the input layer of the neural network, passes through the hidden layer, and finally reaches the output layer of the neural network to output the result; Since there is an error between the output result of the neural network and the actual result, the error between the estimated value and the actual value is calculated, and this error is backpropagated from the output layer to the hidden layer until it reaches the input layer; During the backpropagation process, the values ​​of various parameters are adjusted according to the error; The above process is iterated continuously until convergence.

[0068] Softmax function: The Softmax function is a normalization exponential function that can "compress" a K-dimensional vector z containing arbitrary real numbers into another K-dimensional real vector σ(z), such that each element is in the range (0,1) and the sum of all elements is 1. This function is often used in multi-class classification problems.

[0069] With the development of network, communication, and computer technologies, enterprises are exhibiting characteristics of digitalization, remote operation, virtualization, and networking, leading to a surge in online businesses. Communication and dialogue between customers and enterprises have also evolved from face-to-face consultations and negotiations to remote exchanges and communication based on the internet, telephone, and other means. Against this backdrop, intelligent voice interaction is being widely applied in fields such as finance, logistics, and customer service.

[0070] Currently, voice-interactive financial trading platforms handle a large volume of telephone voice services daily, addressing diverse customer needs including pre-sales inquiries, purchases, after-sales support, and complaints. During these calls, intelligent customer service robots need to respond appropriately to different users. If the intelligent customer service robot fails to accurately identify the user's needs expressed in the voice data, it can lead to inconsistencies between the voice data-based service response and the user's requirements, impacting service quality and customer satisfaction.

[0071] For example, in the process of recommending products through virtual characters on a voice-interactive financial trading platform, when the service recipient has questions and needs consultation and communication, the virtual character can only find answers from pre-set options and cannot accurately identify the service recipient's needs expressed in the voice data, resulting in the phenomenon of "answering the wrong question" and low accuracy of the virtual character's voice interaction.

[0072] Speech recognition primarily involves converting audio into text. Most current speech recognition methods rely on neural network models, but these models often fail to effectively capture useful speech information, resulting in poor training performance. Therefore, improving model training efficiency has become a pressing technical challenge.

[0073] Based on this, embodiments of this application provide a model training method, a speech recognition method, a model training device, a speech recognition device, an electronic device, and a storage medium, aiming to improve the training effect of the model.

[0074] The model training method, speech recognition method, model training device, speech recognition device, electronic device and storage medium provided in the embodiments of this application are specifically described through the following embodiments. First, the model training method in the embodiments of this application is described.

[0075] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0076] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0077] The model training method provided in this application relates to the field of artificial intelligence technology. The model training method provided in this application can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application implementing the model training method, but is not limited to the above forms.

[0078] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0079] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirection to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data required for the proper functioning of these embodiments acquired.

[0080] Figure 1 This is an optional flowchart of the model training method provided in the embodiments of this application. Figure 1 The method may include, but is not limited to, steps S101 to S107.

[0081] Step S101: Obtain sample speech data and sample text data corresponding to the sample speech data;

[0082] Step S102: Input the sample speech data and sample text data into a preset neural network model, wherein the neural network model includes an encoding network and a decoding network;

[0083] Step S103: Extract features from the sample speech data through the convolutional layer of the encoding network to obtain the sample speech feature vector;

[0084] Step S104: The sample speech feature vector is obtained by performing attention calculation on the sample speech feature vector through the sparse attention layer of the encoding network.

[0085] Step S105: Encode the sample text data using a decoding network to obtain the sample text encoding vector;

[0086] Step S106: The sample text encoding vector and sample speech representation vector are reconstructed through the attention layer of the decoding network to obtain the sample relationship feature vector. The sample relationship feature vector is used to characterize the degree of correlation between the sample text encoding vector and the sample speech representation vector.

[0087] Step S107: Update the parameters of the neural network model using a preset loss function and sample relationship feature vector to train the neural network model and obtain the speech recognition model.

[0088] Steps S101 to S107, as illustrated in this embodiment, involve acquiring sample speech data and corresponding sample text data; inputting the sample speech data and sample text data into a preset neural network model, wherein the neural network model includes an encoding network and a decoding network; extracting features from the sample speech data through the convolutional layer of the encoding network to obtain a sample speech feature vector; and performing attention calculation on the sample speech feature vector through the sparse attention layer of the encoding network to obtain a sample speech representation vector. This method can conveniently filter redundant information unrelated to speech recognition in the sample speech data, effectively reducing the total amount of information and improving the model's recognition speed. Further, the sample text data is encoded through the decoding network to obtain a sample text encoding vector, and the sample text encoding vector and sample speech representation vector are reconstructed through the attention layer of the decoding network to obtain a sample relationship feature vector. The sample relationship feature vector is used to characterize the correlation between the sample text encoding vector and the sample speech representation vector, and can reflect the model's recognition performance based on this correlation, thereby improving the model's training efficiency. Finally, the neural network model is trained by updating its parameters using a preset loss function and sample relationship feature vectors, resulting in a speech recognition model. This allows for continuous adjustment of the internal parameters of the neural network model to ensure its recognition performance meets training requirements, thus obtaining a speech recognition model capable of recognizing the text content of target speech data and improving the model's training effectiveness.

[0089] In step S101 of some embodiments, a web crawler can be written, and after setting up a data source, data can be crawled in a targeted manner to obtain sample speech data and corresponding sample text data of the sample speaker. The data source can be various types of online platforms, social media, or specific audio databases, etc. The sample speech data can be music material, speeches, reports, chat dialogues, etc., of the sample speaker, and the text information of the sample text data can characterize the speech content of the sample speech data. This method allows for convenient acquisition of sample speech data and sample text data, improving data acquisition efficiency.

[0090] It should be noted that the sample speech data mentioned above is mainly spectral data. After performing short-time Fourier transform on the acquired raw audio data and filtering it through a Mel-Cepstral Spectrum filter, Mel-Cepstral feature data in spectral form is obtained, and this Mel-Cepstral feature data is used as the target speech data.

[0091] For example, in financial transaction scenarios, sample voice data consists of audio data containing commonly used dialogue phrases in the financial field. In insurance sales scenarios, sample voice data consists of audio data containing descriptions of a particular insurance product, such as its type, fees, and target audience. Sample text data can contain specialized financial terminology, financial business template vocabulary, or product descriptions of insurance products, wealth management products, and commonly used dialogue phrases in the financial field.

[0092] In step S102 of some embodiments, sample speech data and sample text data are input into a preset neural network model through a preset script program or other computer program. The neural network model can be constructed based on the transformer model. The neural network model includes an encoding network and a decoding network. The encoding network is used to encode the input speech data and extract the speech content information in the speech data. Then, the extracted encoded speech content information is input into the decoding network. The decoding network is used to decode the encoded speech content information, extract the text features corresponding to the speech data, and generate the text data corresponding to the speech data, thereby realizing the recognition processing of the input speech data and obtaining the speech text data of the speech data.

[0093] Please see Figure 2 In some embodiments, step S103 may include, but is not limited to, steps S201 to S203:

[0094] Step S201: Perform convolution processing on the sample speech data through a convolutional layer to obtain the sample speech convolution vector;

[0095] Step S202: The sample speech convolution vector is downsampled through a convolutional layer to obtain the sample speech sampling features;

[0096] Step S203: Position encoding is performed on the sample speech sampling features to obtain the sample speech feature vector.

[0097] In step S201 of some embodiments, the sample speech data is processed by convolutional layer to extract speech content features from the sample speech data and obtain sample speech convolution vector. The sample speech convolution vector can be represented in the form of feature map. The convolution kernel and the size of the convolution kernel can be set according to the actual situation. For example, the convolutional layer includes at least one convolution kernel with a convolution size of 3×3, and the number of channels of the convolutional layer can be 256 or 64, etc.

[0098] In step S202 of some embodiments, the sample speech convolution vector is downsampled by a convolutional layer to compress the sample speech convolution vector. During the downsampling process, the maximum value or average value of each feature position of the sample speech convolution vector can be selected as the feature value of that feature position, thereby reducing the feature parameters of the sample speech convolution vector and obtaining the sample speech sampling features.

[0099] In step S203 of some embodiments, the positional encoding of the sample speech sampling features can be either absolute encoding or relative encoding. Specifically, when performing absolute encoding of the sample speech sampling features, the absolute positional encoding of each word vector of the sample speech sampling features is generated using sine and cosine functions. The position of each word vector of the sample speech sampling features is then marked according to this absolute positional encoding, and the absolute positional encoding is used as the position label of the word vector to obtain the sample speech feature vector. When performing relative encoding of the sample speech sampling features, the distance value between every two word vectors of the sample speech sampling features is calculated. This distance value can be Euclidean distance or Manhattan distance, etc. The relationship between every two word vectors is assigned a relationship number based on the magnitude of the distance values. These relationship numbers can be used to represent the semantic order of the word vectors, thereby obtaining the sample speech feature vector.

[0100] Through the above steps S201 to S203, speech content features in sample speech data can be extracted relatively easily, and the speech content features can be simplified to generate sample speech feature vectors that conform to basic grammatical norms. The sample speech feature vectors can then be used for subsequent model training, which can improve the quality of training samples and help improve the training effect of the model.

[0101] Please see Figure 3 In some embodiments, step S104 may include, but is not limited to, steps S301 to S303:

[0102] Step S301: Normalize the sample speech feature vector using a preset function of the sparse attention layer to obtain sample probability features.

[0103] Step S302: The sample speech feature vector is filtered according to the sample probability features and the preset probability threshold to obtain candidate speech feature vectors, wherein the candidate speech feature vectors are the sample speech feature vectors whose sample probability features are greater than the probability threshold.

[0104] Step S303: Perform attention calculation on the candidate speech feature vectors according to the preset weight parameters to obtain the sample speech representation vector.

[0105] In step S301 of some embodiments, the preset function can be the sparsemax function. The sparsemax function can map the Euclidean projection of the sample speech feature vector into a probabilistic simplex p, thereby achieving normalization processing of the sample speech feature vector. The probabilistic simplex reflects the sparsity of the sample speech feature vector. For example, the probabilistic simplex is used as the probabilistic feature of the sample speech feature vector, that is, the sample probability feature of the sample speech feature vector is obtained.

[0106] For example, the process of normalizing the sample speech feature vector through the sparsemax function of the sparse attention layer can be represented as shown in Equation (1);

[0107]

[0108] Where z is the sample speech feature vector; p is the probability simplex, i.e., the sample probability feature; Δ J =p∈R J | p≤0 ; It is a J-dimensional simplex, R is a real number, and j is the word vector number of the sample speech feature vector.

[0109] In step S302 of some embodiments, when filtering the sample speech feature vector according to the sample probability features and the preset probability threshold, the sparsity of the sample speech feature vector is first calculated according to the probability simplex and the probability threshold.

[0110] It should be noted that sparsity can be expressed as p * =[p-τ] + , [] + This means taking the larger of the numbers and 0, where τ is the probability threshold. This method forces the probability features of samples smaller than the probability threshold to zero, while still ensuring...

[0111] Therefore, by comparing the sample probability features with a preset probability threshold, it is easy to select sample speech feature vectors whose probability features are greater than the probability threshold. These sample speech feature vectors are then used as candidate speech feature vectors, while sample speech feature vectors whose probability features are less than or equal to the probability threshold are discarded. The candidate speech feature vectors include information highly relevant to speech recognition, while the discarded sample speech feature vectors contain more redundant information unrelated to speech recognition. This method of filtering sample speech feature vectors effectively preserves some high-probability elements, allowing the neural network model to focus more on these high-probability elements (i.e., the content of the candidate speech feature vectors) during subsequent model training, thereby improving the model's learning performance and training effect.

[0112] In step S303 of some embodiments, the preset weight parameters can be set according to actual business needs and are not limited. When performing attention calculation on candidate speech feature vectors through the attention mechanism and weight parameters of the encoding network, different attention weights are assigned to different candidate speech feature vectors, so that the attention distribution is concentrated on the candidate speech feature vectors with higher importance. By using the attention mechanism and weight parameters to perform attention weighting on the candidate speech feature vectors, a sample speech representation vector is obtained. The sample speech representation vector includes relatively rich semantic information, which can be used to represent the main content of the sample speech data.

[0113] Through the aforementioned steps S301 to S303, redundant information unrelated to speech recognition in the sample speech data can be filtered out relatively easily, effectively reducing the total amount of information, improving the model's recognition speed, and enhancing the model's training effect. Furthermore, compared to the commonly used transformer model network structure, the neural network model in this embodiment employs a sparse attention layer, which simplifies the model structure, making the model more lightweight. This allows the trained speech recognition model to be easily deployed to mobile devices and more network devices, enabling speech recognition in various application scenarios and improving the applicability of the speech recognition model.

[0114] In step S105 of some embodiments, the sample text data can be positionally encoded by a decoding network. This positional encoding process can be absolute encoding or relative encoding. Specifically, when performing absolute encoding on the sample text data, the absolute positional encoding of each word vector in the sample text data is generated using sine and cosine functions. Each word vector in the sample text data is then marked with its position based on this absolute positional encoding, and the absolute positional encoding is used as the position label of the word vector to obtain the sample text encoded vector. When performing relative encoding on the sample text data, the distance value between every two word vectors in the sample text data is calculated. This distance value can be Euclidean distance or Manhattan distance, etc. Each pair of word vectors is assigned a relationship number based on the magnitude of the distance values. These relationship numbers can be used to represent the semantic order of the word vectors, thereby obtaining the sample text encoded vector. This method can preserve the semantic order of the sample text during encoding processing, improve the semantic logic of the sample text encoded vector, and enable the use of higher-quality sample text encoded vectors for subsequent model training, thus improving the model training effect.

[0115] Please see Figure 4 In some embodiments, step S106 may include, but is not limited to, steps S401 to S403:

[0116] Step S401: Perform self-attention calculation on the sample text encoding vector through the attention layer to obtain the sample text representation vector;

[0117] Step S402: Attention is calculated on the sample speech representation vector and sample text representation vector through the attention layer to obtain the initial relation feature vector;

[0118] Step S403: Based on the preset vector feature dimension, map the initial relation feature vector to the preset vector space to obtain the sample relation feature vector.

[0119] In step S401 of some embodiments, when performing self-attention computation on the sample text encoding vector through the attention layer, a scaled dot product can be used to compute each attention head used for self-attention computation, wherein each attention head... i It can be represented as in, h represents the number of attention heads, and d represents the sample text representation vector y. i vector dimension, All are trainable projection matrices, and softmax is the normalized probability function. By concatenating each attention head, a sample text representation vector is obtained, which can be represented as O = concat(head1, ..., head...). h W o concat is a concatenation function, W o This is the projection matrix.

[0120] In step S402 of some embodiments, when performing attention calculation on the sample speech representation vector and the sample text representation vector through the attention layer, a multi-head attention mechanism can be used to construct the relationship between the sample text encoding vector and the sample speech representation vector. First, the multi-head attention mechanism is used to process the sample speech representation vector H = {h1, h2, ..., h...} s Attention calculations are performed to obtain each attention head used for multi-head attention calculations, where each attention head... j It can be represented as in, l represents the number of attention heads, and d represents the number of sample words.

[0121] Sound representation vector h s vector dimension, All are trainable projection matrices, and softmax is a normalized probability function. Further, each attention head is concatenated to obtain the attention result corresponding to the sample speech representation vector. This attention result M can be represented as M = concat(head1, ..., head...).l W M concat is a concatenation function, W M Let be the projection matrix. Finally, the attention results M of the sample text representation vector O and the sample speech representation vector are concatenated to obtain the initial relation feature vector.

[0122] In step S403 of some embodiments, the preset vector feature dimension can be set according to actual business needs. For example, the vector feature dimension can be 64, 512, or 256, etc. Based on the preset vector feature dimension, the initial relation feature vector is subjected to dimension transformation processing, mapping the initial relation feature vector to a preset vector space. The spatial dimension of this vector space is consistent with the preset vector feature dimension, thereby facilitating the transformation of the initial relation feature vector from a high-dimensional space to a low-dimensional space to obtain a sample relation feature vector. The sample relation feature vector is used to characterize the correlation between the sample text encoding vector and the sample speech representation vector.

[0123] Through the above steps S401 to S403, the correlation between the sample speech representation vector and the sample text representation vector can be obtained relatively easily, thereby reflecting the model's recognition performance and improving the model's training efficiency based on this correlation.

[0124] Please see Figure 5 In some embodiments, step S107 may include, but is not limited to, steps S501 to S502:

[0125] Step S501: Calculate the loss of the sample relationship feature vector using a loss function to obtain the target loss value;

[0126] Step S502: Backpropagate the target loss value to update the model parameters of the neural network model and obtain the speech recognition model.

[0127] In step S501 of some embodiments, the loss function can be a cross-entropy loss function, etc. When calculating the loss on the sample relationship feature vector using the loss function, the predicted text content predicted based on the sample speech data in the sample relationship feature vector is compared with the sample text content, and the target loss value is used to characterize the similarity between the predicted text content and the sample text content. The smaller the target loss value, the higher the similarity between the predicted text content and the sample text content, that is, the better the training effect of the model; the larger the target loss value, the lower the similarity between the predicted text content and the sample text content, that is, the worse the training effect of the model.

[0128] In step S502 of some embodiments, backpropagation is performed based on the target loss value, and the internal parameters (i.e., loss parameters) of the neural network model are updated by optimizing the target loss value to obtain the speech recognition model. It is understood that the backpropagation principle can apply conventional backpropagation principles, and this application embodiment does not limit it.

[0129] Through the above steps S501 to S502, the training level and performance of the model can be determined relatively easily. The internal parameters of the neural network model can be continuously adjusted according to the target loss value so that the recognition performance of the neural network model meets the training requirements, thereby obtaining a speech recognition model that can be used to recognize the text content of target speech data.

[0130] The training method of the model in this application embodiment obtains sample speech data and corresponding sample text data; inputs the sample speech data and sample text data into a preset neural network model, wherein the neural network model includes an encoding network and a decoding network; extracts features from the sample speech data through the convolutional layer of the encoding network to obtain sample speech feature vectors; and performs attention calculation on the sample speech feature vectors through the sparse attention layer of the encoding network to obtain sample speech representation vectors. This method can conveniently filter redundant information unrelated to speech recognition in the sample speech data, effectively reducing the total amount of information and improving the model's recognition speed. Furthermore, compared to the commonly used transformer model network structure, the neural network model in this application embodiment uses a sparse attention layer, which simplifies the model structure, making the model more lightweight and allowing the trained speech recognition model to be easily deployed on mobile devices and more network devices, thus improving the model's applicability. Furthermore, the sample text data is encoded using a decoding network to obtain sample text encoding vectors. These vectors and the sample speech representation vectors are then reconstructed using the attention layer of the decoding network to obtain sample relationship feature vectors. These feature vectors characterize the correlation between the sample text encoding vectors and the sample speech representation vectors, reflecting the model's recognition performance and improving training efficiency. Finally, the neural network model is trained by updating its parameters using a pre-defined loss function and the sample relationship feature vectors, resulting in a speech recognition model. This allows for continuous adjustment of the neural network model's internal parameters to ensure its recognition performance meets training requirements. This leads to a speech recognition model capable of recognizing the text content of target speech data, improving training effectiveness. Consequently, intelligent customer service robots can more accurately identify the requests represented by the speech data of service recipients during conversations, improving targeted responses and service feedback. This effectively enhances the quality and effectiveness of dialogue in financial transactions, enabling intelligent voice dialogue services, improving customer service quality and satisfaction, and ultimately increasing business conversion rates.

[0131] Please see Figure 6 This application also provides a speech recognition method, which may include, but is not limited to, steps S601 to S602:

[0132] Step S601: Obtain the target speech data to be processed;

[0133] Step S602: Input the target speech data into the speech recognition model for recognition processing to obtain target text data, wherein the target text data is used to represent the speech content of the target speech data; the speech recognition model is trained according to the training method of the first aspect.

[0134] In step S601 of some embodiments, a web crawler can be written, and after setting the data source, data can be crawled in a targeted manner to obtain the target speech data to be processed. The data source can be various types of network platforms, social media, or certain specific audio databases, etc. The target speech data can be the music material of the target speaker, a speech report, chat dialogue, etc., without limitation.

[0135] In step S602 of some embodiments, the target speech data can be input into a speech recognition model for recognition processing through a preset computer program or other script program. The target speech data is encoded by the encoding network of the speech recognition model to extract the speech content information in the target speech data. The speech content information of the target speech data is filtered by the sparse attention layer of the encoding network to remove redundant and irrelevant information. A target speech representation vector is generated based on the useful speech content information. The target speech representation vector is then input into a decoding network to decode the target speech representation vector, extract the text features corresponding to the target speech data, and generate the target text data corresponding to the target speech data. This achieves the recognition processing of the target speech data and obtains the target text data, thereby representing the speech content of the target speech data.

[0136] For example, when processing financial transactions, if the target dialogue type is "query", the corresponding object intent is "business query" or "user data query". For instance, if the target voice data input by the object is "details query", then the target dialogue type is determined to be "data query", meaning the object intent is "user data query".

[0137] The speech recognition method of this application involves inputting target speech data into a speech recognition model for processing. The model's encoding network encodes the target speech data, extracting speech content information. A sparse attention layer within the encoding network filters this speech content information, removing redundant and irrelevant information. This process effectively reduces the total amount of information and improves speech recognition speed. Furthermore, the decoding network decodes the target speech representation vector, facilitating the generation of target text representing the speech content, thus enhancing both accuracy and efficiency.

[0138] Please see Figure 7This application also provides a model training apparatus that can implement the above-described model training method. The training apparatus includes:

[0139] The sample data acquisition module 701 is used to acquire sample speech data and sample text data corresponding to the sample speech data;

[0140] The data input module 702 is used to input sample speech data and sample text data into a preset neural network model, wherein the neural network model includes an encoding network and a decoding network;

[0141] The feature extraction module 703 is used to extract features from the sample speech data through the convolutional layer of the encoding network to obtain the sample speech feature vector;

[0142] The calculation module 704 is used to perform attention calculation on the sample speech feature vector through the sparse attention layer of the encoding network to obtain the sample speech representation vector.

[0143] The encoding module 705 is used to encode the sample text data through the decoding network to obtain the sample text encoding vector;

[0144] The reconstruction module 706 is used to reconstruct the sample text encoding vector and the sample speech representation vector through the attention layer of the decoding network to obtain the sample relationship feature vector. The sample relationship feature vector is used to characterize the degree of correlation between the sample text encoding vector and the sample speech representation vector.

[0145] The training module 707 is used to update the parameters of the neural network model using a preset loss function and sample relationship feature vectors to train the neural network model and obtain a speech recognition model.

[0146] In some embodiments, the feature extraction module 703 includes:

[0147] A convolutional unit is used to perform convolution processing on sample speech data through convolutional layers to obtain sample speech convolution vectors.

[0148] The sampling unit is used to downsample the convolutional vector of the sample speech through the convolutional layer to obtain the sample speech sampling features;

[0149] The encoding unit is used to perform positional encoding on the sampled speech features to obtain the sample speech feature vector.

[0150] In some embodiments, the computing module 704 includes:

[0151] The normalization unit is used to normalize the sample speech feature vector through a preset function of the sparse attention layer to obtain sample probability features.

[0152] The filtering unit is used to filter the sample speech feature vectors according to the sample probability features and the preset probability threshold to obtain candidate speech feature vectors, wherein the candidate speech feature vectors are the sample speech feature vectors whose sample probability features are greater than the probability threshold.

[0153] The first computing unit is used to perform attention calculation on the candidate speech feature vectors according to preset weight parameters to obtain the sample speech representation vector.

[0154] In some embodiments, the reconfiguration module 706 includes:

[0155] The second computational unit is used to perform self-attention computation on the sample text encoding vector through the attention layer to obtain the sample text representation vector.

[0156] The third computational unit is used to perform attention calculations on the sample speech representation vector and the sample text representation vector through the attention layer to obtain the initial relation feature vector;

[0157] The mapping unit is used to map the initial relation feature vector to a preset vector space according to the preset vector feature dimension, so as to obtain the sample relation feature vector.

[0158] In some embodiments, the training module 707 includes:

[0159] The loss calculation unit is used to calculate the loss of the sample relationship feature vector through a loss function to obtain the target loss value;

[0160] The parameter update unit is used to backpropagate the target loss value to update the model parameters of the neural network model, thereby obtaining the speech recognition model.

[0161] The specific implementation of the training device for this model is basically the same as the specific implementation of the training method for the model described above, and will not be repeated here.

[0162] Please see Figure 8 This application also provides a speech recognition device that can implement the above-described speech recognition method. The device includes:

[0163] The target speech data acquisition module 801 is used to acquire the target speech data to be processed.

[0164] The speech recognition module 802 is used to input the target speech data into the speech recognition model for recognition processing to obtain target text data, wherein the target text data is used to represent the speech content of the target speech data; the speech recognition model is trained according to the above-mentioned training device.

[0165] The specific implementation of this speech recognition device is basically the same as the specific implementation of the speech recognition method described above, and will not be repeated here.

[0166] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described speech recognition method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0167] Please see Figure 9 , Figure 9 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0168] The processor 901 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0169] The memory 902 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902 and is called and executed by the processor 901 using the speech recognition method of the embodiments of this application.

[0170] The input / output interface 903 is used to implement information input and output;

[0171] The communication interface 904 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0172] Bus 905 transmits information between various components of the device (e.g., processor 901, memory 902, input / output interface 903, and communication interface 904);

[0173] The processor 901, memory 902, input / output interface 903, and communication interface 904 are connected to each other within the device via bus 905.

[0174] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described speech recognition method.

[0175] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0176] The training method, speech recognition method, training device, electronic device, and computer-readable storage medium provided in this application embodiment acquire sample speech data and corresponding sample text data; input the sample speech data and sample text data into a preset neural network model, wherein the neural network model includes an encoding network and a decoding network; extract features from the sample speech data through the convolutional layer of the encoding network to obtain sample speech feature vectors; and perform attention calculation on the sample speech feature vectors through the sparse attention layer of the encoding network to obtain sample speech representation vectors. This method can conveniently filter redundant information unrelated to speech recognition in the sample speech data, effectively reducing the total amount of information and improving the recognition speed of the model. Furthermore, compared to the commonly used transformer model network structure, the neural network model in this application embodiment uses a sparse attention layer, which simplifies the model structure, making the model more lightweight and allowing the trained speech recognition model to be easily deployed on mobile devices and more network devices, thus improving the model's applicability. Furthermore, the sample text data is encoded using a decoding network to obtain sample text encoding vectors. These vectors and the sample speech representation vectors are then reconstructed using the attention layer of the decoding network to obtain sample relationship feature vectors. These feature vectors characterize the correlation between the sample text encoding vectors and the sample speech representation vectors, reflecting the model's recognition performance and improving training efficiency. Finally, the neural network model is trained by updating its parameters using a pre-defined loss function and the sample relationship feature vectors, resulting in a speech recognition model. This allows for continuous adjustment of the neural network model's internal parameters to ensure its recognition performance meets training requirements. This leads to a speech recognition model capable of recognizing the text content of target speech data, improving training effectiveness. Consequently, intelligent customer service robots can more accurately identify the requests represented by the speech data of service recipients during conversations, improving targeted responses and service feedback. This effectively enhances the quality and effectiveness of dialogue in financial transactions, enabling intelligent voice dialogue services, improving customer service quality and satisfaction, and ultimately increasing business conversion rates.

[0177] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0178] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0179] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0180] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0181] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0182] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0183] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0184] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0185] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0186] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0187] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A method for training a model, characterized in that, The training method includes: Obtain sample speech data and corresponding sample text data; The sample speech data and the sample text data are input into a preset neural network model, wherein the neural network model includes an encoding network and a decoding network; The sample speech data is feature extracted through the convolutional layers of the encoding network to obtain the sample speech feature vector; Attention is calculated on the sample speech feature vector through the sparse attention layer of the coding network to obtain the sample speech representation vector; The sample text data is encoded using the decoding network to obtain a sample text encoding vector. The sample text encoding vector and the sample speech representation vector are reconstructed by the attention layer of the decoding network to obtain a sample relationship feature vector, which is used to characterize the degree of correlation between the sample text encoding vector and the sample speech representation vector. The neural network model is trained by updating its parameters using a preset loss function and the sample relationship feature vector to obtain a speech recognition model. The step of performing attention calculation on the sample speech feature vector through the sparse attention layer of the encoding network to obtain the sample speech representation vector includes: The Euclidean projection of the sample speech feature vector is mapped into a probability simplex using the sparsemax function of the sparse attention layer. The probability simplex is used as the sample probability feature of the sample speech feature vector. The sample speech features whose sample probability features are greater than a preset probability threshold are determined as candidate speech feature vectors. The candidate speech feature vectors are then attention-weighted according to preset weight parameters to obtain the sample speech representation vector. The probability simplex is used to reflect the sparsity of the sample speech feature vector.

2. The training method according to claim 1, characterized in that, The step of extracting features from the sample speech data through the convolutional layers of the encoding network to obtain the sample speech feature vector includes: The sample speech data is processed by convolutional layer to obtain sample speech convolution vector; The sample speech convolution vector is downsampled by the convolutional layer to obtain the sample speech sampling features. The sampled speech features are positionally encoded to obtain the sampled speech feature vector.

3. The training method according to claim 1, characterized in that, The process of reconstructing the sample text encoding vector and the sample speech representation vector through the attention layer of the decoding network to obtain the sample relation feature vector includes: The sample text representation vector is obtained by performing self-attention calculation on the sample text encoding vector through the attention layer. The attention layer performs attention calculations on the sample speech representation vector and the sample text representation vector to obtain an initial relation feature vector. Based on the preset vector feature dimensions, the initial relation feature vector is mapped to the preset vector space to obtain the sample relation feature vector.

4. The training method according to any one of claims 1 to 3, characterized in that, The step of updating the parameters of the neural network model using a preset loss function and the sample relationship feature vector to train the neural network model and obtain a speech recognition model includes: The target loss value is obtained by calculating the loss of the sample relationship feature vector using the loss function. The target loss value is backpropagated to update the model parameters of the neural network model, thereby obtaining the speech recognition model.

5. A speech recognition method, characterized in that, The method includes: Acquire the target speech data to be processed; The target speech data is input into a speech recognition model for recognition processing to obtain target text data, wherein the target text data is used to represent the speech content of the target speech data; the speech recognition model is trained according to the training method of any one of claims 1 to 4.

6. A training device for a model, characterized in that, The training device includes: The sample data acquisition module is used to acquire sample speech data and sample text data corresponding to the sample speech data. A data input module is used to input the sample speech data and the sample text data into a preset neural network model, wherein the neural network model includes an encoding network and a decoding network; The feature extraction module is used to extract features from the sample speech data through the convolutional layers of the encoding network to obtain sample speech feature vectors; The calculation module is used to perform attention calculation on the sample speech feature vector through the sparse attention layer of the coding network to obtain the sample speech representation vector; The encoding module is used to encode the sample text data through the decoding network to obtain the sample text encoding vector; The reconstruction module is used to reconstruct the sample text encoding vector and the sample speech representation vector through the attention layer of the decoding network to obtain a sample relationship feature vector, which is used to characterize the degree of correlation between the sample text encoding vector and the sample speech representation vector. The training module is used to update the parameters of the neural network model using a preset loss function and the sample relationship feature vector, so as to train the neural network model and obtain a speech recognition model. The step of performing attention calculation on the sample speech feature vector through the sparse attention layer of the encoding network to obtain the sample speech representation vector includes: The Euclidean projection of the sample speech feature vector is mapped into a probability simplex using the sparsemax function of the sparse attention layer. The probability simplex is used as the sample probability feature of the sample speech feature vector. The sample speech features whose sample probability features are greater than a preset probability threshold are determined as candidate speech feature vectors. The candidate speech feature vectors are then attention-weighted according to preset weight parameters to obtain the sample speech representation vector. The probability simplex is used to reflect the sparsity of the sample speech feature vector.

7. A voice recognition device, characterized in that, The device includes: The target speech data acquisition module is used to acquire the target speech data to be processed. A speech recognition module is used to input the target speech data into a speech recognition model for recognition processing to obtain target text data, wherein the target text data is used to represent the speech content of the target speech data; the speech recognition model is trained by the training device according to claim 6.

8. An electronic device, characterized in that, The electronic device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the training method according to any one of claims 1 to 4 or the speech recognition method according to claim 5.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the training method according to any one of claims 1 to 4 or the speech recognition method according to claim 5.

Images