Voice data processing methods, apparatus, devices and readable storage media

Abstract

This application discloses a speech data processing method, apparatus, device, and readable storage medium. The method includes: acquiring speech data; encoding the speech data to obtain an encoding vector; generating N candidate recognition texts based on the encoding vector; aligning the N candidate recognition texts to obtain N aligned recognition texts; ensuring that the N aligned recognition texts have the same character length and / or that the positions of characters representing the same semantic meaning are the same in the N aligned recognition texts; performing vector fusion processing on the text vectors corresponding to the N aligned recognition texts to obtain a fused vector; and generating the recognition result text corresponding to the speech data based on the fused vector and the encoding vector. Using this application can improve the accuracy of speech data recognition.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a voice data processing method, apparatus, device, and readable storage medium. Background Technology

[0002] In existing speech recognition models, the speech recognition model often generates several recognition texts (or feature vectors containing the semantics of the recognition texts) based on speech features, scores the words corresponding to the recognition texts, and determines the recognition text with the highest total score as the final recognition result. Since it only depends on the score of words, if there are inaccurate pronunciations in the user's input speech, the inaccurate pronunciations may cause some words to have excessively high scores. Thus, recognition based on word scores may contain incorrect recognition results, resulting in low recognition accuracy. Summary of the Invention

[0003] This application provides a voice data processing method, apparatus, device, and readable storage medium, which can improve the accuracy of voice data recognition.

[0004] One embodiment of this application provides a voice data processing method, including:

[0005] Acquire speech data, encode the speech data to obtain an encoding vector, and the encoding vector represents the speech features of the speech data;

[0006] N candidate texts are generated based on the encoding vectors; the candidate texts correspond to the speech features, and N is a positive integer.

[0007] The N candidate recognition texts are aligned to obtain N aligned recognition texts. The character lengths of the N aligned recognition texts are all the same and / or the positions of characters representing the same semantics in the N aligned recognition texts are all the same.

[0008] Obtain the text vectors corresponding to the N aligned and recognized texts, and get N text vectors;

[0009] Perform vector fusion processing on N text vectors to obtain a fused vector;

[0010] The recognition result text corresponding to the speech data is generated based on the fusion vector and the encoding vector.

[0011] The speech data is encoded to obtain an encoded vector, which includes:

[0012] Input the speech data into the speech recognition model; the speech recognition model includes a speech coding layer;

[0013] In the speech coding layer, the speech data is segmented into frames to obtain M speech frames corresponding to the speech data; M is a positive integer.

[0014] Feature extraction is performed on M speech frames to obtain M speech embedding vectors;

[0015] Obtain the position embedding vectors corresponding to the M speech embedding vectors respectively. Based on the M position embedding vectors and the M speech embedding vectors, generate M encoded input vectors. The position embedding vectors are used to indicate the position of the speech frame data corresponding to the speech embedding vector in the speech data.

[0016] Generate an encoded vector based on M encoded input vectors.

[0017] Among them, the M encoded input vectors include the encoded input vector E i Where i is a positive integer; generate an encoding vector based on M encoding input vectors, including:

[0018] Obtain the character encoding vector corresponding to each character to be recognized in the character database;

[0019] Concatenate the character encoding vectors to obtain the character encoding sequence;

[0020] Encoding input vector E i Cross-attention processing is performed with the character encoding sequence to obtain the encoded input vector E. i The corresponding unit attention vector A i Unit attention vector A i Including the encoded input vector E i The attention score between each character to be recognized in the character database;

[0021] When the unit attention vectors corresponding to the M encoded input vectors are obtained, the M unit attention vectors are concatenated to obtain the encoded vector.

[0022] Among them, N candidate recognition texts are generated based on the encoding vector, including:

[0023] Based on the attention score corresponding to each character to be identified in the encoding vector, P character combination lists are generated; P is a positive integer, and each character combination list includes M character labels. Each character label in the character combination list is determined by a different unit attention vector.

[0024] Obtain the total attention score for each of the P character combination lists; the total attention score for the character combination list is the sum of the attention scores for each of the M character labels.

[0025] Obtain the candidate text value N, and based on the candidate text value N and the total attention score, obtain a list of N character combinations from a list of P character combinations;

[0026] The N character combination list is time-aligned to obtain N candidate recognition texts.

[0027] Specifically, based on the attention score corresponding to each character to be recognized in the encoding vector, a list of P character combinations is generated, including:

[0028] Obtain the attention threshold, and determine the attention scores that are greater than or equal to the attention threshold among the M unit attention vectors in the encoding vector as candidate attention scores, and determine the characters to be recognized associated with the candidate attention scores as character labels;

[0029] Select and arrange the character labels to obtain a list of P character combinations.

[0030] Among them, the list of N character combinations includes the list of character combinations B. i Character combination list B i Includes the null character C i and the null character C i+1 null character C i and the null character C i+1 The adjacent null characters, null character C i With the null character C i+1 The list contains at least two identical character labels, where i is a positive integer; the list of N character combinations is time-aligned to obtain N candidate recognition texts, including:

[0031] The null character C will be located at i With the null character C i+1 At least two identical character tags are merged into one character tag;

[0032] When character combination list B i When at least two identical character tags located between adjacent empty characters have been merged, delete the character combination list B. i The empty character in the list gives us a list of character combinations B. i The corresponding candidate text for recognition.

[0033] The process involves aligning N candidate texts to obtain N aligned texts, including:

[0034] Input N candidate texts into the speech recognition model; the speech recognition model includes an alignment processing layer.

[0035] In the alignment processing layer, a reference text is determined from N candidate texts for recognition;

[0036] N candidate recognition texts are obtained to align with the editing information corresponding to the reference recognition text. The editing information includes the editing distance and editing processing type corresponding to each candidate recognition text when converted into the reference recognition text. The sum of the editing distances satisfies the minimum editing distance condition.

[0037] Determine the location to be processed among N candidate texts based on editing information;

[0038] Placeholders are inserted at the positions to be processed in the N candidate recognition texts to obtain N aligned recognition texts.

[0039] Among them, N candidate recognition texts include candidate recognition text F i , where i is a positive integer; the editing information includes the candidate recognition text F. i Corresponding editing information G i Based on editing information, determine the location to be processed among N candidate recognition texts, including:

[0040] If you edit information G i If the indicated editing process type is insertion, then the candidate recognition text F will be... i The editing position corresponding to the insertion process is determined as the position to be processed;

[0041] If you edit information G i If the indicated editing process type is deletion, then the candidate identified text F will be... i The corresponding business location in the reference recognition text is determined as the location to be processed; the business location and the candidate recognition text F i Remove the corresponding edit position associated with the process.

[0042] Specifically, the text vectors corresponding to the N aligned and recognized texts are obtained, resulting in N text vectors, including:

[0043] Input N aligned and recognized texts into the speech recognition model; the speech recognition model includes a fusion processing layer;

[0044] In the fusion processing layer, feature extraction is performed on N aligned recognition texts to obtain text vectors corresponding to each of the N aligned recognition texts; the character length of each aligned recognition text is S, where S is a positive integer; the text vector includes S unit character vectors, and each unit character vector is associated with a character in the aligned recognition text;

[0045] Perform vector fusion processing on N text vectors to obtain a fused vector, including:

[0046] Average the unit character vectors at the same position in N text vectors to obtain S unit average vectors. Concatenate the S unit average vectors to obtain a merged vector.

[0047] The fusion vector includes S unit average vectors, where S is a positive integer; the recognition result text corresponding to the speech data is generated based on the fusion vector and the encoding vector, including:

[0048] The fused vector and the encoded vector are input into the speech recognition model; the speech recognition model includes a speech decoding layer.

[0049] In the speech decoding layer, cross-attention processing is performed on the fusion coding vector and the fusion vector to obtain the attention result vector. The attention result vector includes the unit result vectors corresponding to S unit average vectors, and the unit result vector includes the attention score corresponding to each character to be recognized in the character database.

[0050] S recognition characters are generated based on the S unit result vectors in the attention result vector, and the S recognition characters are determined as the recognition result text corresponding to the speech data.

[0051] This application provides another method for voice data processing, including:

[0052] Obtain sample speech data and the corresponding answer text, and input the sample speech data and answer text into the initial recognition model;

[0053] In the initial recognition model, the sample speech data is encoded to obtain a sample encoding vector, and N sample candidate recognition texts are generated based on the sample encoding vector; N is a positive integer; the sample encoding vector represents the speech features of the sample speech data, and the sample candidate recognition texts correspond to the speech features.

[0054] Based on the answer text, the N candidate sample recognition texts are aligned to obtain N aligned sample recognition texts; the character length of the N aligned sample recognition texts is the same as the character length of the answer text, and the positions of characters representing the same semantics in the N aligned sample recognition texts and the answer text are the same;

[0055] Obtain the sample text vectors corresponding to the N sample aligned recognition texts, and obtain N sample text vectors. Perform vector fusion processing on the N sample text vectors to obtain the sample fusion vector. Generate the prediction result text corresponding to the sample speech data based on the sample fusion vector and the sample encoding vector.

[0056] The model loss value is generated based on the predicted result text and the answer text. The model parameters of the initial recognition model are adjusted based on the model loss value until the initial recognition model meets the model convergence condition, thus obtaining the speech recognition model. The speech recognition model is used to generate the recognition result text corresponding to the speech data.

[0057] One embodiment of this application provides a voice data processing apparatus, including:

[0058] The speech feature extraction module is used to acquire speech data, encode the speech data to obtain an encoding vector, and the encoding vector represents the speech features of the speech data.

[0059] The candidate text generation module is used to generate N candidate recognition texts based on the encoding vector. The candidate recognition texts correspond to speech features, and N is a positive integer.

[0060] The alignment module is used to align N candidate recognition texts to obtain N aligned recognition texts; the character lengths of the N aligned recognition texts are all the same and / or the positions of characters representing the same semantics in the N aligned recognition texts are all the same.

[0061] The acquisition module is used to acquire the text vectors corresponding to N aligned recognition texts, and obtain N text vectors;

[0062] The fusion processing module is used to perform vector fusion processing on N text vectors to obtain a fused vector;

[0063] The text generation module is used to generate the recognition result text corresponding to the speech data based on the fusion vector and the encoding vector.

[0064] In one possible implementation, the speech feature extraction module is used to encode the speech data, and when obtaining the encoded vector, it is specifically used to perform the following operations:

[0065] Input the speech data into the speech recognition model; the speech recognition model includes a speech coding layer;

[0066] In the speech coding layer, the speech data is segmented into frames to obtain M speech frames corresponding to the speech data; M is a positive integer.

[0067] Feature extraction is performed on M speech frames to obtain M speech embedding vectors;

[0068] Obtain the position embedding vectors corresponding to the M speech embedding vectors respectively. Based on the M position embedding vectors and the M speech embedding vectors, generate M encoded input vectors. The position embedding vectors are used to indicate the position of the speech frame data corresponding to the speech embedding vector in the speech data.

[0069] Generate an encoded vector based on M encoded input vectors.

[0070] In one possible implementation, the M encoded input vectors include the encoded input vector E. i Where i is a positive integer; the speech feature extraction module is used to generate encoded vectors based on M encoded input vectors, specifically to perform the following operations:

[0071] Obtain the character encoding vector corresponding to each character to be recognized in the character database;

[0072] Concatenate the character encoding vectors to obtain the character encoding sequence;

[0073] Encoding input vector E i Cross-attention processing is performed with the character encoding sequence to obtain the encoded input vector E. i The corresponding unit attention vector A i Unit attention vector A i Including the encoded input vector E i The attention score between each character to be recognized in the character database;

[0074] When the unit attention vectors corresponding to the M encoded input vectors are obtained, the M unit attention vectors are concatenated to obtain the encoded vector.

[0075] In one possible implementation, when the candidate text generation module generates N candidate recognition texts based on the encoding vector, it specifically performs the following operations:

[0076] Based on the attention score corresponding to each character to be identified in the encoding vector, P character combination lists are generated; P is a positive integer, and each character combination list includes M character labels. Each character label in the character combination list is determined by a different unit attention vector.

[0077] Obtain the total attention score for each of the P character combination lists; the total attention score for the character combination list is the sum of the attention scores for each of the M character labels.

[0078] Obtain the candidate text value N, and based on the candidate text value N and the total attention score, obtain a list of N character combinations from a list of P character combinations;

[0079] The N character combination list is time-aligned to obtain N candidate recognition texts.

[0080] In one possible implementation, when the candidate text generation module generates a list of P character combinations based on the attention score corresponding to each character to be recognized in the encoding vector, it specifically performs the following operations:

[0081] Obtain the attention threshold, and determine the attention scores that are greater than or equal to the attention threshold among the M unit attention vectors in the encoding vector as candidate attention scores, and determine the characters to be recognized associated with the candidate attention scores as character labels;

[0082] Select and arrange the character labels to obtain a list of P character combinations.

[0083] In one possible implementation, the list of N character combinations includes the character combination list B. i Character combination list B i Includes the null character C i and the null character C i+1 null character C i and the null character C i+1 The adjacent null characters, null character C i With the null character C i+1 The list includes at least two identical character labels, where i is a positive integer; the candidate text generation module is used to perform temporal alignment processing on the list of N character combinations to obtain N candidate recognition texts, specifically for the following operations:

[0084] The null character C will be located at i With the null character C i+1 At least two identical character tags are merged into one character tag;

[0085] When character combination list B i When at least two identical character tags located between adjacent empty characters have been merged, delete the character combination list B. i The empty character in the list gives us a list of character combinations B. i The corresponding candidate text for recognition.

[0086] In one possible implementation, the alignment module is used to align N candidate recognition texts. When N aligned recognition texts are obtained, it specifically performs the following operations:

[0087] Input N candidate texts into the speech recognition model; the speech recognition model includes an alignment processing layer.

[0088] In the alignment processing layer, a reference text is determined from N candidate texts for recognition;

[0089] N candidate recognition texts are obtained to align with the editing information corresponding to the reference recognition text. The editing information includes the editing distance and editing processing type corresponding to each candidate recognition text when converted into the reference recognition text. The sum of the editing distances satisfies the minimum editing distance condition.

[0090] Determine the location to be processed among N candidate texts based on editing information;

[0091] Placeholders are inserted at the positions to be processed in the N candidate recognition texts to obtain N aligned recognition texts.

[0092] In one possible implementation, the N candidate recognition texts include candidate recognition text F. i , where i is a positive integer; the editing information includes the candidate recognition text F.i Corresponding editing information G i The alignment processing module is used to determine the position to be processed among N candidate recognition texts based on editing information. Specifically, it performs the following operations:

[0093] If you edit information G i If the indicated editing process type is insertion, then the candidate recognition text F will be... i The editing position corresponding to the insertion process is determined as the position to be processed;

[0094] If you edit information G i If the indicated editing process type is deletion, then the candidate identified text F will be... i The corresponding business location in the reference recognition text is determined as the location to be processed; the business location and the candidate recognition text F i Remove the corresponding edit position associated with the process.

[0095] In one possible implementation, the acquisition module is used to acquire the text vectors corresponding to the N aligned recognition texts. When the N text vectors are obtained, the module is specifically used to perform the following operations:

[0096] Input N aligned and recognized texts into the speech recognition model; the speech recognition model includes a fusion processing layer;

[0097] In the fusion processing layer, feature extraction is performed on N aligned recognition texts to obtain text vectors corresponding to each of the N aligned recognition texts; the character length of each aligned recognition text is S, where S is a positive integer; the text vector includes S unit character vectors, and each unit character vector is associated with a character in the aligned recognition text;

[0098] The fusion processing module is used to perform vector fusion processing on N text vectors. When obtaining the fused vector, it specifically performs the following operations:

[0099] Average the unit character vectors at the same position in N text vectors to obtain S unit average vectors. Concatenate the S unit average vectors to obtain a merged vector.

[0100] In one possible implementation, the fusion vector comprises S unit average vectors, where S is a positive integer; the recognition text generation module, when generating the recognition result text corresponding to the speech data based on the fusion vector and the encoding vector, specifically performs the following operations:

[0101] The fused vector and the encoded vector are input into the speech recognition model; the speech recognition model includes a speech decoding layer.

[0102] In the speech decoding layer, cross-attention processing is performed on the fusion coding vector and the fusion vector to obtain the attention result vector. The attention result vector includes the unit result vectors corresponding to S unit average vectors, and the unit result vector includes the attention score corresponding to each character to be recognized in the character database.

[0103] S recognition characters are generated based on the S unit result vectors in the attention result vector, and the S recognition characters are determined as the recognition result text corresponding to the speech data.

[0104] One embodiment of this application provides another voice data processing apparatus, including:

[0105] The sample data input module is used to acquire sample speech data and the corresponding answer text, and input the sample speech data and answer text into the initial recognition model.

[0106] The sample feature extraction module is used to encode the sample speech data in the initial recognition model to obtain the sample encoding vector, and generate N sample candidate recognition texts based on the sample encoding vector; N is a positive integer; the sample encoding vector represents the speech features of the sample speech data, and the sample candidate recognition texts correspond to the speech features;

[0107] The initial alignment processing module is used to align N sample candidate recognition texts based on the answer text to obtain N sample aligned recognition texts. The character length of the N sample aligned recognition texts is the same as the character length of the answer text, and the positions of characters representing the same semantics in the N sample aligned recognition texts and the answer text are the same.

[0108] The initial fusion processing module is used to obtain the sample text vectors corresponding to the N sample aligned recognition texts, obtain N sample text vectors, perform vector fusion processing on the N sample text vectors to obtain sample fusion vectors, and generate the prediction result text corresponding to the sample speech data based on the sample fusion vectors and sample encoding vectors.

[0109] The model training module is used to generate model loss values ​​based on the predicted result text and the answer text. The model loss values ​​are used to adjust the model parameters of the initial recognition model until the initial recognition model meets the model convergence condition, thus obtaining the speech recognition model. The speech recognition model is used to generate the recognition result text corresponding to the speech data.

[0110] One embodiment of this application provides a computer device, including: a processor, a memory, and a network interface;

[0111] The processor is connected to a memory and a network interface. The network interface is used to provide data communication functions, and the memory is used to store computer programs. When the computer program is executed by the processor, the computer device performs the method provided in the embodiments of this application.

[0112] One aspect of this application provides a computer-readable storage medium storing a computer program adapted to be loaded and executed by a processor, so that a computer device having the processor performs the method provided in this application.

[0113] One embodiment of this application provides a computer program product comprising a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, causing the computer device to perform the method provided in this application embodiment.

[0114] This application embodiment generates N candidate recognition texts corresponding to the speech data. Alignment processing is then performed on these N candidate texts to determine characters representing the same semantic meaning at the same character position, resulting in N aligned recognition texts with the same character length. Further fusion processing is performed on these N aligned recognition texts to merge vector elements at the same character position, thereby capturing more dimensions of semantic features. By generating the recognition result text corresponding to the speech data through fused vectors and encoded vectors, the semantic features of the N aligned recognition texts are considered as much as possible while incorporating the speech features contained in the encoded vectors. This reduces the impact of inaccurate user pronunciation, thereby improving the accuracy and stability of speech data recognition. Attached Figure Description

[0115] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0116] Figure 1 This is a schematic diagram of a network architecture provided in an embodiment of this application;

[0117] Figure 2 This is a schematic diagram of a voice data processing scenario provided in an embodiment of this application;

[0118] Figure 3 This is a flowchart illustrating a voice data processing method provided in an embodiment of this application. Figure 1;

[0119] Figure 4 This is a flowchart illustrating a voice data processing method provided in an embodiment of this application. Figure 2 ;

[0120] Figure 5 This is a schematic diagram of the model structure of a speech recognition model provided in an embodiment of this application;

[0121] Figure 6 This is a flowchart illustrating a voice data processing method provided in an embodiment of this application. Figure 3 ;

[0122] Figure 7 This is a schematic diagram of the structure of a voice data processing device provided in an embodiment of this application. Figure 1 ;

[0123] Figure 8 This is a schematic diagram of the structure of a voice data processing device provided in an embodiment of this application. Figure 2 ;

[0124] Figure 9 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0125] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0126] Please see Figure 1 , Figure 1 This is a schematic diagram of a network architecture provided in an embodiment of this application. For example... Figure 1 As shown, the network architecture may include a service server 100 and a terminal device cluster. The terminal device cluster may include terminal devices 10a, 10b, ..., 10n. Each terminal device in the terminal device cluster may have a communication connection with the service server 100. For example, terminal device 10a may have a communication connection with the service server 100, and terminal device 10b may have a communication connection with the service server 100. The communication connection is not limited to a specific method. It may be a direct or indirect connection via wired communication, a direct or indirect connection via wireless communication, or other methods. This application does not impose any restrictions on these methods.

[0127] Each terminal device in the terminal device cluster can include: smartphones, tablets, laptops, desktop computers, intelligent voice interaction devices, smart home appliances (e.g., smart TVs), wearable devices, in-vehicle terminals, aircraft, and other intelligent terminals with data processing capabilities. In-vehicle terminals can be terminal devices used in intelligent transportation scenarios and assisted driving scenarios. It should be understood that, for example... Figure 1 Each terminal device in the terminal device cluster shown can be equipped with an application client that has data processing capabilities. The business server 100 can be the server corresponding to the application client. When the application client runs on each terminal device, it can communicate with the aforementioned... Figure 1 Data interaction is performed between the business servers 100 shown.

[0128] Specifically, the application client may include: in-vehicle client, smart home client, entertainment client (e.g., game client), multimedia client (e.g., video client), social client, and information client (e.g., news client). In this embodiment, the application client may be integrated into a client (e.g., a social client) or may be a standalone client (e.g., a news client). This embodiment does not limit the type of application client.

[0129] like Figure 1 As shown, for ease of understanding, terminal device 10a is used as an example. The user can input voice data in the application client of terminal device 10a, and terminal device 10a can send the voice data to business server 100. Business server 100 can deploy a voice recognition model. The voice recognition model can be implemented by an encoding recognition model and a decoding recognition model. The encoding recognition model and the decoding recognition model can be separate, independent models, or they can be integrated into a single model, i.e., a voice recognition model. This embodiment of the application does not impose any limitations on this.

[0130] Optionally, the encoding recognition model and the decoding recognition model can be deployed separately as independent models. The encoding recognition model can be deployed in the terminal device 10a, and the decoding recognition model can be deployed in the business server 100, improving the security of the original voice data and protecting privacy. At the same time, the decoding recognition model can also be directly applied to systems that have already deployed other recognition models to correct the output results, reducing deployment costs.

[0131] The business server 100 can input speech data into the encoding and recognition model, generate encoding vectors corresponding to the speech data through the encoding and recognition model, and generate N candidate recognition texts based on the encoding vectors. The encoding and recognition model can be a model based on CTC (Connectionist Temporal Classification) technology and ASR (Automatic speech recognition) technology.

[0132] The service server 100 can input the encoded vector and N candidate recognition texts into the decoding recognition model. The decoding recognition model can include an alignment processing layer, a fusion processing layer, and a speech decoding layer. The service server 100 can perform alignment processing on the N candidate recognition texts through the alignment processing layer in the decoding recognition model to obtain N aligned recognition texts. The N aligned recognition texts can satisfy one or more of a first alignment condition or a second alignment condition. The first alignment condition means that the character lengths of the N aligned recognition texts are all the same, and the second alignment condition means that the positions of characters representing the same semantic meaning in the N aligned recognition texts are all the same.

[0133] In the fusion processing layer of the decoding and recognition model, the service server 100 can perform vector fusion processing on the text vectors corresponding to the N aligned recognition texts to obtain a fusion vector. The fusion vector can incorporate the semantics of the characters contained in the N aligned recognition texts. In the speech decoding layer of the decoding and recognition model, the service server 100 can generate the recognition result text corresponding to the speech data based on the fusion vector and the encoding vector. The service server 100 can send the recognition result text to the terminal device 10a, and the terminal device 10a can display the recognition result text corresponding to the speech data in the application client.

[0134] It is understandable that the candidate recognition text can be the preliminary recognition result generated by the encoding recognition model. The decoding recognition model can be an error correction model for the encoding recognition model. The decoding recognition model can fuse N aligned recognition texts to obtain a fusion vector containing the semantics of the N aligned recognition texts. Then, the error correction recognition result (i.e., the recognition result text) can be generated based on the fusion vector and the encoding vector.

[0135] This application embodiment integrates the encoding recognition model and its corresponding error correction model into a single speech recognition model. This integrated speech recognition model reduces data transmission and intermediate processing overhead, and allows for the sharing of certain feature vectors. Therefore, it improves performance, increases computational efficiency, and reduces deployment costs. By generating the recognition result text corresponding to the speech data through fusion vectors and encoding vectors, it considers as many semantic features as possible from the N aligned recognition texts while incorporating the pronunciation information contained in the encoding vectors. This reduces the impact of inaccurate user pronunciation, thereby improving the accuracy and stability of the recognized speech data.

[0136] Please see Figure 2 , Figure 2 This is a schematic diagram of a voice data processing scenario provided in an embodiment of this application. For example... Figure 2 As shown, the application client in terminal device 10a can be a social client. As shown on page 101, in the social client, user 1 can have a voice chat with user 2. User 2 can select the voice data sent by user 1 and click the "recognize" control. Terminal device 10a can respond to the trigger operation of the "recognize" control and send the voice data to business server 100.

[0137] The business server 100 can be deployed with a speech recognition model, which may include a speech coding layer, an alignment processing layer, a fusion processing layer, and a speech decoding layer. The business server 100 can encode speech data through the speech coding layer in the speech recognition model to generate an encoding vector, and generate N candidate recognition texts corresponding to the speech data based on the encoding vector. The N candidate recognition texts may include candidate recognition text 1, ..., candidate recognition text n.

[0138] The business server 100 can perform alignment processing on N candidate recognition texts through the alignment processing layer in the speech recognition model. Alignment processing can involve adding placeholders to the N candidate recognition texts to ensure that the N aligned recognition texts satisfy one or more of a first alignment condition or a second alignment condition. The first alignment condition means that the character lengths of the N aligned recognition texts are all the same, and the second alignment condition means that the positions of characters representing the same semantic meaning in the N aligned recognition texts are all the same. A placeholder can be represented as "φ". For example, when aligning candidate recognition text 1 and candidate recognition text n, candidate recognition text 1 can be represented as "ABC", and candidate recognition text n can be represented as "ACD". After alignment processing, the aligned recognition text 1 corresponding to candidate recognition text 1 can be represented as "ABCφ", and the aligned recognition text n corresponding to candidate recognition text n can be represented as "AφCD". Characters representing the same semantic meaning in aligned recognition text 1 and aligned recognition text n, such as the character "A", are in the same position.

[0139] The business server 100 can perform vector fusion processing on the text vectors corresponding to the N aligned recognition texts through the fusion processing layer in the speech recognition model to obtain a fusion vector. The fusion vector can incorporate the semantics of the characters contained in the N aligned recognition texts. Further, the speech recognition model can input the generated fusion vector and encoding vector into the speech decoding layer, where the speech decoding layer generates the recognition result text corresponding to the speech data based on the fusion vector and encoding vector. The business server 100 can then send the recognition result text to the terminal device 10a. As shown on page 102, the terminal device 10a can display the recognition result text corresponding to the speech data in the application client.

[0140] The speech recognition technology provided by the speech recognition model in this application embodiment can also be applied to the following scenarios: In a virtual assistant scenario, users can control the intelligent assistant to perform functions on a computer device via voice, such as setting reminders or playing music. In a smart transportation scenario, users can control navigation and query traffic routes via voice, improving driving safety and convenience.

[0141] Please see Figure 3 , Figure 3 This is a flowchart illustrating a voice data processing method provided in an embodiment of this application. Figure 1 This voice data processing method can be executed by a computer device, which can be, for example, Figure 1 The business server 100 shown or any terminal device in the terminal device cluster, such as terminal device 10a. The following description will use the example of this voice data processing method being executed by a computer device. This voice data processing method may include at least the following steps S101-S105:

[0142] Step S101: Acquire speech data, encode the speech data to obtain an encoding vector, and the encoding vector represents the speech features of the speech data;

[0143] Specifically, computer devices can be equipped with speech recognition models. The computer devices can acquire speech data and input the speech data into the speech recognition model. The speech recognition model includes a speech coding layer, an alignment processing layer, a fusion processing layer, and a speech decoding layer.

[0144] In the speech coding layer, the computer device can select a certain frame length, such as 15ms, and perform frame segmentation processing on the speech data based on this frame length. Frame segmentation processing can divide the continuous speech signal in the speech data into M speech frames containing short signals.

[0145] Computer devices can extract features from M speech frames separately. For example, they can embed the speech features from the M speech frames into vectors of the same size to obtain M speech embedding vectors.

[0146] A computer device can acquire positional embeddings corresponding to M speech embedding vectors. These positional embeddings indicate the position of the corresponding speech frame data within the speech data. The computer device can concatenate the speech embedding vectors with their associated positional embeddings to obtain M coded input vectors. The computer device can then generate coded vectors based on these M input vectors. These coded vectors can include the association between each input vector and a character database; that is, each input vector can be correlated with each character to be recognized in the character database to obtain a coded vector that characterizes the correlation between each speech frame data and each character to be recognized. These coded vectors can represent the speech features of the speech data, such as phonemes, syllables, and intonation.

[0147] Step S102: Generate N candidate recognition texts based on the encoding vectors; the candidate recognition texts correspond to speech features, and N is a positive integer;

[0148] Specifically, the computer device can determine several character labels corresponding to each encoded input vector based on the association between the encoded input vector contained in the encoded vector and the characters to be recognized in the character database. The computer device can select one character label from the several character labels corresponding to one encoded input vector, and so on, to obtain M character labels corresponding to M encoded input vectors. The M character labels arranged in the order of the M encoded input vectors are determined as a candidate text for recognition. Furthermore, the computer device can select different character labels from the several character labels corresponding to one encoded input vector, and then determine the newly selected M character labels as new candidate texts for recognition, thereby obtaining N distinct candidate texts for recognition. Since the encoded vector can be used to represent speech features, the candidate texts output based on the encoded vector can correspond to speech features.

[0149] Step S103: Align the N candidate recognition texts to obtain N aligned recognition texts; the N aligned recognition texts have the same character length and / or the positions of characters representing the same semantics in the N aligned recognition texts are the same.

[0150] Specifically, the computer device can input N candidate texts into the alignment processing layer of the speech recognition model. The N candidate texts can include candidate text 1, candidate text 2, and candidate text 3. Candidate text 1 can be specifically represented as "ABCD", candidate text 2 can be specifically represented as "ACDE", and candidate text 3 can be specifically represented as "ABC".

[0151] In the alignment processing layer, the computer device can perform alignment processing on N candidate recognition texts. The alignment processing can be done by adding placeholders to the N candidate recognition texts so that the character lengths of the N aligned recognition texts are all the same and / or the positions of characters representing the same semantics in the N aligned recognition texts are all the same.

[0152] It is understood that N aligned texts can satisfy one or more of the first alignment condition or the second alignment condition. The first alignment condition means that the character lengths of the N aligned texts are all the same, and the second alignment condition means that the positions of characters representing the same semantic meaning in the N aligned texts are all the same. That is, the character lengths of the N aligned texts can all be the same, and the positions of characters representing the same semantic meaning in the N aligned texts can all be the same. Optionally, the character lengths of the N aligned texts can all be the same, or the positions of characters representing the same semantic meaning in the N aligned texts can all be the same.

[0153] Placeholders can be represented as "φ". For example, if candidate text 1, candidate text 2, and candidate text 3 are aligned, then after alignment, the aligned text 1 corresponding to candidate text 1 can be represented as "ABCDφ", the aligned text 2 corresponding to candidate text 2 can be represented as "AφCDE", and the aligned text 3 corresponding to candidate text 3 can be represented as "ABCφφ". The position of a character in the aligned text can refer to its index value, which is determined based on the order of characters (such as word order) in the aligned text. For example, if aligned text 1 can be represented as "ABCDφ", and the character "A" is in the starting position, then the index value of the character "A" in aligned text 1 is 1 (i.e., its position is 1), so the position of the character "A" in aligned text 1 is 1. The character "φ" is positioned at the end of the alignment recognition text 1. Therefore, the index value of the character "φ" in the alignment recognition text 1 is 5 (i.e., its position is 5). So, the position of the character "φ" in the alignment recognition text 1 is 5.

[0154] In this context, the positions of characters representing the same semantic meaning in aligned text 1, aligned text 2, and aligned text 3 are all identical. For example, if the length of the aligned text is 5, then the semantic meaning of character A at position 1 in aligned text 1, aligned text 2, and aligned text 3 is identical. Similarly, the semantic meaning of character B at position 2 in aligned text 1 and aligned text 3 is identical. Likewise, the semantic meaning of character C at position 3 in aligned text 1, aligned text 2, and aligned text 3 is identical.

[0155] Step S104: Obtain the text vectors corresponding to the N aligned recognition texts respectively, and perform vector fusion processing on the N text vectors to obtain the fused vector;

[0156] Specifically, the computer device can input N aligned texts into the fusion processing layer of the speech recognition model. In the fusion processing layer, features are extracted from each of the N aligned texts, resulting in text vectors corresponding to each text. Since the character length of each of the N aligned texts is a vector, the vector dimensions of the N text vectors are also the same. The computer device can then fuse the vector elements corresponding to the same character positions to obtain a fused vector. It can be understood that through the alignment process described above, the fusion processing can merge vector elements representing the same semantics. The resulting fused embedding vector can capture more dimensional semantic features in the aligned text, and the semantics it contains are more accurate.

[0157] For example, at character position 1, the vector elements corresponding to character A in aligned recognition text 1, character A in aligned recognition text 2, and character A in aligned recognition text 3 can be averaged to obtain the vector elements. In character position 2, the vector elements corresponding to character B in aligned recognition text 1, character φ in aligned recognition text 2, and character B in aligned recognition text 3 can be averaged to obtain the vector elements. In character position 3, the vector elements corresponding to character C in aligned recognition text 1, character C in aligned recognition text 2, and character C in aligned recognition text 3 can be averaged to obtain the vector elements. In character position 4, the vector elements corresponding to character D in aligned recognition text 1, character D in aligned recognition text 2, and character φ in aligned recognition text 3 can be averaged to obtain the vector elements. In character position 5, the vector elements corresponding to the characters φ in aligned recognition text 1, E in aligned recognition text 2, and φ in aligned recognition text 3 can be averaged to obtain the vector elements. vector elements vector elements vector elements vector elements and vector elements It is determined to be the fusion vector.

[0158] Step S105: Generate the recognition result text corresponding to the speech data based on the fusion vector and the encoding vector.

[0159] Specifically, the computer device can input the fused vector and the encoded vector into the speech decoding layer of the speech recognition model. The vector elements in the fused vector can also be called unit average vectors. Taking a fused vector consisting of S unit average vectors as an example, in the speech decoding layer, the computer device can perform cross-attention processing on the fused-encoded vector and the fused vector to obtain an attention result vector. The attention result vector can include the unit result vectors corresponding to the S unit average vectors, and the unit result vector can include the attention score between each character to be recognized in the character database and the unit average vector.

[0160] Taking S unit result vectors, including unit result vector 1, as an example, unit result vector 1 can include the attention score between the unit average vector and the character to be recognized A, the attention score between the unit average vector and the character to be recognized B, ..., the attention score between the unit average vector and the character to be recognized N. The computer device can determine the character to be recognized with the highest attention score in unit result vector 1 as a recognized character. In this way, the computer device can generate S recognized characters based on the S unit result vectors in the attention result vector, and determine the S recognized characters arranged in the order of the S unit result vectors as the recognition result text corresponding to the speech data.

[0161] This application embodiment generates N candidate recognition texts corresponding to the speech data. Alignment processing is then performed on these N candidate texts to determine characters representing the same semantic meaning at the same position, resulting in N aligned recognition texts with the same character length. Further fusion processing is performed on these N aligned recognition texts to fuse vector elements at the same character position, thereby capturing more dimensions of semantic features. By generating the recognition result text corresponding to the speech data through fused vectors and encoded vectors, the semantic features of the N aligned recognition texts are considered as much as possible while incorporating the pronunciation information contained in the encoded vectors. This reduces the impact of inaccurate user pronunciation, thereby improving the accuracy and stability of speech data recognition.

[0162] Please see Figure 4 , Figure 4 This is a flowchart illustrating a voice data processing method provided in an embodiment of this application. Figure 2 This voice data processing method can be executed by a computer device, which can be, for example, Figure 1 The business server 100 shown or any terminal device in the terminal device cluster, such as terminal device 10a. The following description will use the example of this voice data processing method being executed by a computer device. This voice data processing method may include at least the following steps S201-S208:

[0163] Step S201: Acquire voice data;

[0164] Step S202: Input the speech data into the speech recognition model; the speech recognition model includes a speech coding layer; in the speech coding layer, the speech data is segmented into frames to obtain M speech frames corresponding to the speech data; M is a positive integer; feature extraction is performed on the M speech frames to obtain M speech embedding vectors; obtain the position embedding vectors corresponding to the M speech embedding vectors respectively, and generate M encoded input vectors based on the M position embedding vectors and the M speech embedding vectors; the position embedding vectors are used to indicate the position of the speech frame data corresponding to the speech embedding vector in the speech data;

[0165] Specifically, computer devices can be equipped with speech recognition models. These devices can acquire speech data, input it into the speech recognition model, and then... (Please refer to...) Figure 5 , Figure 5 This is a schematic diagram of the model structure of a speech recognition model provided in an embodiment of this application, as shown below. Figure 5 As shown, the speech recognition model includes a speech coding layer, an alignment processing layer, a fusion processing layer, and a speech decoding layer.

[0166] The speech coding layer can be an encoder in an ASR structure based on CTC-Attention. This encoder can be a Transformer model or a Conformer model (a model that alternates between the self-attention layer of a Transformer and the convolutional layer of a CNN). The attention layer in the speech coding layer can learn dependencies from the entire input sequence, thereby capturing complex semantic structures and focusing on local features in the input sequence. This enhances the model's performance when processing continuous signals (such as speech features) and improves the accuracy of speech recognition.

[0167] like Figure 4As shown, in the speech coding layer, the computer device can select a certain frame length, such as 15ms (milliseconds), and perform frame segmentation processing on the speech data based on this frame length. Frame segmentation processing can divide the continuous speech signal in the speech data into M speech frame data containing short signals. For example, the M speech frame data may include speech frame data X1, speech frame data X2, ..., speech frame data Xm.

[0168] A computer device can extract features from M speech frames separately. For example, it can embed the speech features (such as phonemes, syllables, intonation, etc.) from the M speech frames into vectors of the same size, resulting in M ​​speech embedding vectors. These M speech embedding vectors can include speech embedding vector L1 corresponding to speech frame X1, speech embedding vector L2 corresponding to speech frame X2, ..., speech embedding vector Lm corresponding to speech frame Xm. The speech embedding vectors can contain features such as phonemes, syllables, and intonation.

[0169] A computer device can acquire positional embedding vectors corresponding to M speech embedding vectors. These positional embedding vectors indicate the position of the corresponding speech frame data within the speech data, enabling the capture of the positional order of the speech frame data and the understanding of semantic structure. The M positional embedding vectors may include positional embedding vector P1 corresponding to speech frame data X1, positional embedding vector P2 corresponding to speech frame data X2, ..., positional embedding vector Pm corresponding to speech frame data Xm.

[0170] Computer devices can perform vector concatenation on speech embedding vectors and their associated location embedding vectors. For example, speech embedding vector L1 and location embedding vector P1 can be concatenated to obtain encoded input vector E1, speech embedding vector L2 and location embedding vector P2 can be concatenated to obtain encoded input vector E2, and speech embedding vector Lm and location embedding vector Pm can be concatenated to obtain encoded input vector Em.

[0171] Step S203: Generate an encoding vector based on the M encoding input vectors;

[0172] Specifically, when a computer device acquires M encoded input vectors, it can generate an encoded vector based on these M input vectors. The speech coding layer may include an attention processing layer. The computer device can generate the encoded vector based on the M input vectors within this attention processing layer. The process may involve: acquiring the character encoding vector corresponding to each character to be recognized from the character database; concatenating the character encoding vectors to obtain a character encoding sequence; and then processing the encoded input vector E... i Cross-attention processing is performed with the character encoding sequence to obtain the encoded input vector E. i The corresponding unit attention vector Ai Unit attention vector A i Including the encoded input vector E i The attention scores are calculated for each character to be recognized in the character database. When the unit attention vectors corresponding to the M encoded input vectors are obtained, the M unit attention vectors are concatenated to obtain the encoded vector.

[0173] Specifically, the M encoded input vectors include encoded input vector E i For ease of understanding, let's take the encoded input vector E as an example. i Taking attention processing with the characters to be recognized in the character database as an example, the computer device can obtain the character encoding vector corresponding to each character to be recognized in the character database. Here, the character to be recognized can also be called a token, and the character encoding vector can be, that is, the embedding vector corresponding to the character to be recognized.

[0174] Computer devices can concatenate character encoding vectors to obtain a character encoding sequence. The encoding input vector E... i With the query parameter matrix W in the attention processing layer Q Perform a dot product operation to obtain the query vector (Q, Query), and then combine the character encoding sequence with the key parameter matrix W. K Perform a dot product operation to obtain the key vector (K, Key), and then combine the character encoding sequence with the value parameter matrix W. V Performing a dot product yields the value vector (V, Value).

[0175] Among them, the query parameter matrix W Q Key parameter matrix W K Sum parameter matrix W V Both are matrices composed of learnable parameters from the attention processing layer, used to transform the encoded input vector E through linear transformation. i Character encoding sequences are mapped to different representation spaces.

[0176] Computer devices can transpose the value vector K to obtain the first parameter K. T The query vector Q is compared with the first parameter K. T The result of the dot product operation is determined as the second parameter QK. T Based on the second parameter QK T The vector dimension value d corresponding to the key vector k Generate third parameter The third parameter is normalized to obtain the normalized parameter. normalized parameters The result of the dot product operation with the value vector V is determined as the encoded input vector E. i The corresponding unit attention vector Ai The process can be shown in formula (1):

[0177]

[0178] Wherein, the unit attention vector A i Including the encoded input vector E i The attention score is calculated between each character to be recognized in the character database. The attention score can be a unit attention vector A. i The values ​​of the vector elements are used to represent the encoded input vector E. i The association between each character to be recognized in the character database.

[0179] It is understandable that by performing cross-attention calculations on the encoded input vector and the character encoding sequence, key features in the encoded input vector, such as phonemes, syllables, and intonation, can be captured. This allows for the calculation of the dependency relationship between the encoded input vector and each character to be recognized, thereby understanding the semantic and syntactic structure of the speech data and improving the accuracy of speech recognition.

[0180] Optionally, the input vector E is encoded. i Alternatively, it can be obtained through multi-head attention computation, which can include multiple unit processing spaces, each of which can be an independent attention processing layer, allowing parallel processing of the encoded input vector E. i Attention is calculated with the character encoding sequence, and the attention calculation results obtained from each unit processing space are weighted and averaged to obtain the encoded input vector E. i The corresponding unit attention vector A i Because the encoded input vector E can be captured from multiple different angles. i Because of its different characteristics from character encoded sequences, it can better capture the encoded input vector E. i The dependency on character encoding sequences improves the accuracy of speech recognition.

[0181] The computer device can perform attention calculations on the M encoded input vectors in the same way described above, obtaining M unit attention vectors. When the computer device obtains the unit attention vectors corresponding to the M encoded input vectors, it concatenates the M unit attention vectors to obtain the encoded vector. Since the encoded vector can be used to represent speech features, and the encoded vector is generated from the speech embedding vector and the position embedding vector, the candidate recognition text output based on the encoded vector can correspond to the speech features.

[0182] Step S204: Based on the attention score corresponding to each character to be recognized in the encoding vector, generate P character combination lists; P is a positive integer, and each character combination list includes M character labels, each character label in the character combination list is determined by a different unit attention vector; obtain the total attention score corresponding to each of the P character combination lists; the total attention score of the character combination list is the sum of the attention scores corresponding to the M character labels; obtain the candidate text value N, and based on the candidate text value N and the total attention score in the P character combination lists, obtain N character combination lists; perform temporal alignment processing on the N character combination lists to obtain N candidate recognition texts.

[0183] Specifically, the M encoded input vectors may include encoded input vector 1, and the encoded vectors may include M unit attention vectors, which may include unit attention vector A1, unit attention vector A2, ..., unit attention vector Am. Each unit attention vector may include the attention score between the encoded input vector and each character to be recognized in the character database. For example, unit attention vector A1 may represent an attention score of 87 between encoded input vector 1 and character A, 42 between encoded input vector 1 and character H, 34 between encoded input vector 1 and character K, etc. The attention score can also be a normalized numerical value; this embodiment of the application does not impose any limitations on this.

[0184] It is understood that the encoding vector may include M unit attention vectors, each of which may contain an attention score between itself and the character to be recognized. The character labels decoded by the unit attention vector may be characters with the same semantics as mentioned below in this application. Here, characters with the same semantics may refer to characters that are completely identical or words that are synonyms. Alternatively, the similarity between the semantic features of the characters may be determined by whether the similarity is greater than a similarity threshold.

[0185] A computer device can generate a list of P character combinations based on the attention score corresponding to each character to be recognized in the encoding vector. The process can be as follows: obtain the attention threshold, determine the attention scores that are greater than or equal to the attention threshold among the M unit attention vectors in the encoding vector as candidate attention scores, determine the characters to be recognized associated with the candidate attention scores as character labels, and select, arrange and combine the character labels to obtain a list of P character combinations.

[0186] Specifically, the computer device can obtain an attention threshold, which can be a pre-set value. The attention scores in the M unit attention vectors that are greater than or equal to the attention threshold are determined as candidate attention scores, and the characters to be recognized associated with the candidate attention scores are determined as character labels. Among them, the character label can refer to a single character or a word. For example, in Chinese, it can refer to a single character "我" or a word "我们". The computer device can perform permutations and combinations on the character labels to obtain, for example, one character combination list. Select one character label from the character labels corresponding to one unit attention vector, and so on. One character label corresponding to each of the M unit attention vectors can be obtained. The M character labels arranged in the order of the M unit attention vectors are determined as one character combination list. Similarly, by selecting different character labels from the character labels corresponding to the same unit attention vector, multiple different character combination lists can be generated through permutation and combination. Among them, each character combination list includes M character labels, and each character label in the character combination list is determined by a different unit attention vector. For example, the character labels in character combination list 1 in P character combination lists can be represented as [A, B,..., D], where the character label A can be determined by the unit attention vector A1, the character label B can be determined by the unit attention vector A2, and the character label D can be determined by the unit attention vector Am.

[0187] The computer device can obtain the total attention scores corresponding to the P character combination lists respectively. The total attention score of a character combination list is the sum of the attention scores corresponding to the M character labels. Obtain the candidate text value N, which can be a pre-set value. For example, it can be set based on the context window of the speech decoding layer or the attention threshold. The computer device can obtain N character combination lists with higher total attention scores from the P character combination lists based on the candidate text value N and the total attention scores. For example, the P character combination lists can include character combination list 1, character combination list 2, character combination list 3, and character combination list 4. The total attention score of character combination list 1 can be 431, the total attention score of character combination list 2 can be 422, the total attention score of character combination list 3 can be 417, and the total attention score of character combination list 4 can be 411. When N is 2, the computer device can obtain the character combination lists with higher total attention scores as character combination list 1 and character combination list 2 respectively.

[0188] It is understandable that when the speech coding layer is a CTC model encoder, the CTC model reduces computational complexity by allowing alignment of sequences of different lengths, thus avoiding manual alignment between input and output. Therefore, multiple inputs can correspond to one output. Simultaneously, the CTC model introduces a special blank character in the output label to represent phonemes that have returned to zero in the speech feature code. Taking the blank character as an example, character combination list 1 can be represented as [-, A, B, -, C, C, C, -, D], and character combination list 2 can be represented as [-, A, -, C, C, -, D, E, -]. Therefore, the computer needs to perform temporal alignment processing on the N character combination lists, removing duplicate character labels and blank characters to obtain N candidate recognition texts. The process can be: ... ... i With the null character C i+1 At least two identical character tags between them are merged into one character tag; when the character combination list B i When at least two identical character tags located between adjacent empty characters have been merged, delete the character combination list B. i The empty character in the list gives us a list of character combinations B. i The corresponding candidate text for recognition.

[0189] Specifically, the list of N character combinations includes the character combination list B. i Character combination list B i Includes the null character C i and the null character C i+1 null character C i and the null character C i+1 The adjacent null characters, null character C i With the null character C i+1 The list of character combinations B contains at least two identical character labels; therefore, the list of character combinations in the list of N character combinations is... i It can be represented as [−, A, C] i C, C, C i+1 [,D,E-], the computer device will be located in the null character C i With the null character C i+1 At least two identical character tags between two strings are merged into one character tag, for example, [C, C, C] are merged into character tag C, when the character combination list B... i When at least two identical character tags located between adjacent empty characters have been merged, delete the character combination list B. i The empty character in the list gives us a list of character combinations B. i The corresponding candidate text for recognition. For example, character combination list B. iThe corresponding candidate text can be represented as [A, C, D, E]. A computer device can perform the same temporal alignment process as described above on a list of N character combinations to obtain N candidate texts. Since the encoding vector can be used to represent speech features, the candidate text output based on the encoding vector can correspond to the speech features.

[0190] It is understandable that N candidate texts for recognition can be generated from different attention perspectives. Through corresponding attention processing, different key features in the encoded input vector, such as phonemes, syllables, and intonation, can be captured. The dependency relationship between the encoded input vector and each character to be recognized can be calculated, thereby understanding the semantic and syntactic structure of the speech data and generating N candidate texts for recognition, including features such as different semantic characters, syntactic structures, and semantic relationships. By capturing the semantics contained in the N candidate texts, the accuracy and stability of speech data recognition can be improved.

[0191] Step S205: Input N candidate recognition texts into the speech recognition model; the speech recognition model includes an alignment processing layer; in the alignment processing layer, determine the reference recognition text from the N candidate recognition texts; obtain the editing information corresponding to the N candidate recognition texts for aligning with the reference recognition text; the editing information includes the editing distance and editing processing type corresponding to each candidate recognition text converted into the reference recognition text, and the sum of the editing distances satisfies the minimum editing distance condition;

[0192] Specifically, since the characters in the candidate recognition text are determined by character tags, the characters in the candidate recognition text can also refer to single characters or words. Optionally, the characters in the candidate recognition text can also be obtained by text segmentation based on a tokenizer. The text segmentation method can be word-based, character-based, or subword-based, and this embodiment of the application does not impose any limitations.

[0193] A computer device can input N candidate texts into the alignment layer of a speech recognition model. In the alignment layer, a reference text is determined from the N candidate texts. The reference text can be any one of the N candidate texts, or it can be the candidate text with the highest total attention score among the N candidate texts.

[0194] The computer device can acquire N candidate recognition texts to align with the editing information corresponding to the reference recognition text. The editing information can include the editing distance and editing processing type corresponding to the conversion of each candidate recognition text into the reference recognition text. The sum of the editing distances satisfies the minimum editing distance condition. The editing distance is defined as the minimum number of single-character editing processes required to convert one string text into another. These editing processes can include insertion (inserting a character), deletion (deleting a character), and substitution (replacing one character with another).

[0195] like Figure 4 As shown, taking N candidate texts, including candidate text 1, candidate text 2, and candidate text 3, as an example, candidate text 1 can be represented as [A, B, C, D], candidate text 2 as [A, C, D, E], and candidate text 3 as [A, B, C]. The reference text can be candidate text 1, so the edit distance between candidate text 1 and the reference text is 0. The edit distance between candidate text 2 and the reference text is 2, which includes inserting the character B and deleting the character E in candidate text 2. The edit distance between candidate text 3 and the reference text is 1, which includes inserting the character D in candidate text 3.

[0196] It is understandable that the edit distance between strings a and b can be represented as edit[i][j]. Here, i is the character length of string a, which can also be represented as string a[1:i]. j is the character length of string b, which can also be represented as string b[1:j]. The computer can iterate through the characters a[i] in string a and compare them with the characters b[j] in string b. If characters a[i] and b[j] are the same, then the edit distance edit[i][j] = edit[i-1][j-1]. If characters a[i] and b[j] are different, then the following applies:

[0197] 1. If string a[1:i] is transformed into string b[1:j-1] after multiple editing processes, then inserting character b[j] at the end will transform string a[1:i] into string b[1:j], and its edit distance edit[i][j] = edit[i][j-1] + 1;

[0198] 2. If string a[1:i-1] is transformed into string b[1:j] after multiple editing processes, then deleting the character a[i] from string a[1:i] will transform string a[1:i] into string b[1:j], and its edit distance edit[i][j] = edit[i-1][j] + 1;

[0199] 3. After multiple editing processes, the string a[1:i-1] is transformed into the string b[1:j-1]. Then, by replacing the character a[i] in the string a[1:i] with b[j], the string a[1:i] can be transformed into the string b[1:j]. The edit distance edit[i][j] = edit[i-1][j-1] + 1.

[0200] In this context, `edit[0][j]` represents the number of edit operations required to convert an empty string to `b[1:j]`, and `edit[i][0]` represents the number of edit operations required to convert `a[1:i]` to an empty string, which is `i`. If both `i` and `j` are 0, then `edit[i][j]` = 0. If `i` is greater than 0 and `j` is equal to 0, then `edit[i][j]` = `i`. If `j` is greater than 0 and `i` is equal to 0, then `edit[i][j]` = `j`.

[0201] When both i and j are greater than 0, the process of calculating the edit distance can be shown in formula (2):

[0202]

[0203] When a[i] and b[j] are different, flag is 1. When a[i] and b[j] are the same, flag is 0; flag represents the required increase in edit distance.

[0204] It is understood that, in this embodiment of the application, by calculating the edit distances corresponding to N candidate recognition texts, the processing position corresponding to the alignment reference recognition text can be determined. Based on the processing position, characters representing the same semantics can be identified at the same character position (the character position is the arrangement position of a single character in a text), resulting in N aligned recognition texts with the same character length. Characters with the same semantics can refer to characters that are completely identical or words that are synonyms. This facilitates subsequent fusion processing by fusing vector elements at the same character positions, thereby capturing more dimensional semantic features in the N aligned recognition texts and improving the accuracy and stability of the recognized speech data.

[0205] Step S206: Determine the position to be processed in N candidate recognition texts based on the editing information; insert placeholders in the position to be processed in the N candidate recognition texts to obtain N aligned recognition texts.

[0206] Specifically, the computer device can determine the position to be processed among N candidate recognition texts based on the editing information. The position to be processed can be represented by "&". The candidate recognition text F among the N candidate recognition texts... i The candidate text can be 2, and the editing information can include the candidate text F. iCorresponding editing information G i Edit information G i This could be the edit information 2 corresponding to candidate recognition text 2. The position of a character in the text can refer to its index value, which is determined based on the order of characters in the text. The minimum index value (starting value) is 1, and the maximum index value is the number of characters in the text. The starting position in the text is the position of the first character, and the ending position is the position of the last character. For example, candidate recognition text 1 can be represented as "ABCD". The character "A" is at the starting position in candidate recognition text 1, so its index value is 1 (i.e., position 1), and its position is 1. The character "D" is at the ending position in candidate recognition text 1, so its index value is 4 (i.e., position 4), and its position is 4.

[0207] In the above examples, such as Figure 4 As shown, aligning candidate text 2 with the corresponding edit distance of reference text involves inserting character B and deleting character E into candidate text 2. The computer device can detect the type and location of the edit processing indicated by the edit information. If the edit information G... i If the indicated editing process type is insert processing, then the computer device can identify the candidate text F. i The editing position corresponding to the insertion process is determined as the position to be processed. For example, when candidate recognition text 2 is aligned with reference recognition text, if character B is inserted between character A and character B, the computer device can determine the editing position of this insertion process in candidate recognition text 2 as the position to be processed.

[0208] If you edit information G i If the indicated editing process type is deletion, then the computer device can identify the candidate text F. i The corresponding business location in the reference recognition text is determined as the location to be processed. Specifically, the business location and the candidate recognition text F... i The deletion process is associated with the corresponding editing position. For example, if the character "E" is deleted when candidate recognition text 2 is aligned with reference recognition text, the computer device can determine the editing position of this deletion process in the reference recognition text (i.e., candidate recognition text 1) as the position to be processed.

[0209] If you edit information G i If the indicated editing process type is replacement processing, the computer device does not need to determine the location to be processed.

[0210] It is understandable that when determining the position to be processed in candidate recognition text 3, since the edit distance between candidate recognition text 3 and the corresponding reference recognition text is 1, including the insertion of character D after character C in candidate recognition text 3, the computer device can determine the edit position of this insertion in candidate recognition text 3 as the position to be processed. Furthermore, to align the overall length, the computer device can also determine the position after the aforementioned insertion of character D as the position to be processed.

[0211] A computer device can determine the position to be processed in N candidate recognition texts as the position information to be processed. Based on the position information to be processed, placeholders are inserted at the positions to be processed in the N candidate recognition texts to obtain N aligned recognition texts. The N aligned recognition texts can include aligned recognition text 1 corresponding to candidate recognition text 1, aligned recognition text 2 corresponding to candidate recognition text 2, and aligned recognition text 3 corresponding to candidate recognition text 3. The placeholder can be represented as "φ", aligned recognition text 1 can be represented as [A, B, C, D, φ], aligned recognition text 2 can be represented as [A, φ, C, D, E], and aligned recognition text 3 can be represented as [A, B, C, φ, φ].

[0212] Optionally, embodiments of this application can also determine the processing position by calculating the character similarity between character tags in candidate recognition text. For example, calculating the character similarity between candidate recognition text 1 (represented as [A, B, C, D]) and candidate recognition text 2 (represented as [A, C, D, E]) allows the computer device to determine the processing position in candidate recognition text 1 and candidate recognition text 2 by detecting the character similarity between the character tags in candidate recognition text 1 and candidate recognition text 2. A placeholder is then inserted at the processing position to determine character tags representing the same semantics at the same character position. For example, the computer device can detect the presence of character tags A, C, and D representing the same semantics in candidate recognition text 1 and candidate recognition text 2. This allows the processing position to be determined in candidate recognition text 1 and candidate recognition text 2. A placeholder is then inserted at the processing position to obtain the aligned recognition text 1 corresponding to candidate recognition text 1 and the aligned recognition text 2 corresponding to candidate recognition text 2. The aligned recognition text 1 can be represented as [A, B, C, D, φ], and the aligned recognition text 2 can be represented as [A, φ, C, D, E].

[0213] Step S207: Input N aligned recognition texts into the speech recognition model; the speech recognition model includes a fusion processing layer; in the fusion processing layer, feature extraction is performed on the N aligned recognition texts respectively to obtain text vectors corresponding to the N aligned recognition texts respectively; the character length of the aligned recognition texts is S, where S is a positive integer; the text vector includes S unit character vectors, and each unit character vector is associated with a character in the aligned recognition text; the unit character vectors located at the same position in the N text vectors are averaged to obtain S unit average vectors, and the S unit average vectors are concatenated to obtain a fusion vector.

[0214] Specifically, the computer device can input N aligned recognition texts into the fusion processing layer of the speech recognition model. In the fusion processing layer, features are extracted from each of the N aligned recognition texts to obtain text vectors corresponding to each of the N aligned recognition texts. The text vectors can be the embedding vectors corresponding to the aligned recognition texts. The computer device can use a tokenizer to split and encode the aligned recognition texts into tokens. For example, if the aligned recognition text 1 is [A, B, C, D, φ], the tokens corresponding to the aligned recognition text 1 can be represented as [AB], [C], [D], and [φ]. The text splitting method can be word-based, character-based, or subword-based, and this embodiment of the application does not impose any limitations.

[0215] Since the character lengths of the N aligned text vectors are all vectors, the vector dimensions of the N text vectors are also the same. Computer devices can fuse the vector elements corresponding to the same character positions to obtain a fused vector.

[0216] For example, at character position 1, the vector elements corresponding to character A in aligned recognition text 1, character A in aligned recognition text 2, and character A in aligned recognition text 3 can be averaged to obtain the vector elements. In character position 2, the vector elements corresponding to character B in aligned recognition text 1, character φ in aligned recognition text 2, and character B in aligned recognition text 3 can be averaged to obtain the vector elements. In character position 3, the vector elements corresponding to character C in aligned recognition text 1, character C in aligned recognition text 2, and character C in aligned recognition text 3 can be averaged to obtain the vector elements. In character position 4, the vector elements corresponding to character D in aligned recognition text 1, character D in aligned recognition text 2, and character φ in aligned recognition text 3 can be averaged to obtain the vector elements. In character position 5, the vector elements corresponding to the characters φ in aligned recognition text 1, E in aligned recognition text 2, and φ in aligned recognition text 3 can be averaged to obtain the vector elements. vector elements vector elements vector elements vector elements and vector elements This is determined to be a fusion vector. The vector elements in the fusion vector can also be called unit average vectors.

[0217] Optionally, for vector elements at the same position, this embodiment can also generate an attention weight corresponding to each aligned text based on the total attention score associated with the aligned text. The vector elements of the text vector are then weighted and averaged using the attention weights corresponding to each aligned text to obtain a fused vector. This embodiment does not impose limitations on this. For example, the attention weight of aligned text 1 can be represented as coef1, the attention weight of aligned text 2 as coef2, and the attention weight of aligned text 3 as coef3. The computer device can perform weighted averaging on the vector elements at character position H to obtain the weighted elements. Weighted elements The calculation process is shown in formula (3):

[0218]

[0219] Where H1 represents the vector element of aligned text 1 at character position H, H2 represents the vector element of aligned text 2 at character position H, and H3 represents the vector element of aligned text 3 at character position H. 3 represents the number of aligned texts.

[0220] It is understood that, through fusion processing, the vector elements of the same character position in N aligned recognition texts can be fused to capture different semantic characters, syntactic structures and semantic associations in the N aligned recognition texts, and obtain a fused vector. The fused vector can learn the key features with higher dependencies in the N candidate recognition texts.

[0221] Step S208: Input the fusion vector and the encoding vector into the speech recognition model; the speech recognition model includes a speech decoding layer; in the speech decoding layer, cross-attention processing is performed on the fusion-encoding vector and the fusion vector to obtain an attention result vector; the attention result vector includes unit result vectors corresponding to S unit average vectors, and the unit result vector includes the attention score between each character to be recognized in the character database and the unit average vector; based on the S unit result vectors in the attention result vector, S recognition characters are generated, and the S recognition characters are determined as the recognition result text corresponding to the speech data.

[0222] Specifically, the computer device can input the fusion vector and the encoding vector into the speech decoding layer of the speech recognition model. In the speech decoding layer, the computer device can perform cross-attention processing on the fusion vector and the encoding vector to obtain an attention result vector. Based on the S unit result vectors in the attention result vector, S recognition characters are generated, and the S recognition characters are determined as the recognition result text corresponding to the speech data.

[0223] The details of the cross-attention processing can be found in the specific description of step S202 above, and the details of generating the recognition result text can be found in the above... Figure 3 The specific description of step S105 in the corresponding embodiment will not be repeated here.

[0224] It is understandable that compared to the computational complexity of O(S) for decoding the encoded vector (vector dimension S) using an autoregressive method, the decoding method of generating S recognized characters based on the S unit result vectors in the attention result vector in this application is a non-autoregressive method. Since the S unit result vectors include the preliminary results of the fusion processing of related candidate recognized texts, for example, when decoding the S-th unit result vector, it is not necessary to refer to the S-1 vectors before decoding, and S recognized characters can be decoded at once. The decoding process of the S unit result vectors can be performed in parallel and independently, and its computational complexity is O(1). This can effectively reduce the computational complexity of the decoding process, reduce computational costs, and improve computational efficiency.

[0225] This application embodiment generates M encoded input vectors corresponding to speech data and captures key features in the encoded input vectors, such as phonemes, syllables, and intonation. The dependency relationship between the encoded input vectors and each character to be recognized can be calculated, thereby understanding the semantic and syntactic structure of the speech data and obtaining the encoded vectors. N candidate recognition texts are generated from the encoded vectors. These N candidate recognition texts can be generated from different attention angles and include different semantic characters, syntactic structures, and semantic relationships. Alignment processing is performed on the N candidate recognition texts to determine characters representing the same semantics at the same character position, resulting in N aligned recognition texts with the same character length. Further fusion processing is performed on the N aligned recognition texts to fuse the vector elements at the same character position, thereby capturing different semantic characters, syntactic structures, and semantic relationships in the N aligned recognition texts, as well as capturing more dimensions of semantic features. By generating the recognition result text corresponding to the speech data through the fusion vector and the encoding vector, the semantic features in the N aligned recognition texts can be considered as much as possible while combining the pronunciation information contained in the encoding vector, so as to reduce the impact of the user's inaccurate pronunciation, thereby improving the accuracy and stability of speech data recognition.

[0226] On the other hand, the speech recognition model proposed in this application can be an integrated model (i.e., including both a recognition model and an error correction model). The speech coding layer can be the recognition model, and the alignment processing layer, fusion processing layer, and speech decoding layer can be the error correction model. When the recognition model and the error correction model are deployed separately, the security of the original speech data can be improved, protecting privacy. When the recognition model and the error correction model are integrated into one model, i.e., a speech recognition model, the integrated speech recognition model can reduce data transmission and intermediate processing overhead, and can share certain feature vectors. Therefore, it can improve performance while increasing computational efficiency and reducing deployment costs. Furthermore, since this application uses a non-autoregressive decoding method, the decoding computational complexity can be greatly reduced, improving the efficiency of speech recognition.

[0227] Please see Figure 6 , Figure 6 This is a flowchart illustrating a voice data processing method provided in an embodiment of this application. Figure 3 This voice data processing method can be executed by a computer device, which can be, for example, Figure 1 The business server 100 shown or any terminal device in the terminal device cluster, such as terminal device 10a. The following description will use the example of this voice data processing method being executed by a computer device. This voice data processing method may include at least the following steps S301-S307:

[0228] Step S301: Obtain sample speech data and the corresponding answer text, and input the sample speech data and answer text into the initial recognition model;

[0229] Specifically, the computer device can acquire sample speech data and the corresponding answer text, and input the sample speech data and answer text into the initial recognition model. The initial recognition model may include an initial speech coding layer, an initial alignment processing layer, an initial fusion processing layer, and an initial speech decoding layer. Among them, the initial speech coding layer can be an encoder in an ASR structure based on CTC-Attention, and this encoder can be a Transformer model or a Conformer model.

[0230] Step S302: In the initial recognition model, the sample speech data is encoded to obtain the sample encoding vector;

[0231] Step S303: Generate N candidate text samples for recognition based on the sample encoding vector; N is a positive integer;

[0232] Specifically, in the initial recognition model, the computer device can encode the sample speech data to obtain sample encoding vectors. The details of generating the sample encoding vectors can be found above. Figure 4 The specific description of generating the encoding vector in steps S201 to S203 of the corresponding embodiment will not be repeated here.

[0233] Computer equipment can generate N candidate text samples for recognition based on the sample encoding vectors. The details of generating N candidate text samples can be found above. Figure 4 The specific description of generating N candidate recognition texts in step S204 of the corresponding embodiment will not be repeated here.

[0234] Step S304: Based on the answer text, align the N candidate sample recognition texts to obtain N aligned sample recognition texts; the character length of the N aligned sample recognition texts is the same as the character length of the answer text, and the positions of characters representing the same semantics in the N aligned sample recognition texts and the answer text are the same.

[0235] Specifically, the computer device can determine the answer text as the reference recognition text and obtain N sample candidate recognition texts to align with the editing information corresponding to the reference recognition text. For details regarding the editing information, please refer to the above. Figure 4 The specific description of step S205 in the corresponding embodiment will not be repeated here.

[0236] A computer device can determine the position to be processed in N candidate sample recognition texts based on editing information, and insert placeholders at the position to be processed in the N candidate recognition texts to obtain N sample-aligned recognition texts. The character length of the N sample-aligned recognition texts is the same as the character length of the answer text, and the positions of characters representing the same semantic meaning in the N sample-aligned recognition texts and the answer text are the same. For details on the N sample-aligned recognition texts, please refer to the above. Figure 4 The specific description of the N aligned recognition texts in step S206 of the corresponding embodiment will not be repeated here.

[0237] Step S305: Obtain the sample text vectors corresponding to the N sample aligned recognition texts respectively, and obtain N sample text vectors. Perform vector fusion processing on the N sample text vectors to obtain the sample fusion vector.

[0238] Step S306: Generate the prediction result text corresponding to the sample speech data based on the sample fusion vector and the sample encoding vector;

[0239] Specifically, the computer device can perform vector fusion processing on the sample text vectors corresponding to the N sample aligned recognition texts to obtain a sample fusion vector. This fusion processing can involve averaging or weighted averaging the vector elements. The computer device can then perform cross-attention processing on the sample fusion vector and the sample encoding vector to obtain an attention result vector. Based on this attention result vector, a predicted result text corresponding to the sample speech data is generated. The content of the generated predicted result text can be found above. Figure 3 The specific description of generating the recognition result text in step S105 of the corresponding embodiment will not be repeated here.

[0240] Step S307: Generate model loss value based on the predicted result text and the answer text. Adjust the model parameters of the initial recognition model using the model loss value until the initial recognition model meets the model convergence condition to obtain the speech recognition model. The speech recognition model is used to generate the recognition result text corresponding to the speech data.

[0241] Specifically, the computer device can generate the model loss value based on the prediction result text and the answer text. The model loss value L can be calculated as shown in formula (4):

[0242] L=-ln∏ (x,z)∈S p(z|x) Formula (4)

[0243] Where p(z|x) represents the probability distribution of the output answer text (i.e., the true result path) given the input sample speech data. S is the predicted result text generated by the initial recognition model (i.e., all possible output result paths), z is one of the paths output by the initial recognition model, and x is the input sample speech data and the answer text. (x,z)∈S p(z|x) can represent the product of the probabilities of outputting the answer text after given sample speech data.

[0244] Computer equipment can adjust the model parameters of the initial recognition model using the model loss value until the initial recognition model meets the model convergence condition, thus obtaining the speech recognition model. The model convergence condition can mean that the loss function of the initial recognition model remains stable, and the model loss value no longer fluctuates significantly. Alternatively, the initial recognition model can meet certain performance indicators (e.g., precision, recall, etc.), which are not limited in this embodiment. The speech recognition model is used to generate the above... Figure 3 and Figure 4 The recognition result text corresponding to the voice data in the corresponding embodiment.

[0245] This application embodiment generates N candidate text samples corresponding to speech data through an initial speech recognition model. The initial speech recognition model may include an initial speech encoder using a CTC architecture, which can focus on local features in the input sequence, enhancing the model's performance in processing continuous signals (such as speech features) and improving the accuracy of speech recognition. Simultaneously, the CTC architecture, by allowing alignment of sequences of different lengths, avoids manual alignment between input and output, reducing the computational complexity of generating N candidate text samples. Aligning the N candidate text samples determines characters representing the same semantic meaning at the same character position, resulting in N aligned text samples with the same character length. Further fusing the N aligned text samples merges the vector elements at the same character position, thereby capturing more dimensional semantic features. Generating the predicted text corresponding to the sample speech data through the sample fusion vector and sample encoding vector allows for the incorporation of pronunciation information contained in the sample encoding vector while considering as many semantic features as possible in the N aligned text samples, reducing the impact of inaccurate user pronunciation. The model loss value is generated based on the predicted result text and the answer text. The model parameters of the initial recognition model are adjusted by the model loss value to obtain the speech recognition model. Since the speech recognition model takes into account the semantics contained in the text of the N sample alignment and the pronunciation information contained in the encoding vector, the accuracy and stability of speech data recognition can be improved when the speech data is recognized by the speech recognition model.

[0246] Please see Figure 7, Figure 7 This is a schematic diagram of the structure of a voice data processing device provided in an embodiment of this application. Figure 1 .like Figure 7 As shown, the speech data processing device 1 includes a speech feature extraction module 710, a candidate text generation module 720, an alignment processing module 730, an acquisition module 740, a fusion processing module 750, and a recognition text generation module 760.

[0247] The speech feature extraction module 710 is used to acquire speech data, encode the speech data to obtain an encoding vector, and the encoding vector represents the speech features of the speech data; the specific functions of the speech feature extraction module 710 can be found above. Figure 3 The specific description of step S101 in the corresponding embodiment will not be repeated here.

[0248] Candidate text generation module 720 is used to generate N candidate recognition texts based on the encoding vector; N is a positive integer; the candidate recognition texts correspond to speech features, and the specific functions of candidate text generation module 720 can be found above. Figure 3 The specific description of step S102 in the corresponding embodiment will not be repeated here.

[0249] Alignment processing module 730 is used to align N candidate recognition texts to obtain N aligned recognition texts; the character lengths of the N aligned recognition texts are all the same and / or the positions of characters representing the same semantic meaning in the N aligned recognition texts are all the same; the specific functions of alignment processing module 730 can be found above. Figure 3 The specific description of step S103 in the corresponding embodiment will not be repeated here.

[0250] The acquisition module 740 is used to acquire the text vectors corresponding to the N aligned recognition texts, and obtain N text vectors;

[0251] The fusion processing module 750 is used to perform vector fusion processing on N text vectors to obtain a fused vector;

[0252] The specific functions of the acquisition module 740 and the fusion processing module 750 can be found above. Figure 3 The specific description of step S104 in the corresponding embodiment will not be repeated here.

[0253] The text generation module 760 is used to generate the recognition result text corresponding to the speech data based on the fusion vector and the encoding vector. For details on the functions of the text generation module 760, please refer to the above. Figure 3 The specific description of step S105 in the corresponding embodiment will not be repeated here.

[0254] In one possible implementation, the speech feature extraction module 710 is used to encode the speech data, and when obtaining the encoded vector, it is specifically used to perform the following operations:

[0255] Input the speech data into the speech recognition model; the speech recognition model includes a speech coding layer;

[0256] In the speech coding layer, the speech data is segmented into frames to obtain M speech frames corresponding to the speech data; M is a positive integer.

[0257] Feature extraction is performed on M speech frames to obtain M speech embedding vectors;

[0258] Obtain the position embedding vectors corresponding to the M speech embedding vectors respectively. Based on the M position embedding vectors and the M speech embedding vectors, generate M encoded input vectors. The position embedding vectors are used to indicate the position of the speech frame data corresponding to the speech embedding vector in the speech data.

[0259] Generate an encoded vector based on M encoded input vectors.

[0260] In one possible implementation, the M encoded input vectors include the encoded input vector E. i Where i is a positive integer; the speech feature extraction module 710 is used to generate an encoding vector based on M encoding input vectors, specifically for performing the following operations:

[0261] Obtain the character encoding vector corresponding to each character to be recognized in the character database;

[0262] Concatenate the character encoding vectors to obtain the character encoding sequence;

[0263] Encoding input vector E i Cross-attention processing is performed with the character encoding sequence to obtain the encoded input vector E. i The corresponding unit attention vector A i Unit attention vector A i Including the encoded input vector E i The attention score between each character to be recognized in the character database;

[0264] When the unit attention vectors corresponding to the M encoded input vectors are obtained, the M unit attention vectors are concatenated to obtain the encoded vector.

[0265] For details on the functions of the speech feature extraction module 710, please refer to the above. Figure 4 The specific descriptions of steps S201 to S203 in the corresponding embodiments will not be repeated here.

[0266] In one possible implementation, when the candidate text generation module 720 generates N candidate recognition texts based on the encoding vector, it specifically performs the following operations:

[0267] Based on the attention score corresponding to each character to be identified in the encoding vector, P character combination lists are generated; P is a positive integer, and each character combination list includes M character labels. Each character label in the character combination list is determined by a different unit attention vector.

[0268] Obtain the total attention score for each of the P character combination lists; the total attention score for the character combination list is the sum of the attention scores for each of the M character labels.

[0269] Obtain the candidate text value N, and based on the candidate text value N and the total attention score, obtain a list of N character combinations from a list of P character combinations;

[0270] The N character combination list is time-aligned to obtain N candidate recognition texts.

[0271] In one possible implementation, when the candidate text generation module 720 generates a list of P character combinations based on the attention score corresponding to each character to be recognized in the encoding vector, it specifically performs the following operations:

[0272] Obtain the attention threshold, and determine the attention scores that are greater than or equal to the attention threshold among the M unit attention vectors in the encoding vector as candidate attention scores, and determine the characters to be recognized associated with the candidate attention scores as character labels;

[0273] Select and arrange the character labels to obtain a list of P character combinations.

[0274] In one possible implementation, the list of N character combinations includes the character combination list B. i Character combination list B i Includes the null character C i and the null character C i+1 null character C i and the null character C i+1 The adjacent null characters, null character C i With the null character C i+1 The list includes at least two identical character labels, where i is a positive integer; the candidate text generation module 720 performs temporal alignment processing on the list of N character combinations to obtain N candidate recognition texts, specifically performing the following operations:

[0275] The null character C will be located at i With the null character C i+1At least two identical character tags are merged into one character tag;

[0276] When character combination list B i When at least two identical character tags located between adjacent empty characters have been merged, delete the character combination list B. i The empty character in the list gives us a list of character combinations B. i The corresponding candidate text for recognition.

[0277] For details on the specific functions of the candidate text generation module 720, please refer to the above. Figure 4 The specific description of step S204 in the corresponding embodiment will not be repeated here.

[0278] In one possible implementation, the alignment processing module 730 is used to align N candidate recognition texts. When N aligned recognition texts are obtained, it is specifically used to perform the following operations:

[0279] Input N candidate texts into the speech recognition model; the speech recognition model includes an alignment processing layer.

[0280] In the alignment processing layer, a reference text is determined from N candidate texts for recognition;

[0281] N candidate recognition texts are obtained to align with the editing information corresponding to the reference recognition text. The editing information includes the editing distance and editing processing type corresponding to each candidate recognition text when converted into the reference recognition text. The sum of the editing distances satisfies the minimum editing distance condition.

[0282] Determine the position to be processed among N candidate texts based on editing information;

[0283] Placeholders are inserted at the positions to be processed in the N candidate recognition texts to obtain N aligned recognition texts.

[0284] In one possible implementation, the N candidate recognition texts include candidate recognition text F. i , where i is a positive integer; the editing information includes the candidate recognition text F. i Corresponding editing information G i The alignment processing module 730 is used to determine the position to be processed among N candidate recognition texts based on editing information, specifically to perform the following operations:

[0285] If you edit information G i If the indicated editing process type is insertion, then the candidate recognition text F will be... i The editing position corresponding to the insertion process is determined as the position to be processed;

[0286] If you edit information G iIf the indicated editing process type is deletion, then the candidate identified text F will be... i The corresponding business location in the reference recognition text is determined as the location to be processed; the business location and the candidate recognition text F i Remove the corresponding edit position associated with the process.

[0287] For details on the functions of the alignment processing module 730, please refer to the above. Figure 4 The specific descriptions of steps S205 and S206 in the corresponding embodiments will not be repeated here.

[0288] In one possible implementation, the acquisition module 740 is used to perform vector fusion processing on the text vectors corresponding to the N aligned recognition texts respectively, and when obtaining the fused vector, it is specifically used to perform the following operations:

[0289] Input N aligned and recognized texts into the speech recognition model; the speech recognition model includes a fusion processing layer;

[0290] In the fusion processing layer, feature extraction is performed on N aligned recognition texts to obtain text vectors corresponding to each of the N aligned recognition texts; the character length of each aligned recognition text is S, where S is a positive integer; the text vector includes S unit character vectors, and each unit character vector is associated with a character in the aligned recognition text;

[0291] The fusion processing module 750 is used to perform vector fusion processing on N text vectors. When obtaining the fused vector, it is specifically used to perform the following operations:

[0292] Average the unit character vectors at the same position in N text vectors to obtain S unit average vectors. Concatenate the S unit average vectors to obtain a merged vector.

[0293] The specific functions of the acquisition module 740 and the fusion processing module 750 can be found above. Figure 4 The specific description of step S207 in the corresponding embodiment will not be repeated here.

[0294] In one possible implementation, the fusion vector includes S unit average vectors, where S is a positive integer; when the recognition text generation module 760 generates the recognition result text corresponding to the speech data based on the fusion vector and the encoding vector, it is specifically used to perform the following operations:

[0295] The fused vector and the encoded vector are input into the speech recognition model; the speech recognition model includes a speech decoding layer.

[0296] In the speech decoding layer, cross-attention processing is performed on the fusion coding vector and the fusion vector to obtain the attention result vector. The attention result vector includes the unit result vectors corresponding to S unit average vectors, and the unit result vector includes the attention score corresponding to each character to be recognized in the character database.

[0297] S recognition characters are generated based on the S unit result vectors in the attention result vector, and the S recognition characters are determined as the recognition result text corresponding to the speech data.

[0298] For details on the functions of the text recognition and generation module 760, please refer to the above. Figure 4 The specific description of step S208 in the corresponding embodiment will not be repeated here.

[0299] This application embodiment generates M encoded input vectors corresponding to speech data and captures key features in the encoded input vectors, such as phonemes, syllables, and intonation. The dependency relationship between the encoded input vectors and each character to be recognized can be calculated, thereby understanding the semantic and syntactic structure of the speech data and obtaining the encoded vectors. N candidate recognition texts are generated from the encoded vectors. These N candidate recognition texts can be generated from different attention angles and include different semantic characters, syntactic structures, and semantic relationships. Alignment processing is performed on the N candidate recognition texts to determine characters representing the same semantics at the same character position, resulting in N aligned recognition texts with the same character length. Further fusion processing is performed on the N aligned recognition texts to fuse the vector elements at the same character position, thereby capturing different semantic characters, syntactic structures, and semantic relationships in the N aligned recognition texts, as well as capturing more dimensions of semantic features. By generating the recognition result text corresponding to the speech data through the fusion vector and the encoding vector, the semantic features in the N aligned recognition texts can be considered as much as possible while combining the pronunciation information contained in the encoding vector, so as to reduce the impact of the user's inaccurate pronunciation, thereby improving the accuracy and stability of speech data recognition.

[0300] On the other hand, the speech recognition model proposed in this application can be an integrated model (i.e., including both a recognition model and an error correction model). The speech coding layer can be the recognition model, and the alignment processing layer, fusion processing layer, and speech decoding layer can be the error correction model. When the recognition model and the error correction model are deployed separately, the security of the original speech data can be improved, protecting privacy. When the recognition model and the error correction model are integrated into one model, i.e., a speech recognition model, the integrated speech recognition model can reduce data transmission and intermediate processing overhead, and can share certain feature vectors. Therefore, it can improve performance while increasing computational efficiency and reducing deployment costs. Furthermore, since this application uses a non-autoregressive decoding method, the decoding computational complexity can be greatly reduced, improving the efficiency of speech recognition.

[0301] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0302] Please see Figure 8 , Figure 8 This is a schematic diagram of the structure of a voice data processing device provided in an embodiment of this application. Figure 2 .like Figure 8 As shown, the speech data processing device 2 includes a sample data input module 810, a sample feature extraction module 820, an initial alignment processing module 830, an initial fusion processing module 840, and a model training module 850.

[0303] The sample data input module 810 is used to acquire sample speech data and the corresponding answer text, and input the sample speech data and answer text into the initial recognition model; the specific functions of the sample data input module 810 can be found above. Figure 4 The specific description of step S301 in the corresponding embodiment will not be repeated here.

[0304] The sample feature extraction module 820 is used to encode the sample speech data in the initial recognition model to obtain a sample encoding vector, and to generate N sample candidate recognition texts based on the sample encoding vector; N is a positive integer; the sample encoding vector represents the speech features of the sample speech data, and the sample candidate recognition texts correspond to the speech features; the specific functions of the sample feature extraction module 820 can be found above. Figure 4The specific descriptions of steps S302 and S303 in the corresponding embodiments will not be repeated here.

[0305] The initial alignment processing module 830 is used to align N sample candidate recognition texts based on the answer text, resulting in N sample aligned recognition texts. The character length of the N sample aligned recognition texts is the same as the character length of the answer text, and the positions of characters representing the same semantic meaning in the N sample aligned recognition texts and the answer text are the same. The specific functions of the initial alignment processing module 830 can be found above. Figure 4 The specific description of step S304 in the corresponding embodiment will not be repeated here.

[0306] The initial fusion processing module 840 is used to acquire the sample text vectors corresponding to the N sample aligned recognition texts, obtain N sample text vectors, perform vector fusion processing on the N sample text vectors to obtain a sample fusion vector, and generate the prediction result text corresponding to the sample speech data based on the sample fusion vector and the sample encoding vector; the specific functions of the initial fusion processing module 840 can be found above. Figure 4 The specific descriptions of steps S305 and S306 in the corresponding embodiments will not be repeated here.

[0307] The model training module 850 is used to generate model loss values ​​based on the predicted result text and the answer text. The model parameters of the initial recognition model are adjusted using these loss values ​​until the initial recognition model meets the convergence condition, thus obtaining the speech recognition model. The speech recognition model is used to generate the recognition result text corresponding to the speech data. The specific functions of the model training module 850 can be found above. Figure 4 The specific description of step S307 in the corresponding embodiment will not be repeated here.

[0308] This application embodiment generates N candidate text samples corresponding to speech data through an initial speech recognition model. The initial speech recognition model may include an initial speech encoder using a CTC architecture, which can focus on local features in the input sequence, enhancing the model's performance in processing continuous signals (such as speech features) and improving the accuracy of speech recognition. Simultaneously, the CTC architecture, by allowing alignment of sequences of different lengths, avoids manual alignment between input and output, reducing the computational complexity of generating N candidate text samples. Aligning the N candidate text samples determines characters representing the same semantic meaning at the same character position, resulting in N aligned text samples with the same character length. Further fusing the N aligned text samples merges the vector elements at the same character position, thereby capturing more dimensional semantic features. Generating the predicted text corresponding to the sample speech data through the sample fusion vector and sample encoding vector allows for the incorporation of pronunciation information contained in the sample encoding vector while considering as many semantic features as possible in the N aligned text samples, reducing the impact of inaccurate user pronunciation. The model loss value is generated based on the predicted result text and the answer text. The model parameters of the initial recognition model are adjusted by the model loss value to obtain the speech recognition model. Since the speech recognition model takes into account the semantics contained in the text of the N sample alignment and the pronunciation information contained in the encoding vector, the accuracy and stability of speech data recognition can be improved when the speech data is recognized by the speech recognition model.

[0309] Please see Figure 9 , Figure 9 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Figure 9 As shown, the computer device 1000 may include a processor 1001, a network interface 1004, and a memory 1005. Furthermore, the computer device 1000 may also include a user interface 1003 and at least one communication bus 1002. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen and a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be high-speed RAM or non-volatile memory, such as at least one disk drive. Optionally, the memory 1005 may also be at least one storage device located remotely from the processor 1001. Figure 9As shown, the memory 1005, which is a computer-readable storage medium, may include an operating system, a network communication module, a user interface module, and computer programs.

[0310] In such Figure 9 In the computer device 1000 shown, the network interface 1004 can provide network communication elements; the user interface 1003 is mainly used to provide an input interface for users; and the processor 1001 can be used to call computer programs stored in the memory 1005.

[0311] The network interface 1004 provides network communication elements; the user interface 1003 is mainly used to provide an input interface for users; and the processor 1001 can be used to call computer programs stored in the memory 1005, specifically for executing... Figure 3 , Figure 4 and Figure 5 Each step in the embodiments.

[0312] It should be understood that the computer device 1000 described in the embodiments of this application can execute the foregoing text. Figure 3 , Figure 4 and Figure 5 The description of the voice data processing method in any corresponding embodiment will not be repeated here. Furthermore, the beneficial effects of using the same method will also not be repeated.

[0313] Furthermore, it should be noted that this application embodiment also provides a computer-readable storage medium, which stores a computer program. When the processor executes the computer program, it can execute the aforementioned... Figure 3 , Figure 4 and Figure 5 The description of the above-described voice data processing method in any corresponding embodiment is already provided, and therefore will not be repeated here. Furthermore, the beneficial effects of using the same method will also not be repeated. For technical details not disclosed in the computer-readable storage medium embodiments related to this application, please refer to the description of the method embodiments of this application.

[0314] The aforementioned computer-readable storage medium can be the internal storage unit of the voice data processing apparatus provided in any of the foregoing embodiments or the computer device described above, such as the hard disk or memory of the computer device. The computer-readable storage medium can also be an external storage device of the computer device, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., provided on the computer device. Furthermore, the computer-readable storage medium may include both internal storage units and external storage devices of the computer device. The computer-readable storage medium is used to store the computer program and other programs and data required by the computer device. The computer-readable storage medium can also be used to temporarily store data that has been displayed or will be displayed.

[0315] Furthermore, it should be noted that this application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. The processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, causing the computer device to perform the aforementioned... Figure 3 , Figure 4 and Figure 5 The method provided in any of the corresponding embodiments.

[0316] The terms "first," "second," etc., in the specification, claims, and drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the term "comprising," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, apparatus, product, or device that includes a series of steps or units is not limited to the listed steps or modules, but may optionally include steps or modules not listed, or may optionally include other step units inherent to these processes, methods, apparatuses, products, or devices.

[0317] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the foregoing description as a network element. Whether these network elements are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can implement the described network elements using different methods for each specific application, but such implementation should not be considered beyond the scope of this application.

[0318] The methods and related apparatus provided in this application are described with reference to the method flowcharts and / or structural diagrams provided in this application. Specifically, each block of the method flowcharts and / or structural diagrams, as well as combinations of blocks in the flowcharts and / or block diagrams, can be implemented by a computer program. These computer programs can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable device to create a machine, such that the computer program, executed by the processor of the computer or other programmable device, produces a mechanism for implementing the process... Figure 1 A schematic diagram of one or more processes and / or structures. Figure 1 The computer program may be a means for performing the functions specified in one or more boxes. These computer programs may also be stored in a computer-readable storage medium that can direct a computer or other programmable device to function in a particular manner, causing the computer program stored in the computer-readable storage medium to produce an article of manufacture including the program means, or to be transmitted via a computer-readable storage medium. The computer program can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The program means is implemented in the process. Figure 1 A schematic diagram of one or more processes and / or structures. Figure 1 The functions specified in one or more boxes. These computer programs may also be loaded onto a computer or other programmable device to cause a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, thereby providing the computer program executing on the computer or other programmable device with the means to implement the process. Figure 1 A process or multiple processes and / or structures illustrate the steps of the functions specified in one or more boxes.

[0319] The steps in the method of this application embodiment can be adjusted, combined, or deleted according to actual needs.

[0320] The modules in the device of this application embodiment can be merged, divided, and deleted according to actual needs.

[0321] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, any equivalent variations made in accordance with the claims of this application shall still fall within the scope of this application.

Claims

1. A voice data processing method, characterized in that, include: Acquire speech data, encode the speech data to obtain an encoding vector, and the encoding vector represents the speech features of the speech data; N candidate recognition texts are generated based on the encoded vector, and the candidate recognition texts correspond to the speech features, where N is a positive integer. The N candidate recognition texts are aligned to obtain N aligned recognition texts, wherein the character lengths of the N aligned recognition texts are all the same, and / or the positions of characters representing the same semantics in the N aligned recognition texts are all the same; Obtain the text vectors corresponding to the N aligned and recognized texts to obtain N text vectors; perform vector fusion processing on the N text vectors to obtain a fused vector; The recognition result text corresponding to the speech data is generated based on the fusion vector and the encoding vector.

2. The method according to claim 1, characterized in that, The process of encoding the speech data to obtain an encoding vector includes: The speech data is input into a speech recognition model; the speech recognition model includes a speech coding layer. In the speech coding layer, the speech data is segmented into frames to obtain M speech frames corresponding to the speech data; M is a positive integer. Feature extraction is performed on the M speech frames to obtain M speech embedding vectors; Obtain the position embedding vectors corresponding to the M speech embedding vectors respectively, and generate M coded input vectors based on the M position embedding vectors and the M speech embedding vectors; the position embedding vectors are used to indicate the position of the speech frame data corresponding to the speech embedding vector in the speech data; Generate an encoding vector based on the M encoding input vectors.

3. The method according to claim 2, characterized in that, The M encoded input vectors include encoded input vector E i Where i is a positive integer; the generation of the encoding vector based on the M encoding input vectors includes: Obtain the character encoding vector corresponding to each character to be recognized in the character database; The character encoding vectors are concatenated to obtain the character encoding sequence; For the encoded input vector E i Cross-attention processing is performed on the character encoding sequence to obtain the encoded input vector E. i The corresponding unit attention vector A i The unit attention vector A i Including the encoded input vector E i The attention score between each character to be identified in the character database; When the unit attention vectors corresponding to the M encoded input vectors are obtained, the M unit attention vectors are concatenated to obtain the encoded vector.

4. The method according to claim 3, characterized in that, The generation of N candidate recognition texts based on the encoded vector includes: Based on the attention score corresponding to each character to be identified in the encoding vector, P character combination lists are generated; P is a positive integer, and each character combination list includes M character labels, each character label in the character combination list is determined by a different unit attention vector; Obtain the total attention score corresponding to each of the P character combination lists; the total attention score of the character combination list is the sum of the attention scores corresponding to each of the M character labels; Obtain the candidate text value N, and based on the candidate text value N and the total attention score in the P character combination list, obtain the N character combination list; The list of N character combinations is time-aligned to obtain N candidate recognition texts.

5. The method according to claim 4, characterized in that, The process involves generating a list of P character combinations based on the attention score corresponding to each character to be recognized in the encoding vector, including: Obtain an attention threshold, determine the attention scores that are greater than or equal to the attention threshold among the M unit attention vectors in the encoding vector as candidate attention scores, and determine the characters to be identified associated with the candidate attention scores as character labels; The character labels are selected, arranged, and combined to obtain a list of P character combinations.

6. The method according to claim 4, characterized in that, The N character combination lists include character combination list B. i The character combination list B i Includes the null character C i and the null character C i+1 The null character C i and the empty character C i+1 The adjacent null character, the null character C i With the empty character C i+1 The interval must contain at least two identical character labels, where i is a positive integer; The step of performing time-series alignment processing on the list of N character combinations yields N candidate recognition texts, including: The empty character C will be located at i With the empty character C i+1 At least two identical character tags are merged into one character tag; When the character combination list B i When at least two identical character tags located between adjacent empty characters have been merged, the character combination list B is deleted. i The empty character in the list gives us a list of character combinations B. i The corresponding candidate text for recognition.

7. The method according to claim 1, characterized in that, The alignment process of the N candidate recognition texts to obtain N aligned recognition texts includes: The N candidate texts are input into the speech recognition model; the speech recognition model includes an alignment processing layer. In the alignment processing layer, a reference recognition text is determined from the N candidate recognition texts; The N candidate recognition texts are obtained to align the editing information corresponding to the reference recognition text; the editing information includes the editing distance and editing processing type corresponding to each candidate recognition text when converted to the reference recognition text, and the sum of the editing distances satisfies the minimum editing distance condition; Based on the editing information, the position to be processed is determined among the N candidate recognition texts; Placeholders are inserted at the positions to be processed in the N candidate recognition texts to obtain N aligned recognition texts.

8. The method according to claim 7, characterized in that, The N candidate recognition texts include candidate recognition text F i , where i is a positive integer; the editing information includes the candidate recognition text F. i Corresponding editing information G i ; The step of determining the position to be processed among the N candidate recognition texts based on the editing information includes: If the edit information G i If the indicated editing process type is insertion, then the candidate recognition text F will be... i The editing position corresponding to the insertion process described above is determined as the position to be processed; If the edit information G i If the indicated editing process type is deletion, then the candidate identified text F will be... i The business location in the corresponding reference recognition text is determined as the location to be processed; the business location and the candidate recognition text F i The deletion process described herein is associated with the corresponding editing location.

9. The method according to claim 1, characterized in that, The step of obtaining the text vectors corresponding to the N aligned and recognized texts, resulting in N text vectors, includes: The N aligned and recognized texts are input into the speech recognition model; the speech recognition model includes a fusion processing layer. In the fusion processing layer, feature extraction is performed on the N aligned recognition texts to obtain text vectors corresponding to the N aligned recognition texts respectively; the character length of the aligned recognition texts is S, where S is a positive integer; the text vectors include S unit character vectors, and each unit character vector is associated with a character in the aligned recognition text; The vector fusion process of the N text vectors to obtain the fused vector includes: Average the unit character vectors at the same position in N text vectors to obtain S unit average vectors. Then, concatenate the S unit average vectors to obtain a fused vector.

10. The method according to claim 1, characterized in that, The fusion vector includes S unit average vectors, where S is a positive integer; the step of generating the recognition result text corresponding to the speech data based on the fusion vector and the encoding vector includes: The fusion vector and the encoding vector are input into a speech recognition model; the speech recognition model includes a speech decoding layer; In the speech decoding layer, cross-attention processing is performed on the fusion coding vector and the fusion vector to obtain an attention result vector; the attention result vector includes unit result vectors corresponding to S unit average vectors, and the unit result vector includes the attention score corresponding to each character to be recognized in the character database; Based on the S unit result vectors in the attention result vector, S recognition characters are generated, and the S recognition characters are determined as the recognition result text corresponding to the speech data.

11. A voice data processing method, characterized in that, include: Obtain sample speech data and the corresponding answer text, and input the sample speech data and the answer text into the initial recognition model; In the initial recognition model, the sample speech data is encoded to obtain a sample encoding vector, and N sample candidate recognition texts are generated based on the sample encoding vector; N is a positive integer; the sample encoding vector represents the speech features of the sample speech data, and the sample candidate recognition texts correspond to the speech features; Based on the answer text, the N sample candidate recognition texts are aligned to obtain N sample aligned recognition texts; the character length of the N sample aligned recognition texts is the same as the character length of the answer text, and the positions of characters representing the same semantics in the N sample aligned recognition texts and the answer text are the same; Obtain the sample text vectors corresponding to the N sample aligned recognition texts respectively to obtain N sample text vectors. Perform vector fusion processing on the N sample text vectors to obtain sample fusion vectors. Generate the prediction result text corresponding to the sample speech data based on the sample fusion vectors and the sample encoding vectors. Based on the predicted result text and the answer text, a model loss value is generated. The model parameters of the initial recognition model are adjusted using the model loss value until the initial recognition model meets the model convergence condition, thus obtaining a speech recognition model. The speech recognition model is used to generate the recognition result text corresponding to the speech data.

12. A voice data processing device, characterized in that, include: The speech feature extraction module is used to acquire speech data, encode the speech data to obtain an encoding vector, and the encoding vector represents the speech features of the speech data. The candidate text generation module is used to generate N candidate recognition texts based on the encoding vector, wherein the candidate recognition texts correspond to the speech features, and N is a positive integer. The alignment processing module is used to align the N candidate recognition texts to obtain N aligned recognition texts, wherein the character lengths of the N aligned recognition texts are all the same and / or the positions of characters representing the same semantics in the N aligned recognition texts are all the same. The acquisition module is used to acquire the text vectors corresponding to the N aligned recognition texts respectively, and obtain N text vectors; The fusion processing module is used to perform vector fusion processing on the N text vectors to obtain a fused vector; The text generation module is used to generate the recognition result text corresponding to the speech data based on the fusion vector and the encoding vector.

13. A voice data processing device, characterized in that, include: The sample data input module is used to acquire sample speech data and the corresponding answer text, and input the sample speech data and the answer text into the initial recognition model. The sample feature extraction module is used to encode the sample speech data in the initial recognition model to obtain a sample encoding vector, and generate N sample candidate recognition texts based on the sample encoding vector; N is a positive integer; the sample encoding vector represents the speech features of the sample speech data, and the sample candidate recognition texts correspond to the speech features; The initial alignment processing module is used to align the N sample candidate recognition texts based on the answer text to obtain N sample aligned recognition texts; the character length of the N sample aligned recognition texts is the same as the character length of the answer text, and the positions of characters representing the same semantics in the N sample aligned recognition texts and the answer text are the same; The initial fusion processing module is used to obtain the sample text vectors corresponding to the N sample aligned recognition texts respectively, to obtain N sample text vectors, to perform vector fusion processing on the N sample text vectors to obtain a sample fusion vector, and to generate the prediction result text corresponding to the sample speech data based on the sample fusion vector and the sample encoding vector. The model training module is used to generate a model loss value based on the predicted result text and the answer text, and to adjust the model parameters of the initial recognition model using the model loss value until the initial recognition model meets the model convergence condition, thereby obtaining a speech recognition model; the speech recognition model is used to generate the recognition result text corresponding to the speech data.

14. A computer device, characterized in that, include: Processor, memory, and network interface; The processor is connected to the memory and the network interface, wherein the network interface is used to provide data communication functions, the memory is used to store computer programs, and the processor is used to call the computer programs so that the computer device executes the method according to any one of claims 1-11.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program adapted to be loaded and executed by a processor to cause a computer device having the processor to perform the method according to any one of claims 1-11.

16. A computer program product, characterized in that, The computer program product includes a computer program stored in a computer-readable storage medium and adapted to be read and executed by a processor so that a computer device having the processor performs the method of any one of claims 1-11.

Images