Large model training method and device, voice recognition text processing method and device, equipment and medium

Abstract

The invention discloses a large model training method and device, a voice recognition text processing method and device, equipment and a medium, relates to the technical field of communication, and aims to improve the accuracy of text output obtained by a post-processing task. The method comprises the steps that text data sets used for model training are obtained, the text data sets comprise a first text data set and a second text data set, the first text data set comprises a constructed text smoothing task text data set, a text error correction task text data set, a punctuation recovery task text data set and an ITN task text data set, and the second text data set comprises a constructed text smoothing task text data set and a constructed text error correction task text data set; the second text data set is a text data set manually labeled in a real scene; adding a task label for the text data set; and training the first large model by utilizing the text data set added with the task label. According to the embodiment of the invention, the output accuracy of the text obtained by the post-processing task can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of communication, and particularly relates to a large model training method, a speech recognition text processing method, device and equipment and medium. BACKGROUND

[0002] Due to the input of oral speech often contains some unavoidable unsmooth factors such as repetition, redundancy, stuttering, correction and the like, in addition, there are wrong words and missing words in the recognition result of an automatic speech recognition (ASR) system, and the recognition text does not include punctuation marks and the appearance of long numerical text also greatly affects the reading experience. The existence of these problems leads to unsmooth system output text, affects the integrity and readability of the text semantics, and further causes certain adverse effects on the subsequent downstream tasks of the recognition text.

[0003] Through text smoothing, text error correction, punctuation recovery, inverse text normalization (ITN) and the like post-processing tasks, the phenomena of unsmooth recognition text, wrong words, missing words, no punctuation and long numbers can be optimized. However, the text output accuracy obtained by the existing post-processing scheme is poor. SUMMARY

[0004] The embodiments of the present application provide a large model training method, a speech recognition text processing method, device and equipment and medium to improve the accuracy of the text output obtained by the post-processing task.

[0005] In a first aspect, the embodiments of the present application provide a large model training method, comprising:

[0006] obtaining a text data set for model training, wherein the text data set comprises a first text data set and a second text data set, the first text data set comprises a text smoothing task text data set, a text error correction task text data set, a punctuation recovery task text data set and an ITN task text data set, and the second text data set is a text data set artificially annotated in a real scene;

[0007] adding a task label to the text data set, wherein the task label is used to indicate that at least two of the text smoothing task, the text error correction task, the punctuation recovery task and the ITN task are performed;

[0008] The first large model is trained by using a text data set added with a task label, wherein, during the training, the model parameters of a first part of the Transformer layers of the first large model remain unchanged, the model parameters of a second part of the Transformer layers of the first large model are adjusted, and the proportion of the number of layers of the second part of the Transformer layers to the total number of layers of the Transformer layers of the first large model is less than or equal to a preset value.

[0009] Optionally, the text smoothing task text data set is obtained by the following method:

[0010] The text smoothing task text data set is obtained by performing one or more of the following operations on the plurality of to-be-processed text data:

[0011] At least one character is selected from the to-be-processed text data, and the at least one character is added at a position adjacent to the at least one character to obtain the text smoothing task text data set.

[0012] The to-be-processed text data is segmented to obtain a plurality of segmented words, at least one segmented word is selected from the plurality of segmented words, and the at least one segmented word is added at a position adjacent to the at least one segmented word.

[0013] The to-be-processed text data is segmented to obtain a plurality of segmented words, and a mood pause word and / or a meaningless indicating word are added at one or more target positions, wherein the target position is a position between two adjacent segmented words.

[0014] The to-be-processed text data is segmented to obtain a plurality of segmented words, and one or more segmented words in the plurality of segmented words are deleted.

[0015] Optionally, the text correction task text data set is obtained by the following method:

[0016] The to-be-processed text data is segmented to obtain a plurality of segmented words.

[0017] A target segmented word in the plurality of segmented words is replaced by a near sound word of the target segmented word, wherein the target segmented word is one or more segmented words in the plurality of segmented words, and the near sound word is a homophone of the target segmented word or a homophone of the target segmented word.

[0018] Optionally, the punctuation recovery task text data set is obtained by the following method:

[0019] A target punctuation symbol in the to-be-processed text data, except a comma, a period, a question mark and a colon, is replaced and / or deleted.

[0020] Optionally, the ITN task text data set is obtained by the following method:

[0021] The second large model is used to process the text data to be processed by using a prompt, wherein the prompt includes indication information of a type of number to be converted.

[0022] Optionally, the training of the first large model by using the text data set added with the task label comprises:

[0023] The original weight matrix of the first large model is singular value decomposed to obtain a left singular vector, a diagonal matrix containing singular values, and a right singular vector, wherein the singular values in the diagonal matrix are arranged in descending order on the matrix diagonal line of the diagonal matrix.

[0024] The left singular vector is divided into a first part left singular vector and a second part left singular vector, the diagonal matrix is divided into a first part diagonal matrix and a second part diagonal matrix, and the right singular vector is divided into a first part right singular vector and a second part right singular vector.

[0025] A first part parameter is obtained according to the first part left singular vector, the first part diagonal matrix, and the first part right singular vector, and a second part parameter is obtained according to the second part left singular vector, the second part diagonal matrix, and the second part right singular vector.

[0026] The first part parameter and a model adjustment parameter are summed to obtain an updated weight matrix of the first large model, wherein the model adjustment parameter is determined according to a first low-rank parameter matrix and a second low-rank parameter matrix, and the first low-rank parameter matrix and the second low-rank parameter matrix are initialized by the second part parameter.

[0027] Optionally, the training of the first large model by using the text data set added with the task label comprises:

[0028] The original weight matrix of the first large model is singular value decomposed to obtain a left singular vector, a diagonal matrix containing singular values, and a right singular vector, wherein the singular values in the diagonal matrix are arranged in descending order on the matrix diagonal line of the diagonal matrix.

[0029] The left singular vector is divided into a first part left singular vector and a second part left singular vector, the diagonal matrix is divided into a first part diagonal matrix and a second part diagonal matrix, and the right singular vector is divided into a first part right singular vector and a second part right singular vector.

[0030] a first part parameter is obtained according to the first part left singular vector, the first part diagonal matrix and the first part right singular vector, and a second part parameter is obtained according to the second part left singular vector, the second part diagonal matrix and the second part right singular vector;

[0031] The second part parameter is adjusted by using a dynamic allocation rank method to obtain an adjusted second part parameter;

[0032] The first part parameter and the adjusted second part parameter are used to obtain an updated weight matrix of the first large model.

[0033] Optionally, the method further comprises:

[0034] A contribution value of each singular value in the diagonal matrix and a sum of contribution values of all singular values are obtained;

[0035] The contribution values of the singular values are added one by one from the largest singular value to obtain a contribution value sum;

[0036] If a ratio of a contribution value sum corresponding to the kth singular value to the sum is greater than or equal to a preset value, a first parameter r is obtained, wherein the first parameter r = m-k, m is a total number of singular values in the diagonal matrix and m is an integer greater than 0, k is an integer and 1≤k

[0037] The left singular vector is divided into the first part left singular vector and the second part left singular vector, the diagonal matrix is divided into the first part diagonal matrix and the second part diagonal matrix, and the right singular vector is divided into the first part right singular vector and the second part right singular vector, comprising:

[0038] The left singular vector is divided into the first part left singular vector and the second part left singular vector, the diagonal matrix is divided into the first part diagonal matrix and the second part diagonal matrix, and the right singular vector is divided into the first part right singular vector and the second part right singular vector based on the first parameter r.

[0039] In a second aspect, an embodiment of the present application provides a speech recognition text processing method, comprising:

[0040] Obtaining a speech recognition text;

[0041] Post-processing the speech recognition text by using a first large model, wherein the first large model is a large model trained by using the large model training method.

[0042] In a third aspect, an embodiment of the present application provides a large model training device, comprising:

[0043] The first obtaining module is configured to obtain a text data set for model training, wherein the text data set comprises a first text data set and a second text data set, the first text data set comprises a text smoothing task text data set, a text correction task text data set, a punctuation recovery task text data set, and an ITN task text data set, and the second text data set is a text data set artificially labeled in a real scene.

[0044] The first processing module is configured to add task labels to the text data set, wherein the task labels are used to indicate at least two of the following tasks: a text smoothing task, a text correction task, a punctuation recovery task, and an ITN task.

[0045] The first training module is configured to train a first large model by using the text data set with the added task labels, wherein, during the training, model parameters of a first part of Transformer layers of the first large model remain unchanged, model parameters of a second part of Transformer layers of the first large model are adjusted, and a proportion of a number of layers of the second part of Transformer layers to a total number of layers of the first large model is less than or equal to a preset value.

[0046] Optionally, the first obtaining module is further configured to obtain the text smoothing task text data set in the following manner:

[0047] The following one or more of the following processing is performed on a plurality of to-be-processed text data to obtain the text smoothing task text data set:

[0048] At least one character is selected from the to-be-processed text data, and the at least one character is added at an adjacent position of the at least one character to obtain the text smoothing task text data set.

[0049] The to-be-processed text data is segmented to obtain a plurality of segmented words, at least one segmented word is selected from the plurality of segmented words, and the at least one segmented word is added at an adjacent position of the at least one segmented word.

[0050] The to-be-processed text data is segmented to obtain a plurality of segmented words, and a mood pause word and / or a meaningless demonstrative word are added at one or more target positions, wherein the target positions are positions between adjacent two segmented words.

[0051] The to-be-processed text data is segmented to obtain a plurality of segmented words, and one or more segmented words in the plurality of segmented words are deleted.

[0052] Optionally, the first obtaining module is further configured to obtain the text correction task text data set in the following manner:

[0053] Tokenize the to-be-processed text data to obtain a plurality of tokens;

[0054] Replace a target token in the plurality of tokens with a homophone of the target token, wherein the target token is one or more tokens in the plurality of tokens, and the homophone is a homophone or a homophone of the target token.

[0055] Optionally, the first obtaining module is further configured to obtain the punctuation restoration task text data set in the following manner:

[0056] Replace and / or delete a target punctuation symbol in the to-be-processed text data, except for a comma, a period, a question mark, and a colon.

[0057] Optionally, the first obtaining module is further configured to obtain the ITN task text data set in the following manner:

[0058] Process the to-be-processed text data by using a second large model and a prompt to obtain the ITN task text data set, wherein the prompt includes instruction information about a type of number to be converted.

[0059] Optionally, the first training module is further configured to:

[0060] Perform singular value decomposition on an original weight matrix of the first large model to obtain a left singular vector, a diagonal matrix containing singular values, and a right singular vector, wherein the singular values in the diagonal matrix are arranged in descending order along the matrix diagonal of the diagonal matrix;

[0061] Divide the left singular vector into a first part left singular vector and a second part left singular vector, divide the diagonal matrix into a first part diagonal matrix and a second part diagonal matrix, and divide the right singular vector into a first part right singular vector and a second part right singular vector;

[0062] Obtain a first part parameter according to the first part left singular vector, the first part diagonal matrix, and the first part right singular vector, and obtain a second part parameter according to the second part left singular vector, the second part diagonal matrix, and the second part right singular vector;

[0063] Obtain an updated weight matrix of the first large model by using a sum of the first part parameter and a model adjustment parameter, wherein the model adjustment parameter is determined according to a first low-rank parameter matrix and a second low-rank parameter matrix, and the first low-rank parameter matrix and the second low-rank parameter matrix are initialized by the second part parameter.

[0064] Optionally, the first training module is further configured to:

[0065] perform singular value decomposition on the original weight matrix of the first large model to obtain left singular vectors, a diagonal matrix containing singular values, and right singular vectors, wherein the singular values in the diagonal matrix are arranged in descending order along the matrix diagonal of the diagonal matrix;

[0066] divide the left singular vectors into first-part left singular vectors and second-part left singular vectors, divide the diagonal matrix into a first-part diagonal matrix and a second-part diagonal matrix, and divide the right singular vectors into first-part right singular vectors and second-part right singular vectors;

[0067] obtain first-part parameters according to the first-part left singular vectors, the first-part diagonal matrix, and first-part right singular vectors, and obtain second-part parameters according to the second-part left singular vectors, the second-part diagonal matrix, and second-part right singular vectors;

[0068] adjust the second-part parameters by using a dynamic rank allocation method to obtain adjusted second-part parameters;

[0069] obtain an updated weight matrix of the first large model by using the first-part parameters and the adjusted second-part parameters.

[0070] Optionally, the apparatus can further include a second obtaining module configured to:

[0071] obtain a contribution value of each singular value in the diagonal matrix and a sum of contribution values of all singular values;

[0072] start from the largest singular value, and add the contribution values of the singular values one by one to obtain a contribution value sum;

[0073] if a ratio of the contribution value sum corresponding to the kth singular value to the sum is greater than or equal to a preset value, obtain a first parameter r, wherein the first parameter r = m-k, m is the total number of singular values in the diagonal matrix and m is an integer greater than 0, k is an integer and 1≤k

[0074] The first training module is further configured to:

[0075] divide the left singular vectors into the first-part left singular vectors and the second-part left singular vectors, divide the diagonal matrix into the first-part diagonal matrix and the second-part diagonal matrix, and divide the right singular vectors into the first-part right singular vectors and the second-part right singular vectors based on the first parameter r.

[0076] In a fourth aspect, an embodiment of the present application provides a speech recognition text processing apparatus, including:

[0077] The first obtaining module is configured to obtain a speech recognition text;

[0078] The first processing module is configured to perform post-processing on the speech recognition text by using a first large model, wherein the first large model is a large model trained by using the large model training method.

[0079] In a fifth aspect, an embodiment of the present application provides a large model training apparatus, including a processor and a transceiver;

[0080] The processor is configured to:

[0081] obtain a text data set for model training, wherein the text data set includes a first text data set and a second text data set, the first text data set includes a text smoothing task text data set, a text correction task text data set, a punctuation recovery task text data set, and an ITN task text data set, and the second text data set is a text data set artificially annotated in a real scene;

[0082] add a task label to the text data set, wherein the task label is used to indicate at least two of the following tasks: a text smoothing task, a text correction task, a punctuation recovery task, and an ITN task;

[0083] train a first large model by using the text data set with the task label, wherein in the training process, model parameters of a first part of Transformer layers of the first large model remain unchanged, model parameters of a second part of Transformer layers of the first large model are adjusted, and a proportion of a number of layers of the second part of Transformer layers to a total number of layers of the first large model is less than or equal to a preset value.

[0084] Optionally, the processor is further configured to obtain the text smoothing task text data set in the following manner:

[0085] perform one or more of the following processing on a plurality of to-be-processed text data to obtain the text smoothing task text data set:

[0086] select at least one character from the to-be-processed text data, and add the at least one character at a position adjacent to the at least one character to obtain the text smoothing task text data set;

[0087] perform word segmentation on the to-be-processed text data to obtain a plurality of segmented words, select at least one segmented word from the plurality of segmented words, and add the at least one segmented word at a position adjacent to the at least one segmented word;

[0088] performing word segmentation on the to-be-processed text data to obtain a plurality of segmented words; and adding a mood pause word and / or a meaningless demonstrative word at one or more target positions, wherein the target positions are positions between adjacent two segmented words.

[0089] performing word segmentation on the to-be-processed text data to obtain a plurality of segmented words, and deleting one or more segmented words in the plurality of segmented words.

[0090] Optionally, the processor is further configured to obtain the text error correction task text data set in the following manner:

[0091] performing word segmentation on the to-be-processed text data to obtain a plurality of segmented words;

[0092] replacing a target segmented word in the plurality of segmented words with a near sound word of the target segmented word, wherein the target segmented word is one or more segmented words in the plurality of segmented words, and the near sound word is a homophone or a homonym of the target segmented word.

[0093] Optionally, the processor is further configured to obtain the punctuation recovery task text data set in the following manner:

[0094] replacing and / or deleting a target punctuation symbol in the to-be-processed text data, except for a comma, a period, a question mark and a colon.

[0095] Optionally, the processor is further configured to obtain the ITN task text data set in the following manner:

[0096] processing the to-be-processed text data by using a second large model and a prompt to obtain the ITN task text data set, wherein the prompt includes indication information about a type of number to be converted.

[0097] Optionally, the processor is further configured to:

[0098] performing singular value decomposition on an original weight matrix of the first large model to obtain a left singular vector, a diagonal matrix containing singular values, and a right singular vector, wherein the singular values in the diagonal matrix are arranged in descending order along the matrix diagonal of the diagonal matrix;

[0099] dividing the left singular vector into a first part left singular vector and a second part left singular vector, dividing the diagonal matrix into a first part diagonal matrix and a second part diagonal matrix, and dividing the right singular vector into a first part right singular vector and a second part right singular vector;

[0100] The first part parameters are obtained according to the first part left singular vector, the first part diagonal matrix and the first part right singular vector, and the second part parameters are obtained according to the second part left singular vector, the second part diagonal matrix and the second part right singular vector.

[0101] The first large model updated weight matrix is obtained by using a sum of the first part parameters and model adjustment parameters, wherein the model adjustment parameters are determined according to a first low-rank parameter matrix and a second low-rank parameter matrix, and the first low-rank parameter matrix and the second low-rank parameter matrix are initialized by the second part parameters.

[0102] Optionally, the processor is further configured to:

[0103] The singular value decomposition is performed on the original weight matrix of the first large model to obtain a left singular vector, a diagonal matrix containing singular values and a right singular vector, wherein the singular values in the diagonal matrix are arranged in descending order along the matrix diagonal line of the diagonal matrix.

[0104] The left singular vector is divided into a first part left singular vector and a second part left singular vector, the diagonal matrix is divided into a first part diagonal matrix and a second part diagonal matrix, and the right singular vector is divided into a first part right singular vector and a second part right singular vector.

[0105] The first part parameters are obtained according to the first part left singular vector, the first part diagonal matrix and the first part right singular vector, and the second part parameters are obtained according to the second part left singular vector, the second part diagonal matrix and the second part right singular vector.

[0106] The second part parameters are adjusted by using a dynamic allocation rank method to obtain adjusted second part parameters.

[0107] The first large model updated weight matrix is obtained by using the first part parameters and the adjusted second part parameters.

[0108] Optionally, the processor is further configured to:

[0109] The contribution value of each singular value in the diagonal matrix and the total sum of the contribution values of all singular values are obtained.

[0110] The contribution values of the singular values are added one by one from the largest singular value to obtain a contribution value cumulative sum.

[0111] if a ratio of a sum of contribution values corresponding to the kth singular value to the total sum is greater than or equal to a preset value, a first parameter r is obtained, wherein the first parameter r = m-k, m is a total number of singular values in the diagonal matrix and m is an integer greater than 0, k is an integer and 1≤k

[0112] The left singular vector is divided into the first part left singular vector and the second part left singular vector based on the first parameter r, the diagonal matrix is divided into the first part diagonal matrix and the second part diagonal matrix, and the right singular vector is divided into the first part right singular vector and the second part right singular vector.

[0113] In a sixth aspect, an embodiment of the present application provides a speech recognition text processing apparatus, including: a processor and a transceiver.

[0114] The processor is configured to:

[0115] obtain a speech recognition text;

[0116] perform post-processing on the speech recognition text by using a first large model, wherein the first large model is a large model trained by using the large model training method.

[0117] In a seventh aspect, an embodiment of the present application further provides a communication device, including: a memory, a processor, and a program stored in the memory and executable on the processor, and the processor implements steps of the large model training method or the speech recognition text processing method when executing the program.

[0118] In an eighth aspect, an embodiment of the present application further provides a readable storage medium, and the readable storage medium stores a program, and the program is executed by a processor to implement steps of the large model training method or the speech recognition text processing method.

[0119] In a ninth aspect, an embodiment of the present application further provides a computer program product, including computer instructions, and the computer instructions are executed by a processor to implement steps of the large model training method or the speech recognition text processing method.

[0120] In the embodiment of the present application, the text smoothing task, the text correction task, the punctuation restoration task, and the ITN task can be used to construct a text data set, and the text data set formed by the real text can be used for model training. At the same time, the task label can be added to the text data set, and the large model can be trained. Thus, in the embodiment of the present application, the text smoothing task, the text correction task, the punctuation restoration task, and the ITN task can be integrated in the same large model, and the task category and comprehensiveness are improved. At the same time, the large model can better process the text post-processing task through fine-tuning on the large model, so that the model output meets the requirements of the developer, and the accuracy of the text output by the large model is improved. BRIEF DESCRIPTION OF DRAWINGS

[0121] Figure 1 is a flowchart of the large model training method provided by the embodiment of the present application;

[0122] Figure 2 is a flowchart of the speech recognition text processing method provided by the embodiment of the present application;

[0123] Figure 3 is one of the structure diagrams of the large model training device provided by the embodiment of the present application;

[0124] Figure 4 is one of the structure diagrams of the speech recognition text processing device provided by the embodiment of the present application;

[0125] Figure 5 is another structure diagram of the large model training device provided by the embodiment of the present application;

[0126] Figure 6 is another structure diagram of the speech recognition text processing device provided by the embodiment of the present application. DETAILED DESCRIPTION

[0127] In the embodiment of the present application, the term "and / or" describes the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B, which means that there are three cases of A alone, A and B together, and B alone. The character " / " generally represents an "or" relationship between the associated objects before and after it.

[0128] In the embodiment of the present application, the term "a plurality of" means two or more, and other quantifiers are similar.

[0129] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of the present application.

[0130] Referring to Figure 1 , Figure 1 is a flowchart of the large model training method provided by the embodiments of the present application, as shown in Figure 1 , comprising the following steps:

[0131] Step 101, obtaining a text data set for model training, wherein the text data set comprises a first text data set and a second text data set, the first text data set comprises a constructed text smoothing task text data set, a text correction task text data set, a punctuation recovery task text data set, and an ITN task text data set, and the second text data set is a text data set artificially annotated in a real scene.

[0132] The embodiments of the present application are mainly aimed at the post-processing process of the ASR system recognition, including the text smoothing task, the text correction task, the punctuation recovery task, and the ITN task. Due to the problems commonly existing in spoken language expressions such as repetition, redundancy, correction, and excessive mood words, the text output by the ASR system will appear to be not smooth, such as "I, uh, I watched a movie today", which will affect the fluency and readability of the text. Therefore, the text smoothing task is introduced to remove the not smooth phenomenon in the text. Due to the recognition performance of ASR itself, there are error phenomena such as wrong words and missing words, especially on specific nouns or word groups, which affect the reading experience. Therefore, the text correction task is introduced to correct the wrong characters in the recognized text. Since the output of the ASR system is a continuous text string, the punctuation recovery task is introduced to punctuate the text and add appropriate punctuation marks. Since the numbers output by the ASR are Chinese numbers, such as "today is 2024 July 24", "the phone number is 100086", and the like, the long string of Chinese numbers is not intuitive when reading, which affects the reading efficiency. Therefore, the ITN task is introduced to convert the text related to numbers such as mathematical operations, telephone numbers, and unit measurements.

[0133] Based on the requirements of the above tasks, in the embodiments of the present application, in addition to using real text (for example, the text output by the ASR for different post-processing tasks) as training data, the corresponding text data set can be constructed for the above four tasks in combination with extensive and easily accessible unlabeled corpus, so as to obtain the text data set for model training. In the following, the construction process of the text data set for different post-processing tasks is described respectively.

[0134] 1. The text smoothing task text dataset is obtained by the following method:

[0135] In constructing the text smoothing task text dataset, three kinds of adding disturbances are mainly constructed by repetition, insertion and deletion.

[0136] The text smoothing task text dataset is obtained by processing the plurality of to-be-processed text data in one or more of the following manners:

[0137] 1.1 At least one character is selected from the to-be-processed text data, and the at least one character is added at the adjacent position of the at least one character to obtain the text smoothing task text dataset.

[0138] 1.2 The to-be-processed text data is segmented to obtain a plurality of segmented words, and at least one segmented word is selected from the plurality of segmented words, and the at least one segmented word is added at the adjacent position of the at least one segmented word.

[0139] The to-be-processed text data can be a plurality of sentences. The adjacent position can refer to the position adjacent to the at least one character or the at least one segmented word, which is before or after the at least one character or the at least one segmented word. In the embodiments of the present application, the method of segmentation is not limited.

[0140] The above 1.1 and 1.2 can simulate the possible repetition of the text in the speech recognition text by repeating the text. For example, 1 to 3 non-adjacent characters are randomly selected from the to-be-processed text data for repetition, such as “you can go to see the movie”, which is similar to the phenomenon of speaking stumbling or stuttering; one way is to randomly select 1 to 3 words for repetition after segmentation, such as “this movie is just really good to watch”.

[0141] 1.3 The to-be-processed text data is segmented to obtain a plurality of segmented words, and a mood pause word and / or a meaningless demonstrative word are added at one or more target positions, wherein the target position is the position between two adjacent segmented words.

[0142] Through 1.3, the mood pause word or the meaningless demonstrative word that can appear in the speech recognition text is simulated by inserting the text. Common mood words include “ah, oh, oh, oh, oh, oh” and the like, and common meaningless demonstrative words include “just say, what, this, that” and the like. For example, the to-be-processed text data can be segmented first, and a mood word or a meaningless word is randomly inserted at the position between the segmented words, such as the normal text “we can start now”, and the text after inserting the disturbance can be “uh we can start now just say”.

[0143] 1.4 Tokenize the text data to be processed to obtain a plurality of tokens, and delete one or more tokens in the plurality of tokens.

[0144] Through 1.4, the missing word phenomenon caused by the fuzzy pronunciation of voice utterance or the ASR recognition performance is simulated in the form of deleting text. First, the normal text is tokenized, and 1 to 3 words are randomly selected, and a character is randomly deleted from the word. In this way, the missing word situation is considered in the construction process of the training data set, so as to improve the accuracy and comprehensiveness of model training.

[0145] 2. The text correction task text data set is obtained in the following way:

[0146] Tokenize the text data to be processed to obtain a plurality of tokens. Then, replace the target token in the plurality of tokens with a near sound token of the target token, wherein the target token is one or more tokens in the plurality of tokens, and the near sound token is a homophone or a homophone of the target token.

[0147] In the text correction task, the main concern is homophonic or homophonous errors, as well as errors in specific entity names. Here, a homophone mapping table is first constructed, such as "niu-liu", "min-mi", etc. If the pinyin of two characters is the same (not including tone) or they are a pair of homophones in the homophone mapping table, it is considered that the two characters meet the homophonic or homophonous condition. In practical applications, text tokenization and word frequency statistics processing can be performed on large-scale open general text corpus (such as WuDao corpus), and top K (K≥1) word frequency words are selected to form a word library. Then traverse the word library, if the pinyin of the two words in the corresponding position meets the homophonic or homophonous condition, determine that the two words are near sound or homophone words, such as "niulang" and "liulang", and thus a near sound or homophone mapping table can be obtained. For text data to be processed (such as a normal sentence), tokenization is performed, 1 to 3 words are randomly selected, and the 1-3 words are replaced with the corresponding near sound words (when there is more than one near sound word, a random one is selected), to obtain a sentence after near sound error disturbance.

[0148] 3. The punctuation restoration task text data set is obtained in the following way:

[0149] Replace and / or delete the target punctuation symbol in the text data to be processed, except for comma, period, question mark and colon.

[0150] In the punctuation restoration data task, it is easier to obtain punctuated plain text data. In the post-processing of the ASR system, four punctuation marks, i.e., comma, period, question mark and semicolon, are generally predicted, which can achieve the purpose of breaking up sentences to help reading, and will not lead to high prediction error rate due to too many types of punctuation marks. Therefore, for a wide range of punctuated text corpus, it is necessary to perform preprocessing, retain comma, period, question mark and semicolon, and reasonably delete or replace the remaining punctuation marks to meet the requirements of ASR punctuation prediction training data.

[0151] 4. The ITN task text data set is obtained by the following method:

[0152] The second large model and the prompt are used to process the to-be-processed text data to obtain the ITN task text data set. The prompt includes indication information of the type of number to be converted. The second large model can be different from the first large model, and the indication information can be used to indicate whether the number needs to be converted, which type of number needs to be converted, and the conversion mode.

[0153] In actual application, the ITN task text data set can be generated by means of the prompt and the generation capability of the existing large model. In the prompt, the type of number to be converted is prompted and limited, and the output of the second large model is adjusted to meet the data construction requirements. Generally, the large model used for ITN data construction can be a large model with large parameters and good generation performance, so as to generate more reliable data.

[0154] In addition, in addition to constructing single-task data, multi-task combined training data can also be constructed according to needs, and the types of task combination are set according to actual needs.

[0155] Through the above description, data construction rules can be customized for different tasks, including multi-word, missing word, error correction data containing errors, ITN data and punctuation restoration data, so as to enrich the types of tasks and comprehensiveness.

[0156] Step 102, adding a task label to the text data set, wherein the task label is used to indicate at least two of the text smoothing task, the text error correction task, the punctuation restoration task and the ITN task.

[0157] The large language model takes text as input and output, that is, the input (input) and label (label) of the large model fine-tuning data are both text. The format of a single fine-tuning data is as follows:

[0158] {"conversations":[

[0159] {"from":"user","value":"input"},

[0160] {"from":"assistant","value":"label"}]}

[0161] Specifically, for the text smoothing task, the input is non-smooth text and the label is smooth text; for the text correction task, the input is text with errors and the label is text after correction; for the punctuation restoration task, the input is text without punctuation and the label is text after punctuation restoration; for the ITN task, the input is text before standardization and the label is text after number standardization.

[0162] To enable the large model to distinguish between the four tasks and to flexibly configure which post-processing operations to perform on a given input text, eight special tokens representing the four tasks were designed as input instructions to indicate whether the corresponding post-processing tasks were active. This setup only represents the task combination type and does not add any additional task hints.

[0163] The specific implementation of task tags is not limited here; for example, it can be used...<w_fluent> ,<wo_fluent> These represent the use / non-use of text smoothness, respectively.<w_correct> ,<wo_correct> These respectively represent the use / disuse of text correction, and can be used to...<w_punc> ,<wo_punc> Representing restoration with / without punctuation respectively, can be used<w_itn> ,<wo_itn> These represent the use / non-use of inverse text normalization. Specific token combinations for the four tasks are added to the input as indicative information to help the large model distinguish between different post-processing tasks and combinations thereof. For example, when training punctuation recovery and ITN tasks simultaneously, a training sample example is as follows:

[0164] {"conversations":[

[0165] {"from":"user","value":"Hello, please call 10086 for inquiries"}<wo_fluent><wo_correct><w_punc><w_itn> "},

[0166] {"from":"assistant","value":"Hello, please call 10086 for inquiries."}]}

[0167] By the above mode, the key processing tasks such as text smoothing, text error correction, punctuation recovery, ITN, etc. after speech recognition can be integrated in the same model to be realized. At the same time, by adding the token of the task identifier as the input instruction of multi-task learning, there is no need to design a complex task prompt word, and only a simple label instruction can specify the task category, so that the model can simply and flexibly support a single task or simultaneously support multiple task combinations, simplifying the post-processing process. At the same time, by adding the task label, the processing results of multiple tasks can be obtained when multiple tasks are executed at the same time, so that the speech recognition text post-processing does not need to be performed in the manner of multiple single tasks in cascade as in the prior art, thereby saving computing resources.

[0168] Through multi-task combination learning of multiple tasks, the model input is the text to be processed, and the output is directly the text after superposition of multiple task processing, so there is no need to perform various rule conversion and adjustment on the model output, so that the model does not need to excessively rely on preset features, and is more flexible and scalable.

[0169] Step 103, training the first large model by using the text data set with added task labels, wherein in the training process, the model parameters of the first part of the Transformer layers of the first large model remain unchanged, the model parameters of the second part of the Transformer layers of the first large model are adjusted, and the proportion of the number of layers of the second part of the Transformer layers to the total number of layers of the Transformer layers of the first large model is less than or equal to a preset value.

[0170] Since the parameter amount of the large model is large, the training and inference speed of the large model is slow, and considering that the fine-tuning task types in the scene to which the embodiments of the present application are applied are less, a large language model with a smaller parameter amount is selected as the first large model, such as the Qwen1.8B model. The input and output of the first large model are both texts, so there is no need to manually define and extract features.

[0171] The current large language model is mostly composed of multiple layers of Transformer layers. The front layers close to the input and the back layers close to the output have different effects during fine-tuning. The front layers focus on capturing shallow text features, such as the morphology of words and local relationships, and focus on basic information in the text. The back layers focus on capturing advanced semantic features and longer distance dependencies, and learn knowledge related to the domain or task. Because the large model itself has learned rich text representation capabilities, including word meaning, grammatical relationships, context, etc., here, only the back layers of the model can be fine-tuned. The back layers, i.e., the second part of the Transformer layers, can be understood as one or more layers that account for less than or equal to a preset proportion of the total number of Transformer layers in the first large model, and can or can not include the output layer. The preset value can be set as needed. In the first large model, the layers other than the second part of the Transformer layers can be considered as the first part of the Transformer layers.

[0172] For example, an hyperparameter α can be set to control how to determine the back layers that need to be fine-tuned. For example, for a large model composed of 32 layers of Transformer, set α = 0.5, i.e., fine-tune the back 16 layers of the large model, and keep the front 16 layers frozen. In this way, the rich representation capability of the base of the large model can be preserved, and the requirements specific to the text post-processing task can be adapted.

[0173] When selecting the fine-tuning method, the size of the data, the available resources, and the pursuit of performance are mainly considered. Here, a partial low-rank fine-tuning method based on singular value decomposition (SVD) is adopted to fine-tune the first large model. Compared with fine-tuning of all parameters of the model, the low-rank fine-tuning method based on singular value decomposition keeps the main parameter part of the original model fixed during fine-tuning, and only updates the low-rank matrix of the secondary parameter part. This method reduces the parameter amount of model training, has smaller requirements for computing resources and storage resources than full-parameter fine-tuning, and has faster training speed. Since the large language model itself has rich understanding and generation capabilities for text, only a small amount of adjustment is needed for the first large model during the text post-processing task fine-tuning process in the embodiments of the present application, and the effect of being able to generate corresponding output for specific task combinations can be achieved without destroying the basic knowledge ability of the large model. Since the low-rank fine-tuning method based on singular value decomposition only updates the secondary parameter part during fine-tuning, it allows the model to learn task-specific information without changing its general knowledge, and the adjustment amplitude of the original model is relatively small.

[0174] In the embodiments of the present application, based on the dynamic low-rank fine-tuning method of singular value decomposition, the original weight matrix W of the large model is singular value decomposed, the main part parameters and the secondary part parameters are obtained according to the adaptive rank division, the main part parameters are frozen, and the low-rank fine-tuning is performed on the secondary part parameters. The method can be mainly applied to the linear modules in each layer network of the large model, including Query, Key, Value in the attention mechanism, feed-forward layer and output project layer.

[0175] In the following, the training method in the embodiments of the present application is described in detail:

[0176] In one way, the following process can be included:

[0177] S1, singular value decomposition is performed on the original weight matrix of the first large model to obtain left singular vectors, a diagonal matrix containing singular values and right singular vectors, wherein the singular values in the diagonal matrix are arranged in descending order on the matrix diagonal line of the diagonal matrix.

[0178] For an original weight matrix singular value decomposition can be performed on it:

[0179] W=UΣV T

[0180] wherein, is the left singular vector; is the right singular vector; is a diagonal matrix containing singular values, wherein the singular values are arranged in descending order on the matrix diagonal line. Without loss of generality, in order to simplify the representation, it is assumed that m≤n.

[0181] S2, the left singular vectors are divided into first part left singular vectors and second part left singular vectors, the diagonal matrix is divided into first part diagonal matrix and second part diagonal matrix, and the right singular vectors are divided into first part right singular vectors and second part right singular vectors.

[0182] According to the corresponding singular value division, the left singular vector U can be divided into [U a ,U b ], wherein Σ and V are divided in the same way, wherein, Then the singular value decomposition of W can be expressed as:

[0183]

[0184] As can be seen from the above description, in this step, the left singular vectors are divided into the first part left singular vectors and the second part left singular vectors based on the first parameter r, the diagonal matrix is divided into the first part diagonal matrix and the second part diagonal matrix, and the right singular vectors are divided into the first part right singular vectors and the second part right singular vectors.

[0185] Since the optimal segmentation points of different weights on singular values are not necessarily the same, after the original weights are subjected to singular value decomposition calculation, for the descending singular value sequence in Σ, the most appropriate r is found by using the method of cumulative contribution rate, in the following specific manner:

[0186] (1) Obtain the contribution value of each singular value in the diagonal matrix and the sum of the contribution values of all singular values.

[0187] Suppose that each singular value in Σ is represented as σ i The contribution value of each singular value is defined as The total contribution value of each singular value is defined as

[0188] (2) Starting from the largest singular value, the contribution values of the singular values are added one by one to obtain the cumulative sum of the contribution values. If the ratio of the cumulative sum of the contribution values corresponding to the kth singular value to the sum is greater than or equal to a preset value, the first parameter r is obtained, where the first parameter r = m-k, m is the total number of singular values in the diagonal matrix and m is an integer greater than 0, and k is an integer and 1≤k

[0189] For example, starting from the largest singular value (i.e., the first singular value), the contribution values of the singular values are added one by one to obtain the cumulative sum of the contribution values. Suppose that when the kth singular value is added, the following condition is met, and it is considered that the rank under the current weight (i.e., the first parameter) r = m-k:

[0190]

[0191] Where θ represents the preset value.

[0192] S3, obtaining a first part parameter according to the first part left singular vector, the first part diagonal matrix and the first part right singular vector, and obtaining a second part parameter according to the second part left singular vector, the second part diagonal matrix and the second part right singular vector.

[0193] Since the singular values in Σ are arranged in descending order, the larger the singular value, the more important the feature in the corresponding direction. Therefore, it can be considered that the main parameter part (i.e., the first part parameter) in the above formula (1) is and the secondary parameter part (i.e., the second part parameter) is​

[0194] S4, obtaining a weight matrix of the first large model update by using a sum of the first partial parameter and a model adjustment parameter, wherein the model adjustment parameter is determined according to a first low-rank parameter matrix and a second low-rank parameter matrix, and the first low-rank parameter matrix and the second low-rank parameter matrix are initialized by the second partial parameter.

[0195] Here, the main parameter part W a is frozen, and two low-rank parameter matrices A and B (respectively, a first low-rank parameter matrix and a second low-rank parameter matrix, which can be obtained by performing low-rank decomposition on the original weight matrix) are added on the basis of the frozen main parameter part. Wherein A and B are initialized by the secondary partial parameter, and then W b can be represented as:

[0196]

[0197] Wherein BA is a model adjustment parameter. In the fine-tuning process, the original weight matrix is divided into a frozen main parameter part and a secondary partial parameter (i.e., a fine-tuning part). After training, the two parts can be combined into an updated weight matrix through weight merging. For example, the input of a certain layer in the training process is represented as x, and the output is represented as h. The original pre-training weight of the layer is W, W a is the main parameter part thereof, and two low-rank matrices are represented as A and B, respectively. Then, the update manner in the training process is:

[0198] h = W a x + BAx

[0199] The low-rank matrix weight is merged into the main parameter part weight, that is, the update manner is:

[0200] h = (W a + BA)x

[0201] W a + BA is the merged weight. After weight merging, the inference time delay is not increased during model inference. During the inference process, the default is to process all four tasks. If it is necessary to infer a single task or some combined tasks, a task instruction tag needs to be added during the inference.

[0202] In another way, the following process can be included:

[0203] S1, performing singular value decomposition on an original weight matrix of the first large model to obtain a left singular vector, a diagonal matrix containing singular values, and a right singular vector, wherein the singular values in the diagonal matrix are arranged in descending order on the matrix diagonal line of the diagonal matrix.

[0204] For an original weight matrix The singular value decomposition can be performed on it:

[0205] W = UΣV T

[0206] where, is the left singular vector; is the right singular vector; is a diagonal matrix containing singular values, where the singular values are arranged in descending order along the diagonal of the matrix. Without loss of generality, to simplify the representation, it is assumed that m ≤ n.

[0207] S2, dividing the left singular vector into a first part of left singular vectors and a second part of left singular vectors, dividing the diagonal matrix into a first part of diagonal matrices and a second part of diagonal matrices, and dividing the right singular vector into a first part of right singular vectors and a second part of right singular vectors.

[0208] According to the corresponding singular value division, the left singular vector U can be divided into [U a ,U b ], where Σ and V are divided in the same way, where The singular value decomposition of W can be expressed as:

[0209]

[0210] Since the optimal segmentation points of different weights on singular values are not necessarily the same, after the original weights are calculated by singular value decomposition, for the descending singular value sequence in Σ, the cumulative contribution rate method is used to find the most suitable r, which is as follows: as previously described.

[0211] S3, obtaining a first part of parameters according to the first part of left singular vectors, the first part of diagonal matrices, and the first part of right singular vectors, and obtaining a second part of parameters according to the second part of left singular vectors, the second part of diagonal matrices, and the second part of right singular vectors.

[0212] Since the singular values in Σ are arranged in descending order, the larger the singular value, the more important the feature in the corresponding direction. Therefore, in the above formula, is considered as the main parameter part (i.e., the first part of parameters), and is considered as the secondary parameter part (i.e., the second part of parameters).

[0213] S4, adjusting the second part of parameters using a dynamic rank allocation method to obtain an adjusted second part of parameters.

[0214] Here, in the secondary parameter part In the fine-tuning process, the importance of each parameter participating in fine-tuning to the training result is learned, and the ranks of the parameter matrices are dynamically adjusted according to the importance degree. For example, the fine-tuning method of the incremental PΛQ fine-tuning in AdaLora can be used to fine-tune the secondary parameter part The main parameter part W is kept a Freezing improves training efficiency while effectively preserving the knowledge of the pre-trained model.

[0215] S5, obtaining an updated weight matrix of the first large model by using the first part of parameters and the adjusted second part of parameters.

[0216] For example, the sum of the first part of parameters and the adjusted second part of parameters can be used as the updated weight matrix.

[0217] Through the above fine-tuning method, the generation ability specific to the speech recognition post-processing task can be learned while the rich understanding and generation ability of the large model itself is kept, so that the model output is more in line with the requirements of the developer.

[0218] In the embodiments of the present application, the text smoothing task, the text correction task, the punctuation restoration task and the ITN task can be used to construct a text data set, and the text data set formed by the real text can be combined to form a text data set for model training. At the same time, the text data set can be added with task labels and the large model can be trained. Therefore, in the embodiments of the present application, the text smoothing task, the text correction task, the punctuation restoration task and the ITN task can be integrated in the same large model, and the task category and comprehensiveness are improved. At the same time, through fine-tuning on the large model, the large model can better handle the text post-processing task, keep the strong ability of the large model itself, make the model output more in line with the requirements of the developer, and improve the accuracy of the output text of the large model.

[0219] Referring to Figure 2 , Figure 2 is a flowchart of a speech recognition text processing method provided by the embodiments of the present application, as Figure 2 shown, comprising the following steps:

[0220] Step 201, obtaining a speech recognition text.

[0221] The speech recognition text can be the text output by the ASR system.

[0222] Step 202, using a first large model to post-process the speech recognition text, wherein the first large model is a large model trained by using the large model training method described above.

[0223] In the post-processing process, through task combination learning of multiple tasks in the training process, the input of the first large model is speech recognition text, and the output is directly the text after multiple task processing superposition, so that various rule conversion and adjustment of the model output are not needed, so that the model does not rely too much on preset features, and has more flexibility and scalability.

[0224] Since the training of the large model maintains the strong ability of the large model itself while making the model output more in line with the requirements of the developer, the accuracy of the text output by the large model can be improved.

[0225] Referring to Figure 3 , Figure 3 is a structural diagram of a large model training device provided by an embodiment of the present application. As shown in Figure 3 , the device comprises:

[0226] The first acquisition module 301 is configured to acquire a text data set for model training, wherein the text data set comprises a first text data set and a second text data set, the first text data set comprises a text smoothing task text data set, a text correction task text data set, a punctuation recovery task text data set, and an ITN task text data set, and the second text data set is a text data set artificially annotated in a real scene. The first processing module 302 is configured to add a task label to the text data set, wherein the task label is used to indicate at least two of the following tasks: a text smoothing task, a text correction task, a punctuation recovery task, and an ITN task. The first training module 303 is configured to train a first large model by using the text data set with the added task label, wherein in the training process, the model parameters of a first part of the Transformer layers of the first large model remain unchanged, the model parameters of a second part of the Transformer layers of the first large model are adjusted, and the proportion of the number of layers of the second part of the Transformer layers to the total number of layers of the Transformer layers of the first large model is less than or equal to a preset value.

[0227] Optionally, the first acquisition module 301 is further configured to obtain the text smoothing task text data set in the following manner:

[0228] The following one or more of the following is performed on a plurality of to-be-processed text data to obtain the text smoothing task text data set:

[0229] At least one character is selected from the to-be-processed text data, and the at least one character is added at the adjacent position of the at least one character to obtain the text smoothing task text data set.

[0230] performing word segmentation on the to-be-processed text data to obtain a plurality of segmented words; selecting at least one segmented word from the plurality of segmented words, and adding the at least one segmented word at adjacent positions of the at least one segmented word;

[0231] performing word segmentation on the to-be-processed text data to obtain a plurality of segmented words; adding a mood pause word and / or a meaningless demonstrative word at one or more target positions, wherein the target positions are positions between adjacent two segmented words;

[0232] performing word segmentation on the to-be-processed text data to obtain a plurality of segmented words, and deleting one or more segmented words in the plurality of segmented words.

[0233] Optionally, the first obtaining module 301 is further configured to obtain the text correction task text data set in the following manner:

[0234] performing word segmentation on the to-be-processed text data to obtain a plurality of segmented words;

[0235] replacing a target segmented word in the plurality of segmented words with a near sound word of the target segmented word, wherein the target segmented word is one or more segmented words in the plurality of segmented words, and the near sound word is a homophone word or a homonym word of the target segmented word.

[0236] Optionally, the first obtaining module 301 is further configured to obtain the punctuation recovery task text data set in the following manner:

[0237] replacing and / or deleting a target punctuation symbol in the to-be-processed text data, except for a comma, a period, a question mark and a colon.

[0238] Optionally, the first obtaining module 301 is further configured to obtain the ITN task text data set in the following manner:

[0239] processing the to-be-processed text data by using a second large model and a prompt to obtain the ITN task text data set, wherein the prompt includes indication information about a type of number to be converted.

[0240] Optionally, the first training module 303 is further configured to:

[0241] performing singular value decomposition on an original weight matrix of the first large model to obtain a left singular vector, a diagonal matrix containing singular values, and a right singular vector, wherein the singular values in the diagonal matrix are arranged in descending order on a matrix diagonal line of the diagonal matrix;

[0242] the left singular vector is divided into a first part left singular vector and a second part left singular vector, the diagonal matrix is divided into a first part diagonal matrix and a second part diagonal matrix, and the right singular vector is divided into a first part right singular vector and a second part right singular vector;

[0243] a first part parameter is obtained according to the first part left singular vector, the first part diagonal matrix and the first part right singular vector, and a second part parameter is obtained according to the second part left singular vector, the second part diagonal matrix and the second part right singular vector;

[0244] a weight matrix of the first large model update is obtained by using a sum of the first part parameter and a model adjustment parameter, wherein the model adjustment parameter is determined according to a first low-rank parameter matrix and a second low-rank parameter matrix, and the first low-rank parameter matrix and the second low-rank parameter matrix are initialized by the second part parameter.

[0245] Optionally, the first training module 303 is further configured to:

[0246] singular value decomposition is performed on the original weight matrix of the first large model to obtain a left singular vector, a diagonal matrix containing singular values and a right singular vector, wherein the singular values in the diagonal matrix are arranged in descending order on the matrix diagonal line of the diagonal matrix;

[0247] the left singular vector is divided into a first part left singular vector and a second part left singular vector, the diagonal matrix is divided into a first part diagonal matrix and a second part diagonal matrix, and the right singular vector is divided into a first part right singular vector and a second part right singular vector;

[0248] a first part parameter is obtained according to the first part left singular vector, the first part diagonal matrix and the first part right singular vector, and a second part parameter is obtained according to the second part left singular vector, the second part diagonal matrix and the second part right singular vector;

[0249] the second part parameter is adjusted by using a dynamic rank allocation method to obtain an adjusted second part parameter;

[0250] an updated weight matrix of the first large model is obtained by using the first part parameter and the adjusted second part parameter.

[0251] Optionally, the apparatus can further include a second acquisition module configured to:

[0252] a contribution value of each singular value in the diagonal matrix and a total sum of contribution values of all singular values are acquired;

[0253] starting from the largest singular value, the contribution values of the singular values are accumulated one by one to obtain a contribution value accumulation sum;

[0254] if a ratio of the contribution value accumulation sum corresponding to the kth singular value to the total sum is greater than or equal to a preset value, a first parameter r is obtained, wherein the first parameter r = m-k, m is a total number of singular values in the diagonal matrix and m is an integer greater than 0, k is an integer and 1≤k

[0255] The first training module is further configured to divide the left singular vector into the first part left singular vector and the second part left singular vector, divide the diagonal matrix into the first part diagonal matrix and the second part diagonal matrix, and divide the right singular vector into the first part right singular vector and the second part right singular vector based on the first parameter r.

[0256] The apparatus provided in the embodiments of the present application can execute the method embodiments described above, and has similar implementation principles and technical effects, which will not be described here again in the embodiments.

[0257] Referring to Figure 4 , Figure 4 is a structural diagram of a speech recognition text processing apparatus provided by an embodiment of the present application. As shown in Figure 4 , the apparatus includes:

[0258] The first acquisition module 401 is configured to acquire a speech recognition text, and the first processing module 402 is configured to perform post-processing on the speech recognition text by using a first large model, wherein the first large model is a large model trained by using the large model training method.

[0259] The apparatus provided in the embodiments of the present application can execute the method embodiments described above, and has similar implementation principles and technical effects, which will not be described here again in the embodiments.

[0260] Referring to Figure 5 , Figure 5 is a structural diagram of a large model training apparatus provided by an embodiment of the present application. As shown in Figure 5 , the apparatus includes a processor 501 and a transceiver 502.

[0261] The processor 501 is configured to:

[0262] acquire a text data set for model training, wherein the text data set includes a first text data set and a second text data set, the first text data set includes a constructed text smoothing task text data set, a text correction task text data set, a punctuation recovery task text data set, and an ITN task text data set, and the second text data set is a text data set artificially annotated in a real scene.

[0263] adding a task label to the text dataset, wherein the task label is used to represent at least two of the text smoothing task, the text correction task, the punctuation restoration task, and the ITN task;

[0264] training the first large model using the text dataset with the task label, wherein during the training, model parameters of a first part of the Transformer layers of the first large model remain unchanged, and model parameters of a second part of the Transformer layers of the first large model are adjusted, and a proportion of the number of layers of the second part of the Transformer layers to the total number of layers of the Transformer layers of the first large model is less than or equal to a preset value.

[0265] Optionally, the processor 501 is further configured to obtain the text smoothing task text dataset in the following manner:

[0266] performing one or more of the following on the plurality of to-be-processed text data to obtain the text smoothing task text dataset:

[0267] selecting at least one character from the to-be-processed text data, and adding the at least one character at a position adjacent to the at least one character to obtain the text smoothing task text dataset;

[0268] performing word segmentation on the to-be-processed text data to obtain a plurality of segmented words, and selecting at least one segmented word from the plurality of segmented words and adding the at least one segmented word at a position adjacent to the at least one segmented word;

[0269] performing word segmentation on the to-be-processed text data to obtain a plurality of segmented words, and adding a mood pause word and / or a meaningless demonstrative word at one or more target positions, wherein the target positions are positions between adjacent two segmented words;

[0270] performing word segmentation on the to-be-processed text data to obtain a plurality of segmented words, and deleting one or more segmented words from the plurality of segmented words.

[0271] Optionally, the processor 501 is further configured to obtain the text correction task text dataset in the following manner:

[0272] performing word segmentation on the to-be-processed text data to obtain a plurality of segmented words;

[0273] replacing a target segmented word in the plurality of segmented words with a near sound word of the target segmented word, wherein the target segmented word is one or more segmented words in the plurality of segmented words, and the near sound word is a homophone of the target segmented word or a homophone of the target segmented word.

[0274] Optionally, the processor 501 is further configured to obtain the punctuation restoration task text data set in the following manner:

[0275] Replace and / or delete target punctuation symbols in the to-be-processed text data, except for commas, periods, question marks, and semicolons.

[0276] Optionally, the processor 501 is further configured to obtain the ITN task text data set in the following manner:

[0277] Process the to-be-processed text data using a second large model and a prompt, to obtain the ITN task text data set, wherein the prompt includes indication information about the type of numbers to be converted.

[0278] Optionally, the processor 501 is further configured to:

[0279] Perform singular value decomposition on the original weight matrix of the first large model, to obtain a left singular vector, a diagonal matrix containing singular values, and a right singular vector, wherein the singular values in the diagonal matrix are arranged in descending order along the matrix diagonal of the diagonal matrix;

[0280] Divide the left singular vector into a first-part left singular vector and a second-part left singular vector, divide the diagonal matrix into a first-part diagonal matrix and a second-part diagonal matrix, and divide the right singular vector into a first-part right singular vector and a second-part right singular vector;

[0281] Obtain a first-part parameter according to the first-part left singular vector, the first-part diagonal matrix, and the first-part right singular vector, and obtain a second-part parameter according to the second-part left singular vector, the second-part diagonal matrix, and the second-part right singular vector;

[0282] Obtain an updated weight matrix of the first large model using the sum of the first-part parameter and a model adjustment parameter, wherein the model adjustment parameter is determined according to a first low-rank parameter matrix and a second low-rank parameter matrix, and the first low-rank parameter matrix and the second low-rank parameter matrix are initialized by the second-part parameter.

[0283] Optionally, the processor 501 is further configured to:

[0284] Perform singular value decomposition on the original weight matrix of the first large model, to obtain a left singular vector, a diagonal matrix containing singular values, and a right singular vector, wherein the singular values in the diagonal matrix are arranged in descending order along the matrix diagonal of the diagonal matrix;

[0285] divide the left singular vectors into first part left singular vectors and second part left singular vectors, divide the diagonal matrix into first part diagonal matrix and second part diagonal matrix, and divide the right singular vectors into first part right singular vectors and second part right singular vectors;

[0286] obtain first part parameters according to the first part left singular vectors, the first part diagonal matrix and first part right singular vectors, and obtain second part parameters according to the second part left singular vectors, the second part diagonal matrix and second part right singular vectors;

[0287] adjust the second part parameters by using a dynamic allocation rank method to obtain adjusted second part parameters;

[0288] obtain an updated weight matrix of the first large model by using the first part parameters and the adjusted second part parameters.

[0289] Optionally, the processor 501 is further configured to:

[0290] obtain a contribution value of each singular value in the diagonal matrix and a sum of contribution values of all singular values;

[0291] start from the largest singular value, and accumulate the contribution values of the singular values one by one to obtain a contribution value accumulation sum;

[0292] if a ratio of a contribution value accumulation sum corresponding to the kth singular value to the sum is greater than or equal to a preset value, obtain first parameters r, wherein the first parameters r = m-k, m is the total number of singular values in the diagonal matrix and m is an integer greater than 0, k is an integer and 1≤k

[0293] divide the left singular vectors into the first part left singular vectors and the second part left singular vectors, divide the diagonal matrix into the first part diagonal matrix and the second part diagonal matrix, and divide the right singular vectors into the first part right singular vectors and the second part right singular vectors based on the first parameters r.

[0294] The apparatus provided in the embodiments of the present application can execute the method embodiments described above, and has similar implementation principles and technical effects, which will not be described here in detail.

[0295] Referring to Figure 6 , Figure 6 is a structural diagram of a speech recognition text processing apparatus provided by the embodiments of the present application. As shown in the figure, Figure 6 the apparatus includes a processor 601 and a transceiver 602;

[0296] The processor 601 is configured to:

[0297] obtaining speech recognition text;

[0298] post-processing the speech recognition text by using a first large model, wherein the first large model is a large model trained by using the large model training method.

[0299] The apparatus provided in the embodiments of the present application can execute the method embodiments described above, and has similar implementation principles and technical effects. Therefore, the apparatus will not be described here again.

[0300] It should be noted that the division of units in the embodiments of the present application is illustrative, and is only a logical function division. In actual implementation, another division manner can be used. In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0301] If the integrated unit is realized in the form of a software functional unit and sold or used as an independent product, the integrated unit can be stored in a processor-readable storage medium. Based on this understanding, the technical solutions of the present application, essentially or in part, or all or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to execute all or part of the steps of the methods described in the various embodiments of the present application. The foregoing storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, and various other media that can store program codes.

[0302] The embodiments of the present application provide a communication device, including a memory, a processor, and a program stored in the memory and executable on the processor; the processor is configured to read the program in the memory to implement the steps in the information processing method described above.

[0303] The embodiment of the application further provides a readable storage medium, and a program is stored on the readable storage medium. The program is executed by a processor to implement each process of the large model training method or the speech recognition text processing method embodiment, and the same technical effects can be achieved. To avoid repetition, details are not described herein. The readable storage medium can be any available medium or data storage device accessible by the processor, including but not limited to a magnetic memory (for example, a floppy disk, a hard disk, a magnetic tape, a magneto-optical disk (MO), etc.), an optical memory (for example, a CD, a DVD, a BD, a HVD, etc.), and a semiconductor memory (for example, a ROM, an EPROM, an EEPROM, a nonvolatile memory (NAND FLASH), a solid state disk (SSD)), etc.

[0304] The embodiment of the application further provides a computer program product, including computer instructions, which are executed by a processor to implement each process of the large model training method or the speech recognition text processing method embodiment, and the same technical effects can be achieved. To avoid repetition, details are not described herein.

[0305] It should be noted that in this paper, the term "including", "containing" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the statement "including a" does not exclude the presence of another identical element in the process, method, article or device including the element.

[0306] From the above description of the embodiments, those skilled in the art can clearly understand that the above embodiment method can be realized by software and necessary general hardware platform, of course, it can also be realized by hardware, but in many cases, the former is a better embodiment. According to such understanding, the technical solutions of the application can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, disk, optical disk), including a plurality of instructions for making a terminal (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) execute the method described in each embodiment of the application.

[0307] The embodiments of the application are described above in combination with the drawings, but the application is not limited to the above specific embodiments, which are only illustrative and not restrictive. Those skilled in the art can make many forms without departing from the purpose of the application and the scope protected by the claims under the guidance of the application.

Claims

1. A large model training method, characterized in that, The method comprises the following steps: obtaining a text data set for model training, wherein the text data set comprises a first text data set and a second text data set, the first text data set comprises a text smoothing task text data set, a text correction task text data set, a punctuation recovery task text data set, and an inverse text normalization (ITN) task text data set, and the second text data set is a text data set artificially annotated in a real scene; adding task labels to the text data set, wherein the task labels are used to indicate at least two of the following tasks: text smoothing, text correction, punctuation recovery, and ITN; training a first large model using the text data set with the task labels, wherein, during the training, the model parameters of a first part of the first large model remain unchanged, the model parameters of a second part of the first large model are adjusted, and the proportion of the number of layers of the second part of the first large model to the total number of layers of the first large model is less than or equal to a preset value.

2. The method of claim 1, wherein, The text smoothing task text data set is obtained by the following method: performing one or more of the following operations on a plurality of to-be-processed text data to obtain the text smoothing task text data set: selecting at least one character from the to-be-processed text data and adding the at least one character at the adjacent position of the at least one character to obtain the text smoothing task text data set; performing word segmentation on the to-be-processed text data to obtain a plurality of segmented words, selecting at least one segmented word from the plurality of segmented words, and adding the at least one segmented word at the adjacent position of the at least one segmented word; performing word segmentation on the to-be-processed text data to obtain a plurality of segmented words, and adding a mood pause word and / or a meaningless indicator word at one or more target positions, wherein the target positions are positions between adjacent segmented words; performing word segmentation on the to-be-processed text data to obtain a plurality of segmented words, and deleting one or more segmented words from the plurality of segmented words.

3. The method of claim 1, wherein, The text correction task text data set is obtained by the following method: performing word segmentation on the to-be-processed text data to obtain a plurality of segmented words; replacing a target segmented word in the plurality of segmented words with a near sound word of the target segmented word, wherein the target segmented word is one or more segmented words in the plurality of segmented words, and the near sound word is a homophone of the target segmented word or a homophone of the target segmented word.

4. The method of claim 1, wherein, The punctuation recovery task text data set is obtained by the following method: replacing and / or deleting a target punctuation symbol in the to-be-processed text data, except for a comma, a period, a question mark, and a colon.

5. The method of claim 1, wherein, The ITN task text data set is obtained by the following method: processing the to-be-processed text data using a second large model and a prompt to obtain the ITN task text data set, wherein the prompt includes indication information about the type of number to be converted.

6. The method of claim 1, wherein, The training of the first large model using the text data set with the task labels comprises: performing singular value decomposition on the original weight matrix of the first large model to obtain left singular vectors, a diagonal matrix containing singular values, and right singular vectors, wherein the singular values in the diagonal matrix are arranged in descending order along the matrix diagonal of the diagonal matrix; dividing the left singular vectors into first part left singular vectors and second part left singular vectors, dividing the diagonal matrix into first part diagonal matrix and second part diagonal matrix, and dividing the right singular vectors into first part right singular vectors and second part right singular vectors; obtaining first part parameters according to the first part left singular vectors, the first part diagonal matrix, and the first part right singular vectors, and obtaining second part parameters according to the second part left singular vectors, the second part diagonal matrix, and the second part right singular vectors; obtaining an updated weight matrix of the first large model by using the sum of the first part parameters and model adjustment parameters, wherein the model adjustment parameters are determined according to first low-rank parameter matrix and second low-rank parameter matrix, and the first low-rank parameter matrix and the second low-rank parameter matrix are initialized by the second part parameters.

7. The method of claim 1, wherein, The training of the first large model by using the text data set added with the task label comprises: performing singular value decomposition on the original weight matrix of the first large model to obtain left singular vectors, a diagonal matrix containing singular values, and right singular vectors, wherein the singular values in the diagonal matrix are arranged in descending order along the matrix diagonal of the diagonal matrix; dividing the left singular vectors into first part left singular vectors and second part left singular vectors, dividing the diagonal matrix into first part diagonal matrix and second part diagonal matrix, and dividing the right singular vectors into first part right singular vectors and second part right singular vectors; obtaining first part parameters according to the first part left singular vectors, the first part diagonal matrix, and the first part right singular vectors, and obtaining second part parameters according to the second part left singular vectors, the second part diagonal matrix, and the second part right singular vectors; adjusting the second part parameters by using a dynamic rank allocation method to obtain adjusted second part parameters; obtaining an updated weight matrix of the first large model by using the first part parameters and the adjusted second part parameters.

8. The method according to claim 6 or 7, characterized in that, The method further comprises: obtaining the contribution value of each singular value in the diagonal matrix and the sum of the contribution values of all singular values; starting from the largest singular value, the contribution values of the singular values are added one by one to obtain a contribution value cumulative sum; if the ratio of the contribution value cumulative sum corresponding to the kth singular value to the sum is greater than or equal to a preset value, a first parameter r is obtained, wherein the first parameter r = m-k, m is the total number of singular values in the diagonal matrix and m is an integer greater than 0, k is an integer and 1≤k The dividing the left singular vector into a first part left singular vector and a second part left singular vector, the dividing the diagonal matrix into a first part diagonal matrix and a second part diagonal matrix, and the dividing the right singular vector into a first part right singular vector and a second part right singular vector comprises: The dividing the left singular vector into the first part left singular vector and the second part left singular vector, the dividing the diagonal matrix into the first part diagonal matrix and the second part diagonal matrix, and the dividing the right singular vector into the first part right singular vector and the second part right singular vector are based on the first parameter r.

9. A speech recognition text processing method, characterized by, Comprise: Obtaining speech recognition text; Post-processing the speech recognition text by using a first large model, wherein the first large model is a large model trained by using the large model training method in any one of claims 1-8.

10. A large model training apparatus, comprising: Comprise: A first obtaining module is configured to obtain a text data set for model training, wherein the text data set comprises a first text data set and a second text data set, the first text data set comprises a text smoothing task text data set, a text correction task text data set, a punctuation recovery task text data set, and an ITN task text data set, and the second text data set is a text data set artificially annotated in a real scene. A first processing module is configured to add task labels to the text data set, wherein the task labels are used to indicate at least two of the following tasks: a text smoothing task, a text correction task, a punctuation recovery task, and an ITN task. A first training module is configured to train a first large model by using the text data set with the added task labels, wherein, during the training, model parameters of a first part of Transformer layers of the first large model remain unchanged, model parameters of a second part of Transformer layers of the first large model are adjusted, and a proportion of a number of layers of the second part of Transformer layers to a total number of layers of all Transformer layers of the first large model is less than or equal to a preset value.

11. A speech recognition text processing apparatus, characterized by comprising: Comprise: A first obtaining module is configured to obtain speech recognition text. A first processing module is configured to post-process the speech recognition text by using a first large model, wherein the first large model is a large model trained by using the large model training method in any one of claims 1-8.

12. A large model training apparatus, comprising: Comprise: A processor and a transceiver; The processor is configured to: Obtain a text data set for model training, wherein the text data set comprises a first text data set and a second text data set, the first text data set comprises a text smoothing task text data set, a text correction task text data set, a punctuation recovery task text data set, and an ITN task text data set, and the second text data set is a text data set artificially annotated in a real scene. Add task labels to the text data set, wherein the task labels are used to indicate at least two of the following tasks: a text smoothing task, a text correction task, a punctuation recovery task, and an ITN task. The first large model is trained by using a text data set added with a task label, wherein, in the training process, model parameters of a first part of a Transformer layer of the first large model remain unchanged, model parameters of a second part of the Transformer layer of the first large model are adjusted, and a proportion of a number of layers of the second part of the Transformer layer to a total number of layers of the first large model is less than or equal to a preset value.

13. A speech recognition text processing apparatus, characterized by comprising: Comprise: A processor and a transceiver; Wherein, the processor is used for: Obtaining speech recognition text; Using a first large model to post-process the speech recognition text, wherein the first large model is a large model trained by using the large model training method of any one of claims 1-8.

14. A communication device comprising: A memory, a processor, and a program stored in the memory and executable on the processor; characterized in that the processor is used to read the program in the memory to implement the steps in the large model training method of any one of claims 1 to 8; or implement the steps in the speech recognition text processing method of claim 9.

15. A computer readable storage medium for storing a program, characterized in that, The program is executed by the processor to implement the steps in the large model training method of any one of claims 1 to 8; or implement the steps in the speech recognition text processing method of claim 9.

16. A computer program product, characterised in that, Comprise computer instructions, which are executed by the processor to implement the steps in the large model training method of any one of claims 1 to 8; or implement the steps in the speech recognition text processing method of claim 9.

Images