Automatic speech recognition method and system based on large model

Abstract

The present invention belongs to the field of speech recognition and provides an automatic speech recognition method and system based on a large model. The method obtains speech data to be recognized and converts it into text data. The method uses a pre-trained language detection model to perform a rough screening on the text data converted from the speech data to be recognized to identify whether there are problems in the text data. If there are problems, the method uses the trained large model to perform a fine screening, performs in-depth semantic and structural analysis on the text data, and identifies the problem type to which the problem belongs. If it is a single problem type, the method calls a preset knowledge base and preset rules, and uses the trained large model to correct the problems in the text data. If it is a complex problem type, the method determines the correction priority through a policy priority matrix, sorts the complex problem types, and calls the preset knowledge base and preset rules according to the sorting priority, and uses the trained large model to sequentially correct the problems in the text data. The present invention improves the user experience.

Description

Technical Field

[0001] The present invention belongs to the field of speech recognition, and in particular relates to an automatic speech recognition method and system based on a large model. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] Automatic speech recognition is an important branch of natural language processing and can be seen everywhere in production and life.

[0004] However, user voice commands often contain strong colloquial characteristics (such as informal language, omissions, and simplifications) and regional characteristics (such as differences in logic and intonation between different dialects), and do not fully conform to standard language logic and grammar. Current automatic speech recognition systems often use voice data that cannot cover all dialects and accents when training their models. This results in significantly reduced recognition accuracy for some uncommon dialects or accents. For example, the system may not be able to accurately recognize some lesser-known dialects or the unique accents of specific regions.

[0005] On the other hand, people often use colloquial expressions, abbreviations, idioms, etc. in daily communication. These expressions often do not conform to standard grammatical and vocabulary rules. Users' voice commands may contain repeated information, missing information, invalid information, incorrect word order, unconventional expressions, pronunciation errors, and homophones. When processing these colloquial expressions, current automatic speech recognition systems lack an in-depth understanding of language habits and context, and it is difficult to accurately convert them into correct text. As a result, the computer is unable to effectively provide correct feedback based on user instructions, resulting in interaction failure or wrong decisions. Summary of the Invention

[0006] In order to solve the above problems, the present invention proposes an automatic speech recognition method and system based on a large model. The present invention optimizes the speech-to-text conversion process through intelligent correction technology, and can better understand and process complex speech input such as informal, omitted and dialectal speech, ensuring smoother and more effective interaction with users and improving user experience.

[0007] According to some embodiments, the present invention adopts the following technical solutions:

[0008] A large model-based automatic speech recognition method comprises the following steps:

[0009] Acquire the speech data to be recognized and pre-process the speech data to be recognized;

[0010] Extract information features of pre-processed speech data and convert it into text data;

[0011] Use the pre-trained language detection model to perform a rough screening of the text data converted from the speech data to identify whether there are any problems in the text data. If not, the process ends. Otherwise, use the trained large model for fine screening, conduct in-depth semantic and structural analysis of the text data, and identify the problem type to which the problem belongs. If it is a single problem type, call the preset knowledge base and preset rules, and use the trained large model to correct the problems in the text data;

[0012] If it is a complex problem type, the correction priority is determined through the policy priority matrix, and the complex problem types are sorted. According to the sorting priority, the preset knowledge base and preset rules are called, and the problems in the text data are corrected in sequence using the trained large model.

[0013] As an optional implementation manner, the problem types include repeated information, missing information, invalid information, word order errors, unconventional expressions, pronunciation errors and homophones.

[0014] As an optional implementation, the training process of the large model includes: obtaining text data converted from speech data with different problems as samples, and a corresponding labeling difficulty coefficient for each sample, and adjusting the proportion of sample data of the corresponding problem type according to the corresponding labeling difficulty coefficient of each sample to ensure that the proportion of samples with a labeling difficulty coefficient greater than a set value in the sample is greater than or equal to the set value;

[0015] Obtain the question type of each sample and the corrected text information, and build a corresponding knowledge base to store the corresponding corrected knowledge and rules;

[0016] Use samples, question types and corrected text information to build a training set to train and fine-tune the large model.

[0017] As a further implementation method, in the process of obtaining text data converted from speech data with different problems as samples, text data that is difficult to understand due to incomplete information is not selected. Based on the preliminary training results, the difficulty of the large model in identifying various types of problems is determined, and the proportion of sample data for the corresponding problem types is preliminarily determined, so that the sample includes text data for each type of problem, and the more difficult it is for the large model to identify a certain type of problem, the higher the proportion of sample data for this type of problem is within the set range.

[0018] As a further embodiment, in the process of obtaining text data converted from speech data with different problems as samples, the annotation difficulty coefficient is comprehensively determined based on the average single sentence annotation time, the inter-annotator agreement rate, and the expert review and modification rate. Based on the annotation difficulty coefficient, the process of adjusting the proportion of sample data of the corresponding problem type is as follows:

[0019] ;

[0020] in, For the i The adjusted proportion of the problem type, For the i The original proportion of class problems, For the i The difficulty coefficient of labeling class problems, is the average marking difficulty coefficient, is the intensity adjustment factor, is the conservation factor, where:

[0021] ;

[0022] in, To correct the adjustment amount, ensure that the proportion of data increase is equal to the proportion of data decrease;

[0023] And the adjusted sample data ratio is set with an upper limit for the single-category ratio.

[0024] As a further implementation method, before constructing the training set, all text data are converted into a unified encoding format and the text structure is ensured to be consistent.

[0025] As a further implementation method, the process of constructing a training set and training and fine-tuning the large model includes: shuffling the corrected data set and dividing it into a training set, a validation set, and a test set according to a set ratio; inputting the training set into the large model; combining it with pre-designed prompt words, which are used to instruct the large model to focus on features of different question types, and fine-tuning the large model to adapt to the corresponding question types;

[0026] During fine-tuning, a multi-task learning framework is established to enable the large model to learn and handle various types of problems at the same time. The large model is verified using the validation set based on the joint loss function. The loss weight of the joint loss function is dynamically adjusted according to the performance of the large model. After fine-tuning is completed, the model is evaluated using the test set to verify the model's performance on the corresponding task.

[0027] As a further embodiment, the large model includes a shared layer and a task layer, wherein the shared layer is used to extract common features of the text, and the task layer is used to correct subtasks for each type of problem;

[0028] Construct a dynamic task weighting mechanism to assign a loss to each task Introducing learnable parameters , to automatically balance task importance:

[0029] For detection tasks involving duplicate and invalid information, we use the class-balanced cross entropy loss to address the sample imbalance problem.

[0030] For generation tasks with missing information, incorrect word order, unconventional expressions, phonetic errors, and homophones, we use reinforcement learning to optimize the fluency of generated text by combining cross-entropy loss with BLEU reward signals.

[0031] The gradient projection algorithm is used to eliminate conflicts between tasks and ensure the stability of joint training.

[0032] As a further implementation, the joint loss function is as follows:

[0033]

[0034] in, is the total loss, is the task weight, is the loss of each subtask;

[0035] Perform multi-task alternating training on large models, training different tasks in batches, prioritizing tasks with high uncertainty. Then, perform joint training, in which all tasks participate in gradient calculations simultaneously, and optimize parameters uniformly through backpropagation.

[0036] The loss weight is dynamically adjusted according to the performance of the validation set. When the overall loss of the validation set does not decrease in consecutive rounds, the parameters of the shared layer used to extract common features of the text are frozen, and only the parameters of the task layer are fine-tuned.

[0037] As an optional implementation method, a policy priority matrix is ​​used to determine the correction priority, sort the complex problem types, and repair the problems in sequence according to the sorted priority. The process includes: designing a policy priority matrix based on the semantic impact and correction difficulty. In the policy priority matrix, the semantic impact and correction difficulty of each type of problem are comprehensively evaluated, and sorted according to the evaluation results. The problem type with higher semantic impact and greater correction difficulty has a higher priority.

[0038] As an optional implementation method, after using the trained large model to correct problems in the text data, editable tags with contextual anchors are inserted into the generated corrected text, the user's decision behavior on the editable tags is obtained, and it is synchronized to the knowledge base.

[0039] An automatic speech recognition system based on a large model, comprising:

[0040] A preprocessing module is configured to obtain speech data to be recognized and preprocess the speech data to be recognized;

[0041] a text extraction module configured to extract information features of the preprocessed speech data and convert it into text data;

[0042] The large model correction module is configured to use the pre-trained language detection model to perform a rough screening of the text data converted from the speech data to be recognized, and identify whether there are problems in the text data. If not, the process ends. Otherwise, the trained large model is used for fine screening, and the text data is subjected to in-depth semantic and structural analysis to identify the problem type. If it is a single problem type, the preset knowledge base and preset rules are called to use the trained large model to correct the problem in the text data.

[0043] If it is a complex problem type, the correction priority is determined through the policy priority matrix, and the complex problem types are sorted. According to the sorting priority, the preset knowledge base and preset rules are called, and the problems in the text data are corrected in sequence using the trained large model.

[0044] Compared with the prior art, the present invention has the following beneficial effects:

[0045] The present invention constructs a training set through historical data sets to train and fine-tune the large model. When constructing the training set, it specifically extracts seven types of problems, such as information duplication, information missing, information invalidity, word order errors, unconventional expressions, voice errors and homophones, as well as the corrected text information. This can solve common problems in speech recognition, thereby improving the accuracy and reliability of speech recognition. The intelligent correction technology optimizes the speech-to-text conversion process, enabling the system to better understand and process complex speech inputs such as informal, omitted and dialectal speech, ensuring smoother and more effective interaction with users and improving the user experience.

[0046] In the process of constructing the training set, the present invention comprehensively considers the average single sentence annotation time, the consistency rate between annotators and the expert review and modification rate, calculates the annotation difficulty coefficient, and adjusts the sample data proportion of the corresponding question type based on the annotation difficulty coefficient. This can provide the large model with more learning samples of high-difficulty questions to make up for its natural lack of learning difficulty, facilitate the large model to explore its internal laws, and at the same time set an upper limit on the single-class proportion to prevent excessive distortion of the data distribution from causing overfitting of the large model.

[0047] When generating the final revised text, the present invention inserts an editable tag with a contextual anchor at each key semantic node that has been automatically corrected, displays suspected problem points in comparison with the original text through a visual interface, records user decision-making behavior in real time, and synchronously updates the knowledge base feedback mechanism. This interactive design not only realizes the closed-loop verification of human-computer collaboration, but also feeds back model training through user behavior data, and combines incremental learning technology to realize dynamic update of model parameters without retraining the entire model; by continuously optimizing the granularity of problem identification and the accuracy of correction strategies, it ultimately realizes intelligent text correction with dynamic learning capabilities.

[0048] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0050] Figure 1 This is a schematic diagram of the speech information recognition and text data conversion process in an embodiment;

[0051] Figure 2 Schematic diagram of a large model correction process in an embodiment. DETAILED DESCRIPTION

[0052] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0053] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention belongs.

[0054] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0055] In the absence of conflict, the embodiments and features in the embodiments of this application can be combined with each other.

[0056] Example 1

[0057] A large model-based automatic speech recognition method comprises the following steps:

[0058] Acquire the speech data to be recognized and pre-process the speech data to be recognized;

[0059] Extract information features of pre-processed speech data and convert it into text data;

[0060] Use the pre-trained language detection model to perform a rough screening of the text data converted from the speech data to identify whether there are any problems in the text data. If not, the process ends. Otherwise, use the trained large model for fine screening, conduct in-depth semantic and structural analysis of the text data, and identify the problem type to which the problem belongs. If it is a single problem type, call the preset knowledge base and preset rules, and use the trained large model to correct the problems in the text data;

[0061] If it is a complex problem type, the correction priority is determined through the policy priority matrix, and the complex problem types are sorted. According to the sorting priority, the preset knowledge base and preset rules are called, and the problems in the text data are corrected in sequence using the trained large model.

[0062] First, let me explain, Figure 1 As shown, the process of obtaining speech data to be recognized, preprocessing the speech data to be recognized, extracting information features of the preprocessed speech data, and converting it into text data can use existing technologies.

[0063] In this embodiment, the process of preprocessing the speech data to be recognized includes noise reduction and enhancement of speech information, extracting information features from the preprocessed speech data, and converting it into text data. Existing acoustic models or deep learning models are used to describe the statistical features of the speech information, and existing language models are used to predict word sequence probabilities. Combining the results of these two methods, a search algorithm is used to search for the optimal word sequence, thereby obtaining text data. The above processes all use existing technologies and are not described in detail here.

[0064] After obtaining preliminary text data, such as Figure 2 As shown, text correction is performed in the trained large model, and the corrected text information is then input into the downstream task to complete subsequent instructions.

[0065] In this embodiment, the specific process is to use the pre-trained language detection model to perform a rough screening on the text data converted from the speech data to be recognized, and identify whether there is a problem in the text data. If not, the process ends. Otherwise, the trained large model is used for fine screening to perform in-depth semantic and structural analysis on the text data to identify the problem type. If it is a single problem type, the preset knowledge base and preset rules are called, and the trained large model is used to correct the problem in the text data.

[0066] If it is a complex problem type, the correction priority is determined through the policy priority matrix, and the complex problem types are sorted. According to the sorting priority, the preset knowledge base and preset rules are called, and the problems in the text data are corrected in sequence using the trained large model.

[0067] The details of the whole process are introduced below.

[0068] In this embodiment, a large amount of voice data is first obtained and converted into text data using speech recognition technology. The text data set contains seven types of language problems: repeated information, missing information, invalid information, incorrect word order, unconventional expressions, phonetic errors, and homophones.

[0069] In order to make the present solution more clear to those skilled in the art, each speech problem is described with an example.

[0070] (1) Information duplication. For example, the voice message is “I went to the supermarket yesterday and bought beef, eggs, beer, apples, beef, and rice.” When describing specific items, due to the large variety of items, the user may have short-term forgetfulness during the description, so “beef” is mentioned twice. The normal sentence is: “I went to the supermarket yesterday and bought beef, eggs, beer, apples, and rice.”

[0071] 2. Missing information. For example, the voice message "Good morning, have you eaten?" lacks a subject, that is, a specific person. However, based on the context, it can be inferred that the normal sentence is: "Good morning, have you eaten?"

[0072] 3. Invalid information. For example, if the voice message is "Um, have you bought your ticket home?", "Um," is invalid information and will not have any impact on downstream tasks in speech recognition. A normal sentence is: "Have you bought your ticket home?"

[0073] 4. Incorrect word order. For example, the voice message is "Will you go to the library with me?" This reverses the standard "subject, predicate, and object" word order in Chinese. The normal sentence is: "Will you go to the library with me?"

[0074] 5. Unconventional expressions. For example, a voice message might read, "Please don't slack off at work." "Slack off" is an online slang term that actually means "to be lazy." The correct sentence is, "Please don't slack off at work."

[0075] 6. Voice errors. For example, the voice message reads, "Tomorrow I'm going to drive to Zhengzhou, the Netherlands." In southern dialects, "Henan" and "Holland" are pronounced similarly. Considering the context, the correct sentence here should be, "Tomorrow I'm going to drive to Zhengzhou, Henan."

[0076] 7. Homophones with different characters. For example, the voice message reads, "I bought a pound of loquats. Do you want to eat some?" "Pipa" and "loquat" have the same pronunciation but different meanings. Based on the context, the correct sentence here should be, "I bought a pound of loquats. Do you want to eat some?"

[0077] In this embodiment, the acquired voice data can come from a public data set or be collected through authorized partners and users. It should be noted that in order to improve the learning efficiency of the model, when fine-tuning the model, the present invention does not select sentences that are difficult to understand due to incomplete information. According to the difficulty of distinguishing the above seven types of language problems, the data proportion of the corresponding category is adjusted. If a certain type of language problem is not easy to distinguish, the data proportion is appropriately increased to facilitate the large model to mine its internal laws. Then, all text data is converted into a unified encoding format, and the text structure is ensured to be consistent for easy subsequent processing.

[0078] When constructing the training dataset, to ensure that the model can effectively learn and evenly handle all seven types of language problems, this embodiment adopts a dynamic data ratio adjustment strategy based on annotation difficulty assessment and model learning effect feedback.

[0079] The specific implementation plan is as follows:

[0080] During the data annotation stage, the difficulty of initial identification and correction of each type of language problem is quantified through statistical analysis of key indicators of the annotation team's performance of the task, including the average single-sentence annotation time (longer the time, the higher the difficulty), the agreement rate between annotators (lower the agreement rate, the more ambiguous the problem boundaries or the more subjective it is, and the higher the difficulty), and the expert review and modification rate (higher the modification rate, the more challenging it is, the more challenging it is, the more challenging it is, and the more difficult ...

[0081] Based on the above indicators, a quantitative labeling difficulty coefficient is calculated for each of the seven types of questions, which is a natural number ranging from 1 to 10. The larger the value, the higher the difficulty, so as to objectively distinguish the difficulty of each type of question.

[0082] There are many ways to quantify this, such as using a weighted approach. This involves assigning different weights to the average sentence annotation time, inter-annotator agreement, and expert review and revision rate. The calculated score is then multiplied by the corresponding weight to obtain the final result, the annotation difficulty coefficient. This process can utilize existing technical solutions, and we will not elaborate on them here.

[0083] Based on the calculated "labeling difficulty coefficient", the proportion of each type of question in the original data set is actively adjusted: Based on the natural distribution ratio of the data, in this embodiment, the proportion of samples of high-difficulty question types can be increased according to the difficulty coefficient according to the following formula, and the proportion of samples of low-difficulty question types can be correspondingly reduced:

[0084] ;

[0085] in, For the i The adjusted proportion of the problem type, For the i The original proportion of class problems, For the i The difficulty coefficient of labeling class problems, is the average marking difficulty coefficient, is the intensity adjustment factor, is the conservation factor, where:

[0086] ;

[0087] in, To correct the adjustment amount, ensure that the proportion of data increase is equal to the proportion of data decrease;

[0088] And the adjusted sample data ratio is set with an upper limit for the single-category ratio.

[0089] This strategy aims to provide the model with more learning samples of high-difficulty problems to make up for its inherent difficulty in learning, so as to facilitate the model to explore its internal laws. At the same time, it sets an upper limit on the proportion of a single class. For example, the increase in the most difficult samples does not exceed 30%, and the part exceeding 30% is inherited by the sample category with the lowest difficulty, so as to prevent excessive distortion of the data distribution and cause overfitting of the model.

[0090] On the other hand, during the fine-tuning and evaluation phase of the large model, this embodiment continuously monitors the large model's independent detection accuracy and correction effect on various types of problems on the validation set and test set. If it is found that the performance of the large model for a certain type of problem with higher initial difficulty is significantly lower than that of other categories or a preset threshold, it indicates that the model learning is still insufficient and the proportion of samples of this type of problem needs to be further increased in subsequent training rounds or supplementary data. Conversely, if the performance of a certain type of problem has reached saturation and is significantly better than that of other types, its proportion can be appropriately fine-tuned. This dynamic adjustment is based on feedback from the model's actual learning bottlenecks and is more targeted, aiming to achieve balanced optimization of the large model's capabilities.

[0091] This example also requires the pre-assembly of an annotation team comprised of linguists, data scientists, and industry experts. A set of rules is established to define the annotation standards and examples for each type of problem. The annotation team then manually annotates the dataset, classifying data that meets the seven language problem categories using a predetermined method. They then manually correct each data set, recording the corrected text information and the language problem type number.

[0092] The predetermined method can define the data label sequence number, original text, and revised text, and use different labels to represent the corresponding language problem types in order, that is, if the original text contains a certain type of language problem, the label of this type is assigned a value of 1, otherwise it is 0.

[0093] The above is the pre-training process.

[0094] During specific execution, the voice data to be recognized is obtained and preprocessed; the information features of the preprocessed voice data are extracted and converted into text data. For the text data, the existing pre-trained language model, such as the Bert model, is first used to detect and classify the text data. The scale of the text data can be reduced as needed to the extent of identifying whether there is a problem with the text.

[0095] The pre-trained language model is trained on a binary classification task using the constructed dataset, i.e., “there is a problem” or “there is no problem”. The input of the pre-trained language model is an ASR text fragment, and the output is a preliminary judgment on whether there is a problem.

[0096] Next, make corrections using the large model trained / fine-tuned on the dataset.

[0097] A general large model is chosen instead of an ordinary pre-trained model in the fine screening stage because the general large model already has excellent natural language understanding capabilities. In the fine-tuning stage, the model can be directly professionally trained to quickly improve its ability to understand the above seven types of language problems.

[0098] In this embodiment, a large language model that can be fine-tuned is selected, such as a general large model such as Qwen or GLM, and the order of the corrected data set is shuffled and divided into a training set, a validation set, and a test set in a ratio of 8:1:1.

[0099] The training set is fed into the selected large model, and the model is fine-tuned using the designed prompts to adapt to specific question types. During fine-tuning, a multi-task learning framework is established, allowing the large model to simultaneously learn to handle multiple error types, rather than handling each type separately. For the seven types of language problems after speech-to-text conversion, instead of training a separate correction model for each, a unified multi-task model is designed to simultaneously learn to detect and correct all types of errors.

[0100] This technical solution allows multi-task models to jointly correct for multiple types of errors that occur simultaneously through shared contextual understanding. Furthermore, shared parameters constrain model complexity, preventing over-reliance on specific data features for a single task and thus overfitting the model. After fine-tuning, the model is evaluated using a test set to verify its performance on specific tasks.

[0101] The following is the design idea of ​​model training:

[0102] The model architecture is divided into two layers: a shared layer (used to extract common features of text) and a task layer (used to perform each subtask, with an independent output layer designed for each subtask). Weights are then assigned based on task importance or the proportion of training data to balance the optimization objectives of different tasks.

[0103] Since the learning difficulty and convergence speed of the seven types of problems vary significantly, such as "information missing" requires contextual reasoning, "homophones" rely on local features, etc., fixed weights will cause the model to over-focus on simple tasks and ignore complex tasks. Therefore, this embodiment adopts dynamic weighted multi-task loss + task customized loss function + regularization strategy to build a dynamic task weighting mechanism, and set a loss function for each task. Introducing learnable parameters , automatically balance task importance.

[0104] Specifically, for detection tasks (such as those dealing with duplicate or invalid information), a class-balanced cross-entropy loss is used to address sample imbalance. For generation tasks (such as those dealing with missing information, incorrect word order, unconventional expressions, phonetic errors, and homophones), a cross-entropy loss is combined with a BLEU reward signal to optimize the fluency of generated text through reinforcement learning. Furthermore, a gradient projection algorithm is used to eliminate conflicts between tasks and ensure the stability of joint training. The joint loss function is as follows:

[0105] ;

[0106] Among them, is the total loss, is the task weight, and are the losses of each subtask.

[0107] Subsequently, multi-task alternating training is performed on the large model, training different tasks in rotation by batch, and preferentially learning tasks with high uncertainty; then joint training is carried out, allowing all tasks to participate in the gradient calculation simultaneously, and uniformly optimizing the parameters through backpropagation); finally, the loss weights are dynamically adjusted according to the performance of the validation set. When the overall loss of the validation set does not decrease for 3 consecutive rounds, the parameters of the shared layer are frozen, and only the task layer is fine-tuned.

[0108] Finally, independent evaluation metrics (such as statistical parameters like accuracy, F1 value, etc.) are defined for each subtask, and the overall correction effect is evaluated simultaneously.

[0109] By constructing correct prompt words, the large model is required to deeply analyze the semantics, context, grammatical structure, and common expression patterns of the input text to identify whether one or more of the above seven types of problems exist. For example, the prompt words will instruct the model to pay attention to features such as repetitive information, missing subject / object, redundant words, abnormal word order, non-standard expressions, words with easily confused pronunciations, and homophones.

[0110] Based on the degree of semantic impact and the difficulty of correction, a fixed priority matrix is designed, where the smaller the number, the higher the priority, as shown in Table 1:

[0111] Table 1 Language Problem Priority Matrix

[0112]

[0113] It should be noted that this priority can be flexibly adjusted according to the requirements of speech recognition. This embodiment only gives a setting principle and example, and is not limited thereto. For example, in Table 1, both information repetition and information invalidity are low in terms of semantic impact degree and correction difficulty. However, according to the requirements of speech recognition, the user may want to delete the repetitive information more and consider that colloquial languages such as "that thing" have relatively less impact. Then, the user can design the priority matrix by themselves and set the priority of information repetition higher than that of information invalidity.

[0114] According to the above priority matrix, the previously identified problem types are sorted from high to low according to the priority, and then handed over to the large model for text correction in sequence.

[0115] Corresponding correction knowledge bases and rules are constructed for various types of problems. According to the detected problem types (single or combined), the large model will call the repair methods for storing various problems in priority order and combine its own capabilities to correct the problems. The following is an example for each type of problem:

[0116] Information repetition: First, the large model breaks down the sentence, identifies core semantic units such as subject, predicate, and object, and detects consecutive repetitions. For non-consecutive repetitions, the large model combines word vectors to calculate the cosine similarity of cross-unit semantics. In this embodiment, when the cosine similarity is greater than 0.85, it is determined to be synonymous repetition. The large model also verifies the contextual coherence after deleting the repeated units. Finally, the first valid unit is retained, and a standard sentence structure that conforms to linguistic habits is reconstructed.

[0117] Missing information: First, use the big model to locate the component type of the missing information, then compare the context, use the big model to analyze the potential referents based on semantic coherence, and complete the text.

[0118] Invalid information: The fine-tuned large model is used to reconstruct the text in combination with the context, and the original sentence containing invalid information is converted into a more concise and standardized expression.

[0119] Word order errors: By defining standard grammatical rules such as subject-verb-object structure and time / place adverbial position, a rule library can be established, and the word order can be adjusted using a large model.

[0120] Unconventional expressions: Collect internet buzzwords, slang, industry terms, and their corresponding standard semantic interpretations (such as "moyu = being lazy" and "driving the night car = staying up late"), and convert them into conventional expressions through a large model.

[0121] Speech errors: We sort out the correct words corresponding to easily confused pronunciations in various dialects (such as "Holland" and "Henan", "older" and "company commander"), and use the semantic logical reasoning capabilities of the large model to perform global verification of recognition results to improve recognition efficiency.

[0122] Homophones with different characters: The large model has the ability to understand semantics across sentences and can reason based on the logical relationship between the previous and next context. When it identifies questions in a certain field, it gives priority to choosing words that are suitable for the scenario in that field.

[0123] When generating the final revised text, in this embodiment, an editable mark with a context anchor is inserted at each key semantic node that has been automatically corrected, such as color labeling, underline marking or interactive pop-up window. The form is not limited here, and the suspected problem points are compared with the original text through a visual interface.

[0124] Users can perform actions such as "Accept," "Reject," and "Customize" on these marked areas. This embodiment records user decision-making behavior in real time and simultaneously updates the knowledge base feedback mechanism. This interactive design not only enables closed-loop verification of human-machine collaboration but also feeds back model training through user behavior data. Combined with incremental learning techniques, it enables dynamic updating of large model parameters without retraining the entire model. By continuously optimizing the granularity of problem identification and the accuracy of correction strategies, intelligent text correction with dynamic learning capabilities is ultimately achieved.

[0125] Example 2

[0126] An automatic speech recognition system based on a large model, comprising:

[0127] A preprocessing module is configured to obtain speech data to be recognized and preprocess the speech data to be recognized;

[0128] a text extraction module configured to extract information features of the preprocessed speech data and convert it into text data;

[0129] The large model correction module is configured to use the pre-trained language detection model to perform a rough screening of the text data converted from the speech data to be recognized, and identify whether there are problems in the text data. If not, the process ends. Otherwise, the trained large model is used for fine screening, and the text data is subjected to in-depth semantic and structural analysis to identify the problem type. If it is a single problem type, the preset knowledge base and preset rules are called to use the trained large model to correct the problem in the text data.

[0130] If it is a complex problem type, the correction priority is determined through the policy priority matrix, and the complex problem types are sorted. According to the sorting priority, the preset knowledge base and preset rules are called, and the problems in the text data are corrected in sequence using the trained large model.

[0131] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of one or more computer-usable storage media (including but not limited to disk storage, CD - ROM , optical storage, etc.).

[0132] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0133] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0134] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the function specified in one or more boxes.

[0135] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made by those skilled in the art that fall within the spirit and principles of the present invention and do not require creative effort are intended to be within the scope of protection of the present invention.

Claims

1. A large model-based automatic speech recognition method, characterized in that: The following steps are involved: Acquire the speech data to be recognized and pre-process the speech data to be recognized; Extract information features of pre-processed speech data and convert it into text data; Use the pre-trained language detection model to perform a rough screening of the text data converted from the speech data to identify whether there are any problems in the text data. If not, the process ends. Otherwise, use the trained large model for fine screening, conduct in-depth semantic and structural analysis of the text data, and identify the problem type to which the problem belongs. If it is a single problem type, call the preset knowledge base and preset rules, and use the trained large model to correct the problems in the text data; The training process of the large model includes: obtaining text data converted from speech data with different problems as samples, and the labeling difficulty coefficient corresponding to each sample, and adjusting the proportion of sample data of the corresponding problem type according to the labeling difficulty coefficient corresponding to each sample to ensure that the proportion of samples with a labeling difficulty coefficient greater than a set value in the sample is greater than or equal to the set value; In the process of obtaining text data converted from speech data with different problems as samples, the annotation difficulty coefficient is comprehensively determined based on the average single sentence annotation time, the inter-annotator agreement rate, and the expert review and modification rate. Based on the annotation difficulty coefficient, the process of adjusting the sample data proportion of the corresponding problem type is as follows: ； in, For the i The adjusted proportion of the problem type, For the i The original proportion of class problems, For the i The difficulty coefficient of labeling class problems, is the average marking difficulty coefficient, is the intensity adjustment factor, is the conservation factor, where: ； in, To correct the adjustment amount, ensure that the proportion of data increase is equal to the proportion of data decrease; And the adjusted sample data ratio is set with a single-class ratio upper limit; If it is a complex problem type, the correction priority is determined through the policy priority matrix, and the complex problem types are sorted. According to the sorting priority, the preset knowledge base and preset rules are called, and the problems in the text data are corrected in sequence using the trained large model; The process of determining the correction priority through the policy priority matrix, sorting the complex problem types, and repairing the problems in sequence according to the sorted priority includes: designing a policy priority matrix based on the semantic impact degree and correction difficulty, in which the semantic impact degree and correction difficulty of various types of problems are comprehensively evaluated, and sorted according to the evaluation results. The problem type with higher semantic impact and greater correction difficulty has a higher priority.

2. The large model-based automatic speech recognition method according to claim 1, wherein: The problem types include repeated information, missing information, invalid information, word order errors, unconventional expressions, pronunciation errors and homophones.

3. The large model-based automatic speech recognition method according to claim 1, wherein: The training process of the large model also includes: obtaining the question type of each sample and the corrected text information, and building a corresponding knowledge base to store the corresponding corrected knowledge and rules; Use samples, question types and corrected text information to build a training set to train and fine-tune the large model.

4. The large model-based automatic speech recognition method according to claim 3, wherein: In the process of obtaining text data converted from speech data with different problems as samples, text data that is difficult to understand due to incomplete information is not selected. Based on the preliminary training results, the difficulty of the large model in identifying various types of problems is determined, and the proportion of sample data for the corresponding problem types is preliminarily determined, so that the sample includes text data for each type of problem. The more difficult it is for the large model to identify a certain type of problem, the higher the proportion of sample data for this type of problem is within the set range.

5. The large model-based automatic speech recognition method according to claim 3, wherein: The process of constructing a training set and training and fine-tuning the large model includes: shuffling the corrected dataset and dividing it into a training set, a validation set, and a test set according to a set ratio; inputting the training set into the large model; and fine-tuning the large model to suit the corresponding question type by combining it with pre-designed prompt words that instruct the large model to focus on features of different question types. During fine-tuning, a multi-task learning framework is established to enable the large model to learn and handle various types of problems at the same time. The large model is verified using the validation set based on the joint loss function. The loss weight of the joint loss function is dynamically adjusted according to the performance of the large model. After fine-tuning is completed, the model is evaluated using the test set to verify the model's performance on the corresponding task.

6. The large model-based automatic speech recognition method according to claim 1 or 3, wherein: The large model includes a shared layer and a task layer, wherein the shared layer is used to extract common features of the text, and the task layer is used to correct the subtasks of each type of problem; Construct a dynamic task weighting mechanism to assign a loss to each task Introducing learnable parameters , to automatically balance task importance: For detection tasks involving duplicate and invalid information, we use the class-balanced cross entropy loss to address the sample imbalance problem. For generation tasks with missing information, incorrect word order, unconventional expressions, phonetic errors, and homophones, we use reinforcement learning to optimize the fluency of generated text by combining cross-entropy loss with BLEU reward signals. Adopting gradient projection algorithm to eliminate conflicts between tasks and ensure the stability of joint training; The joint loss function is as follows: in, is the total loss, is the task weight, is the loss of each subtask; Perform multi-task alternating training on large models, training different tasks in batches, prioritizing tasks with high uncertainty. Then, perform joint training, in which all tasks participate in gradient calculations simultaneously, and optimize parameters uniformly through backpropagation. The loss weight is dynamically adjusted according to the performance of the validation set. When the overall loss of the validation set does not decrease in consecutive rounds, the parameters of the shared layer used to extract common features of the text are frozen, and only the parameters of the task layer are fine-tuned.

7. The large model-based automatic speech recognition method according to claim 1, wherein: After using the trained large model to correct problems in the text data, editable tags with contextual anchors are inserted into the generated corrected text, and the user's decision-making behavior on the editable tags is obtained and synchronized to the knowledge base.

8. A large-model-based automatic speech recognition system, using the large-model-based automatic speech recognition method according to any one of claims 1 to 7, characterized in that: include: A preprocessing module is configured to obtain speech data to be recognized and preprocess the speech data to be recognized; a text extraction module configured to extract information features of the preprocessed speech data and convert it into text data; The large model correction module is configured to use the pre-trained language detection model to perform a rough screening of the text data converted from the speech data to be recognized, and identify whether there are problems in the text data. If not, the process ends. Otherwise, the trained large model is used for fine screening, and the text data is subjected to in-depth semantic and structural analysis to identify the problem type. If it is a single problem type, the preset knowledge base and preset rules are called to use the trained large model to correct the problem in the text data. If it is a complex problem type, the correction priority is determined through the policy priority matrix, and the complex problem types are sorted. According to the sorting priority, the preset knowledge base and preset rules are called, and the problems in the text data are corrected in sequence using the trained large model.

Images