A method, system and storage medium for reducing semantic recognition computation

Abstract

This invention relates to the field of natural language processing, and particularly to a method, system, and storage medium for reducing the computational cost of semantic recognition. The method for reducing the computational cost of semantic recognition includes the following steps: obtaining a pre-trained model; adding training optimization objectives to the pre-trained model to obtain a final trained model; inputting the text to be recognized into the final trained model; and outputting the semantics of the text to be recognized. By adding training optimization objectives, the accuracy and robustness of the training can be improved without changing the model's structure, while also reducing the model's complexity and computational cost, thus solving the problem of excessive computational cost in existing models.

Description

[Technical Field]

[0001] This invention relates to the field of natural language processing, and in particular to a method, system, and storage medium for reducing the computational load of semantic recognition. [Background Technology]

[0002] In recent years, pre-trained models have become a new paradigm in the field of Natural Language Processing (NLP). Pre-trained models are an application of transfer learning, which trains a task-independent pre-trained model from large-scale data through self-supervised learning. Utilizing virtually unlimited text, it learns context-dependent representations of each word unit in the input sentence, thereby implicitly learning general syntactic and semantic knowledge, and can be fine-tuned for any downstream task.

[0003] Discriminative pre-trained language models, represented by BERT, employ a mask-based pre-training method (MLM), also known as a denoising strategy. The process is divided into two stages. First, training samples are constructed by using ennoising methods, such as masking a certain percentage of words in a sentence. Then, the language model is trained to reconstruct the corrupted sentence.

[0004] Because random destruction methods are typically used, the difficulty of predicting samples varies, leading to a lack of consistency between the gradient update strength and sample difficulty during denoising training, resulting in training instability. Furthermore, these methods can introduce false negatives—sentences that are different from the original but are actually valid. Since models typically use cross-entropy training, these samples will be judged as incorrect predictions, causing inaccurate training.

[0005] Existing solutions guide models to learn human language structures through linguistic tasks, including both language representation and attention architecture levels. Specifically, the former integrates linguistic information such as part-of-speech tags, entity tags, or language role tags into word representations via word embeddings, serving as linguistic label enhancement; the latter uses syntactic structure parsing to obtain relationships between words, constructs a relation matrix, and integrates it into the weights of the attention mechanism. Both of these methods require modifications to the model architecture, leading to excessive computational costs. [Summary of the Invention]

[0006] To address the problem of excessive computational cost in existing models, this invention provides a method, system, and storage medium for reducing the computational cost of semantic recognition.

[0007] The solution to the technical problem of this invention is to provide a method for reducing the computational load of semantic recognition, comprising the following steps:

[0008] Obtain a pre-trained model;

[0009] The pre-trained model is further trained to optimize the training objective, thus becoming the final trained model.

[0010] Input the text to be recognized into the final training model;

[0011] Output the semantics of the text to be recognized.

[0012] Preferably, the specific steps for obtaining the pre-trained model include:

[0013] Obtain the original sentence W and set the first training objective;

[0014] The original sentence W is split into a sub-word original sequence, and the sub-word original sequence of the original sentence W is randomly masked to generate the masked sentence W' and the masked sequence.

[0015] The masked sentence W' is input into the language model to obtain the encoded context representation H, and the model is trained through the first training objective to obtain the pre-trained model.

[0016] Preferably, the training optimization objective is to evaluate the degree of damage of the denoising strategy to the original sentence and / or to evaluate the semantic distance between the predicted sentence and the original sentence.

[0017] Preferably, evaluating the destructive effect of the denoising strategy on the sentence specifically involves calculating the KL divergence between the original sequence and the masked sequence and using it as a second training objective.

[0018] Preferably, assessing the semantic distance between the predicted sentence and the original sentence includes the following steps:

[0019] The predicted words at the occluded positions in the original sentence sample W are obtained through a pre-trained model and then filled back into the occluded positions in the original sentence sample W to generate a predicted sentence.

[0020] The predicted sentence is split into a sequence of predicted sub-words; the KL divergence between the predicted sequence and the original sequence is calculated and used as a third training objective.

[0021] Preferably, the first training objective is combined with the second training objective to optimize the model and obtain the final training model; or, the first training objective is combined with the third training objective to optimize the model and obtain the final training model; or, the first training objective, the second training objective and the third training objective are combined to optimize the model and obtain the final training model.

[0022] Preferably, the first training objective is set as follows: The second training objective calculates the KL divergence between the original sequence and the masked sequence as follows: The third training objective calculates the KL divergence between the predicted sequence and the original sequence as follows: The training objective of the final training model is L = L D +L C +L S ; where L D L is the loss function of the pre-trained model. C Let L be the loss function for predicting the KL divergence between the original sequence and the masked sequence. S The prediction loss function for the KL divergence between the predicted sequence and the original sequence is w. k Let W be the k-th input sentence, W′ be the masked sentence, and H be the... i For the sequence distribution after masking, The original sequence distribution, To predict the sequence distribution.

[0023] Preferably, the training objective is a loss function, which is set as the cross-entropy loss function.

[0024] To address the aforementioned technical problems, this invention also provides a system for reducing the computational load of semantic recognition, comprising:

[0025] The training module is used to obtain a pre-trained model and add training optimization to the pre-trained model to become the final trained model;

[0026] The input module is used to input the text to be recognized into the final training model;

[0027] The output module is used to output the semantics of the text to be recognized.

[0028] To solve the above-mentioned technical problems, the present invention also provides a storage medium including a computer program, which, when executed by a processor, implements the steps of the above-described method.

[0029] Compared with the prior art, the method, system, and storage medium for reducing the computational load of semantic recognition provided by the present invention have the following beneficial effects:

[0030] 1. The method for reducing the computational load of semantic recognition in this invention first obtains a pre-trained model; then, it adds training optimization objectives to the pre-trained model to become the final trained model; finally, it inputs the text to be recognized into the final trained model; and outputs the semantics of the text to be recognized. By adding training optimization objectives, the accuracy and robustness of the training can be improved without changing the model structure, while also reducing the complexity and computational load of the model, thus solving the problem of excessive computational load in existing technology models.

[0031] 2. The specific steps of obtaining the pre-trained model in the method for reducing the computational load of semantic recognition according to the present invention include: obtaining the original sentence W and setting a first training objective; splitting the original sentence W into a sub-word sequence, randomly masking the sub-word sequence of the original sentence W to generate a masked sentence W' and a masked sequence; inputting the masked sentence W' into a language model to obtain the encoded context representation H, and training the model through the first training objective to obtain the pre-trained model. By training the model through the masked word sequence, the model initially acquires the ability to reconstruct semantics.

[0032] 3. The training optimization objective of the method for reducing the computational cost of semantic recognition in this invention is to evaluate the degree of destruction of the original sentence by the denoising strategy and / or to evaluate the semantic distance between the predicted sentence and the original sentence. By evaluating the degree of destruction of the original sentence by the denoising strategy and / or evaluating the semantic distance between the predicted sentence and the original sentence, as a training objective at the sentence or sequence level, it can effectively supplement the conventional word-level cross-entropy objective, thereby enhancing the model's learning efficiency of knowledge. That is, under the same model size and training resource consumption, it produces a model with stronger performance, higher training efficiency, and greater robustness in application.

[0033] 4. The method for reducing the computational cost of semantic recognition in this invention specifically evaluates the sentence-destructiveness of the denoising strategy by calculating the KL divergence between the original sequence and the masked sequence and using it as a second training objective. By evaluating the sentence-destructiveness of the denoising strategy as a training objective at the sentence or sequence level, it can effectively supplement the conventional word-level cross-entropy objective, thereby enhancing the model's learning efficiency.

[0034] 5. The method for reducing the computational cost of semantic recognition in this invention, which evaluates the semantic distance between the predicted sentence and the original sentence, includes the following steps: obtaining predicted words at the occluded positions in the original sentence sample W through a pre-trained model, and backfilling these words into the occluded positions in the original sentence sample W to generate a predicted sentence; splitting the predicted sentence into a sub-word prediction sequence; calculating the KL divergence between the predicted sequence and the original sequence and using it as a third training objective. By evaluating the semantic distance between the predicted sentence and the original sentence as a sentence or sequence-level training objective, this effectively complements conventional word-level cross-entropy objectives, thereby enhancing the model's learning efficiency.

[0035] 6. The method for reducing the computational load of semantic recognition in this invention combines a first training objective with a second training objective to optimize the model and obtain a final trained model; or, combines a first training objective with a third training objective to optimize the model and obtain a final trained model; or, combines the first, second, and third training objectives to optimize the model and obtain a final trained model. By adding a second and third training objective and combining the first and second training objectives to optimize the model, or combining the first and third training objectives to optimize the model, or combining the first, second, and third training objectives to optimize the model, the trained model can obtain additional accuracy and robustness without changing its own framework.

[0036] 7. The first training objective of the method for reducing the computational load of semantic recognition in this invention is set as follows: The second training objective is to calculate the KL divergence between the original sequence and the masked sequence as follows: The third training objective is to calculate the KL divergence between the predicted sequence and the original sequence, set as follows: The final training objective of the model is L = L D +L C +L S ; where L D L is the loss function of the pre-trained model. C Let L be the loss function for predicting the KL divergence between the original sequence and the masked sequence. S The prediction loss function for the KL divergence between the predicted sequence and the original sequence is w. k Let W be the k-th input sentence, W′ be the masked sentence, and H be the... i For the sequence distribution after masking, The original sequence distribution, To predict sequence distribution, by optimizing the design of the training objective through increased training, a model with stronger performance, higher training efficiency, and greater robustness in application can be obtained with the same model and training costs.

[0037] 8. The method for reducing the computational cost of semantic recognition in this invention uses a loss function as the training objective, specifically the cross-entropy loss function. Using the cross-entropy loss function simplifies differentiation and allows for faster model training.

[0038] 9. This invention also provides a system for reducing the computational load of semantic recognition, comprising: a training module for acquiring a pre-trained model and training the pre-trained model to optimize the training objective into a final training model; an input module for inputting the text to be recognized into the final training model; and an output module for outputting the semantics of the text to be recognized. It has the same beneficial effects as the methods described above, and will not be elaborated further here.

[0039] 10. The present invention also provides a storage medium including a computer program, which, when executed by a processor, implements the steps of the above-described method. It has the same beneficial effects as the above-described method, and will not be repeated here. [Attached Image Description]

[0040] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 This is a flowchart of the steps of the method for reducing the computational load of semantic recognition provided in the first embodiment of the present invention.

[0042] Figure 2 This is a detailed flowchart of the steps involved in obtaining a pre-trained model, as provided in the first embodiment of the present invention, to reduce the computational load of semantic recognition.

[0043] Figure 3 This is a schematic diagram of the training optimization training objective principle of the method for reducing the computational load of semantic recognition provided in the first embodiment of the present invention.

[0044] Figure 4 These are comparison diagrams showing the effects of an example provided in the first embodiment of the present invention.

[0045] Figure 5 This is a block diagram of a system for reducing the computational load of semantic recognition provided in the second embodiment of the present invention.

[0046] Explanation of reference numerals in the attached diagram:

[0047] 300. Systems that reduce the computational load of semantic recognition;

[0048] 301. Training module; 302. Input module; 303. Output module.

Detailed Implementation Methods

[0049] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0050] In the embodiments provided by this invention, it should be understood that "B corresponding to A" means that B is associated with A, and B can be determined based on A. However, it should also be understood that determining B based on A does not mean determining B solely based on A; B can also be determined based on A and / or other information.

[0051] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of the invention. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Those skilled in the art should also recognize that the embodiments described in the specification are optional embodiments, and the actions and modules involved are not necessarily essential to the invention.

[0052] In various embodiments of the present invention, it should be understood that the sequence number of each process does not necessarily imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0053] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It is particularly important to note that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0054] Please see Figure 1 The first embodiment of the present invention provides a method for reducing the computational load of semantic recognition, comprising the following steps:

[0055] S1: Obtain the pre-trained model;

[0056] S2: Add training to the pre-trained model to optimize the training objective and become the final trained model;

[0057] S3: Input the text to be recognized into the final training model;

[0058] S4: Output the semantics of the text to be recognized.

[0059] By adding training optimization to the design of training objectives, which can serve as sentence or sequence-level training objectives, this effectively complements conventional word-level cross-entropy objectives. This enhances the model's learning efficiency, improves training accuracy and robustness without altering the model's structure, and reduces model complexity and computational cost, thus addressing the problem of excessive computation in existing models.

[0060] Further, please refer to Figure 2 The specific steps to obtain a pre-trained model include:

[0061] S101: Obtain the original sentence W = {w1, w2, ..., w n Establish the primary training objective;

[0062] S102: Split the original sentence W into a sub-word sequence, and select a specific proportion of tokens (representing language units W in the sentence) from the original sentence W. i The sentence W′ is randomly masked with a special symbol (such as [MASK]) to obtain the masked sentence W′ and the masked sequence.

[0063] S103: Let D = {k1,k2,…,k m} represents the position of the masked token, where m represents the total number of times the token is masked;

[0064] S104: Input the masked sentence W′ into the language model to obtain the encoded context representation H={h1,h2,…,h n};

[0065] S105: Obtain the pre-trained model through the first training objective.

[0066] Specifically, the first training objective is set as follows:

[0067]

[0068] Among them, L D Here, m is the loss function of the pre-trained model, D is the total number of occluded tokens, and w is the position of the occluded token. k Let W be the k-th input sentence, and W′ be the masked sentence.

[0069] By training the model with masked word sequences, the model can initially acquire the ability to restore semantics.

[0070] Further, please refer to Figure 3 The training optimization objective is to evaluate the degree of damage the denoising strategy does to the original sentence and / or to evaluate the semantic distance between the predicted sentence and the original sentence.

[0071] Furthermore, the evaluation of the denoising strategy's impact on sentence destruction is specifically achieved by calculating the KL divergence between the original sequence and the masked sequence and using it as a second training objective.

[0072] Specifically, the second training objective calculates the KL divergence between the original sequence and the masked sequence as follows:

[0073]

[0074] Where L C H is the loss function for predicting the KL divergence between the original sequence and the masked sequence. i For the sequence distribution after masking, This represents the original sequence distribution.

[0075] Furthermore, assessing the semantic distance between the predicted sentence and the original sentence includes the following steps:

[0076] The predicted words at the occluded positions in the original sentence sample W are obtained through a pre-trained model and then filled back into the occluded positions in the original sentence sample W to generate a predicted sentence.

[0077] The predicted sentence is split into a sequence of predicted sub-words; the KL divergence between the predicted sequence and the original sequence is calculated and used as a third training objective.

[0078] Specifically, the third training objective calculates the KL divergence between the predicted sequence and the original sequence as follows:

[0079]

[0080] Among them, L S The prediction loss function is used to predict the KL divergence between the predicted sequence and the original sequence. To predict the sequence distribution, This represents the original sequence distribution.

[0081] By evaluating the degree to which denoising strategies disrupt the original sentences and / or assessing the semantic distance between the predicted sentences and the original sentences, as training objectives at the sentence or sequence level, this can effectively complement conventional word-level cross-entropy objectives, thereby enhancing the model's learning efficiency. In other words, it results in a model with stronger performance, higher training efficiency, and greater robustness in application, all within the same model size and training resource consumption.

[0082] Furthermore, the model can be optimized by combining the first training objective with the second training objective to obtain the final training model; or, the model can be optimized by combining the first training objective with the third training objective to obtain the final training model; or, the model can be optimized by combining the first training objective, the second training objective, and the third training objective to obtain the final training model.

[0083] Specifically, the method for reducing the computational cost of semantic recognition in this invention uses a loss function as the training objective, specifically the cross-entropy loss function. Using the cross-entropy loss function simplifies differentiation and allows for faster model training, thus improving training efficiency.

[0084] For example, we use the BERT-base-uncased model as the baseline and compare it with the model after training optimization. After pre-training on the English Wikipedia corpus and BookCorpus corpus, the trained language model's performance is validated on eight datasets (development set) of the English GLUE benchmark platform. The comparison results are shown in the figure below. Figure 4 As shown, Figure 4 The performance comparison between the trained and optimized model and the baseline model is shown, demonstrating a significant performance improvement.

[0085] For example, input the original sentence W:

[0086] Thank you for inviting me to your party last week

[0087] During preprocessing, the language model uses a special marker [MASK] to randomly mask words in the sentence, resulting in the masked sentence W′:

[0088] Thank you for[MASK][MASK]to your[MASK]last week

[0089] The training objective of the language model is to reconstruct the words masked by the [MASK] marker. The output of a typical model is shown below:

[0090] {inviting,us,home} corresponds to the content covered by the three [MASK] tags, and the predicted sentence is:

[0091] Thank you for inviting us to your home last week

[0092] Because the language model is trained using cross-entropy, which determines whether each word perfectly matches the target word, both "us" and "home" will be incorrectly predicted, leading to inaccurate training.

[0093] In reality, language expression is diverse. After the model's prediction results are backfilled into sentences, the new sentences are standard sentences that conform to the logic of language expression and are semantically similar to the original sentences. Therefore, they should be judged as correct predictions.

[0094] Therefore, this invention employs an increased evaluation of the denoising strategy's impact on the sentence and / or the evaluation of the semantic distance between the predicted sentence and the original sentence as training objectives at the sentence or sequence level. This can effectively supplement conventional word-level cross-entropy objectives. The smaller the loss should be when the semantics of the predicted sentence are similar to those of the original sentence, thereby alleviating the above problems.

[0095] Continuing with the example above, the masked sentence W′ is input into the language model to obtain the masked sequence H:

[0096] Thank you for[MASK][MASK]to your[MASK]last week

[0097] The original sentence is input into the language model to obtain the original sequence.

[0098] Thank you for inviting me to your party last week

[0099] At this point, it can be done according to the formula Calculate H and The divergence between the two is used to assess the degree to which the denoising strategy damages the sentence.

[0100] Input the predicted sentence into the language model to obtain the predicted sequence.

[0101] Thank you for inviting us to your home last week

[0102] At this point, it can be done according to the formula calculate and The divergence between the predicted and original sentences is used to assess the semantic distance between them.

[0103] By enhancing the evaluation of the denoising strategy's impact on the sentence and / or assessing the semantic distance between the predicted and original sentences, the methods described above can avoid misclassifying "us" and "home" as incorrect predictions, thus preventing inaccurate training. This improves the performance of the judgment, such as... Figure 4 As shown, the model's judgment performance was significantly improved after adding the evaluation of the denoising strategy's impact on the sentence and the evaluation of the semantic distance between the predicted sentence and the original sentence.

[0104] Please see Figure 5 The second embodiment of the present invention provides a system 300 for reducing the computational load of semantic recognition, comprising:

[0105] Training module 301 is used to obtain a pre-trained model and add training optimization to the pre-trained model to become the final trained model;

[0106] Input module 302 is used to input the text to be recognized into the final training model;

[0107] Output module 303 is used to output the semantics of the text to be recognized.

[0108] It has the same beneficial effects as the methods described above, and will not be repeated here.

[0109] A third embodiment of the present invention provides a storage medium including a computer program that, when executed by a processor, implements the steps of the above-described method. It has the same beneficial effects as the method described above, and will not be repeated here.

[0110] Compared with existing technologies, the method, system, and storage medium for reducing the computational load of semantic recognition provided by this invention have the following beneficial effects:

[0111] 1. The method for reducing the computational load of semantic recognition in this invention first obtains a pre-trained model; then, it adds training optimization objectives to the pre-trained model to become the final trained model; finally, it inputs the text to be recognized into the final trained model; and outputs the semantics of the text to be recognized. By adding training optimization objectives, the accuracy and robustness of the training can be improved without changing the model structure, while also reducing the complexity and computational load of the model, thus solving the problem of excessive computational load in existing technology models.

[0112] 2. The specific steps of obtaining the pre-trained model in the method for reducing the computational load of semantic recognition according to the present invention include: obtaining the original sentence W and setting a first training objective; splitting the original sentence W into a sub-word sequence, randomly masking the sub-word sequence of the original sentence W to generate a masked sentence W' and a masked sequence; inputting the masked sentence W' into a language model to obtain the encoded context representation H, and training the model through the first training objective to obtain the pre-trained model. By training the model through the masked word sequence, the model initially acquires the ability to reconstruct semantics.

[0113] 3. The training optimization objective of the method for reducing the computational cost of semantic recognition in this invention is to evaluate the degree of destruction of the original sentence by the denoising strategy and / or to evaluate the semantic distance between the predicted sentence and the original sentence. By evaluating the degree of destruction of the original sentence by the denoising strategy and / or evaluating the semantic distance between the predicted sentence and the original sentence, as a training objective at the sentence or sequence level, it can effectively supplement the conventional word-level cross-entropy objective, thereby enhancing the model's learning efficiency of knowledge. That is, under the same model size and training resource consumption, it produces a model with stronger performance, higher training efficiency, and greater robustness in application.

[0114] 4. The method for reducing the computational cost of semantic recognition in this invention specifically evaluates the sentence-destructiveness of the denoising strategy by calculating the KL divergence between the original sequence and the masked sequence and using it as a second training objective. By evaluating the sentence-destructiveness of the denoising strategy as a training objective at the sentence or sequence level, it can effectively supplement the conventional word-level cross-entropy objective, thereby enhancing the model's learning efficiency.

[0115] 5. The method for reducing the computational cost of semantic recognition in this invention, which evaluates the semantic distance between the predicted sentence and the original sentence, includes the following steps: obtaining predicted words at the occluded positions in the original sentence sample W through a pre-trained model, and backfilling these words into the occluded positions in the original sentence sample W to generate a predicted sentence; splitting the predicted sentence into a sub-word prediction sequence; calculating the KL divergence between the predicted sequence and the original sequence and using it as a third training objective. By evaluating the semantic distance between the predicted sentence and the original sentence as a sentence or sequence-level training objective, this effectively complements conventional word-level cross-entropy objectives, thereby enhancing the model's learning efficiency.

[0116] 6. The method for reducing the computational load of semantic recognition in this invention combines a first training objective with a second training objective to optimize the model and obtain a final trained model; or, combines a first training objective with a third training objective to optimize the model and obtain a final trained model; or, combines the first, second, and third training objectives to optimize the model and obtain a final trained model. By adding a second and third training objective and combining the first and second training objectives to optimize the model, or combining the first and third training objectives to optimize the model, or combining the first, second, and third training objectives to optimize the model, the trained model can obtain additional accuracy and robustness without changing its own framework.

[0117] 7. The first training objective of the method for reducing the computational load of semantic recognition in this invention is set as follows: The second training objective is to calculate the KL divergence between the original sequence and the masked sequence as follows: The third training objective is to calculate the KL divergence between the predicted sequence and the original sequence, set as follows: The final training objective of the model is L = L D +L C +L S ; where L D L is the loss function of the pre-trained model. C Let L be the loss function for predicting the KL divergence between the original sequence and the masked sequence. S The prediction loss function for the KL divergence between the predicted sequence and the original sequence is w. k Let W be the k-th input sentence, W′ be the masked sentence, and H be the... i For the sequence distribution after masking, The original sequence distribution, To predict sequence distribution, by optimizing the design of the training objective through increased training, a model with stronger performance, higher training efficiency, and greater robustness in application can be obtained with the same model and training costs.

[0118] 8. The method for reducing the computational cost of semantic recognition in this invention uses a loss function as the training objective, specifically the cross-entropy loss function. Using the cross-entropy loss function simplifies differentiation and allows for faster model training.

[0119] 9. This invention also provides a system for reducing the computational load of semantic recognition, comprising: a training module for acquiring a pre-trained model and training the pre-trained model to optimize the training objective into a final training model; an input module for inputting the text to be recognized into the final training model; and an output module for outputting the semantics of the text to be recognized. It has the same beneficial effects as the methods described above, and will not be elaborated further here.

[0120] 10. The present invention also provides a storage medium including a computer program, which, when executed by a processor, implements the steps of the above-described method. It has the same beneficial effects as the above-described method, and will not be repeated here.

[0121] The foregoing has provided a detailed description of a method, system, and storage medium for reducing the computational load of semantic recognition according to embodiments of the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention. Any modifications, equivalent substitutions, and improvements made within the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for reducing the computational load of semantic recognition, characterized in that: Includes the following steps: Obtain the original sentence W and set the first training objective; The original sentence W is split into a sub-word sequence, and the sub-word sequence of the original sentence W is randomly masked to generate the masked sentence. And the masked sequence, the masked sentence Input the language model to obtain the encoded context representation H, and train the model through the first training objective to obtain the pre-trained model; The first training objective is set to , Let D be the loss function of the pre-trained model, where D represents the position of the masked token, and m represents the total number of masked tokens. Indicates that when it appears In the case of w k The probability, For the k-th input sentence, To cover the following sentence; The pre-trained model is further optimized by adding training objectives to become the final trained model; The training optimization objective includes a second training objective and / or a third training objective. The second training objective is to calculate the KL divergence between the original sequence and the masked sequence, and is set as follows: The third training objective is to calculate the KL divergence between the predicted sequence and the original sequence, set as follows: ; in, Let KL divergence be the prediction loss function between the original sequence and the masked sequence. The prediction loss function for the KL divergence between the predicted sequence and the original sequence. For the sequence distribution after masking, The original sequence distribution, To predict the sequence distribution, i represents the i-th sequence, and n represents the total number of sequences; Input the text to be recognized into the final training model; Output the semantics of the text to be recognized.

2. The method for reducing the computational load of semantic recognition as described in claim 1, characterized in that: The prediction sequence is obtained by splitting the prediction sentence into a sub-word prediction sequence. The prediction sentence is generated by obtaining the predicted words at the covered positions in the original sentence W through a pre-trained model and backfilling them into the covered positions in the original sentence W.

3. The method for reducing the computational load of semantic recognition as described in claim 1, characterized in that: The second training objective is used to evaluate the degree to which the denoising strategy disrupts the sentence; the third training objective is used to evaluate the semantic distance between the predicted sentence and the original sentence.

4. A system for reducing the computational load of semantic recognition, characterized in that: include: The training module is used to obtain a pre-trained model and add training optimization to the pre-trained model to become the final trained model; The training process includes: Obtain the original sentence W and set the first training objective; The original sentence W is split into a sub-word sequence, and the sub-word sequence of the original sentence W is randomly masked to generate the masked sentence. And the masked sequence, the masked sentence Input the language model to obtain the encoded context representation H, and train the model through the first training objective to obtain the pre-trained model; The first training objective is set to , Let D be the loss function of the pre-trained model, where D represents the position of the masked token, and m represents the total number of masked tokens. Indicates that when it appears In the case of w k The probability, For the k-th input sentence, To cover the following sentence; The pre-trained model is further optimized by adding training objectives to become the final trained model; The training optimization objective includes a second training objective and / or a third training objective. The second training objective is to calculate the KL divergence between the original sequence and the masked sequence, and is set as follows: The third training objective is to calculate the KL divergence between the predicted sequence and the original sequence, set as follows: ; in, Let KL divergence be the prediction loss function between the original sequence and the masked sequence. The prediction loss function for the KL divergence between the predicted sequence and the original sequence. For the sequence distribution after masking, The original sequence distribution, To predict the sequence distribution, i represents the i-th sequence, and n represents the total number of sequences; The input module is used to input the text to be recognized into the final training model; The output module is used to output the semantics of the text to be recognized.

5. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the method described in any one of claims 1-3.

Images