Scenarized adaptive semantic recognition method, system and equipment and storage medium

Abstract

According to the scenarized adaptive semantic recognition method, system and device and the storage medium, the model can be further improved on a better semantic representation base through fine tuning and retraining on the pre-trained language model; through a comparative learning method, the model can better represent the semantics of the whole sentence; through adaptive keyword extraction and positive sample generation, the model can be better distinguished and adjusted in the training process; the large model capability is called through prompt for data verification and automatic marking, the data preparation stage and the training stage are separated, and learning of large model knowledge by a small model is achieved. And finally, low-cost, self-adaptive and self-supervised training of the high-quality semantic representation model is realized. And recognizing semantics by using the trained semantic representation model.

Description

Technical Field

[0001] The present invention belongs to the technical field of natural language processing, and specifically relates to a scenario-based adaptive semantic recognition method, system, device and storage medium. Background Art

[0002] Language model is a technology used for natural language processing, whose main task is to predict the next word or character for a given text sequence. In the past few decades, the technology of language model has developed a lot, and the technical background behind it has become more and more complex and advanced. The earliest language model was based on the traditional n-gram model, which makes predictions by analyzing n adjacent words or characters in a text sequence. With the development of machine learning technology, language models based on neural networks have also been widely adopted, among which the most famous are the recurrent neural network (RNN) model and the long short-term memory (LSTM) model. These models can model long-term dependencies, making the language model more predictive and the quality of generated text higher. In recent years, the rapid development of deep learning technology has brought new breakthroughs to language models, among which the most representative are Transformer-based language models, such as GPT-3 and BERT. The Transformer model uses the self-attention mechanism to capture the dependencies between text sequences, and adopts a hierarchical structure and pre-training technology, which greatly improves the performance and effect of the language model. Today, there are LLMs (large-scale pre-trained language models) with billions or even tens of billions of parameters, such as T5, LLaMA, GPT-4, etc. These large models are trained using large amounts of text data and can perform a wide range of tasks, including text summarization, text translation, sentiment analysis, text summarization, code generation, mathematical computing, logical reasoning, text understanding, dialogue systems, etc. Large language models are characterized by the ability to generate high-quality text and coherent output based on the input context.

[0003] Glossary: ​​Prompt: In the field of large language models, prompt refers to a piece of text that guides the model to generate a specific output. The prompt can be a question, a sentence, a paragraph, or an article. The design of the prompt is very important for generating high-quality text because it can affect the content and style of the model's generation.

[0004] LLM: Large Language Model, which is a language model built by a deep neural network with more than tens of billions of parameters, and trained with a large amount of unlabeled text using self-supervised learning methods. Representative ones include: ChatGPT, GPT4, ChatGLM, LLaMA, BaiChuan, Wenxin Yiyan, Xinghuo, Tongyi Qianwen, etc.

[0005] F1: It is the harmonic mean of precision and recall, and is used to measure the accuracy of the classifier.

[0006] Auc: Accuracy refers to the ratio of the number of samples correctly classified by the classifier to the total number of samples.

[0007] At present, if you want to use language models to represent text semantics, on the one hand, you can use traditional language models, but due to limitations such as technical architecture, the actual effect is poor; on the other hand, you can use large-scale pre-trained language models, but they will also face huge requirements in terms of hardware, computing power, and data, which are extremely difficult to meet; if you compromise and choose some models with acceptable parameter numbers, in the process of combining with downstream tasks, due to the misalignment of the two stages, a large amount of supervised data will be required for fine-tuning, and this requirement often consumes a lot of manpower and material resources, and even lacks the possibility of rapid reuse in different scenarios, which invisibly increases the cost of use. Summary of the invention

[0008] In view of the deficiencies in the prior art, the present invention provides a scenario-based adaptive semantic recognition method, system, device and storage medium. Through fine-tuning and retraining on the pre-trained language model, the model can be further improved on a good semantic representation base; through the comparative learning method, the model can better represent the semantics of the entire sentence; through adaptive keyword extraction and positive sample generation, the model can obtain better differentiation and adjustment during the training process; through prompt calling the large model capability for data verification and automatic labeling, the data preparation and training stages are separated to achieve the learning of the large model knowledge by the small model. Ultimately, a low-cost, adaptive, self-supervised training high-quality semantic representation model is achieved.

[0009] To achieve the above object, the present invention adopts the following technical solutions:

[0010] A scenario-based adaptive semantic recognition method comprises the following steps:

[0011] S1. Clean the scenario data to obtain corpus data;

[0012] S2, compile prompt templates based on corpus data;

[0013] S3, dividing the corpus data into supervised data, weakly supervised data and unsupervised data; the supervised data has labels, and the labels are used to distinguish the similarity of the corpus data, and similar text pairs and non-similar text pairs are obtained according to the labels; the weakly supervised data has labels but is not used to mark similarity; the unsupervised data is corpus data without labels;

[0014] Perform cluster analysis on weakly supervised data and unsupervised data to obtain several cluster data. Use the cluster data and prompt templates to let the large model generate a set of refined words corresponding to each corpus data. Use the refined word set to distinguish the similarities between corpus data to obtain similar text pairs and dissimilar text pairs.

[0015] Extract triplets from all similar text pairs and dissimilar text pairs, and divide them into training sets, validation sets, and test sets;

[0016] S4. Configure the parameters of the semantic representation model and the parameters of comparative learning, and use the training set to train the model; use the validation set and the test set to verify and test the semantic representation model obtained in each training round, and judge the effect of the model training by comparing the indicators among the training set, validation set, and test set. If the performance is not improved, the training ends;

[0017] S5. Use the trained semantic representation model to perform semantic recognition.

[0018] To optimize the above technical solutions, the specific measures taken also include:

[0019] Furthermore, in S1, the cleaning includes filtering of sensitive corpus, unification of punctuation marks, conversion of simplified and traditional Chinese characters, filtering of confidential information and alignment of values.

[0020] Furthermore, in S2, the language used for compiling the prompt template is LISP language.

[0021] Furthermore, in S3, the specific process of performing cluster analysis on the weakly supervised data and the unsupervised data to obtain a number of cluster data, and using the cluster data and the prompt template to allow the large model to generate a set of refined words corresponding to each corpus data is as follows:

[0022] Perform cluster analysis on weakly supervised data to obtain several cluster data Among them, k is the kth label among all K labels in the weak supervision data D2, and j is the jth cluster of all J clusters under the kth label;

[0023] From each cluster data The corpus data is extracted evenly, and the prompt template is used to let the large model generate the refined word set corresponding to each corpus data;

[0024] Perform cluster analysis on unsupervised data to obtain several cluster data Where p is the pth label among all P labels in the unsupervised data D3;

[0025] From each cluster data The corpus data is evenly extracted from the dataset, and the prompt template is used to let the large model generate a set of refined words corresponding to each corpus data.

[0026] Furthermore, in S3, the specific process of using the refined word set to distinguish the similarities between the corpus data and obtain similar text pairs and dissimilar text pairs is:

[0027] The initial text pair (d m ,d n ), the rule is the mth corpus data d m The corresponding refined word set w m The length of the nth corpus data d n The corresponding refined word set w n The difference in length is less than or equal to 1;

[0028] Calculate the mth corpus data d m The corresponding refined word set w m With the nth corpus data d n The corresponding refined word set w n The intersection of mn , calculate the intersection w mn The length of the corresponding w m and n The ratio of the lengths of the two ratios is taken as the minimum value l mn ;

[0029] If l mn If it is greater than 0.8, the text is considered m ,d n ) is a similar text pair, if l mn Less than or equal to 0.8, then the text is considered to be m ,d n ) are non-similar text pairs.

[0030] Furthermore, in S4, the parameters of the semantic representation model include batch size and GPU, and the parameters of the contrastive learning include temperature coefficient and dropout ratio.

[0031] Furthermore, the indicators include the loss value Loss, the harmonic mean F1 of the precision and recall rate, and the accuracy Auc.

[0032] The present invention also proposes a scenario-based adaptive semantic recognition system, comprising:

[0033] Data cleaning module, used to clean scenario data and obtain corpus data;

[0034] A prompt preparation module is used to prepare prompt templates in combination with corpus data;

[0035] The data set preparation module is used to divide the corpus data into supervised data, weakly supervised data and unsupervised data; the supervised data has labels, and the labels are used to distinguish the similarity of the corpus data, and similar text pairs and non-similar text pairs are obtained according to the labels; the weakly supervised data has labels but is not used to mark similarity; the unsupervised data is corpus data without labels;

[0036] Perform cluster analysis on weakly supervised data and unsupervised data to obtain several cluster data. Use the cluster data and prompt templates to let the large model generate a set of refined words corresponding to each corpus data. Use the refined word set to distinguish the similarities between corpus data to obtain similar text pairs and dissimilar text pairs.

[0037] Extract triplets from all similar text pairs and dissimilar text pairs, and divide them into training sets, validation sets, and test sets;

[0038] The model training module is used to configure the parameters of the semantic representation model and the parameters of comparative learning, and use the training set to train the model; the validation set and the test set are used to verify and test the semantic representation model obtained in each training round, and the effect of the model training is judged by comparing the indicators among the training set, validation set, and test set. If the performance does not improve, the training ends;

[0039] The semantic recognition module is used to perform semantic recognition using the trained semantic representation model.

[0040] The present invention also proposes an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the scenario-based adaptive semantic recognition method as described above is implemented.

[0041] The present invention also proposes a computer-readable storage medium storing a computer program, wherein the computer program enables a computer to execute the scenario-based adaptive semantic recognition method as described above.

[0042] The beneficial effects of the present invention are:

[0043] A programming language-style prompt compilation method is provided to fully stimulate the text understanding and text reasoning capabilities of the large language model, enabling the large model to reason about samples with high quality, thereby ensuring that the large model's summary of text clusters and key information of the text is correct.

[0044] A cost-effective data collection method combining clustering algorithm and big model is proposed. It not only eliminates the need to extract samples from the corpus one by one like the traditional solution to let the big model generate corresponding training data, but also integrates the text embedding ability of the open source cutting-edge clustering algorithm model and the text understanding ability of the big model, so that this solution can efficiently utilize the capabilities of the big model in a low-cost and adaptive manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 This is an overall flow chart of the scenario-based adaptive semantic recognition method proposed in the present invention. DETAILED DESCRIPTION

[0046] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0047] Embodiment 1

[0048] The present invention proposes a scenario-based adaptive semantic recognition method. The overall process is as follows: Figure 1 As shown, the following steps are included:

[0049] S1. Clean the scenario data to obtain corpus data. Specifically, perform cleaning on the scenario data such as filtering sensitive corpus, unifying punctuation marks, converting simplified and traditional Chinese characters, filtering confidential information, and aligning values ​​to obtain corpus data D, where each data is recorded as d, d i That is the i-th data in D.

[0050] S2. Prepare a prompt template in combination with the corpus data; the present invention prepares a programming language prompt with reference to the LISP language, and the template is as follows:

[0051] ;;Purpose: Extract key information from the given content

[0052] ;;Set the following to your *SystemPrompt*

[0053] (defun<*>field expert()

[0054] "You are an expert in <*> field. You can always extract key information from a text, and try to use words that have been extracted in previous texts."

[0055] (Good at refining and summarizing)

[0056] (Expression. Concise)

[0057] (output.vocabulary))

[0058] (defun refine fusion (user input)

[0059] "You can succinctly summarize the key information in a text and extract and integrate new vocabulary based on the existing vocabulary."

[0060] (let(merge(succinct(summarize user input))))

[0061] (few-shots(<sample corpus>.<word 1, word 2, word 3>)))

[0062] (defunstart()

[0063] "Run at startup"

[0064] (let(system-role<*>field expert))

[0065] ;;Operation rules

[0066] ;;1. The (start) function must be run at startup

[0067] ; ; 2. Then call the main function (refining and integrating user input)

[0068] In the above template, <*> needs to be customized according to the field or scenario to which the corpus belongs. <Sample corpus> and <word 1, word 2, word 3> also need to provide customized examples based on the actual corpus.

[0069] S3, dividing the corpus data into supervised data, weakly supervised data and unsupervised data; the supervised data has labels, and the labels are used to distinguish the similarity of the corpus data, and similar text pairs and non-similar text pairs are obtained according to the labels; the weakly supervised data has labels but is not used to mark similarity; the unsupervised data is corpus data without labels;

[0070] Perform cluster analysis on weakly supervised data and unsupervised data to obtain several cluster data. Use the cluster data and prompt templates to let the large model generate a set of refined words corresponding to each corpus data. Use the refined word set to distinguish the similarities between corpus data to obtain similar text pairs and dissimilar text pairs.

[0071] The specific process of performing cluster analysis on weakly supervised data and unsupervised data to obtain a number of cluster data, and using the cluster data and prompt templates to allow the large model to generate a set of refined words corresponding to each corpus data is as follows:

[0072] Perform cluster analysis on weakly supervised data to obtain several cluster data Among them, k is the kth label among all K labels in the weak supervision data D2, and j is the jth cluster of all J clusters under the kth label;

[0073] From each cluster data The corpus data is extracted evenly, and the prompt template is used to let the large model generate the refined word set corresponding to each corpus data;

[0074] Perform cluster analysis on unsupervised data to obtain several cluster data Where p is the pth label among all P labels in the unsupervised data D3;

[0075] From each cluster data The corpus data is evenly extracted from the dataset, and the prompt template is used to let the large model generate a set of refined words corresponding to each corpus data.

[0076] The specific process of using the refined word set to distinguish the similarities between corpus data and obtain similar text pairs and non-similar text pairs is as follows:

[0077] The initial text pair (d m ,d b ), the rule is the mth corpus data d m The corresponding refined word set w m The length of the nth corpus data d n The corresponding refined word set w n The difference in length is less than or equal to 1;

[0078] Calculate the mth corpus data d m The corresponding refined word set w m With the nth corpus data d n The corresponding refined word set w n The intersection of mn , calculate the intersection w mn The length of the corresponding w m and n The ratio of the lengths of the two ratios is taken as the minimum value l mn ;

[0079] If l mn If it is greater than 0.8, the text is considered m ,d n ) is a similar text pair, if l mn Less than or equal to 0.8, then the text is considered to be m ,d n ) are non-similar text pairs.

[0080] Triples are extracted from all similar text pairs and dissimilar text pairs, and then divided into training set, validation set and test set in the ratio of 7:2:1;

[0081] S4. Configure the parameters of the semantic representation model (such as batch size, GPU, etc.) and the parameters of contrastive learning (such as temperature coefficient, dropout ratio, etc.) as well as the path of the base model, and use the training set to train the model; use the validation set and test set to verify and test the semantic representation model obtained in each training round, output the performance indicators of the current model on the validation set in each epoch of training, and output the performance indicators of the current model on the test set after each epoch.

[0082] The effect of model training is judged by comparing the indicators of the training set, validation set, and test set. If the performance does not improve, the training ends. The indicators include the loss value, the harmonic mean F1 of the precision and recall rate, and the accuracy rate Auc. If there is no performance improvement after multiple batch nodes (recommended to be 1 / 5 of the total batch number) within an epoch (the standard for improvement is recommended to be set to 1 / 10), the training ends.

[0083] S5. Create a relevant line graph of the performance parameters of the model after each batch training, save the model file after the key batch or epoch training, and the file name contains the corresponding epoch and batch number for easy identification. Use the trained semantic representation model for semantic recognition.

[0084] Embodiment 2

[0085] The present invention proposes a scenario-based adaptive semantic recognition system corresponding to the method of embodiment 1, comprising:

[0086] Data cleaning module, used to clean scenario data and obtain corpus data;

[0087] A prompt preparation module is used to prepare prompt templates in combination with corpus data;

[0088] The data set preparation module is used to divide the corpus data into supervised data, weakly supervised data and unsupervised data; the supervised data has labels, and the labels are used to distinguish the similarity of the corpus data, and similar text pairs and non-similar text pairs are obtained according to the labels; the weakly supervised data has labels but is not used to mark similarity; the unsupervised data is corpus data without labels;

[0089] Perform cluster analysis on weakly supervised data and unsupervised data to obtain several cluster data. Use the cluster data and prompt templates to let the large model generate a set of refined words corresponding to each corpus data. Use the refined word set to distinguish the similarities between corpus data to obtain similar text pairs and dissimilar text pairs.

[0090] Extract triplets from all similar text pairs and dissimilar text pairs, and divide them into training sets, validation sets, and test sets;

[0091] The model training module is used to configure the parameters of the semantic representation model and the parameters of comparative learning, and use the training set to train the model; the validation set and the test set are used to verify and test the semantic representation model obtained in each training round, and the effect of the model training is judged by comparing the indicators among the training set, validation set, and test set. If the performance does not improve, the training ends;

[0092] The semantic recognition module is used to perform semantic recognition using the trained semantic representation model.

[0093] The implementation methods of each module and module function in the system are completely consistent with the steps of the method in Example 1, so they will not be repeated here.

[0094] Embodiment 3

[0095] The present invention proposes an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the scenario-based adaptive semantic recognition method as described in Example 1 is implemented.

[0096] Embodiment 4

[0097] The present invention provides a computer-readable storage medium storing a computer program, wherein the computer program enables a computer to execute the scenario-based adaptive semantic recognition method as described in the first embodiment.

[0098] In the embodiments disclosed in the present application, the computer storage medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. The computer storage medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the above. More specific examples of computer storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above.

[0099] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0100] The above are only preferred embodiments of the present invention. The protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should be regarded as the protection scope of the present invention.

Claims

1. A scenario-based adaptive semantic recognition method, characterized in that: The following steps are involved: S1. Clean the scenario data to obtain corpus data; S2, compile prompt templates based on corpus data; S3, dividing the corpus data into supervised data, weakly supervised data and unsupervised data; the supervised data has labels, and the labels are used to distinguish the similarity of the corpus data, and similar text pairs and non-similar text pairs are obtained according to the labels; the weakly supervised data has labels but is not used to mark similarity; the unsupervised data is corpus data without labels; Perform cluster analysis on weakly supervised data and unsupervised data to obtain several cluster data. Use the cluster data and prompt templates to let the large model generate a set of refined words corresponding to each corpus data. Use the refined word set to distinguish the similarities between corpus data to obtain similar text pairs and dissimilar text pairs. Extract triplets from all similar text pairs and dissimilar text pairs, and divide them into training sets, validation sets, and test sets; S4. Configure the parameters of the semantic representation model and the parameters of comparative learning, and use the training set to train the model; use the validation set and the test set to verify and test the semantic representation model obtained in each training round, and judge the effect of the model training by comparing the indicators among the training set, validation set, and test set. If the performance is not improved, the training ends; S5. Use the trained semantic representation model to perform semantic recognition.

2. The scenario-based adaptive semantic recognition method according to claim 1, characterized in that: In S1, the cleaning includes filtering of sensitive corpus, unification of punctuation marks, conversion of simplified and traditional Chinese characters, filtering of confidential information and alignment of values.

3. The scenario-based adaptive semantic recognition method according to claim 1, characterized in that: In S2, the language used for compiling the prompt template is LISP language.

4. The scenario-based adaptive semantic recognition method according to claim 1, characterized in that: In S3, the specific process of performing cluster analysis on weakly supervised data and unsupervised data to obtain a number of cluster data, and using the cluster data and prompt templates to allow the large model to generate a set of refined words corresponding to each corpus data is as follows: Perform cluster analysis on weakly supervised data to obtain several cluster data Among them, k is the kth label among all K labels in the weak supervision data D2, and j is the jth cluster of all J clusters under the kth label; From each cluster data The corpus data is extracted evenly, and the prompt template is used to let the large model generate the refined word set corresponding to each corpus data; Perform cluster analysis on unsupervised data to obtain several cluster data Where p is the pth label among all P labels in the unsupervised data D3; From each cluster data The corpus data is evenly extracted from the dataset, and the prompt template is used to let the large model generate a set of refined words corresponding to each corpus data.

5. The scenario-based adaptive semantic recognition method according to claim 1, characterized in that: In S3, the specific process of using the refined word set to distinguish the similarities between the corpus data and obtain similar text pairs and dissimilar text pairs is as follows: The initial text pair (d m ,d n ), the rule is the mth corpus data d m The corresponding refined word set w m The length of the nth corpus data d n The corresponding refined word set w n The difference in length is less than or equal to 1; Calculate the mth corpus data d m The corresponding refined word set w m With the nth corpus data d n The corresponding refined word set w n The intersection of mn , calculate the intersection w mn The length of the corresponding w m and n The ratio of the lengths of the two ratios is taken as the minimum value l mn ; If l mn If it is greater than 0.8, the text is considered m ,d n ) is a similar text pair, if l mn Less than or equal to 0.8, then the text is considered to be m ,d n ) are non-similar text pairs.

6. The scenario-based adaptive semantic recognition method according to claim 1, characterized in that: In S4, the parameters of the semantic representation model include batch size and GPU, and the parameters of the contrastive learning include temperature coefficient and dropout ratio.

7. The scenario-based adaptive semantic recognition method according to claim 1, characterized in that: In S4, the indicators include the loss value Loss, the harmonic mean F1 of the precision and recall rate, and the accuracy Auc.

8. A scenario-based adaptive semantic recognition system, characterized in that: include: Data cleaning module, used to clean scenario data and obtain corpus data; A prompt preparation module is used to prepare prompt templates in combination with corpus data; The data set preparation module is used to divide the corpus data into supervised data, weakly supervised data and unsupervised data; the supervised data has labels, and the labels are used to distinguish the similarity of the corpus data, and similar text pairs and non-similar text pairs are obtained according to the labels; the weakly supervised data has labels but is not used to mark similarity; the unsupervised data is corpus data without labels; Perform cluster analysis on weakly supervised data and unsupervised data to obtain several cluster data. Use the cluster data and prompt templates to let the large model generate a set of refined words corresponding to each corpus data. Use the refined word set to distinguish the similarities between corpus data to obtain similar text pairs and dissimilar text pairs. Extract triplets from all similar text pairs and dissimilar text pairs, and divide them into training sets, validation sets, and test sets; The model training module is used to configure the parameters of the semantic representation model and the parameters of comparative learning, and use the training set to train the model; the validation set and the test set are used to verify and test the semantic representation model obtained in each training round, and the effect of the model training is judged by comparing the indicators among the training set, validation set, and test set. If the performance does not improve, the training ends; The semantic recognition module is used to perform semantic recognition using the trained semantic representation model.

9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the scenario-based adaptive semantic recognition method as described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: The computer program enables a computer to execute the scenario-based adaptive semantic recognition method as described in any one of claims 1 to 7.

Images