Semantic detection method and system based on prompt tuning and medium

Through the method based on prompt tuning, effective prompts and optimized languageizers are automatically designed, which solves the problem that malicious speech detection relies on high-complexity methods and external resources in the existing technology, and realizes efficient and stable malicious speech detection and classification Effect.

CN119443110AInactive Publication Date: 2025-02-14GUANGDONG OCEAN UNIVERSITY
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411469075.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-10-21
Publication Date
2025-02-14
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

The prior art relies on high-complexity methods or external resources in malicious speech detection and classification, and its performance is unstable and difficult to adapt to rapidly changing network languages.

Method used

Through the method based on prompt tuning, effective prompts are automatically designed to improve the model's utilization efficiency of small-scale annotated data, reduce dependence on large-scale annotated data, and optimize the languageizer to improve detection performance.

Benefits of technology

It realizes significant advantages in the field of malicious speech detection and classification, reduces the cost of data collection and labeling, improves the timeliness and effectiveness of detection, and retains the interpretability of the linguist.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119443110A_ABST
    Figure CN119443110A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of semantic detection, in particular to a semantic detection method and system based on prompt tuning and a medium, and the method comprises the steps: crawling data from each network platform, and screening out malicious speech data from the data to form a malicious speech data set; randomly selecting a part of data from the malicious speech data set as a training data set, and taking the remaining data as a test data set; generating a malicious speech detection model according to the training data set, and testing the malicious speech detection model by using the test data set; and performing semantic detection by using the tested malicious speech detection model. According to the method, effective prompts are automatically designed, so that the utilization efficiency of the model on small-scale annotation data is improved, the dependence on large-scale annotation data is reduced, the cost of data collection and annotation is reduced, and the instability and cost of manual design prompts are avoided.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of semantic detection, and in particular to a semantic detection method, system and medium based on prompt tuning. Background Art

[0002] In the context of the rapid development of the Internet and social media, malicious speech detection and classification applications are particularly necessary. They have a profound impact on national security, social stability, and the image of the platform. Effective malicious speech detection and classification methods can timely curb online speech, improve the content quality and user experience of social platforms, and enhance the image of the platform.

[0003] Prompt tuning is a technique for optimizing deep learning models. Its core is to adjust the prompt text of the model input to affect the performance of the model in specific tasks or scenarios. This method is usually used in pre-trained language models (PLMs). By changing the content, language expression or format of the prompt text, the model is guided to generate more expected output or improve the small sample performance of tasks such as classification and regression.

[0004] Pre-trained language model is a method of self-supervised learning using large-scale unlabeled text data. By pre-training on a large amount of text, the model learns a general language representation. This technology mainly uses the BERT series of pre-trained language models, which are trained through two tasks: Masked Language Model (MLM) and Next Sentence Prediction (NSP). The MLM task requires the model to predict randomly masked words in a sentence, thereby prompting the model to understand the contextual information of each word; the NSP task requires the model to determine whether two sentences are semantically adjacent, which helps the model understand the relationship and logic between sentences.

[0005] Hinted tuning has achieved great success in enabling pre-trained language models (PLMs) to perform different downstream NLP (Natural Language Processing) tasks, especially when the amount of downstream data is limited. Reframing the task as a fill-in-the-blank problem is an effective way to hint tuning. This method is implemented by adding additional text (templates) to the output and mapping the output to one or more tokens. It directly applies the MLM pre-training task to the downstream task, eliminating the gap between pre-training and performing downstream tasks, and greatly reducing the number of samples required for fine-tuning. However, the performance of such methods is very dependent on verbalizers. Therefore, the choice of verbalizer is a key issue when using this method.

[0006] Based on the above background, many researchers have used this method to solve various text classification tasks. In recent years, research on this type of method has made significant progress, and several representative technologies include:

[0007] Manual Verbalizer: Schick and A manual linguist solution was used, where a linguist was manually written based on knowledge of the task and the pre-trained model.

[0008] Automatic discrete linguistics: Gao et al. adopted the idea of ​​enumeration to automatically obtain discrete linguistics. They sorted the words by testing the performance of each word in the PLMs vocabulary in its corresponding category, then screened the top K words in each category, performed Cartesian product on them by category, and tested them to select discrete linguistics; Shin et al. automatically obtained discrete linguistics by training a classifier. They selected words with high relevance to a given label to form a linguistics by training a classifier; Hu et al. proposed a method based on knowledge graph enhancement. They obtained words related to a given label from the vocabulary through some existing dictionary tools. A knowledge base was introduced, and the triples in the knowledge base were used to measure the relevance.

[0009] Automatic Continuous Linguisticizer: Hambardzumyan et al. directly optimized word embeddings to form a linguisticizer. Instead of selecting discrete label words, they directly used word embeddings to form a linguisticizer and optimized them during training through gradient descent. Cui et al. used the idea of ​​prototypical network. They first defined a fixed template such as "A[MASK]news." and concatenated it to the original input sentence, input it into the pre-trained language model, obtained the representation vector of the [mask] position to represent the sentence vector, and used it as the label word to form a linguisticizer.

[0010] However, current technologies still have many shortcomings, such as:

[0011] (1) Using only discrete label words to form a verbalizer usually performs poorly due to the limitations of natural language.

[0012] (2) The languageizer is composed of only continuous label words, which is incomprehensible to humans and has poor interpretability, making it difficult to improve it in a targeted manner.

[0013] (3) Using highly complex methods or introducing external resources such as knowledge bases requires high time and labor costs and has unstable performance, which is not suitable for malicious online speech that updates quickly. Summary of the invention

[0014] The purpose of the present invention is to provide a semantic detection method, system and medium based on prompt tuning, which improves the model's utilization efficiency of small-scale annotated data by automatically designing effective prompts, thereby reducing dependence on large-scale annotated data, which not only reduces the cost of data collection and annotation, but also avoids the instability and cost of manually designed prompts.

[0015] In order to achieve the above object, the present invention provides the following technical solutions:

[0016] In a first aspect, an embodiment of the present invention provides a semantic detection method based on prompt tuning, the method comprising the following steps:

[0017] Crawl data from various online platforms and filter out malicious speech data to form a malicious speech dataset;

[0018] Randomly select part of the data from the malicious speech dataset as a training dataset, and use the remaining data as a verification dataset and a test dataset;

[0019] Training a malicious speech detection model based on the training data set and the verification data set, and testing the malicious speech detection model using the test data set;

[0020] Use the tested malicious speech detection model to perform semantic detection;

[0021] The training of the malicious speech detection model specifically includes:

[0022] Generate and optimize a languageizer based on a pre-trained language model;

[0023] During the model training process, hyperparameters are selected based on the performance of the model on the validation dataset, including cue word integration and vocabulary purification.

[0024] Preferably, the generating and optimizing the languageizer based on the pre-trained language model specifically includes:

[0025] Given a pre-trained language model And the model input text x to be classified in ;

[0026] Input the model to text x in The input text is packaged into a cloze task through template T, represented as: T =T(x in );

[0027] Enter the cloze task into text x T Input to pre-trained language model In the example above, we get the hidden vector h of [MASK] [MASK] ;

[0028] Given all categories Y of a classification task, map them into a linguisticizer V consisting of tokens in the vocabulary and their weights, and calculate the classification probability;

[0029] After calculating the classification probability, fine-tune the pre-trained language model with the correct labels And adjust the weight of the languageizer V to minimize the cross entropy loss, and obtain the selected and optimized languageizer V;

[0030] Among them, [MASK] represents the input text x of the cloze task T The blank position to be filled in.

[0031] Preferably, after obtaining the hidden vector h [MASK] After that, the probability that [MASK] is filled with token v is calculated, expressed as:

[0032]

[0033] in, express The token v represents a specific word in the vocabulary, and the token v' represents an arbitrary word in the vocabulary, and e v represents the embedding of token v, e v’ Represents the embedding of token v'.

[0034] Preferably, the classification probability is calculated as:

[0035]

[0036] Among them, for each category y∈Y, its mapping V y =V(y).

[0037] Preferably, the prompt word integration includes:

[0038] For each pre-trained language model Both exist in the vocabulary

[0039] Let the mapping V of each category y be y All are expressed as: And assign a weight w to each token v∈V in the linguistizer V v .

[0040] Preferably, a shadow weight is introduced, specifically:

[0041] At each parameter update, the new weight value With a certain ratio β and the old shadow weight Perform exponential moving average, expressed as:

[0042] Preferably, the classification probability is improved based on the shadow weight, which is expressed as:

[0043]

[0044] Preferably, the vocabulary purification includes the following contents:

[0045] Use the full vocabulary for training;

[0046] Select the mapping V for each category y y All corresponding weights w v The first k tokens form the formally trained linguist.

[0047] The purified V y V y ′, expressed as;

[0048] V y ′=V y (Top-k{w v}),v∈V y ;

[0049] Among them, V y (w v ) represents the token corresponding to the weight; Top-k is the operator.

[0050] In a second aspect, an embodiment of the present invention provides a semantic detection system based on prompt tuning, the system comprising:

[0051] at least one processor;

[0052] at least one memory for storing at least one program;

[0053] When the at least one program is executed by the at least one processor, the at least one processor implements the semantic detection method based on prompt tuning as described in any one of the above.

[0054] In a third aspect, an embodiment of the present invention provides a computer-readable storage medium, which stores a program executable by a processor, characterized in that the program executable by the processor is used to execute the semantic detection method based on prompt tuning as described in any one of the above when executed by the processor.

[0055] The beneficial effects of the present invention are:

[0056] (1) The present invention makes targeted improvements based on the prompt tuning method and has significant advantages in the field of malicious speech detection and classification. By automatically designing effective prompts, the model's efficiency in utilizing small-scale annotated data is improved, thereby reducing dependence on large-scale annotated data. This not only reduces the cost of data collection and annotation, but also eliminates the instability and cost of manually designed prompts; in addition, due to its excellent small sample performance, it can adapt to emerging malicious speech and changing network language more quickly, improving the timeliness and effectiveness of detection.

[0057] (2) The present invention maps discrete label words into a continuous space, thereby improving the performance of the discrete linguisticizer. Compared with other continuous optimization methods, the method of the present invention retains the interpretability of the final linguisticizer. It does not introduce any external resources or prior knowledge, and adopts a relatively simple solution, which not only saves the cost of use, but also makes it easy to combine with other methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0059] Figure 1 is a flow chart of a semantic detection method based on prompt tuning in an embodiment of the present invention;

[0060] Figure 2 It is a structural diagram of a semantic detection system based on prompt tuning in an embodiment of the present invention. DETAILED DESCRIPTION

[0061] The following will be combined with the embodiments and drawings to clearly and completely describe the concept, specific structure and technical effects of the present invention, so as to fully understand the purpose, scheme and effect of the present invention. It should be noted that the embodiments and features in the embodiments of the present invention can be combined with each other without conflict.

[0062] See also Figure 1 The present invention provides a semantic detection method based on prompt tuning, the method comprising the following steps:

[0063] Crawl data from various online platforms and filter out malicious speech data to form a malicious speech dataset;

[0064] Randomly select part of the data from the malicious speech dataset as a training dataset, and use the remaining data as a verification dataset and a test dataset;

[0065] Training a malicious speech detection model based on the training data set and the verification data set, and testing the malicious speech detection model using the test data set;

[0066] Use the tested malicious speech detection model to perform semantic detection;

[0067] The training of the malicious speech detection model specifically includes:

[0068] Generate and optimize a languageizer based on a pre-trained language model;

[0069] During the model training process, hyperparameters are selected based on the performance of the model on the validation dataset, including cue word integration and vocabulary purification.

[0070] In an embodiment provided in the present application, the generating and optimizing the languageizer based on the pre-trained language model specifically includes:

[0071] Given a pre-trained language model And the model input text x to be classified in ;

[0072] Input the model to text x in The input text is packaged into a cloze task through template T, represented as: T =T(x in );

[0073] Enter the cloze task into text x T Input to pre-trained language model In the example above, we get the hidden vector h of [MASK] [MASK] ;

[0074] Given all categories Y of a classification task, map them into a linguisticizer V consisting of tokens in the vocabulary and their weights, and calculate the classification probability;

[0075] After calculating the classification probability, fine-tune the pre-trained language model with the correct labels And adjust the weight of the languageizer V to minimize the cross entropy loss, and obtain the selected and optimized languageizer V;

[0076] Among them, [MASK] represents the input text x of the cloze task T The blank position to be filled in.

[0077] It should be noted that [MASK] is the blank that needs to be filled in PLMs. For example, if the sentence "[MASK] is the capital of China." is used as the input of the model, the task of the model is to determine what word should be in the position of [MASK] (the output is the probability that each word in the model vocabulary should be filled in [MASK], in this case, the probability corresponding to 'Beijing' should be the largest), which can be understood as a blank in a cloze test.

[0078] In one embodiment provided in the present application, after obtaining the hidden vector h [MASK] After that, the probability that [MASK] is filled with token v is calculated, expressed as:

[0079]

[0080] in, Represents a pre-trained language model The token is a general term for all words in the vocabulary, then token v represents a specific word in the vocabulary, token v' represents any word in the vocabulary, and e v represents the embedding of token v, e v’ Represents the embedding of token v'.

[0081] It should be noted that the above formula is used to calculate the probability of [MASK] being filled with token v, which is similar to normalization. If there are three words in the vocabulary, 'apple', 'banana', and 'orange', and you need to calculate the probability of [MASK] being filled with 'apple', you need to divide the probability of being filled with 'apple' (unnormalized) by the sum of the probabilities of 'apple', 'banana', and 'orange' (unnormalized). Applying this formula to all three words ensures that the sum of the three probabilities is 1. In this example, v is 'apple', and v' represents any one of the three words.

[0082] In an embodiment provided in the present application, the classification probability is calculated and expressed as:

[0083]

[0084] Among them, for each category y∈Y, its mapping V y =V(y).

[0085] Specifically, the present application provides a method for selecting a languageizer through the above content, which specifically includes the following contents:

[0086] Given a pre-trained language model And the model input text x to be classified in .

[0087] First, x in Wrap the input text x into a cloze task through template T T =T ( x in ). If x in = "Wishing you a wonderful week." and T(x) = x([MSAK]), then the corresponding cloze input text is represented as:

[0088] x T =Wishing you a wonderful week.([MASK]),

[0089] Then, enter the cloze task into the text x T Input to pre-trained language model In the example above, we can get the hidden vector h of [MASK] [MASK] . Command word list For pre-trained language models The vocabulary of [MASK] is filled with tokens The probability can be expressed as:

[0090]

[0091] where e v is the embedding of token v.

[0092] Given all categories Y of a classification task, map them to a languageizer V consisting of tokens; for each category y∈Y, its mapping V y =V(y). Then the text x in The probability of being classified as category y is as follows:

[0093]

[0094] After calculating the classification probability, the pre-trained language model can be fine-tuned with the correct labels. And adjust the weight of the linguisticizer V to minimize the cross entropy loss. The linguisticizer selection method proposed in this embodiment improves the above process by affecting the process of the above classification probability formula.

[0095] In an embodiment provided in the present application, the prompt word integration includes the following contents:

[0096] For each pre-trained language model Both exist in the vocabulary

[0097] Let the mapping V of each category y be y Both are expressed as And assign a weight w to each token v∈V in the linguistizer V v .

[0098] In an embodiment provided in the present application, a shadow weight is introduced, specifically:

[0099] At each parameter update, the new weight value With a certain ratio β and the old shadow weight Perform exponential moving average, expressed as:

[0100] In an embodiment provided in the present application, the above classification probability is improved based on the shadow weight, which is expressed as:

[0101]

[0102] Specifically, regarding the prompt word integration, it includes the following contents:

[0103] For each pre-trained language model There are vocabulary lists Let the mapping V of each category y Can be expressed as: And assign a weight w to each token v∈V in the linguist v That is, use it as a weight matrix Perform a dot multiplication operation with the mapped model output. | Y | and |V y | are the number of categories and the size of the languageizer, respectively. In addition, in order to make the weight update smoother and reduce the jitter and instability of the parameters, this application also introduces shadow weights: when each parameter is updated, the new weight value is With a certain ratio β and the old shadow weight Perform exponential moving average. According to this idea, we first randomly initialize a weight matrix to integrate the discrete tokens in the languageizer. At this point, we can map each V y Each of them is considered as a whole, that is, a token that can be continuously optimized. Therefore, only the weight matrix needs to be involved in the training to achieve continuous optimization of the discrete languageizer. In this embodiment, the formula of classification probability is improved as follows:

[0104]

[0105] In an embodiment provided in the present application, the vocabulary purification includes the following contents:

[0106] Use the full vocabulary for training;

[0107] Select the mapping V for each category y y All corresponding weights w v The first k tokens form the formally trained linguist.

[0108] The purified V y V y ′, expressed as;

[0109] V y ′=V y (Top-k{w v}),v∈V y ;

[0110] Among them, V y (w v ) represents the token corresponding to the weight; Top-k is the operator.

[0111] It should be noted that Top-k is a whole, an operator that takes all w v The words corresponding to the first k weights in the middle are used to select words that have a greater impact on the classification results. The left side of the equal sign represents a purified vocabulary.

[0112] Specifically, as the scale of models continues to increase, the vocabulary of many models has also reached a very large scale. However, under the condition of few samples, it is difficult to optimize the huge weight matrix using only the previously described method, and in most tasks, such a large-scale languageizer may not be needed, which introduces a lot of unnecessary noise and causes the performance of the model to be unsatisfactory. In view of the above problems, this embodiment uses the complete vocabulary for training, and then selects each mapping V y All corresponding w v The first k tokens form the formally trained languageizer. This process is called purification in this embodiment. yIt can be expressed as: V y ′=V y (Top-k{w v}),v∈V y .

[0113] Where V y (w v ) represents the token corresponding to the weight; Top-k is the operator.

[0114] By filtering out unnecessary words, the model loss can be reduced more steadily and computing resources can be saved.

[0115] To sum up, the idea of ​​the present application is to map the probability of each word in the pre-trained language model prediction vocabulary being filled with [MASK] with the languageizer to obtain the probability of each label word in the languageizer being filled with [MASK], then multiply it with the weight matrix and sum it row by row to obtain the probability corresponding to each category, and then update the weight matrix after calculating the loss with the label.

[0116] In addition, it should be noted that this application first crawls data from various online platforms and selects malicious speech data that does not involve privacy as the data set for subsequent use; then, randomly selects part of the obtained data as the training data set, and trains the model using the above method; finally, uses the remaining data to form a test data set and uses it to test and verify the trained model, specifically by implementing comparison and ablation experiments.

[0117] This application adopts a prompt-based tuning method, which has powerful small sample performance and can adapt to changeable and obscure malicious online speech; it develops integrated optimized languageizer technology, which is efficient and concise, reduces the cost of technology use and improves system flexibility; it retains the interpretability of the languageizer to facilitate targeted optimization of the system.

[0118] On the other hand, this application can use pre-trained models in different languages ​​and fields to adapt to various needs.

[0119] In summary, this application is based on the prompt tuning method and has targeted improvements, which has significant advantages in the field of malicious speech detection and classification. By automatically designing effective prompts, the model's efficiency in utilizing small-scale annotated data is improved, thereby reducing dependence on large-scale annotated data. This not only reduces the cost of data collection and annotation, but also eliminates the instability and cost of manually designed prompts. In addition, due to its excellent small sample performance, this application can adapt to emerging malicious speech and changing network language more quickly, improving the timeliness and effectiveness of detection.

[0120] Therefore, the present application provides the above-mentioned prompt tuning method that can integrate a languageizer, which improves the malicious speech detection method and has the following beneficial effects:

[0121] (1) This application maps discrete label words into a continuous space, improving the performance of the discrete linguisticizer.

[0122] (2) Compared with other continuous optimization methods, the semantic detection method provided in this application retains the interpretability of the final languageizer.

[0123] (3) This application does not introduce any external resources or prior knowledge, and adopts a relatively simple solution, which not only saves the cost of use, but also makes it easy to combine with other methods.

[0124] and Figure 1 Corresponding to the method, refer to Figure 2 , an embodiment of the present invention provides a semantic detection system based on prompt tuning, including:

[0125] at least one processor;

[0126] at least one memory for storing at least one program;

[0127] When the at least one program is executed by the at least one processor, the at least one processor implements the above method.

[0128] It can be seen that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0129] In addition, an embodiment of the present invention further discloses a computer program product or a computer program, which is stored in a computer-readable storage medium. A processor of a computer device can read the computer program from a computer-readable storage medium, and the processor executes the computer program, so that the computer device executes the above method. Similarly, the contents of the above method embodiment are all applicable to the storage medium embodiment, and the functions specifically implemented by the storage medium embodiment are the same as those of the above method embodiment, and the beneficial effects achieved are also the same as those achieved by the above method embodiment.

[0130] It will be appreciated by those skilled in the art that all or some of the methods disclosed above, the system can be implemented as software, firmware, hardware and appropriate combinations thereof. Some physical components or all physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, and the computer-readable medium can include a computer storage medium (or a non-transitory medium) and a communication medium (or a temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassette, magnetic tape, disk storage or other magnetic storage device, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0131] The above is a specific description of the preferred implementation of the present disclosure, but the present disclosure is not limited to the above-mentioned implementation mode. Technical personnel familiar with the field can also make various equivalent deformations or substitutions without violating the spirit of the present disclosure. These equivalent deformations or substitutions are all included in the scope defined by the claims of the present disclosure.

Claims

1. A semantic detection method based on prompt tuning, characterized in that: The method comprises the following steps: Crawl data from various online platforms and filter out malicious speech data to form a malicious speech dataset; Randomly select part of the data from the malicious speech dataset as a training dataset, and use the remaining data as a verification dataset and a test dataset; Training a malicious speech detection model based on the training data set and the verification data set, and testing the malicious speech detection model using the test data set; Use the tested malicious speech detection model to perform semantic detection; The training of the malicious speech detection model specifically includes: Generate and optimize a languageizer based on a pre-trained language model; During the model training process, hyperparameters are selected based on the performance of the model on the validation dataset, including cue word integration and vocabulary purification.

2. The semantic detection method based on prompt tuning according to claim 1, characterized in that: The generating and optimizing the languageizer based on the pre-trained language model specifically includes: Given a pre-trained language model And the model input text x to be classified in ; Input the model to text x in The input text is packaged into a cloze task through template T, represented as: T =T(x in ); Enter the cloze task into text x T Input to pre-trained language model In the example above, we get the hidden vector h of [MASK] [MASK] ; Given all categories Y of a classification task, map them into a linguisticizer V consisting of tokens in the vocabulary and their weights, and calculate the classification probability; After calculating the classification probability, fine-tune the pre-trained language model with the correct labels And adjust the weight of the languageizer V to minimize the cross entropy loss, and obtain the selected and optimized languageizer V; Among them, [MASK] represents the input text x of the cloze task T The blank position to be filled in.

3. The semantic detection method based on prompt tuning according to claim 2, characterized in that: After obtaining the hidden vector h [MASK] After that, the probability that [MASK] is filled with token v is calculated, expressed as: in, express The token v represents a specific word in the vocabulary, and the token v' represents an arbitrary word in the vocabulary, and e v represents the embedding of token v, e v’ Represents the embedding of token v'.

4. The semantic detection method based on prompt tuning according to claim 3, characterized in that: The classification probability is calculated and expressed as: Among them, for each category y∈Y, its mapping V y =V(y).

5. The semantic detection method based on prompt tuning according to claim 1, characterized in that: The prompt word integration includes: For each pre-trained language model Both exist in the vocabulary Let the mapping V of each category y be y All are expressed as: And assign a weight w to each token v∈V in the linguistizer V v .

6. The semantic detection method based on prompt tuning according to claim 5, characterized in that: Introduce shadow weights, specifically: At each parameter update, the new weight value With a certain ratio β and the old shadow weight Perform exponential moving average, expressed as:

7. The semantic detection method based on prompt tuning according to claim 6, characterized in that: The classification probability is improved based on the shadow weight, expressed as:

8. The semantic detection method based on prompt tuning according to claim 1, characterized in that: The vocabulary purification includes the following contents: Use the full vocabulary for training; Select the mapping V for each category y y All corresponding weights w v The first k tokens form the formally trained linguist. The purified V y V y ′, expressed as; V y ′=V y (Top-k{w v }),v∈V y ; Among them, V y (w v ) represents the token corresponding to the weight; Top-k is the operator.

9. A semantic detection system based on prompt tuning, characterized in that: The system comprises: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the semantic detection method based on prompt tuning as described in any one of claims 1 to 8.

10. A computer-readable storage medium storing a program executable by a processor, characterized in that: The processor-executable program is used to perform the method according to any one of claims 1 to 8 when executed by the processor.

Citation Information

Cited By

  • Toxic speech detection method and system based on forgetting learning

    CN121501995A