Text intention recognition method based on deep soft hint
Through the combination of deep soft prompt network and residual module, the problem of low recognition accuracy in text intention recognition is solved, efficient recognition under complex semantic relationships is achieved, and the recognition accuracy and stability of the model are improved.
Patent Information
- Application Number
- CN202411974108.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-30
- Publication Date
- 2025-04-18
AI Technical Summary
Existing pre-trained language models have low recognition accuracy in text intent recognition, especially in scenarios where specific data are sparse or small sample sizes are limited.
The deep soft prompt method is adopted to build a target deep soft prompt network containing several soft prompt layers and residual modules, combine the pre-trained language model for text intention recognition, and use the connection between each layer of soft prompt layer and residual module to improve the flexibility and accuracy of the model in complex semantic relationship processing.
It improves the accuracy of text intent recognition, reduces the noise problem caused by random initialization, stabilizes the capture of deep semantic information, and improves the recognition performance of the model in a sparse sample environment.
Smart Images
Figure CN119848255A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of natural language processing, and in particular to a text intention recognition method based on deep soft prompts. Background Art
[0002] In the field of Natural Language Processing (NLP), intent recognition has been a research focus that has received much attention in recent years. Intent recognition aims to identify and understand the core intent or purpose in user discourse. Simply put, it accurately captures the user's real needs by classifying the user's input sentences into predefined intent categories. For example, in an intelligent customer service system, intent recognition can help the system understand whether the user wants to check the balance, handle business, or make complaints and suggestions, etc., so as to provide more accurate and personalized services. Intent recognition is essentially a classification problem, which can be achieved through technologies such as machine learning and deep learning. In actual applications, it is necessary to build an intent recognition model based on specific scenarios and needs, and optimize the model through a lot of training.
[0003] Pre-trained language models (PLMs) are a major technological breakthrough in the field of intent recognition in recent years. In the pre-training stage, PLMs are trained using a large amount of unsupervised data (such as text information on the Internet) to learn the common features of text. These features can be expressed as knowledge in terms of vocabulary, grammar, semantics, etc. In the fine-tuning stage, PLMs are trained on specific NLP tasks using a small amount of supervised data to adjust the parameters of the model to make it better suited to specific tasks. Although PLM models demonstrate a deep understanding and processing capability of language, they still face severe challenges such as adaptability to specific downstream tasks and high training costs. These challenges mainly stem from the large parameter size of large language models and their dependence on a large amount of annotated data. Especially in specific data-sparse fields or scenarios with few samples, directly fine-tuning pre-trained language models is not only costly but also has limited effects. To address this issue, Prompt Learning (Prompt Learning) is a new type of learning method that uses a lot of supervised data to train the model. Prompt Learning (PL for short) came into being. Prompt learning re-expresses NLP tasks into a form that language models can understand (prompts), so that pre-trained language models can be directly applied to these tasks without large-scale fine-tuning. As an emerging method to adapt PLM models to downstream tasks, the core idea of prompt learning is to guide the model to handle specific tasks in a similar way to meta-learning by designing task-specific prompts (such as question templates), thereby effectively utilizing the knowledge and capabilities of the pre-trained model without significantly adjusting the model parameters. Prompt learning has shown excellent performance in many fields, especially when dealing with few-sample problems, it can significantly reduce the demand for labeled data, improve the generalization ability of the model, and reduce dependence on labeled data.
[0004] For example, the conversational recommendation model UniCRS (prompt learning + conversational recommendation model) based on PLM+prompt learning can effectively complete the recommendation and conversation subtasks, where DialoGPT (conversational model) is used as PLM to remodel these two subtasks using knowledge-enhanced prompt learning. By optimizing only these prompts, the UniCRS model can effectively complete the recommendation and conversation subtasks. The results show that UniCRS outperforms the existing CRS model (conversational recommendation model) and PLM model, and has excellent application prospects. In addition, prompt learning has excellent performance in multiple tasks in the NLP field by combining with PLM. Summary of the invention
[0005] The embodiment of the present invention provides a method for identifying text intent based on deep soft prompts, so as to at least solve the technical problem in the prior art that the pre-trained language model introduces prompt learning to identify the intent of the text and the recognition accuracy is low.
[0006] According to one aspect of an embodiment of the present invention, a text intent recognition method based on deep soft prompts is provided. The method may include: obtaining a target input text, wherein the target input text carries an intent category label; segmenting the target input text using a segmentation rule to obtain a segmented text; converting the segmented text using an id-token to obtain an initial text vector; encoding and converting the initial text vector to obtain a target text vector; constructing a target deep soft prompt network; wherein the target deep soft prompt network includes several soft prompt layers and several residual modules, and each soft prompt layer is connected to a residual module; inputting the target text vector into the target deep soft prompt network to obtain a target vector; inputting the target vector into a pre-trained language model to obtain a predicted intent category of the target input text.
[0007] Optionally, before obtaining the target input text, the method further includes: obtaining a natural language classification data set; preprocessing the natural language classification data set to obtain a plurality of input texts; and taking any one of the plurality of input texts as the target input text.
[0008] Optionally, the plurality of soft prompt layers are 12 soft prompt layers, and the dimension of each soft prompt layer is 768.
[0009] Optionally, the optimizer of the target deep soft hint network is an Adam optimizer.
[0010] Optionally, the learning rate of the target deep soft hint network is 0.0001.
[0011] Optionally, the pre-trained language model is a BERT model.
[0012] Beneficial effects of the present invention: The present invention proposes a text intent recognition method based on deep soft prompts. By connecting each soft prompt layer with each residual module, each soft prompt layer not only improves the flexibility and accuracy of the BERT model in processing complex semantic relationships, but also enables the BERT model to integrate and utilize semantic information at a deeper level. Each residual module processes the data of each soft prompt layer, so that the BERT model can capture semantic information more stably when processing soft prompts, reduces the noise problem caused by random initialization of prompts, and thus improves the accuracy of predicting the intent category of the input text. BRIEF DESCRIPTION OF THE DRAWINGS
[0013] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings: Figure 1is a flow chart of a text intent recognition method based on deep soft prompts according to an embodiment of the present invention; Figure 2 is a framework diagram of a text intent recognition method based on deep soft prompts according to an embodiment of the present invention; Figure 3 is a schematic diagram of a residual module according to an embodiment of the present invention. DETAILED DESCRIPTION
[0014] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only embodiments of a part of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.
[0015] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and to describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0016] Example 1 According to an embodiment of the present invention, a method for text intent recognition based on deep soft prompts is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system comprising at least one set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in an order different from that shown here.
[0017] Figure 1 is a flowchart of a text intent recognition method based on deep soft prompts according to an embodiment of the present invention. Figure 1 As shown, the method may include the following steps: Step S101, obtaining a target input text, wherein the target input text carries an intent category label.
[0018] In the technical solution provided in the above step S101 of the present invention, the target input text is obtained. For example, the target input text may be "I wish to hold your hand in the evening, and drink tea and porridge in the winter while it is still warm."
[0019] Step S102, segmenting the target input text using the segmentation rule to obtain a segmented text.
[0020] In the technical solution provided in step S102 of the present invention, the target input text may be 'I wish to hold your hand and stand at dusk, and drink tea and porridge in winter while it is still warm.' The word segmentation rules are used to perform word segmentation, and the word segmentation text ['wish', 'hold', 'you', 'hand', 'stand', 'yellow', 'dusk', ', ', ', 'winter', 'day', 'taste', 'tea', 'porridge', 'still', 'warm', '.'] is obtained.
[0021] Step S103, using id-token to convert the segmented text to obtain an initial text vector.
[0022] In the technical solution provided in the above step S103 of the present invention, the above-mentioned word segmentation text is converted using id-token to obtain an initial text vector, and the initial text vector is [2703, 2889, 2094, 2797, 4989, 7942, 3218, 8024, 182, 1100, 3189, 1501, 5751, 5114, 2213, 3946, 119].
[0023] Step S104, performing encoding conversion on the initial text vector to obtain a target text vector.
[0024] In the technical solution provided in the above step S104 of the present invention, the above initial text vector is encoded and converted to obtain a target text vector, and the target text vector is [101, 151, 8993, 8357, 11677, 8370, 102, 10288, 9059, 8913, 8175, 102].
[0025] Step S105, constructing a target depth soft prompt network; wherein the target depth soft prompt network includes a plurality of soft prompt layers and a plurality of residual modules, and each soft prompt layer is connected to a residual module.
[0026] In the technical solution provided in the above step S105 of the present invention, Figure 2 is a framework diagram of a text intent recognition method based on deep soft prompts according to an embodiment of the present invention, Figure 2 The yellow in the figure represents deep soft hints, the orange is the output parameters of the target deep soft hint network, and the blue is the parameter distribution of the pre-trained language model. Figure 2 The detailed distribution of the residual network (residual module) in Figure 3 As shown, Figure 3 is a schematic diagram of a residual module according to an embodiment of the present invention, and each soft prompt layer is connected to a residual module, the soft prompt layers include 12, and the residual modules include 12.
[0027] Step S106, inputting the target text vector into the target deep soft prompt network to obtain the target vector.
[0028] In the technical solution provided in the above step S106 of the present invention, the target depth soft prompt network processes the target text vector to obtain a target vector.
[0029] Step S107, input the target vector into the pre-trained language model to obtain the predicted intent category of the target input text.
[0030] In the technical solution provided in the above step S107 of the present invention, the pre-trained language model processes the target vector to obtain the predicted intent category of the target input text.
[0031] The above method of this embodiment is further introduced below.
[0032] As an optional implementation manner, in step S101, before obtaining the target input text, the method further includes: obtaining a natural language classification data set; preprocessing the natural language classification data set to obtain a plurality of input texts; and taking any one of the plurality of input texts as the target input text.
[0033] In this embodiment, the natural language classification data set is SST-2, a sentiment analysis related data set. After data preprocessing, there are 1758 input texts in total. Each input text has a label, 0 indicates negative sentiment, and 1 indicates positive sentiment. Store related data in a format.
[0034] Using Lists Represents several input texts in the dataset, where Representation list Middle Input text, The number of input texts; list Represents the label of each input text in the dataset, where Representation list Middle Input text Tags, is the number of tags, and the tag elements are composed of 0 and 1. Indicates that the input text is a positive sample, Indicates that the input text is a negative sample.
[0035] As an optional implementation manner, in step S105, the plurality of soft prompt layers are 12 soft prompt layers, and the dimension of each soft prompt layer is 768.
[0036] As an optional implementation manner, in step S105, the optimizer of the target depth soft prompt network is an Adam optimizer.
[0037] As an optional implementation manner, in step S105, the learning rate of the target deep soft prompt network is 0.0001.
[0038] As an optional implementation manner, in step S107, the pre-trained language model is a BERT model.
[0039] Experimental part: The training data set consists of 10,000 text data and the test data set consists of 3,000 text data, of which 1,000 are in Class 1, 1,000 are in Class 2, and 1,000 are in Class 3. The test results are shown in the following table.
[0040] Table 1 Performance indicators of this method on the test dataset
[0041] From the performance indicators in Table 1, we can see that the text intent recognition method based on deep soft hints has excellent performance in accuracy, F1 score, precision, and recall of the three categories. This is mainly due to (1) in the sample sparse environment, the intention recognition stability problem and performance problem faced by traditional soft hints are solved by re-parameterizing the residual network; (2) inserting trainable soft hint embedding into the prefix part of each layer of the original meta-learning-based pre-trained model, and using the residual network to re-parameterize the deep soft hint; (3) the vector of the original input text after quantization is sent to the model, and the knowledge of meta-learning of the pre-trained language model is used to calculate the semantic information representation vector.
[0042] In order to verify that this method is currently a more advanced intent recognition method, the performance of this method is compared with other commonly used methods. The results are shown in Table 2.
[0043] Table 2 Performance comparison of different intent recognition methods
[0044] From the performance comparison in Table 2, it can be found that the present method exceeds the existing intent recognition methods in all four indicators and has good application prospects. In summary, the present invention designs a lightweight and efficient intent recognition method based on a pre-trained language model from the perspective of lightweight and practicality. The experimental results show that the sample meta-learning intent recognition method based on deep soft prompts proposed in the present invention can utilize deep soft prompts processed by residual networks, so that the model can stably capture deep semantic information, thereby effectively solving the noise problem caused by random initialization of prompts. The prediction results have high accuracy and small errors, can be applied to various intent recognition scenarios, and have high practical value.
[0045] In an embodiment of the present invention, a target input text is obtained, wherein the target input text carries an intent category label; the target input text is segmented using a segmentation rule to obtain a segmented text; the segmented text is converted using an ID-token to obtain an initial text vector; the initial text vector is encoded and converted to obtain a target text vector; a target deep soft prompt network is constructed; wherein the target deep soft prompt network includes a plurality of soft prompt layers and a plurality of residual modules, and each soft prompt layer is connected to a residual module; the target text vector is input into the target deep soft prompt network to obtain a target vector; the target vector is input into a pre-trained language model to obtain a predicted intent category of the target input text, thereby solving the technical problem of low recognition accuracy in that the pre-trained language model in the prior art introduces prompt learning to recognize the intent of the text, and achieves the technical effect of connecting each soft prompt layer with each residual module, so that the BERT model can integrate and utilize semantic information at a deeper level, thereby improving the recognition accuracy of the intent of the sample.
[0046] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.
[0047] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0048] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.
[0049] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed over multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.
[0050] In addition, each functional unit in each embodiment of the present invention may be integrated into a first processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0051] The above are only preferred embodiments of the present invention. It should be pointed out that, for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.
Claims
1. A text intent recognition method based on deep soft prompts, characterized in that: include: Obtaining a target input text, wherein the target input text carries an intent category label; Segment the target input text using the word segmentation rules to obtain the segmented text; Use id-token to convert the segmented text to obtain the initial text vector; Encode and convert the initial text vector to obtain the target text vector; Constructing a target depth soft prompt network; wherein the target depth soft prompt network includes a plurality of soft prompt layers and a plurality of residual modules, and each soft prompt layer is connected to a residual module; Input the target text vector into the target deep soft prompt network to obtain the target vector; The target vector is input into the pre-trained language model to obtain the predicted intent category of the target input text.
2. The method according to claim 1, characterized in that Before obtaining the target input text, the method further includes: Get the natural language classification dataset; Preprocess the natural language classification data set to obtain several input texts; Take any one of several input texts as the target input text.
3. The method according to claim 1, characterized in that The plurality of soft prompt layers are 12 soft prompt layers, and the dimension of each soft prompt layer is 768.
4. The method according to claim 1, characterized in that The optimizer of the target depth soft prompt network is the Adam optimizer.
5. The method according to claim 1, characterized in that The learning rate of the target deep soft hint network is 0.0001.
6. The method according to claim 1, characterized in that The pre-trained language model is a BERT model.
7. A computer system, characterized in that include: One or more processors, and a computer-readable storage medium for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors are enabled to implement the method of claim 1.
8. A computer-readable storage medium, characterized in that Computer executable instructions are stored, and when the instructions are executed, they are used to implement the method of claim 1.
9. A computer program product, characterized in that The invention comprises computer executable instructions, which are used to implement the method of claim 1 when being executed.