A black-box text classification method and device based on multi-scale prompt memory enhancement

Abstract

The application discloses a black box text classification method and device based on multi-scale prompt memory enhancement, and belongs to the technical field of text data processing. The method comprises the following steps: obtaining text data, processing the text data, and obtaining training data and test data; in the training stage, obtaining multi-scale prompts according to the training data; after the training data and the multi-scale prompts are spliced, obtaining instance-level knowledge of different levels through a pre-trained language model, and storing the instance-level knowledge in a memory module; in the test stage, obtaining multi-scale prompts according to the test data; according to the obtained prompts, calculating instance-level scores and class-level scores for test samples by using the memory module, and comprehensively calculating a final score matrix according to the two scores of different levels, so that the class of the test sample is predicted. The application calculates the class score matrix for the test sample by using the memory module, thereby efficiently solving the text classification problem in the black box scene.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of text data processing, and particularly relates to a black-box text classification method and device based on multi-scale prompt memory enhancement. BACKGROUND

[0002] Text classification is a natural language processing task, which aims to divide text data into different categories. Existing text classification algorithms can be divided into traditional methods and deep learning methods. Traditional methods usually perform feature engineering on text, such as Bag of Words, to extract formats that are easy for machines to understand. Then a simple algorithm is used to classify the features. The deep learning method does not need to extract features manually, and it can learn more complex and abstract representations from raw data through end-to-end learning. In particular, with the emergence of recurrent neural networks (RNN) and Transformers, pre-trained language models based on Transformers have achieved remarkable results in text classification.

[0003] Although pre-trained language models have achieved great success in text classification, as their parameter size continues to grow, their training time and space overheads also grow rapidly. In low-resource scenarios, i.e., with little labeled data and lack of computing resources, how to deploy pre-trained language models to specific downstream tasks becomes a very challenging problem. A very intuitive method is to fine-tune the model by freezing part of the model's parameters. Other methods include adding a low-rank fine-tunable parameter module to the branch next to the model to adapt to the downstream task, and obtaining a smaller model through knowledge distillation. Researchers have found that large-scale pre-trained language models have initially acquired natural language understanding capabilities and can understand the semantic information of text. Therefore, the method based on prompt learning has become one of the mainstream methods for pre-trained language models to adapt to downstream tasks in low-resource scenarios. It pays more attention to mining the semantic information learned by pre-trained language models, and shows great advantages in text classification in low-resource scenarios.

[0004] In the black box scenario (unable to access the internal parameters and gradient information of the pre-trained language model), the existing method optimizes the prompt to adapt to the downstream task through a gradient-free optimization algorithm such as an evolutionary algorithm, and the internal parameters of the pre-trained language model remain unchanged. This adaptation algorithm that regards the pre-trained language model as a black box and only calls its API is a response to the increasing size and effectiveness of pre-trained language models. It can efficiently deploy pre-trained large models to individuals and small and micro enterprises, enabling them to be utilized by a wider group to help humans solve automated text classification tasks. However, the existing technical solutions still have the following technical problems: (1) In the black box scenario, the existing algorithm still needs to call the API of the pre-trained large language model a large number of times to search for the optimal prompt or needs a larger language model to assist in generating a prompt set to assist the model in classification when solving the few-shot text classification problem. This is contrary to the original intention of the black box algorithm, i.e., it is impossible to widely deploy pre-trained large language models to specific downstream tasks. (2) The existing memory enhancement method based on pre-trained language models still needs gradient for parameter updating, or there is no method that combines in-context learning to improve the effect of text classification. In the black box scenario, memory model related methods still need to be explored. (3) The existing memory enhancement method does not consider combining class-level information and instance-level information to jointly assist the downstream text classification task, and does not consider the role of knowledge of different scales in text classification. (4) The existing algorithm lacks consideration of different levels of different levels, and only considers instance-level information when classifying downstream text classification tasks, so most of the scoring modules simply use the nearest neighbor algorithm. SUMMARY

[0005] To at least partially solve one of the technical problems existing in the prior art, the purpose of the present application is to provide a black box text classification method and device based on multi-scale prompt memory enhancement.

[0006] The technical solution adopted by the present application is:

[0007] A black box text classification method based on multi-scale prompt memory enhancement, comprising the following steps:

[0008] Obtaining text data, processing the text data to obtain training data and test data;

[0009] In the training phase, the training data is arranged and combined according to a predetermined rule to obtain multi-scale prompts; after the training data and the multi-scale prompts are spliced, different levels of instance-level knowledge are obtained through a pre-trained language model, and the instance-level knowledge is stored in a memory module;

[0010] In the test phase, the test data is arranged and combined according to the preset rules to obtain multi-scale prompts; according to the obtained prompts, the memory module is used to calculate the instance-level score and the class-level score for the test sample, and the final score matrix is calculated according to the two scores at different levels, so as to predict the category of the test sample.

[0011] Further, the memory module is used to store the output representation extracted by the pre-trained language model combined with the context prompt in the training set;

[0012] For each training sample, the training sample is connected to the prompt , wherein c represents the number of connecting each class context example;

[0013] The memory module is represented as which contains the key-value pair of the corresponding sample output representation, the key is the output representation, and the value is the corresponding label;

[0014] The memory module The instance-level knowledge stored in the memory module is calculated according to the following formula:

[0015]

[0016] In the formula, v represents the words of the vocabulary table of the pre-trained language model, x i is the training sample; represents the model output representation obtained under the prompt of x i .

[0017] Further, the class-level knowledge is introduced into the memory module, and the formula is as follows:

[0018]

[0019] In the formula, j represents the corresponding category, represents the number of instance-level representations stored in the memory module under single scale, represents that the output is 1 when the input is true, otherwise the output is 0, and y i is the category of the training sample i.

[0020] Further, the knowledge stored in the memory module is extracted according to the prompt and the pre-trained language model, wherein the prompt used is the prompt form in the context learning; the representation is extracted according to the following formula:

[0021]

[0022] In the formula, ​denotes a pre-trained language model, and the parameters and gradient information inside the model cannot be accessed, i.e. a black-box scenario; v denotes a word of a vocabulary of the pre-trained language model, and the size is the same as that of the vocabulary; denotes a context prompt, and π denotes a test sample x test is converted into a specific format according to a preset prompt template; * is a placeholder.

[0023] Further, the prompt template is constructed as follows:

[0024]

[0025] wherein represents a simple concatenation symbol, denotes the i-th sample selected from the training set, and the category is denotes a label word of the category .

[0026] Further, the multi-scale prompt is obtained in the following manner:

[0027] In each category, examples are selected in a hierarchical sampling manner to constitute the prompt of each scale;

[0028] is adopted denotes the i-th key-value pair for storing knowledge when the scale size is c, wherein c is a scale size from 2 0 to 2 m-1 , and m denotes the number of multi-scales; the calculation formula of the key-value pair is as follows:

[0029]

[0030] s.t.c=2 m-1

[0031] wherein v denotes a word of a vocabulary of the pre-trained language model, and x i is a training sample.

[0032] Further, the instance-level score of the test sample x test is calculated in the following manner:

[0033]

[0034]

[0035]

[0036] wherein D KL denotes the distance of KL divergence calculation, p test denotes the representation of the test sample, denotes the prompt of the scale size of 1, denotes all the vocabulary; Top k (D ins ) denotes the selection of the first j values with the smallest relative distance, denotes the instance-level score of the test sample in the j-th class.

[0037] Further, the class-level score of the test sample x test is calculated in the following way:

[0038]

[0039] In the formula, is the class-level knowledge stored in the memory module.

[0040] Further, the calculation formula of the score matrix S is as follows:

[0041]

[0042] In the formula, λ is the weight, and || ||1 is the L1 norm.

[0043] Another technical solution adopted by the present application is:

[0044] A black box text classification device based on multi-scale prompt memory enhancement, comprising:

[0045] At least one processor;

[0046] At least one memory for storing at least one program;

[0047] When the at least one program is executed by the at least one processor, the at least one processor implements the above method.

[0048] Another technical solution adopted by the present application is:

[0049] A computer readable storage medium, wherein a processor executable program is stored, the processor executable program is used to execute the method as described above when executed by the processor.

[0050] The beneficial effects of the present application are: the present application extracts knowledge of different levels of instance level and class level by designing a multi-scale prompt auxiliary pre-training language model. In the test stage, it calculates the class score matrix for the test sample by using the memory module, thereby efficiently solving the text classification problem in the black box scene, and realizing the automatic text classification technology. BRIEF DESCRIPTION OF DRAWINGS

[0051] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following introduces the drawings of the related technical solutions in the embodiments of the present application or the prior art. It should be understood that the drawings in the following introduction are only for the convenience of clearly describing part of the embodiments of the technical solutions of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the premise that there is no creative labor.

[0052] Figure 1 is the model overall architecture diagram of the black box text classification based on multi-scale prompt memory enhancement in the embodiments of the present application;

[0053] Figure 2 is the flowchart of a black box text classification method based on multi-scale prompt memory enhancement in the embodiments of the present application. DETAILED DESCRIPTION

[0054] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the drawings, wherein the same or similar reference signs represent the same or similar elements or elements with the same or similar functions throughout. The embodiments described below by referring to the drawings are exemplary and are only used to explain the present application, and cannot be understood as a limitation of the present application. For the step numbers in the following embodiments, they are only set for the convenience of description and explanation, and the order between the steps is not limited in any way, and the execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0055] In the description of the present application, it should be understood that the orientation description, such as the orientation or position relationship indicated by up, down, front, back, left, right, etc. is based on the orientation or position relationship shown in the drawings, and is only for the convenience of describing the present application and simplifying the description, and therefore cannot be understood as indicating or implying that the device or element indicated must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation of the present application.

[0056] In the description of the present application, the meaning of several is one or more, the meaning of multiple is two or more than two, greater than, less than, more than, etc. are understood as not including the number, above, below, etc. are understood as including the number. If the first, second is described for the purpose of distinguishing technical features, it cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features or implicitly indicating the order of indicated technical features. In addition, "and / or" describes the association relationship of the associated objects, which means that there can be three kinds of relationships, for example, A and / or B can represent: A exists alone, A and B exist together, and B exists alone. The character " / " generally represents a "or" relationship between the front and rear associated objects.

[0057] In the description of the present application, the words such as arrangement, installation, connection, etc. should be understood broadly unless otherwise explicitly limited, and the skilled in the art can reasonably determine the specific meaning of the above words in the present application in combination with the specific content of the technical solution.

[0058] As shown in Figure 1 and Figure 2 The present embodiment provides a black box text classification method based on multi-scale prompt memory enhancement. The method can efficiently utilize the multi-scale prompts generated by a small number of labeled samples of a downstream task to construct a memory enhancement module, thereby helping the model to improve the text classification ability in a black box scenario through a scoring algorithm. The method has the advantages of no parameters, fast training, high accuracy, and small space overhead when solving the problem of text classification in a black box scenario. The method specifically includes the following steps:

[0059] S1, training phase: a small number of labeled samples of a downstream task are divided into four different scale prompts according to simple permutation and combination. Then concatenate in front of each training sample, extract the corresponding features through the prompt, and store the corresponding labels in the memory module. Calculate the class-level knowledge using the instance-level knowledge stored in the memory module.

[0060] As an optional implementation, step S1 specifically includes steps S11-S13:

[0061] S11, collect text data such as product reviews in the corresponding field, organize user comments on products and scoring attitudes. And clean up the text data such as removing special characters and punctuation, removing stop words, converting to lowercase, etc. For the training set, the training samples need to be labeled according to the requirements.

[0062] S12, arrange and combine the processed training data according to the specified rules to get different scale prompts, that is, to construct a multi-scale prompt module.

[0063] S13, concatenate the training data and the multi-scale prompt to get different levels of instance-level knowledge through the pre-trained language model. And store it in the memory module. Use the memory module to store instance-level knowledge to learn class-level knowledge and store it in the memory module.

[0064] It should be noted that the pre-trained language model can be a pre-trained language model based on Transformer such as BERT, GPT and T5, and the backbone model can also be replaced by various other variant models such as language models based on RNN.

[0065] S2, test phase: The scoring module of the present algorithm calculates the relative distance of the semantics of the test sample and the memory module in the training phase, thereby obtaining the predicted class of the test sample.

[0066] As an optional implementation, step S2 specifically comprises steps S21-S23:

[0067] S21, the same preprocessing is performed on the input test text data to obtain cleaned test data.

[0068] S22, the multi-scale module is used to obtain representations of different scales in the same way as in the training phase.

[0069] S23, the scoring module calculates instance-level scores and class-level scores for the test sample using the memory module, and finally calculates a final score matrix according to the scores of different levels to predict the class of the test sample.

[0070] The above method will be explained and described in detail in combination with specific embodiments.

[0071] The black box text classification method based on multi-scale prompt memory enhancement proposed in this embodiment can extract knowledge of different levels by designing a multi-scale prompt module, and then introduce a class-level knowledge calculation method, and finally design a scoring module that can consider scores of different levels and different levels. In the black box scenario, it provides a new solution for text classification tasks based on pre-trained language models. Compared with previous methods, it has higher data utilization efficiency, smaller time and memory overhead, and also has the advantages of no parameterization and prompt interpretability.

[0072] The black box text classification method based on multi-scale prompt memory enhancement is composed of three key modules, which can use pre-trained language models to solve the text classification task in the black box scenario.

[0073] (1) Single-scale memory module

[0074] In this embodiment, the knowledge stored in the memory module is extracted by the prompt module and the pre-trained language model, and the prompt used is the form of prompt in context learning. The representation is extracted by the following formula:

[0075]

[0076] wherein, is defined as a pre-trained language model, and the parameters and gradient information inside the model cannot be accessed, i.e. a black box scenario. v represents the words of the vocabulary of the pre-trained language model, and its size is the same as that of the vocabulary. represents the context prompt, and π represents the transformation of the test sample x test According to the specified template, it is converted into a specific format. The specific prompt template is constructed as follows:

[0077]

[0078] where, represents a simple concatenation symbol, and the samples are selected from the training set, and represents the i-th sample, and the class of the sample is

[0079] The embodiment of the application constructs a memory module to store the output representation extracted from the training set by combining the pre-training language model with the context prompt. For each training sample, the embodiment of the application connects it to the prompt , where c represents the number of connections of each class context example. The memory module is represented as , which contains the key-value pairs of the corresponding sample output representation, and the key is the output representation and the value is the corresponding label. The memory module stores the instance-level knowledge, and the calculation formula is as follows:

[0080]

[0081] The embodiment of the application proposes that the memory module not only needs to store instance-level knowledge but also needs to store class-level knowledge. In the text classification task, only relying on single instance-level knowledge may not be able to understand. Therefore, the embodiment of the application introduces class-level knowledge in the memory module, and the formula is as follows:

[0082]

[0083] where, j represents the corresponding class, represents the number of instance-level representations stored in the memory module under a single scale.

[0084] (2) Multi-scale prompt module

[0085] Through the construction of the above module, the embodiment of the application successfully realizes the memory module containing instance-level knowledge and class-level knowledge. However, in the few-shot text classification task, due to the limited number of samples available for extracting knowledge, the extracted knowledge may not be typical and diverse. In addition, existing research shows that different sizes of context have a great influence on the prediction of the target word by the pre-training language model. Longer context prompts may provide more comprehensive information, while shorter context prompts may pay more attention to local knowledge. Therefore, when selecting the context prompt, the advantages and disadvantages of different sizes of context prompts need to be weighed to obtain the best prediction effect. Therefore, in order to assist the language model to extract more diverse knowledge to construct the memory module, the embodiment of the application designs a multi-scale prompt. In each class, the embodiment of the application uses hierarchical sampling to select as many diverse examples as possible to form the prompt of each scale. The embodiment of the application uses represents the i-th key-value pair of stored knowledge with scale size c, where c is a scale size from 2 0 to 2 m-1 (m represents the number of multi-scale). The following is the calculation formula of the corresponding key:

[0086]

[0087] Through the above formula, the number of knowledge stored by the memory module is increased to m times the original. Intuitively, it can be regarded as a multi-scale diversity knowledge enhancement module under few-shot learning. And the whole process does not optimize the prompt in any way. Its context examples can be arranged simply, so the calculation efficiency is very high.

[0088] (3) Non-parametric scoring module

[0089] In the scoring module, the embodiment of the present application does not need to be parameterized. It guides the pre-trained language model to comprehensively utilize the different levels of knowledge stored in the memory module to calculate the score matrix S of the test example on each category. The embodiment of the present application first calculates the score matrix S under a single scale prompt , wherein the corresponding memory module is represented as In the reasoning stage, for each test example, the embodiment of the present application calculates the output representation p test by formula (3). test and k i are the output probabilities about the entire vocabulary, which may contain other redundant information. For this purpose, the embodiment of the present application proposes to use Kullback-Leibler (KL) divergence to measure the information difference between the two, and gives the corresponding score. The embodiment of the present application utilizes the instance-level knowledge stored in to calculate the score of x test on the instance level:

[0090]

[0091]

[0092]

[0093] where Top k (D ins ) represents the selection of the top k values with the smallest relative distance, represents the instance-level score of the test sample in class j. Similarly, the score formula of the test sample class level is as follows:

[0094]

[0095] After the instance-level and class-level scores are calculated, the two scores are combined, and the calculation formula is as follows:

[0096]

[0097] wherein, lambda represents the weight size of the class-level information, and the L1 norm is used to normalize the score matrix so that it can be added. The value of lambda can be changed to adjust the importance of the class-level knowledge.

[0098]

[0099] Finally, the embodiment of the application simply weighted averages the scores of different levels to obtain the final score matrix. Then, the class with the highest score of the test sample is determined as its predicted class.

[0100] As another optional implementation, the above-mentioned scoring module can also be implemented by using other conventional machine learning algorithms, such as the nearest neighbor algorithm and support vector machine.

[0101] In summary, the present application has at least the following advantages and beneficial effects compared with the prior art:

[0102] (1) The present application proposes a memory enhancement method based on a pre-trained large language model, which only calls the API of the pre-trained large language model less than ten times to optimize the memory model to realize the downstream text classification task. It does not need to use a larger language model to assist text classification, but only uses a small amount of labeled samples as prompts to assist downstream text classification.

[0103] (2) The present application proposes a memory enhancement method for a black box scenario, which can realize efficient text classification tasks without optimizing any parameters in a black box scenario. It reduces the memory overhead and the cost of manual data labeling, and can quickly adapt to downstream classification tasks.

[0104] (3) The present application stores instance-level information and class-level information of different levels in the memory module to assist the model to complete the downstream text classification task. For example, in the news classification task, simply considering the instance-level article information of military news cannot learn the information shared by the military news class from the class level of military news, so as to efficiently complete the news classification task. Similarly, only considering the shared information of the military news class cannot comprehensively understand the class to which the test sample belongs. The more information the memory module stores, the better the effect it can achieve. Using multi-scale prompts to help extract more rich downstream information will be able to realize more effective text classification.

[0105] (4) The application designs a scoring module capable of comprehensively considering various levels of information, which can comprehensively consider various and representative information to help improve the effect of the text classification task.

[0106] The embodiment also provides a black-box text classification device based on multi-scale prompt memory enhancement, comprising:

[0107] at least one processor;

[0108] at least one memory for storing at least one program;

[0109] When the at least one program is executed by the at least one processor, the at least one processor implements Figure 2 the method.

[0110] The black-box text classification device based on multi-scale prompt memory enhancement can execute the black-box text classification method based on multi-scale prompt memory enhancement provided by the method embodiment, can execute the implementation steps of any combination of the method embodiment, and has the corresponding functions and beneficial effects of the method.

[0111] The embodiment of the application also discloses a computer program product or a computer program, which comprises computer instructions stored in a computer readable storage medium. A processor of a computer device can read the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer device executes Figure 2 the method.

[0112] The embodiment also provides a storage medium storing instructions or programs capable of executing the black-box text classification method based on multi-scale prompt memory enhancement provided by the method embodiment of the application, and when the instructions or programs are executed, the implementation steps of any combination of the method embodiment can be executed, and the corresponding functions and beneficial effects of the method are possessed.

[0113] In some alternative embodiments, the functions / operations mentioned in the block diagram can not occur in the order mentioned in the operation diagram. For example, depending on the functions / operations involved, two blocks shown in succession can actually be executed substantially simultaneously or the blocks can sometimes be executed in reverse order. In addition, the embodiments presented and described in the flowcharts of the application are provided by way of example, with the purpose of providing a more comprehensive understanding of the technology. The disclosed method is not limited to the operations and logical flows presented herein. Alternative embodiments are contemplated in which the order of various operations is changed and in which sub-operations described as part of larger operations are independently executed.

[0114] Furthermore, although the present application is described in the context of functional modules, it is to be understood that one or more of the described functions and / or features can be integrated in a single physical device and / or software module, or one or more functions and / or features can be implemented in separate physical devices or software modules. It will also be appreciated that detailed discussion of the actual implementation of each module is not necessary to an understanding of the application. Rather, the actual implementation of the modules, in combination with their attributes, functions, and internal relationships, are to be understood within the context of the devices disclosed herein. Thus, those skilled in the art with access to patents, scientific journals, and other public sources known by those skilled in the art will be able to practice the application as set forth in the claims without undue experimentation, using ordinary skill in the art along with the present disclosure. It is also to be understood that the specific concepts disclosed are merely illustrative and that the scope of the present application is to be determined by the entire scope of the claims, along with all equivalents of the claims and their equivalents.

[0115] If the functions are implemented in software, the functions can be stored in or implemented as one or more computer program products, which can be incorporated into a computer-readable medium for use by or in connection with an apparatus, method or system as described herein. The computer-readable medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, or a computer-readable signal. The computer-readable medium can be, for example, but is not limited to, volatile or non-volatile memory, a floppy diskette, a compact disc read-only memory (CD-ROM), a magnetic tape, a flash drive, a hard disk drive, a programmable read-only memory (PROM), a random access memory (RAM), a reprogrammable read-only memory (REPROM) or a similar medium.

[0116] The logic and / or steps represented in the flowcharts and / or otherwise described herein, for example, can be embodied in non-transitory computer-readable media, which can direct one or more computing devices to function in a particular manner, such as executing instructions. The instructions executed by the computing devices can be part of an operating system or a separate application, and the instructions can be executed across one or more processors. The logic and / or steps represented in the flowcharts and / or otherwise described herein can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch instructions from a instruction execution system, apparatus, or device and execute instructions. For the purposes of this specification, a "computer-readable medium" can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. The computer-readable medium can be, for example but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device. More specific examples (a non-exhaustive list) of the computer-readable medium include an electrical connection, hard-wired ROM, random access memory (RAM), nonvolatile memory, magnetic or optical diskette, memory chip, firmware, or compact disc read-only memory (CD-ROM).

[0117] More specific examples (a non-exhaustive list) of the computer-readable medium include the following: an electrical connection (electronic) having one or more wires, a portable computer diskette (magnetic), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium can also be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, for example, via optical scanning of the paper or other medium, then compiled, interpreted, or otherwise processed in a suitable manner, if necessary, and then stored in a computer memory.

[0118] It should be understood that aspects of the application can be implemented in hardware, software, firmware or combinations thereof. In the above described embodiments, various steps or methods can be implemented in software or firmware that is stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any of the following technologies, known in the art, or combinations thereof, can be used: a discrete logic circuit having logic gates for implementing logic functions upon data signals, an application specific integrated circuit having appropriate combinational logic gates, a programmable gate array (PGA), a field programmable gate array (FPGA), and / or the like.

[0119] In the above description of the present specification, the description referring to the terms "one embodiment", "another embodiment", or "certain embodiments" or the like means that a specific feature, structure, material or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the present application. The illustrative expressions do not necessarily refer to the same embodiment or example throughout the present specification. Also, the specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0120] Although embodiments of the present application have been shown and described, it would be recognized by those of ordinary skill in the art that various changes, modifications, alternatives, and variations can be made to the embodiments without departing from the spirit and scope of the application, which is defined by the claims and their equivalents.

[0121] The above is a specific description of the preferred embodiments of the present application, but the present application is not limited to the above-described embodiments, and those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present application, and these equivalent modifications or substitutions are included in the scope defined by the claims of the present application.

Claims

1. A black-box text classification method based on multi-scale prompt memory enhancement, characterized in that, The method comprises the following steps: Obtaining text data, processing the text data, and obtaining training data and test data; In the training stage, the training data is arranged and combined according to a preset rule to obtain a multi-scale prompt; after the training data and the multi-scale prompt are spliced, different levels of instance-level knowledge is obtained through a pre-trained language model, and the instance-level knowledge is stored in a memory module; In the test stage, the test data is arranged and combined according to a preset rule to obtain a multi-scale prompt; According to the obtained prompt, the memory module is used to calculate the instance-level score and the class-level score for the test sample, and the final score matrix is calculated according to the two scores of different levels, so as to predict the class of the test sample; The memory module is configured to store output representations extracted from the pre-trained language model combined with the context prompts in the training set; for each training sample, the training sample is concatenated to the back of the prompt , wherein represents the number of category context examples concatenated to each category. The memory module is represented as wherein the key-value pairs contain the output representation of the corresponding sample, the key being the output representation and the value being the corresponding label; The multi-scale prompt is obtained by the following method: In each class, a hierarchical sampling method is used to select examples to form a prompt of each scale; Adopt to represent the size of the scale The first key-value pair of stored knowledge when is the size of the scale from to , represent the number of multi-scale; The formula for calculating the key value is as follows: wherein, represent words of a vocabulary of a pre-trained language model, is a training sample.

2. The black-box text classification method based on multi-scale prompt memory enhancement according to claim 1, characterized in that, Memory module The stored instance-level knowledge computation formula is as follows: wherein, represent words of a vocabulary of a pre-trained language model, is a training sample; represent under a prompt the model output representation obtained under the prompt.

3. The black-box text classification method based on multi-scale prompt memory enhancement according to claim 2, characterized in that, The memory module introduces class-level knowledge, and the formula is as follows: wherein, denotes the corresponding class, denotes the number of instance-level representations stored by the memory module at a single scale, denotes an output of 1 if the input is true, otherwise 0, is the class of the training sample .

4. The black-box text classification method based on multi-scale prompt memory enhancement according to claim 1, characterized in that, The knowledge stored in the memory module is obtained according to the prompt and the pre-trained language model, wherein the prompt is in the form of context learning; and the representation is extracted by the following formula: In the formula, denotes a pre-trained language model; denotes a word of the vocabulary of the pre-trained language model; denotes a context prompt, denotes that the test sample is converted into a specific format according to a preset prompt template; is a placeholder.

5. The method of claim 4, wherein the method is a black-box text classification method based on multi-scale prompt memory enhancement. The construction of the prompt template is as follows: In the formula, represents a simple concatenation symbol, represents the i-th sample selected from the training set, and the category of the sample is ; ; represents the label word of the category .

6. The black-box text classification method based on multi-scale prompt memory enhancement according to claim 2, characterized in that, The example level score for the test sample was calculated by the following way: Example 1 : wherein, denotes the distance of KL divergence computation, denotes the representation of the test sample, denotes the hint with scale size 1, denotes the whole vocabulary; denotes the value before selection the smallest value of relative distance, denotes the instance-level score of the test sample in class j.

7. The method of claim 6, wherein the method is a black-box text classification method based on multi-scale prompt memory enhancement. The class-level score for the test sample was calculated by the following way: In the formula, is the class-level knowledge stored in the memory module.

8. The black-box text classification method based on multi-scale prompt memory enhancement according to claim 7, characterized in that, The score matrix The formula for calculating the score matrix is as follows: wherein is a weight, is norm.

9. A black-box text classification apparatus based on multi-scale cue memory enhancement, characterized in that, It comprises: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the black box text classification method based on multi-scale prompt memory enhancement according to any one of claims 1-8.

Images