Prompt word generation method and related equipment

Abstract

The invention relates to the technical field of cue word generation, and provides a cue word generation method and related equipment. The method comprises the following steps: acquiring a plurality of target texts and a real prompt word corresponding to each target text; generating a plurality of prediction cue words of each target text based on each target text by using an improved LLaMA model; for each target text, determining a positive sample similar to a real cue word and a negative sample not similar to the real cue word from a plurality of predicted cue words of the target text according to the real cue word corresponding to the target text; constructing a contrast loss function according to all the positive samples and all the negative samples, and optimizing the improved LLaMA model by using the contrast loss function to obtain an optimized LLaMA model; and generating a final cue word of the to-be-generated text based on the to-be-generated text by using the optimized LLaMA model. According to the method, the cue word generation accuracy can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of prompt word generation, and in particular to a prompt word generation method and related equipment. Background Art

[0002] In the field of artificial intelligence (AI), a prompt is a text or instruction that provides input to a model to guide it to generate a specific output. It is a text paragraph provided by the user when interacting with the model, which is used to describe the information, answer, text, etc. that the user wants to obtain from the model. The purpose of the prompt is to guide the model to produce the desired response so as to better control the generated output.

[0003] For language models, the prompt can be a short question, a complete paragraph, or a set of instructions, depending on the user's needs and scenario.

[0004] The generative language model is a generative neural network based on large-scale pre-training, built on the transformer model, and uses an attention mechanism. This mechanism is inspired by human vision and can quickly scan the whole world and focus on key areas, so as to better capture the information of long-distance language dependencies. This technology is currently used in the field of text generation to generate prompts, and use prompts to provide input to other models and systems to guide them to generate text or instructions for specific outputs. It is also used in multiple fields, such as intelligent customer service, chatbots, virtual assistants, etc. However, the existing prompt word generation method has the problem of low accuracy in prompt word generation. Summary of the invention

[0005] The present application provides a prompt word generation method and related devices, which can solve the problem of low accuracy in prompt word generation.

[0006] In a first aspect, an embodiment of the present application provides a prompt word generation method, the prompt word generation method comprising:

[0007] Obtain multiple target texts and the real prompt words corresponding to each target text;

[0008] Using the improved LLaMA model, multiple prediction prompt words for each target text are generated based on each target text; the improved LLaMA model is obtained by replacing the first multiple transformers of the LLaMA model with improved lightweight transformers, and the improved lightweight transformer is a lightweight transformer that introduces a singular value decomposition algorithm;

[0009] For each target text, according to the real prompt word corresponding to the target text, determine the positive samples similar to the real prompt word and the negative samples dissimilar to the real prompt word from multiple predicted prompt words of the target text;

[0010] A contrast loss function is constructed according to all positive samples and all negative samples, and the improved LLaMA model is optimized using the contrast loss function to obtain an optimized LLaMA model; the contrast loss function is used to describe the accuracy of all predicted prompt words;

[0011] The optimized LLaMA model is used to generate the final prompt words of the text to be generated based on the text to be generated.

[0012] Optionally, according to the real prompt word corresponding to the target text, determining a positive sample similar to the real prompt word and a negative sample dissimilar to the real prompt word from multiple predicted prompt words of the target text, including:

[0013] For each predicted prompt word of the target text, the distance between the predicted prompt word and the corresponding real prompt word of the target text is calculated;

[0014] Based on all distances, positive samples similar to the true prompt words and negative samples dissimilar to the true prompt words are determined from multiple predicted prompt words of the target text.

[0015] Optionally, based on all distances, positive samples similar to the actual prompt words and negative samples dissimilar to the actual prompt words are determined from multiple predicted prompt words of the target text, including:

[0016] The distance with the smallest value is taken as the first distance, and the predicted prompt word corresponding to the first distance is taken as the positive sample;

[0017] The distance with the largest value is taken as the second distance, and the predicted prompt word corresponding to the second distance is taken as the negative sample.

[0018] Optionally, construct a contrastive loss function based on all positive samples and all negative samples, including:

[0019] The positive sample and negative sample corresponding to the same target text are regarded as a positive-negative sample pair;

[0020] For each positive and negative sample pair, calculate the adjustment parameters of the positive and negative sample pairs respectively;

[0021] Construct a contrastive loss function based on all positive and negative sample pairs and all tuning parameters.

[0022] Optionally, calculate adjustment parameters for positive and negative sample pairs, including:

[0023] By formula:

[0024]

[0025] Calculate adjustment parameter ξ;

[0026] Among them, γ(t) represents the hyperparameter, y- Represents the negative sample in the positive and negative sample pair, y + Represents the positive sample in the positive and negative sample pairs, rank() represents the ranking function, and Var() represents the variance calculation.

[0027] Optionally, the contrastive loss function is:

[0028]

[0029] Among them, L N-Pairs represents the value of the contrast loss function, P represents the set of all positive and negative sample pairs, d() represents the Mahalanobis distance, z x Represents a positive and negative sample pair (y + ,y - ) corresponding to the real prompt word, Represents y - The corresponding test prompt words, Represents y + The corresponding test prompt word.

[0030] Optionally, the improved LLaMA model is optimized using a contrast loss function to obtain an optimized LLaMA model, including:

[0031] Determine whether the value of the contrast loss function is less than the preset loss value;

[0032] If yes, the improved LLaMA model is used as the optimized LLaMA model;

[0033] Otherwise, adjust the parameters of the improved LLaMA model, and return to the step of using the improved LLaMA model to generate multiple prediction prompt words for each target text based on each target text.

[0034] Optionally, the prompt word generating method further includes:

[0035] An improved lightweight transformer is used to decompose the input data of the improved lightweight transformer into multiple non-overlapping groups, and a singular value decomposition algorithm is used to decompose the weight matrix corresponding to each non-overlapping group into a low-rank matrix. All low-rank matrices are used to perform linear changes on all non-overlapping groups to obtain the output of the improved lightweight transformer.

[0036] Optionally, all non-overlapping groups are linearly transformed using all weight matrices to obtain the output of the improved lightweight transformer, including:

[0037] By formula:

[0038] Y=(X1W1 X2W2 … X g W g )

[0039] Calculate the output Y of the improved lightweight converter;

[0040] Among them, X1 represents the first non-overlapping group, W1 represents the first low-rank matrix, X2 represents the second non-overlapping group, W2 represents the second low-rank matrix, X g represents the g-th non-overlapping group, W g represents the g-th low-rank matrix.

[0041] In a second aspect, an embodiment of the present application provides a prompt word generating device, comprising:

[0042] An acquisition module is used to acquire multiple target texts and the real prompt words corresponding to each target text;

[0043] The first generation module uses an improved LLaMA model to generate multiple prediction prompt words for each target text based on each target text; the improved LLaMA model is obtained by replacing the first multiple transformers of the LLaMA model with improved lightweight transformers, and the improved lightweight transformer is a lightweight transformer that introduces a singular value decomposition algorithm;

[0044] A determination module determines, for each target text, positive samples similar to the real prompt words and negative samples dissimilar to the real prompt words from multiple predicted prompt words of the target text according to the real prompt words corresponding to the target text;

[0045] The optimization module constructs a contrast loss function based on all positive samples and all negative samples, and uses the contrast loss function to optimize the improved LLaMA model to obtain the optimized LLaMA model; the contrast loss function is used to describe the accuracy of all predicted prompt words;

[0046] The second generation module uses the optimized LLaMA model to generate the final prompt words of the text to be generated based on the text to be generated.

[0047] In a third aspect, an embodiment of the present application provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the prompt word generation method when executing the computer program.

[0048] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the above-mentioned prompt word generation method is implemented.

[0049] The above solution of the present application has the following beneficial effects:

[0050] In an embodiment of the present application, multiple target texts and real prompt words corresponding to each target text are obtained, and then multiple predicted prompt words of each target text are generated based on each target text using an improved LLaMA model. Then, for each target text, positive samples similar to the real prompt words and negative samples dissimilar to the real prompt words are determined from the multiple predicted prompt words of the target text according to the real prompt words corresponding to the target text. Then, a contrast loss function is constructed based on all positive samples and all negative samples, and the improved LLaMA model is optimized using the contrast loss function to obtain an optimized LLaMA model. Finally, the optimized LLaMA model is used to generate the final prompt words of the text to be generated based on the text to be generated. Among them, improving the LLaMA model can effectively improve the performance of the LLaMA model, and using the LLaMA model with improved performance to generate the predicted prompt words of the target text can improve the accuracy of the predicted prompt words. By constructing a contrast loss function to optimize the improved LLaMA model, the performance of the LLaMA model can be further improved, so that the accuracy of the final prompt words generated using the LLaMA model is further improved.

[0051] Other beneficial effects of the present application will be described in detail in the subsequent specific implementation section. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0053] Figure 1 A flowchart of a method for generating a prompt word provided in an embodiment of the present application;

[0054] Figure 2 A schematic diagram of the structure of a prompt word generating device provided in an embodiment of the present application;

[0055] Figure 3 A schematic diagram of the structure of a terminal device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0056] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0057] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0058] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0059] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0060] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0061] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0062] In view of the problem of low accuracy of existing prompt word generation, an embodiment of the present application provides a prompt word generation method, which obtains multiple target texts and real prompt words corresponding to each target text, and then uses an improved LLaMA model to generate multiple predicted prompt words for each target text based on each target text, and then for each target text, according to the real prompt words corresponding to the target text, from the multiple predicted prompt words of the target text, positive samples similar to the real prompt words and negative samples dissimilar to the real prompt words are determined, and then a contrast loss function is constructed according to all positive samples and all negative samples, and the improved LLaMA model is optimized by using the contrast loss function to obtain an optimized LLaMA model, and finally the optimized LLaMA model is used to generate the final prompt words of the text to be generated based on the text to be generated. Among them, improving the LLaMA model can effectively improve the performance of the LLaMA model. Using the LLaMA model with improved performance to generate predicted prompt words for the target text can improve the accuracy of the predicted prompt words. By optimizing the improved LLaMA model by constructing a contrast loss function, the performance of the LLaMA model can be further improved, so that the accuracy of the final prompt words generated by the LLaMA model is further improved.

[0063] Next, the prompt word generation method provided in this application is exemplified.

[0064] like Figure 1 As shown, the prompt word generation method provided by the present application includes the following steps:

[0065] Step 11, obtaining multiple target texts and the real prompt words corresponding to each target text.

[0066] In some embodiments of the present application, the target text and the corresponding real prompt words can be obtained by accessing a public prompt word library.

[0067] In some embodiments of the present application, after obtaining the target text, the target text can be preprocessed, such as using an open source large model to perform data augmentation operations on the target text to increase the diversity of vocabulary and sentence structure to obtain augmented text, retaining the original key concepts and semantic information, and performing fidelity filtering on the augmented text to filter out unfaithful augmented texts, and using balanced sampling to control the ratio of target text and augmented text. Finally, all sampled target texts and all augmented texts are used as target texts for calculation in the next step.

[0068] Step 12: Generate multiple prediction prompt words for each target text based on each target text using the improved LLaMA model.

[0069] The above-mentioned improved Large Language Model MetaAI (LLaMA) model is obtained by replacing the first multiple transformers of the LLaMA model with improved lightweight transformers (delighT, i.e., lightweight transformers), and the improved lightweight transformers are lightweight transformers that introduce singular value decomposition algorithms. For example, for an LLaMA model formed by stacking 32 layers of transformers, the transformers from layers 1 to 6 starting from the model input are replaced with improved lightweight transformers.

[0070] In some embodiments of the present application, the target text is input into the improved LLaMA model for multiple processings. After each processing, the improved LLaMA model outputs a predicted prompt word of the target text.

[0071] It should be noted that when the improved lightweight converter in the improved LLaMA model processes the input data, the improved lightweight converter is used to decompose the input data of the improved lightweight converter into multiple non-overlapping groups, and the weight matrix corresponding to each non-overlapping group is decomposed into a low-rank matrix using a singular value decomposition algorithm, and all low-rank matrices are used to perform linear changes on all non-overlapping groups to obtain the output of the improved lightweight converter. Specifically, the improved LLaMA model is expressed by the formula:

[0072] Y=(X1W1 X2W2 … X g W g )

[0073] Calculate the output Y of the improved lightweight converter.

[0074] Among them, X1 represents the first non-overlapping group, W1 represents the first low-rank matrix, X2 represents the second non-overlapping group, W2 represents the second low-rank matrix, X g represents the g-th non-overlapping group, W g represents the g-th low-rank matrix.

[0075] The expression of singular value transformation is W≈UΣV T , where W represents the weight matrix, U and V both represent orthogonal matrices, ∑ represents a diagonal matrix, () T represents the transpose operation, U∑V T is the result of singular value transformation, which contains multiple singular values. All singular values ​​are sorted from large to small, and only the first multiple singular values ​​in the sorted result are retained to obtain a low-rank matrix.

[0076] Exemplarily, the attention mechanism in the improved LLaMA model can be replaced with Flash attention, and a memory pool can be preset to dynamically allocate video memory at runtime according to the length of the input sequence and the computing requirements, so that Flash attention can be allocated and released according to actual needs, and a model that can dynamically guide the attention mechanism such as a content-aware mixer can be used to dynamically adjust the sparsity and sparse position of the attention calculation according to the learning situation of the model and the data distribution. The process of the improved LLaMA model processing the target text is specifically as follows: the text input is passed through the embedding layer (embedding) to obtain an embedded vector, the embedded vector is passed through the normalization layer, and the normalized embedded vector is input into Flash attention. Next, the vector output by Flash attention is passed through the residual connection and the normalization layer again, and then through the feedforward activation function (FFS, FeedForward SwiGLU) layer. In this layer, an improved lightweight transformer architecture is used to reduce the number of parameters of the model as a whole without losing accuracy. Finally, the vector is normalized, mapped by a linear layer, and distributed by softmax probability to obtain the final output.

[0077] It is worth mentioning that in the traditional lightweight converter, after the input data is decomposed into multiple non-overlapping groups, the weight matrix is ​​directly used to perform linear changes on the non-overlapping groups. In this application, the singular value decomposition algorithm is added to decompose the weight matrix into a low-rank matrix, which reduces the number of parameters and effectively improves the computational efficiency of the lightweight converter.

[0078] Step 13, for each target text, according to the real prompt word corresponding to the target text, determine the positive sample similar to the real prompt word and the negative sample dissimilar to the real prompt word from the multiple predicted prompt words of the target text.

[0079] In some embodiments of the present application, the above step of determining, according to the real prompt word corresponding to the target text, a positive sample similar to the real prompt word and a negative sample dissimilar to the real prompt word from multiple predicted prompt words of the target text is specifically:

[0080] In the first step, for each predicted prompt word of the target text, the distance between the predicted prompt word and the corresponding true prompt word of the target text is calculated.

[0081] Exemplarily, the above distance can be calculated using a cosine similarity calculation formula.

[0082] In the second step, based on all distances, positive samples similar to the true prompt words and negative samples dissimilar to the true prompt words are determined from multiple predicted prompt words of the target text.

[0083] The distance with the smallest value is taken as the first distance, and the predicted prompt word corresponding to the first distance is taken as the positive sample.

[0084] The distance with the largest value is taken as the second distance, and the predicted prompt word corresponding to the second distance is taken as the negative sample.

[0085] Exemplarily, the target text has 5 predicted prompt words, and the distances between them and the real prompt words are 1.3, 2.1, 2.8, 3.2, and 0.7 respectively. The predicted prompt word corresponding to 3.2 is used as a negative sample, and the predicted prompt word corresponding to 0.7 is used as a positive sample.

[0086] Step 14: construct a contrast loss function based on all positive samples and all negative samples, and use the contrast loss function to optimize the improved LLaMA model to obtain an optimized LLaMA model.

[0087] The above contrast loss function is used to describe the accuracy of all predicted prompt words.

[0088] In some embodiments of the present application, the above steps of constructing a contrast loss function according to all positive samples and all negative samples, and optimizing the improved LLaMA model using the contrast loss function to obtain the optimized LLaMA model are specifically as follows:

[0089] In the first step, the positive samples and negative samples corresponding to the same target text are regarded as a positive-negative sample pair.

[0090] In the second step, for each positive and negative sample pair, the adjustment parameters of the positive and negative sample pairs are calculated.

[0091] Specifically, through the formula:

[0092]

[0093] Calculate the tuning parameter ξ.

[0094] Among them, γ(t) represents the hyperparameter, y - Represents the negative sample in the positive and negative sample pair, y + Represents the positive sample in the positive and negative sample pairs, rank() represents the ranking function, and Var() represents the variance calculation.

[0095] In the third step, a contrastive loss function is constructed based on all positive and negative sample pairs and all adjustment parameters.

[0096] Specifically, the contrast loss function is:

[0097]

[0098] Among them, L N-Pairsrepresents the value of the contrast loss function, P represents the set of all positive and negative sample pairs, d() represents the Mahalanobis distance, z x Represents a positive and negative sample pair (y + ,y - ) corresponding to the real prompt word, Represents y - The corresponding test prompt words, Represents y + The corresponding test prompt word.

[0099] The fourth step is to determine whether the value of the contrast loss function is less than the preset loss value.

[0100] If yes, the improved LLaMA model is used as the optimized LLaMA model.

[0101] Otherwise, adjust the parameters of the improved LLaMA model, and return to the step of using the improved LLaMA model to generate multiple prediction prompt words for each target text based on each target text.

[0102] Exemplarily, if the value of the contrast loss function is 1.6, which is greater than the preset loss value of 0.9, the parameters of the improved LLaMA model are adjusted, and the step of using the improved LLaMA model to generate multiple predicted prompt words for each target text based on each target text is returned. At this time, the value of the contrast loss function is 0.7, which is less than the preset loss value, and the improved LLaMA model at this time is used as the optimized LLaMA model.

[0103] It is worth mentioning that by constructing a contrast loss function to optimize the improved LLaMA model, the performance of the LLaMA model can be further improved.

[0104] Step 15, using the optimized LLaMA model, generating the final prompt word of the text to be generated based on the text to be generated.

[0105] The above-mentioned text to be generated is the text for which the corresponding prompt word needs to be generated. The prompt word can be used to guide other models to generate the results required by the user. For example, in the field of dynamic image generation, the text to be generated is the content description of the dynamic image to be generated. The prompt word is generated based on the generated text, and then a model capable of generating dynamic images (such as GTP4) is used to guide the model to generate a dynamic image that meets the content description using the prompt word.

[0106] Specifically, the text to be generated is input into the optimized LLaMA model, and the optimized LLaMA model outputs the final prompt word of the text to be generated.

[0107] For example, the text to be generated is "gloomy and damp riverbank, waterwaves lapping at the shore, distant sky with a hazy gray mist. Rong Gote stands by the river, looking at the turbulent river with a olemn gaze. Behind is a simple wooden shed, which is a temporary station for retrieving corpses." The corresponding prompt words generated by the optimized LLaMA model are "gloomy and damp riverbank, waterwaves lapping at the shore, distant sky with a hazy gray mist. Rong Gote stands by the river, looking at the turbulent river with a olemn gaze. Behind is a simple wooden shed, which is a temporary station for retrieving corpses.".

[0108] It is worth mentioning that improving the LLaMA model can effectively improve the performance of the LLaMA model. Using the LLaMA model with improved performance to generate predicted prompt words for the target text can improve the accuracy of the predicted prompt words. By optimizing the improved LLaMA model by constructing a contrast loss function, the performance of the LLaMA model can be further improved, and the accuracy of the final prompt words generated by the LLaMA model can be further improved.

[0109] The following is an exemplary description of the prompt word generating device provided in the present application.

[0110] like Figure 2 As shown, the embodiment of the present application provides a prompt word generating device, and the prompt word generating device 200 includes:

[0111] An acquisition module 201 acquires a plurality of target texts and a real prompt word corresponding to each target text;

[0112] The first generation module 202 generates multiple prediction prompt words for each target text based on each target text using an improved LLaMA model; the improved LLaMA model is obtained by replacing the first multiple transformers of the LLaMA model with improved lightweight transformers, and the improved lightweight transformer is a lightweight transformer that introduces a singular value decomposition algorithm;

[0113] The determination module 203 determines, for each target text, a positive sample similar to the real prompt word and a negative sample dissimilar to the real prompt word from multiple predicted prompt words of the target text according to the real prompt word corresponding to the target text;

[0114] The optimization module 204 constructs a contrast loss function according to all positive samples and all negative samples, and optimizes the improved LLaMA model using the contrast loss function to obtain an optimized LLaMA model; the contrast loss function is used to describe the accuracy of all predicted prompt words;

[0115] The second generating module 205 uses the optimized LLaMA model to generate the final prompt words of the text to be generated based on the text to be generated.

[0116] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0117] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0118] like Figure 3 As shown, an embodiment of the present application provides a terminal device. The terminal device D10 of this embodiment includes: at least one processor D100 ( Figure 3 Only one processor is shown in the figure), a memory D101, and a computer program D102 stored in the memory D101 and executable on the at least one processor D100, wherein the processor D100 implements the steps of any of the above-mentioned method embodiments when executing the computer program D102.

[0119] Specifically, when the processor D100 executes the computer program D102, it obtains multiple target texts and the real prompt words corresponding to each target text, and then uses the improved LLaMA model to generate multiple predicted prompt words for each target text based on each target text, and then for each target text, according to the real prompt words corresponding to the target text, from the multiple predicted prompt words of the target text, determine the positive samples similar to the real prompt words and the negative samples dissimilar to the real prompt words, and then construct a contrast loss function according to all positive samples and all negative samples, and optimize the improved LLaMA model using the contrast loss function to obtain an optimized LLaMA model, and finally use the optimized LLaMA model to generate the final prompt words of the text to be generated based on the text to be generated. Among them, improving the LLaMA model can effectively improve the performance of the LLaMA model, and using the LLaMA model with improved performance to generate the predicted prompt words of the target text can improve the accuracy of the predicted prompt words, and optimizing the improved LLaMA model by constructing the contrast loss function can further improve the performance of the LLaMA model, so that the accuracy of the final prompt words generated by the LLaMA model is further improved.

[0120] The processor D100 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0121] In some embodiments, the memory D101 may be an internal storage unit of the terminal device D10, such as a hard disk or memory of the terminal device D10. In other embodiments, the memory D101 may also be an external storage device of the terminal device D10, such as a plug-in hard disk, a smart memory card (SMC, SmartMedia Card), a secure digital (SD, Secure Digital) card, a flash card (Flash Card), etc. equipped on the terminal device D10. Further, the memory D101 may also include both an internal storage unit of the terminal device D10 and an external storage device. The memory D101 is used to store an operating system, an application program, a boot loader (BootLoader), data and other programs, such as the program code of the computer program, etc. The memory D101 may also be used to temporarily store data that has been output or is to be output.

[0122] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0123] An embodiment of the present application provides a computer program product. When the computer program product is run on a terminal device, the terminal device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0124] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium that can carry the computer program code to the prompt word generation method device / terminal device. For example, a USB flash drive, a mobile hard disk, a disk or an optical disk.

[0125] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0126] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0127] The above is a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles described in the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for generating a prompt word, characterized in that: include: Acquire multiple target texts and real prompt words corresponding to each of the target texts; Using an improved LLaMA model, multiple prediction prompt words for each target text are generated based on each target text; the improved LLaMA model is obtained by replacing the first multiple transformers of the LLaMA model with improved lightweight transformers, and the improved lightweight transformer is a lightweight transformer that introduces a singular value decomposition algorithm; For each target text, according to the real prompt word corresponding to the target text, determine a positive sample similar to the real prompt word and a negative sample dissimilar to the real prompt word from multiple predicted prompt words of the target text; Constructing a contrast loss function according to all positive samples and all negative samples, and optimizing the improved LLaMA model using the contrast loss function to obtain an optimized LLaMA model; the contrast loss function is used to describe the accuracy of all predicted prompt words; The optimized LLaMA model is used to generate a final prompt word of the text to be generated based on the text to be generated.

2. The method for generating prompt words according to claim 1, characterized in that: The step of determining, according to the real prompt word corresponding to the target text, a positive sample similar to the real prompt word and a negative sample dissimilar to the real prompt word from a plurality of predicted prompt words of the target text comprises: For each predicted prompt word of the target text, respectively, calculating the distance between the predicted prompt word and the real prompt word corresponding to the target text; Based on all the distances, positive samples similar to the true prompt words and negative samples dissimilar to the true prompt words are determined from the multiple predicted prompt words of the target text.

3. The method for generating prompt words according to claim 2, characterized in that: The step of determining, based on all distances, a positive sample similar to the real prompt word and a negative sample dissimilar to the real prompt word from a plurality of predicted prompt words of the target text comprises: The distance with the smallest value is taken as the first distance, and the predicted prompt word corresponding to the first distance is taken as the positive sample; The distance with the largest value is taken as the second distance, and the predicted prompt word corresponding to the second distance is taken as a negative sample.

4. The method for generating prompt words according to claim 1, characterized in that: The contrast loss function is constructed based on all positive samples and all negative samples, including: The positive sample and negative sample corresponding to the same target text are regarded as a positive-negative sample pair; For each of the positive and negative sample pairs, respectively, calculating the adjustment parameters of the positive and negative sample pairs; Construct a contrastive loss function based on all positive and negative sample pairs and all tuning parameters.

5. The method for generating prompt words according to claim 4, characterized in that: The calculating the adjustment parameters of the positive and negative sample pairs includes: By formula: Calculate adjustment parameter ξ; Among them, γ(t) represents the hyperparameter, y - Represents the negative sample in the positive and negative sample pair, y + Represents the positive sample in the positive and negative sample pairs, rank() represents the ranking function, and Var() represents the variance calculation.

6. The method for generating prompt words according to claim 5, characterized in that: The contrast loss function is: Among them, L N-Pairs represents the value of the contrast loss function, P represents the set of all positive and negative sample pairs, d( ) represents the Mahalanobis distance, z x Represents a positive and negative sample pair (y + ,y - ) corresponding to the real prompt word, Represents y - The corresponding test prompt words, Represents y + The corresponding test prompt word.

7. The method for generating prompt words according to claim 6, characterized in that: The step of optimizing the improved LLaMA model by using the contrast loss function to obtain an optimized LLaMA model includes: Determine whether the value of the contrast loss function is less than a preset loss value; If yes, the improved LLaMA model is used as the optimized LLaMA model; Otherwise, the parameters of the improved LLaMA model are adjusted, and the process returns to the step of using the improved LLaMA model to generate a plurality of prediction prompt words for each target text based on each target text.

8. The method for generating prompt words according to claim 1, characterized in that: The prompt word generation method also includes: The improved lightweight converter is used to decompose the input data of the improved lightweight converter into multiple non-overlapping groups, and the weight matrix corresponding to each non-overlapping group is decomposed into a low-rank matrix using a singular value decomposition algorithm. All low-rank matrices are used to perform linear changes on all non-overlapping groups to obtain the output of the improved lightweight converter.

9. The method for generating prompt words according to claim 8, characterized in that: The method of using all weight matrices to linearly change all non-overlapping groups to obtain the output of the improved lightweight converter includes: By formula: Y=(X1W1 X2W2…X g W g ) Calculate the output Y of the improved lightweight converter; Among them, X1 represents the first non-overlapping group, W1 represents the first low-rank matrix, X2 represents the second non-overlapping group, W2 represents the second low-rank matrix, X g represents the g-th non-overlapping group, W g represents the g-th low-rank matrix.

10. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the prompt word generating method according to any one of claims 1 to 9 is implemented.

Images