TF-IDF and cross entropy-based cue word compression method and system

Abstract

The invention discloses a cue word compression method and system based on TF-IDF and cross entropy, belongs to the technical field of large model cue word compression, and aims to solve the problems that redundant information is introduced into long cue words, the model efficiency is reduced and the cost is increased. To-be-compressed content is divided into sentences at the sentence level and then converted into embedded vectors, and the Euclidean distance is calculated in combination with problem vectors so as to screen related sentences; calculating a TF-IDF value at the word level through a word frequency and an inverse document frequency to extract keywords and recombine sentences; and selecting a reference model and a basic model at the Token level, identifying the key Token based on a cross entropy loss difference value, and splicing the key Token in sequence to generate a compressed cue word. According to the method, a complex calculation structure is avoided, the inference efficiency is improved while the semantic integrity is maintained, and the resource consumption is reduced.

Description

Technical Field

[0001] The present invention belongs to the technical field of large model prompt word compression, and in particular relates to a prompt word compression method and system based on TF-IDF and cross entropy. Background Art

[0002] With the advancement of large-scale model technology, users are increasingly using more detailed prompts to fine-tune model behavior, such as limiting the scope of responses, emphasizing specific data, or specifying output formats. However, overly long prompts introduce redundant information, making it difficult for the model to focus on the core problem, reducing inference efficiency, and increasing training costs. For companies relying on third-party APIs, excessively long prompts can significantly increase API call costs. Existing prompt compression technologies have significant limitations: while some existing technologies achieve dynamic compression at the sentence and word levels, they lack fine-grained optimization at the token level. Others extract text fragments based on relevance, similarly failing to address token-level compression. Some methods employ syntax tree construction and recursive pruning, which is complex and computationally expensive. Others rely solely on attention mechanisms for token scoring, which struggles to accurately identify key information and is burdensome for processing long texts. These existing technologies either lack fine-grained compression capabilities or are difficult to implement due to their complex computational structures. Consequently, they fail to balance compression efficiency and semantic integrity in real-world applications. Summary of the Invention

[0003] To solve the above technical problems, the present invention proposes a prompt word compression method and system based on TF-IDF and cross entropy to solve the problems existing in the above-mentioned prior art.

[0004] In a first aspect, to achieve the above-mentioned objectives, the present invention provides a prompt word compression method based on TF-IDF and cross entropy, comprising the following steps:

[0005] Divide the prompt word into the question part and the content to be compressed, and keep the question part intact;

[0006] Perform sentence-level compression on the content to be compressed: convert the sentences to be compressed and the questions into embedding vectors, select semantically related sentences based on Euclidean distance and retain them in their original order;

[0007] Perform word-level compression on the sentence-level compression results: extract keywords based on TF-IDF values and reorganize the sentences in their original order;

[0008] Perform token-level compression on the word-level compression results: Select the reference model and the base model, calculate the difference in cross-entropy loss between the two models for the processed text, identify key tokens based on the difference, and concatenate them in the original order;

[0009] Output the concatenated compressed prompt words.

[0010] Optionally, the process of dividing the content to be compressed includes:

[0011] Identify the background information, example, and question sections of a prompt;

[0012] Keep the answers to the questions and examples intact.

[0013] The content to be compressed is determined to be the background information portion and the background information in the example portion.

[0014] Optionally, the sentence-level compression process includes:

[0015] Treat compressed content as sentences according to punctuation marks;

[0016] Convert the sentence into an embedding vector after segmentation;

[0017] Convert the problem into an embedding vector;

[0018] Calculate the Euclidean distance between each sentence vector and the question vector;

[0019] Sorting by distance keeps related sentences and maintains the original order.

[0020] Optionally, the word-level compression process includes:

[0021] Count the frequency of each word in the retained sentence;

[0022] Calculate the inverse document frequency of each word;

[0023] The TF-IDF value is obtained by fusing the term frequency and the inverse document frequency;

[0024] Sort by TF-IDF value, retain the keywords and reorganize the sentences in the original order.

[0025] Optionally, the token-level compression process includes:

[0026] Select reference model and base model;

[0027] Input the word-level compression results into the reference model and the base model;

[0028] Calculate the cross entropy loss of the reference model and the cross entropy loss of the base model;

[0029] Calculate the difference between the two losses;

[0030] Identify the key tokens based on the difference and concatenate them in the original order.

[0031] Optionally, the reference model is a high-performance model optimized for the task domain;

[0032] The basic model is a low-parameter model of the same series.

[0033] In a second aspect, the present invention further provides a prompt word compression system based on TF-IDF and cross entropy, which is used to implement a prompt word compression method based on TF-IDF and cross entropy, and the system includes:

[0034] The content segmentation module is used to identify the question part and the content to be compressed in the prompt word, and retain the question part intact;

[0035] The semantic filtering module is used to perform sentence-level compression on the content to be compressed. It converts the sentences and questions to be compressed into embedding vectors, filters semantically related sentences based on Euclidean distance, and retains them in their original order.

[0036] The keyword extraction module is used to perform word-level compression on the sentence-level compression results: extract keywords based on TF-IDF values and reorganize the sentences in their original order;

[0037] The sensitivity analysis module is used to perform token-level compression on word-level compression results. It selects a reference model and a base model, calculates the difference in cross-entropy loss between the two, identifies key tokens based on the difference, and concatenates them in the original order.

[0038] The output generation module is used to output compressed prompt words.

[0039] Optionally, the content division module includes:

[0040] A background recognition unit for recognizing the background information part, the example part and the question part in the prompt word;

[0041] A content retention unit is used to completely retain the answer content of the question part and the example part;

[0042] The compression range determining unit is configured to determine that the content to be compressed is the background information in the background information part and the example part.

[0043] In a third aspect, the present invention further provides a computer terminal device, comprising:

[0044] one or more processors;

[0045] a memory, coupled to the processor, for storing one or more programs;

[0046] When the one or more programs are executed by the one or more processors, the one or more processors implement a prompt word compression method based on TF-IDF and cross entropy.

[0047] In a fourth aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it implements a prompt word compression method based on TF-IDF and cross entropy.

[0048] Compared with the prior art, the present invention has the following advantages and technical effects:

[0049] The present invention provides a prompt word compression method and system based on TF-IDF and cross entropy. The present invention significantly improves the prompt word processing efficiency through three-layer progressive compression: at the sentence level, irrelevant content is screened out based on semantic similarity to reduce information redundancy; at the word level, key semantic units are extracted in combination with TF-IDF values to shorten the text length; at the token level, sensitive tokens are identified using cross entropy loss differences to achieve fine-grained optimization. This solution does not require a complex attention mechanism or pre-trained model, avoids the computational burden of syntax tree construction, and maintains semantic integrity. Through model-independent design, it adapts to diverse application scenarios, reduces API call costs, improves inference speed, and prevents memory overflow problems. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The accompanying drawings, which constitute part of the present invention, are provided to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are provided to explain the present invention and do not constitute an undue limitation of the present invention. In the accompanying drawings:

[0051] Figure 1 This is a flowchart of a prompt word compression system based on TF-IDF and cross entropy according to an embodiment of the present invention;

[0052] Figure 2 This is a diagram showing the F1 comparison results after the prompt word is compressed according to an embodiment of the present invention. DETAILED DESCRIPTION

[0053] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments of the present invention can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0054] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0055] Example 1

[0056] like Figure 1 As shown, this embodiment provides a prompt word compression method based on TF-IDF and cross entropy, including:

[0057] Divide the prompt word into the question part and the content to be compressed, and keep the question part intact;

[0058] Perform sentence-level compression on the content to be compressed: convert the sentences to be compressed and the questions into embedding vectors, select semantically related sentences based on Euclidean distance and retain them in their original order;

[0059] Perform word-level compression on the sentence-level compression results: extract keywords based on TF-IDF values and reorganize the sentences in their original order;

[0060] Perform token-level compression on the word-level compression results: Select the reference model and the base model, calculate the difference in cross-entropy loss between the two models for the processed text, identify key tokens based on the difference, and concatenate them in the original order;

[0061] Output the concatenated compressed prompt words.

[0062] Specifically, S1. Segment the paragraph to be compressed into sentences according to punctuation marks and convert each sentence into a corresponding embedding vector. Simultaneously, the question is also converted into an embedding vector and the semantic similarity of the question is obtained using Euclidean distance in the vector space, thereby filtering out the core content related to the question.

[0063] S2. After filtering out relevant sentences, we calculate the importance score of each word by combining word frequency and inverse document frequency methods, thereby extracting keywords with key semantic functions in sentences related to the question, and achieving information simplification at the word level;

[0064] S3. At the token level, we use the cross-entropy loss difference between the basic model and the reference model to perform sensitivity analysis on the model output, identify key tokens that have a significant impact on the final result, further remove redundant information, complete fine-grained compression, and ultimately generate a compact and semantically complete prompt word.

[0065] As an implementation method of this embodiment, the process of dividing the content to be compressed includes:

[0066] Identify the background information, example, and question sections of a prompt;

[0067] Keep the answers to the questions and examples intact.

[0068] The content to be compressed is determined to be the background information portion and the background information in the example portion.

[0069] Specifically, S1.1, first, the prompt word content is structurally divided to identify the background information, example part and question part. For large language models, the question part has a strong correlation with the final output result. If the keywords in the question are compressed or the semantic expression is unclear, it may cause deviations in model understanding, thereby affecting the accuracy of the output. Therefore, the complete question part is retained during the compression process, and no task compression processing is performed on it. At the same time, the example part mainly plays the role of guiding the model to understand the output format. Therefore, in this part, the example answer content is fully retained to ensure that the model can accurately refer to the expected output structure. Based on the above analysis, it is finally determined that the content that needs to be compressed is the background information part, and the background information content in the example part;

[0070] As an implementation method in this embodiment, the sentence-level compression process includes:

[0071] Treat compressed content as sentences according to punctuation marks;

[0072] Convert the sentence into an embedding vector after segmentation;

[0073] Convert the problem into an embedding vector;

[0074] Calculate the Euclidean distance between each sentence vector and the question vector;

[0075] Sorting by distance keeps related sentences and maintains the original order.

[0076] Specifically, S1.2, for the content to be compressed, sentence processing is performed according to punctuation marks, and each sentence to be compressed is converted into an embedding vector, denoted as {S1, S2, ..., Sn}; at the same time, the problem is also converted into an embedding vector, denoted as T.

[0077] S1.3. Calculate the Euclidean distance between the embedding vector {S1, S2, …, Sn} of each sentence to be compressed and the question vector T in the vector space. The calculation formula is as follows:

[0078]

[0079] Where d represents the Euclidean distance between two vectors; n is the number of dimensions of the vector; Represents the first vector, that is, the embedding vector with compressed sentence, the value in the kth dimension; Represents the value of the second vector, the problem vector, in the kth dimension.

[0080] By calculating the Euclidean distance between each sentence and the question and sorting them, we filter out sentences with high semantic relevance to the question in the background information, remove the remaining redundant content that is irrelevant to the question, and retain the remaining sentences related to the question in the original order.

[0081] As an implementation method in this embodiment, the word-level compression process includes:

[0082] Count the frequency of each word in the retained sentence;

[0083] Calculate the inverse document frequency of each word;

[0084] The TF-IDF value is obtained by fusing the term frequency and the inverse document frequency;

[0085] Sort by TF-IDF value, retain the keywords and reorganize the sentences in the original order.

[0086] Specifically, S2.1, based on the sentences retained in S1, uses the word frequency analysis method to identify keywords on a sentence-by-sentence basis. Specifically, this method counts the frequency of each word appearing in the current sentence to select high-frequency words as candidate keywords;

[0087] S2.2, also based on the sentences retained in S1, use the inverse word frequency method to identify keywords in each sentence. Inverse word frequency is an important indicator to measure the global discrimination ability of words, reflecting the prevalence of word t in all sentences. Its calculation formula is as follows:

[0088]

[0089] This method can identify words that appear less frequently throughout the corpus but have strong discriminative power. These words have high inverse frequency values, indicating that they are more representative of the information expressed in a specific sentence. Conversely, words with low inverse frequency values are considered to be widely distributed throughout the corpus and contain less information. Words with high inverse frequency values are selected as candidate keywords.

[0090] S2.3. Based on the two previous methods, to further improve the accuracy and representativeness of keyword recognition, this paper combines term frequency and inverse term frequency to comprehensively evaluate the importance of words using the TF-IDF (Term Frequency-Inverse Document Frequency) method. The TF-IDF weight reflects the local importance of a word in the current sentence and its global discriminability in the entire corpus. The calculation formula of this method is as follows:

[0091]

[0092] in, is the word frequency-inverse word frequency, is the word frequency, TF-IDF is the inverse document frequency. It effectively identifies words that appear frequently in the current sentence but are relatively rare in the entire corpus, thereby selecting more representative and discriminative keywords as the final keyword set. These keywords are then arranged and combined according to their order in the original sentence to form a compressed sentence, thus concisely retaining key information while also streamlining the text.

[0093] As an implementation method in this embodiment, the token-level compression process includes:

[0094] Select reference model and base model;

[0095] Input the word-level compression results into the reference model and the base model;

[0096] Calculate the cross entropy loss of the reference model and the cross entropy loss of the base model;

[0097] Calculate the difference between the two losses;

[0098] Identify the key tokens based on the difference and concatenate them in the original order.

[0099] Specifically, S3.1, select a reference model with strong performance and a basic model as the basis for evaluating the importance of tokens.

[0100] S3.2. After completing the comprehensive deletion of non-keywords in S2, the processed sentences are input into the reference model and the basic model for inference, and the cross-entropy loss (Cross-Entropy Loss) output on the two models is calculated and recorded as and .

[0101] S3.3. Calculate the difference between the cross entropy loss values obtained from the reference model and the base model. The formula is as follows:

[0102]

[0103] This difference reflects the degree of influence each token has on the reference model's output. A larger difference indicates a more critical token to the reference model's performance. Therefore, these tokens are identified as more critical and retained, while the remaining non-critical tokens are deleted. The remaining important tokens are arranged according to their order in the original sentence.

[0104] S3.4. The various parts of the content that have been compressed at the token level are concatenated in sequence according to the original word order, and finally a compressed prompt word with a complete structure and coherent semantics is generated.

[0105] As an implementation method in this embodiment, the reference model is a high-performance model optimized for the task domain;

[0106] The basic model is a low-parameter model of the same series.

[0107] This example applies the present invention's prompt word compression method based on TF-IDF and cross entropy to effectively compress long legal documents. This method significantly reduces the number of input tokens while ensuring the model's output remains accurate and reliable. To quantify model performance, this example uses the F1 value to test the compression results. The F1 calculation formula is as follows:

[0108]

[0109] Taking a specific legal task as an example, the question is: "How does Foley v. Interactive Data Corp. reflect a shift in judicial attitudes toward the application of tort law in contractual relationships?" The original input prompt is a long legal text: "The freedom to choose our associates — like other personal liberties — has often been suppressed by overreaching government officials, including judges. Recent judicial trends of imposing tort law onto contractual relationships exemplify this overreach. Fortunately, this approach is being reconsidered. In Foley v. Interactive Data Corp., 47 Cal.3d 654 (1988), the California Supreme Court limited the reach of Seaman's by holding that tort remedies are not always available in contract." disputes." (The freedom to choose interpersonal relationships—like other personal freedoms—is often suppressed by overreaching and authoritarian government officials, including judges. The recent judicial tendency to infiltrate tort law into contractual relationships is a manifestation of this overreach. Fortunately, this practice is being re-examined. In the 1988 case of Faure v. Interactive Data Corporation (47 Cal. 3d, p. 654), the California Supreme Court, by limiting the scope of the "Seaman case," established the rule that tort remedies are not automatically applicable in contract disputes.) contains three complex long sentences.

[0110] The method proposed by the present invention compresses the original prompt words by sequentially executing three steps. The specific steps of the method include:

[0111] S1. Divide and compress the long legal documents into sentence-level prompt words to ensure that the compressed information is relevant to the question. The specific steps are:

[0112] S1.1. Divide the entire long legal notice into background information, output examples, and legal questions, and divide the output examples into background information, questions, and answers;

[0113] S1.2. Divide the prompt words and background information in the example into sentences based on punctuation marks. Convert each sentence to be compressed into an embedding vector, denoted as {S1, S2, …, Sn}. Also convert the legal issues in the prompt words into an embedding vector, denoted as T.

[0114] S1.3. Calculate the Euclidean distance between the embedding vector {S1, S2, …, Sn} of each sentence to be compressed and the question vector T in the vector space to obtain sentences that are more relevant to the legal question. Redundant sentences irrelevant to the question are filtered out, and the remaining sentences are arranged in their original order. The background information in this example is compressed in the same manner according to its question.

[0115] After sentence-level compression, the original input legal text deleted sentences that are not highly relevant to the question. The compressed text is: "Recent judicial trends of imposing tort law onto contractual relationships exemplify this overreach. In Foley v. Interactive Data Corp., 47 Cal.3d 654 (1988), the California Supreme Court limited the reach of Seaman's by holding that tort remedies are not always available in contract disputes." (The recent judicial trend of imposing tort law onto contractual relationships is a manifestation of overreach. In Foley v. Interactive Data Corp., 47 Cal.3d 654 (1988), the California Supreme Court limited the reach of Seaman's by holding that tort remedies are not always available in contract disputes.), which contains two complex sentences related to the question.

[0116] S2. Based on the sentences related to legal issues obtained by screening in S1, further compression is performed to identify keywords and delete non-keywords in the sentences. The specific steps are:

[0117] S2.1. Based on the sentences retained in S1, take each sentence as a unit and screen high-frequency words as candidate keywords by calculating the frequency of each word in the current sentence;

[0118] S2.2, also based on the sentences retained in S1, use the inverse word frequency to identify the words with the highest degree of distinction in the entire sentence as candidate keywords;

[0119] S2.3. Based on the above two methods, the TF-IDF method is used to comprehensively evaluate the importance of words and select keywords for retention by combining word frequency and inverse word frequency.

[0120] After word-level compression and filtering out non-keywords in the sentence, the compressed text is: "Recent judicial trends of imposing tort law onto contractual relationships. In Foley v.Interactive Data Corp. 47 Cal.3d 654 1988 the California Supreme Court limited the reach of Seaman's by holding that tort remedies are not." (Recent judicial trends of imposing tort law onto contractual relationships. In Foley v.Interactive Data Corp. 47 Cal.3d 654 1988 the California Supreme Court limited the reach of Seaman's by holding that tort remedies are not.), containing 46 tokens.

[0121] S3. At the token level, the cross entropy loss difference between the reference model and the base model is used to further perform fine-grained compression. The specific steps are:

[0122] S3.1. Select a reference model with strong performance and a basic model. Here, the reference model is Llama3.2-3B-Instruct, which is more sensitive to legal information, and the basic model is Llama3.2-1B-Instruct.

[0123] S3.2, after completing the comprehensive deletion of non-keywords in stage S2, the processed sentences are inferred on the Llama3.2-3B-Instruct and Llama3.2-1B-Instrcut models respectively, and the cross entropy loss is obtained respectively. and ;

[0124] S3.3, and Subtract to get the cross entropy loss difference ,The larger the difference, the more critical the token’s performance on the reference model is, that is, it is more sensitive to legal information, so the token with a large difference is retained;

[0125] S3.4. The parts compressed at the token level are spliced in their original order, and finally combined to form a compressed prompt word with complete structure and coherent semantics.

[0126] After token compression, the non-important tokens in the text were deleted. The compressed text is: "Recent judicial trends of imposing tort onto relationships. In Foley v. Data Corp. 47 Cal.d654 1988 the California Supreme Court limited the reach of Seaman's byholding that tort not." (Recent judicial trends of imposing tort onto relationships. In Foley v. Data Corp. 47 Cal.d654 1988 the California Supreme Court limited the reach of Seaman's byholding that tort not.), containing 40 tokens.

[0127] The original dataset data will cause out-of-memory (OOM) problems during inference on Qwen2.5-3B due to the length of the prompt word. The compression mechanism proposed in this paper not only avoids the out-of-memory problem, but also significantly reduces the video memory usage and inference latency. The output quality after compression remains basically stable, verifying the effectiveness of this method. Figure 2 shown.

[0128] Based on this, the embodiment of the present invention provides a prompt word compression method based on TF-IDF and cross entropy, which has the following beneficial effects:

[0129] (1) This paper filters at the sentence level to remove redundant sentences that are irrelevant to the target question. Specifically, each sentence is subjected to vector similarity analysis to determine whether it is directly related to the question. This process not only improves the overall refinement of the text but also ensures a high degree of information focus.

[0130] (2) The present invention comprehensively uses word frequency and inverse document frequency weight analysis at the word level to significantly reduce unnecessary words while retaining key information, thereby improving the conciseness and readability of the text.

[0131] (3) The present invention identifies important tokens at the token level through an evaluation mechanism that evaluates model output differences. Specifically, the difference in cross-entropy loss between the reference model and the base model output is calculated to determine the degree of influence of each token on the model output, and thus identify important tokens. This method can retain information that is critical to understanding the context to the greatest extent possible while removing redundant information, achieving efficient information compression and optimization.

[0132] In summary, the present invention implements a progressive compression strategy at three levels: sentence level, word level, and token level. This layered approach not only achieves efficient data compression while ensuring semantic integrity, but also allows for fine-grained control of text content, effectively preserving key information within sentences.

[0133] The compression strategy also eliminates the need for pre-training a specific model and avoids the excessive consumption of computational resources associated with the use of attention mechanisms. Furthermore, because it is independent of a specific model structure, the method demonstrates strong generalization capabilities and is applicable to a variety of application scenarios.

[0134] Example 2

[0135] Based on the same general inventive concept, the present invention further provides a prompt word compression system based on TF-IDF and cross entropy. The prompt word compression system based on TF-IDF and cross entropy provided by the present invention is described below. The prompt word compression system based on TF-IDF and cross entropy described below can be used in conjunction with the prompt word compression method based on TF-IDF and cross entropy described above. The system includes:

[0136] Sentence-level compression module, word-level compression module, and token-level compression module.

[0137] The sentence-level compression module is primarily responsible for identifying and compressing sentences that are irrelevant to the question. By calculating the Euclidean distance between the sentence and the question in the vector space, sentences with low semantic similarity to the question are filtered out and removed, thereby reducing unnecessary information redundancy.

[0138] The word-level compression module focuses on removing non-critical keywords while preserving the relevance of the sentence to the question. By calculating the TF-IDF metric in the sentence and ranking each word's contribution to the entire sentence, it determines which words in the sentence are non-critical and removes them accordingly.

[0139] The token-level compression module identifies and compresses tokens that have little impact on the final output based on the cross-entropy loss difference between the base model and the reference model, thereby compressing non-critical tokens.

[0140] It should be understood that the prompt word compression system based on TF-IDF and cross entropy provided in the embodiment of the present invention has all the advantages of the prompt word compression method based on TF-IDF and cross entropy provided in the above embodiment.

[0141] Example 3

[0142] In this embodiment, a computer terminal device is provided, including:

[0143] one or more processors;

[0144] a memory, coupled to the processor, for storing one or more programs;

[0145] When the one or more programs are executed by the one or more processors, the one or more processors implement the methods in the above embodiments.

[0146] In this embodiment, a computer-readable storage medium is further provided, on which a computer program is stored. When the computer program is executed by a processor, the method in the above embodiment is implemented.

[0147] In this embodiment, an electronic device is further provided, including a memory and a processor. The memory stores a computer program, and the processor is configured to run the computer program to execute the method in the above embodiment.

[0148] The above program can be executed in a processor or stored in a memory (or computer-readable medium). Computer-readable media includes both permanent and non-permanent, removable and non-removable media, and can be implemented using any method or technology to store information. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible by a computing device.

[0149] These computer programs can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable device to implement the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps of the functions specified in one or more blocks can be implemented by different modules corresponding to different steps.

[0150] This embodiment provides such a device or system. The system is called a prompt word compression system based on TF-IDF and cross entropy, and includes:

[0151] The content segmentation module is used to identify the question part and the content to be compressed in the prompt word, and retain the question part intact;

[0152] The semantic filtering module is used to perform sentence-level compression on the content to be compressed. It converts the sentences and questions to be compressed into embedding vectors, filters semantically related sentences based on Euclidean distance, and retains them in their original order.

[0153] The keyword extraction module is used to perform word-level compression on the sentence-level compression results: extract keywords based on TF-IDF values and reorganize the sentences in their original order;

[0154] The sensitivity analysis module is used to perform token-level compression on word-level compression results. It selects a reference model and a base model, calculates the difference in cross-entropy loss between the two, identifies key tokens based on the difference, and concatenates them in the original order.

[0155] The output generation module is used to output compressed prompt words.

[0156] As an implementation method of this embodiment, the content division module includes:

[0157] A background recognition unit for recognizing the background information part, the example part and the question part in the prompt word;

[0158] A content retention unit is used to completely retain the answer content of the question part and the example part;

[0159] The compression range determining unit is configured to determine that the content to be compressed is the background information in the background information part and the example part.

[0160] As an implementation method of this embodiment, the semantic screening module includes:

[0161] Sentence segmentation unit, used to process compressed content into sentences according to punctuation marks;

[0162] Vector conversion unit, used to convert the sentence into an embedding vector after clause segmentation, and convert the question into an embedding vector;

[0163] A distance calculation unit, used to calculate the Euclidean distance between each sentence vector and the question vector;

[0164] Sentence filtering unit, which is used to retain related sentences by distance sorting and maintain the original order.

[0165] As an implementation in this embodiment, the keyword extraction module includes:

[0166] A word frequency counting unit is used to count the word frequencies of each word in the retained sentence;

[0167] An inverse frequency calculation unit, used to calculate the inverse document frequency of each word;

[0168] Weight fusion unit, used to fuse term frequency and inverse document frequency to obtain TF-IDF value;

[0169] The keyword reorganization unit is used to retain keywords sorted by TF-IDF values and reorganize sentences in the original order.

[0170] As an implementation in this embodiment, the sensitivity analysis module includes:

[0171] Model pairing unit, used to select reference model and basic model;

[0172] The loss calculation unit is used to input the word-level compression results into the two models and calculate the cross-entropy loss;

[0173] The difference analysis unit is used to calculate the difference between the losses of the two models;

[0174] The token screening unit is used to identify key tokens based on the difference size and splice them in the original order.

[0175] As an implementation method in this embodiment, the model pairing unit is configured as follows:

[0176] Select a high-performance model optimized for the task domain as the reference model;

[0177] The low-parameter model of the same series was selected as the basic model.

[0178] The system or device is used to implement the functions of the method in the above-mentioned embodiment. Each module in the system or device corresponds to each step in the method, which has been explained in the method and will not be repeated here.

[0179] Through the above implementation, the problem of prompt word compression based on TF-IDF and cross entropy in the related art is solved, thereby ensuring that the problems existing in the existing technology are solved.

[0180] The above are merely preferred embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A prompt word compression method based on TF-IDF and cross entropy, characterized in that: The following steps are involved: Divide the prompt word into the question part and the content to be compressed, and keep the question part intact; Perform sentence-level compression on the content to be compressed: convert the sentences to be compressed and the questions into embedding vectors, select semantically related sentences based on Euclidean distance and retain them in their original order; Perform word-level compression on the sentence-level compression results: extract keywords based on TF-IDF values and reorganize the sentences in their original order; Perform token-level compression on the word-level compression results: Select the reference model and the base model, calculate the difference in cross-entropy loss between the two models for the processed text, identify key tokens based on the difference, and concatenate them in the original order; Output the concatenated compressed prompt words.

2. The method according to claim 1, characterized in that The process of dividing the content to be compressed includes: Identify the background information, example, and question sections of a prompt; Keep the answers to the questions and examples intact. The content to be compressed is determined to be the background information portion and the background information in the example portion.

3. The method according to claim 1, characterized in that The sentence-level compression process includes: Treat compressed content as sentences according to punctuation; Convert the sentence into an embedding vector after segmentation; Convert the problem into an embedding vector; Calculate the Euclidean distance between each sentence vector and the question vector; Sorting by distance keeps related sentences and maintains the original order.

4. The method according to claim 1, wherein The word-level compression process includes: Count the frequency of each word in the retained sentence; Calculate the inverse document frequency of each word; The TF-IDF value is obtained by fusing the term frequency and the inverse document frequency; Sort by TF-IDF value, retain the keywords and reorganize the sentences in the original order.

5. The method according to claim 1, characterized in that The Token-level compression process includes: Select reference model and base model; Input the word-level compression results into the reference model and the base model; Calculate the cross entropy loss of the reference model and the cross entropy loss of the base model; Calculate the difference between the two losses; Identify the key tokens based on the difference and concatenate them in the original order.

6. The method according to claim 5, characterized in that The reference model is a high-performance model optimized for the task domain; The basic model is a low-parameter model of the same series.

7. A prompt word compression system based on TF-IDF and cross entropy, characterized by: The system comprises: The content segmentation module is used to identify the question part and the content to be compressed in the prompt word, and retain the question part intact; The semantic filtering module is used to perform sentence-level compression on the content to be compressed. It converts the sentences and questions to be compressed into embedding vectors, filters semantically related sentences based on Euclidean distance, and retains them in their original order. The keyword extraction module is used to perform word-level compression on the sentence-level compression results: extract keywords based on TF-IDF values and reorganize the sentences in their original order; The sensitivity analysis module is used to perform token-level compression on word-level compression results. It selects a reference model and a base model, calculates the difference in cross-entropy loss between the two, identifies key tokens based on the difference, and concatenates them in the original order. The output generation module is used to output compressed prompt words.

8. The system according to claim 7, characterized in that The content division module includes: A background recognition unit for recognizing the background information part, the example part and the question part in the prompt word; A content retention unit is used to completely retain the answer content of the question part and the example part; The compression range determining unit is configured to determine that the content to be compressed is the background information in the background information part and the example part.

9. A computer terminal device, characterized in that: include: one or more processors; a memory, coupled to the processor, for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the prompt word compression method based on TF-IDF and cross entropy according to any one of claims 1 to 6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the prompt word compression method based on TF-IDF and cross entropy according to any one of claims 1 to 6 is implemented.

Images