Prompt word compression method and system for edge device large model

Abstract

The invention is suitable for the technical field of cue word compression, and provides a cue word compression method and system for an edge device large model. According to the method, the performance test is performed on the edge device large model, the regression model of the input length and the processing time is established, and the longest Prompt length is determined based on the preset expected processing time; dividing the input Prompt into a plurality of key paragraphs and a plurality of non-key paragraphs, and calculating semantic association degrees between the plurality of non-key paragraphs and the plurality of key paragraphs; and calculating the key total length of the plurality of key paragraphs, comparing the key total length with the longest Prompt length, and performing multiple rounds of Prompt compression on the input Prompt. Through the key paragraph-oriented compression method, the workload of the compression process can be reduced, reasoning precision and delay requirements are considered, the semantic retention performance is excellent, memory consumption and reasoning delay can be limited, and subsequent reasoning work is effectively supported.

Description

Technical Field

[0001] The present invention belongs to the technical field of prompt word compression, and in particular, relates to a prompt word compression method and system for a large model of an edge device. Background Art

[0002] The technologies for prompt compression can be divided into two categories: traditional natural language processing technology and language model technology. Among them, traditional natural language processing technology usually includes: (1) rule-based methods, which use predefined language rules and patterns to identify and remove redundant information, such as data dictionaries and stop word removal technology; (2) extractive summarization, which generates a compressed version by selecting key paragraphs in the text, such as using term frequency-inverse document frequency (TF-IDF) to sort and compress sentences; (3) topic modeling, which uses latent Dirichlet allocation (LDA) to identify the main topics in the prompt, thereby retaining relevant information and removing irrelevant content. Language model technology relies on language models to capture the key parts of the text and achieve text compression, aiming to retain important information while removing redundancy.

[0003] Traditional natural language processing technologies have mostly been eliminated due to their poor performance in retaining semantics.

[0004] Language model technology is currently widely used. For example, Microsoft's LLMLingua uses the LLaMa2 7b model for prompt compression, while some other cutting-edge works use FLAN-T5-XXL or GPT2-XL. However, due to the limitations of end-side computing power and available memory, the use of advanced language model compression may not be suitable for instant prompt compression during inference. This is because these compression techniques require additional computing resources during execution, resulting in increased latency in inference, which offsets the time savings brought by compression. Although traditional semantic extraction or segmented truncation can quickly shorten the prompt length, it will lose a lot of key information and significantly increase the zero-sample perplexity of the model.

[0005] Therefore, the existing technology has two major defects: on the one hand, the technology with faster compression speed performs poorly in terms of semantic preservation; on the other hand, although the use of language models for compression can retain most semantics, it will bring high memory consumption and inference delay. Summary of the invention

[0006] The purpose of the embodiments of the present invention is to provide a prompt word compression method and system for a large model of an edge device, aiming to solve the technical problems existing in the prior art mentioned in the background technology.

[0007] The embodiment of the present invention is implemented as follows: A prompt word compression method for a large model of an edge device, the method specifically comprising the following steps: Perform performance tests on large models of edge devices, build a regression model of input length and processing time, and determine the maximum prompt length based on the preset expected processing time; Receive an input prompt, divide the input prompt into a plurality of key paragraphs and a plurality of non-key paragraphs, and calculate semantic associations between the plurality of non-key paragraphs and the plurality of key paragraphs; The key total lengths of the multiple key paragraphs are calculated, the key total length is compared with the longest Prompt length, and multiple rounds of Prompt compression are performed on the input Prompt.

[0008] As a further limitation of the technical solution of the embodiment of the present invention, the performance test of the edge device large model, establishing a regression model of input length and processing time, and determining the longest Prompt length based on a preset expected processing time specifically include the following steps: Perform performance tests on large models of edge devices and obtain performance test data; According to the performance test data, random forest regression is selected, and a regression model of input length and processing time is established through cross-validation and parameter tuning; Importing the preset expected processing time into the regression model; Export the maximum prompt length.

[0009] As a further limitation of the technical solution of the embodiment of the present invention, the receiving of the input Prompt, dividing the input Prompt into a plurality of key paragraphs and a plurality of non-key paragraphs, and calculating the semantic association between the plurality of non-key paragraphs and the plurality of key paragraphs specifically comprises the following steps: Receive input prompt; Performing text preprocessing and sentence similarity analysis on the input prompt, and dividing the input prompt into multiple topic paragraphs; Calculating the attention scores of the plurality of topic paragraphs, and dividing the plurality of topic paragraphs into a plurality of key paragraphs and a plurality of non-key paragraphs; A set of non-key paragraphs and an average vector of key paragraphs are obtained, and semantic associations between a plurality of the non-key paragraphs and a plurality of the key paragraphs are calculated.

[0010] As a further limitation of the technical solution of the embodiment of the present invention, the text preprocessing and sentence similarity analysis of the input prompt and dividing the input prompt into multiple topic paragraphs specifically include the following steps: Performing text preprocessing on the input prompt to eliminate noise that affects segmentation, and dividing the input prompt into multiple sentences; Converting the plurality of sentences into fixed-length vectors; Calculating the cosine similarity between the plurality of sentences according to the plurality of fixed-length vectors; Compare the multiple cosine similarities with a preset similarity threshold, and record the comparison result; According to the comparison result, multiple sentences are aggregated to obtain multiple topic paragraphs.

[0011] As a further limitation of the technical solution of the embodiment of the present invention, the expression of the fixed-length vector is: ; in, Representative Sentences, is the total number of sentences; The calculation formula for calculating the cosine similarity between the multiple sentences is: ; in, represents the inner product of vectors, and Represents the bi-norm of a vector.

[0012] As a further limitation of the technical solution of the embodiment of the present invention, the calculation of the attention scores of the plurality of topic paragraphs and the division of the plurality of topic paragraphs into a plurality of key paragraphs and a plurality of non-key paragraphs specifically include the following steps: Get the attention weight of each attention head in each layer through the hook function; Calculating global attention scores for a plurality of the topic paragraphs; Normalizing the multiple global attention scores to obtain multiple attention scores; According to the plurality of attention scores, the plurality of topic paragraphs are divided into a plurality of key paragraphs and a plurality of non-key paragraphs.

[0013] As a further limitation of the technical solution of the embodiment of the present invention, the calculation formula of the attention weight is: ; in, represent Layer Attention head, and are the matrices of query and key respectively, is the dimension of the key; The calculation formula of the global attention score is: ; in, Representative topic paragraphs, topic paragraphs Contains a token collection , for About Location The attention matrix; The calculation formula for the normalization process is: ; in, Topic paragraph Attention score.

[0014] As a further limitation of the technical solution of the embodiment of the present invention, the set of non-critical paragraphs is expressed as: ; in, is the collection of all thematic paragraphs; The average vector of the key paragraph is expressed as: ; in, is a collection of key paragraphs, for Embedded vector representation of ; The calculation formula of the semantic relevance is: ; in, For the Non-critical paragraphs The embedded vector representation of .

[0015] As a further limitation of the technical solution of the embodiment of the present invention, the calculating of the total key lengths of the multiple key paragraphs, comparing the total key lengths with the longest Prompt length, and performing multiple rounds of Prompt compression on the input Prompt specifically includes the following steps: Calculate the total key length of the plurality of key paragraphs; Compare the key total length with the longest prompt length to calculate the total allocation length; If the total length of the allocation is not greater than 0, retaining multiple key sections and compressing multiple non-key sections; If the total allocated length is greater than 0, then calculating the total non-critical length of the plurality of non-critical sections; comparing the non-critical total length to the allocated total length; If the total length of the non-critical paragraphs is not less than 0, multiple non-critical paragraphs are retained; If the total length of the non-critical paragraphs is less than 0, multiple non-critical paragraphs are selectively compressed according to multiple semantic associations.

[0016] A prompt word compression system for a large model of an edge device, the system comprising a maximum prompt length determination module, a key paragraph recognition processing module and a multi-round prompt compression module, wherein: The longest prompt length determination module is used to perform performance testing on the edge device large model, establish a regression model of input length and processing time, and determine the longest prompt length based on the preset expected processing time; A key paragraph identification processing module, used for receiving an input prompt, dividing the input prompt into a plurality of key paragraphs and a plurality of non-key paragraphs, and calculating the semantic association between the plurality of non-key paragraphs and the plurality of key paragraphs; The multi-round Prompt compression module is used to calculate the key total length of multiple key paragraphs, compare the key total length with the longest Prompt length, and perform multi-round Prompt compression on the input Prompt.

[0017] Compared with the prior art, the present invention has the following beneficial effects: The embodiment of the present invention performs a performance test on a large model of edge devices, establishes a regression model of input length and processing time, determines the longest Prompt length based on the preset expected processing time, divides the input Prompt into multiple key paragraphs and multiple non-key paragraphs, and calculates the semantic association between multiple non-key paragraphs and multiple key paragraphs, calculates the key total length of multiple key paragraphs, compares the key total length with the longest Prompt length, and performs multiple rounds of Prompt compression on the input Prompt. Through the key paragraph-oriented compression method, the workload of the compression process can be reduced, and the reasoning accuracy and delay requirements can be taken into account. It can not only perform well in semantic retention, but also limit memory consumption and reasoning delay, and effectively support subsequent reasoning work. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A flowchart of a prompt word compression method for an edge device large model provided by an embodiment of the present invention is shown; Figure 2 An application architecture diagram of a prompt word compression system for a large model of an edge device provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0020] It is understandable that in the existing technology, prompt compression is divided into two major categories of technology: traditional natural language processing technology, which has been mostly eliminated due to its poor effect in semantic retention; language model technology, due to the limitations of terminal computing power and available memory, requires additional computing resources during execution, resulting in increased reasoning delays, which offsets the time savings brought by compression. Although traditional semantic extraction or segment truncation can quickly shorten the prompt length, it will lose a lot of key information and significantly increase the zero-sample perplexity of the model. Therefore, the existing technology has two major defects: on the one hand, technologies with faster compression speeds perform poorly in semantic retention; on the other hand, although using language models for compression can retain most semantics, it will result in high memory consumption and reasoning delays.

[0021] To solve the above problems, the embodiment of the present invention discloses a prompt word compression method and system for edge device large model, which performs performance testing on edge device large model, establishes a regression model of input length and processing time, and determines the longest prompt length based on the preset expected processing time; receives input prompt, divides the input prompt into multiple key paragraphs and multiple non-key paragraphs, and calculates the semantic association between multiple non-key paragraphs and multiple key paragraphs; calculates the key total length of multiple key paragraphs, compares the key total length with the longest prompt length, and performs multiple rounds of prompt compression on the input prompt. Through the key paragraph-oriented compression method, the workload of the compression process can be reduced, and the reasoning accuracy and delay requirements can be taken into account. It can not only perform well in semantic retention, but also limit memory consumption and reasoning delay, and effectively support subsequent reasoning work.

[0022] Specifically, Figure 1 A flow chart of a prompt word compression method for an edge device large model provided by an embodiment of the present invention is shown.

[0023] In a preferred embodiment of the present invention, a prompt word compression method for a large model of an edge device is provided, and the method specifically comprises the following steps: Step S101: perform performance testing on the edge device large model, establish a regression model of input length and processing time, and determine the longest Prompt length based on the preset expected processing time.

[0024] In an embodiment of the present invention, a comprehensive test and evaluation is performed on the performance of the edge device large model on the target edge device, including but not limited to key performance indicators such as the response time, memory usage and energy consumption of the model under different Prompt lengths, and performance test data is obtained. Based on the performance test data, random forest regression is selected, and a regression model of input length and processing time is established through cross-validation and parameter tuning. The preset expected processing time is imported into the regression model for processing, and the longest Prompt length is derived.

[0025] It is understandable that through systematic performance testing, the operating limits of the large edge device model on the target edge device can be clarified, thereby providing data support for the formulation of the Prompt compression strategy.

[0026] It is understandable that in terms of model selection, random forest regression is selected to improve the accuracy and robustness of prediction; through cross-validation and parameter tuning, the generalization ability of the regression model on different data sets is ensured.

[0027] It is understandable that the maximum prompt length is not a constant. It varies with the computing power of the end-side device and the deployed model, and needs to be determined dynamically before inference. Specifically, by designing an adaptive mechanism, the maximum prompt length is adjusted in real time according to the current operating status and load of the device to adapt to different application requirements and environmental changes.

[0028] Step S102: receiving an input Prompt, dividing the input Prompt into a plurality of key paragraphs and a plurality of non-key paragraphs, and calculating the semantic association between the plurality of non-key paragraphs and the plurality of key paragraphs.

[0029] In an embodiment of the present invention, by receiving an input Prompt, performing text preprocessing on the input Prompt to eliminate noise that affects segmentation, dividing the input Prompt into multiple sentences, and then converting the multiple sentences into fixed-length vectors, and then calculating the cosine similarity between the multiple sentences according to the multiple fixed-length vectors, by comparing the multiple cosine similarities with a preset similarity threshold, recording the comparison results, and then performing aggregation processing on the multiple sentences according to the comparison results to obtain multiple topic paragraphs, and then obtaining the attention weight of each attention head in each layer through a hook function, calculating the global attention scores of the multiple topic paragraphs, and then normalizing the multiple global attention scores to obtain multiple attention scores, and then comparing the multiple attention scores according to the preset score threshold, dividing the multiple topic paragraphs into multiple key paragraphs and multiple non-key paragraphs, and then obtaining a set of non-key paragraphs and an average vector of key paragraphs, and calculating the semantic association between multiple non-key paragraphs and multiple key paragraphs. Specifically, the expression of the fixed-length vector is: ; in, Representative Sentences, is the total number of sentences; The formula for calculating the cosine similarity between multiple sentences is: ; in, represents the inner product of vectors, and represents the two-norm of a vector; The calculation formula of attention weight is: ; in, represent Layer Attention head, and are the matrices of query and key respectively, is the dimension of the key; The calculation formula for the global attention score is: ; in, Representative topic paragraphs, topic paragraphs Contains a token collection , for About Location The attention matrix; The calculation formula for normalization is: ; in, Topic paragraph Attention score; The set of non-key paragraphs is represented as: ; in, is the set of all topic paragraphs, is the score threshold; The average vector of the key paragraph is expressed as: ; in, is a collection of key paragraphs, for Embedded vector representation of ; The calculation formula of semantic relevance is: ; in, For the Non-critical paragraphs The embedded vector representation of .

[0030] It can be understood that the text preprocessing of the input prompt to eliminate the noise that affects the segmentation is to remove meaningless symbols such as extra spaces in the input prompt, but it is necessary to retain punctuation marks, modal particles and emoticons to reflect the user's emotions and the structural distribution of the prompt.

[0031] It is understandable that cosine similarity The final value range of will be [-1,1]. The closer the value is to 1, the higher the similarity is, and it is suitable to be classified into the same paragraph. By setting the similarity threshold , control the criteria for sentence aggregation and ensure the rationality of paragraph division.

[0032] It is understandable that the difficulty of identifying key paragraphs depends on the quality of the prompt. High-quality prompts usually have "question" paragraphs at the beginning and end. You only need to quickly locate the question sentence to identify the key paragraph. For prompts with complex content, the key paragraphs are determined by extracting the attention weights of the first few layers of the large language model. That is, the large language model selects the important paragraphs it considers by itself. The specific method is to obtain the attention weights of each layer through the hook function. Each attention head The attention matrix To express About Location The attention weight.

[0033] It can be understood that by calculating the semantic correlation between multiple non-critical paragraphs and multiple critical paragraphs, the semantic correlation between non-critical paragraphs and critical paragraphs can be quantified, thereby providing a basis for subsequent compression or deletion decisions. Non-critical paragraphs with high correlation may contain important information related to critical paragraphs, while non-critical paragraphs with low correlation can be given priority for compression or deletion to minimize the loss of semantic information.

[0034] Step S103: Calculate the total key lengths of the multiple key paragraphs, compare the total key length with the longest Prompt length, and perform multiple rounds of Prompt compression on the input Prompt.

[0035] In the embodiment of the present invention, the total key length of multiple key paragraphs is calculated, and then the total key length is compared with the longest prompt length to calculate the total allocation length. When the total allocation length is not greater than 0, multiple key paragraphs are retained and multiple non-key paragraphs are compressed; when the total allocation length is greater than 0, the total non-key length of multiple non-key paragraphs is calculated, and the total non-key length is compared with the total allocation length. When the total non-key length is not less than 0, multiple non-key paragraphs are retained; when the total non-key length is less than 0, multiple non-key paragraphs are selected and compressed according to multiple semantic associations. Specifically, the calculation formula of the total key length is: ; in, It is a paragraph The number of tokens; The total length of the allocation is calculated as: ; in, is the longest prompt length; The calculation formula for the non-critical total length is: ; in, It is a paragraph The number of tokens.

[0036] It can be understood that by selectively compressing multiple non-key paragraphs, the non-key paragraphs can be effectively compressed while ensuring the key paragraph information, thereby reducing the overall prompt length. The corresponding compression ratio is calculated as follows: ; It is understandable that if the maximum prompt length cannot be met after processing non-critical paragraphs, , and the highly correlated non-key paragraphs are difficult to delete, then it is necessary to consider compressing the key paragraphs. The corresponding compression ratio calculation formula is: .

[0037] Furthermore, Figure 2 An application architecture diagram of a prompt word compression system for a large model of an edge device provided by an embodiment of the present invention is shown.

[0038] Among them, in another preferred embodiment provided by the present invention, a prompt word compression system for a large model of an edge device includes: The longest prompt length determination module 101 is used to perform performance testing on the edge device large model, establish a regression model of input length and processing time, and determine the longest prompt length based on a preset expected processing time.

[0039] In an embodiment of the present invention, the longest Prompt length determination module 101 performs a comprehensive test and evaluation on the performance of the edge device large model on the target edge device, including but not limited to key performance indicators such as the response time, memory usage and energy consumption of the model under different Prompt lengths, obtains performance test data, selects random forest regression based on the performance test data, establishes a regression model of input length and processing time through cross-validation and parameter tuning, and exports the longest Prompt length by importing the preset expected processing time into the regression model for processing.

[0040] The key paragraph identification processing module 102 is used to receive an input Prompt, divide the input Prompt into a plurality of key paragraphs and a plurality of non-key paragraphs, and calculate the semantic association between the plurality of non-key paragraphs and the plurality of key paragraphs.

[0041] In an embodiment of the present invention, the key paragraph identification processing module 102 receives an input Prompt, performs text preprocessing on the input Prompt to eliminate noise that affects segmentation, divides the input Prompt into multiple sentences, and then converts the multiple sentences into fixed-length vectors, and then calculates the cosine similarity between the multiple sentences according to the multiple fixed-length vectors, compares the multiple cosine similarities with a preset similarity threshold, records the comparison results, and then aggregates the multiple sentences according to the comparison results to obtain multiple topic paragraphs. After that, the attention weight of each attention head in each layer is obtained through a hook function, and the global attention scores of the multiple topic paragraphs are calculated. The multiple global attention scores are normalized to obtain multiple attention scores, and then the multiple attention scores are compared according to the preset score threshold, and the multiple topic paragraphs are divided into multiple key paragraphs and multiple non-key paragraphs. Then, the set of non-key paragraphs and the average vector of the key paragraphs are obtained, and the semantic association between the multiple non-key paragraphs and the multiple key paragraphs is calculated. Specifically, the expression of the fixed-length vector is: ; in, Representative Sentences, is the total number of sentences; The formula for calculating the cosine similarity between multiple sentences is: ; in, represents the inner product of vectors, and represents the two-norm of a vector; The calculation formula of attention weight is: ; in, represent Layer Attention head, and are the matrices of query and key respectively, is the dimension of the key; The calculation formula for the global attention score is: ; in, Representative topic paragraphs, topic paragraphs Contains a token collection , for About Location The attention matrix; The calculation formula for the normalization process is: ; in, Topic paragraph Attention score; The set of non-key paragraphs is represented as: ; in, is the set of all topic paragraphs, is the score threshold; The average vector of the key paragraph is expressed as: ; in, is a collection of key paragraphs, for Embedded vector representation of ; The calculation formula of semantic relevance is: ; in, For the Non-critical paragraphs The embedded vector representation of .

[0042] The multi-round Prompt compression module 103 is used to calculate the total key length of the multiple key paragraphs, compare the total key length with the longest Prompt length, and perform multi-round Prompt compression on the input Prompt.

[0043] In the embodiment of the present invention, the multi-round prompt compression module 103 calculates the total key length of multiple key paragraphs, and then compares the total key length with the longest prompt length to calculate the total allocation length. When the total allocation length is not greater than 0, multiple key paragraphs are retained and multiple non-key paragraphs are compressed; when the total allocation length is greater than 0, the non-key total length of multiple non-key paragraphs is calculated, and the non-key total length is compared with the total allocation length. When the non-key total length is not less than 0, multiple non-key paragraphs are retained; when the non-key total length is less than 0, multiple non-key paragraphs are selected and compressed according to multiple semantic associations. Specifically, the calculation formula of the total key length is: ; in, It is a paragraph The number of tokens; The total length of the allocation is calculated as: ; in, is the longest prompt length; The calculation formula for the non-critical total length is: ; in, It is a paragraph The number of tokens.

[0044] It should be understood that, although each step in the flow chart of each embodiment of the present invention is shown in sequence according to the indication of the arrow, these steps are not necessarily performed in sequence according to the order indicated by the arrow. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be performed in other orders. Moreover, at least a portion of the steps in each embodiment may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these sub-steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with at least a portion of other steps or sub-steps or stages of other steps.

[0045] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0046] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A prompt word compression method for a large model of an edge device, characterized in that: The method specifically comprises the following steps: Perform performance tests on large models of edge devices, build a regression model of input length and processing time, and determine the maximum prompt length based on the preset expected processing time; Receive an input prompt, divide the input prompt into a plurality of key paragraphs and a plurality of non-key paragraphs, and calculate semantic associations between the plurality of non-key paragraphs and the plurality of key paragraphs; The key total lengths of the multiple key paragraphs are calculated, the key total length is compared with the longest Prompt length, and multiple rounds of Prompt compression are performed on the input Prompt.

2. The prompt word compression method for edge device large model according to claim 1, characterized in that: The performance test of the edge device large model is performed to establish a regression model of input length and processing time, and based on the preset expected processing time, the longest prompt length is determined, specifically including the following steps: Perform performance tests on large models of edge devices and obtain performance test data; According to the performance test data, random forest regression is selected, and a regression model of input length and processing time is established through cross-validation and parameter tuning; Importing the preset expected processing time into the regression model; Export the maximum prompt length.

3. The prompt word compression method for edge device large model according to claim 1, characterized in that: The receiving of the input Prompt, dividing the input Prompt into a plurality of key paragraphs and a plurality of non-key paragraphs, and calculating the semantic association between the plurality of non-key paragraphs and the plurality of key paragraphs specifically comprises the following steps: Receive input prompt; Performing text preprocessing and sentence similarity analysis on the input prompt, and dividing the input prompt into multiple topic paragraphs; Calculating the attention scores of the plurality of topic paragraphs, and dividing the plurality of topic paragraphs into a plurality of key paragraphs and a plurality of non-key paragraphs; A set of non-key paragraphs and an average vector of key paragraphs are obtained, and semantic associations between a plurality of the non-key paragraphs and a plurality of the key paragraphs are calculated.

4. The prompt word compression method for edge device large model according to claim 3 is characterized in that: The performing text preprocessing and sentence similarity analysis on the input prompt and dividing the input prompt into a plurality of topic paragraphs specifically comprises the following steps: Performing text preprocessing on the input prompt to eliminate noise that affects segmentation, and dividing the input prompt into multiple sentences; Converting the plurality of sentences into fixed-length vectors; Calculating the cosine similarity between the plurality of sentences according to the plurality of fixed-length vectors; Compare the multiple cosine similarities with a preset similarity threshold, and record the comparison result; According to the comparison result, multiple sentences are aggregated to obtain multiple topic paragraphs.

5. The prompt word compression method for edge device large model according to claim 4 is characterized in that: The expression of the fixed-length vector is: ； in, Representative Sentences, is the total number of sentences; The calculation formula for calculating the cosine similarity between the multiple sentences is: ； in, represents the inner product of vectors, and Represents the bi-norm of a vector.

6. The prompt word compression method for edge device large model according to claim 3, characterized in that: The step of calculating the attention scores of the plurality of topic paragraphs and dividing the plurality of topic paragraphs into a plurality of key paragraphs and a plurality of non-key paragraphs specifically comprises the following steps: Get the attention weight of each attention head in each layer through the hook function; Calculating global attention scores for a plurality of the topic paragraphs; Normalizing the multiple global attention scores to obtain multiple attention scores; According to the plurality of attention scores, the plurality of topic paragraphs are divided into a plurality of key paragraphs and a plurality of non-key paragraphs.

7. The prompt word compression method for edge device large model according to claim 6, characterized in that: The calculation formula of the attention weight is: ； in, represent Layer Attention head, and are the matrices of query and key respectively, is the dimension of the key; The calculation formula of the global attention score is: ； in, Representative topic paragraphs, topic paragraphs Contains a token collection , for About Location The attention matrix; The calculation formula for the normalization process is: ； in, Topic paragraph Attention score.

8. The prompt word compression method for edge device large model according to claim 7, characterized in that: The set of non-critical paragraphs is expressed as: ； in, is the collection of all thematic paragraphs; The average vector of the key paragraph is expressed as: ； in, is a collection of key paragraphs, for Embedded vector representation of ; The calculation formula of the semantic relevance is: ； in, For the Non-critical paragraphs The embedded vector representation of .

9. The prompt word compression method for edge device large model according to claim 1, characterized in that: The calculating of the total key lengths of the plurality of key paragraphs, comparing the total key length with the longest Prompt length, and performing multiple rounds of Prompt compression on the input Prompt specifically comprises the following steps: Calculate the total key length of the plurality of key paragraphs; Compare the key total length with the longest prompt length to calculate the total allocation length; If the total length of the allocation is not greater than 0, retaining multiple key sections and compressing multiple non-key sections; If the total allocated length is greater than 0, then calculating the total non-critical length of the plurality of non-critical sections; comparing the non-critical total length to the allocated total length; If the total length of the non-critical paragraphs is not less than 0, multiple non-critical paragraphs are retained; If the total length of the non-critical paragraphs is less than 0, multiple non-critical paragraphs are selectively compressed according to multiple semantic associations.

10. A prompt word compression system for a large model of edge devices, characterized in that: The system includes a longest prompt length determination module, a key paragraph identification processing module and a multi-round prompt compression module, wherein: The longest prompt length determination module is used to perform performance testing on the edge device large model, establish a regression model of input length and processing time, and determine the longest prompt length based on the preset expected processing time; A key paragraph identification processing module, used for receiving an input prompt, dividing the input prompt into a plurality of key paragraphs and a plurality of non-key paragraphs, and calculating the semantic association between the plurality of non-key paragraphs and the plurality of key paragraphs; The multi-round Prompt compression module is used to calculate the key total length of multiple key paragraphs, compare the key total length with the longest Prompt length, and perform multi-round Prompt compression on the input Prompt.

Images