Edge semantic caching method and device based on content delivery network and storage medium

Abstract

The embodiment of the invention relates to the field of artificial intelligence, and discloses an edge semantic caching method and device based on a content distribution network, and a storage medium, and the method comprises the steps: obtaining a current prompt inputted by a user, and judging the correlation between the current prompt and the context of the current prompt; after it is judged that the current prompt is related to the context of the current prompt, the current prompt and the context of the current prompt serve as retrieval conditions, and a first target answer is retrieved in a semantic cache pre-deployed in an edge node to which the user belongs; if the first target answer is retrieved, returning the first target answer to the user; and if the first target answer is not retrieved, submitting a current prompt to the large language model to obtain the answer, returning the answer to the user, and storing a prompt of dialogue information determined according to the dialogue information of the user and the large language model, a context and the corresponding answer in a semantic cache pre-deployed in an edge node to which the user belongs. According to the scheme, the hit rate and the accuracy rate of semantic caching can be considered and improved at the same time.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to an edge semantic caching method, device and storage medium based on a content distribution network. Background Art

[0002] With the gradual development of science and technology, the large language model technology has developed rapidly since its emergence, and applications based on large language models are emerging in an endless stream. Large language models can be briefly divided into two parts: model training and model reasoning. Generally speaking, the application of large language models is mainly based on model reasoning. Due to the huge number of parameters in the large language model itself, the reasoning of the model itself requires a large computing power cost, and the computing power cost will eventually be converted into the usage cost of the model reasoning user.

[0003] In order to reduce computing power costs, existing technologies generally introduce a cache mechanism to reduce computing power costs by caching reusable intermediate states of model reasoning or directly caching the final output results of model reasoning. However, the method of caching the reusable states of the intermediate process of model reasoning is strongly dependent on the algorithm of the model itself. The algorithms of different models may not be directly reused, and the same model must be deployed in the same location, which is not flexible and has great limitations. If the method of directly caching the final output results of model reasoning adopts a static cache method, it is only applicable to scenarios with relatively fixed questions, while the scenarios in which users ask questions are diverse. Therefore, static caches are almost impossible to hit, and the effect of reducing usage costs cannot be achieved; if semantic caches are adopted, existing technologies usually directly discard the context of prompts, which reduces the accuracy of the cache; or directly and rigidly store all information related to the prompts, which reduces the hit rate of the cache. It can be seen that how to balance the hit rate and accuracy of semantic cache while reducing computing power costs and enhancing flexibility has become an urgent problem to be solved. Summary of the invention

[0004] The purpose of the embodiments of the present invention is to provide an edge semantic caching method, device and storage medium based on a content distribution network. By executing this solution to perform semantic caching of edge nodes, the hit rate and accuracy of the semantic cache can be taken into account at the same time, meeting various application scenarios and having strong versatility.

[0005] To solve the above technical problems, an embodiment of the present invention provides an edge semantic caching method based on a content distribution network, comprising: obtaining a current prompt input by a user, and determining the correlation between the current prompt and the context of the current prompt when the current prompt has a context; after determining that the current prompt is correlated with the context of the current prompt, using the current prompt and the context of the current prompt as retrieval conditions to retrieve a first target answer in a semantic cache pre-deployed at an edge node to which the user belongs; wherein a semantic similarity between a first target prompt corresponding to the first target answer and the current prompt is greater than a first threshold, and a semantic similarity between the context of the first target prompt and the context of the current prompt is greater than a second threshold; if the first target answer is retrieved, the first target answer is returned to the user; if the first target answer is not retrieved, the current prompt is submitted to a large language model to obtain an answer and returned to the user, and the prompt, context and corresponding answer of the dialogue information determined according to the dialogue information between the user and the large language model are stored in the semantic cache pre-deployed at the edge node to which the user belongs.

[0006] An embodiment of the present invention also provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the edge semantic caching method based on the content distribution network as described above.

[0007] An embodiment of the present invention further provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the edge semantic caching method based on a content distribution network as described above.

[0008] Compared with the prior art, the embodiments of the present invention deploy semantic caches at each edge node of the content distribution network, and perform cache retrieval in the semantic cache of the edge node to which the user belongs with different retrieval conditions according to the correlation between the current prompt input by the user and its context; if the correlation between the current prompt input by the user and the context is large, the first target answer is retrieved based on the current prompt of the user and the context of the current prompt as cache retrieval conditions in the semantic cache, and the semantic similarity between the prompt corresponding to the first target answer and the current prompt is greater than a first threshold, and the semantic similarity between the context of the first target prompt and the context of the current prompt is greater than a second threshold. If the first target answer cannot be retrieved in the semantic cache of the current node, the answer is obtained from the large language model according to the current prompt and returned to the user. When using the semantic cache, combining the correlation between the prompt and its context to perform retrieval under different conditions can simultaneously take into account and improve the hit rate and accuracy of the semantic cache to meet various application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 The specific process of the edge semantic caching method based on the content distribution network according to the embodiment of the present invention is as follows: Figure 1 ;

[0010] Figure 2 is an example of a conversation with a large language model according to an embodiment of the present invention Figure 1 ;

[0011] Figure 3 is an example of a conversation with a large language model according to an embodiment of the present invention Figure 2 ;

[0012] Figure 4 The specific process of the edge semantic caching method based on the content distribution network according to the embodiment of the present invention is as follows: Figure 2 ;

[0013] Figure 5 is a schematic structural diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0014] In order to make the purpose, technical scheme and advantages of the embodiments of the present invention clearer, the following will be described in detail with reference to the accompanying drawings. However, it will be appreciated by those skilled in the art that in the various embodiments of the present invention, many technical details are provided in order to enable the reader to better understand the present application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical scheme claimed in the present application can be implemented.

[0015] Before describing the present solution in detail, the prior art in the field to which the present solution belongs is described first.

[0016] Large Language Model (LLM) refers to a deep learning model trained with a large amount of text data that can generate natural language text or understand the meaning of language text. With the rapid development of artificial intelligence and natural language processing technology, large language models have shown strong performance in many natural language tasks. However, query processing of LLM consumes a lot of computing resources, which makes efficient semantic caching essential during real-time queries.

[0017] In order to perform efficient semantic caching, the existing technology has adopted a variety of solutions. The most common solution is to create a cache library for a specific large language model. For example, GPTCache in the Python package is a library that creates semantic caches specifically for LLM queries. Each query and its corresponding LLM output result are stored in the cache, and the cache is obtained by performing cosine similarity matching. However, when this cache retrieves prompts input by the user, it only considers the similarity between the prompt itself and the statement in the cache, and does not consider relevant information such as the context of the prompt. Therefore, if the prompt does not reflect a clear purpose, determining the cache only by the semantics of the prompt itself will greatly reduce the cache hit rate and accuracy. Moreover, this solution of customizing a cache layer or cache library for each large language model cannot be applied to all large language models with one library.

[0018] In response to the above problems, an embodiment of the present invention intends to provide an edge semantic caching method based on a content distribution network. By deploying semantic caches at each edge node of the content distribution network, cache retrieval is performed based on the correlation between the prompt and its context, thereby improving the cache hit rate and accuracy, and the method has strong versatility.

[0019] One embodiment of the present invention relates to an edge semantic caching method based on a content distribution network, such as Figure 1 As shown, the following steps are included.

[0020] Step 101: Obtain a current prompt input by a user, and determine the relevance between the current prompt and the context of the current prompt when the current prompt has a context.

[0021] In the field of artificial intelligence, prompt is a very important concept. It refers to a given text or sentence that is used to start and guide the machine learning model to generate output of a specific type, theme or format. In the field of natural language processing, prompt usually consists of a question or task description, such as "write me an article about artificial intelligence", "translate this English sentence into French", etc. In the field of image recognition, prompt can be a picture description, label or classification information. This embodiment does not limit the specific form of the current prompt input by the user.

[0022] Specifically, first obtain the current prompt input by the user, and determine the correlation between the current prompt and the context of the current prompt when the current prompt has a context. Among them, the current prompt refers to the prompt currently input by the user. The context in this embodiment refers to the semantic environment (context) of the current prompt, and does not explicitly require that there must be a previous context and a following context. Generally speaking, the user obtains a satisfactory answer by inputting one or more prompts to the large language model. When the user successively inputs multiple prompts to the large language model to obtain a satisfactory answer, it will be conducted in the form of multiple conversations with the large language model. During each conversation, the content of each latest input by the user to the large language model is the current prompt. If the current prompt is the prompt input by the user for the first time in this conversation, the current prompt has no context; only when the current prompt is the prompt input by the user for a non-first time in this conversation, the current prompt has context; in actual applications, if the current prompt has a semantic environment in the current conversation, it generally has only a specific previous context, but no following context.

[0023] like Figure 2 As shown, Figure 2 An example of a conversation with a large language model is given. Among them, "Please use a short language of about 300 words to help me generate a fable similar to "The Monkey Catching the Moon"." is the current prompt, and "In a deep forest,...pursuing something truly valuable." is the answer given by the large language model. At this time, if the user is satisfied with the current answer, the conversation ends, and the current prompt has no context. If the user is not satisfied with the current answer, he can continue to input new prompts to the large language model. At this time, the newly input prompt is the "current prompt", and the prompt input by the user in the previous round of conversation and the answer given by the large language model are the context of the "current prompt". Figure 3 As shown, users Figure 2 The student is not satisfied with the answer given by the large language model, "In a deep forest,...pursuing something of real value.", and enters a new prompt "More brief". At this time, "More brief" is the current prompt. The previous round of dialogue, "Please use about 300 words to help me generate a fable similar to "The Monkey Catching the Moon"." and "In a deep forest,...pursuing something of real value." are the context of the current prompt, and "Deep in the forest,...pursuing truth." is the final answer given by the large language model.

[0024] In this embodiment, there is no specific restriction on whether the current prompt must have a context, but the method of this proposal is mainly performed when the current prompt has a context. When the current prompt has a context, since the current prompt of the user and the prompt input in the previous round of conversation may be two completely unrelated topics, if you want to introduce context to improve the accuracy of the cache, you need to judge the correlation between the current prompt and the context of the current prompt. The correlation can reflect whether there is a semantic dependency between the current prompt and the context of the current prompt, which can be judged by semantic correlation / similarity, temporal correlation, etc. The specific judgment method can be to judge the completeness of the current prompt itself, or to calculate the semantic correlation by dividing the length of the text, or to extract a semantic-based feature model for calculation and judgment. A threshold can be set for the correlation to obtain a relevant or irrelevant judgment result, which will not be repeated in this application.

[0025] In some embodiments, the specific method for determining the correlation between the current prompt and the context of the current prompt may be: determining the semantic completeness of the current prompt; if complete, determining that the current prompt is not correlated with the context of the current prompt; if incomplete, determining that the current prompt is correlated with the context of the current prompt.

[0026] Specifically, the correlation between the current prompt and the context of the current prompt can be determined by analyzing the semantic completeness of the current prompt itself. If the current prompt itself has already elaborated the demand in a relatively detailed, complete and comprehensive manner, then the necessity of the context corresponding to the current prompt is relatively weak. The semantic completeness can be roughly judged by the number of words in the prompt. If the number of words in the prompt is more, the meaning to be expressed by the prompt itself is relatively complete. On the contrary, it is relatively incomplete and needs to be considered in combination with the context. The semantic completeness of the prompt can also be evaluated by analyzing the weighted scores of dimensions such as the amount of information, sentence structure complexity, and vocabulary richness of the prompt, and the completeness or incompleteness can be judged by setting the corresponding completeness threshold. This embodiment does not specifically limit the method for judging the semantic completeness of the current prompt.

[0027] Accordingly, in addition to reflecting whether the current prompt is related to the context of the current prompt according to the semantic completeness of the current prompt, the temporal correlation between the current prompt and the context of the current prompt may also be considered.

[0028] In some embodiments, after determining that the semantic integrity of the current prompt is incomplete, it can be further determined whether the time interval between the current prompt and the context of the current prompt is greater than a preset third threshold; if so, it is determined that the current prompt and the context of the current prompt are unrelated; if not, it is determined that the current prompt and the context of the current prompt are related.

[0029] Specifically, the time interval between the current prompt and the context of the current prompt can be calculated based on the time when the current prompt appears in the current conversation interface with the large language model and the time when the previous conversation message adjacent to the current prompt in the current conversation interface appears in the current conversation interface. A correlation coefficient can be designed for the time interval and the correlation, so that the longer the time interval between the current prompt and the context of the current prompt, the weaker the correlation between the two. Alternatively, a third threshold is set for the time interval. When the time interval between the current prompt and the context of the current prompt is less than or equal to the specified third threshold, the prompt and the context are considered to be relevant; otherwise, the current prompt and the context of the current prompt are considered to be irrelevant.

[0030] It should be noted that the means of judging the time interval can be combined with the semantic integrity of the current prompt, and a weight can be set for each of the time interval and the semantic integrity, and whether the current prompt is related to the context of the current prompt can be judged through the comprehensive result of weight calculation; or after judging that the semantic integrity of the current prompt is incomplete, continue to judge whether the time interval is greater than a preset third threshold to judge whether the current prompt is related to the context of the current prompt; or judge whether the current prompt is related to the context of the current prompt based solely on the time interval between the current prompt and the context of the current prompt.

[0031] Accordingly, in addition to reflecting whether the current prompt is related to the context of the current prompt based on the semantic completeness of the current prompt and the time interval between the current prompt and the context of the current prompt, the semantic relevance between the current prompt and the context of the current prompt may also be considered.

[0032] In some embodiments, after determining that the time interval between the current prompt and the context of the current prompt is not greater than the third threshold, it is also possible to: determine whether the semantic correlation between the current prompt and the context of the current prompt is less than a preset fourth threshold; if so, determine that the current prompt and the context of the current prompt are not related; if not, determine that the current prompt and the context of the current prompt are related.

[0033] Specifically, the semantic relevance can be judged by using some pre-trained semantic relevance models, for example, a general domain Chinese semantic relevance model based on the ROM-Base pre-trained model is used for judgment, the model takes a source sentence and a sentence list as input, and finally outputs the relevance score of the source sentence and each sentence in the list (0-1, the higher the score, the more relevant the two are). In this embodiment, the current prompt and the context of the current prompt can be input into a pre-trained semantic relevance model, a fourth threshold value is set for the semantic relevance, and it is determined whether the semantic relevance between the current prompt and the context of the current prompt is less than the preset fourth threshold value; if so, it is determined that the current prompt and the context of the current prompt are not relevant; if not, it is determined that the current prompt and the context of the current prompt are relevant.

[0034] It should be noted that the judgment method of the semantic relevance can be combined with the aforementioned semantic integrity of the current prompt and the time interval between the current prompt and the context of the current prompt to judge, set a weight for each of the semantic relevance, time interval and semantic integrity, and judge whether the current prompt and the context of the current prompt are related by weight calculation and comprehensive results; or after judging that the semantic integrity of the current prompt is incomplete, continue to judge whether the semantic relevance between the current prompt and the context of the current prompt is less than the preset fourth threshold to determine whether the current prompt and the context of the current prompt are related; or after judging that the time interval is not greater than the preset third threshold, continue to judge whether the semantic relevance between the current prompt and the context of the current prompt is less than the preset fourth threshold to determine whether the current prompt and the context of the current prompt are related; or after judging that the semantic integrity of the current prompt is incomplete and the time interval is not greater than the preset third threshold, further judge whether the semantic relevance between the current prompt and the context of the current prompt is less than the preset fourth threshold to determine whether the current prompt and the context of the current prompt are related; or it can also be determined whether the current prompt and the context of the current prompt are related based solely on the semantic relevance between the current prompt and the context of the current prompt. This application does not make specific restrictions on this.

[0035] Step 102: after determining that the current prompt is related to the context of the current prompt, the first target answer is retrieved from the semantic cache pre-deployed at the edge node to which the user belongs using the current prompt and the context of the current prompt as retrieval conditions;

[0036] Among them, the semantic similarity between the first target prompt corresponding to the first target answer and the current prompt is greater than a first threshold, and the semantic similarity between the context of the first target prompt and the context of the current prompt is greater than a second threshold.

[0037] Specifically, this solution needs to deploy semantic caches in advance at each edge node of the content distribution network before execution. Semantic cache is a method that uses vector space models (such as BERT or GloVe) to represent the semantics of text and performs cache retrieval by calculating similarity. It stores and retrieves specific semantic information, concepts, and knowledge in a structured manner, which helps to improve the performance and efficiency of AI applications. When the deployment is just completed, there is no cache in each semantic cache. If the user initiates a session to the large language model after the deployment is completed, the corresponding session information will be stored (write cache) in the semantic cache in the edge node to which the user belongs after the session is completed. It should be noted that when writing the cache, it will be cached in the corresponding relationship of prompt-prompt context-answer. Each prompt cache has a corresponding answer. If the prompt has no context, the prompt is only cached in the corresponding relationship of prompt-answer.

[0038] After determining that the current prompt is related to the context of the current prompt, indicating that the semantics of the current prompt needs to be considered in conjunction with its context, the current prompt and the context of the current prompt are used as retrieval conditions to retrieve the first target answer in the cache in the semantic cache pre-deployed at the edge node to which the user belongs. The first target answer needs to satisfy the conditions that the semantic similarity between its corresponding first target prompt and the current prompt is greater than a first threshold, and the semantic similarity between the context of the first target prompt and the context of the current prompt is greater than a second threshold. If there is no prompt cache in the semantic cache whose semantic similarity with the current prompt is greater than the first threshold, the first target answer cannot be retrieved; if there is no context cache in the semantic cache whose semantic similarity with the context of the current prompt is greater than the second threshold, the first target answer cannot be retrieved.

[0039] In some embodiments, Figure 4 As shown, step 102 specifically includes the following steps:

[0040] Step 1021: Retrieve and read the prompt cache that is greater than a first threshold value and is consistent with the current prompt.

[0041] Specifically, after obtaining the current prompt input by the user, all prompt caches whose semantic similarity with the current prompt is greater than a first threshold are retrieved and read from the semantic cache of the edge node to which the user belongs, wherein the semantic similarity can be determined by methods such as vector cosine similarity. Before using this method, the text needs to be vectorized, the parameter scale of the text embedding model is small, and the computing cost is controllable. This solution does not limit the specific semantic similarity calculation method.

[0042] Step 1022: traverse all prompt caches and select prompt caches with context as preliminary prompts.

[0043] Specifically, after filtering out all the prompt caches, since each prompt cache is cached in the correspondence relationship of prompt-prompt context-answer or prompt-answer when storing, if there are multiple prompt caches, not every prompt cache has a context, and the current prompt needs to be compared with the context, so all the prompt caches are traversed in turn, and the prompt caches with context are filtered out as preliminary prompts.

[0044] Step 1023: Calculate the semantic similarity between the context of the current prompt and the context of each preliminary prompt in turn.

[0045] Specifically, after screening in step 1022, all the preliminary prompts meet the conditions that the semantic similarity with the current prompt is greater than the first threshold and has a context. At this time, all the preliminary prompts can be screened again according to the context of the current prompt. The semantic similarity between the context of the current prompt and the context of each preliminary prompt is calculated in turn, and the screening is performed according to whether each semantic similarity calculation result is greater than the second threshold.

[0046] Step 1024: Select a preliminary prompt whose context has a semantic similarity with the context of the current prompt greater than a second threshold as the first target prompt, and determine a corresponding first target answer based on the first target prompt.

[0047] Specifically, among all the preliminary prompts, the preliminary prompt whose context has a semantic similarity greater than the second threshold with the context of the current prompt is selected as the first target prompt. If there are multiple preliminary prompts whose contexts have a semantic similarity greater than the second threshold with the context of the current prompt, the corresponding preliminary prompt with the maximum semantic similarity is selected as the first target prompt, or any one preliminary prompt is selected as the first target prompt, and this embodiment does not impose specific restrictions on this. After determining the first target prompt, the answer stored corresponding to the first target prompt in the cache is the first target answer to be returned to the user, and the cache hits.

[0048] Step 103: If the first target answer is retrieved, the first target answer is returned to the user.

[0049] Specifically, if the first target answer can be retrieved in the semantic cache pre-deployed in the edge node to which the user belongs, it is a cache hit, and there is no need to request resources from the server where the large language model is located, and the first target answer can be directly returned to the user.

[0050] Step 104: If the first target answer is not retrieved, the current prompt is submitted to the large language model to obtain the answer and returned to the user, and the prompt, context and corresponding answer of the conversation information determined based on the conversation information between the user and the large language model are stored in the semantic cache pre-deployed by the edge node to which the user belongs.

[0051] Specifically, if the first target answer is not retrieved, it proves that there is a cache miss in the semantic cache of the current edge node, and it is necessary to request resources from the server where the large language model is located. Among them, the large language model can be: ChatGPT, LaMDA, PaLM, Gopher, GLM and other common large language models at home and abroad, and this application does not impose specific restrictions on this. If the first target answer is not retrieved, the current prompt is submitted to the large language model to obtain the answer and return it to the user. In addition, the prompt, context and corresponding answer of the dialogue information determined based on the dialogue information between the user and the large language model are stored in the semantic cache pre-deployed by the edge node to which the user belongs to enrich the semantic cache of the edge node. In other embodiments, as a supplementary embodiment of step 102, when it is determined that the current prompt is not related to the context of the current prompt, the method of this embodiment also includes:

[0052] The context of the current prompt is discarded, and the current prompt is used as a retrieval condition to retrieve a second target answer in the semantic cache pre-deployed at the edge node to which the user belongs; wherein the semantic similarity between the second target prompt corresponding to the second target answer and the current prompt is greater than a first threshold; if the second target answer is retrieved, the second target answer is returned to the user; if the second target answer is not retrieved, the current prompt is submitted to the large language model to obtain an answer and returned to the user, and the prompt of the dialogue information determined according to the dialogue information between the user and the large language model and the corresponding answer are stored in the semantic cache pre-deployed at the edge node to which the user belongs.

[0053] Specifically, when it is determined that the current prompt is not related to the context of the current prompt, it is proved that the context has no effect on the expression of the meaning of the current prompt, and the context of the current prompt does not need to be considered at this time. The context can be discarded, and only the current prompt is used as the retrieval condition to retrieve the second target answer. The second target answer only needs to satisfy that the semantic similarity between its corresponding second target prompt and the current prompt is greater than the first threshold. In other words, the second target prompt can have a context or not, and this application does not impose specific restrictions on this. If the cache is hit and the second target answer is retrieved, the second target answer can be directly returned to the user. If the cache is not hit, a request is made to the server where the large language model is located, and the obtained answer and the current prompt are written into the cache in a prompt-answer correspondence. The specific details about semantic similarity and retrieval are the same as those in the above embodiment, and this embodiment will not be repeated here.

[0054] Compared with the related art, the embodiment of the present invention deploys a semantic cache at each edge node of the content distribution network, and performs cache retrieval in the semantic cache of the edge node to which the user belongs according to different retrieval conditions based on the correlation between the current prompt input by the user and its context; if the correlation between the current prompt input by the user and the context is large, the first target answer is retrieved according to the current prompt of the user and the context of the current prompt as the cache retrieval condition in the semantic cache, and the semantic similarity between the prompt corresponding to the first target answer and the current prompt is greater than the first threshold, and the semantic similarity between the context of the first target prompt and the context of the current prompt is greater than the second threshold. If the first target answer cannot be retrieved in the semantic cache of the current node, the answer is obtained from the large language model according to the current prompt and returned to the user. When using the semantic cache, combining the correlation between the prompt and its context to perform retrieval under different conditions can take into account the hit rate and accuracy of the semantic cache at the same time, and there is no need to establish a corresponding adaptive cache library for each large language model. The semantic cache deployed at each edge node of the content distribution network can be docked with various large language models to meet various application scenarios, and has strong versatility.

[0055] Another embodiment of the present invention relates to an electronic device, such as Figure 5 As shown, it includes at least one processor 202; and a memory 201 that is communicatively connected to the at least one processor 202; wherein the memory 201 stores instructions that can be executed by the at least one processor 202, and the instructions are executed by the at least one processor 202 so that the at least one processor 202 can execute any of the above method embodiments.

[0056] Among them, the memory 201 and the processor 202 are connected in a bus manner, and the bus may include any number of interconnected buses and bridges, and the bus connects one or more processors 202 and various circuits of the memory 201 together. The bus can also connect various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art, and therefore, are not further described herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be one element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices on a transmission medium. The data processed by the processor 202 is transmitted on a wireless medium through an antenna, and further, the antenna also receives data and transmits the data to the processor 202.

[0057] The processor 202 is responsible for managing the bus and general processing, and can also provide various functions, including timing, peripheral interfaces, voltage regulation, power management and other control functions. The memory 201 can be used to store data used by the processor 202 when performing operations.

[0058] Another embodiment of the present invention relates to a computer-readable storage medium storing a computer program, which implements any of the above method embodiments when executed by a processor.

[0059] That is, those skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a program, and the program is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0060] Those skilled in the art will appreciate that the above-mentioned embodiments are specific examples for implementing the present invention, and in actual applications, various changes may be made thereto in form and detail without departing from the spirit and scope of the present invention.

Claims

1. An edge semantic caching method based on a content distribution network, characterized in that: include: Obtaining a current prompt input by a user, and determining a correlation between the current prompt and the context of the current prompt when the current prompt has a context; After determining that the current prompt is related to the context of the current prompt, the first target answer is retrieved from the semantic cache pre-deployed at the edge node to which the user belongs using the current prompt and the context of the current prompt as retrieval conditions; wherein the semantic similarity between the first target prompt corresponding to the first target answer and the current prompt is greater than a first threshold, and the semantic similarity between the context of the first target prompt and the context of the current prompt is greater than a second threshold; If the first target answer is retrieved, returning the first target answer to the user; If the first target answer is not retrieved, the current prompt is submitted to the large language model to obtain the answer and returned to the user, and the prompt, context and corresponding answer of the conversation information determined based on the conversation information between the user and the large language model are stored in the semantic cache pre-deployed by the edge node to which the user belongs.

2. The edge semantic caching method based on a content distribution network according to claim 1, characterized in that: The determining the correlation between the current prompt and the context of the current prompt includes: Determining the semantic completeness of the current prompt; If it is complete, it is determined that there is no correlation between the current prompt and the context of the current prompt; if it is incomplete, it is determined that there is correlation between the current prompt and the context of the current prompt.

3. The edge semantic caching method based on content distribution network according to claim 2 is characterized in that: After determining that the semantic integrity of the current prompt is incomplete, the method further includes: Determining whether a time interval between the current prompt and the context of the current prompt is greater than a preset third threshold; If so, it is determined that the current prompt is not related to the context of the current prompt; if not, it is determined that the current prompt is related to the context of the current prompt.

4. The edge semantic caching method based on a content distribution network according to claim 3 is characterized in that: After determining that the time interval is not greater than the third threshold, the method further includes: Determining whether the semantic relevance between the current prompt and the context of the current prompt is less than a preset fourth threshold; If so, it is determined that the current prompt is not related to the context of the current prompt; if not, it is determined that the current prompt is related to the context of the current prompt.

5. The edge semantic caching method based on a content distribution network according to claim 1, characterized in that: When it is determined that the current prompt is not related to the context of the current prompt, the method further includes: The context of the current prompt is discarded, and the second target answer is retrieved from the semantic cache pre-deployed by the edge node to which the user belongs using the current prompt as a retrieval condition; wherein the semantic similarity between the second target prompt corresponding to the second target answer and the current prompt is greater than the first threshold; If the second target answer is retrieved, returning the second target answer to the user; If the second target answer is not retrieved, the current prompt is submitted to the large language model to obtain the answer and returned to the user, and the prompt of the dialogue information determined based on the dialogue information between the user and the large language model and the corresponding answer are stored in the semantic cache pre-deployed by the edge node to which the user belongs.

6. The edge semantic caching method based on a content distribution network according to claim 1, characterized in that: Retrieving a first target answer from a semantic cache pre-deployed at an edge node to which the user belongs using the current prompt and the context of the current prompt as retrieval conditions, including: Retrieving and reading the prompt cache whose semantic similarity with the current prompt is greater than the first threshold; Traversing all the prompt caches, and selecting the prompt caches with context as preliminary prompts; Calculating the semantic similarity between the context of the current prompt and the context of each of the preliminary prompts in sequence; The preliminary prompt whose context has a semantic similarity with the context of the current prompt greater than the second threshold is selected as the first target prompt, and the corresponding first target answer is determined according to the first target prompt.

7. The edge semantic caching method based on a content distribution network according to claim 6, characterized in that: The selecting the preliminary prompt whose semantic similarity between the context and the context of the current prompt is greater than the second threshold as the first target prompt comprises: If there are multiple contexts of the preliminary prompts whose semantic similarity with the context of the current prompt is greater than the second threshold; The preliminary prompt corresponding to the calculated maximum value of the semantic similarity is selected as the first target prompt.

8. The edge semantic caching method based on a content distribution network according to claim 1, characterized in that: Before obtaining the current prompt input by the user, the method includes: deploying semantic caches at each edge node of the content distribution network.

9. An electronic device, characterized in that: include: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the edge semantic caching method based on a content distribution network according to any one of claims 1 to 8.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the edge semantic caching method based on a content distribution network according to any one of claims 1 to 8 is implemented.

Images