Large language model inference result adaptive reuse method and electronic device

Abstract

The embodiment of the application is suitable for the field of artificial intelligence technology, and provides a large language model inference result adaptive reuse method and an electronic device. The method comprises the following steps: acquiring a current first feature of a first user and an adaptive precision parameter; determining whether target inference data exists in a preset cache based on the precision parameter and the current first feature; if the target inference data exists in the preset cache, reusing the target inference data as an inference result of the current first feature; if the target inference data does not exist in the preset cache, determining an inference strategy based on the current first feature and a historical first feature, and determining an inference result of the current first feature based on the inference strategy; the inference strategy comprises differential inference, full-amount inference and inference reuse; the embodiment of the application can determine the precision parameter of the target inference data identified from the cache for reuse through the double dimensions of cache hit rate and result complaint rate, and avoids identifying reusable inference data by using a single index.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence technology, and in particular relates to an adaptive reuse method and electronic device for reasoning results of large language models. Background Technology

[0002] Large Language Models (LLMs) are a type of artificial intelligence model trained on massive amounts of text data using deep learning. They are capable of understanding and generating natural language. Under compliant conditions, LLMs can be used to infer from users' raw behavioral data and generate structured user feature descriptions, i.e., user profiles.

[0003] SaaS (Software as a Service) platforms face extreme cost pressures when using LLM (Local Level Management) to generate user profiles with hundreds of millions of users: Based on an estimated 100 million users and an average of 400 keywords per user, a single full-scale inference requires processing approximately 40 billion keywords, which, according to mainstream commercial API pricing, ranges from tens of thousands to hundreds of thousands of yuan. Daily incremental updates (millions of users changing their profiles) also generate consistently high costs.

[0004] To reduce resource consumption, one existing technique is to convert the user's natural language query into an embedding vector, use approximate nearest neighbor to search for similar questions, and return a cached response as a reused result if the cosine similarity exceeds a fixed threshold. However, this technique only sets a fixed threshold for a single metric for comparison and cannot adjust the returned results in real time in terms of accuracy and resource consumption.

[0005] Therefore, there is a need for an adaptive reuse method for inference results based on large language models. Summary of the Invention

[0006] In view of this, embodiments of this application provide an adaptive reuse method and electronic device for large language model inference results, which can realize adaptive multi-level reuse of inference results output by large language models that match user features under both co-tenant and cross-tenant conditions, avoiding full inference of all user features and reducing user profile inference costs.

[0007] The first aspect of this application provides an adaptive reuse method for large language model inference results, including: Obtain the first user's current first feature and adaptive precision parameters; Based on the accuracy parameter and the current first feature, determine whether target inference data exists in the preset cache; If the target inference data exists in the preset cache, then the target inference data is reused as the inference result of the current first feature.

[0008] If the target inference data is not present in the preset cache, an inference strategy is determined based on the current first feature and the historical first feature, and the inference result of the current first feature is determined based on the inference strategy; the inference strategy includes differential inference, full inference, and inference reuse; Wherein, the historical first feature is the feature of the first user's last successful inference; the precision parameter is determined by the cache hit rate and the result complaint rate, and the cache hit rate is determined by the success rate of the target inference data.

[0009] In some implementations of this application, the preset cache includes a tenant private cache corresponding to the first user; determining whether target inference data exists in the preset cache based on the precision parameter and the current first feature includes: Based on the accuracy parameter and the current first feature, it is determined whether the target inference data exists in the tenant private cache; If the target inference data exists in the tenant's private cache, it is determined that the target inference data exists in the preset cache. If the target inference data is not present in the tenant's private cache, it is determined that the target inference data is not present in the preset cache.

[0010] In some implementations of this application, the preset cache further includes a cross-tenant shared cache; determining that the target inference data does not exist in the preset cache when the target inference data does not exist in the tenant's private cache includes: If the target inference data is not present in the tenant's private cache, determine the differential privacy parameters; Based on the precision parameter, the current first feature, and the differential privacy parameter, determine whether the target inference data exists in the cross-tenant shared cache; If the target inference data exists in the cross-tenant shared cache, it is determined that the target inference data exists in the preset cache. If the target inference data is not present in the cross-tenant shared cache, it is determined that the target inference data is not present in the preset cache.

[0011] In some implementations of this application, the tenant private cache includes a first-level cache and a second-level cache; the first-level cache records corresponding second feature fingerprints and first historical inference data; the second-level cache records corresponding second features and historical inference data; determining whether the target inference data exists in the tenant private cache based on the precision parameter and the current first feature includes: Using the precision parameters and the current first feature, a first feature fingerprint is generated; If a target second feature fingerprint that is consistent with the first feature fingerprint exists in the first-level cache, it is determined that the target inference data exists in the tenant private cache, and the target inference data is the historical inference data corresponding to the target second feature fingerprint. If no target second feature fingerprint matching the first feature fingerprint exists in the first-level cache, the first similarity threshold is determined using the precision parameter, and the first feature similarity between the current first feature and the second feature in the second-level cache is calculated. If the first similarity of the feature reaches the first similarity threshold, it is determined that the target inference data exists in the tenant private cache. The target inference data is the historical inference data corresponding to the second feature whose first similarity reaches the first similarity threshold. If the first similarity of the feature does not reach the first similarity threshold, it is determined that the target inference data does not exist in the tenant private cache.

[0012] In some implementations of this application, determining whether the target inference data exists in the cross-tenant shared cache based on the precision parameter, the current first feature, and the differential privacy parameter includes: Noise is added to the current first feature using the differential privacy parameter to obtain a first perturbation feature; Based on the first perturbation feature, a corresponding first perturbation feature fingerprint is generated; If a target second perturbation feature fingerprint that is consistent with the first perturbation feature fingerprint exists in the cross-tenant shared cache, it is determined that the target inference data exists in the cross-tenant shared cache, and the target inference data is the historical inference data corresponding to the target second perturbation feature fingerprint; If no target second perturbation feature fingerprint that matches the first perturbation feature fingerprint exists in the cross-tenant shared cache, the second similarity threshold is determined using the precision parameter, and the feature second similarity between the first perturbation feature and each second perturbation feature in the cross-tenant shared cache is calculated. If the second similarity of the feature reaches the second similarity threshold, it is determined that the target inference data exists in the cross-tenant shared cache. The target inference data is the historical inference data corresponding to the second perturbation feature whose second similarity of the feature reaches the second similarity threshold. If the second similarity of the feature does not reach the second similarity threshold, it is determined that the target inference data does not exist in the cross-tenant shared cache.

[0013] In some implementations of the embodiments of this application, determining that the target inference data does not exist in the preset cache when the target inference data does not exist in the tenant private cache further includes: Determine the differential privacy budget pool corresponding to the first user; If the differential privacy budget pool does not reach the differential privacy parameter, skip the step of determining whether the target inference data exists in the cross-tenant shared cache, and determine that the target inference data does not exist in the preset cache.

[0014] In some implementations of this application, determining the reasoning strategy based on the current first feature and the historical first feature includes: Determine the first feature value of the current first feature and the second feature value of the historical first feature, and determine the difference value; If the difference value is less than a first difference threshold, the reasoning strategy is determined to be reasoning reuse; If the difference value is not less than a first difference threshold and the difference value is not greater than a second difference threshold, the reasoning strategy is determined to be differential reasoning. If the difference value is greater than the second difference threshold, the reasoning strategy is determined to be full-scale reasoning.

[0015] In some implementations of this application, if the inference strategy is differential inference, determining the inference result of the current first feature based on the inference strategy includes: Determine the differences between the current first feature and the historical first feature; Generate difference prompts for the differences; The target inference data is generated by using the differential prompt words and historical inference data corresponding to the historical first feature.

[0016] A second aspect of this application provides an apparatus for adaptive reuse of large language model inference results, comprising: The acquisition module is used to acquire the current first feature of the first user and the adaptive precision parameters.

[0017] The cache determination module is used to determine whether target inference data exists in the preset cache based on the precision parameter and the current first feature.

[0018] The cached inference data reuse module is used to reuse the target inference data as the inference result of the current first feature if the target inference data exists in the preset cache.

[0019] The inference strategy determination and execution module is used to determine an inference strategy based on the current first feature and the historical first feature if the target inference data is not present in the preset cache; and to determine the inference result of the current first feature based on the inference strategy; the inference strategy includes differential inference, full inference and inference reuse.

[0020] A third aspect of this application provides an electronic device including a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the electronic device enables the large language model inference result adaptive reuse method as described in the first aspect above.

[0021] A fourth aspect of this application provides a computer program product, including a computer program that, when run, causes the adaptive reuse method for large language model inference results as described in the first aspect above to be executed.

[0022] A fifth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the adaptive reuse method for large language model inference results as described in the first aspect above.

[0023] Compared with the prior art, the embodiments of this application have the following beneficial effects: In this embodiment, the current first feature of a first user and an adaptive precision parameter are obtained. The precision parameter is determined by the cache hit rate and the result complaint rate, and the cache hit rate is determined by the success rate of the target inference data. Based on the precision parameter and the current first feature, it is determined whether target inference data exists in a preset cache. If the target inference data exists in the preset cache, the target inference data is reused as the inference result of the current first feature. Thus, when it is determined that the target inference data in the cache is the same as or similar to the expected inference data of the current first feature, the target inference data is reused without calling a large language model to infer the current first feature, reducing the waste of computing resources. If the target inference data is not present in the preset cache, an inference strategy is determined based on the current first feature and the historical first feature, and the inference result of the current first feature is determined based on the inference strategy. The inference strategy includes differential inference, full inference, and inference reuse. The historical first feature is the feature of the first user's last successful inference. In the case that the target inference data is not present in the cache, the current first feature and the historical first feature of the first user are compared, and one of differential inference, full inference, and inference reuse is determined as the inference strategy for the current first feature. Thus, a multi-level inference strategy is adopted to balance the quality of the inference result of the current first feature and the consumption of computing resources. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a schematic diagram of an adaptive reuse method for reasoning results of a large language model provided in an embodiment of this application; Figure 2 This is a schematic diagram of an adaptive reuse device for reasoning results of a large language model provided in an embodiment of this application; Figure 3 This is a schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0026] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0027] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0028] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0029] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0030] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0031] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0032] The technical solution of this application will be described below through specific embodiments.

[0033] Reference Figure 1 The diagram illustrates an adaptive reuse method for reasoning results of a large language model provided in an embodiment of this application, which may specifically include the following steps: Step 101: Obtain the current first feature of the first user and the adaptive precision parameters.

[0034] The first user refers to the user whose user profile (i.e., the inference result mentioned below) needs to be determined. We can first obtain the full feature set of the first user. Then, select the most discriminative features from the full feature set. Each feature term generates a corresponding feature vector, which is the current first feature.

[0035] As an example, the 10 most distinctive features can be selected to generate the current first feature.

[0036] This application does not limit the method for determining the most distinctive features. As one example, it can be that specified features are manually selected based on the specific business scenario of the user profile. As another example, it can also process the feature set of some or all users, calculate the variance of the feature values ​​corresponding to each feature, and select the K feature items with the largest variance as the most distinctive features.

[0037] The accuracy parameter can be determined by the cache hit rate and result complaint rate fed back from downstream systems. The cache hit rate is the success rate of determining the target inference data from a preset cache. The result complaint rate is the proportion of target inference data determined from the cache and marked as "inaccurate" by downstream systems.

[0038] Step 102: Based on the accuracy parameter and the current first feature, determine whether target inference data exists in the preset cache.

[0039] Based on the accuracy parameter and the current first feature, it can be determined whether there is target inference data in the preset cache that matches the current first feature.

[0040] The target inference data can be historical inference data that is the same as or has a certain similarity to the expected inference data of the current first feature. The historical inference data can be user profiles generated by the large language model for other feature vectors before the current time. The expected inference data is the inference data obtained if the large language model is used to infer the current first feature. Step 103: If the target inference data exists in the preset cache, then the target inference data is reused as the inference result of the current first feature.

[0041] If the target inference data exists in the cache, the target inference data is directly reused as the inference result of the current feature, avoiding the need to regenerate the same or similar inference results using a large language model, thus reducing the waste of computing resources.

[0042] Step 104: If the target inference data is not present in the preset cache, then a reasoning strategy is determined based on the current first feature and the historical first feature, and the reasoning result of the current first feature is determined based on the reasoning strategy.

[0043] The first historical feature is the feature of the first user's last successful reasoning; the reasoning strategy includes differential reasoning, full reasoning, and reasoning reuse.

[0044] If the target inference data cannot be determined in the cache, the feature from the first user's last successful inference, i.e., the historical first feature, is obtained. By comparing the differences between the current first feature and the historical first feature, the inference strategy for the current first inference feature is determined. Among them, differential inference uses a large language model to infer some feature items of the current first feature, full inference uses all feature items of the current first feature for inference, and inference reuse uses the inference data corresponding to the historical first feature as the inference result of the current first feature. Through multi-level inference strategies, the quality of the inference result of the current first feature and the consumption of computing resources are balanced, avoiding the direct use of the large language model for re-inference in existing technologies.

[0045] In this embodiment, the current first feature of the first user and an adaptive precision parameter are obtained. The precision parameter is determined by the cache hit rate and the result complaint rate, and the cache hit rate is determined by the success rate of the target inference data. Based on the precision parameter and the current first feature, it is determined whether target inference data exists in the preset cache. If the target inference data exists in the preset cache, the target inference data is reused as the inference result of the current first feature. Thus, when it is determined that the target inference data in the cache is the same as or similar to the expected inference data of the current first feature, the target inference data is reused without calling the large language model to infer the current first feature, reducing the waste of computing resources. If the target inference data is not present in the preset cache, an inference strategy is determined based on the current first feature and the historical first feature, and the inference result of the current first feature is determined based on the inference strategy. The inference strategy includes differential inference, full inference, and inference reuse. The historical first feature is the feature of the first user's last successful inference. In the case that the target inference data is not present in the cache, the current first feature and the historical first feature of the first user are compared, and one of differential inference, full inference, and inference reuse is determined as the inference strategy for the current first feature. Thus, a multi-level inference strategy is adopted to balance the quality of the inference result of the current first feature and the consumption of computing resources.

[0046] In some implementations of this application, the preset cache includes a tenant private cache corresponding to the first user; determining whether target inference data exists in the preset cache based on the precision parameter and the current first feature includes: determining whether the target inference data exists in the tenant private cache based on the precision parameter and the current first feature; if the target inference data exists in the tenant private cache, determining that the target inference data exists in the preset cache; if the target inference data does not exist in the tenant private cache, determining that the target inference data does not exist in the preset cache.

[0047] In SaaS scenarios, data between different tenants typically needs to be isolated to ensure data privacy. Therefore, a tenant-specific cache can be set up for each user's corresponding tenant to store inference data within the same tenant.

[0048] Based on the precision parameter and the current first feature, determine whether there is target inference data in the tenant's private cache that matches the current first feature.

[0049] If the target inference data exists in the tenant's private cache, it is determined that the target inference data also exists in the preset cache, and as described in step 102 above, the target inference data is reused as the inference result of the current first feature. If the target inference data does not exist in the tenant's private cache, it is determined that the target inference data does not exist in the preset cache.

[0050] In some implementations of this application, the tenant private cache includes a first-level cache (L1 cache) and a second-level cache (L2 cache); the first-level cache records corresponding second feature fingerprints and first historical inference data; the second-level cache records corresponding second features and historical inference data; determining whether the target inference data exists in the tenant private cache based on the precision parameter and the current first feature includes: Using the precision parameter and the current first feature, a first feature fingerprint is generated. If a target second feature fingerprint identical to the first feature fingerprint exists in the first-level cache, it is determined that the target inference data exists in the tenant's private cache; the target inference data is historical inference data corresponding to the target second feature fingerprint. If no target second feature fingerprint identical to the first feature fingerprint exists in the first-level cache, a first similarity threshold is determined using the precision parameter, and a first feature similarity is calculated between the current first feature and the second feature in the second-level cache. If the first feature similarity reaches the first similarity threshold, it is determined that the target inference data exists in the tenant's private cache; the target inference data is historical inference data corresponding to the second feature whose first feature similarity reaches the first similarity threshold. If the first feature similarity does not reach the first similarity threshold, it is determined that the target inference data does not exist in the tenant's private cache.

[0051] Precision parameters can be used For each feature value in the current first feature, quantize and round it, then serialize it and calculate the SHA-256 digest to obtain the first feature fingerprint. :

[0052] in This indicates rounding to the nearest integer.

[0053] The system determines whether the target inference data exists in the first-level cache and the second-level cache separately. In the specific implementation, the determination of the target inference data in the first-level cache and the second-level cache can be performed asynchronously, or the first-level cache can be determined first and then the second-level cache can be determined.

[0054] For the first-level cache, it is determined whether a second feature fingerprint exists in the first-level cache that matches the first feature fingerprint. If it exists, this second feature fingerprint is determined as the target second feature fingerprint, and the historical inference data corresponding to the target second feature fingerprint is read as the target inference data. If the target inference data is in the first-level cache, the determination of the target inference data in the second-level cache can be terminated.

[0055] For the second-level cache, a first similarity threshold is determined using the precision parameter, and a first feature similarity is calculated between the current first feature and the second feature in the second-level cache. If the first feature similarity reaches the first similarity threshold, it is determined that target inference data exists in the second-level cache. The target inference data is the historical inference data corresponding to the second feature whose first feature similarity reaches the first similarity threshold. For example, the second feature with the highest feature similarity among those reaching the first similarity threshold can be used as the target second feature, and the historical inference data corresponding to the target second feature can be determined as the target inference data. If none of the first feature similarities reach the first similarity threshold, it is determined that target inference data does not exist in the tenant's private cache.

[0056] If the target inference data cannot be determined in either the first-level cache or the second-level cache, it is determined that the target inference data does not exist in the tenant's private cache.

[0057] The accuracy parameters in the embodiments of this application can be adaptively updated according to a certain update strategy.

[0058] In the specific implementation, the cache hit rate can be defined. Hit rate in the t-th evaluation period; Result quality complaint rate The percentage of downstream target inference data determined from the cache that is marked as "inaccurate".

[0059] Precision parameters The following rules can be used for adaptive adjustment:

[0060] in For independent learning rate, This is an indicator function. (Constraints) .

[0061] The specific controls are as follows: like and Increase precision parameters This increases the scope of similarity judgments and improves the hit rate of target feature data. If Then reduce the precision parameter This reduces the scope of data reuse for inference and prevents quality degradation. If and If the accuracy parameter remains unchanged, it indicates that the current judgment of the target inference data has reached the Pareto optimal neighborhood. Steady-state condition: Make and Simultaneously, the expected value of the control law exponent term is zero, and the precision parameter automatically converges without the need for manual parameter adjustment. This is the lower bound for hit rate. The upper limit of the quality complaint rate, and It can be preset.

[0062] To maintain consistency between the first-level cache and the second-level cache, the first similarity threshold is adjusted when the precision parameter is changed. Adjust according to the following formula:

[0063] in This is the linkage coefficient. This is the initial value for the first similarity threshold. This is the initial value for the precision parameter. Increasing the precision parameter lowers the first similarity threshold, expanding the matching range for the second feature in the second-level cache; conversely, decreasing it narrows the range.

[0064] Regarding the aforementioned fingerprint generation methods based on fingerprint features, this application provides a theoretical model for fingerprint collision probability: if each core feature The sample distribution is approximately as follows Then any two users The probability of sharing the same fingerprint feature is:

[0065] The fingerprint collision probability theoretical model reveals that the hit rate of feature fingerprints varies with the accuracy parameter. Monotonically increasing, but the standard deviation of each characteristic distribution is... The hit rate of the feature fingerprint determines the accuracy parameter. Sensitivity. When the distribution of business data undergoes seasonal changes, the fixed precision parameter... This will cause a systematic shift in the hit rate. By adaptively adjusting the accuracy parameters mentioned above, the accuracy parameters can be adapted to the seasonal drift of the data distribution.

[0066] As can be seen from the above, the embodiments of this application reuse historical reasoning data in the cache as the reasoning result of the current first feature, in order to quantify the effect of reusing historical reasoning data.

[0067] Specifically, the total expected inference cost can be defined as follows:

[0068] in, For the cost of a single large language model inference, To avoid business losses due to the degradation in quality caused by reusing inference data, For penalty weights.

[0069] Based on this formula, the optimal accuracy parameter can be determined as follows:

[0070] As can be seen from the above adaptive accuracy parameter, the accuracy parameter will approximate this optimal solution in a gradient-free manner. Therefore, there is no need to... and Building explicit models reduces the cost of full-scale inference. .

[0071] In other implementations of this application, the preset cache includes, in addition to the tenant-private cache described above, a cross-tenant shared cache; determining that the target inference data does not exist in the preset cache when the target inference data does not exist in the tenant-private cache includes: If the target inference data is not present in the tenant private cache, a differential privacy parameter is determined; based on the precision parameter, the current first feature, and the differential privacy parameter, it is determined whether the target inference data exists in the cross-tenant shared cache; if the target inference data exists in the cross-tenant shared cache, it is determined that the target inference data exists in the preset cache; if the target inference data is not present in the cross-tenant shared cache, it is determined that the target inference data does not exist in the preset cache.

[0072] In scenarios where the preset cache includes a tenant-private cache and a cross-tenant shared cache, if the target inference data is not present in the tenant-private cache, a differential privacy parameter is determined; based on the precision parameter, the current first feature, and the differential privacy parameter, it is determined whether the target inference data exists in the cross-tenant shared cache.

[0073] If the target inference data exists in the cross-tenant shared cache (i.e., if the target inference data does not exist in the tenant's private cache but exists in the cross-tenant shared cache), then the target inference data is determined to exist in the preset cache.

[0074] If the target inference data is not present in the cross-tenant shared cache (i.e., if the target inference data is not present in both the tenant's private cache and the cross-tenant shared cache), it is determined that the target inference data is not present in the preset cache.

[0075] In cross-tenant data access scenarios, the strength of data privacy protection is an important consideration. The privacy threat model is as follows: Assume the attacker is the tenant who owns the property. Internal personnel with legitimate access aim to infer tenant information by analyzing fingerprints in a cross-tenant shared index. User characteristic distribution.

[0076] Further explanation is provided regarding the differential privacy requirements for cross-tenant shared cache data access in this application embodiment: Perturbation fingerprint generation mechanism: The Laplace mechanism is applied to fingerprint generation. Specifically, let the features... The global sensitivity is (i.e., the maximum change of this feature across any two adjacent datasets), the differential privacy budget is Then add noise:

[0077] Perturbation feature fingerprint:

[0078] The above-mentioned perturbation feature fingerprinting mechanism satisfy - Differential privacy, that is, for any adjacent feature vectors (Only one component difference) and any set of outputs :

[0079] Laplace mechanism It is known that the following conditions are met. - Differential privacy. SHA-256 hashing is a deterministic post-processing technique. Due to the post-processing invariance of differential privacy, the overall mechanism remains unchanged. - Differential privacy: Based on the above-mentioned generation of perturbation feature fingerprints, the present application embodiment also satisfies differential privacy by determining whether target inference data exists in the cross-tenant shared cache based on the perturbation feature fingerprints.

[0080] In some implementations of this application, the step of basing the precision parameter, the current first feature, and the differential privacy parameter (i.e., the differential privacy budget mentioned above) is... The method for determining whether the target inference data exists in the cross-tenant shared cache includes: adding noise to the current first feature using the differential privacy parameter to obtain a first perturbation feature; generating a corresponding first perturbation feature fingerprint based on the first perturbation feature; if a target second perturbation feature fingerprint consistent with the first perturbation feature fingerprint exists in the cross-tenant shared cache, the target inference data is historical inference data corresponding to the target second perturbation feature fingerprint; if no target second perturbation feature fingerprint consistent with the first perturbation feature fingerprint exists in the cross-tenant shared cache, a second similarity threshold is determined using the precision parameter, and a second feature similarity is calculated between the first perturbation feature and each second perturbation feature in the cross-tenant shared cache; if the second feature similarity reaches the second similarity threshold, the target inference data exists in the cross-tenant shared cache, the target inference data is historical inference data corresponding to the second perturbation feature whose second feature similarity reaches the second similarity threshold; if the second feature similarity does not reach the second similarity threshold, the target inference data does not exist in the cross-tenant shared cache.

[0081] Following the aforementioned feature fingerprinting mechanism, a first perturbation feature fingerprint corresponding to the current first feature can be generated. Then, it is determined whether a second perturbation feature fingerprint, consistent with the first perturbation feature fingerprint, exists in the cross-tenant shared cache.

[0082] If a second perturbation fingerprint that is consistent with the first perturbation fingerprint exists in the cross-tenant shared cache, then the second perturbation fingerprint is determined to be the target second perturbation fingerprint, and the historical inference data corresponding to the target second perturbation fingerprint is determined to be the target inference data.

[0083] If no second perturbation fingerprint matching the first perturbation fingerprint exists in the cross-tenant shared cache, the precision parameter is used to determine the second similarity threshold. Since the process of generating the perturbation feature introduces noise, the second similarity threshold can be greater than the first similarity threshold.

[0084] Calculate the second similarity between the first perturbation feature and each of the second perturbation features in the cross-tenant shared cache. If any feature's second similarity reaches a second similarity threshold, it is determined that target inference data exists in the cross-tenant shared cache. The target inference data is the historical inference data corresponding to the second feature whose second similarity reaches the second similarity threshold. For example, the second feature with the highest feature similarity among those reaching the second similarity threshold can be used as the target second feature, and the historical inference data corresponding to the target second feature can be determined as the target inference data. If none of the feature's second similarity reaches the second similarity threshold, it is determined that target inference data does not exist in the cross-tenant shared cache.

[0085] As can be seen from the above perturbation fingerprint generation mechanism, the hit rate (utility) of cross-tenant shared cache varies with privacy parameters. Increased size leads to improved fingerprint stability (reduced noise, more stable fingerprints), and enhanced privacy protection. Decrease and increase.

[0086] Let the original fingerprint be The fingerprint after disturbance is The probability of a collision between the two (i.e., the upper bound of the cross-tenant hit rate):

[0087] optimal This can be determined by maximizing the cross-tenant hit rate under privacy protection constraints (configurable by the business).

[0088] In some implementations of this application, to ensure privacy strength, this application provides a cumulative privacy loss management mechanism for continuous... For cross-tenant queries, the cumulative loss is defined using the Basic Composition theorem:

[0089] Using advanced synthesis (Rényi differential privacy) can achieve tighter boundaries:

[0090] Therefore, a privacy budget pool can be maintained for each tenant. Each cross-tenant query consumes ,when Automatically stop cross-tenant sharing.

[0091] Based on the cumulative privacy loss management mechanism, in the case where the target inference data does not exist in the tenant's private cache, after determining the differential privacy parameter, the differential privacy budget pool corresponding to the first user is determined; if the differential privacy budget pool does not reach the differential privacy parameter, the determination of whether the target inference data exists in the cross-tenant shared cache is skipped, and it is determined that the target inference data does not exist in the preset cache. In some implementations of this application, determining the inference strategy based on the current first feature and the historical first feature includes: determining a first feature value of the current first feature and a second feature value of the historical first feature, and determining a difference value; if the difference value is less than a first difference threshold, determining the inference strategy as inference reuse; if the difference value is not less than the first difference threshold and the difference value is not greater than a second difference threshold, determining the inference strategy as differential inference; and if the difference value is greater than the second difference threshold, determining the inference strategy as full inference.

[0092] This application embodiment can maintain a feature snapshot of subsuccessful inference. This is the first characteristic of history.

[0093] The difference can be calculated using the following formula:

[0094] in, The first feature is the weight vector. The values ​​are determined based on the historical correlation coefficients between each feature and the output quality of the large language model (initial values ​​are set by domain experts and then automatically calibrated).

[0095] When the difference value is less than the first difference threshold ( In the case of ), the reasoning strategy is determined to be reasoning reuse; when the difference value is not less than the first difference threshold and the difference value is not greater than the second difference threshold ( In the case of ), the reasoning strategy is determined to be difference reasoning; when the difference value is greater than the second difference threshold ( In the case of ), the reasoning strategy is determined to be full-scale reasoning.

[0096] In practical implementation, the semantic similarity can be calculated by periodically sampling the inference results obtained from differential inference and the full inference results obtained during the same period (sampling rate 0.1%). (Using a lightweight embedding model). If (Default 0.90), automatically adjusted downwards. Increase the proportion of full-scale reasoning to prevent quality degradation.

[0097] For multi-level inference strategies, the expected cost-saving formula is... set up , , , .

[0098] Daily incremental update expected LLM cost:

[0099] Savings percentage:

[0100] Where γ represents the proportion of re-inference required for differential inference relative to full inference for the same current first feature.

[0101] In some implementations of this application, if the reasoning strategy is differential reasoning, determining the reasoning result of the current first feature based on the reasoning strategy includes: determining the difference between the current first feature and the historical first feature; generating differential prompt words for the difference; and using the differential prompt words and historical reasoning data corresponding to the historical first feature to perform reasoning and generate target reasoning data.

[0102] For differential reasoning, a minimal modification guidance strategy can be adopted. First, differential prompt words are generated, and then a large language model is used to reason about the historical reasoning data corresponding to the differential prompt words and the historical first feature to obtain the target reasoning data.

[0103] For example: [Existing portrait (historical inference data corresponding to the first historical feature)] {old_narrative}.

[0104]

Feature Changes (Differences)

[0105] [Output Format (Differential Prompts)] [Original Sentence] → [Modified Sentence], each item changes on one line, only the affected part is modified.

[0106] The average output length of differential inference is 15-25% of that of full inference, and the response speed is improved by 3-5 times.

[0107] In the specific implementation, a version chain can also be generated for each user's inference results. Supports version rollback. Globally saved token count. It can be used for cost auditing:

[0108] It should be noted that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0109] Reference Figure 2 The diagram illustrates an adaptive reuse device for large language model inference results provided in an embodiment of this application, which may specifically include: The acquisition module 201 is used to acquire the current first feature of the first user and the adaptive precision parameters.

[0110] The cache determination module 202 is used to determine whether target inference data exists in the preset cache based on the precision parameter and the current first feature.

[0111] The cached inference data reuse module 203 is used to reuse the target inference data as the inference result of the current first feature if the target inference data exists in the preset cache.

[0112] The inference strategy determination and execution module 204 is used to determine an inference strategy based on the current first feature and the historical first feature if the target inference data is not present in the preset cache; and to determine the inference result of the current first feature based on the inference strategy; the inference strategy includes differential inference, full inference and inference reuse.

[0113] Wherein, the historical first feature is the feature of the first user's last successful inference; the precision parameter is determined by the cache hit rate and the result complaint rate, and the cache hit rate is determined by the success rate of the target inference data.

[0114] In some implementations of this application, the preset cache includes a tenant private cache corresponding to the first user; the cache determination module 202 includes: The tenant private cache determination submodule is used to determine whether the target inference data exists in the tenant private cache based on the precision parameter and the current first feature; if the target inference data exists in the tenant private cache, determine that the target inference data exists in the preset cache; if the target inference data does not exist in the tenant private cache, determine that the target inference data does not exist in the preset cache.

[0115] In some implementations of this application, the preset cache further includes a cross-tenant shared cache; the target inference data determination submodule includes: The differential privacy parameter determination unit is used to determine differential privacy parameters when the target inference data does not exist in the tenant private cache. A cross-tenant shared cache determination unit is used to determine whether the target inference data exists in the cross-tenant shared cache based on the precision parameter, the current first feature, and the differential privacy parameter; if the target inference data exists in the cross-tenant shared cache, determine that the target inference data exists in the preset cache; if the target inference data does not exist in the cross-tenant shared cache, determine that the target inference data does not exist in the preset cache.

[0116] In some implementations of this application, the tenant private cache includes a first-level cache and a second-level cache; the first-level cache records corresponding second feature fingerprints and first historical inference data; the second-level cache records corresponding second features and historical inference data; the tenant private cache determination submodule includes: The first feature fingerprint generation unit is used to generate a first feature fingerprint using the precision parameter and the current first feature; The first feature fingerprint matching unit is used to determine that the target inference data exists in the tenant private cache when a target second feature fingerprint that is consistent with the first feature fingerprint exists in the first-level cache. The target inference data is historical inference data corresponding to the target second feature fingerprint. The feature first similarity determination unit is used to determine a first similarity threshold using the precision parameter when there is no target second feature fingerprint in the first-level cache that is consistent with the first feature fingerprint, and to calculate the feature first similarity between the current first feature and the second feature in the second-level cache. The feature first similarity processing unit is configured to determine, when the feature first similarity reaches the first similarity threshold, that the target inference data exists in the tenant private cache, wherein the target inference data is historical inference data corresponding to the second feature whose feature first similarity reaches the first similarity threshold; and to determine, when the feature first similarity does not reach the first similarity threshold, that the target inference data does not exist in the tenant private cache.

[0117] In some implementations of this application, the cross-tenant shared cache determination unit includes: The first perturbation feature generation subunit is used to add noise to the current first feature using the differential privacy parameter to obtain the first perturbation feature; The first perturbation feature fingerprint generation subunit is used to generate a corresponding first perturbation feature fingerprint based on the first perturbation feature. The first perturbation feature fingerprint matching subunit is used to determine, when the cross-tenant shared cache contains a target second perturbation feature fingerprint that is consistent with the first perturbation feature fingerprint, that the target inference data exists in the cross-tenant shared cache, wherein the target inference data is historical inference data corresponding to the target second perturbation feature fingerprint; The feature second similarity determination subunit is used to determine a second similarity threshold using the precision parameter when there is no target second perturbation feature fingerprint consistent with the first perturbation feature fingerprint in the cross-tenant shared cache, and to calculate the feature second similarity between the first perturbation feature and each second perturbation feature in the cross-tenant shared cache. The feature second similarity processing subunit is configured to determine, when the feature second similarity reaches the second similarity threshold, that the target inference data exists in the cross-tenant shared cache, wherein the target inference data is the historical inference data corresponding to the second perturbation feature whose feature second similarity reaches the second similarity threshold; and to determine, when the feature second similarity does not reach the second similarity threshold, that the target inference data does not exist in the cross-tenant shared cache.

[0118] In some implementations of this application, the tenant private cache determination submodule further includes: The differential privacy budget pool determination submodule is used to determine the differential privacy budget pool corresponding to the first user; The differential privacy budget processing submodule is used to skip the step of determining whether the target inference data exists in the cross-tenant shared cache and to determine that the target inference data does not exist in the preset cache when the differential privacy budget pool does not reach the differential privacy parameter.

[0119] In some implementations of this application, the inference strategy determination and execution module 204 includes: The difference value determination submodule is used to determine the first feature value of the current first feature and the second feature value of the historical first feature, and to determine the difference value. The inference strategy determination submodule is used to determine the inference strategy as inference reuse when the difference value is less than a first difference threshold; to determine the inference strategy as differential inference when the difference value is not less than the first difference threshold and not greater than a second difference threshold; and to determine the inference strategy as full inference when the difference value is greater than the second difference threshold.

[0120] In some implementations of this application, if the inference strategy is differential inference, the inference strategy determination and execution module 204 is used to call the following sub-modules: The difference item determination submodule is used to determine the difference items between the current first feature and the historical first feature; The difference prompt word generation submodule is used to generate difference prompt words for the difference items; The differential reasoning submodule is used to perform reasoning using the differential prompt words and historical reasoning data corresponding to the historical first feature to generate target reasoning data.

[0121] This application provides an adaptive reuse device for large language model reasoning results. By using this device, the steps in the aforementioned method embodiments can be implemented.

[0122] As the apparatus embodiments are basically similar to the method embodiments, they are described in a relatively simple manner. For relevant details, please refer to the description in the method embodiment section.

[0123] Reference Figure 3 The diagram illustrates an electronic device according to an embodiment of this application. Figure 3 As shown, the electronic device 300 in this embodiment includes: a processor 310, a memory 320, and a computer program 321 stored in the memory 320 and executable on the processor 310. When the processor 310 executes the computer program 321, it implements the steps in various embodiments of the adaptive reuse method for large language model inference results described above, for example... Figure 1 Steps 101 to 104 are shown. Alternatively, when the processor 310 executes the computer program 321, it implements the functions of each module / unit in the above-described device embodiments, for example... Figure 2 The functions of modules 201 to 204 are shown.

[0124] For example, the computer program 321 can be divided into one or more modules / units, which are stored in the memory 320 and executed by the processor 310 to complete this application. The one or more modules / units can be a series of computer program instruction segments capable of performing specific functions, which can be used to describe the execution process of the computer program 321 in the electronic device 300. For example, the computer program 321 can be divided into an acquisition module, a cache determination module, a cache inference data reuse module, and an inference strategy determination and execution module, with the specific functions of each module as follows: The acquisition module is used to acquire the current first feature of the first user and the adaptive precision parameters.

[0125] The cache determination module is used to determine whether target inference data exists in the preset cache based on the precision parameter and the current first feature.

[0126] The cached inference data reuse module is used to reuse the target inference data as the inference result of the current first feature if the target inference data exists in the preset cache.

[0127] The inference strategy determination and execution module is used to determine an inference strategy based on the current first feature and the historical first feature if the target inference data is not present in the preset cache; and to determine the inference result of the current first feature based on the inference strategy; the inference strategy includes differential inference, full inference and inference reuse.

[0128] Wherein, the historical first feature is the feature of the first user's last successful inference; the precision parameter is determined by the cache hit rate and the result complaint rate, and the cache hit rate is determined by the success rate of the target inference data. The electronic device 300 can be a desktop computer, cloud server, or other computing device. The electronic device 300 may include, but is not limited to, a processor 310 and a memory 320. Those skilled in the art will understand that... Figure 3 This is merely one example of electronic device 300 and does not constitute a limitation on electronic device 300. It may include more or fewer components than shown, or combine certain components, or different components. For example, electronic device 300 may also include input / output devices, network access devices, buses, etc.

[0129] The processor 310 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0130] The memory 320 can be an internal storage unit of the electronic device 300, such as a hard disk or memory of the electronic device 300. The memory 320 can also be an external storage device of the electronic device 300, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD) card, flash card, etc., equipped on the electronic device 300. Furthermore, the memory 320 can include both internal and external storage units of the electronic device 300. The memory 320 is used to store the computer program 321 and other programs and data required by the electronic device 300. The memory 320 can also be used to temporarily store data that has been output or will be output.

[0131] This application also discloses a computer-readable storage medium storing a computer program that, when executed by a processor, implements the adaptive reuse method for large language model inference results as described in the foregoing embodiments.

[0132] This application also discloses a computer program product, including a computer program that, when run, causes the adaptive reuse method for large language model inference results as described in the foregoing embodiments to be executed.

[0133] The embodiments described above are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. An adaptive reuse method for reasoning results of a large language model, characterized in that, The method includes: Obtain the first user's current first feature and adaptive precision parameters; Based on the accuracy parameter and the current first feature, determine whether target inference data exists in the preset cache; If the target inference data exists in the preset cache, then the target inference data is reused as the inference result of the current first feature; If the target inference data is not present in the preset cache, an inference strategy is determined based on the current first feature and the historical first feature, and the inference result of the current first feature is determined based on the inference strategy; the inference strategy includes differential inference, full inference, and inference reuse; Wherein, the historical first feature is the feature of the first user's last successful inference; the precision parameter is determined by the cache hit rate and the result complaint rate, and the cache hit rate is determined by the success rate of the target inference data.

2. The method according to claim 1, characterized in that, The preset cache includes a tenant-private cache corresponding to the first user; determining whether target inference data exists in the preset cache based on the precision parameter and the current first feature includes: Based on the accuracy parameter and the current first feature, it is determined whether the target inference data exists in the tenant private cache; If the target inference data exists in the tenant's private cache, it is determined that the target inference data exists in the preset cache. If the target inference data is not present in the tenant's private cache, it is determined that the target inference data is not present in the preset cache.

3. The method according to claim 2, characterized in that, The preset cache also includes a cross-tenant shared cache; determining that the target inference data does not exist in the preset cache when the target inference data does not exist in the tenant's private cache includes: If the target inference data is not present in the tenant's private cache, determine the differential privacy parameters; Based on the accuracy parameter, the current first feature, and the differential privacy parameter, determine whether the target inference data exists in the cross-tenant shared cache; If the target inference data exists in the cross-tenant shared cache, it is determined that the target inference data exists in the preset cache. If the target inference data is not present in the cross-tenant shared cache, it is determined that the target inference data is not present in the preset cache.

4. The method according to claim 2 or 3, characterized in that, The tenant private cache includes a first-level cache and a second-level cache; the first-level cache records corresponding second feature fingerprints and first historical inference data; the second-level cache records corresponding second features and historical inference data; determining whether the target inference data exists in the tenant private cache based on the precision parameter and the current first feature includes: Using the precision parameters and the current first feature, a first feature fingerprint is generated; If a target second feature fingerprint that is consistent with the first feature fingerprint exists in the first-level cache, it is determined that the target inference data exists in the tenant private cache, and the target inference data is the historical inference data corresponding to the target second feature fingerprint. If no target second feature fingerprint matching the first feature fingerprint exists in the first-level cache, the first similarity threshold is determined using the precision parameter, and the first feature similarity between the current first feature and the second feature in the second-level cache is calculated. If the first similarity of the feature reaches the first similarity threshold, it is determined that the target inference data exists in the tenant private cache. The target inference data is the historical inference data corresponding to the second feature whose first similarity reaches the first similarity threshold. If the first similarity of the feature does not reach the first similarity threshold, it is determined that the target inference data does not exist in the tenant private cache.

5. The method according to claim 3, characterized in that, The step of determining whether the target inference data exists in the cross-tenant shared cache based on the precision parameter, the current first feature, and the differential privacy parameter includes: Noise is added to the current first feature using the differential privacy parameter to obtain a first perturbation feature; Based on the first perturbation feature, a corresponding first perturbation feature fingerprint is generated; If a target second perturbation feature fingerprint that is consistent with the first perturbation feature fingerprint exists in the cross-tenant shared cache, it is determined that the target inference data exists in the cross-tenant shared cache, and the target inference data is the historical inference data corresponding to the target second perturbation feature fingerprint; If no target second perturbation feature fingerprint that matches the first perturbation feature fingerprint exists in the cross-tenant shared cache, the second similarity threshold is determined using the precision parameter, and the feature second similarity between the first perturbation feature and each second perturbation feature in the cross-tenant shared cache is calculated. If the second similarity of the feature reaches the second similarity threshold, it is determined that the target inference data exists in the cross-tenant shared cache. The target inference data is the historical inference data corresponding to the second perturbation feature whose second similarity of the feature reaches the second similarity threshold. If the second similarity of the feature does not reach the second similarity threshold, it is determined that the target inference data does not exist in the cross-tenant shared cache.

6. The method according to claim 3 or 5, characterized in that, The step of determining that the target inference data does not exist in the preset cache when the target inference data does not exist in the tenant private cache further includes: Determine the differential privacy budget pool corresponding to the first user; If the differential privacy budget pool does not reach the differential privacy parameter, skip the step of determining whether the target inference data exists in the cross-tenant shared cache, and determine that the target inference data does not exist in the preset cache.

7. The method according to claim 1, characterized in that, The method of determining the reasoning strategy based on the current first feature and the historical first feature includes: Determine the first feature value of the current first feature and the second feature value of the historical first feature, and determine the difference value; If the difference value is less than a first difference threshold, the reasoning strategy is determined to be reasoning reuse; If the difference value is not less than a first difference threshold and the difference value is not greater than a second difference threshold, the reasoning strategy is determined to be differential reasoning. If the difference value is greater than the second difference threshold, the reasoning strategy is determined to be full-scale reasoning.

8. The method according to claim 7, characterized in that, If the reasoning strategy is differential reasoning, determining the reasoning result of the current first feature based on the reasoning strategy includes: Determine the differences between the current first feature and the historical first feature; Generate difference prompts for the differences; The target inference data is generated by using the differential prompt words and historical inference data corresponding to the historical first feature.

9. An electronic device, characterized in that, The device includes a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the electronic device performs the method as described in any one of claims 1-8.

10. A computer program product, characterized in that, Includes a computer program, which, when run, causes the method as described in any one of claims 1-8 to be performed.

Images