Data processing method and device for resource-constrained equipment and computer equipment

Abstract

The invention provides a data processing method and device for resource-constrained equipment and computer equipment, and relates to the technical field of computers and artificial intelligence, in particular to the fields of computer data processing, large language models and the like, and the implementation scheme is as follows: determining a resource budget constraint based on a resource use state of target hardware equipment; obtaining to-be-processed data, wherein the to-be-processed data comprises weight parameters associated with the target large language model; sampling the plurality of hierarchies based on the hierarchical index, performing precision reduction processing on the weight parameter of each hierarchy layer by layer according to a sampling sequence, and determining a marginal contribution value of each hierarchy; generating an interaction matrix based on the plurality of marginal contribution values of each level; constructing an objective function based on the marginal contribution value and the interaction matrix; and based on the solution of the target function, performing precision reduction processing on the weight parameter to obtain a processed weight parameter, and deploying a large language model represented by the processed weight parameter on the target hardware equipment.

Description

Technical Field

[0001] This disclosure relates to the fields of computer and artificial intelligence technology, particularly to the fields of computer data processing and large language models, and especially to a data processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product for resource-constrained devices. Background Technology

[0002] As the scale of large language model parameters continues to expand, the number of model parameters and computational load increase significantly, leading to problems such as high storage consumption and large computational overhead when deployed on resource-constrained devices. To reduce model deployment costs, existing technologies typically use quantization to compress the model. However, these methods allocate precision based on independent metrics for each layer, ignoring the inter-layer interaction effects that are prevalent in neural networks. In reality, when multiple layers are compressed to low precision simultaneously, the quantization noise they generate can superimpose or amplify, causing the actual performance of the model to be far lower than expected. Therefore, accurately assessing the impact of precision reduction on each layer and the inter-layer interaction relationships under limited computing resources, and accordingly deploying the reduced-precision weight parameters on resource-constrained devices to improve hardware resource utilization, has become an important technical direction. Summary of the Invention

[0003] This disclosure provides a data processing method, apparatus, computer device, computer-readable storage medium, and computer program product for resource-constrained devices.

[0004] According to one aspect of this disclosure, a data processing method for resource-constrained devices is provided, comprising: determining a resource budget constraint for limiting the resource occupancy limit of the target hardware device based on the resource usage status of the target hardware device; acquiring data to be processed, wherein the data to be processed includes weight parameters associated with a target large language model, the weight parameters being distributed across multiple levels; sampling multiple levels based on a hierarchical index to generate multiple sampling sequences, wherein in each sampling sequence, precision reduction processing is performed on the weight parameters of each level in the order of sampling, and a marginal contribution value for each level is determined, the marginal contribution value being used to characterize the impact of precision reduction processing on the performance indicators of the target large language model; constructing a covariance matrix based on the multiple marginal contribution values ​​of each level, and processing the covariance... The matrix undergoes diagonal shrinkage and the interaction terms in the covariance matrix are sparsified to generate an interaction matrix. This interaction matrix characterizes the impact of weight parameters at two different levels on the performance metrics of the target large language model when both are at low precision. Based on the marginal contribution value and the interaction matrix, an objective function is constructed and solved. This objective function quantifies the impact of single-level precision reduction on the performance metrics of the target large language model and the interaction effects between different levels. The solution to the objective function corresponds to the precision configuration of each level under resource budget constraints. Based on the solution to the objective function, precision reduction is performed on the weight parameters associated with the target large language model to obtain processed weight parameters. The large language model represented by the processed weight parameters is deployed on the target hardware device.

[0005] In some embodiments, determining a resource budget constraint to limit the upper limit of resource usage of the target hardware device based on the resource usage status of the target hardware device includes: obtaining the hardware specification parameters and operating status of the target hardware device, wherein the operating status is used to describe the memory usage status and computing resource status of the target hardware device; and determining the resource budget constraint based on the hardware specification parameters and operating status.

[0006] In some embodiments, in each sampling sequence, the weight parameters of each level are subjected to precision reduction processing layer by layer according to the sampling order, and the marginal contribution value of each level is determined, including: in the current sampling sequence, starting from the state where the weight parameters of all levels are in a high precision state, the weight parameters of the current level are reduced from high precision to low precision layer by layer according to the sampling order; and based on the calibration dataset, the difference between the performance index of the target large language model before the precision reduction processing of the current level and the performance index of the target large language model after the precision reduction processing is performed, wherein the difference is used as the marginal contribution value of the current level in the current sampling sequence, and the performance index is calculated using the negative log-likelihood method.

[0007] In some embodiments, the data processing method for resource-constrained devices further includes: averaging multiple marginal contribution values ​​calculated for each level in multiple sampling sequences to obtain the comprehensive influence of each level, wherein the comprehensive influence is used to characterize the impact on the performance index of the target large language model when the weight parameters of a single level are subjected to precision reduction processing.

[0008] In some embodiments, the data processing method for resource-constrained devices further includes: calculating the uncertainty of the overall influence at each level in real time during the sampling process, wherein the uncertainty is used to assess the stability of the overall influence; stopping sampling in response to the uncertainty satisfying a preset convergence condition; and adjusting the intensity of the diagonal shrinkage processing based on the number of samplings.

[0009] In some embodiments, constructing and solving an objective function based on marginal contribution values ​​and an interaction matrix includes: constructing an objective function based on the comprehensive influence at each level and the interaction terms in the interaction matrix, wherein the objective function is... , and This indicates the precision of the weight parameters selected at the i-th or j-th level. This represents the overall influence of the i-th level. This represents the impact on the performance metrics of the target large language model when the accuracy of the i-th and j-th levels is reduced simultaneously; and the objective function is solved under resource budget constraints with the target accuracy set as the variable range, where the target accuracy set is used to limit the range of candidate accuracy values ​​when performing accuracy reduction processing on the weight parameters associated with the target large language model.

[0010] According to one aspect of this disclosure, a data processing apparatus for resource-constrained devices is provided, comprising: a constraint determination module configured to determine a resource budget constraint for limiting the resource occupancy limit of the target hardware device based on the resource usage status of the target hardware device; a data acquisition module configured to acquire data to be processed, wherein the data to be processed includes weight parameters associated with a target large language model, the weight parameters being distributed across multiple levels; a hierarchical sampling module configured to sample multiple levels based on a hierarchical index to generate multiple sampling sequences, wherein in each sampling sequence, precision reduction processing is performed on the weight parameters of each level in the order of sampling, and a marginal contribution value for each level is determined, the marginal contribution value being used to characterize the impact of precision reduction processing on the performance indicators of the target large language model; and a matrix construction module configured to construct a matrix based on the multiple marginal contribution values ​​of each level. The system employs a covariance matrix, which undergoes diagonal shrinkage and sparsification of interaction terms to generate an interaction matrix. This interaction matrix characterizes the impact of weight parameters at two different levels simultaneously being of low precision on the performance metrics of the target large language model. A function construction module is configured to construct and solve an objective function based on marginal contribution values ​​and the interaction matrix. This objective function quantifies the impact of single-level precision reduction on the performance metrics of the target large language model and the interaction effects between different levels. The solution to the objective function corresponds to the precision configuration of each level under resource budget constraints. A data deployment module is configured to perform precision reduction on the weight parameters associated with the target large language model based on the solution to the objective function, resulting in processed weight parameters. The large language model represented by these processed weight parameters is then deployed on the target hardware device.

[0011] According to another aspect of this disclosure, a computer device is provided, comprising: at least one processor; and a memory having a computer program stored thereon, wherein the computer program, when executed by the at least one processor, causes the at least one processor to perform the methods provided above in this disclosure.

[0012] According to another aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, causes the processor to perform the methods provided above in this disclosure.

[0013] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, causes the processor to perform the methods provided above in this disclosure.

[0014] According to one or more embodiments of this disclosure, it is possible to reduce hardware memory usage and memory bandwidth pressure when running large language models, and optimize the resource utilization of hardware devices.

[0015] These and other aspects of this disclosure will be apparent from the embodiments described below, and will be elucidated with reference to the embodiments described below. Attached Figure Description

[0016] The accompanying drawings exemplify embodiments and form part of the specification, serving together with the textual description to explain exemplary implementations of the embodiments. The illustrated embodiments are for illustrative purposes only and do not limit the scope of this disclosure. Throughout the drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0017] Figure 1 This is a flowchart illustrating a data processing method for a resource-constrained device according to an exemplary embodiment.

[0018] Figure 2 This is a schematic diagram illustrating the marginal contribution value calculation process according to an exemplary embodiment.

[0019] Figure 3 This is a schematic block diagram illustrating a data processing apparatus for a resource-constrained device according to an exemplary embodiment.

[0020] Figure 4 An example computer device is shown in which any of the embodiments described herein may be implemented. Detailed Implementation

[0021] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0022] In this disclosure, unless otherwise stated, the use of terms such as "first," "second," etc., to describe various elements is not intended to limit the positional, temporal, or importance relationships of these elements; such terms are merely used to distinguish one element from another. In some examples, the first element and the second element may refer to the same instance of that element, while in other cases, based on the context, they may refer to different instances.

[0023] The terminology used in the description of the various examples described in this disclosure is for the purpose of describing particular examples only and is not intended to be limiting. Unless the context explicitly indicates otherwise, an element may be one or more unless the number of elements is specifically limited. As used herein, the term "multiple" means two or more, and the term "based on" should be interpreted as "at least partially based on". Furthermore, the terms "and / or" and "at least one of..." cover any one of the listed items and all possible combinations thereof.

[0024] In related technologies, as the scale of large language model parameters grows, their demand for video memory capacity and memory bandwidth during the inference stage has exceeded the hardware capacity of most edge devices. To reduce deployment costs, model quantization techniques are often used to reduce the precision of weight parameters to reduce storage consumption and computational overhead. However, existing quantization methods are mostly based on the independent statistical features of each layer or the sensitivity index of a single layer for precision configuration, lacking a holistic consideration of inter-layer dependencies and error propagation effects.

[0025] First, existing technical solutions rely on heuristic strategies to estimate the quantization impact of each layer, only considering the local impact of single-layer accuracy reduction on model performance, while ignoring the coupling effect generated when different layers reduce accuracy simultaneously. This leads to the accuracy allocation decision deviating from the optimal configuration, making it difficult to balance overall performance and hardware utilization efficiency under limited resource conditions.

[0026] Secondly, some existing technologies attempt to improve estimation accuracy by utilizing second-order information or large-scale matrix analysis. However, such methods are usually computationally complex and resource-intensive, making them difficult to apply in practical scenarios with large-scale models. They are especially unsuitable for deployment environments with limited computing power, bandwidth, or storage. Therefore, they still cannot meet the need for efficient and stable deployment on limited hardware platforms.

[0027] To this end, embodiments of this disclosure propose a more effective data processing method for resource-constrained devices. By performing multi-sequence sampling on each level of the target large language model and calculating the marginal contribution value after precision reduction, a covariance matrix reflecting the inter-layer correlation is constructed and an interaction matrix is ​​generated. Under resource budget constraints, an objective function is established and solved to obtain the optimal precision configuration for each level. This reduces the performance loss of the large language model on resource-constrained devices while optimizing hardware storage and bandwidth usage and improving hardware resource utilization.

[0028] Exemplary embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0029] Figure 1 This is a flowchart illustrating a data processing method for a resource-constrained device according to an exemplary embodiment.

[0030] like Figure 1As shown, the data processing method for resource-constrained devices includes steps S101 to S106.

[0031] In step S101, based on the resource usage status of the target hardware device, a resource budget constraint is determined to limit the upper limit of resource usage of the target hardware device.

[0032] In this example, since subsequent data processing needs to be performed within predetermined hardware resource boundaries, it is necessary to determine the maximum allowed hardware resource threshold. First, information about the resource usage status of the target hardware device can be obtained, including parameters such as the device's current available storage capacity and allocable runtime memory, to determine the maximum range of available resources that the target hardware device can provide during operation. Then, resource budget constraints are generated based on the maximum available resource range to limit the upper limit of resources allowed to be used by the target hardware device during program execution, thus deriving resource constraints that match the target hardware device.

[0033] In step S102, data to be processed is obtained, wherein the data to be processed includes weight parameters associated with the target large language model, and the weight parameters are distributed in multiple levels.

[0034] In this example, the target large language model can refer to the large language model to be subjected to precision reduction processing. First, the data to be processed must be obtained. For example, a pre-trained model file of the target large language model can be loaded from storage media via a data interface, and the weight parameters corresponding to the target large language model can be parsed to obtain them. These weight parameters are structured and distributed across multiple layers according to the model's network architecture. In this way, hierarchical weight data associated with the target large language model can be obtained, providing a data foundation for subsequent layer-based processing.

[0035] In step S103, multiple levels are sampled based on the hierarchical index to generate multiple sampling sequences. In each sampling sequence, the weight parameters of each level are processed by reducing precision in the order of sampling, and the marginal contribution value of each level is determined. The marginal contribution value is used to characterize the impact of reducing precision on the performance index of the target large language model after processing each level.

[0036] In the example, a unique hierarchical index can be assigned to each layer based on the hierarchical structure of the target large language model. Multiple different hierarchical permutations can then be generated as sampling sequences based on these indexes. For instance, if the target large language model has four layers, one sampling sequence could be layer 2-layer 4-layer 3-layer 1. Subsequently, for each sampling sequence, starting with all layers' weight parameters in a high-precision state, the weight parameters corresponding to the current layer are converted from high-precision to low-precision representation layer by layer according to the sequence's order. Using the above sampling sequence as an example, the weight parameters of layer 2 can first be reduced from high precision to low precision, and then the weight parameters of layers 4, 3, and 1 are reduced in precision sequentially. After each layer's precision reduction, the change in the target large language model's performance metrics can be determined, i.e., the marginal contribution value of each layer. This marginal contribution value represents the impact of precision reduction on the target large language model's performance metrics. By constructing multiple sampling sequences and repeating the process of reducing precision, the impact of each level on the overall model performance can be determined, avoiding the bias caused by a single sampling sequence.

[0037] In step S104, a covariance matrix is ​​constructed based on multiple marginal contribution values ​​of each level. The covariance matrix is ​​then diagonally shrunk and the interaction terms in the covariance matrix are sparsified to generate an interaction matrix. The interaction matrix is ​​used to characterize the impact of the weight parameters of two different levels on the performance index of the target large language model when both are of low precision.

[0038] In the example, some layers experience significant impacts on performance metrics when precision is reduced simultaneously. Therefore, to determine the nonlinear coupling relationship between different layers in the large language model during precision reduction, feature extraction methods can be employed. Specifically, a multidimensional covariance matrix can be constructed using multiple marginal contribution values ​​calculated from various sampling sequences for each layer as sample data. This allows for an initial assessment of the performance loss when different layers experience simultaneous precision reduction. In this covariance matrix, the diagonal elements represent the variance of the marginal contribution value for each layer. A larger variance indicates greater fluctuation in the marginal contribution value of that layer across different sampling sequences, meaning the layer's impact on overall performance is more unstable. The interaction terms in the covariance matrix are the off-diagonal elements, representing the covariance of the marginal contribution values ​​between corresponding two layers. A positive covariance indicates a high correlation between the two layers, and simultaneous precision reduction of these two layers would result in greater performance loss. A negative covariance indicates relative independence between the two layers, with a smaller impact on performance.

[0039] In the example, due to the limited number of samples, noise may be present. Therefore, a diagonal shrinkage operation can be performed on the covariance matrix to increase the proportion of diagonal elements and suppress estimation noise introduced by limited sampling. Furthermore, the interaction terms of the off-diagonal elements are sparsified, retaining only those that meet the threshold conditions, thus obtaining the final interaction matrix. This process reduces noise interference while highlighting the mutual influence between layers, enabling the obtained interaction matrix to effectively characterize the joint impact of different layer weight parameters on model performance metrics when they are simultaneously in a low-precision state, thereby improving the reliability of subsequent precision reduction processing.

[0040] In step S105, an objective function is constructed and solved based on the marginal contribution value and the interaction matrix. The objective function is used to quantify the impact of single-level precision reduction processing on the performance index of the target large language model and the interaction between each level. The solution of the objective function corresponds to the precision configuration of each level under the resource budget constraint.

[0041] In this example, to determine which layers to reduce precision under limited hardware resources, an objective function aimed at minimizing performance loss can be constructed. Specifically, the marginal contribution value corresponding to each layer can be used as the single-layer impact quantity to quantify the direct impact of reducing the precision of a single layer on the target large language model's performance metrics. Simultaneously, the off-diagonal elements (i.e., interaction terms) in the interaction matrix are used as interaction impact quantities to quantify the performance impact when two specific layers are simultaneously configured to low precision. Subsequently, the objective function is solved according to resource budget constraints to obtain the layer precision configuration that minimizes the overall performance loss. In this way, the solution can satisfy the resource budget constraint while minimizing the performance loss of the large language model, thus avoiding the overall performance bias caused by decisions based solely on single-layer metrics.

[0042] In step S106, based on the solution of the objective function, the weight parameters associated with the target large language model are subjected to a precision reduction process to obtain the processed weight parameters, wherein the large language model represented by the processed weight parameters is deployed on the target hardware device.

[0043] In the example, after obtaining the solution to the objective function, the precision configuration corresponding to each level can be determined based on the solution. Then, based on this precision configuration, the weight parameters associated with the target large language model are subjected to precision reduction processing. The processing formula can be... Where W represents the original weight parameters, This represents the weight parameters after precision reduction, where s is the scaling factor, clamp() is the truncation function, round() is the rounding function, and b is the target precision. The processed set of weight parameters is then obtained. Finally, the processed weight parameters are loaded onto the target hardware device, enabling the large language model represented by these weight parameters to run in the target hardware device environment.

[0044] Therefore, by evaluating the hierarchical marginal contribution value based on multi-sequence sampling and solving the objective function in combination with the interaction matrix, the storage usage and data transmission bandwidth of the target hardware device can be optimized while ensuring the performance stability of the target large language model, thereby improving the computational efficiency and resource utilization of the target hardware device.

[0045] In some embodiments, determining a resource budget constraint to limit the upper limit of resource usage of the target hardware device based on the resource usage status of the target hardware device includes: obtaining the hardware specification parameters and operating status of the target hardware device, wherein the operating status is used to describe the memory usage status and computing resource status of the target hardware device; and determining the resource budget constraint based on the hardware specification parameters and operating status.

[0046] In the example, to ensure that the weight parameters after precision reduction are adaptable to the specific deployment environment, hardware specifications, such as memory capacity and memory bandwidth limit, can first be read from the target hardware device's system interface or monitoring module. The dynamic operating status of the current hardware device can also be collected, such as current memory usage, remaining free memory, and computing unit load. Based on this collected raw data, calculations can be performed using preset hardware metrics, such as removing system process reserved space and currently occupied space from the total memory, thereby deriving the maximum available resource limit that the device can provide for the target large language model inference at the current moment.

[0047] Therefore, by considering both static hardware specifications and real-time operating status, resource budget constraints can be aligned with the device's hardware load capacity, thereby improving adaptability under actual deployment conditions.

[0048] In some embodiments, in each sampling sequence, the weight parameters of each level are subjected to precision reduction processing layer by layer according to the sampling order, and the marginal contribution value of each level is determined, including: in the current sampling sequence, starting from the state where the weight parameters of all levels are in a high precision state, the weight parameters of the current level are reduced from high precision to low precision layer by layer according to the sampling order; and based on the calibration dataset, the difference between the performance index of the target large language model before the precision reduction processing of the current level and the performance index of the target large language model after the precision reduction processing is performed, wherein the difference is used as the marginal contribution value of the current level in the current sampling sequence, and the performance index is calculated using the negative log-likelihood method.

[0049] In the example, after determining the current sampling sequence, the weight parameters of all levels of the target large language model can be initialized to full high precision. Then, according to the order in the sampling sequence, precision reduction is performed level by level, for example, by using a quantization formula to truncate the weight parameters of each level from high precision. Each time precision reduction is performed for the current level, the same calibration dataset can be used as input to the target large language model for inference computation, and the model's output can be evaluated. During this process, negative log-likelihood can be used as a performance evaluation metric. The first performance metric value before precision reduction and the second performance metric value after precision reduction are calculated, and the difference between the second and first performance metric values ​​is taken as the marginal contribution value of the current level at the current sampling sequence position.

[0050] By dynamically reducing the accuracy layer by layer from full high precision, it is possible to simulate the continuous loss of state of the model during the accuracy reduction process. This allows for the quantification of the actual impact of single-level accuracy reduction on model performance, providing a reliable basis for subsequent analysis based on hierarchical effects.

[0051] Figure 2 This is a schematic diagram illustrating the marginal contribution value calculation process according to an exemplary embodiment.

[0052] like Figure 2 As shown, firstly, the weight parameters of all levels are set to high precision, and the current sampling sequence is obtained. Then, it is determined whether all levels in the sampling sequence have been reduced in precision. If there are levels that have not been reduced in precision, they are selected according to the order of the levels in the sampling sequence. For example, in a sampling sequence with 4 levels, the order of the levels is level 2-level 3-level 4-level 1. Then, the precision reduction is performed in the order of level 2-level 3-level 4-level 1. After selecting the level that has not been reduced in precision, the performance index of the target large language model before the precision reduction operation is determined as P1. Then, the precision reduction operation is performed, and the performance index of the target large language model after the precision reduction operation is calculated as P2. The marginal contribution of this level in the current sampling sequence is the difference between P2 and P1. If it is determined that all levels have been processed, the operation on the current sampling sequence can be ended.

[0053] In some embodiments, the data processing method for resource-constrained devices further includes: averaging multiple marginal contribution values ​​calculated for each level in multiple sampling sequences to obtain the comprehensive influence of each level, wherein the comprehensive influence is used to characterize the impact on the performance index of the target large language model when the weight parameters of a single level are subjected to precision reduction processing.

[0054] In the example, due to the coupling effect between layers in the target large language model, the evaluation results under a single order may be biased. Therefore, it is necessary to average the multiple marginal contribution values ​​calculated for each layer in multiple sampling sequences to eliminate random noise introduced by different arrangement orders. First, for any specific layer in the model, the multiple marginal contribution values ​​calculated for that layer in all executed sampling sequences can be retrieved. Then, these marginal contribution values ​​can be averaged, and the result is the comprehensive influence degree corresponding to that layer. This comprehensive influence degree can be used to characterize the actual impact on the performance index of the target large language model when the weight parameters of a single layer are reduced in precision.

[0055] Therefore, by calculating the average of multiple marginal contribution values, we can eliminate random fluctuations caused by differences in sampling order, and at the same time accurately represent the correlation between changes in the accuracy of a single level and model performance.

[0056] In some embodiments, the data processing method for resource-constrained devices further includes: calculating the uncertainty of the overall influence at each level in real time during the sampling process, wherein the uncertainty is used to assess the stability of the overall influence; stopping sampling in response to the uncertainty satisfying a preset convergence condition; and adjusting the intensity of the diagonal shrinkage processing based on the number of samplings.

[0057] In the example, to save computational costs, the standard error, or uncertainty, of the overall influence at each level can be calculated in real time during the generation of the sampling sequence, thereby quantifying the convergence state of the overall influence. The uncertainty formula can be... Where M is the number of samples, Let be the standard error of the i-th layer. Let Var() be the marginal contribution value of layer i in the m-th sampling, and let Var() be the variance calculation function. Then, the uncertainty is compared with a pre-defined convergence condition. When the uncertainty meets the convergence condition, it can be determined that the overall influence of the current layer has reached a stable estimation state, thus stopping further sampling to avoid redundant computational overhead. This convergence condition can be... ,in, Let be the standard error of the i-th layer. This represents the overall influence of the i-th layer. This indicates the maximum permissible relative standard error. This is a set constant.

[0058] In the example, the intensity coefficient of the diagonal shrinkage of the covariance matrix can be determined by the number of samples completed at the end. That is, when the number of samples is small, a larger shrinkage intensity can be configured to suppress estimation noise; conversely, when the number of samples is large, a smaller shrinkage intensity can be configured.

[0059] Therefore, by calculating the uncertainty of the comprehensive influence of each level in real time during the sampling process and using it as the basis for convergence, invalid repeated calculations can be avoided, thereby reducing computational overhead and shortening evaluation time. At the same time, by dynamically adjusting the intensity of diagonal shrinkage processing according to the number of samplings, the random noise of the covariance matrix in the case of small samples can be corrected, making the data more reliable.

[0060] In some embodiments, constructing and solving an objective function based on marginal contribution values ​​and an interaction matrix includes: constructing an objective function based on the comprehensive influence at each level and the interaction terms in the interaction matrix, wherein the objective function is... , and This indicates the precision of the weight parameters selected at the i-th or j-th level. This represents the overall influence of the i-th level. This represents the impact on the performance metrics of the target large language model when the accuracy of the i-th and j-th levels is reduced simultaneously; and the objective function is solved under resource budget constraints with the target accuracy set as the variable range, where the target accuracy set is used to limit the range of candidate accuracy values ​​when performing accuracy reduction processing on the weight parameters associated with the target large language model.

[0061] In the example, we can construct an objective function containing linear terms and quadratic interaction terms, which consists of two parts: one part This section represents the correspondence between the comprehensive influence of each level and its accuracy variable, used to characterize the independent impact of single-level accuracy reduction on performance indicators; another part... This is a combination of interaction terms and paired precision variables in the interaction matrix, used to characterize the coupled impact on performance indicators when precision is reduced simultaneously at two levels. This allows the objective function to uniformly represent both single-level and inter-level synergistic effects. Based on this, a target precision set can be determined first, representing the range of candidate precision values. For example, the precision of weight parameters may need to be reduced to 2 or 4 digits. Then, using the target precision set as the variable domain, resource budget constraints can be introduced into the solution process, creating hardware resource limitations on the variable combination. Since smaller precision of weight parameters results in smaller memory usage, the memory size occupied by all weight parameters must meet the resource budget constraints of resource-constrained devices. The objective function is then solved under these constraints. For example, in the binary case, it can be transformed into a mixed-integer linear programming problem for global solution, obtaining the precision values ​​for each level under the resource constraints.

[0062] Therefore, by solving the objective function under the condition of satisfying the resource budget constraint, the performance index change trend and inter-layer correlation can be considered while determining the accuracy configuration of each level, so as to obtain the accuracy configuration result that meets the resource conditions while ensuring that the loss of model performance index is small.

[0063] Embodiments of this disclosure also provide a data processing apparatus for resource-constrained devices.

[0064] Figure 3 This is a schematic block diagram illustrating a data processing apparatus 300 for a resource-constrained device according to an exemplary embodiment.

[0065] In some embodiments, a data processing apparatus 300 for a resource-constrained device may include a constraint determination module 301, a data acquisition module 302, a hierarchical sampling module 303, a matrix construction module 304, a function construction module 305, and a data deployment module 306.

[0066] The constraint determination module 301 is configured to determine a resource budget constraint for limiting the upper limit of resource usage of the target hardware device based on the resource usage status of the target hardware device.

[0067] The data acquisition module 302 is configured to acquire data to be processed, wherein the data to be processed includes weight parameters associated with the target large language model, and the weight parameters are distributed in multiple levels.

[0068] The hierarchical sampling module 303 is configured to sample multiple levels based on the hierarchical index to generate multiple sampling sequences. In each sampling sequence, the weight parameters of each level are processed by reducing precision in the order of sampling, and the marginal contribution value of each level is determined. The marginal contribution value is used to characterize the impact of reducing precision on the performance index of the target large language model after processing each level.

[0069] The matrix construction module 304 is configured to construct a covariance matrix based on multiple marginal contribution values ​​at each level, and to perform diagonal shrinkage on the covariance matrix and sparsify the interaction terms in the covariance matrix to generate an interaction matrix. The interaction matrix is ​​used to characterize the impact of the weight parameters of two different levels on the performance index of the target large language model when both are of low precision.

[0070] The function construction module 305 is configured to construct an objective function and solve the objective function based on the marginal contribution value and the interaction matrix. The objective function is used to quantify the impact of single-level precision reduction processing on the performance index of the target large language model and the interaction effects between each level. The solution of the objective function corresponds to the precision configuration of each level under the resource budget constraint.

[0071] The data deployment module 306 is configured to perform precision reduction processing on the weight parameters associated with the target large language model based on the solution of the objective function to obtain processed weight parameters, wherein the large language model represented by the processed weight parameters is deployed on the target hardware device.

[0072] It should be understood that Figure 3 The constraint determination module 301, data acquisition module 302, hierarchical sampling module 303, matrix construction module 304, function construction module 305, and data deployment module 306 shown can respectively correspond to, as follows: Figure 1 The operations of steps S101, S102, S103, S104, S105, and S106 are shown. Therefore, the details of each aspect will not be elaborated here.

[0073] In some embodiments, the constraint determination module 301 may include a status acquisition module 3011 and a budget determination module 3012. The status acquisition module 3011 may be configured to acquire the hardware specifications and operating status of the target hardware device, wherein the operating status describes the memory usage and computing resource status of the target hardware device. The budget determination module 3012 may be configured to determine resource budget constraints based on the hardware specifications and operating status.

[0074] In some embodiments, the hierarchical sampling module 303 may include a precision reduction module 3031 and a difference calculation module 3032. The precision reduction module 3031 may be configured to, in the current sampling sequence, reduce the weight parameters of the current level from high precision to low precision layer by layer, starting from when all levels' weight parameters are in a high-precision state, according to the sampling order. The difference calculation module 3032 may be configured to, based on a calibration dataset, calculate the difference between the performance index of the target large language model before and after precision reduction processing of the current level, where the difference serves as the marginal contribution value of the current level in the current sampling sequence, and the performance index is calculated using negative log-likelihood.

[0075] In some embodiments, the data processing apparatus 300 for resource-constrained devices may further include a mean calculation module 307. The mean calculation module 307 may be configured to calculate the average of multiple marginal contribution values ​​calculated for each level in multiple sampling sequences to obtain the comprehensive influence of each level, wherein the comprehensive influence is used to characterize the impact on the performance index of the target large language model when the weight parameters of a single level are subjected to precision reduction processing.

[0076] In some embodiments, the data processing apparatus 300 for resource-constrained devices may further include an uncertainty determination module 308, a convergence judgment module 309, and an intensity adjustment module 310. The uncertainty determination module 308 may be configured to calculate the uncertainty of the overall influence at each level in real time during the sampling process, wherein the uncertainty is used to assess the stability of the overall influence. The convergence judgment module 309 may be configured to stop sampling in response to the uncertainty meeting a preset convergence condition. The intensity adjustment module 310 may be configured to adjust the intensity of the diagonal contraction processing based on the number of samplings.

[0077] In some embodiments, the function construction module 305 may include a target construction module 3051 and a result determination module 3052. The target construction module 3051 may be configured to construct a target function based on the comprehensive influence degree at each level and the interaction terms in the interaction matrix, wherein the target function is... , and This indicates the precision of the weight parameters selected at the i-th or j-th level. This represents the overall influence of the i-th level. This represents the impact on the performance metrics of the target large language model when the accuracy of both the i-th and j-th levels is reduced simultaneously. The result determination module 3052 can be configured to solve the objective function under resource budget constraints, with the target accuracy set as the variable range. The target accuracy set is used to limit the range of candidate accuracy values ​​when performing accuracy reduction processing on the weight parameters associated with the target large language model.

[0078] According to one aspect of this disclosure, a computer device is also provided, including a memory, a processor, and a computer program stored in the memory. The processor is configured to execute the computer program to implement the steps of any of the method embodiments described above.

[0079] According to one aspect of this disclosure, a non-transitory computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the steps of any of the method embodiments described above.

[0080] According to one aspect of this disclosure, a computer program product is also provided, which includes a computer program that, when executed by a processor, implements the steps of any of the method embodiments described above.

[0081] Figure 4An example computer device 400 is shown in which any of the embodiments described herein may be implemented. Computer device 400 may be used to implement one or more components of the systems and methods described above. Computer device 400 may include a bus 402 or other communication mechanism for communicating information, and one or more processors 404 coupled to the bus 402 for processing information. Processor 404 may be, for example, one or more general-purpose microprocessors.

[0082] Computer device 400 may also include main memory 406, such as random access memory (RAM), cache, and / or other dynamic storage devices, coupled to bus 402, for storing information and instructions to be executed by processor 404. Main memory 406 may also be used to store temporary variables or other intermediate information during the execution of instructions to be executed by processor 404. Such instructions, when stored in a storage medium accessible to processor 404, can make computer device 400 a special-purpose machine customized to perform the operations specified in the instructions. Main memory 406 may include non-volatile media and / or volatile media. Non-volatile media may include, for example, optical discs or magnetic disks. Volatile media may include dynamic memory. Common media formats may include, for example, floppy disks, collapsible disks, hard disks, solid-state drives, magnetic tapes or any other magnetic data storage media, CD-ROMs (read-only optical disc drives), any other optical data storage media, any physical media with a perforated arrangement, RAM (random access memory), DRAM (dynamic random access memory), PROM (programmable read-only memory) and EPROM (erasable programmable read-only memory), FLASH-EPROM (fast erase programmable read-only memory), NVRAM (non-volatile random access memory), any other memory chips or tape cartridges, or network versions of the above.

[0083] Computer device 400 may implement the techniques described herein using custom hardwired logic, one or more ASICs (Application-Specific Integrated Circuits) or FPGAs (Field-Programmable Gate Arrays), firmware, and / or program logic, which, when combined with computer device 400, enable computer device 400 to become a special-purpose machine or to be programmed therein. According to one embodiment, the techniques described herein are executed by computer device 400 in response to processor 404 executing one or more sequences of one or more instructions contained in main memory 406. Such instructions may be read into main memory 406 from another storage medium, such as storage device 408. Executing the sequence of instructions contained in main memory 406 causes processor 404 to perform the processing steps described herein. For example, the processes / methods disclosed herein may be implemented by computer program instructions stored in main memory 406. When these instructions are executed by processor 404, they may perform the steps shown in the corresponding figures and as described above. In alternative embodiments, hardwired circuitry may be used in place of or in combination with software instructions.

[0084] Computer device 400 also includes a network interface 410 coupled to bus 402. Network interface 410 can provide bidirectional data communication coupled to one or more network links connected to one or more networks. As another example, network interface 410 can be a local area network (LAN) card to provide data communication connectivity with a compatible LAN (or a WAN component communicating with a WAN (wide area network)). Wireless links can also be implemented.

[0085] The performance of certain operations can be distributed across processors, not just residing within a single machine, but deployed across many machines. In some exemplary embodiments, the processor or the processor-implemented engine may reside in a single geographic location (e.g., in a home environment, office environment, or server farm). In other exemplary embodiments, the processor or the processor-implemented engine may be distributed across many geographic locations.

[0086] Each process, method, and algorithm described in the preceding sections can be embodied in a code module executed by one or more computer systems or computer processors including computer hardware, and can be fully or partially automated by them. These processes and algorithms can be implemented, in part or in whole, in a specific application circuit.

[0087] When the functions disclosed herein are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Specific technical solutions (all or part) disclosed herein, or aspects contributing to the prior art, can be embodied in the form of a software product. This software product can be stored in a storage medium and includes instructions to cause a computer device (which may be a personal computer, server, network device, etc.) to perform all or part of the steps of the methods described in the embodiments of this application. The storage medium may include a flash drive, a portable hard drive, ROM, RAM, a magnetic disk, an optical disk, another medium suitable for storing program code, or any combination thereof.

[0088] The embodiments disclosed herein can be implemented via a cloud platform, server, or group of servers that interact with a client. The client can be a terminal device or a client registered by a user on the platform, wherein the terminal device can be a mobile terminal, a personal computer (PC), or any device that can install platform applications.

[0089] The various features and processes described above can be used independently or combined in various ways. All possible combinations and sub-combinations are intended to fall within the scope of this disclosure. Furthermore, certain method or process blocks may be omitted in some embodiments. The methods and processes described herein are not limited to any particular order, and associated blocks or states may be executed in other suitable orders. For example, described blocks or states may be executed in a non-specifically disclosed order, or multiple blocks or states may be combined in a single block or state. Exemplary blocks or states may be executed serially, in parallel, or otherwise. Blocks or states may be added to or removed from the disclosed exemplary embodiments. The exemplary systems and components described herein may be configured differently from those described. For example, elements may be added, removed, or rearranged compared to the disclosed exemplary embodiments.

[0090] The various operations of the exemplary methods described herein can be performed at least in part by an algorithm. An algorithm may consist of program code or instructions stored in memory (such as the non-transitory computer-readable storage medium described above). Such an algorithm may include a machine learning algorithm. In some embodiments, the machine learning algorithm may not be explicitly programmed into the computer to perform the function, but may learn from training data to obtain a predictive model for performing that function.

[0091] The various operations of the exemplary methods described herein can be performed at least in part by one or more processors, which are temporarily configured (e.g., by software) or permanently configured to perform the relevant operations. Whether temporarily or permanently configured, such processors can constitute the engine of a processor implementation whose operation is to perform one or more of the operations or functions described herein.

[0092] Similarly, the methods described herein can be implemented at least partially by a processor, where a specific processor or one or more processors are examples of hardware. For example, at least some operations of the methods can be performed by one or more processors or an engine implemented by a processor. Furthermore, one or more processors can also run in a “cloud computing” environment or as “Software as a Service” (SaaS) to support the execution of the relevant operations. For example, at least some operations can be performed by a group of computers (as an example of a machine including processors), which can be accessed via a network (e.g., the Internet) and through one or more appropriate interfaces (e.g., application programming interfaces (APIs)).

[0093] The performance of certain operations can be distributed across processors, not just residing within a single machine, but deployed across many machines. In some exemplary embodiments, the processor or the processor-implemented engine may reside in a single geographic location (e.g., in a home environment, office environment, or server farm). In other exemplary embodiments, the processor or the processor-implemented engine may be distributed across many geographic locations.

[0094] In this specification, multiple instances may implement components, operations, or structures described as a single instance. Although individual operations of one or more methods are described and illustrated as independent operations, one or more individual operations may be performed concurrently, and these operations are not required to be performed in the order shown. Structures and functionalities presented as independent components in the example configuration may be implemented as combined structures or components. Similarly, structures and functionalities presented as individual components may be implemented as independent components. These and other variations, modifications, additions, and improvements are all within the scope of this document.

[0095] As used herein, “or” is inclusive rather than exclusive unless explicitly stated or indicated by context. Furthermore, “and” is both common and individual unless explicitly stated or indicated by context. Moreover, multiple instances may be provided for the resources, operations, or structures described herein as a single example. Furthermore, the boundaries between various resources, operations, engines, and data stores are somewhat arbitrary, and specific operations are illustrated within the context of a particular illustrative configuration. The allocation of other functionalities is conceivable and may fall within the scope of various embodiments of this disclosure. Generally, structures and functionalities presented as independent resources in example configurations may be implemented as combined structures or resources. Similarly, structures and functionalities presented as individual resources may be implemented as independent resources. These and other variations, modifications, additions, and improvements are all within the scope of embodiments of this disclosure. Therefore, this specification and accompanying drawings should be viewed in an illustrative rather than restrictive sense.

[0096] The terms “comprising” or “including” are used to indicate the presence of a subsequently stated feature, but do not preclude the addition of other features. Conditional language, in particular, such as “may,” “can,” or “may,” unless specifically stated or otherwise understood in the context of use, is generally intended to express that certain embodiments include certain features, elements, and / or steps, while other embodiments do not. Therefore, such conditional language generally does not imply that a feature, element, and / or step is necessary in any way for one or more embodiments, or that one or more embodiments must include logic that, with or without user input or prompting, determines whether such features, elements, and / or steps are included in any particular embodiment, or whether they are to be performed in any particular embodiment.

Claims

1. A data processing method for resource-constrained devices, characterized in that, The method includes: Based on the resource usage status of the target hardware device, determine the resource budget constraint used to limit the upper limit of resource consumption of the target hardware device; Obtain data to be processed, wherein the data to be processed includes weight parameters associated with the target large language model, and the weight parameters are distributed across multiple levels; Based on the hierarchical index, the multiple levels are sampled to generate multiple sampling sequences. In each sampling sequence, the weight parameters of each level are reduced in precision according to the sampling order, and the marginal contribution value of each level is determined. The marginal contribution value is used to characterize the impact of the reduction in precision on the performance index of the target large language model. Based on multiple marginal contribution values ​​at each level, a covariance matrix is ​​constructed, and the covariance matrix is ​​diagonally shrunk and the interaction terms in the covariance matrix are sparsified to generate an interaction matrix. The interaction matrix is ​​used to characterize the impact of the weight parameters of two different levels on the performance index of the target large language model when both are of low precision. Based on the marginal contribution value and the interaction matrix, an objective function is constructed and solved. The objective function quantifies the impact of single-level precision reduction processing on the performance metrics of the target large language model and the interaction effects between different levels. The solution to the objective function corresponds to the precision configuration of each level under the resource budget constraint. Based on the solution of the objective function, the weight parameters associated with the target large language model are subjected to a precision reduction process to obtain the processed weight parameters, wherein the large language model represented by the processed weight parameters is deployed on the target hardware device.

2. The method according to claim 1, characterized in that, The process of determining the resource budget constraint for limiting the resource consumption limit of the target hardware device based on the resource usage status of the target hardware device includes: Obtain the hardware specifications and operating status of the target hardware device, wherein the operating status describes the memory usage and computing resource status of the target hardware device; and Based on the hardware specifications and the operating status, the resource budget constraint is determined.

3. The method according to claim 1 or 2, characterized in that, In each sampling sequence, the weight parameters of each level are processed layer by layer according to the sampling order to reduce precision, and the marginal contribution value of each level is determined, including: In the current sampling sequence, starting from when all levels of weight parameters are in a high-precision state, the weight parameters of the current level are progressively reduced from high precision to low precision according to the sampling order; and Based on the calibration dataset, the difference between the performance index of the target large language model before and after the precision reduction processing is performed on the current level is calculated. The difference is used as the marginal contribution value of the current level under the current sampling sequence. The performance index is calculated using the negative log-likelihood method.

4. The method according to claim 1 or 2, characterized in that, The method further includes: The average value of the multiple marginal contribution values ​​calculated for each level in the multiple sampling sequences is calculated to obtain the comprehensive influence of each level. The comprehensive influence is used to characterize the impact on the performance index of the target large language model when the weight parameters of a single level are subjected to precision reduction processing.

5. The method according to claim 4, characterized in that, The method further includes: During the sampling process, the uncertainty of the overall influence at each level is calculated in real time, wherein the uncertainty is used to assess the stability of the overall influence. In response to the uncertainty satisfying a preset convergence condition, the sampling is stopped; and The intensity of the diagonal shrinkage process is adjusted based on the number of samples taken.

6. The method according to claim 4, characterized in that, The step of constructing an objective function based on the marginal contribution value and the interaction matrix, and solving the objective function, includes: Based on the comprehensive influence degree at each level and the interaction terms in the interaction matrix, an objective function is constructed, wherein the objective function is: , and This indicates the precision of the weight parameters selected at the i-th or j-th level. This represents the overall influence of the i-th level. This indicates the impact of simultaneously reducing the precision of the i-th and j-th levels on the performance metrics of the target large language model; and Using the target precision set as the variable range, the objective function is solved under the resource budget constraint, wherein the target precision set is used to limit the range of candidate precision values ​​when performing precision reduction processing on the weight parameters associated with the target large language model.

7. A data processing apparatus for resource-constrained devices, characterized in that, The device includes: The constraint determination module is configured to determine a resource budget constraint for limiting the upper limit of resource usage of the target hardware device based on the resource usage status of the target hardware device. The data acquisition module is configured to acquire data to be processed, wherein the data to be processed includes weight parameters associated with the target large language model, and the weight parameters are distributed in multiple levels; The hierarchical sampling module is configured to sample the multiple levels based on the hierarchical index to generate multiple sampling sequences. In each sampling sequence, the weight parameters of each level are processed by reducing precision layer by layer according to the sampling order, and the marginal contribution value of each level is determined. The marginal contribution value is used to characterize the impact of reducing precision layer on the performance index of the target large language model. The matrix construction module is configured to construct a covariance matrix based on multiple marginal contribution values ​​at each level, and to perform diagonal shrinkage on the covariance matrix and sparsify the interaction terms in the covariance matrix to generate an interaction matrix, wherein the interaction matrix is ​​used to characterize the impact of the weight parameters of two different levels on the performance index of the target large language model when both are of low precision. A function construction module is configured to construct an objective function and solve the objective function based on the marginal contribution value and the interaction matrix. The objective function quantifies the impact of single-level precision reduction processing on the performance metrics of the target large language model and the interaction effects between different levels. The solution to the objective function corresponds to the precision configuration of each level under the resource budget constraint. The data deployment module is configured to perform precision reduction processing on the weight parameters associated with the target large language model based on the solution of the objective function to obtain processed weight parameters, wherein the large language model represented by the processed weight parameters is deployed on the target hardware device.

8. A computer device, characterized in that, The computer device includes: At least one processor; A memory having a computer program stored thereon, wherein, when executed by the at least one processor, the computer program causes the at least one processor to perform the method of any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to perform the method of any one of claims 1-6.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, causes the processor to perform the method of any one of claims 1-6.

Images