A compression method and electronic device for embodied intelligent models

Abstract

This invention discloses a compression method and electronic device for embodied intelligent models, relating to the field of artificial intelligence technology. The embodiments of this invention acquire multiple standard samples; obtain the predicted action sequence corresponding to each standard sample through the embodied intelligent model; for each channel, calculate the action sensitivity based on the channel gradient and channel weights; based on the hardware parameters of the embodied device and the action sensitivity of all channels, determine the target quantization precision for each channel from the candidate quantization precisions; quantize the embodied intelligent model according to the target quantization precision for each channel to obtain the compressed embodied intelligent model. This invention can quickly calculate action sensitivity through channel gradients and channel weights, taking into account both hardware parameters and action sensitivity when quantizing the embodied intelligent model, ensuring efficient reduction of the embodied intelligent model's size while avoiding accuracy loss due to over-compression.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and more specifically to a method for compressing embodied intelligence models and an electronic device. Background Technology

[0002] Embodied intelligence refers to an intelligent form of artificial intelligence that relies on physical or virtual carriers to perceive the environment through multiple modalities such as vision, language, and sensing, autonomously completes task decision-making and motion control, and achieves real-time interaction with the physical environment and autonomous execution of tasks, such as robots. Embodied intelligence models are neural network models that can be given physical or virtual carriers, enabling them to interact with the environment and complete tasks.

[0003] With the rapid development of embodied intelligence, Vision-Language-Action (VLA) models have become the mainstream architecture for generalized control of robots in complex environments. These models typically have billions to tens of billions of parameters, achieving end-to-end mapping from environmental perception to natural language understanding and then to low-level action output by absorbing massive amounts of multimodal data. However, when deployed on hardware-constrained devices, the sheer number of parameters leads to extremely severe inference latency. Traditional coarse-grained uniform quantization results in over-compression and significant performance degradation. Therefore, there is an urgent need for a method to compress embodied intelligence models to efficiently reduce model size and inference latency while avoiding the accuracy loss caused by over-compression. Summary of the Invention

[0004] To address the aforementioned problems, the present invention aims to provide a compression method and electronic device for embodied intelligent models, which can ensure efficient reduction of the size of embodied intelligent models while avoiding the loss of accuracy caused by excessive compression.

[0005] To solve the above-mentioned technical problems, the present invention provides the following technical solution: On one hand, the present invention provides a method for compressing embodied intelligence models, comprising: Acquire multiple standard samples, wherein the standard samples include sample instructions, sample visual information, and corresponding sample action sequences; The embodied intelligent model is used to obtain the predicted action sequence corresponding to each standard sample. The embodied intelligent model includes multiple model layers, each model layer includes multiple channels, and each channel has a corresponding channel weight. For each channel, an action sensitivity is calculated based on the channel gradient and the channel weight, wherein the channel gradient is calculated based on the deviation between the predicted action sequence and the sample action sequence; Based on the hardware parameters of the embodied device and the motion sensitivity of all channels, the target quantization precision for each channel is determined from the candidate quantization precisions. The embodied intelligence model is quantized according to the target quantization precision corresponding to each channel to obtain a compressed embodied intelligence model.

[0006] On the other hand, the present invention also provides a compression device for an embodied intelligent model, comprising: The sample acquisition module is used to acquire multiple standard samples, wherein the standard samples include sample instructions, sample visual information and corresponding sample action sequences. The prediction module is used to obtain the predicted action sequence corresponding to each standard sample through the embodied intelligence model. The embodied intelligence model includes multiple model layers, each model layer includes multiple channels, and each channel has a corresponding channel weight. The calculation module is used to calculate the action sensitivity for each channel based on the channel gradient and the channel weight, wherein the channel gradient is calculated based on the deviation between the predicted action sequence and the sample action sequence; The target determination module is used to determine the target quantization precision for each channel from the candidate quantization precisions based on the hardware parameters of the embodied device and the motion sensitivity of all channels. The compression module is used to quantize the embodied intelligence model according to the target quantization precision corresponding to each channel, so as to obtain the compressed embodied intelligence model.

[0007] On the other hand, the present invention also provides an electronic device including a processor and a memory, the memory storing a plurality of instructions; the processor loads instructions from the memory to execute steps in any of the embodied intelligence model compression methods provided by the present invention.

[0008] On the other hand, the present invention also provides a computer-readable storage medium storing a plurality of instructions adapted for loading by a processor to perform steps in any of the compression methods for embodied intelligence models provided by the present invention.

[0009] On the other hand, the present invention also provides a computer program product, including a computer program / instructions, which, when executed by a processor, implement the steps in the compression method of any embodied intelligence model provided by the present invention.

[0010] The beneficial effects of the technical solution provided by this invention include at least the following: In this embodiment of the invention, an embodied intelligence model is used to infer from standard samples to obtain predicted action sequences. These sequences are then combined with the sample action sequences to calculate channel gradients. The action sensitivity of each channel is calculated using the channel gradients and channel weights. Finally, the target quantization precision for each channel is determined using hardware parameters and action sensitivity. The embodied intelligence model is then quantized according to the target quantization precision for each channel. This invention can quickly calculate action sensitivity using channel gradients and channel weights. When quantizing the embodied intelligence model, both hardware parameters and action sensitivity are considered, ensuring efficient reduction of the embodied intelligence model's size while avoiding accuracy loss due to over-compression. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a schematic diagram illustrating an application scenario of the compression method for embodied intelligent models provided in this embodiment of the invention; Figure 2 This is a flowchart illustrating the compression method for embodied intelligent models provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of the compression device for the embodied intelligent model provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0013] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0014] It is understood that in specific embodiments of the present invention, data involving user information and related data requires user permission or consent, and the collection, use and processing of such data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0015] See also Figure 1The diagram illustrates an application scenario for the compression method of embodied intelligent models. This application scenario may include a terminal 101 and a server 102, which can exchange data via a network. The terminal 101 can be a mobile phone, tablet, smart Bluetooth device, computer, large screen, robot, etc.; the server 102 can be a single server or a server cluster consisting of multiple servers.

[0016] The user can send a compression start command to the server 102 via terminal 101. After receiving the command, the server 102 can acquire multiple standard samples, which include sample commands, sample visual information, and corresponding sample action sequences. It then uses an embodied intelligent model to obtain the predicted action corresponding to each standard sample. The embodied intelligent model includes multiple model layers, each layer containing multiple channels, and each channel has corresponding channel weights. For each channel, the action sensitivity is calculated based on the channel gradient and channel weights. The channel gradient is calculated based on the deviation between the predicted action sequence and the sample action sequence. Based on the hardware parameters of the embodied device and the action sensitivity of all channels, the target quantization precision for each channel is determined from the candidate quantization precisions. The embodied intelligent model is then quantized according to the target quantization precision for each channel to obtain the compressed embodied intelligent model. After completing the quantization processing of the embodied intelligent model, the server 102 can send a completion notification to the terminal 101, which will then display it to the user.

[0017] In this embodiment, a compression method for embodied intelligence models is provided, such as... Figure 2 As shown, the specific process of the compression method for this embodied intelligence model can be as follows: S110. Obtain multiple standard samples.

[0018] Embodied intelligence refers to an intelligent form of artificial intelligence that relies on physical or virtual carriers to perceive the environment through multiple modalities such as vision, language, and sensing, autonomously completes task decision-making and motion control, and achieves real-time interaction with the physical environment and autonomous execution of tasks, such as robots. Embodied intelligence models are neural network models that can be given physical or virtual carriers, enabling them to interact with the environment and complete tasks.

[0019] In this embodiment of the invention, the embodied intelligence model can refer to the Vision-Language-Action (VLA) model, and the standard samples can be considered as the training set used to train the VLA model.

[0020] Optionally, since the training set used to train the VLA model is large, a small number of demonstration trajectories covering multiple stages can be extracted from it, such as demonstration trajectories covering the grasping, moving, and placing stages. These data can be extracted from the training set as standard samples. It is understood that the VLA model can comprehensively process user input commands and collected visual information to output the sequence of actions to be executed. Therefore, the standard samples may include sample commands, sample visual information, and the corresponding sample action sequences.

[0021] S120. Obtain the predicted action sequence corresponding to each of the standard samples through the embodied intelligence model.

[0022] For each standard sample, the sample instructions and visual information in the standard sample can be input into the embodied intelligence model so that the embodied intelligence model can analyze and reason about the input information to obtain the predicted action sequence corresponding to each standard sample.

[0023] It should be noted that an embodied intelligence model can include multiple model layers, and each model layer can include multiple channels, each with its corresponding channel weights. The model layers can include linear layers, fully connected layers, and convolutional layers; for convolutional layers, the channels are the output channels of the convolutional layer; for linear and fully connected layers, the channels are the rows of the weight matrix.

[0024] S130. For each of the channels, calculate the action sensitivity based on the channel gradient and the channel weight.

[0025] For each channel in the embodied intelligence model, the channel gradient can be calculated, and combined with the channel weights, the action sensitivity of the channel can be calculated. The channel gradient is calculated through backpropagation on the embodied intelligence model, and the action sensitivity characterizes the degree of influence on the action sequence output by the embodied intelligence model; the greater the action sensitivity, the greater the influence of that channel on the action prediction result.

[0026] As one implementation method, when calculating action sensitivity based on channel gradients and channel weights, the action loss can be calculated based on the deviation between the predicted action sequence and the sample action sequence; for each channel, the gradient of the channel weight with respect to the action loss is calculated to obtain the channel gradient; and the action sensitivity of the channel is obtained by multiplying the channel gradient with the channel weight.

[0027] In S120, the predicted action sequence corresponding to the standard sample can be calculated through forward propagation. The sample action sequence in the standard sample is equivalent to the standard answer. The deviation between the predicted action sequence and the sample action sequence can be calculated, and the action loss can be constructed based on this deviation. Common loss methods can be used for the action loss; in this embodiment of the invention, L1 smoothing error can be used as the action loss.

[0028] Each model layer in the embodied intelligence model is expanded according to its output dimension and backpropagated to calculate the gradient of each channel weight with respect to the action loss, thus obtaining the channel gradient. Since the VLA model has a large number of parameters, performing forward and backward propagation across the entire network would consume significant computational power and be time-consuming. In this embodiment of the invention, for each standard sample, only one forward and backward propagation is needed on the embodied intelligence model, eliminating the need for multiple repetitions. This not only saves computational power but also rapidly improves the efficiency of action sensitivity evaluation.

[0029] The action sensitivity of a channel can be calculated by multiplying its gradient by its weight. See the formula below for details: ; in, D represents the motion sensitivity of the c-th channel in the l-th layer; D represents the set of multiple standard samples. The representation takes the expectation across multiple standard samples; The Hadamard product is represented by multiplying the channel weights and channel gradients element by element. The representation weight tensor is expanded along the channel dimension, i.e., channel weights; Characterizes action loss; Characterizes the channel gradient.

[0030] Therefore, the motion sensitivity for each channel can be calculated using the formula above. Considering motion sensitivity in subsequent quantization compression can prevent critical motion channels from being over-compressed.

[0031] S140. Based on the hardware parameters of the device and the motion sensitivity of all channels, determine the target quantization precision for each channel from the candidate quantization precisions.

[0032] Embodied devices refer to devices that run embodied intelligent models. These embodied devices can be integrated with devices that perform actions, or they can be set up separately. Among them, the devices that perform actions can refer to devices such as robots, robot dogs, and robotic arms, while embodied devices can refer to edge devices, end-side devices, etc. of robots.

[0033] The hardware parameters of the embodied device may include video memory, system memory, etc., which can be set according to actual needs. In this embodiment of the invention, the hardware parameters may refer to the physical video memory actually available in the embodied device. Model quantization refers to a model compression technique that maps and compresses the original high-precision floating-point values ​​such as weights and activation values ​​in the model into low-bit integer representations.

[0034] The candidate quantization precision is determined by its corresponding candidate quantization bit width. Bit width refers to the number of bits used to store a weight or activation value, commonly 16-bit, 8-bit, 4-bit, 2-bit, etc. Understandably, a larger quantization bit width results in higher quantization precision but also requires more GPU memory. In this embodiment of the invention, the candidate quantization precision can be represented as a set B = {0, 2, 4, 8, 16}, where 0 indicates pruning and 2 indicates quantization with a 2-bit bit width.

[0035] Due to the limitations of their hardware configuration, deploying a full-precision embodied intelligence model on an embodied device results in extremely severe inference latency and memory overflow due to the large number of parameters. Therefore, based on the hardware parameters of the embodied device and the action sensitivity of the channels, the target quantization precision of a channel can be accurately determined from the candidate quantization precisions, so as to preserve the model's effectiveness to the maximum extent while compressing the embodied intelligence model.

[0036] Optionally, when determining the target quantization precision for each channel from the candidate quantization precisions based on the hardware parameters of the embodied device and the motion sensitivity of all channels, the global contribution of the embodied intelligent model can be calculated based on the motion sensitivity of all channels and the quantization benefit table, where the quantization benefit table includes the quantization benefit coefficient of each channel at each candidate quantization precision; the hardware parameters of the embodied device are obtained, and hardware constraints are constructed based on the hardware parameters; and the target quantization precision for each channel is calculated based on the hardware constraints, assuming the global contribution is maximized.

[0037] The quantization gain table can include the quantization gain coefficient for each channel at each candidate quantization precision. This quantization gain coefficient characterizes the information retention between the feature distribution before and after quantization. The quantization gain coefficient represents the amount of information retained; the larger the quantization gain coefficient, the higher the information retention. For example, when using 4-bit quantization, the corresponding quantization gain coefficient is 0.85, indicating that 85% of the information is retained when using 4-bit quantization.

[0038] As one implementation method, when calculating the global contribution of the embodied intelligence model based on the channel's operational sensitivity and the quantization benefit table, the following steps can be taken: for each channel, determine the quantization benefit coefficient of the channel from the quantization benefit table based on the target quantization accuracy corresponding to the channel; multiply the channel's action sensitivity by the quantization benefit coefficient to obtain the channel contribution of the channel; and sum the channel contributions of all channels to obtain the global contribution of the embodied intelligence model.

[0039] For each channel, if the target quantization precision for that channel is determined, the quantization benefit coefficient for that channel can be determined from the quantization benefit table. Multiplying the quantization benefit coefficient by the channel's motion sensitivity yields the channel contribution. Summing up the contributions of all channels calculates the global contribution. It should be noted that since the target quantization precision for each channel has not yet been solved, the quantization benefit coefficient is also unknown; therefore, the global contribution obtained here is an expression containing unknown terms.

[0040] Specifically, the global contribution can be expressed by the following formula: ; Where J represents the global contribution; L represents the number of model layers in the embodied intelligence model, and l represents the l-th model layer; The l-th model layer represents the number of channels, c represents the c-th channel, and K represents the number of candidate quantization precisions. Characterizes the motion sensitivity of the c-th channel in the l-th model layer; Characterize the precision of the selected candidate quantization; This represents the quantitative return coefficient retrieved from the quantitative return table. Among them, For binary decision variables, ;when When = 1, it indicates that the c-th channel of the l-th layer is assigned a candidate quantization precision. ;when When =0, it means that the candidate quantization precision has not been assigned.

[0041] As one implementation method, the quantitative profit table can be pre-calculated and stored in a designated location, and can be read from the designated location when needed, thereby improving quantitative efficiency.

[0042] The global contribution is an expression containing unknown terms, where the unknown term is the target quantization precision for each channel. In this embodiment of the invention, it is necessary to calculate the target quantization precision for each channel when the global contribution is maximized.

[0043] The solution process requires certain constraints. This can be achieved by obtaining the hardware parameters of the embodied device and constructing hardware constraints based on these parameters. In some implementations, constructing hardware constraints may involve obtaining the hardware parameters of the embodied device and setting resource thresholds based on these parameters; calculating the total resource requirements for all channels based on the target quantization accuracy corresponding to each channel and the type of model layer to which the channel belongs; and then constructing the hardware constraints using the total resource requirements, the system fixed resource requirements, and the resource thresholds.

[0044] The hardware parameters of the embodied device are obtained, and a resource threshold is set based on these parameters. The resource threshold refers to the upper limit of the embodied device's resources, and the resources consumed should be less than or equal to this threshold. Optionally, the resource threshold can be a resource value with reserved safety redundancy. Taking video memory as an example, if the obtained video memory is A, the set resource threshold can be less than or equal to the value of A. The specific setting can be determined according to actual needs.

[0045] After determining the resource threshold, the resources consumed by all selected channels must be less than or equal to the resource threshold. Since each model layer contains channels, the total resource requirement of all channels can be calculated based on the type of model layer to which the channel belongs.

[0046] As one implementation method, when calculating the total resource requirement for all channels, the channel parameter quantity can be determined for each channel based on the type of the model layer to which the channel belongs; the channel resource requirement can be calculated based on the channel parameter quantity and the target quantization accuracy; and the total resource requirement can be obtained by summing the resource requirements of all channels.

[0047] Here, the model layers can be convolutional layers or linear layers. In a convolutional layer, one input channel corresponds to one convolutional kernel, and its channel parameters can be the product of the kernel height, kernel width, and the number of input channels. In a linear layer, one output channel corresponds to the fully connected weights of one neuron, and its channel parameters can be the number of input channels. Using the channel parameter count and the target quantization precision, the channel resource requirement can be calculated. Adding the channel resource requirements of all channels yields the total resource requirement.

[0048] The hardware constraint is obtained by ensuring that the sum of the total resource requirement and the system's fixed resource requirement is less than the resource threshold.

[0049] In this embodiment of the invention, the hardware parameter is illustrated using video memory as an example. The video memory of a specific device has a certain upper limit. Ultimately, the total video memory usage of all channels, plus the system's fixed overhead, cannot exceed the physical video memory threshold with reserved safety redundancy.

[0050] Therefore, the constructed hardware constraints can be: ; Where L represents the number of model layers in the embodied intelligence model, and l represents the l-th model layer; The l-th model layer represents the number of channels, c represents the c-th channel, and K represents the number of candidate quantization precisions. For binary decision variables, ;when When = 1, it indicates that the c-th channel of the l-th layer is assigned a candidate quantization precision. ;when When = 0, it indicates that the candidate quantization precision has not been assigned; it represents the candidate quantization precision used for the c-th channel of the l-th layer. The amount of video memory used, in MB; The fixed video memory overhead of the system represents the fixed resource requirements of the system, which can be set according to the actual situation. The actual usable video memory of the device, i.e., the obtained hardware parameters; The safety redundancy coefficient can be set according to actual needs; This represents the upper limit of available video memory, i.e., the resource threshold; This represents the channel resource requirement, i.e., the video memory occupied by that channel.

[0051] In some implementations, the video memory calculation function may specifically be: ; in, The number of parameters is for a single channel, and varies depending on whether the model layer is a convolutional or linear layer. If the l-th layer is a convolutional layer, then... ;in, The kernel height is the height of the convolution kernel. The kernel width is the width of the convolution kernel. This represents the number of input channels. If the l-th layer is a linear layer, then... Multiply the single-channel parameter value by the selected candidate quantization precision, divide by 8 to convert it to bytes, then divide by... Convert to MB.

[0052] Meanwhile, the target quantization precision for each channel is selected from candidate quantization precisions, with an implicit constraint that each channel can only select one from all candidate quantization precisions. This implicit constraint can be denoted as the channel configuration uniqueness constraint, specifically as follows: .

[0053] Under hardware constraints and the uniqueness constraint of channel configuration, we solve for the target quantization accuracy of each channel while maximizing the global contribution.

[0054] Based on the above, solving for the target quantization precision for each channel under constraints transforms it into a multidimensional knapsack problem. When the number of channels and model layers is large, exhaustively searching for all candidate quantization precision combinations is computationally intensive and inefficient. In this embodiment, a pseudo-polynomial-time solver or a branch-and-bound solver based on dynamic programming is deployed. During the solution process, when the number of model layers is deep, leading to a large variable scale, the Lagrange relaxation method can be used to convert the hardware constraints into a penalty term for global contribution, thereby decoupling the global problem into hierarchical subproblems for parallel solution. The solver outputs the optimal solution vector; decoding the optimal solution vector yields the target quantization precision for each channel.

[0055] Specifically, decoding the optimal solution vector yields the precision used by each channel of each model layer, for example, in the following form: "Layer_01_Conv": [4,0,8,4,…] "Layer_02_Linear": [2,2,0,4,…] "Layer_03_ActionHead": [16,16,16,…] Taking the output of the first layer as an example, the solution for its corresponding channel is: x 1,1,0 =0, indicating that the first channel of the first layer is not selected; x 1,1,4 =1 indicates that the first channel of the first layer is selected as 4-bit, and 4-bit quantization needs to be performed; x 1,2,0 =1 indicates that the second channel of the first layer is selected as 0-bit, and pruning is required; x 1,3,8 =1 indicates that the third channel of the first layer is selected as 8-bit and 8-bit quantization needs to be performed.

[0056] When determining the target quantization accuracy of a channel, we consider not only the limitations of hardware parameters but also the motion sensitivity of each channel to ensure adaptive compression of each channel, maximizing the preservation of model performance and minimizing the impact on the accuracy of motion prediction.

[0057] In some implementations, the aforementioned quantization benefit table can be pre-processed and obtained as follows: a specified sample is extracted from the plurality of standard samples; the output features of the specified sample at each model layer are calculated when the embodied intelligent model is in full-precision state; the baseline features of each model layer are obtained; each model layer of the embodied intelligent model is labeled as a specified model layer, and the specified model layer is quantized according to the candidate quantization precision to obtain a quantized label model for each specified model layer; for each specified model layer, the information retention amount of each candidate quantization precision is calculated based on the quantized label model, baseline features, and layer type of the specified model layer; all information retention amounts are normalized to obtain the quantization benefit table.

[0058] For the multiple standard samples that have been obtained, some samples can be extracted from the multiple standard samples as designated samples. The number of designated samples extracted can be set according to actual needs, and no specific limit is made here.

[0059] When the embodied intelligence model is in full-precision mode, the output features of a specified sample at each model layer are calculated. In full-precision mode, no quantization or compression processing is performed on the model; the specified sample is input into the embodied intelligence model, and the output features corresponding to each model layer are obtained and used as the baseline features for each model layer.

[0060] To predict the feature fidelity of each model layer under different candidate quantization precipitates, each model layer of the embodied intelligence model can be labeled as a designated model layer. These designated model layers are then quantized according to the candidate quantization precipitates to obtain quantized labeled models for each designated model layer. The designated model layer is determined from all model layers; each model layer is identified as a designated model layer once, and then quantized according to the candidate quantization precipitates to obtain multiple quantized labeled models. For example, if there are 5 model layers and 4 candidate quantization precipitates, a total of 20 quantized labeled models can be obtained.

[0061] As one implementation method, when each model layer of the embodied intelligence model is labeled as a designated model layer, and the designated model layer is quantized according to the candidate quantization precision to obtain the quantized labeled model of each designated model layer, it can be that each model layer in the embodied intelligence model is labeled as a designated model layer to obtain the labeled model corresponding to each designated model layer; for each labeled model, the designated model layer is quantized according to each candidate quantization precision to obtain the quantized labeled model.

[0062] Specifically, a model layer is selected from the embodied intelligence model and marked as a specified model layer to obtain a labeled model. The number of labeled models is the same as the number of model layers, and a labeled model has one and only one specified model layer.

[0063] For each labeled model, a specified model layer can be quantized according to each candidate quantization precision, resulting in multiple quantized labeled models corresponding to that labeled model. Specifically, after quantizing a specified model layer within a labeled model to each candidate quantization precision, multiple corresponding quantized labeled models can be obtained. For each specified model layer, the information retention amount for each candidate quantization precision can be calculated based on the quantized labeled model of the specified model layer, baseline features, and layer type.

[0064] As one implementation method, calculating the information retention of each candidate quantization precision can be done by, for each quantization label model of a specified model layer, inputting the specified sample into the quantization label model and obtaining the output features of the specified model layer to obtain quantization features; if the layer type of the specified model layer is an intermediate feature layer, calculating the information retention based on the cosine similarity between the quantization features and the baseline features; if the layer type of the specified model layer is a probability output layer, calculating the information retention based on the relative entropy between the quantization features and the baseline features.

[0065] Each specified model layer corresponds to multiple quantization label models, and each quantization label model corresponds to a candidate quantization precision. For each quantization label model, specified samples can be input into the quantization label model for forward inference, and the output features of the specified model layer can be obtained as quantization features.

[0066] By utilizing quantized features and baseline features, the information retention of a specified model layer before and after quantization can be calculated. For embodied intelligence models, different layer types can be categorized according to their position and function. In this embodiment of the invention, layer types may include intermediate feature layers and probabilistic output layers. The intermediate feature layer is typically located after the input and before the final output, primarily responsible for extracting and fusing multimodal features, such as convolutional layers and linear projection layers. The probabilistic output layer, located at the very end of the model, maps the intermediate features to the final output of the task, such as an action classification head.

[0067] Depending on the layer type, the information retention can be calculated in different ways. Specifically, if the specified model layer type is an intermediate feature layer, the cosine similarity between the quantized features and the baseline features can be calculated. A higher cosine similarity indicates a greater similarity between the quantized and baseline features, meaning quantization has not caused excessive feature changes, and thus, a higher information retention. Conversely, a lower cosine similarity indicates a less similarity between the quantized and baseline features, meaning quantization has caused excessive feature changes, and thus, a lower information retention.

[0068] Specifically, the information retention amount can be calculated using the following formula: ; in, Characterizes the corresponding candidate quantization precision; The amount of information retained for the corresponding candidate quantization precision; N represents the number of specified samples. Characterizes the baseline feature corresponding to the i-th specified sample; Characterizes the quantization feature corresponding to the i-th specified sample.

[0069] If the specified model layer type is a probability output layer, then both the quantized features and the baseline features are probability distributions. Relative entropy, or KL divergence, can be used to measure the difference in distribution before and after quantization. A larger KL divergence indicates a greater distribution difference between the quantized features and the baseline features, and a smaller amount of information retained; conversely, a smaller KL divergence indicates a smaller distribution difference between the quantized features and the baseline features, and a larger amount of information retained.

[0070] Specifically, the information retention amount can be calculated using the following formula: ; in, The amount of information retained corresponding to the progress of candidate quantization; Characterize the smooth adjustment hyperparameters, which can be set according to actual needs; Characterizing the benchmark features; Characterize quantitative features.

[0071] The final calculated information retention amount is normalized to the [0,1] interval to form the quantitative return coefficient corresponding to the specified model layer. By performing the above operation on each specified model layer, the quantitative return table can be obtained.

[0072] It should be noted that the quantization return table contains the quantization return coefficients of each model layer at different candidate quantization precipitates. Since a model layer usually contains multiple channels, the quantization return coefficients of all channels corresponding to that layer at different candidate quantization precipitates are the same, and the quantization return coefficients corresponding to that layer are used.

[0073] S150. According to the target quantization precision corresponding to each channel, the embodied intelligent model is quantized to obtain the compressed embodied intelligent model.

[0074] Based on the target quantization precision for each channel, the embodied intelligent model can be quantized. It should be noted that when the target quantization precision is 0-bit, the quantization process is a pruning process, and when the target quantization precision is less than the original precision, the quantization process is a low-bit conversion process.

[0075] After quantizing each channel according to the target quantization precision, the compressed embodied intelligence model can be obtained.

[0076] The compressed embodied intelligent model can be loaded into the embodied device. The embodied device can capture image sequences of the external environment in real time and receive natural language commands from the user. The compressed embodied intelligent model uses the computing power of the embodied device to perform high-speed autoregressive forward inference and output multi-dimensional continuous motion commands. The underlying kinematic controller converts the motion commands into torque or position control signals of the joint motors, driving the device to perform the motion to complete the corresponding task.

[0077] The embodied intelligence model compression scheme provided in this invention can be applied to various scenarios. For example, taking robots as an example, the scheme provided in this invention can compress embodied intelligence models more efficiently and retain the model's performance to the maximum extent during compression, enabling stable and efficient operation even on robots and edge devices with limited hardware resources.

[0078] The method provided by this invention can obtain the predicted action sequence of the embodied intelligent model using standard samples, obtain the channel gradient through forward and backward propagation of a single data stream, and combine it with the channel weights to calculate the action sensitivity of the entire network channel, saving time. Considering the limitations of action sensitivity and hardware resources, the optimal quantization strategy is determined comprehensively, which can ensure that the quantized and compressed embodied intelligent model can run on low-configuration devices while maintaining good model performance, thereby achieving fast and effective compression of the embodied intelligent model.

[0079] To better implement the above methods, embodiments of the present invention also provide a compression device for embodied intelligent models. This compression device can be integrated into an electronic device, such as a terminal or server. The terminal can be a mobile phone, tablet computer, smart Bluetooth device, laptop computer, or personal computer; the server can be a single server or a server cluster composed of multiple servers.

[0080] For example, in this embodiment, the method of the present invention will be described in detail by taking the compression device of the embodied intelligent model specifically integrated into the server as an example.

[0081] For example, such as Figure 3 As shown, the compression device for the embodied intelligent model may include a sample acquisition module 210, a prediction module 220, a calculation module 230, a target determination module 240, and a compression module 250.

[0082] The sample acquisition module 210 is used to acquire multiple standard samples, wherein the standard samples include sample instructions, sample visual information and corresponding sample action sequences. Prediction module 220 is used to obtain the predicted action sequence corresponding to each standard sample through an embodied intelligent model. The embodied intelligent model includes multiple model layers, each model layer includes multiple channels, and each channel has a corresponding channel weight. The calculation module 230 is used to calculate the action sensitivity for each channel based on the channel gradient and the channel weight, wherein the channel gradient is calculated based on the deviation between the predicted action sequence and the sample action sequence; The target determination module 240 is used to determine the target quantization precision for each channel from the candidate quantization precisions based on the hardware parameters of the embodied device and the motion sensitivity of all channels. Compression module 250 is used to quantize the embodied intelligent model according to the target quantization accuracy corresponding to each channel to obtain the compressed embodied intelligent model.

[0083] In some embodiments, the calculation module 230 is specifically used for: The action loss is calculated based on the deviation between the predicted action sequence and the sample action sequence; For each channel, the gradient of the channel weight with respect to the action loss is calculated to obtain the channel gradient; The action sensitivity of a channel is obtained by multiplying the channel gradient by the channel weight.

[0084] In some embodiments, the target determination module 240 is specifically used for: Based on the action sensitivity of all channels and the quantization benefit table, the global contribution of the embodied intelligence model is calculated. The quantization benefit table includes the quantization benefit coefficient of each channel at each candidate quantization precision. Obtain the hardware parameters of the embodied device, and construct hardware constraints based on the hardware parameters; Based on the hardware constraints, calculate the target quantization accuracy for each channel under the condition of maximizing the global contribution.

[0085] In some embodiments, the target determination module 240 is specifically used for: For each channel, the quantization revenue coefficient of the channel is determined from the quantization revenue table based on the target quantization accuracy corresponding to the channel; Multiply the action sensitivity of the channel by the quantitative return coefficient to obtain the channel contribution degree corresponding to the channel; The global contribution of the embodied intelligent model is obtained by summing the channel contributions of all channels.

[0086] In some embodiments, the target determination module 240 is specifically used for: Obtain the hardware parameters of the embodied device, and set resource thresholds based on the hardware parameters; Based on the target quantization accuracy corresponding to each channel and the type of model layer to which the channel belongs, calculate the total resource requirement of all channels; Hardware constraints are constructed using the total resource requirement, the system fixed resource requirement, and the resource threshold.

[0087] In some embodiments, the target determination module 240 is specifically used for: For each channel, the number of channel parameters is determined based on the type of the model layer to which the channel belongs; Calculate the channel resource requirements based on the channel parameters and the target quantization accuracy; Sum the resource requirements of all channels to obtain the total resource requirement.

[0088] In some embodiments, the compression device 200 for embodied intelligent models further includes a quantization table acquisition module, which is specifically used for: Extract a specified sample from the plurality of standard samples; With the embodied intelligent model in full-precision state, the output features of the specified sample in each model layer are calculated to obtain the baseline features of each model layer; Each model layer of the embodied intelligent model is labeled as a designated model layer, and the designated model layer is quantized according to the candidate quantization precision to obtain the quantized labeled model of each designated model layer; For each specified model layer, the information retention of each candidate quantization precision is calculated based on the quantization label model, baseline features, and layer type of the specified model layer; The amount of information retained is normalized to obtain a quantitative revenue table.

[0089] In some embodiments, the quantization table acquisition module is specifically used for: For each quantization labeling model of a specified model layer, the specified sample is input into the quantization labeling model, and the output features of the specified model layer are obtained to obtain the quantization features; If the specified model layer is an intermediate feature layer, the information retention is calculated based on the cosine similarity between the quantized feature and the baseline feature; If the specified model layer is a probability output layer, the information retention is calculated based on the relative entropy between the quantized feature and the benchmark feature.

[0090] In some embodiments, the quantization table acquisition module is specifically used for: Each model layer in the embodied intelligent model is labeled as a specified model layer, thus obtaining the labeled model corresponding to each specified model layer; For each of the labeled models, the specified model layer is quantized according to each candidate quantization precision to obtain a quantized labeled model.

[0091] In practice, the above modules can be implemented as independent entities or combined in any way to be implemented as the same or several entities. For the specific implementation of the above modules, please refer to the previous method implementation examples, which will not be repeated here.

[0092] As shown above, the compression device for the embodied intelligent model in this embodiment can use the embodied intelligent model to infer from standard samples, obtain predicted action sequences, and combine them with the sample action sequences to calculate channel gradients. The action sensitivity of each channel is then calculated using the channel gradients and channel weights. Finally, the target quantization accuracy for each channel is determined using hardware parameters and action sensitivity, and the embodied intelligent model is quantized according to the target quantization accuracy for each channel. By quickly calculating action sensitivity using channel gradients and channel weights, and by considering both hardware parameters and action sensitivity when quantizing the embodied intelligent model, the device ensures efficient reduction of the embodied intelligent model's size while avoiding accuracy loss due to over-compression.

[0093] This invention also provides an electronic device, which can be a terminal, a server, or other similar devices. The terminal can be a mobile phone, tablet computer, smart Bluetooth device, laptop computer, personal computer, etc.; the server can be a single server or a server cluster composed of multiple servers, etc.

[0094] In some embodiments, the compression device for the embodied intelligent model can also be integrated into multiple electronic devices. For example, the compression device for the embodied intelligent model can be integrated into multiple servers, and the compression method for the embodied intelligent model of the present invention can be implemented by multiple servers.

[0095] In this embodiment, a server will be used as an example for detailed description. For example, ... Figure 4 As shown, it illustrates a structural schematic diagram of the electronic device involved in an embodiment of the present invention, specifically: The electronic device may include components such as a processor 310 with one or more processing cores, a memory 320 with one or more computer-readable storage media, a power supply 330, an input module 340, and a communication module 350. Those skilled in the art will understand that... Figure 4 The electronic device structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein: The processor 310 is the control center of the electronic device, connecting various parts of the device via various interfaces and lines. It executes various functions and processes data by running or executing software programs and / or modules stored in the memory 320, and by calling data stored in the memory 320. In some embodiments, the processor 310 may include one or more processing cores; in some embodiments, the processor 310 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may also not be integrated into the processor 310.

[0096] The memory 320 can be used to store software programs and modules. The processor 310 executes various functional applications and data processing by running the software programs and modules stored in the memory 320. The memory 320 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 320 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 320 may also include a memory controller to provide the processor 310 with access to the memory 320.

[0097] The electronic device also includes a power supply 330 that supplies power to the various components. In some embodiments, the power supply 330 can be logically connected to the processor 310 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 330 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0098] The electronic device may also include an input module 340, which can be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0099] The electronic device may also include a communication module 350. In some embodiments, the communication module 350 may include a wireless module, through which the electronic device can perform short-range wireless transmission, thereby providing users with wireless broadband internet access. For example, the communication module 350 can be used to help users send and receive emails, browse web pages, and access streaming media.

[0100] Although not shown, the electronic device may also include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 310 in the electronic device loads the executable files corresponding to the processes of one or more applications into the memory 320 according to the following instructions, and the processor 310 runs the applications stored in the memory 320, thereby implementing the steps in the methods of the various embodiments of the present invention.

[0101] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0102] As can be seen from the above, the electronic device provided in this embodiment of the invention can use an embodied intelligent model to infer from standard samples, obtain predicted action sequences, and combine them with sample action sequences to calculate channel gradients. The action sensitivity of each channel is then calculated using the channel gradients and channel weights. Finally, the target quantization accuracy for each channel is determined using hardware parameters and action sensitivity, and the embodied intelligent model is quantized according to the target quantization accuracy for each channel. By using channel gradients and channel weights to quickly calculate action sensitivity, and by considering both hardware parameters and action sensitivity when quantizing the embodied intelligent model, the size of the embodied intelligent model can be efficiently reduced while avoiding accuracy loss due to excessive compression.

[0103] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0104] To this end, embodiments of the present invention provide a computer-readable storage medium storing a plurality of instructions that can be loaded by a processor to execute steps in any of the embodied intelligence model compression methods provided in the embodiments of the present invention. The storage medium may include: read-only memory (ROM), random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0105] According to one aspect of the present invention, a computer program product or computer program is provided, comprising a computer program / instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer program / instructions from the computer-readable storage medium and executes the computer program / instructions, causing the electronic device to perform the methods provided in various optional implementations of the compression aspect of the embodied intelligence model or the acquisition aspect of the quantified revenue table provided in the above embodiments.

[0106] Since the instructions stored in the storage medium can execute the steps in any of the embodied intelligent model compression methods provided in the embodiments of the present invention, the beneficial effects that any of the embodied intelligent model compression methods provided in the embodiments of the present invention can achieve can be realized, as detailed in the preceding embodiments, and will not be repeated here.

[0107] The foregoing has provided a detailed description of a compression method and electronic device for an embodied intelligent model provided by embodiments of the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for compressing embodied intelligence models, characterized in that, The method includes: Acquire multiple standard samples, wherein the standard samples include sample instructions, sample visual information, and corresponding sample action sequences; The embodied intelligent model is used to obtain the predicted action sequence corresponding to each standard sample. The embodied intelligent model includes multiple model layers, each model layer includes multiple channels, and each channel has a corresponding channel weight. For each channel, an action sensitivity is calculated based on the channel gradient and the channel weight, wherein the channel gradient is calculated based on the deviation between the predicted action sequence and the sample action sequence; Based on the hardware parameters of the embodied device and the motion sensitivity of all channels, the target quantization precision for each channel is determined from the candidate quantization precisions. This includes: calculating the global contribution of the embodied intelligent model based on the motion sensitivity of all channels and a quantization benefit table, where the quantization benefit table includes the quantization benefit coefficient of each channel at each candidate quantization precision; obtaining the hardware parameters of the embodied device and constructing hardware constraints based on the hardware parameters; and calculating the target quantization precision for each channel under the condition of maximizing the global contribution, according to the hardware constraints. The embodied intelligence model is quantized according to the target quantization precision corresponding to each channel to obtain a compressed embodied intelligence model.

2. The method according to claim 1, characterized in that, For each of the channels, the motion sensitivity is calculated based on the channel gradient and the channel weight, including: The action loss is calculated based on the deviation between the predicted action sequence and the sample action sequence; For each channel, the gradient of the channel weight with respect to the action loss is calculated to obtain the channel gradient; The action sensitivity of a channel is obtained by multiplying the channel gradient by the channel weight.

3. The method according to claim 1, characterized in that, The global contribution of the embodied intelligence model is calculated based on the action sensitivity of all channels and the quantified benefit table, including: For each channel, the quantization revenue coefficient of the channel is determined from the quantization revenue table based on the target quantization accuracy corresponding to the channel; Multiply the action sensitivity of the channel by the quantitative return coefficient to obtain the channel contribution degree corresponding to the channel; The global contribution of the embodied intelligent model is obtained by summing the channel contributions of all channels.

4. The method according to claim 1, characterized in that, The process of obtaining the hardware parameters of the embodied device and constructing hardware constraints based on the hardware parameters includes: Obtain the hardware parameters of the embodied device, and set resource thresholds based on the hardware parameters; Based on the target quantization accuracy corresponding to each channel and the type of model layer to which the channel belongs, calculate the total resource requirement of all channels; Hardware constraints are constructed using the total resource requirement, the system fixed resource requirement, and the resource threshold.

5. The method according to claim 4, characterized in that, The calculation of the total resource requirements for all channels, based on the target quantization precision for each channel and the type of model layer to which the channel belongs, includes: For each channel, the number of channel parameters is determined based on the type of the model layer to which the channel belongs; Calculate the channel resource requirements based on the channel parameters and the target quantization accuracy; Sum the resource requirements of all channels to obtain the total resource requirement.

6. The method according to claim 1, characterized in that, The quantitative return table is obtained through the following method: Extract a specified sample from the plurality of standard samples; With the embodied intelligent model in full-precision state, the output features of the specified sample in each model layer are calculated to obtain the baseline features of each model layer; Each model layer of the embodied intelligent model is labeled as a designated model layer, and the designated model layer is quantized according to the candidate quantization precision to obtain the quantized labeled model of each designated model layer; For each specified model layer, the information retention of each candidate quantization precision is calculated based on the quantization label model, baseline features, and layer type of the specified model layer; The amount of information retained is normalized to obtain a quantitative revenue table.

7. The method according to claim 6, characterized in that, For each specified model layer, based on the quantization label model, baseline features, and layer type of the specified model layer, the information retention of each candidate quantization precision is calculated, including: For each quantization labeling model of a specified model layer, the specified sample is input into the quantization labeling model, and the output features of the specified model layer are obtained to obtain the quantization features; If the specified model layer is an intermediate feature layer, the information retention is calculated based on the cosine similarity between the quantized feature and the baseline feature; If the specified model layer is a probability output layer, the information retention is calculated based on the relative entropy between the quantized feature and the benchmark feature.

8. The method according to claim 6, characterized in that, The step of labeling each model layer of the embodied intelligence model as a designated model layer, and quantizing the designated model layer according to the candidate quantization precision to obtain the quantized labeled model of each designated model layer includes: Each model layer in the embodied intelligent model is labeled as a specified model layer, thus obtaining the labeled model corresponding to each specified model layer; For each of the labeled models, the specified model layer is quantized according to each candidate quantization precision to obtain a quantized labeled model.

9. An electronic device, characterized in that, It includes a processor and a memory, the memory storing multiple instructions; the processor loads instructions from the memory to perform the steps in the compression method of the embodied intelligence model as described in any one of claims 1-8.

Images