Flexible job production line control method and system based on vertical domain large model

Abstract

This invention provides a flexible production line control method and system based on a large vertical domain model. The method includes: constructing a pruning index combining temporal stability and spatial similarity for the original large vertical domain model to evaluate the redundancy of the activation tensors in the intermediate layers, and performing structured pruning accordingly to obtain a sparse structure model; performing mixed-precision quantization on the pruned model: allocating quantization bits according to the sensitivity of each network layer weight to task loss, and calculating scaling factors and zeros based on the range of non-zero weights to represent sparse weights; calculating the output deviation between the quantized pruned model and the original model under the same input, and training a feedforward network with the former's output as input to fit and compensate for the output deviation; combining the quantized pruned model and the feedforward network to form a composite control model; after deployment, adding the output command of the quantized pruned model to the compensation vector during inference to generate control commands.

Description

Technical Field

[0001] This application belongs to the field of control, and in particular relates to a flexible production line control method and system based on a large vertical domain model. Background Technology

[0002] Large-scale deep learning models can process farmland environmental data from various sensors, including cameras, radar, and GPS, and generate control commands.

[0003] However, these large models typically contain hundreds of millions of parameters, exhibiting high computational complexity and large memory footprint. This makes it difficult to deploy the original large models on agricultural machinery terminals, where computing power, storage, and power consumption are limited, hindering the practical application of advanced algorithms in real-world agricultural production environments. Model compression techniques include model pruning and model quantization. Model pruning reduces complexity by removing redundant parameters or structural units, while model quantization reduces model size and accelerates computation by lowering the numerical precision of weights and activation values. However, in agricultural machinery operation scenarios, existing compression methods often fail to fully utilize the unique temporal and spatial characteristics of agricultural machinery operation data, potentially omitting structures important for environmental perception. Conventional quantization strategies can lead to significant accuracy loss, and common quantization range settings may not represent non-zero values ​​in sparse weights, exacerbating performance degradation. Compression processes often lack mechanisms to compensate for the accuracy loss accumulated during pruning and quantization, resulting in significant deviations between the optimized model's output control commands and the original large-scale model, making it difficult to meet the requirements of agricultural machinery operations. Summary of the Invention

[0004] To address the problem that existing technologies fail to fully utilize the unique temporal and spatial characteristics of agricultural machinery operation data, and lack mechanisms to compensate for the accuracy loss accumulated during pruning and quantification processes.

[0005] In the first aspect, the present invention proposes a flexible production line control method based on a large vertical domain model, comprising: Obtain the original vertical domain large model of the agricultural machinery operation task, wherein the agricultural machinery operation task is to generate a set of control command vectors based on the input farmland environment data; construct a pruning measure index to represent the redundancy of the model structural units; perform structured pruning on the original vertical domain large model according to the pruning measure index, remove redundant neurons or convolution kernels, thereby obtaining a sparse structure model. The sparse structure model is subjected to mixed precision quantization to generate a large model after quantization pruning. The quantization includes: calculating the sensitivity of each network layer weight to a preset task loss function, and allocating a higher number of quantization bits to the network layer with higher sensitivity based on the sensitivity; when calculating the scaling factor and zero point of each weight tensor, the quantization range is determined based on the maximum and minimum values ​​of the non-zero weights in the tensor. Collect the control command vector output by the quantized pruned large model on the calibration dataset, and calculate the deviation between the vector and the corresponding output of the original vertical domain large model; use the output of the quantized pruned large model as input, and use the deviation as training label to train a lightweight feedforward network for fitting and compensating the deviation. The large model after quantitative pruning is combined with the lightweight feedforward network to form a composite agricultural machinery operation control model, which is then deployed to the agricultural machinery vehicle terminal. The control command vector output by the large model after quantitative pruning is summed with the compensation vector generated by the lightweight feedforward network based on the command vector to obtain the control command.

[0006] In another aspect, the present invention also proposes a flexible production line control system based on a large vertical domain model, comprising the following modules: The pruning module is used to obtain the original vertical domain large model of the agricultural machinery operation task, which is to generate a set of control command vectors based on the input farmland environment data; construct a pruning metric to represent the redundancy of the model structural units; and perform structured pruning on the original vertical domain large model according to the pruning metric to remove redundant neurons or convolution kernels, thereby obtaining a sparse structure model. A determination module is used to perform mixed-precision quantization on the sparse structure model to generate a large model after quantization pruning. The quantization includes: calculating the sensitivity of each network layer weight to a preset task loss function, and allocating a higher number of quantization bits to the network layer with higher sensitivity based on the sensitivity; when calculating the scaling factor and zero point of each weight tensor, determining the quantization range based on the maximum and minimum values ​​of the non-zero weights in the tensor. The training module is used to collect the control command vector output by the quantized pruned large model on the calibration dataset, and calculate the deviation between the vector and the corresponding output of the original vertical domain large model; using the output of the quantized pruned large model as input and the deviation as training label, a lightweight feedforward network is trained to fit and compensate for the deviation. The summation module is used to combine the quantized pruned large model with the lightweight feedforward network to form a composite agricultural machinery operation control model, and deploy it to the agricultural machinery vehicle terminal. The control command vector output by the quantized pruned large model is summed with the compensation vector generated by the lightweight feedforward network based on the command vector to obtain the control command.

[0007] This invention constructs a pruning index that comprehensively evaluates the temporal stability and spatial similarity of activation tensors. This index can identify and remove redundant structural units in the original large-scale vertical model, reducing the computational complexity and number of parameters. The hybrid precision quantization strategy employs different quantization bits based on the sensitivity of network layers and optimizes the quantization range for sparse weights, compressing the model size while preserving the precision of key weight information. By training a lightweight feedforward network to fit and compensate for performance deviations caused by the compression process, the output precision of the model is restored with minimal additional overhead. The resulting composite model enables the deployment and operation of large models on resource-constrained agricultural machinery terminal devices, while ensuring the reliability of agricultural machinery operation control commands, thus improving the overall quality and efficiency of agricultural machinery operations. Attached Figure Description

[0008] Figure 1 This is a flowchart of the first embodiment. Detailed Implementation

[0009] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples.

[0010] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0011] In the first embodiment, the present invention proposes a flexible production line control method based on a large vertical domain model, such as... Figure 1 As shown, it includes: S1. Obtain the original vertical domain large model of the agricultural machinery operation task, wherein the agricultural machinery operation task is to generate a set of control command vectors based on the input farmland environment data; construct a pruning measure index to represent the redundancy of the model structural units; perform structured pruning on the original vertical domain large model according to the pruning measure index, remove redundant neurons or convolution kernels, thereby obtaining a sparse structure model. Using the PyTorch or TensorFlow deep learning framework, a large neural network model file pre-trained on a cloud server is loaded. The original vertical domain control model is a visual neural network structure containing deep convolutional layers, such as ResNet or a convolution-based multimodal fusion network. The function for loading the model is torch.load or tf.keras.models.load_model. The farmland environment data is a multimodal data stream that integrates point cloud data acquired by vehicle-mounted LiDAR, RGB-D images captured by a depth camera, and real-time coordinate information provided by the Global Positioning System. The control command vector is a one-dimensional floating-point tensor containing specific values ​​for the agricultural machinery's steering angle, throttle opening, and hydraulic suspension lifting height.

[0012] A representative calibration dataset is input into the original vertical domain large model, and the activation tensors of the fully connected or convolutional layers to be pruned are detected. Temporal stability is measured by calculating the temporal variance of the activation value output sequence of each neuron or convolutional kernel when processing consecutive data frames. This can be calculated using the `torch.var` function; a smaller variance indicates higher stability. Spatial similarity is measured by calculating the cosine similarity between the activation map vectors of any two neurons or convolutional kernels within the same layer. This can be calculated using the `torch.nn.functional.cosine_similarity` function; a higher similarity indicates closer functional similarity. Pruning metrics... The combination is performed using a weighted summation method. Considering that smaller variance indicates higher redundancy, and higher similarity also indicates higher redundancy, to ensure consistency between the two, the time series stability index needs to be reversed. The formula is as follows: = × (1 - normalized time series stability index) + × Spatial similarity index, weight and For preset hyperparameters; according to Sort all neurons or convolutional kernels in descending order by value and remove... For structural units whose values ​​rank in the preset pruning ratio, such as the top 30%, the corresponding rows or columns of the weight matrix are set to zero to complete the structured pruning.

[0013] The calibration dataset is constructed by collecting multimodal data streams in a real farmland environment using the same sensor suite as during deployment. Samples covering different terrains, crop types, lighting conditions, and operational stages are selected to fully reflect the data distribution of real-world operational scenarios. Data acquisition must ensure temporal continuity; for example, each scenario should contain at least 30 consecutive data frames. Finally, N samples are selected from the original collected data using a distance- or variance-based sample selection algorithm, such as the Kennard-Stone algorithm. N is preferably between 500 and 2000, for example, 1024 samples, to form the calibration dataset.

[0014] In an optional embodiment, the construction of a pruning metric representing the redundancy of model structural units includes: For each activation channel in the intermediate layer of the model, with farmland environmental data input at continuous time steps, the variance of the output activation value sequence is calculated; For each convolutional kernel in the convolutional layer, calculate the average cosine similarity between the convolutional kernel and all other convolutional kernels in the layer; For non-convolutional layer structural units, the pruning metric is based solely on temporal redundancy; for convolutional layer structural units, the pruning metric is obtained by weighted summation of the normalized temporal redundancy metric and spatial redundancy metric.

[0015] Select a farmland environment data sequence containing T consecutive time steps from the calibration dataset. For example, T is preferably 10 to 50, such as 30 consecutive video frames. Input this sequence into the original vertical domain large model. For the i-th activation channel or neuron in any intermediate layer of the model, record the activation value output at T time steps, forming an activation value sequence { , ,..., }. Calculate the variance of the sequence. The variance value represents the timing stability of the channel; the smaller the variance, the higher the redundancy.

[0016] The temporal redundancy and spatial redundancy indices calculated for all units to be pruned in the model are subjected to min-max normalization, mapping their numerical ranges to the [0,1] interval. For neurons in fully connected layers, the pruning metric is the inverse value of the normalized temporal redundancy index, i.e. ,in This is a normalized temporal stability metric; a higher metric indicates higher redundancy. For convolutional kernels in convolutional layers, the pruning metric is... From the normalized temporal redundancy index and spatial redundancy index The weighted summation is obtained, and the calculation formula is: ,in The reverse processing is performed because a smaller variance indicates higher redundancy, while higher similarity also indicates higher redundancy. To unify the direction, [the process is reversed]. Convert to Weight and To balance the importance of the two redundancies, the sum is 1, preferably... , The obtained indicators The higher the value of a cell, the higher its priority for pruning.

[0017] S2, perform mixed-precision quantization on the sparse structure model to generate a large model after quantization pruning. The quantization includes: calculating the sensitivity of each network layer weight to a preset task loss function, and allocating a higher number of quantization bits to the network layer with higher sensitivity based on the sensitivity; when calculating the scaling factor and zero point of each weight tensor, determining the quantization range based on the maximum and minimum values ​​of the non-zero weights in the tensor. Sensitivity is calculated using an empirical perturbation method. This involves temporarily quantizing the weights of each network layer in the model to a low bit depth, such as 4 bits. The mean squared error (MSE) loss between the model's output control command vector and the original model output is evaluated on a calibration dataset. Layers with higher MSE losses have higher sensitivity. Based on the sensitivity ranking, different quantization precisions are assigned to the model. For example, the top 20% of layers with the highest sensitivity are assigned FP16 half-precision floating-point numbers, the middle 50% are assigned INT8 eight-bit integers, and the bottom 30% are assigned INT4 four-bit integers. For weight tensors requiring integer fixed-point quantization, the `torch.where` function is used to filter out all non-zero elements. The `torch.max` and `torch.min` functions are then called to calculate the maximum and minimum values ​​of these non-zero elements, and the results are determined based on this range and the target bit count. And the scaling factor S and zero Z are calculated using the asymmetric quantization formula, which is as follows: , The S and Z values ​​are used to perform quantization operations on the weight tensor.

[0018] In an optional embodiment, calculating the sensitivity of each network layer weight to a preset task loss function includes: The preset task loss function is set as the mean square error loss between the control command vector output by the model and the baseline control command vector. Input the calibration dataset into the model and use the backpropagation algorithm to calculate the partial derivative of the mean squared error loss with respect to the weights of each network layer; For a specific network layer, the sensitivity of the network layer is obtained by averaging the absolute values ​​of all weight gradients of that network layer.

[0019] A calibration dataset containing N samples, where N is preferably between 500 and 2000, for example, 1024 samples. Each sample contains farmland environmental data. The entire calibration dataset is input into the original vertical domain large model, the output is recorded, and a corresponding set of baseline control command vectors is generated. Assuming the control command vectors are three-dimensional, containing steering angle, throttle opening, and hydraulic suspension lift height, i.e. For any sample i in the calibration dataset, the control command vector output by the model is: The corresponding reference control command vector is Then the mean squared error loss function The calculation formula is: .

[0020] The calibration dataset is input into the sparse structure model in batches, and forward propagation is performed to obtain the predicted values. Then calculate the loss according to the above formula. The loss is calculated using the backpropagation algorithm within the differential framework. For the weight tensor of each network layer l in the model The partial derivative, i.e., the gradient. The process is repeated across the entire calibration dataset, accumulating gradients. After traversing all data, for the l-th layer, the sensitivity... By calculating the gradient tensor The weight tensor is determined by the average of the absolute values ​​of all elements in the tensor. For example, if the size of the weight tensor of the l-th layer is ( , Given a variable K, which contains M weight parameters, the sensitivity is... .

[0021] In an optional embodiment, allocating a higher number of quantization bits to network layers with higher sensitivity based on the sensitivity includes: The sensitivity values ​​of all network layers are linearly mapped to the [0,1] interval to obtain the normalized sensitivity scores; Each network layer is assigned a quantization bit number B, calculated using the following formula: ,in is the sensitivity score for layer normalization, and round is the rounding function.

[0022] Calculate the sensitivity of all L unquantized network layers in the model { , ,..., After that, find the maximum value. and minimum value Sensitivity to each layer l Perform min-max normalization and calculate the normalized score. Through this step, the sensitivity of all layers is mapped to a unified range of [0,1], and the higher the score, the greater the impact of that layer on the model performance.

[0023] Based on normalized sensitivity score Allocate an integer number of bits to each layer. Set the minimum number of bits. Maximum number of bits For example, the normalization sensitivity score of a layer that is important for model performance is... Then the number of bits allocated is Bits. Conversely, a less sensitive layer has a normalized score of Then the number of bits allocated is Bits. For the layer with the lowest sensitivity, Approaching 0, the number of bits will be 2 bits. This method achieves the allocation of computing resources, maximizing the compression rate of non-critical layers while ensuring the accuracy of critical layers.

[0024] In an optional embodiment, determining the quantization range based on the maximum and minimum values ​​of the non-zero weights within the tensor when calculating the scaling factor and zeros of each weight tensor includes: For a weight tensor to be quantized, extract all non-zero weight values ​​and determine the maximum value among them. and minimum value ; Set the lower bound of the actual quantization range of the tensor to min(0, The upper bound is set to max(0, ); Based on the lower and upper bounds of the actual quantization range, and the number of bits B allocated to the layer, the scaling factor S and the zero point Z of the tensor are calculated.

[0025] Given a weight tensor, iterate through all elements and collect all non-zero weight values ​​into a list. For example, if the range of non-zero elements in a weight tensor is [-0.85, 1.23], then the maximum value in the list is... The minimum value is 1.23. The value is -0.85. This eliminates the interference of zero values ​​on the quantization range and avoids the problem of excessively large quantization range and reduced precision caused by the presence of zero values.

[0026] After determining the maximum value of the non-zero weights and minimum value Then, the range of asymmetric quantization is determined. The Lower Realm Upper Realm As in the example above. , If all non-zero weights of a tensor are positive, for example, the range is [-0.85, 1.23]. If all non-zero weights of a tensor are positive, for example, the range is [0.1, 0.9], then the quantization range is [0, 0.9]. This depends on the number of bits in the layer. ,For example Calculate the scaling factor and zero point ,in This is the minimum value of the quantized value, usually 0. Therefore, For a range of [-0.85, 1.23] and 4-bit quantization, S ≈ 0.1387, Z = 6. The parameters S and Z will be used to map the floating-point weights of this tensor to 4-bit integer quantized values.

[0027] S3, Collect the control command vector output by the quantized pruned large model on the calibration dataset, and calculate the deviation between the vector and the corresponding output of the original vertical domain large model; use the output of the quantized pruned large model as input, and use the deviation as training label to train a lightweight feedforward network for fitting and compensating the deviation. Iterate through each farmland environment data point in the calibration dataset, and obtain the output control command vectors of the original vertical domain large model and the quantized pruned large model, denoted as... and ; Calculate the element-wise difference vector d between the two, i.e. Construct a new training dataset, where the data samples are... The data is labeled d; a lightweight feedforward network is planned, consisting of one input layer, two hidden layers, and one output layer. The number of neurons in the hidden layers can be set to 64 or 32, the activation function is ReLU, and the output layer has linear activation; this network is trained using this new dataset, the loss function is mean squared error loss, and the optimizer is Adam; iterative training is performed using the standard backpropagation algorithm until the network can respond to the input... The compensation vector d is predicted.

[0028] In an optional embodiment, the step of using the output of the quantized pruned large model as input and the bias as training labels to train a lightweight feedforward network for fitting and compensating for the bias includes: The calibration dataset is input into the original vertical domain large model and the large model after quantization pruning. The output vector of the latter is used as the training input data, and the element-wise difference between the output vectors of the two is used as the training label data to form the bias fitting training set. Construct a fully connected feedforward network, wherein the input layer dimension is the same as the control command vector dimension, the network contains at least one hidden layer, and the output layer dimension is the same as the control command vector dimension; An optimizer is trained on the bias fitting training set, with the training objective being to minimize the mean square error between the network output compensation vector and the training label data.

[0029] The lightweight feedforward network is a fully connected neural network consisting of an input layer, two hidden layers, and an output layer connected in series. If the control instruction vector has dimension D, both the input and output layers contain D neurons. The first hidden layer contains 8D neurons, and the second hidden layer contains 4D neurons. All hidden layers use the Modified Linear Unit (ReLU) as the activation function, and the output layer uses a linear activation function. The network input is a vector of dimension D. The vector represents the initial control instructions from the quantized, pruned large model for the current environmental data output. The network output is a compensation vector of dimension D. It is used to correct the initial control commands.

[0030] Training data is generated using a calibration dataset of size N, for example, N=1024. For each farmland environment data point in the dataset... The data are then input into the original vertical domain large model. Large model after quantization pruning In this process, two control command vectors are obtained: and At this point, the training input data for the lightweight network is... The corresponding training labels are Repeat this process to generate a dataset containing N sample pairs. , ,..., The biased training set is fitted. During training, the Adam optimizer is used with an initial learning rate of 1e-4 and a learning rate decay strategy. The batch size can be set to 64. The training objective is to minimize the compensation vector predicted by the network. Deviation from reality The mean squared error (MSE) between the training and validation sets. The training process lasts for 50 epochs, or is stopped early when the loss on the validation set no longer decreases for 5 consecutive epochs.

[0031] To enable the lightweight feedforward network to perceive the current agricultural machinery operation environment, optionally, the intermediate layer feature maps of the quantized pruning model are extracted, pooled for dimensionality reduction, and then concatenated with the control command vector before being input into the compensation network. The training samples of the lightweight feedforward network are also synchronized.

[0032] S4. The large model after quantitative pruning is combined with the lightweight feedforward network to form a composite agricultural machinery operation control model, which is then deployed to the agricultural machinery vehicle terminal. The control command vector output by the large model after quantitative pruning is summed with the compensation vector generated by the lightweight feedforward network based on the command vector to obtain the control command.

[0033] A new model encapsulation class is created, which instantiates a quantized pruned large model and a trained lightweight feedforward network. In the forward propagation function of this encapsulation class, the data flow path is defined as follows: the quantized pruned large model processes the input environmental data to obtain preliminary control commands, which are then passed as input to the lightweight feedforward network to generate compensation vectors. The `torch.add` function is used to add the preliminary control commands and compensation vectors element-wise to obtain the output. This encapsulated composite model is exported as an ONNX open neural network exchange format file using the `torch.onnx.export` function. On an agricultural machinery vehicle terminal such as the NVIDIA Jetson AGX Orin, the ONNX model is parsed and optimized using NVIDIA TensorRT tools to generate a locally hardware-accelerated inference engine file. The vehicle's main control program calls the TensorRT runtime API to load this engine, performs real-time inference, and sends the inferred control command vectors to the vehicle's CAN bus to drive the actuators to complete the operation.

[0034] In an optional embodiment, the step of summing the control command vector output by the quantized pruned large model with the compensation vector generated by the lightweight feedforward network based on the command vector to obtain the control command includes: When performing inference at the vehicle terminal, the acquired farmland environment data is input into the quantized and pruned large model to obtain a preliminary control command vector. The initial control command vector is used as input and passed to the lightweight feedforward network to generate a compensation vector. The initial control command vector is added element by element to the compensation vector to obtain the output control command.

[0035] The model deployed on the agricultural machinery vehicle terminal consists of two modules that work in series, quantizing the large model after pruning. and lightweight feedforward network In actual operation, the vehicle-mounted sensor system collects real-time farmland environmental data at a specific frequency. In each inference cycle, Input to In the process, through reasoning and calculation, a preliminary control command vector is obtained. For example, at a certain moment, The output vector is , respectively represent a steering angle of 0.12 radians, a throttle opening of 55%, and a hydraulic suspension lift height of 0.

[0036] The preliminary control command vector obtained above As a lightweight feedforward network Input. Perform fast forward propagation on the vector to output a compensation vector. As a response to Prediction and compensation for systematic errors caused by compression. For example, Possible output =[-0.015,0.03,0.01]. The initial control command vector and the compensation vector are added element-wise to obtain the control command vector. Taking the above values ​​as an example, the instruction is: =[0.105,0.58,0.01]. The command vector is sent to the underlying actuator of the agricultural machinery to control it. The entire inference process is short and meets the real-time requirements of agricultural machinery operations.

[0037] In the second embodiment, the present invention also proposes a flexible production line control system based on a large vertical domain model, comprising the following modules: The pruning module is used to obtain the original vertical domain large model of the agricultural machinery operation task, which is to generate a set of control command vectors based on the input farmland environment data; construct a pruning metric to represent the redundancy of the model structural units; and perform structured pruning on the original vertical domain large model according to the pruning metric to remove redundant neurons or convolution kernels, thereby obtaining a sparse structure model. A determination module is used to perform mixed-precision quantization on the sparse structure model to generate a large model after quantization pruning. The quantization includes: calculating the sensitivity of each network layer weight to a preset task loss function, and allocating a higher number of quantization bits to the network layer with higher sensitivity based on the sensitivity; when calculating the scaling factor and zero point of each weight tensor, determining the quantization range based on the maximum and minimum values ​​of the non-zero weights in the tensor. The training module is used to collect the control command vector output by the quantized pruned large model on the calibration dataset, and calculate the deviation between the vector and the corresponding output of the original vertical domain large model; using the output of the quantized pruned large model as input and the deviation as training label, a lightweight feedforward network is trained to fit and compensate for the deviation. The summation module is used to combine the quantized pruned large model with the lightweight feedforward network to form a composite agricultural machinery operation control model, and deploy it to the agricultural machinery vehicle terminal. The control command vector output by the quantized pruned large model is summed with the compensation vector generated by the lightweight feedforward network based on the command vector to obtain the control command.

[0038] In an optional embodiment, the construction of a pruning metric representing the redundancy of model structural units includes: For each activation channel in the intermediate layer of the model, with farmland environmental data input at continuous time steps, the variance of the output activation value sequence is calculated; For each convolutional kernel in the convolutional layer, calculate the average cosine similarity between the convolutional kernel and all other convolutional kernels in the layer; For non-convolutional layer structural units, the pruning metric is based solely on temporal redundancy; for convolutional layer structural units, the pruning metric is obtained by weighted summation of the normalized temporal redundancy metric and spatial redundancy metric.

[0039] In an optional embodiment, calculating the sensitivity of each network layer weight to a preset task loss function includes: The preset task loss function is set as the mean square error loss between the control command vector output by the model and the baseline control command vector. Input the calibration dataset into the model and use the backpropagation algorithm to calculate the partial derivative of the mean squared error loss with respect to the weights of each network layer; For a specific network layer, the sensitivity of the network layer is obtained by averaging the absolute values ​​of all weight gradients of that network layer.

[0040] In an optional embodiment, allocating a higher number of quantization bits to network layers with higher sensitivity based on the sensitivity includes: The sensitivity values ​​of all network layers are linearly mapped to the [0,1] interval to obtain the normalized sensitivity scores; Each network layer is assigned a quantization bit number B, calculated using the following formula: ,in is the sensitivity score for layer normalization, and round is the rounding function.

[0041] In an optional embodiment, determining the quantization range based on the maximum and minimum values ​​of the non-zero weights within the tensor when calculating the scaling factor and zeros of each weight tensor includes: For a weight tensor to be quantized, extract all non-zero weight values ​​and determine the maximum value among them. and minimum value ; Set the lower bound of the actual quantization range of the tensor to min(0, The upper bound is set to max(0, ); Based on the lower and upper bounds of the actual quantization range, and the number of bits B allocated to the layer, the scaling factor S and the zero point Z of the tensor are calculated.

[0042] In an optional embodiment, the step of using the output of the quantized pruned large model as input and the bias as training labels to train a lightweight feedforward network for fitting and compensating for the bias includes: The calibration dataset is input into the original vertical domain large model and the large model after quantization pruning. The output vector of the latter is used as the training input data, and the element-wise difference between the output vectors of the two is used as the training label data to form the bias fitting training set. Construct a fully connected feedforward network, wherein the input layer dimension is the same as the control command vector dimension, the network contains at least one hidden layer, and the output layer dimension is the same as the control command vector dimension; An optimizer is trained on the bias fitting training set, with the training objective being to minimize the mean square error between the network output compensation vector and the training label data.

[0043] In an optional embodiment, the step of summing the control command vector output by the quantized pruned large model with the compensation vector generated by the lightweight feedforward network based on the command vector to obtain the control command includes: When performing inference at the vehicle terminal, the acquired farmland environment data is input into the quantized and pruned large model to obtain a preliminary control command vector. The initial control command vector is used as input and passed to the lightweight feedforward network to generate a compensation vector. The initial control command vector is added element by element to the compensation vector to obtain the output control command.

[0044] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0045] The functional modules shown in the above-described block diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0046] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0047] The aspects of this application have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by dedicated hardware performing the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0048] The above description is merely a specific implementation of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.

Claims

1. A flexible production line control method based on a vertical domain large model, characterized in that, Includes the following steps: Obtain the original vertical domain large model of the agricultural machinery operation task, wherein the agricultural machinery operation task is to generate a set of control command vectors based on the input farmland environment data; construct a pruning measure index to represent the redundancy of the model structural units; perform structured pruning on the original vertical domain large model according to the pruning measure index, remove redundant neurons or convolution kernels, thereby obtaining a sparse structure model. The sparse structure model is subjected to mixed precision quantization to generate a large model after quantization pruning. The quantization includes: calculating the sensitivity of each network layer weight to a preset task loss function, and allocating a higher number of quantization bits to the network layer with higher sensitivity based on the sensitivity; when calculating the scaling factor and zero point of each weight tensor, the quantization range is determined based on the maximum and minimum values ​​of the non-zero weights in the tensor. Collect the control command vector output by the quantized pruned large model on the calibration dataset, and calculate the deviation between the vector and the corresponding output of the original vertical domain large model; use the output of the quantized pruned large model as input, and use the deviation as training label to train a lightweight feedforward network for fitting and compensating the deviation. The large model after quantitative pruning is combined with the lightweight feedforward network to form a composite agricultural machinery operation control model, which is then deployed to the agricultural machinery vehicle terminal. The control command vector output by the large model after quantitative pruning is summed with the compensation vector generated by the lightweight feedforward network based on the command vector to obtain the control command.

2. The method according to claim 1, characterized in that, The pruning metrics for constructing the redundancy of model structural units include: For each activation channel in the intermediate layer of the model, with farmland environmental data input at continuous time steps, the variance of the output activation value sequence is calculated; For each convolutional kernel in the convolutional layer, calculate the average cosine similarity between the convolutional kernel and all other convolutional kernels in the layer; For non-convolutional layer structural units, the pruning metric is based solely on temporal redundancy; for convolutional layer structural units, the pruning metric is obtained by weighted summation of the normalized temporal redundancy metric and spatial redundancy metric.

3. The method according to claim 1, characterized in that, The calculation of the sensitivity of each network layer weight to the preset task loss function includes: The preset task loss function is set as the mean square error loss between the control command vector output by the model and the baseline control command vector. Input the calibration dataset into the model and use the backpropagation algorithm to calculate the partial derivative of the mean squared error loss with respect to the weights of each network layer; For a specific network layer, the sensitivity of the network layer is obtained by averaging the absolute values ​​of all weight gradients of that network layer.

4. The method according to claim 2, characterized in that, The method of allocating a higher number of quantization bits to network layers with higher sensitivity based on the sensitivity includes: The sensitivity values ​​of all network layers are linearly mapped to the [0,1] interval to obtain the normalized sensitivity scores; Each network layer is assigned a quantization bit number B, calculated using the following formula: ,in is the sensitivity score for layer normalization, and round is the rounding function.

5. The method according to claim 1, characterized in that, The step of determining the quantization range based on the maximum and minimum values ​​of the non-zero weights within the tensor when calculating the scaling factor and zero point of each weight tensor includes: For a weight tensor to be quantized, extract all non-zero weight values ​​and determine the maximum value among them. and minimum value ; Set the lower bound of the actual quantization range of the tensor to min(0, The upper bound is set to max(0, ); Based on the lower and upper bounds of the actual quantization range, and the number of bits B allocated to the layer, the scaling factor S and the zero point Z of the tensor are calculated.

6. The method according to claim 1, characterized in that, The process of using the output of the large model after quantization pruning as input and the bias as training labels to train a lightweight feedforward network for fitting and compensating for the bias includes: The calibration dataset is input into the original vertical domain large model and the large model after quantization pruning. The output vector of the latter is used as the training input data, and the element-wise difference between the output vectors of the two is used as the training label data to form the bias fitting training set. Construct a fully connected feedforward network, wherein the input layer dimension is the same as the control command vector dimension, the network contains at least one hidden layer, and the output layer dimension is the same as the control command vector dimension; An optimizer is trained on the bias fitting training set, with the training objective being to minimize the mean square error between the network output compensation vector and the training label data.

7. The method according to claim 5, characterized in that, The control command vector output by the quantized pruned large model is summed with the compensation vector generated by the lightweight feedforward network based on the command vector to obtain the control command, including: When performing inference at the vehicle terminal, the acquired farmland environment data is input into the quantized and pruned large model to obtain a preliminary control command vector. The initial control command vector is used as input and passed to the lightweight feedforward network to generate a compensation vector. The initial control command vector is added element by element to the compensation vector to obtain the output control command.

8. A flexible production line control system based on a vertical domain large model, characterized in that, Includes the following modules: The pruning module is used to obtain the original vertical domain large model of the agricultural machinery operation task, which is to generate a set of control command vectors based on the input farmland environment data; construct a pruning metric to represent the redundancy of the model structural units; and perform structured pruning on the original vertical domain large model according to the pruning metric to remove redundant neurons or convolution kernels, thereby obtaining a sparse structure model. A determination module is used to perform mixed-precision quantization on the sparse structure model to generate a large model after quantization pruning. The quantization includes: calculating the sensitivity of each network layer weight to a preset task loss function, and allocating a higher number of quantization bits to the network layer with higher sensitivity based on the sensitivity; when calculating the scaling factor and zero point of each weight tensor, determining the quantization range based on the maximum and minimum values ​​of the non-zero weights in the tensor. The training module is used to collect the control command vector output by the quantized pruned large model on the calibration dataset, and calculate the deviation between the vector and the corresponding output of the original vertical domain large model; using the output of the quantized pruned large model as input and the deviation as training label, a lightweight feedforward network is trained to fit and compensate for the deviation. The summation module is used to combine the quantized pruned large model with the lightweight feedforward network to form a composite agricultural machinery operation control model, and deploy it to the agricultural machinery vehicle terminal. The control command vector output by the quantized pruned large model is summed with the compensation vector generated by the lightweight feedforward network based on the command vector to obtain the control command.

9. The system according to claim 8, characterized in that, The pruning metrics for constructing the redundancy of model structural units include: For each activation channel in the intermediate layer of the model, with farmland environmental data input at continuous time steps, the variance of the output activation value sequence is calculated; For each convolutional kernel in the convolutional layer, calculate the average cosine similarity between the convolutional kernel and all other convolutional kernels in the layer; For non-convolutional layer structural units, the pruning metric is based solely on temporal redundancy; for convolutional layer structural units, the pruning metric is obtained by weighted summation of the normalized temporal redundancy metric and spatial redundancy metric.

10. The system according to claim 8, characterized in that, The calculation of the sensitivity of each network layer weight to the preset task loss function includes: The preset task loss function is set as the mean square error loss between the control command vector output by the model and the baseline control command vector. Input the calibration dataset into the model and use the backpropagation algorithm to calculate the partial derivative of the mean squared error loss with respect to the weights of each network layer; For a specific network layer, the sensitivity of the network layer is obtained by averaging the absolute values ​​of all weight gradients of that network layer.

Images