Robot decision control method based on gradient sparsification and robot

Abstract

The application relates to a robot decision control method and robot based on gradient sparsification. The method comprises the following steps: acquiring multi-modal sensor data of the robot; generating a dynamic mask corresponding to a gradient matrix of a multi-modal large model based on a preset sparsification strategy; screening effective gradients in a back propagation process of the dynamic mask based on the generated dynamic mask; updating parameters corresponding to the effective gradients in real time, and acquiring an output of the multi-modal large model based on the updated parameters; performing feature fusion according to the obtained multi-modal sensor data and the output of the multi-modal large model based on the updated parameters, and generating joint state encoding including environment state, robot body state and historical decision information; and generating dynamic decision instructions including action sequence, force control parameters and path planning of the robot based on the determined joint state encoding and a time sequence model, so that the robot can act based on the generated dynamic decision instructions, thereby realizing decision control of the robot.

Description

Technical Field

[0001] This application relates to the field of robot decision control technology, and in particular to a robot decision control method and robot based on gradient sparsity. Background Technology

[0002] In the field of intelligent robotics, utilizing large models to process multimodal data for real-time decision-making is a current research hotspot. However, existing sparsification methods in large model processing are mostly based on fixed threshold gradient selection, which can easily lead to problems such as missed updates of key parameters or redundant calculations of non-key parameters, affecting the balance between model accuracy and efficiency. Summary of the Invention

[0003] Therefore, it is necessary to provide a gradient sparsity-based robot decision control method and robot that can balance the accuracy and efficiency of robot decision-making, addressing the aforementioned technical problems.

[0004] Firstly, this application provides a robot decision control method based on gradient sparsity, including:

[0005] Acquire multimodal sensor data from the robot; this multimodal sensor data includes visual images, force and tactile signals, inertial navigation data, and environmental semantic information.

[0006] Based on a preset sparsification strategy, a dynamic mask corresponding to the gradient matrix of the multimodal large model is generated. The multimodal large model is a decision model that reflects the robot's decision-making.

[0007] Based on dynamic masks, effective gradients are selected during the backpropagation process of dynamic masks;

[0008] Update the parameters corresponding to the effective gradient in real time, and obtain the output of the multimodal large model based on the updated parameters;

[0009] Based on multimodal sensor data and the output of the multimodal large model based on updated parameters, feature fusion is performed to generate a joint state code; the joint state code includes environmental state, robot body state, and historical decision information;

[0010] Based on the joint state coding and the timing model, dynamic decision instructions are generated; the dynamic decision instructions include the robot's action sequence, force control parameters and path planning.

[0011] Secondly, this application also provides robots, including:

[0012] A sensor data acquisition module, which is used to acquire multimodal sensor data of the robot;

[0013] A control module, which is connected to the sensor data acquisition module, is used to execute any step in the above-described gradient sparsity-based robot decision control method.

[0014] The aforementioned gradient sparsity-based robot decision-making and control method and robot acquire multimodal sensor data from the robot; then, based on a preset sparsity strategy, generate a dynamic mask corresponding to the gradient matrix of the multimodal large model; based on the generated dynamic mask, filter the effective gradients during the backpropagation process of the dynamic mask; update the parameters corresponding to the effective gradients in real time, and obtain the output of the multimodal large model based on the updated parameters; perform feature fusion based on the acquired multimodal sensor data and the output of the multimodal large model based on the updated parameters to generate a joint state code including environmental state, robot body state, and historical decision information; based on the determined joint state code and a time-series model, generate dynamic decision instructions including the robot's action sequence, force control parameters, and path planning, so that the robot can act based on the generated dynamic decision instructions, thereby realizing the robot's decision-making and control. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is an application environment diagram of a robot decision-making and control method based on gradient sparsity in one embodiment.

[0017] Figure 2 This is a flowchart illustrating a robot decision control method based on gradient sparsity in one embodiment;

[0018] Figure 3 This is a structural block diagram of the robot in one embodiment;

[0019] Figure 4 This is a structural block diagram of the control module in a robot in one embodiment;

[0020] Figure 5 This is a structural block diagram of a robot decision control device based on gradient sparsity in one embodiment;

[0021] Figure 6 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0023] It should be noted that the terms "comprising" and "having," and any variations thereof, as used in this application, are intended to cover non-exclusive inclusion. The term "multiple" as used in this application refers to two or more.

[0024] The robot decision control method based on gradient sparsity provided in this application can be applied to, for example... Figure 1 In the application environment shown, the data storage system stores the data that server 102 needs to process. The data storage system can be integrated onto server 102 or placed on the cloud or other network servers. Server 102 acquires multimodal sensor data from the robot; generates a dynamic mask corresponding to the gradient matrix of the multimodal large model based on a preset sparsity strategy; filters effective gradients during the backpropagation process of the generated dynamic mask based on the generated dynamic mask; updates the parameters corresponding to the effective gradients in real time and acquires the output of the multimodal large model based on the updated parameters; performs feature fusion based on the acquired multimodal sensor data and the output of the multimodal large model based on the updated parameters to generate a joint state code including environmental state, robot body state, and historical decision information; based on the determined joint state code and a time-series model, generates a dynamic decision command including the robot's action sequence, force control parameters, and path planning, so that the robot can act based on the generated dynamic decision command, thereby achieving robot decision control. Server 102 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0025] Traditional robot decision-making and control methods involve large-scale models with enormous parameter sizes, and require updating the entire gradient during backpropagation. Therefore, traditional robot decision-making and control methods have high computational complexity, and the fine-tuning delay during execution can exceed the robot's control cycle, failing to meet the real-time requirements of dynamic environments. Specifically, the fine-tuning delay during execution typically needs to be less than or equal to 50ms.

[0026] Traditional robot decision control methods often use gradient sparsification based on fixed thresholds. This gradient sparsification strategy is simplistic and does not take into account task relevance and historical parameter update frequency. This can easily lead to some key parameters in the gradient being missed or non-key parameters being redundantly calculated, which seriously affects the balance between model accuracy and efficiency.

[0027] Traditional robot decision control methods simply splice sensor data and the output of large models, lacking deep semantic fusion of environmental state, ontological state, and historical decisions. This results in insufficient fusion of decision information, which leads to the neglect of dynamic contextual information during temporal decision reasoning, resulting in insufficient decision robustness.

[0028] It is evident that traditional robot decision control methods, by using fixed amplitude thresholds to filter gradients, fail to consider dynamic evaluation of task differences and parameter importance. Traditional robot decision control methods are prone to parameter update imbalances in complex tasks. Although traditional robot decision control methods propose simple fusion decision models based on multimodal features, they fail to address the delay conflict between large model fine-tuning and real-time control, resulting in decision lag risks in practical applications.

[0029] In one exemplary embodiment, such as Figure 2 As shown, a robot decision control method based on gradient sparsity is provided, which is applied to... Figure 1 Taking server 102 as an example, the explanation includes:

[0030] S202, acquire multimodal sensor data of the robot; the multimodal sensor data includes visual images, force and tactile signals, inertial navigation data and environmental semantic information.

[0031] Visual images can be acquired by deploying visual cameras. For example, a visual camera could be an RGB (Red, Green, Blue) camera capable of capturing and displaying image colors, or a depth camera capable of capturing distance information of objects in a scene. Force and tactile signals can be acquired by deploying a six-axis force sensor. Inertial navigation data can be acquired by deploying an IMU (Inertial Measurement Unit). Inertial environmental semantic information can be acquired by deploying environmental semantic sensors. For example, environmental semantic sensors could be LiDAR or microphone arrays.

[0032] Furthermore, synchronous sampling of multimodal sensor data can be achieved through a real-time operating system (ROS) or a dedicated hardware interface. The sampling frequency can be set to match the robot's control cycle. For example, with a sampling frequency ≥ 200Hz and a control cycle ≤ 5ms, data loss is significantly reduced, and may even be eliminated entirely.

[0033] After acquiring multimodal sensor data, the raw multimodal sensor data can be preprocessed. This includes extracting edge features from visual images and compressing the resolution using a lightweight convolutional network to reduce computational load. For example, the lightweight convolutional network could be MobileNetV3 (MobileNet Version 3); the compressed resolution could be 320×240. A moving average filter is used to remove high-frequency noise from the force tactile signal while preserving the contact force trend characteristics. A Kalman filter algorithm is fused to estimate the robot pose, and the global coordinates after coordinate system transformation are output based on the inertial navigation data. A pre-trained language model is used to parse voice commands or text maps from the environmental semantic information to generate structured semantic vectors. For example, the pre-trained language model could be BERT (Bidirectional Encoder Representations from Transformers); the structured semantic vectors could be one-hot encoded task labels. Furthermore, nanosecond-level timestamps are added to the multimodal sensor data to align the data in chronological order via a synchronization buffer. This avoids temporal misalignment during subsequent data fusion, thereby improving the reliability of the gradient sparsity-based robot decision-making and control method. The synchronization buffer can be a FIFO (First In First Out) queue.

[0034] S204 generates a dynamic mask corresponding to the gradient matrix of the multimodal large model based on a preset sparsity strategy. The multimodal large model is a decision model that reflects the robot's decision-making.

[0035] S206, based on dynamic masking, filters the effective gradients during the backpropagation process of dynamic masking.

[0036] S208 updates the parameters corresponding to the effective gradients in real time and obtains the output of the multimodal large model based on the updated parameters.

[0037] During backpropagation, gradients G, including zero and non-zero gradients, can be obtained. A dynamic mask M is applied to gradient G during backpropagation, and the gradients at positions where the dynamic mask is 1 (M=1) are calculated, thus reducing the computational complexity from O(N) to O(N). 2)It is reduced to O(K), where K is the number of non - zero gradients and K << N. N is the number of parameters in the full - gradient matrix. Further, a mixed - precision training (FP16 (Half - Precision Floating - Point) / FP32 (Full - Precision Floating - Point)) method can be adopted to further reduce the computation time consumption, so as to ensure that the gradient calculation is completed within the control period (≤50ms).

[0038] An improved Adam (Adaptive Moment Estimation) optimizer can be used, and the learning rate is dynamically adjusted according to the sparsity of the current dynamic mask.

[0039] The sparsity s can be determined according to s = K / N; the learning rate η can be determined according to ηt = η0*exp(-λ*(1 - st)). Here, η0 is the initial learning rate, λ is the sparsity - sensitive coefficient, and t is the time. When the sparsity s decreases, that is, when fewer gradients are retained, the learning rate η can be automatically increased to compensate for the parameter update amplitude, thus balancing the problem of insufficient updates caused by sparsification.

[0040] Compared with the update of traditional full - gradients, after sparsification, the computation time consumption of this robot decision - making control method based on gradient sparsification is reduced by 70% - 90%. And the adjustment of the adaptive learning rate can avoid low precision under sparse gradients, and at the same time ensure that the fine - tuning delay of the robot is ≤20ms, thus meeting the real - time control period requirements of the robot.

[0041] S210, perform feature fusion based on the multi - modal sensor data and the output of the multi - modal large model based on the updated parameters to generate a joint state encoding; the joint state encoding includes the environmental state, the robot's own state, and historical decision information.

[0042] S212, generate a dynamic decision instruction based on the joint state encoding and based on the time - series model; where the dynamic decision instruction includes the robot's action sequence, force - control parameters, and path planning.

[0043] Therefore, the aforementioned robot decision control method based on gradient sparsity acquires multimodal sensor data of the robot; then, based on a preset sparsity strategy, generates a dynamic mask corresponding to the gradient matrix of the multimodal large model; based on the generated dynamic mask, it filters the effective gradients in the backpropagation process of the dynamic mask; it updates the parameters corresponding to the effective gradients in real time and acquires the output of the multimodal large model based on the updated parameters; based on the acquired multimodal sensor data and the output of the multimodal large model based on the updated parameters, it performs feature fusion to generate a joint state code including environmental state, robot body state, and historical decision information; based on the determined joint state code and a time-series model, it generates a dynamic decision command including the robot's action sequence, force control parameters, and path planning, so that the robot can act based on the generated dynamic decision command, thereby realizing the robot's decision control.

[0044] This gradient-sparse-based robot decision-making and control method combines fine-tuning of a multimodal large model with real-time robot control. By using dynamic sparsity techniques to reduce computational complexity, it achieves efficient real-time decision-making. Specifically, based on gradient sparsity and time-slice scheduling, this method can control the fine-tuning latency to within 20ms, thus meeting the low-latency requirements of real-time robot control. The structured sparsity strategy reduces the computational complexity of a single fine-tuning operation to a linear level, reducing computational power consumption and adapting to embedded hardware deployments. The joint state encoding, which integrates multimodal sensor data and historical decision information, combined with a time-series model, enhances the robustness of dynamic decision-making in complex environments.

[0045] In an exemplary embodiment, the step of generating a dynamic mask corresponding to the gradient matrix of a multimodal large model based on a preset sparsity strategy includes:

[0046] S2041, based on the preset gradient filtering mechanism, calculates the full gradient matrix.

[0047] The formula for calculating the total gradient matrix G is: G∈R N×D Where N is the number of parameters, D is the batch dimension, and R is the set of real numbers.

[0048] S2042, determine and generate an initial binary mask based on the absolute value and standard deviation of each batch gradient in the full gradient matrix and a preset threshold.

[0049] The mean value μ of the absolute value of the current batch gradient is calculated based on the absolute value of the gradient of each batch. The formula for calculating the mean value μ of the absolute value of the current batch gradient is: μ = 1 / N∑∣G∣.

[0050] The formula for calculating the standard deviation σ is:

[0051]

[0052] Among them, G i Let represent the absolute value of the gradient of the i-th parameter in the full gradient matrix G, and let σ represent the standard deviation σ, which can be used to describe the dispersion of the absolute values ​​of the gradients in the current batch.

[0053] Therefore, the preset threshold Threshold can be determined based on the mean μ and standard deviation σ of the absolute values ​​of the current batch gradients: Threshold = μ + α * σ. Here, α is the task sensitivity coefficient, which can be optimized online through reinforcement learning. Of course, the preset threshold can also be a pre-set value.

[0054] Therefore, the initial binary mask can be determined based on the full gradient matrix G and the preset threshold Threshold. Specifically, gradients |G|≥Threshold can be retained to generate the corresponding initial binary mask M1, where M1=1 indicates retention and M1=0 indicates filtering.

[0055] S2043, update the frequency vector based on maintenance parameters; where maintenance parameters are parameters filtered based on historical update frequencies.

[0056] EMA (Exponential Moving Average) can be used to record the historical update frequency of each parameter. The cumulative importance score / mask effectiveness index can be used to reflect the activity level of the corresponding parameter in historical updates; a higher cumulative importance score / mask effectiveness index indicates that the parameter is more important or updated more frequently. The formula for determining the cumulative importance score / mask effectiveness index Ft(i) of the i-th parameter at time t is:

[0057] Ft(i)=β*Ft-1(i)+(1-β)*I(Mt-1(i)=1)

[0058] Where β is the forgetting factor; I is the indicator function, which is updated when the mask retains the indicator function I in the last time; t is the time step, representing the current training / inference iteration step, for example, the t-th backpropagation; i is the parameter index, representing the i-th trainable parameter in the neural network, for example, the weight can be represented as wi and the bias can be represented as bi; Mt(i) is the dynamic mask state (Binary Mask) of the i-th parameter at time t, used to mark whether the parameter is valid, where Mt(i)=1 indicates valid and Mt(i)=0 indicates invalid; Ft-1(i) is the cumulative value of the i-th parameter at the previous time (t-1), which can retain the corresponding historical information through exponential smoothing; Mt-1(i) is the dynamic mask state of the i-th parameter at the previous time (t-1).

[0059] Specifically, maintaining parameters refers to updating the frequency vector (frequency matrix) F∈R through parameters. N Key parameters that are recorded and have a low historical update frequency. The parameter update frequency vector F∈R N The record refers to the frequency vector F updated in real time using an exponential moving average, and the formula is: Ft(i)=βFt-1(i)+(1-β)I(Mt-1(i)=1); where the forgetting factor β controls the influence weight of the historical update frequency). When the indicator function I was retained in the previous dynamic mask, the indicator function I was 1, and conversely, when the indicator function I was not retained in the previous dynamic mask, the indicator function I was 0.

[0060] S2044, Generate a correction mask based on the frequency vector.

[0061] If the corresponding Ft(i) < γ, the parameter is determined to be a "low-frequency updated parameter." Therefore, based on the initial binary mask M1, a modified mask M2 is used to forcibly retain its gradient, preventing key parameters from being filtered out due to accidental low-amplitude gradients. Here, γ is a frequency threshold, which can be determined based on the staff's historical experience. The maintained parameters are "low-frequency but important" parameters dynamically selected through historical update frequencies, ensuring that key parameters that have not been updated for a long time continue to receive gradient updates. For example, a key parameter that has not been updated for a long time could be the contact stiffness parameter in a force control model.

[0062] S2045, Obtain and generate a region mask based on the task label.

[0063] S2046, Generate a dynamic mask based on the initial binary mask, the modified mask, and the region mask.

[0064] Predefine tasks and group their corresponding parameters. For example, a grasping task can be grouped into stress control parameters, and a navigation task into kinematic parameters. Then, activate the parameter region corresponding to the parameter value using the task label T. For example, when the task is "precision assembly," preserve the gradients of the relevant parameters in the robotic arm's end-effector coordinate system and generate a region mask M3.

[0065] More specifically, the force control parameter set relates to the stress tactile signal and related control parameters. The force tactile signal can be a 6-dimensional force or a 6-dimensional torque. For example, parameters such as the stiffness matrix and damping coefficient in the force control model of the robotic arm's end effector can directly affect the contact force control accuracy. Therefore, the force control parameter set can directly affect tasks that require precise force control, such as grasping and assembly tasks.

[0066] The kinematic parameter set corresponds to inertial navigation data (3D coordinates + 4D quaternions) and kinematic model parameters. The inertial navigation data can be a combination of 3D coordinates and 4D quaternions. Examples include the coordinate transformation matrix from the robot's joint space to Cartesian space, and inverse kinematics solution parameters. Therefore, inertial navigation data enables the planning of the robot arm's end-effector pose. For instance, navigation planning tasks and path planning tasks include the planning of the robot arm's end-effector pose.

[0067] Visual images correspond to visual perception-related tasks, such as object detection and pose estimation tasks like recognizing object positions in grasping tasks. Therefore, the parameter set corresponding to visual images is the parameter set of the convolutional layers and feature extraction layers in the visual encoder.

[0068] If the robot includes a semantic parsing module for understanding natural language instructions, then the environmental semantic information corresponds to a complex task planning task, such as assembling parts according to semantic instructions. Therefore, the parameter set corresponding to the environmental semantic information includes the semantic embedding layer and the task planning network parameters.

[0069] Accordingly, after the precision assembly task is activated, the relevant parameters of the robotic arm's end-effector coordinate system are retained. Since the relevant parameters of the robotic arm's end-effector coordinate system involve both force control parameter sets (contact force control of the robotic arm) and kinematic parameter sets (positional accuracy control of the robotic arm), precision assembly is a force-controlled composite task. It is necessary to retain both force control parameters related to force tactile signals and kinematic parameters related to inertial navigation data to ensure contact force accuracy and positional accuracy.

[0070] Therefore, based on the initial binary mask M1, the modified mask M2, and the region mask M3, a dynamic mask can be generated by performing a logical AND operation. The dynamic mask M = M1⊙M2⊙M3, thus ensuring the retention of gradients with high amplitude, low-frequency updates, and relevance to the current task, achieving an optimal balance between accuracy and efficiency. Compared with traditional fixed threshold directions, this dynamic mask in the gradient sparsity-based robot decision-making and control method can combine real-time gradient distribution with historical update patterns, avoiding the omission of key parameters such as low-frequency but important force control parameters, and redundant calculation of high-frequency but non-task-related visual feature parameters, thereby improving the computational accuracy of the gradient sparsity-based robot decision-making and control method. Experiments show that in precision assembly tasks, the update accuracy of key force control parameters increases from 68% to 92%, and the model convergence speed is accelerated by 30%.

[0071] In the aforementioned real-time fine-tuning process, the gradient matrix is ​​structured and sparsified so that the gradients of non-critical parameters are filtered during backpropagation, reducing the computational complexity of a single fine-tuning to O(kn), where k is the sparsification ratio coefficient and n is the total number of model parameters; the sparsification ratio coefficient is the threshold for the proportion of non-critical parameters in the gradient matrix.

[0072] The parameters of a large multimodal model are divided into different sub-matrices according to the network layer structure. These sub-matrices can include visual encoders, force-controlled decoders, and temporal inference layers. Then, a sparsity strategy is applied independently to each sub-matrix. For example, the visual encoder parameters account for 60%, but depending on the task type, such as a haptic-dominated grasping task, its sparsity ratio can be set to 90%, thus retaining only the gradients of key convolutional kernels.

[0073] In an exemplary embodiment, the step of generating a joint state code by performing feature fusion based on multimodal sensor data and the output of a multimodal large model based on updated parameters includes:

[0074] S2102, concatenates multimodal sensor data into a sensor state vector.

[0075] S2103, obtain the current decisions and historical experience of the multimodal large model.

[0076] S2106, determine the joint state code based on the sensor state vector, current decision, historical experience and scaling factor.

[0077] The preprocessed multimodal sensor data is concatenated into sensor state variables. Specifically, the preprocessed multimodal sensor data includes 128-dimensional visual features, 6-dimensional force / torque tactile signals, and 3-dimensional coordinates + 4-dimensional quaternions for inertial navigation pose, which can be represented as S∈R. 141The sensor state vector is further compressed to 64 dimensions using two fully connected layers. After processing by a multimodal large model, the 64-dimensional sensor state vector can output a preliminary decision including the action category and force control target value. This preliminary decision can be converted into a 32-dimensional vector O through an embedding layer.

[0078] Based on historical experience, corresponding historical decision codes can be obtained. Historical decision codes H∈R 64 It refers to the circular buffer that maintains the historical 5-step decision, which can be obtained by extracting temporal features through LSTM (Long Short-Term Memory networks).

[0079] Therefore, attention mechanisms can be used to achieve cross-modal interaction, generating a joint state code C∈R that replaces the traditional simple splicing and includes environment (environment state), ontology (robot ontology state), and historical association (historical decision information). 64 This enables the modeling of semantic dependencies in multimodal data.

[0080] This involves mapping inputs from different modalities to the same dimension through a Linear layer, based on the formula: Attn = Softmax(QK). T The joint state code is obtained from ` / dk)V`. This dimension can be a Q / K / V dimension, where Q is the query vector, K is the key vector, and V is the value vector, facilitating cross-modal interaction. `dk` (dimension of keys) is a scaling factor used to control attention computation. For example, the key vector K can be 64-dimensional. `Q` represents the current environment (environmental state) and the ontology state (robot ontology state); `H` is the multi-dimensional historical decision encoding (encoding corresponding to historical decision information); `V` represents the temporal dependence of historical decisions; and `Attn` is the final joint state code, i.e., the joint state code C.

[0081] Therefore, joint state coding, which integrates multimodal semantic associations, can generate more robust control decisions. T It calculates the correlation or similarity between the sensor state and the current decision, QK TDividing by dk yields a scaled dot product attention mechanism to prevent gradient vanishing. A joint state code is obtained through weighted summation of V, modeling the semantic dependency between "environmental state (S) - current decision (O) - historical experience (H)," thus replacing the implicit association of simple concatenation. Compared to traditional concatenation fusion, this gradient sparsity-based robot decision control method captures dynamic contextual relationships through an attention mechanism. For example, force and tactile signals have higher weight in decisions during high-speed motion, enabling the joint state code to contain temporal causal relationships and improving the robustness of decisions in complex scenarios such as dynamic obstacle avoidance. In complex scenarios such as unstructured environment navigation, the decision error rate is reduced by 55% compared to traditional concatenation fusion methods, and the robot's task completion success rate increases from 72% to 89%.

[0082] In an exemplary embodiment, the step of generating dynamic decision instructions based on joint state coding and a timing model includes:

[0083] S2122, based on a preset machine model, processes joint state encoding to generate temporal features; wherein, the temporal features include action sequences, force control parameters, and path planning. The preset machine model can be a lightweight TransformerEncoder with 3 encoder layers and 128 hidden layers.

[0084] S2124, based on a time series model, captures the temporal correlation of action sequences.

[0085] S2126, Determine the temporal characteristics of the current cycle based on the temporal correlation of the action sequence.

[0086] S2128 generates dynamic decision instructions based on the timing characteristics of the current cycle.

[0087] By processing the joint state code C using the TransformerEncoder, a temporal feature T∈R containing predictions for the next three steps can be generated. 3×128 This allows for the capture of long-range dependencies in the action sequence. For example, such long-range dependencies exhibit high smoothness of joint trajectories during continuous grasping actions.

[0088] Specifically, action sequences can be generated using convolutional layers to produce joint angle trajectories θ(t)∈R. nj×10The algorithm determines and predicts the action sequence for the next 10 control cycles. Here, nj represents the number of joints in the robot (e.g., nj=6 for a 6-DOF robot arm), with each joint corresponding to a row in the trajectory matrix. 10 represents the number of future control cycles to be predicted; each joint needs to predict the next 10 cycles, forming the joint angle trajectory θ(t) over time. For example, if each cycle is 10ms, the motion for the next 100ms can be predicted as θ(t+1), ..., θ(t+9). The physical meaning of this matrix is ​​to generate angle planning values ​​for the next 10 control cycles for each joint, thus ensuring the foresight and continuity of motion control and avoiding trajectory jumps caused by single-step decisions. It also ensures the smoothness of joint motion, avoiding sudden speed changes, and directly driving the robot arm's body motion. Therefore, this gradient-sparse robot decision control method can solve the trajectory planning problem in joint space through motion control oriented towards the robot body, and solve the coordination problem of continuous actions by relying on a pre-set machine model to capture the temporal correlation of action sequences.

[0089] Based on force-tactile encoding, and by outputting the expected value of the contact force Fref∈R6 through Gaussian process regression, the corresponding force control parameters are obtained. Specifically, the force control parameters are used for force control tasks of the end effector, such as grip force control during grasping and contact force constraints during assembly. Thus, this gradient sparsity-based robot decision control method can solve the force / position hybrid control problem in Cartesian space through interactive force control, ensuring the safety and accuracy of interaction with the environment, thereby avoiding collisions and force closure situations in precision assembly.

[0090] By combining an environmental semantic map, the A* algorithm (A Search Algorithm) is used to generate local navigation path points, which are then fitted into continuous trajectories through linear interpolation, thus obtaining path planning. The environmental semantic map includes obstacle locations and target point coordinates; the path planning guides the robot's movement direction in the global environment, such as a collision-free path from point A to point B. Therefore, this gradient sparsity-based robot decision-making and control method can solve the path optimization problem in the environmental space through global motion planning, outputting a pose sequence in Cartesian space, forming a "task objective → body motion" mapping relationship with joint trajectories.

[0091] Then, the MPC (Model Predictive Control) framework is adopted. While executing the current cycle action, the decision of the next cycle can be optimized by rolling to avoid the time consumption of long sequence calculations, thereby ensuring that the decision generation time of each step is ≤15ms.

[0092] It is important to clarify that this gradient sparsity-based robot decision-making and control method does not directly generate the final predicted decision. The dynamic decision command is the output of parallel task branches of action sequences, force control parameters, and path planning, each serving a different control dimension. These are ultimately integrated into a real-time execution strategy through a model predictive control framework. Specifically, the real-time execution strategy executes the action of the current control cycle, such as the joint angle of the first cycle and the force control target of the current cycle. Simultaneously, based on the latest environmental feedback, such as real-time sensor data, it re-optimizes the decision for the next cycle to avoid error accumulation and time consumption caused by pre-calculating long sequences, thus ensuring that the decision generation time for each step is ≤15ms.

[0093] The temporal model in the robot decision control method based on gradient sparsity models the temporal correlation of action sequences. Combined with the rolling optimization of the MPC framework, the dynamic decision corresponding to the dynamic decision command has both a global planning perspective and meets the requirements of real-time execution. Compared with the traditional single-step decision method, the trajectory smoothness in the dynamic environment is improved by 40%.

[0094] The temporal correlation of the action sequences modeled by the time series model is specifically reflected in the following aspects:

[0095] First, the Transformer's self-attention mechanism explicitly calculates the dependencies between actions at different time steps, such as the joint angle at step t and the velocity constraint at step t+2, thereby capturing long-term temporal correlations, such as the coordinated movement of the wrist joint and the end effector in continuous grasping actions. For example, within the joint trajectory prediction range, it can generate joint angle trajectories θ(t)∈R for the next 10 control cycles. nj×10 Instead of making single-step decisions, the time dimension of 10 cycles is directly embedded into the output, forcing the model to learn the smoothness constraints of the action sequence on the time axis. The smoothness constraints include that the joint angle changes in adjacent cycles cannot exceed the speed limit.

[0096] Second: Capturing the long-range dependencies of action sequences can directly demonstrate the Transformer Encoder's modeling of temporal dependencies. The velocity abrupt changes in the temporal dependencies of action sequences in the temporal model correspond to the constraint of the difference in joint angles between adjacent time steps, which can be achieved by predicting multi-step trajectories and optimizing the overall smoothness.

[0097] Third: Traditional single-step decision-making methods ignore the temporal continuity of action sequences, which may lead to skipping between adjacent cycles and problems such as sudden changes in speed; while this gradient sparsity-based robot decision-making and control method predicts multi-step trajectories and introduces temporal modeling of TransformerEncoder to explicitly constrain the temporal correlation of action sequences, ultimately improving the smoothness of the trajectory.

[0098] This gradient sparsity-based robot decision control method reduces fine-tuning time from over 200ms in traditional methods to less than or equal to 30ms by dynamically masking gradients and combining adaptive learning rate with mixed precision. This fully meets the robot control cycle requirement (≤50ms) and supports real-time policy updates in dynamic environments. Specifically, dynamic masking of gradients can achieve sparsity exceeding 95%.

[0099] Through multi-level optimization including sparse computation, adaptive learning rate, and MPC rolling planning, the system still meets the real-time control requirements of robots even with ≥1 billion parameters in a multimodal large model, breaking the traditional bottleneck that "large models are difficult to implement in real-time robot systems." Multimodal fusion and temporal model inference based on attention mechanisms enable the decision-making process to incorporate environmental context understanding and historical action memory, driving the robot's upgrade from "reactive control" to "cognitive decision-making." Thus, this gradient-sparse robot decision-making and control method systematically solves the three core pain points of multimodal large models in real-time robot control through technological innovation, providing key technical support for the application of intelligent robots in dynamic and complex scenarios.

[0100] In one exemplary embodiment, after generating the dynamic decision instruction, the method further includes:

[0101] Obtain the robot's actual operating data and expected operating data.

[0102] Determine the corresponding error based on actual operating data and expected operating data.

[0103] If the error exceeds the preset task threshold, a feedback signal is generated.

[0104] Based on the error gradient in the feedback signal, the target parameters are determined and corrected; where the target parameters are the parameters in the full gradient matrix that correspond to the error gradient.

[0105] Feedback signals are used for real-time fine-tuning of large multimodal models.

[0106] After the robot executes dynamic decision-making commands, the sensor data acquisition module acquires the robot's actual operating data in real time, including joint angles, end effector position / velocity, real-time contact force values, and dynamic coordinates of environmental obstacles. This data is then used to construct a projected state vector containing the theoretical joint trajectories corresponding to the decision-making commands, force control target values, and path planning coordinates. The actual state vector is calculated using the root mean square error. With the expected state vector Error:

[0107]

[0108] Where n is the dimension of the robot state vector. For example, when there are 6 joint angles and 3 position coordinates, n=9.

[0109] When the error E exceeds the task-defined threshold, an error signal is generated to trigger the backpropagation process. For precision assembly, the task-defined threshold can be a positional error of 0.5 mm. Specifically, the error signal is mapped to a gradient correction signal, and the gradient contribution of each layer parameter is calculated using the differential chain rule. Error components are weighted according to the task type to avoid ineffective interference from non-critical errors on the model. In the force control task, the contact force error weight is set to 0.7, and the position error weight is set to 0.3.

[0110] Specifically, let's take pose control as an example: Inertial navigation data is used to directly provide pose information in 3D coordinates and quaternions, which serves as one of the inputs to the joint state code C. The temporal reasoning layer (temporal model) models the temporal changes of the continuity of the pose sequence through a self-attention mechanism. Environmental semantic information indirectly affects the temporal reasoning through local path points (Cartesian coordinate sequences) in path planning. These path points serve as the result of generating the expected trajectory to constrain the joint trajectory. The temporal reasoning layer needs to ensure that the actual motion trajectory is time-aligned with the planned path so that the robot can reach the designated path point in a cycle.

[0111] Specifically, the corresponding parameter acquisition steps (implicit temporal feature extraction process) include: multimodal input concatenation: concatenating 128-dimensional visual features, 6-dimensional force and tactile signals, and 7-dimensional inertial navigation pose into a 141-dimensional sensor state S, and generating a 64-dimensional joint state code C through an attention mechanism with the 32-dimensional preliminary decision O and the 64-dimensional historical decision code H; then performing temporal modeling: inputting the joint state code C into a 3-layer TransformerEncoder with a 128-dimensional hidden layer, where each layer calculates the dependency relationship of different time steps through self-attention, such as the association between the current decision and the previous 2 decisions, and then outputs the temporal features T for the next 3 steps; generating the joint angle trajectory θ(t) through convolutional layers based on the temporal features T, where temporal constraints are explicitly introduced in the time dimension of 10 control cycles, and the attention weights of the TransformerEncoder need to be optimized during parameter updates to minimize the abrupt changes in joint angles between adjacent cycles, i.e., temporal correlation modeling.

[0112] Among them, the parameters of the temporal inference layer, such as the weight matrix of TransformerEncoder, are cross-modal shared temporal feature extractors. They do not directly correspond to a certain type of sensor data, but indirectly learn the temporal evolution of these data by processing the joint state code C, which includes inertial navigation and environmental semantics, such as the time series of path points in navigation tasks and the dynamic changes of contact forces in force control tasks.

[0113] The sparsity scaling factor k can be dynamically determined by defining non-critical parameters as those whose gradient magnitude is lower than the global mean and whose historical update frequency is lower than the median. Specifically, k is calculated as the proportion of non-critical parameters by statistically analyzing the gradient matrix of the current batch. When k=0.8, it means that 80% of the parameters are filtered out.

[0114] Block sparse technology can be used to threshold the entire submatrix, thereby ensuring that the sparse gradient matrix retains the complete structured features of attention heads or convolutional layer channels, and avoiding computational irregularities caused by unstructured sparsity.

[0115] Backpropagation is performed only on the sparsified non-zero gradients (accounting for 1-k), reducing the computational complexity from O(n^2) for full update. 2 The computational complexity is reduced to O(kn). Hypothetically, when the gradient matrix is ​​a sparse vector, matrix operations degenerate into vector operations. For example, when k=0.95, the computational complexity is reduced by 95%, and the time for a single fine-tuning operation is reduced from 200ms to 10ms.

[0116] By employing structured sparsity, the quadratic complexity bottleneck of traditional full gradient updates is overcome, making fine-tuning of large models feasible on embedded hardware such as Jetson AGX (Jetson Accelerated Graphics Port), reducing computational resource consumption by over 80%. Compared to random sparsity, structured sparsity preserves the functional integrity of network layers, improving model accuracy by 30% at the same sparsity. This functional integrity includes force-controlled fully connected layer parameters.

[0117] In a shared feature space for real-time fine-tuning and multi-step decision reasoning, a time-slicing scheduling algorithm is used to achieve asynchronous synchronization between the fine-tuning points corresponding to real-time fine-tuning and the decision nodes corresponding to multi-step decision reasoning. This ensures that the fine-tuning latency for real-time fine-tuning is ≤20ms, meeting the real-time control requirements of the robot. A common embedding space for multimodal features is defined, allowing the gradient calculation for real-time fine-tuning and multi-step decision reasoning to share the encoded vectors of underlying feature extraction, including visual and force / tactile features. Memory mapping technology is used to avoid redundant feature extraction calculations and reduce data transmission latency. The common embedding space can be a 128-dimensional vector.

[0118] In an exemplary embodiment, determining and correcting the target parameter based on the error gradient in the feedback signal includes:

[0119] Obtain the robot's task type.

[0120] Determine the error type based on the task type.

[0121] Determine the error parameter set based on the error type.

[0122] Based on the error gradient and the set of error parameters, the target parameters are determined and corrected.

[0123] The feedback signal is input into the gradient sparsity mechanism to link with the preceding gradient selection process, and then the parameter subspace related to the current error is updated. For example, if the error originates from the force control parameters, the network layer corresponding to the force haptic encoder is updated primarily.

[0124] Accordingly, based on the task types (grabbing, navigation, precision assembly) and their core control objectives mentioned above, the weight allocation of the error components (contact force error, position error, joint trajectory error, etc.) is as follows:

[0125] The core objective of force control tasks such as grasping and precision assembly is to accurately control contact force, preventing the object being grasped by the robot from slipping or being overloaded, while ensuring the accuracy of the end effector position. Error weights can include torque. Specifically, the weight of contact force error can be 0.6~0.7 to avoid task failure due to insufficient gripping force in grasping operations dominated by force control parameters; position error, i.e., end effector coordinates: the weight of position error can be 0.2~0.3 to ensure the positioning accuracy of the grasped object, but slight positional deviations are allowed to prioritize force control stability; the weight of joint trajectory error is 0.1~0.2, making joint movement smoothness a secondary requirement, and allowing for a certain range of angle adjustments to meet force control requirements.

[0126] The core objective of kinematically driven tasks such as navigation and contactless path planning is to quickly and collision-free reach the target pose, prioritizing joint trajectory smoothness and path accuracy. The weight of position / attitude errors, i.e., the weight of inertial navigation pose, can be 0.5~0.6, so that Cartesian space pose accuracy directly affects navigation success rate. Joint trajectory errors affect velocity / acceleration smoothness; therefore, the weight of joint trajectory errors can be 0.3~0.4 to avoid joint mechanical damage such as vibrations during high-speed motion. The weight of contact force errors can be 0~0.1; in contactless tasks, force control errors can be ignored.

[0127] For precision assembly, the core objective of complex tasks combining force control and kinematic accuracy is to achieve high-precision force control, such as component insertion force constraints, and high positional accuracy for hole-axis alignment. The weight of contact force error can be 0.4~0.5 to prevent excessive force from causing component deformation; the weight of position error, i.e., the weight of the XYZ coordinates of the end coordinate system, can be 0.4~0.5 to keep axial alignment error within 0.5mm; the weight of attitude error, i.e., the weight of quaternion deviation, can be 0.1~0.2 to make rotational accuracy secondary to translational accuracy, but the magnitude of the quaternion deviation weight affects the assembly success rate.

[0128] For purely visual guidance tasks such as object recognition and localization-driven grasping, the core objective is the accuracy of visual feature matching. Purely visual guidance tasks indirectly affect end-effector positioning. The weight of visual feature errors, including object center coordinate deviation, is 0.5–0.6 to correlate position errors with visual encoder parameters; the weight of position errors can be 0.3–0.4 to ensure accurate mapping of joint movements through visual localization; the weight of force control errors can be 0–0.1, as force control is unnecessary when there is no contact.

[0129] It should be noted that the feedback signal does not update the entire matrix, but only applies to the parameter subspace related to the current error, i.e., the target parameters. Specifically, the corresponding parameter group is determined according to the error type. For example, when the error is determined to be force control error, it can be determined to be the force haptic encoder parameter group; when the error is determined to be position error, it can be determined to be the kinematic parameter group. The task parameter grouping (M3) and low-frequency parameter protection (M2) in the previous gradient sparsity mechanism are reused, but the filtering condition is changed from gradient magnitude and historical frequency to error correlation. For example, if the error originates from force control parameters, only the subspace corresponding to the force control parameter group is activated. At this time, the mask of that group in M3 is set to 1, and the gradients of other groups, such as the gradient of the visual encoder, are directly filtered. Combined with the parameter update frequency vector F, the gradients of low-frequency update parameters within the force control parameter group, such as the contact stiffness coefficient, are forcibly retained, i.e., the corresponding M2 is effective, thereby preventing key parameters from being filtered out due to weak error signals.

[0130] Compared to the pre-gradient selection (training phase) which is based on "real-time gradient distribution + historical frequency", the feedback fine-tuning (online phase) is based on the selection of "error correlation + task parameter grouping". It is essentially a task-oriented dynamic mask secondary selection, rather than recalculating the mask of the entire matrix. In this way, the gradient update range is narrowed through the error signal to achieve precise fine-tuning, rather than global update.

[0131] Therefore, the gradient sparsity-based robot decision control method forms a closed-loop control of "decision-execution-feedback-fine-tuning" through error feedback from actual operating data. This enables the model to adapt online to environmental changes such as changes in the robot arm's load and sensor noise drift. Compared to open-loop fine-tuning, the task success rate of this gradient sparsity-based robot decision control method is improved by 25%. Error allocation based on task weights avoids redundant updates of irrelevant parameters, improving the update efficiency of key parameters by 40% while maintaining real-time performance and accelerating model convergence.

[0132] In an exemplary embodiment, after the step of determining and correcting the target parameter based on the error gradient in the feedback signal, the method further includes:

[0133] The consistency of parameters between the fine-tuning point and the decision node is verified based on the timestamp; where the fine-tuning point is the time node when the target parameters are corrected, and the decision node is the time node when the dynamic decision instruction is generated.

[0134] When the parameters of the fine-tuning point and the decision node are inconsistent, the dynamic decision command of the previous cycle is output so that the robot can move based on the dynamic decision command of the previous cycle.

[0135] The control period can be divided into fine-tuning points and decision nodes. With a control period of 50ms, the fine-tuning point can be 10ms, and the decision node can be 30ms. Task scheduling is then implemented using an asynchronous queue: in the first 10ms of the control period, gradient sparsity updates are performed using the error feedback from the previous period, and the results are stored in a parameter buffer; in the remaining 30ms, multi-step decision inference for the current period is executed based on the updated model parameters in the buffer. A double-buffering mechanism (current parameters / parameters to be updated) avoids resource conflicts between fine-tuning and decision-making, ensuring that the fine-tuning delay is strictly controlled within 20ms, while also reserving a 10ms fault tolerance time.

[0136] The gradient sparsity update, performed using the error feedback from the previous cycle, corresponds to the update of the feedback signal. The specific linkage logic is as follows: After making a decision in the previous cycle, the robot's actual operating data is acquired, and the error E between this error and the expected state is calculated. This error E includes a task-weighted error component. Then, the error signal corresponding to error E is input into the gradient sparsity mechanism to generate a correction signal containing only the gradients of key parameters, such as updating only the force haptic encoder parameters in a force control task. During the first 10ms of the current control cycle's fine-tuning, this correction signal is used to perform parameter updates, and the results are stored in a buffer for use by subsequent decision nodes. Therefore, the essence of this control algorithm is the concrete engineering implementation of the feedback signal. Gradient sparsity transforms error feedback into parameter updates, forming a closed loop with the dynamic mask sparsity gradient and the weighted error component.

[0137] Before the end of each control cycle, the consistency of parameters between the fine-tuning point and the decision node is checked by timestamp. If an asynchronous deviation occurs, it can be regarded as the fine-tuning is not completed. At this time, the system automatically reverts to the parameters of the previous cycle so that the robot can move based on the dynamic decision instructions of the previous cycle and triggers priority scheduling to ensure the robustness of the system.

[0138] The rollback mechanism for asynchronous deviations involves rolling back immediately upon the existence of a deviation, thus eliminating the need for additional threshold checks. Asynchronous deviations specifically refer to time synchronization issues, not threshold issues related to error magnitude. Examples include situations where fine-tuning points fail to update within 10ms due to excessive computational load. Specific scenarios include: sudden spikes in sparsity computation causing fine-tuning points to take longer than 10ms, preventing decision nodes from obtaining updated parameters on time; and interruptions during fine-tuning leading to inconsistencies between the timestamps of parameters awaiting updates and current parameters in the double-buffering mechanism.

[0139] The rollback strategy is triggered when parameter consistency verification fails. The time stamp is used to determine whether the update is complete, which is a state boolean judgment (complete / incomplete) rather than a threshold comparison of continuous quantities. This ensures that the system still uses known and valid parameters to make decisions in extreme cases such as instantaneous computational overload, thus avoiding the risk of loss of control.

[0140] By using time slicing, the asynchronous execution of fine-tuning and decision-making can be transformed into logical synchronization, ensuring that the two do not block each other. On heterogeneous computing platforms of CPU (Central Processing Unit) / GPU (Graphics Processing Unit), resource utilization is improved by 60%, and end-to-end latency is stabilized within 50ms. Double buffering and asynchronous verification avoid decision failures caused by incomplete fine-tuning. In high-load scenarios such as navigation in complex environments, the failure rate of systems applying this gradient sparsity-based robot decision control method is reduced by 70%.

[0141] During the aforementioned real-time fine-tuning process, the sparsity strategy is dynamically adjusted based on task priority. In high real-time tasks, the update frequency of key perception module parameters is automatically increased, and in low-latency sensitive tasks, the gradient sparsity ratio is expanded to achieve intelligent allocation of computing resources.

[0142] Specifically, a predefined task priority queue is used, and key perception modules are labeled for each task. The priority of the task priority queue can be arranged as follows: emergency obstacle avoidance (highest) > precision assembly > normal handling > standby mode; the key perception module corresponding to the obstacle avoidance task is the visual obstacle detection module, and the key perception module corresponding to the assembly task is the force and tactile sensor encoder. The current task type can be parsed through the task descriptor, thereby activating the parameter group of the corresponding key module. For example, in the obstacle avoidance task, the visual convolutional layer and motion planning layer are marked as high-priority update areas. The task descriptor can be an Action message in ROS (Robot Operating System).

[0143] The correspondence between task priorities and key perception modules is shown in Table 1 below:

[0144]

[0145] Table 1. Correspondence between task priority and key perception modules

[0146] Among them, emergency obstacle avoidance and precision assembly are high-priority tasks. Their key modules benefit from reduced sparsity during gradient sparsification, ensuring exemption from parameter updates for high-frequency tasks. Each task's key module directly corresponds to its core control objective: emergency obstacle avoidance corresponds to environmental perception, precision assembly to stress control feedback, and ordinary handling to path planning. Therefore, the corresponding parameter groups can be dynamically activated through task descriptors, enabling targeted allocation of computational resources.

[0147] When performing high real-time tasks such as emergency obstacle avoidance, the sparsity ratio of critical module parameters is reduced to 30%, retaining 70% of the gradient, while the sparsity ratio of non-critical modules is increased to 95%. A dynamic threshold lower bound, Threshold = μ - 0.5σ, is adopted to ensure that even low-amplitude gradients can update the corresponding critical parameters, thus avoiding missed obstacle detection. When performing low-latency sensitive tasks such as ordinary material handling, the global sparsity ratio is increased to 98%, retaining only the kinematic parameter gradients relevant to the task. In this case, the computation time can be compressed to less than 15ms. The task scheduler monitors the CPU / GPU load in real time, and when multiple tasks are concurrently executed, computing resources are dynamically allocated according to priority to ensure that critical parameter updates are not delayed. Specifically, 50% of GPU computing power can be reserved for high-priority tasks.

[0148] Therefore, the sparse strategy can be dynamically adjusted according to task requirements. In obstacle avoidance scenarios, the update frequency of key visual parameters is increased by 5 times, obstacle detection latency is reduced from 40ms to 15ms, and reaction speed is improved by 60%. In handling scenarios, computing resources are saved by 70%, and battery life is extended by 25%. Through priority-driven resource allocation, the system equipped with a robot decision control method based on gradient sparsity can simultaneously handle complex tasks such as navigation and grasping. Compared with a fixed strategy, the task switching latency is reduced from 200ms to 30ms, significantly improving practicality.

[0149] In an exemplary embodiment, the dynamic mask determination step further includes: implementing threshold screening based on gradient magnitude by dynamically calculating the ratio of the absolute value of each element in the gradient matrix to the global gradient mean; calculating an exponentially weighted moving average of the parameter update frequency during historical fine-tuning to determine dynamic weight allocation; and identifying the key network layers and functional sub-modules corresponding to the current task to preserve local gradient regions. The dynamic mask generation process combines gradient statistical characteristics, parameter update history, and task semantic features to achieve a three-dimensional joint constraint.

[0150] Specifically, calculate the global mean μ of the absolute values of the gradient matrices in the current batch, μ = 1 / N ∑|Gij|, and calculate the proportionality coefficient rij = |Gij| / μ for each element Gij. Set a dynamic threshold ratio τ, where the dynamic threshold ratio τ can be optimized online through reinforcement learning, and the typical value is 1.2 - 1.5. Only retain the gradients where rij ≥ τ to generate the amplitude mask Mmag.

[0151] Maintain a parameter update frequency table F ∈ RN, and update it using exponential weighted moving average: Ft(i) = αFt-1(i) + (1 - α)*I(Mt-1(i) = 1), where α = 0.9 is the forgetting factor, and the update frequencies in the past 10 steps are weighted. For parameters where Ft(i) < freq_threshold (such as 0.3), force them to be set to 1 in the mask to generate the frequency mask Mfreq. Exemplarily, Ft(i) < 0.3 can be forced to be set to 1 in the mask.

[0152] Map through the task label to a predefined set of key network layers. Exemplarily, the assembly task corresponds to the fully connected layer of the force control decoder, and the navigation task corresponds to the path planning Transformer Encoder layer. Use a semantic parsing model to extract task keywords, activate the parameter regions of the corresponding modules, and generate the task mask Mtask.

[0153] Finally, generate the corrected dynamic mask M = Mmag ⊙ Mfreq ⊙ Mtask. Specifically, through element-wise logical AND operation, only retain the gradients with high amplitude, low-frequency updates, and related to the current task.

[0154] Compared with the single-threshold method, the three-dimensional constraint avoids the omission of key low-frequency parameters and the redundant calculation of non-task-related parameters, increasing the update accuracy of the force control parameters in the assembly task from 65% to 93%, and also increasing the gradient filtering rate of the visual semantic module during navigation to 98%. Through task semantic perception, the system equipped with the robot decision control method based on gradient sparsification can automatically identify the key parameter regions of the current task, quickly adjust the mask strategy during task switching, and the pertinence of parameter updates is increased by 40%, enhancing the generalization ability.

[0155] In an exemplary embodiment, after the step of screening the effective gradients in the backpropagation process based on the dynamic mask and before the step of updating the parameters corresponding to the effective gradients in real time, it further includes:

[0156] Weighted correction is performed on the screened effective gradients based on the parameter importance scoring matrix; where the parameter importance scoring matrix is determined based on historical sensitivity, task contribution degree, and the position weight of the network structure.

[0157] Among them, historical sensitivity refers to the gradient variance of the calculated parameters over the past 100 updates; the higher the variance, the more sensitive the parameter. Its weight can be set to 40%. Task contribution refers to the impact of parameter removal on task performance, evaluated through ablation experiments; the greater the impact, the higher the contribution. Its weight can be set to 30%. Structural position weight refers to parameters closer to the output layer, such as the fully connected layer in the decision head; its weight can be set to 30%. Historical sensitivity, task contribution, and structural position weights can be dynamically allocated based on the network structure hierarchy.

[0158] Parameter importance rating matrix W∈R N After normalization, the value range is [0, 1], where high scores correspond to important parameters. Gradient weighting correction is achieved by weighting the non-zero gradient G′ after dynamic masking according to the scoring matrix. Specifically, the corrected effective gradient G... 修正 =G′*WG, where the gradients of critical parameters with W>0.8 are multiplied by 1.5 for amplification and non-critical parameters with W<0.2 are multiplied by 0.8 for attenuation, thus avoiding insufficient updates of critical parameters due to sparsity. Therefore, by activating only the masked parameter paths and dynamically constructing the backpropagation subgraph using dynamic computation graph techniques, the computation of filtered parameters can be skipped, thereby reducing memory usage and computation time. Dynamic computation graph techniques can be PyTorch's autograd (Automatic Differentiation).

[0159] Important parameters are those with a score W > 0.8 in the parameter importance score matrix W. These parameters play a crucial role in model performance, task accuracy, or sensitivity to dynamic updates. Non-critical parameters are those with a score W < 0.2. These parameters typically have a smaller impact on the model or are redundant / auxiliary parameter connections.

[0160] For example, in a robot control model, key parameters may include the weights of the fully connected layers corresponding to the force control parameters of the end effector. The closer these weights are to the output layer, the higher their contribution to the task. In the vision branch, key parameters may include the convolutional kernel parameters responsible for obstacle edge detection. The higher the historical gradient variance in the convolutional kernel parameters, the more frequently updates are triggered due to environmental changes. Non-critical parameters may include redundant connection weights in early feature extraction layers and bias parameters of the BatchNorm (Batch Normalization Layer) used for stable training.

[0161] In the gradient correction process, the distinction between important and non-critical parameters is not a simple binary classification, but rather a segmented and differentiated adjustment based on a continuous importance score W ranging from [0, 1]. Although there is no absolute "line" boundary, refined gradient correction for parameters of different importance is achieved through preset scoring intervals and corresponding correction rules.

[0162] To determine G based on important or non-critical parameters 修正 Gradient correction is not an independent operation; it works in conjunction with the dynamic mask M, which determines whether parameters participate in the update, to form a two-layer "filtering-correction" control.

[0163] Specifically: A dynamic mask is used for filtering, retaining parameters corresponding to valid gradients. Important parameters with W > 0.8 have a high probability of being retained by the dynamic mask, while non-critical parameters with W < 0.2 have a high probability of being filtered out. However, this is not absolute; the gradient sparsity-based robot decision control method allows a small number of non-critical parameters (parameters with W < 0.2) to temporarily participate in the update to maintain network stability. Gradient correction is performed only on the parameters retained by the dynamic mask (M(i) = 1), while filtered parameters (M(i) = 0) are directly skipped from the update (gradient set to zero). Here, i represents the i-th parameter, and M(i) represents the dynamic mask corresponding to the i-th parameter.

[0164] Then, perform an element-wise correction operation on each parameter i. If it is preserved by the mask (M(i)=1), then the coefficient is corrected according to the corresponding W(i):

[0165]

[0166] Among them, G 修正 (i) represents the effective gradient after the correction of the i-th parameter, G'(i) represents the parameter of the i-th non-zero gradient after dynamic masking, and W(i) represents the parameter importance score matrix corresponding to the i-th parameter.

[0167] The correction coefficients are dynamically generated for each parameter. Therefore, each parameter has an independent W(i) and correction rule. In other words, the parameters are not adjusted globally, thus achieving gradient optimization accurate to a single parameter.

[0168] By adding importance weighting to gradient screening, the effective update volume of key parameters is increased by 50%, and redundant updates of non-key parameters are reduced by 30%. Experiments show that in complex force control tasks, the contact force control accuracy is improved by 22%, and the model fluctuation amplitude is reduced by 40%. Through path activation technology, the memory usage of backpropagation is reduced by 60%, supporting real-time fine-tuning of larger-scale models (such as 10-B parameters) on edge devices.

[0169] In an exemplary embodiment, before the step of generating a dynamic mask corresponding to the gradient matrix of a multimodal large model based on a preset sparsity strategy, the method further includes:

[0170] A three-dimensional adjustment factor is constructed, which includes the sparsity ratio, the current gradient norm, and the historical parameter fluctuation amplitude. Among them, s is used to characterize the retention ratio of the current gradient (s=1-k), with a value range of [0.01, 0.3], where k is the sparsity ratio; the gradient norm g is used to select the L2 norm (Euclidean norm) of the gradient, which can be normalized to [0, 1]; the historical fluctuation amplitude v is used to represent the standard deviation based on the historical 5-step parameter update, reflecting the parameter stability, with a value range of [0, 0.5].

[0171] The learning rate of the multimodal large model is determined based on the three-dimensional adjustment factor.

[0172] The gradient sparsity-based robot decision control method can automatically improve the learning rate compensation accuracy in the information loss region caused by gradient sparsity, and dynamically reduce the learning rate in the high-density gradient update region to avoid parameter oscillation. It forms a learning rate intelligent adjustment strategy based on dynamic environment perception, so as to balance the fine-tuning accuracy and computational efficiency through the mechanism corresponding to the multimodal large model by adaptive learning rate adjustment.

[0173] Specifically, the fuzzy set of input variables is defined as follows: s: {low, medium, high}, v: {stable, fluctuating}; where s: {low, medium, high} corresponds to [0.01, 0.1), [0.1, 0.2), [0.2, 0.3]; v: {stable, fluctuating} corresponds to the standard deviation threshold.

[0174] When s is low (i.e., high sparsity) and g is low, the learning rate is increased by 20% to compensate for information loss; when v is high (i.e., parameter oscillation), the learning rate is decreased by 15% and gradient clipping is triggered. The adjustment coefficient β can be obtained through fuzzy inference. The current learning rate ηt = η0 * β, where η0 is the base learning rate and β is output by the fuzzy controller. The typical range of β is [0.5, 1.5].

[0175] In the case of low sparsity (s ∈ [0.01, 0.1]), the gradient retention rate is low, meaning high sparsity, retaining only 1%-10% of the gradient. In the case of medium sparsity (s ∈ [0.1, 0.2]), s is in the middle range, retaining 10%-20% of the gradient. In the case of high sparsity (s ∈ [0.2, 0.3]), the gradient retention rate is high, but sparsity is low, retaining 20%-30% of the gradient. Therefore, s directly corresponds to the gradient retention rate; the smaller the value, the fewer gradients are retained, and the higher the sparsity. Thus, low s corresponds to high sparsity and fewer retained gradients, while high s corresponds to low sparsity and more retained gradients.

[0176] Based on the normalization range g∈[0,1] and the conventional partitioning logic of fuzzy logic, typically using the midpoint as the boundary, it can be determined that in the case of a small gradient norm (i.e., small g), g∈[0,0.5), the overall magnitude of the gradient after filtering is low, resulting in less update information; while in the case of a large gradient norm (i.e., large g), g∈[0.5,1], the gradient magnitude is high, containing key update information. After gradient norm normalization, 0.5 is used as an intermediate value to distinguish between "small" and "large," which conforms to the implicit partitioning of the intermediate interval in fuzzy logic and is consistent with the strategy of increasing the learning rate when s is low and g is small. For example, in the case of low sparsity but small gradient magnitude, it indicates insufficient effective information, requiring compensation.

[0177] The low / high range of historical fluctuation amplitude v is defined based on the standard deviation of parameter updates over the past 5 steps in the range v∈[0, 0.5], combined with the fuzzy set description of "stable" and "fluctuating". Specifically, in the case of low fluctuation, i.e., low / stable v, v∈[0, 0.25), the standard deviation is small, parameter updates are stable, and there is no significant oscillation; in the case of high fluctuation, i.e., high / fluctuating v, v∈[0.25, 0.5], the standard deviation is large, parameter updates oscillate significantly, and gradient clipping needs to be triggered. The midpoint of the interval, 0.25, is used as the boundary to distinguish the stability of parameter updates, which matches the strategy of reducing the learning rate when v is high, i.e., the greater the fluctuation, the more necessary it is to suppress oscillations.

[0178] It should be noted that a 20% increase in the learning rate corresponds to β = 1.2 (i.e., ηt = η0 × 1.2), while a 15% decrease corresponds to β = 0.85 (i.e., ηt = η0 × 0.85). However, β is not directly determined by a single condition, but rather is a continuous value dynamically calculated based on a preset rule base by the fuzzy inference system, which integrates the membership degrees of multiple input variables. The typical range is [0.5, 1.5]. Input variables include the sparsity ratio, the current gradient norm, and the historical parameter fluctuation amplitude.

[0179] Therefore, the gradient sparsity-based robot decision control method can automatically increase the learning rate by 15%-20% even when information loss occurs due to sparsity, such as when s=0.05, thus avoiding performance degradation caused by insufficient gradients. Compared to a fixed learning rate, the fine-tuning accuracy is improved by 18%. When parameter fluctuations are too large, such as when sudden environmental changes cause gradient anomalies, the fuzzy control logic can promptly reduce the learning rate to suppress parameter oscillations, improving the convergence stability of the model in dynamic environments by 35%.

[0180] In an exemplary embodiment, the step of generating a joint state code by performing feature fusion based on multimodal sensor data and the output of a multimodal large model based on updated parameters includes:

[0181] Spatial feature pyramid extraction is performed on visual images to obtain image features.

[0182] Force characteristics are obtained by performing time-frequency domain feature transformation on the force tactile signal.

[0183] Spatiotemporal trajectory modeling is performed on inertial navigation data to obtain navigation features.

[0184] Semantic features are obtained by embedding environmental semantic information into a knowledge graph.

[0185] Based on image features, force features, navigation features, semantic features, and the output of the multimodal large model based on updated parameters, feature fusion is performed to generate joint state codes.

[0186] Multi-scale spatial features (16×16, 32×32, 64×64) of visual images are extracted using FPN (Feature Pyramid Networks), and then unified to a 128-dimensional vector through adaptive pooling to preserve spatial details and global semantics. Force tactile signals are converted into time-frequency maps using STFT (Short-Time Fourier Transform), extracting 10-dimensional time-frequency features such as energy spectrum and dominant frequency, and then combined with time-domain mean / variance to form a 20-dimensional force tactile encoding. LSTM modeling is performed on historical 50ms inertial navigation data to extract multiple spatiotemporal trajectory features such as acceleration trends and steering curvature to generate force features of a 32-dimensional dynamic trajectory vector. The LiDAR point cloud containing environmental semantic information is converted into an occupancy grid map, and obstacle relationship features are extracted using GCN (Graph Convolutional Network). Combined with BERT embedding of natural language commands, semantic features of a 40-dimensional semantic vector are formed.

[0187] Cross-modal attention fusion unifies multimodal features, including 128-dimensional visual images, 20-dimensional force and tactile signals, 32-dimensional inertial navigation data, and 40-dimensional environmental semantic information, into 64 dimensions through a linear layer, resulting in X=[Xvis, Xft, Ximu, Xsem].

[0188] Here, Xvis is a 128-dimensional image feature, which is a 128-dimensional vector extracted from a visual image using FPN (multi-scale spatial features, 16×16, 32×32, 64×64), and then uniformly converted into a 128-dimensional vector through adaptive pooling. It includes object position, contour, and global semantics. For example, the image feature can be obstacle feature or target object feature.

[0189] Xft is a 20-dimensional force feature, which is generated by converting the force tactile signal into a time-frequency diagram via STFT. It extracts the energy spectrum, the dominant frequency of the 10-dimensional time-frequency feature, and the time-domain mean / variance of the 10-dimensional time-domain statistical feature, totaling 20 dimensions, reflecting the magnitude, direction, and time-varying characteristics of the contact force. For example, the force feature can be an impact vibration feature.

[0190] Ximu is a 32-dimensional navigation feature. It is created by modeling inertial navigation data (including acceleration, angular velocity, Euler angles, etc.) over a historical 50ms period using LSTM, extracting spatiotemporal trajectory features, such as acceleration trends and steering curvature, which characterize the robot's dynamic state. For example, navigation features can determine whether the robot is in an acceleration / deceleration phase.

[0191] Xsem is a 40-dimensional semantic feature based on environmental semantic information. It is converted into an occupancy grid map through LiDAR point cloud and the spatial relationship of obstacles is extracted through GCN. For example, obstacle A is 1m in front of the robot on the left. Combined with BERT embedding of natural language instructions, it forms a joint encoding of environmental semantics and task instructions.

[0192] Therefore, a gating network G=σ(Linear(X)) can be constructed to output the weight vectors of each modality. The joint state code is generated by weighted summation: C=∑(Gi*Attni), where Gi is the gating weight of the i-th modality. The modality dominated by the current task is dynamically selected, such as the force tactile weight being increased to 0.6 when grasping.

[0193] Through targeted processing such as spatial pyramids, time-frequency transformation, and spatiotemporal modeling, the discriminative power of each modality feature is improved by more than 30%. Compared with the original data splicing, the accuracy of environmental semantic understanding is improved from 75% to 89%. The combination of adaptive gating and cross-modal attention enables the system to automatically focus on key modalities according to task requirements, such as increasing the weight of force and tactile sensation in dark environments, and improving decision robustness by 50% in unstructured environments.

[0194] In an exemplary embodiment, the step of generating dynamic decision instructions based on joint state coding and a timing model includes:

[0195] Based on a hybrid architecture, a hierarchical model is performed on the spatiotemporal correlation features and task constraint relationships in joint state coding.

[0196] Multiple candidate action sequences are generated within a preset decision window using a rolling time-domain optimization method. Multi-objective game theory is then performed by combining robot kinematic and dynamic constraints, semantic information of environmental obstacles, and energy consumption optimization objectives to output the optimal dynamic decision instruction set that meets real-time requirements.

[0197] The hybrid architecture design includes low-level temporal modeling and high-level relational modeling. In the low-level temporal modeling, LSTM is used to process continuous temporal signals such as joint angles and force control parameters to capture the dynamic inertial characteristics of the robot's motion; for example, dynamic inertial characteristics can be joint acceleration constraints. In the high-level relational modeling, a GNN is used to construct a task constraint graph, where nodes are action sub-objectives and edges are constraint relationships, outputting task-level dependency features; action sub-objectives include approaching objects and adjusting posture, and constraint relationships include temporal sequence relationships and force control priority.

[0198] The hierarchical modeling process includes spatiotemporal feature extraction. Specifically, the joint state code is input into an LSTM to generate a hidden state ht containing motion continuity, capturing dynamic changes in the robot's velocity, acceleration constraints, etc. A GNN is used to parse the semantic constraints of the current task, generating a constraint vector ct, which is then concatenated with ht and input into the decision head. A semantic constraint could be that the assembly requires maintaining an end-effector velocity ≤ 0.1 m / s.

[0199] Within a preset decision window, i.e., within the next 5 steps, 10 candidate action sequences are generated. Each sequence includes joint trajectories, force control targets, and waypoints. Constraints include kinematic constraints, dynamic constraints, and environmental constraints. Kinematic constraints include joint angle ranges and velocity limits; dynamic constraints include torque limits and inertia matrix constraints; environmental constraints include obstacle distance ≥ a safety threshold. Each step can last 10 ms. The safety threshold can be set to 0.2 m.

[0200] The candidate action sequence is generated based on the aforementioned dynamic changes and the concatenated constraint vector. Specifically, the joint state encoding `ht` contains the temporal dependencies of real-time states such as joint angles, velocities, and accelerations, representing the robot's dynamic characteristics and reflecting its current motion inertia; for example, if the robot's second joint is rotating at high speed, a sudden stop must be avoided. The constraint vector `ct` is the task constraint output by the GNN, used to define the physical limitations and task objectives of the action. `ht` and `ct` are concatenated and used as input to the decision head. `ht` provides the dynamic state of the robot's motion, such as joint torque limits, while `ct` provides task-level semantic constraints, such as safe distances for path planning. `ht` and `ct` together define the feasible solution space for the candidate actions. The generated candidate action sequence must satisfy the kinematic / dynamic constraints implicit in `ht` and the explicit task constraints in `ct`; otherwise, it is discarded.

[0201] Furthermore, the optimization objective function can be set as min = w1 * energy consumption + w2 * path smoothness + w3 * constraint violation. Here, w1, w2, and w3 are weights dynamically assigned based on the task type. w1 is the energy consumption weight, accounting for 40%-60%, dynamically adjusted by the task, used to measure the energy efficiency of the robot's actions. Priority is given to optimizing energy-intensive tasks such as material handling, favoring low-power action sequences. The energy efficiency of the robot's actions can be measured by motor torque consumption and battery power output. w1 can be set to 0.6.

[0202] w2 is the path smoothness weight, accounting for 20%-40%, used to quantify the motion trajectory and avoid violent movements such as mechanical wear caused by high-speed start-stop. In precision assembly tasks, w2 is increased to ensure trajectory smoothness. The continuity of the motion trajectory can be specifically calculated by the sum of squares of the second derivatives of the joint angle changes. Here, w2 can be set to 0.5.

[0203] w3 is the constraint violation weight, accounting for 10%-30%, used to calculate the degree of violation of constraints by the action sequence. w3 is never 0 in any task to ensure that safety constraints take priority. w3 is highest in emergency obstacle avoidance tasks; for example, w3 can be set to 0.4 in an emergency obstacle avoidance task. The degree of constraint violation can be the penalty value when the obstacle distance is less than the safety threshold of 0.2m.

[0204] Real-time optimization employs parallel computing to generate candidate sequences and uses a pruning algorithm to eliminate obviously infeasible paths, keeping the solution time within 15ms to meet real-time decision-making requirements. The pruning algorithm can be Rapidly-exploring Random Trees.

[0205] The hybrid architecture simultaneously models the temporal continuity of robot motion and task semantic constraints, improving decision rationality by 60% and path planning success rate from 70% to 92% in scenarios involving multi-objective optimization. For example, a multi-objective optimization scenario could be a task scenario that simultaneously includes obstacle avoidance and energy-efficient navigation.

[0206] Compared to traditional global planning methods, rolling temporal optimization combined with pruning algorithms can control computation time while ensuring solution quality, and improves reaction speed by 3 times in dynamic environments, making it suitable for high-speed motion scenarios such as obstacle avoidance for mobile robots.

[0207] In one exemplary embodiment, such as Figure 3 As shown, a robot 300 includes: a sensor data acquisition module 302 and a control module 304.

[0208] The sensor data acquisition module 302 is used to acquire multimodal sensor data of the robot.

[0209] The control module 304 is connected to the sensor data acquisition module 302, and the control module 304 is used to execute the steps of the above-described robot decision control method based on gradient sparsity.

[0210] like Figure 4 As shown, specifically, the control module 304 can be divided into a mask generation unit 3042, a real-time update unit 3044, a feature fusion unit 3046, and a decision generation unit 3048.

[0211] The mask generation unit 3042 is used to generate a dynamic mask corresponding to the gradient matrix of the multimodal large model based on a preset sparsity strategy. The multimodal large model is a decision model that reflects the robot's decision-making. Based on the dynamic mask, the effective gradients in the backpropagation process of the dynamic mask are selected.

[0212] The real-time update unit 3044 is used to update the parameters corresponding to the effective gradient in real time and obtain the output of the multimodal large model based on the updated parameters.

[0213] The feature fusion unit 3046 is used to acquire multimodal sensor data of the robot; based on the multimodal sensor data and the output of the multimodal large model based on the updated parameters, feature fusion is performed to generate joint state codes; wherein, the multimodal sensor data includes visual images, force and tactile signals, inertial navigation data and environmental semantic information; the joint state codes include environmental state, robot body state and historical decision information.

[0214] The decision generation unit 3048 is used to generate dynamic decision instructions based on joint state encoding and a timing model; wherein, the dynamic decision instructions include the robot's action sequence, force control parameters, and path planning.

[0215] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0216] Based on the same inventive concept, this application also provides a gradient-sparse robot decision control device for implementing the gradient-sparse robot decision control method described above. The solution provided by this device is similar to the implementation described in the above method; therefore, the specific limitations of one or more gradient-sparse robot decision control device embodiments provided below can be found in the limitations of the gradient-sparse robot decision control method described above, and will not be repeated here.

[0217] In one exemplary embodiment, such as Figure 5 As shown, a robot decision control device 500 based on gradient sparsity is provided, including: an acquisition module 502, a dynamic mask generation module 504, a filtering module 506, a real-time update module 508, a joint state code generation module 510, and a dynamic decision instruction generation module 512, wherein:

[0218] The acquisition module 502 is used to acquire multimodal sensor data of the robot; wherein, the multimodal sensor data includes visual images, force and tactile signals, inertial navigation data and environmental semantic information.

[0219] The dynamic mask generation module 504 is used to generate a dynamic mask corresponding to the gradient matrix of the multimodal large model based on a preset sparsity strategy. The multimodal large model is a decision model that reflects the robot's decision-making.

[0220] The filtering module 506 is used to filter effective gradients in the backpropagation process of the dynamic mask based on the dynamic mask.

[0221] The real-time update module 508 is used to update the parameters corresponding to the effective gradient in real time and obtain the output of the multimodal large model based on the updated parameters.

[0222] The joint state coding generation module 510 is used to perform feature fusion based on multimodal sensor data and the output of the multimodal large model based on updated parameters to generate joint state codes; the joint state codes include environmental state, robot body state and historical decision information.

[0223] The dynamic decision instruction generation module 512 is used to generate dynamic decision instructions based on joint state encoding and a timing model; wherein, the dynamic decision instructions include the robot's action sequence, force control parameters, and path planning.

[0224] The modules in the aforementioned gradient sparsity-based robot decision-making and control device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the computer device's memory as software, so that the processor can call and execute the corresponding operations of each module.

[0225] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 6 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores multimodal sensor data. The I / O interfaces are used for information exchange between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When executed by the processor, the computer program implements a gradient sparsity-based robot decision-making and control method.

[0226] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components, or combine certain components, or have different component arrangements.

[0227] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of any of the methods described above for the robot decision control method based on gradient sparsity.

[0228] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the methods described above for robot decision control based on gradient sparsity.

[0229] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of any of the methods described above for the gradient-sparsed robot decision control method.

[0230] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0231] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0232] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A robot decision control method based on gradient sparsification, characterized in that, The method comprises: acquiring multi-modal sensor data of a robot; wherein the multi-modal sensor data comprises visual images, force tactile signals, inertial navigation data, and environmental semantic information; based on a preset sparsification strategy, generating a dynamic mask corresponding to the gradient matrix of a multi-modal large model, the multi-modal large model being a decision model reflecting the decision of the robot; based on the dynamic mask, screening effective gradients in the backpropagation process of the dynamic mask; updating the parameters corresponding to the effective gradients in real time, and acquiring the output of the multi-modal large model based on the updated parameters; performing feature fusion based on the multi-modal sensor data and the output of the multi-modal large model based on the updated parameters to generate a joint state code; the joint state code comprises environmental state, robot body state and historical decision information; based on the joint state code and based on a time series model, generating a dynamic decision instruction; wherein the dynamic decision instruction comprises the action sequence, force control parameter and path planning of the robot.

2. The method of claim 1, wherein, The method comprises: based on a preset sparsification strategy, generating a dynamic mask corresponding to the gradient matrix of a multi-modal large model, the multi-modal large model being a decision model reflecting the decision of the robot; based on a preset gradient screening mechanism, calculating a full-quantity gradient matrix; determining and generating an initial binary mask based on the absolute values and standard deviations of each batch of gradients in the full-quantity gradient matrix and a preset threshold; maintaining a parameter update frequency vector; wherein the maintained parameter is a parameter screened based on historical update frequency; generating a correction mask according to the frequency vector; acquiring and generating a region mask according to the task label; 3. The method of claim 1, wherein, generating the dynamic mask according to the initial binary mask, the correction mask and the region mask. The method comprises: concatenating the multi-modal sensor data into a sensor state vector; acquiring the current decision and historical experience of the multi-modal large model; 4. The method of claim 2, wherein, determining the joint state code according to the sensor state vector, the current decision, the historical experience and a scaling factor. The method comprises: processing the joint state code based on a preset machine model to generate time series features; wherein the time series features comprise action sequence, force control parameter and path planning; based on the time series model, capturing the time correlation of the action sequence; determining the time series features in the current period according to the time correlation of the action sequence; 5. The method of claim 4, wherein, generating a dynamic decision instruction according to the time series features in the current period. After generating the dynamic decision instruction, the method further comprises: acquiring actual running data and expected running data of the robot; determining the corresponding error according to the actual running data and the expected running data; generating a feedback signal in the case that the error is greater than a preset task setting threshold; determining and correcting a target parameter according to the error gradient in the feedback signal; wherein the target parameter is a parameter in the full-quantity gradient matrix corresponding to the error gradient.

6. The method of claim 5, wherein, The step of determining and correcting the target parameter according to the error gradient in the feedback signal comprises: acquiring a task type of the robot; determining an error type according to the task type; determining an error parameter group according to the error type; determining and correcting the target parameter according to the error gradient and the error parameter group.

7. The method of claim 5, wherein, After the step of determining and correcting the target parameter according to the error gradient in the feedback signal, the method further comprises: verifying the parameter consistency of the fine-tuning node and the decision node based on the timestamp, wherein the fine-tuning node is a time node at which the target parameter is corrected, and the decision node is a time node at which the dynamic decision instruction is generated; in the case that the parameters of the fine-tuning node and the parameters of the decision node are inconsistent, outputting the dynamic decision instruction of the previous period to make the robot move based on the dynamic decision instruction of the previous period.

8. The method of claim 1, wherein, After the step of screening the effective gradient in the back propagation process based on the dynamic mask, and before the step of updating the parameter corresponding to the effective gradient in real time, the method further comprises: weighting and correcting the screened effective gradient based on a parameter importance scoring matrix, wherein the parameter importance scoring matrix is determined based on historical sensitivity, task contribution degree, and position weight of network structure.

9. The method of claim 1, wherein, Before the step of generating the dynamic mask corresponding to the gradient matrix of the multi-modal large model based on the preset sparsification strategy, the method further comprises: constructing a three-dimensional adjustment factor, wherein the three-dimensional adjustment factor comprises a sparsification ratio, a current gradient norm, and a historical parameter fluctuation amplitude; determining the learning rate of the multi-modal large model according to the three-dimensional adjustment factor.

10. A robot, characterized in that The method comprises: a sensor data acquisition module, configured to acquire multi-modal sensor data of a robot; a control module, connected with the sensor data acquisition module, configured to execute the steps of the method according to any one of claims 1 to 9.

Images