Robot operation trajectory dynamic reconstruction system, method and computer program product under large language model task interaction

Abstract

This invention provides a system, method, and computer program product for dynamic reconstruction of robot trajectories under large language model task interaction. It includes: constructing a global coordinate system point cloud based on sensor data, obtaining point clusters through spatial clustering, constructing a local geometric tensor matrix and calculating hardness coefficients to identify hard agglomeration regions; dividing the global point cloud into point cloud voxels, selecting task-related voxel sets, and calculating voxel physical resistance feature weights and comprehensive cost values. It plans an environmental reconstruction path from the robot to the agglomeration region, sets a breaking parameter sequence based on resistance weights, and drives the breaking mechanism. Based on torque data during the breaking process, it identifies the critical point of agglomeration breakage through time-domain feature analysis, calculates the environmental reverse action gain signal, and corrects the corresponding voxel resistance weights. Based on the corrected resistance weights, it again achieves dynamic trajectory reconstruction through a heuristic path search algorithm. This invention realizes intelligent, closed-loop, and adaptive robot trajectory planning.

Description

Technical Field

[0001] This invention relates to the field of embodied intelligence and autonomous system control technology, specifically to a system, method, and computer program product for dynamic reconstruction of robot operation trajectory under large language model task interaction. Background Technology

[0002] With the deep integration of large language models and embodied intelligence technology, the operational efficiency of autonomous unmanned systems in unstructured and complex scenarios (such as environments with non-uniform particle packing and dynamically evolving physical spaces) has become a key indicator for measuring the level of robot intelligence. In these highly challenging tasks, the environment not only exhibits complex topological geometry but also demonstrates significant nonlinear mechanical resistance. Existing avoidance-based path planning schemes often neglect the embodied device's ability to actively intervene in the environment, making it difficult to achieve deep alignment between task instructions and physical reconstruction. To improve the autonomy of operations and the optimality of paths, there is an urgent need for a planning method that can deeply analyze the high-level semantics of large models, actively drive environmental topological evolution, and achieve bidirectional optimization of the environment and trajectory.

[0003] First, existing trajectory planning techniques primarily focus on kinematic parameter coordination and state robustness, lacking the ability to deeply couple the physical actions of the actuator with high-level task semantics. Current advanced solutions largely aim to optimize the robot's stability in geometric space by establishing complex motion mathematical models. For example, Chinese National Patent No. CN202511972350.9 discloses a dynamic trajectory planning method for autonomous vehicles based on lateral and longitudinal coordination. This technique calculates lateral and longitudinal coordination indices and uses the changing trend of trajectory planning deviation values ​​to dynamically correct control parameters. While this approach effectively solves the motion smoothness problem in dynamic environments, it essentially remains within the realm of simple kinematic control and fails to establish a logical mapping between high-level semantic tasks and low-level physical intervention actions. When facing complex scenarios requiring active changes to the environmental topology, this technique cannot analyze the semantic benefits behind the actions, resulting in the system only being able to perform passive obstacle avoidance planning when facing unstructured obstacles, making it difficult to achieve the perception-interaction-remodeling closed loop required for embodied intelligence.

[0004] Secondly, traditional planning operators exhibit significant passive adaptability in environmental interaction logic, making it difficult to capture the topological benefits brought about by changes in environmental state. Existing dynamic cost map update mechanisms typically treat physical obstacles as constant negative constraints, lacking a mechanism for predicting and utilizing the evolution of obstacle physical states. For example, Chinese National Patent No. CN202511947733.0 discloses a robot path planning method that uses static maps and dynamic cost refeeding to achieve global path correction. However, in actual unstructured interaction scenarios (such as the material accumulation environment involved in this embodiment), this technology cannot identify the physical critical point of intervention and passage, ignoring potential paths that reduce global operational cost by actively reshaping the environmental topology, resulting in the planned trajectory falling far short of the globally optimal balance point in terms of spatiotemporal efficiency and energy consumption.

[0005] Finally, existing embodied control frameworks lack a physical-level causal closed loop between actuator actions and environmental evolution logic. Even advanced models employing multimodal token inference often only act as observers for static predictions, failing to establish a causal relationship model that triggers environmental state transitions based on actions. For example, Chinese National Patent No. CN202512026290.8 discloses an action trajectory planning method, device, embodied intelligent system, and autonomous vehicle. While it optimizes inference efficiency by selecting highly correlated visual tokens, its logical core does not integrate a quantization operator for physical feedback gain. This results in the system's inability to perceive the dynamic evolution of environmental resistance after physical operations (such as the transition from a high-resistivity state to a flow-optimized state), and its inability to perform secondary iterative optimization of the path based on the real-time feedback gain signal from the environment. This makes it highly susceptible to decision feedback lag and operational accuracy drift when executing long-sequence tasks.

[0006] Therefore, developing a trajectory planning method driven by the semantics of a large language model, which enables proactive reshaping of environmental topology and the construction of a physical feedback optimization mechanism, is of vital value for improving the operational autonomy of embodied intelligent systems in extremely complex scenarios. Summary of the Invention

[0007] The purpose of this invention is to address the shortcomings of existing technologies by providing a dynamic reconstruction system for robot operation trajectories under large language model task interaction, comprising: The multimodal recognition module is used to construct a global coordinate system point cloud based on sensor data, perform spatial clustering on the global coordinate system point cloud to obtain multiple point clusters, construct a local geometric tensor matrix for each point cluster, calculate the hardness coefficient of each point cluster based on the local geometric tensor matrix, determine the hard nodule region corresponding to the point cluster based on the hardness coefficient, divide the global coordinate system point cloud into multiple point cloud voxels, and select a set of point cloud voxels related to the robot's current task instruction from the global coordinate system point cloud through a cross-modal self-attention mechanism. Based on the physical characteristics of the hard nodule material of each point cloud voxel in the set of point cloud voxels, calculate the physical resistance feature weight of the corresponding point cloud voxel, and calculate the comprehensive cost value of each point cloud voxel in the set of point cloud voxels based on the physical resistance feature weight. The environmental reconstruction decision module is used to calculate the set of points with the minimum comprehensive cost from the robot's current position to the hard nodule area among all points contained in the point cloud voxel set using a heuristic path search algorithm. This set of points serves as the robot's environmental reconstruction path. The module sets a breaking parameter sequence based on the physical resistance characteristic weight of the hard nodule area, controls the robot to reach the hard nodule area according to the environmental reconstruction path, and drives the robot's breaking mechanism to break the hard nodule according to the breaking parameter sequence. The reverse optimization gain module is used to determine the critical point of the hard nodule's transformation from a hard constraint state to a loose particle state based on the torque data during the process of breaking the hard nodule. When the critical point is reached, the environmental reverse action gain signal is calculated using a nonlinear function based on torque release efficiency, and the physical resistance feature weight of the corresponding point cloud voxel is corrected based on the environmental reverse action gain signal. The topology trajectory optimization module is used to calculate the combination of point clouds that minimizes the overall cost of the robot's journey from its current position to the target location among all point clouds contained in the point cloud voxel set, based on the physical resistance feature weights corrected for the corresponding point cloud voxels, using a heuristic path search algorithm. This combination serves as the robot's path.

[0008] Furthermore, in the multimodal recognition module, a local geometric tensor matrix is ​​constructed for each point cluster, and the hardness coefficient of each point cluster is calculated based on the local geometric tensor matrix. The specific method for determining the hard nodal region corresponding to the point cluster based on the hardness coefficient is as follows: For each point cluster, construct a local geometric tensor matrix. As shown below: in, This represents the number of points contained in the current cluster after spatial clustering. For the first point in the cluster The coordinate vector of a point, Let the point cluster be the centroid. It is the transpose symbol; For local geometric tensor matrix Perform eigenvalue decomposition to obtain eigenvalues ​​representing the extent of extension of the longest principal direction of the point cluster. , eigenvalues ​​representing the extent of extension of the secondary principal direction of a point cluster The eigenvalues ​​representing the extent of extension of the shortest principal direction of the point cluster. ; The formula for calculating the hardness coefficient is as follows: in, The hardness coefficient, , , These are the first-direction weights, the second-direction weights, and the third-direction weights, respectively. Hardness coefficient Clusters of points that are greater than or equal to the hardness coefficient threshold are identified as hard agglomerates.

[0009] Furthermore, in the multimodal recognition module, the specific method for calculating the physical resistance feature weight of the corresponding point cloud voxel based on the hard agglomerate material physical characteristics of each point cloud voxel in the point cloud voxel set is as follows: in, For the first The physical resistance feature weights of each point cloud voxel The hardness coefficient is preset based on the type of hard agglomerated material. These are the eigenvalues ​​representing the extent of extension along the longest principal direction of the point cluster. , eigenvalues ​​representing the extent of extension of the secondary principal direction of a point cluster The eigenvalues ​​representing the extent of extension of the shortest principal direction of the point cluster. , For the first Terrain height gradient along each main direction, It is an exponential function.

[0010] Furthermore, in the multimodal recognition module, the specific method for calculating the comprehensive cost value of each point cloud voxel in the point cloud voxel set based on the physical resistance feature weights is as follows: in, For the first The comprehensive value of a point cloud voxel For the first The comprehensive value of a point cloud voxel As the first balance factor, As the second balance factor, As the third balance factor, For point cloud voxels spatial coordinates, For the robot's spatial coordinates, for and The straight-line distance between them It is the corresponding point cloud voxel height The compensation function.

[0011] Furthermore, in the environmental reconstruction decision module, the specific method for calculating the set of points with the minimum comprehensive cost from the robot's current position to the hard nodule region among all points contained in the point cloud voxel set using a heuristic path search algorithm is as follows: A heuristic pathfinding algorithm is employed, with a cost function To optimize the objective and minimize the total cost, the path is solved. Cost function As shown below: in, For path Cost function For the first Time of the first A point cloud voxel, path It is the first Time of the first A point cloud voxel The set, For smoothness weights, For the first Time of the first A point cloud voxel instantaneous speed, That is the total time.

[0012] Furthermore, in the environmental reconfiguration decision module, the specific method for setting the agglomeration breaking driving parameter sequence based on the physical resistance characteristic weights of the hard agglomeration region is as follows: Based on the physical resistance characteristic weights of the corresponding point cloud voxels in the hard agglomerate region, a preset driving parameter sequence is generated through a dynamic mapping operator. The dynamic mapping operator is based on the physical resistance feature weights of point cloud voxels. A family of linear mapping functions for input variables; According to the preset drive parameter sequence Set the control vector for block breaking ,in, For a moment The angular velocity of the drill bit rotation, axial depth axial feed pressure, For a moment The robot's movement speed; in, It is the lower limit of the drill bit's rotational angular velocity. Upper limit of drill bit rotational angular velocity It is the acceleration coefficient of the drill bit's rotational angular velocity. It is a natural constant; in, It is the axial depth to which the drill bit penetrates the agglomerate. It is a proportionality coefficient. It is the expected preset torque. It is a damping adjustment factor. It is the torque collected in real time by the drill bit motor; Furthermore, in the reverse optimization gain module, based on the torque data during the process of breaking up hard agglomerates, a rule-based state identification algorithm using time-domain feature analysis is employed to determine the critical point at which the hard agglomerate transforms from a hard-constrained state to a granular state. The specific method is as follows: in, for The status indicator shows the breaking mechanism in various states: 2 indicates the breaking mechanism is under high load and requires intervention; 1 indicates the breaking mechanism is in progress and operating normally; and 0 indicates the breaking mechanism has completed breaking, meaning the hard agglomerate has been broken up. High load torque threshold The rate of change of torque. The peak torque within the time window. The threshold for the rate of change of torque. For logical NOT operator, The residual torque threshold, The torque variance within the time window. This is the torque variance threshold.

[0013] Furthermore, in the reverse optimization gain module, the environmental reverse action gain signal is calculated using a nonlinear function based on torque release efficiency. The specific method for correcting the physical resistance feature weights of the corresponding point cloud voxels based on the environmental reverse action gain signal is as follows: in, This is the environmental reverse-action gain signal. This is the first gain adjustment coefficient. This is the maximum torque during the breaking process. This is the second gain adjustment coefficient. To break through the drilling depth of the mechanism, Thickness of hard nodules; in, For the first Physical resistance feature weights corrected for each point cloud voxel To optimize the adjustment coefficient.

[0014] A method for dynamically reconstructing robot operation trajectories under large language model task interaction includes: A global coordinate system point cloud is constructed based on sensor data. Spatial clustering is performed on the global coordinate system point cloud to obtain multiple point clusters. For each point cluster, a local geometric tensor matrix is ​​constructed. The hardness coefficient of each point cluster is calculated based on the local geometric tensor matrix. The hardness nodule region corresponding to the point cluster is determined based on the hardness coefficient. The global coordinate system point cloud is divided into multiple point cloud voxels. A set of point cloud voxels related to the robot's current task instruction is selected from the global coordinate system point cloud through a cross-modal self-attention mechanism. The physical resistance feature weight of the corresponding point cloud voxel is calculated based on the physical characteristics of the hardness nodule material of each point cloud voxel in the set of point cloud voxels. The comprehensive cost value of each point cloud voxel in the set of point cloud voxels is calculated based on the physical resistance feature weight. A heuristic path search algorithm is used to calculate the set of points with the minimum comprehensive cost from the robot's current position to the hard nodule region among all points contained in the point cloud voxel set. This set is used as the robot's environment reconstruction path. The breaking parameter sequence is set according to the physical resistance characteristic weight of the hard nodule region. The robot is controlled to reach the hard nodule region according to the environment reconstruction path. The breaking mechanism of the robot is driven to break the hard nodule according to the breaking parameter sequence. Based on the torque data during the process of breaking up hard agglomerates, a rule-based state identification algorithm based on time-domain feature analysis is used to determine the critical point at which the hard agglomerates transform from a hard constrained state to a loose particle state. When the critical point is reached, the environmental reverse action gain signal is calculated using a nonlinear function based on torque release efficiency, and the physical resistance feature weights of the corresponding point cloud voxels are corrected based on the environmental reverse action gain signal. Based on the physical resistance feature weights corrected for the corresponding point cloud voxels, a heuristic path search algorithm is used to calculate the point cloud combination with the minimum comprehensive cost from the robot's current position to the target location among all point clouds contained in the point cloud voxel set, which is then used as the robot's path.

[0015] A computer program product includes a computer program / instruction that, when executed by a processor, implements the above-described method for dynamic reconstruction of robot operation trajectory under large language model task interaction.

[0016] The beneficial effects of this invention are as follows: 1. Unlike traditional trajectory planning methods that rely solely on initial static perception and cannot adapt to environmental changes after operation, this invention utilizes a reverse optimization gain module to convert torque and depth feedback during agglomeration breaking into real-time environmental reverse effect gain signals. This dynamically corrects the physical resistance feature weights of point cloud voxels, enabling real-time updates of the 3D semantic cost map. Based on the updated cost field, the topology trajectory optimization module automatically reconstructs the globally optimal operation trajectory, resolving the engineering pain points of traditional solutions such as lagging trajectory planning, repetitive detours, and low operation efficiency. This ensures that the robot's operation trajectory always conforms to the actual state of the grain warehouse environment, achieving true intelligent dynamic adaptation.

[0017] 2. The local geometric tensor features of the point cluster (eimetric values ​​representing the extent of extension along the longest principal direction of the point cluster) , eigenvalues ​​representing the extent of extension of the secondary principal direction of a point cluster The eigenvalues ​​representing the extent of extension of the shortest principal direction of the point cluster. ), Main direction terrain height gradient, material hardness coefficient Deep integration and customized calculation of physical resistance feature weights This technology enables precise differentiation and resistance quantification of hard clumps and loose grain surfaces, solving the problem of high misjudgment rates associated with traditional single-vision and geometric feature recognition. By integrating physical resistance, spatial distance, and height compensation into a comprehensive cost formula, a three-dimensional semantic cost map is constructed, providing accurate and realistic cost input for path planning and improving the accuracy of initial trajectory planning.

[0018] 3. It distinguishes between three operational stages in real time: high load requiring intervention, normal disintegration, and disintegration completed. It accurately captures the critical point of hard agglomerates transforming from a hard constrained state to a loose particle state, solving the pain point of traditional solutions that cannot determine the agglomerate disintegration status in real time. Based on the critical point triggering reverse optimization and trajectory reconstruction, it avoids mechanical damage and energy waste caused by excessive operation, while ensuring the thoroughness of agglomerate disintegration.

[0019] 4. Through a cross-modal self-attention mechanism, precise association between task instructions and the physical features of point cloud voxels is achieved, automatically filtering task-related voxel sets and dynamically adjusting the balance factor (first balance factor) of the overall cost value. Second balance factor The third balance factor It is adapted to the needs of different tasks (breaking up clumps, rapid inspection, grain turning, etc.). Attached Figure Description

[0020] Figure 1 A schematic diagram of the overall logical framework of the trajectory planning method provided in the embodiments of the present invention; Figure 2Example diagram of three-dimensional semantic cost map construction provided by the present invention; Figure 3 A schematic diagram of the cost map feature extraction process based on visual token filtering provided by the present invention; Figure 4 A schematic diagram illustrating the correlation analysis between the torque feedback feature vector and the environmental evolution state provided by this invention; Figure 5 This is a schematic diagram illustrating the effect of the inverse optimization gain operator provided by the present invention on the local cost function correction; Figure 6 A schematic diagram of the path iterative search algorithm with a bidirectional optimization mechanism provided by the present invention; Figure 7 A logical block diagram of the hardware operating environment for the trajectory planning and control system provided by this invention; Figure 8 The system block diagram of the robot operation trajectory dynamic reconstruction system under large language model task interaction provided by the present invention. Detailed Implementation

[0021] To make the technical problems, technical solutions, and beneficial effects to be solved by this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and are not intended to limit the scope of this application.

[0022] Definitions: IMU: Inertial Measurement Unit, is the core motion attitude sensor for a robot. It integrates a three-axis gyroscope, a three-axis accelerometer, and a three-axis magnetometer, and can collect the robot's angular velocity, linear acceleration, and geomagnetic direction data in real time.

[0023] A voxel is a regular cubic three-dimensional basic unit formed by dividing a point cloud space in a three-dimensional global coordinate system according to a preset three-dimensional size. It is the smallest discretized and structured unit of analysis and computation in three-dimensional space. Each voxel contains all the point cloud data within its spatial range.

[0024] RViz: short for ROS Visualization, is a standard open-source visualization tool in the Robot Operating System (ROS) ecosystem. It is a universal tool in the global robot research and development, simulation, and debugging fields, and is used in the research and development of almost all mobile robots, robotic arms, and special-purpose robots.

[0025] Example 1 As the core scenario for grain storage, grain silos are prone to forming hard clumps of varying hardness and distribution due to factors such as gravity, temperature, and humidity changes. These clumps become obstacles for robots performing tasks such as grain silo inspection and grain turning. Existing grain silo robots generally suffer from the following technical pain points: First, it is difficult to accurately identify hard clumping areas and quantify their physical resistance, making it impossible to accurately correlate task instructions with point cloud perception data; second, the operation trajectory planning is mostly based on fixed paths without dynamic adjustment based on clumping resistance characteristics, easily leading to poor path traversal and insufficient targeting of clumping removal; third, during the clumping removal process, it is impossible to determine the clumping's fragmentation state in real time, nor can it dynamically correct environmental resistance characteristics based on removal feedback, resulting in delayed trajectory reconstruction, low subsequent operation efficiency, and difficulty in achieving adaptive dynamic optimization of the robot's operation trajectory.

[0026] To address the aforementioned practical engineering problems, this embodiment uses a grain silo environment as a specific application scenario to detail the implementation of the robot operation trajectory dynamic reconstruction system under the large language model task interaction of this invention. This system can accurately identify, quantify, adaptively break up, and dynamically reconstruct the trajectory of hard lumps within the grain silo, adapting to the actual operational needs of grain silo robots. Its specific structure and workflow are as follows: refer to Figure 8 A dynamic reconstruction system for robot operation trajectory under large language model task interaction, comprising: The multimodal recognition module is used to construct a global coordinate system point cloud based on sensor data, perform spatial clustering on the global coordinate system point cloud to obtain multiple point clusters, construct a local geometric tensor matrix for each point cluster, calculate the hardness coefficient of each point cluster based on the local geometric tensor matrix, determine the hard nodule region corresponding to the point cluster based on the hardness coefficient, divide the global coordinate system point cloud into multiple point cloud voxels, and select a set of point cloud voxels related to the robot's current task instruction from the global coordinate system point cloud through a cross-modal self-attention mechanism. Based on the physical characteristics of the hard nodule material of each point cloud voxel in the set of point cloud voxels, calculate the physical resistance feature weight of the corresponding point cloud voxel, and calculate the comprehensive cost value of each point cloud voxel in the set of point cloud voxels based on the physical resistance feature weight. The environmental reconstruction decision module is used to calculate the set of points with the minimum comprehensive cost from the robot's current position to the hard nodule area among all points contained in the point cloud voxel set using a heuristic path search algorithm. This set of points serves as the robot's environmental reconstruction path. The module sets a breaking parameter sequence based on the physical resistance characteristic weight of the hard nodule area, controls the robot to reach the hard nodule area according to the environmental reconstruction path, and drives the robot's breaking mechanism to break the hard nodule according to the breaking parameter sequence. The reverse optimization gain module is used to determine the critical point of the hard nodule's transformation from a hard constraint state to a loose particle state based on the torque data during the process of breaking the hard nodule. When the critical point is reached, the environmental reverse action gain signal is calculated using a nonlinear function based on torque release efficiency, and the physical resistance feature weight of the corresponding point cloud voxel is corrected based on the environmental reverse action gain signal. The topology trajectory optimization module is used to calculate the combination of point clouds that minimizes the overall cost of the robot's journey from its current position to the target location among all point clouds contained in the point cloud voxel set, based on the physical resistance feature weights corrected for the corresponding point cloud voxels, using a heuristic path search algorithm. This combination serves as the robot's path.

[0027] In the above technical solution, a single point cluster, such as a point cloud of a hard agglomerate, is spatially distributed among several adjacent point cloud voxels. These voxels, due to containing the characteristics of hard agglomerate point clusters, are calculated to have a higher hardness coefficient and physical resistance feature weight, and are ultimately determined to be point cloud voxels corresponding to the hard agglomerate region. Conversely, point clusters containing only loose grain surface point clouds have their corresponding voxels calculated to have a low resistance feature weight, and are thus considered passable voxels. In this embodiment, the point cloud voxels are 0.05m × 0.05m × 0.05m in size.

[0028] The trajectory planning function in this embodiment relies on the collaborative support of the hardware system, and its hardware operating environment logic is as follows: Figure 7 As shown: the various hardware modules form a closed-loop interaction: the 3D mapping and voice input module collects sensor data from millimeter-wave radar and depth cameras, as well as human voice task instructions, and transmits them to the robot's built-in multimodal large language model and trajectory planning module; the robot's built-in multimodal large language model and trajectory planning module calls the knowledge and strategy support module, and outputs trajectory planning instructions and breaking execution parameters in combination with preset algorithms; the execution drive module receives instructions and controls the walking mechanism and breaking mechanism (specifically the drill bit in this embodiment) to complete the operation; the real-time motion control module and torque feedback closed-loop module collect drill bit torque data and robot motion state data, and feed them back to the robot's built-in multimodal large language model and trajectory planning module, providing real-time hardware data support for software algorithms (multimodal recognition, reverse optimization, etc.), and realizing a hardware-level closed loop of perception, planning, execution, and feedback.

[0029] The trajectory dynamic reconstruction logic of this invention can be achieved through... Figure 1 The overall framework can be understood intuitively: Figure 1The entire process of multimodal perception, cluster identification, initial path planning, cluster removal, torque feedback, resistance correction, and trajectory reconstruction is presented: the input layer consists of sensor data and task instructions; the core processing layer corresponds sequentially to the multimodal identification module (cluster identification and cost calculation), the environmental reconstruction decision module (initial path planning and cluster removal control), the reverse optimization gain module (torque feedback and resistance correction), and the topology trajectory optimization module (dynamic trajectory reconstruction); the output layer is the robot's final operating trajectory and execution parameters. Each link is progressive, forming a complete logical closed loop of perception, planning, execution, feedback, and optimization.

[0030] As a specific implementation method, the method for constructing a global coordinate system point cloud based on sensor data in the multimodal recognition module is as follows: In this embodiment, the sensors include a millimeter-wave radar and a depth camera. Based on IMU data, a complementary filtering algorithm is used to calculate the robot's roll angle, pitch angle, and yaw angle. For the ranging and angular positioning data of the millimeter-wave radar, an adaptive Kalman filter is performed based on a uniform acceleration motion model to obtain the robot's denoised position, velocity, and acceleration. The robot's roll angle, pitch angle, and yaw angle, along with the denoised position, velocity, and acceleration, are fused to obtain a six-degree-of-freedom pose matrix. Motion compensation is performed on the original point cloud of the millimeter-wave radar based on the six-degree-of-freedom pose matrix. The original point cloud of the depth camera and the motion-compensated point cloud of the millimeter-wave radar are fused to obtain a fused point cloud in the sensor's local coordinate system. Based on the robot's denoised position and the robot's roll angle, pitch angle, and yaw angle, the fused point cloud is transformed from the sensor's local coordinate system to the global coordinate system to obtain a global coordinate system point cloud.

[0031] The specific method for calculating the robot's roll angle, pitch angle, and yaw angle based on IMU data and using a complementary filtering algorithm is as follows: Set initial attitude angle , The initial roll angle (around the X-axis, tilted left and right). The initial pitch angle (around the Y-axis, pitch forward and backward). This is the initial yaw angle (around the Z-axis, heading). (Usually, the initial level...) = =0, (Initial heading).

[0032] At time k, the gyroscope angular velocity of the IMU is synchronously acquired. x-axis acceleration y-axis acceleration z-axis acceleration and magnetometer yaw angle .

[0033] The current attitude prediction value is calculated by integrating the gyroscope. , (This is the IMU sampling time interval).

[0034] Roll correction value Pitch correction value The yaw correction value is the yaw angle of the magnetometer. Construct the correction vector .

[0035] Calculate the attitude angle at the current moment , As a complementary weighting factor, this embodiment uses 0.98.

[0036] The specific method for obtaining the robot's denoised position, velocity, and acceleration by performing adaptive Kalman filtering based on a uniform acceleration motion model using the ranging and angular positioning data from millimeter-wave radar is as follows: Millimeter-wave radar operates in FMCW (Frequency Modulated Continuous Wave) mode, and its raw echo is analyzed by FFT (Fast Fourier Transform) to obtain the target range, velocity and angle.

[0037] The robot's state-space equations are constructed using a uniformly accelerated motion model, which is detailed below: in, For robots in The complete motion state at any given moment For robots in The complete motion state at any given moment , For the robot's position, For robot speed, Accelerate the robot For input control items, For robot control commands, This is a control matrix, used to map control commands to state changes. Noise during uniformly accelerated motion of the robot, To make the first The state vector at time t becomes the first The state transition matrix used to predict the state vector at time step [time]. , The sampling interval for the adaptive Kalman filter; The formula for updating the predicted covariance is as follows: in, For the first The posterior covariance at time t on the first The prior prediction covariance matrix of the state at time step. For the first The posterior covariance matrix at time t. It is the transpose of the state transition matrix. Let be the noise covariance matrix of the robot during uniformly accelerated motion. To achieve adaptive denoising, an observation noise covariance corrected for time k based on echo intensity E is introduced. The formula is shown below: Among them, it is directly used for Kalman gain calculation, replacing the fixed base noise. The pre-defined baseline observation noise covariance, It is an exponential function. For adjustment coefficients, The current millimeter-wave radar echo intensity. This represents the minimum echo intensity of millimeter-wave radar. This represents the maximum echo intensity of the millimeter-wave radar. based on and Calculate the Kalman gain to filter the edges of moving and variable obstacles, and obtain the robot's denoised position, velocity, and acceleration.

[0038] The specific method for motion compensation of the original point cloud of millimeter-wave radar based on the six-degree-of-freedom pose matrix is ​​as follows: First, a unified time reference moment for a single frame of point cloud is selected. Using the attitude angles (roll, pitch, and yaw) calculated by the IMU complementary filtering, the rotation matrix at the moment of acquisition for each millimeter-wave radar point is interpolated. Simultaneously, based on the velocity and acceleration output by the adaptive Kalman filter, the translational offset at the moment of acquisition for each point is calculated using a uniform acceleration motion model. The rotational distortion of the point cloud caused by attitude turbulence is offset by transposing the rotation matrix, and the point cloud trailing distortion caused by robot movement is eliminated by backtracking the translational difference. Coordinate correction and outlier removal are performed point-by-point for all original radar points. Finally, a millimeter-wave radar point cloud with eliminated motion distortion and preserved spatial structure is output, providing a reliable data foundation for subsequent accurate fusion with depth camera point clouds.

[0039] The specific method for fusing the original point cloud from the depth camera and the point cloud from the motion-compensated millimeter-wave radar is as follows: First, offline extrinsic parameter calibration is performed on the millimeter-wave radar and depth camera to obtain a fixed rotation and translation relationship between their coordinate systems. Then, time synchronization of the point clouds of the two is achieved based on high-precision timestamps to ensure that the data acquisition time is aligned. Denoising preprocessing is performed on the point clouds of both to remove invalid noise and outliers. The depth camera coordinate system is selected as the local coordinate system of the sensor. Based on the calibration extrinsic parameters, the point cloud of the motion-compensated millimeter-wave radar is uniformly transformed to the depth camera coordinate system to achieve spatial registration. In the near-range area, the depth camera point cloud is used as the main component, preserving the edges of clumps and the fine contours of grain surfaces. In the far-range or dust-covered areas, the point cloud of the motion-compensated millimeter-wave radar is used to fill holes and supplement distant targets. Finally, voxel downsampling is performed on the superimposed hybrid point cloud to remove duplicate and redundant points and filter out sporadic noise generated after fusion, generating a fused point cloud in the local coordinate system that balances density, integrity, and robustness.

[0040] The specific method for transforming the fused point cloud from the sensor's local coordinate system to the global coordinate system is as follows: First, a 3×3 rotation matrix R (in the order of Z, Y, X, i.e., yaw, pitch, roll) is calculated from the attitude angles. This matrix is ​​then used to spatially rotate the fused point cloud to align it with the global gravity coordinate system. Finally, the robot's denoised position is used... Using this as a translation reference, spatial translation is completed. Through this transformation process, global coordinate unification of multi-source sensing point clouds is achieved. in, It is the yaw rotation matrix. It is a pitch and rotation matrix. It is a roll rotation matrix.

[0041] in, These are the coordinates of any point in the global coordinate system. These are the coordinates of any point in the sensor's local coordinate system.

[0042] This gives us the global coordinate system of the grain warehouse (X and Y are the horizontal axes within the grain surface, and Z is the vertical axis perpendicular to the grain surface).

[0043] As a specific implementation method, the specific method for spatial clustering of point clouds in a global coordinate system is as follows: A typical spatial clustering algorithm, Euclidean clustering, is employed. First, a KD-tree spatial index structure is constructed to accelerate the lookup of neighboring points. Then, a distance threshold is set, and each point in the point cloud is traversed, with connected points whose spatial distance is less than the distance threshold grouped into the same local point cluster. Finally, a point count threshold is set to filter the clustering results, removing tiny isolated clusters composed of dust clutter and retaining the set of local point clusters containing effective geometric structures.

[0044] As a specific implementation method, in the multimodal recognition module, a local geometric tensor matrix is ​​constructed for each point cluster, the hardness coefficient of each point cluster is calculated based on the local geometric tensor matrix, and the hardness coefficient is used to determine the hard nodal region corresponding to the point cluster. The specific method is as follows: For each point cluster, construct a local geometric tensor matrix. As shown below: in, This represents the number of points contained in the current cluster after spatial clustering. For the first point in the cluster The coordinate vector of a point, Let the point cluster be the centroid. It is the transpose symbol; For local geometric tensor matrix Perform eigenvalue decomposition to obtain eigenvalues ​​representing the extent of extension of the longest principal direction of the point cluster. , eigenvalues ​​representing the extent of extension of the secondary principal direction of a point cluster The eigenvalues ​​representing the extent of extension of the shortest principal direction of the point cluster. Using the global coordinate system of the grain warehouse (X and Y are the horizontal axes within the grain surface, and Z is the vertical axis perpendicular to the grain surface), for a two-dimensional planar point cluster on a loose grain surface, , Corresponding to the two main horizontal extension directions within the grain surface. Corresponding to the vertical Z-axis direction perpendicular to the grain surface ( ≈0); for the three-dimensional cluster of protruding points in hard nodules, , The corresponding horizontal main extension direction of the cluster. Corresponding to the vertical Z-axis direction perpendicular to the grain surface ( >0 indicates the degree of vertical protrusion of the agglomerate); among the three principal directions, the shortest principal direction is... To distinguish the core features of two-dimensional loose grain surfaces from three-dimensional hard lumps, and to provide core support for subsequent hardness coefficient calculation and lump identification.

[0045] For local geometric tensor matrix The specific method for performing eigenvalue decomposition is as follows: Local geometric tensor matrix Let be the 3×3 real symmetric positive semidefinite covariance matrix corresponding to the point cluster, satisfying the orthogonal diagonalization condition for real symmetric matrices. For the local geometric tensor matrix... Eigenvalues ​​are calculated to obtain all eigenvalues; simultaneously, the corresponding eigenvectors are solved, and these eigenvectors are normalized to obtain pairwise orthogonal unit eigenvector sets. The three eigenvalues ​​obtained are sorted in descending order of value, corresponding to the longest, second longest, and shortest principal directions of the point cluster, respectively. The magnitude of the eigenvalue represents the spatial extension of the point cluster along the corresponding principal direction, while the magnitude of the eigenvalue represents the discreteness of the point cloud along the corresponding principal direction. Due to the local geometric tensor matrix... Let be a 3rd order real symmetric matrix in 3-dimensional space. According to the linear algebra spectrum theorem, eigenvalue decomposition will necessarily yield 3 eigenvalues ​​and eigenvectors corresponding to the orthogonal principal directions.

[0046] The formula for calculating the hardness coefficient is as follows: in, The hardness coefficient, , , These are the first-direction weights, the second-direction weights, and the third-direction weights, respectively. In this embodiment... , , The values ​​are 1, 2, and 3 respectively.

[0047] , representing the longest principal direction With the secondary main direction The extended difference, divided by Normalization is performed to completely eliminate the influence of the overall scale of point clusters. For loose grain surfaces, which are almost two-dimensional planes, ≈ , ≈0; hard nodules, which are three-dimensional protrusions. > , >0 indicates a contribution to basic geometric features. First-direction weights. The value is 1, which serves as the basic term, corresponding to the difference between the longest and second longest principal directions.

[0048] Characterizing the secondary principal direction With the shortest main direction The differences in extension are also normalized. For loose grain flour, ≈0, ≈ , ≈ For hard lumps, >0, < , <2, providing geometric features for intermediate levels. Second direction weights. The value is 2, which is higher than the weight in the first direction. The difference between the second longest principal direction and the shortest principal direction is the key to distinguishing between two-dimensional planes and three-dimensional structures: the plane's... Extremely small, with extremely large differences; three-dimensional clusters The difference is relatively large, but the difference is small.

[0049] Characterized by the shortest principal direction The relative extension of the point cluster relative to the longest principal direction directly reflects its spatial proportion along the shortest principal direction. For loose grain surfaces, ≈0, ≈0, for hardness coefficient Almost no contribution; for hard lumps, >0, >0, and The larger the value (the more fully the shortest principal direction is extended), the greater the contribution, directly increasing the hardness coefficient. Values ​​accurately identify 3D protruding clumps. Third-direction weights. The value is 3, because it is the shortest main direction. It is to distinguish two-dimensional planes ( ≈0) and three-dimensional structure ( The core feature of >0 is to strengthen its influence through high weighting, so as to completely avoid misjudging the two-dimensional plane as a block.

[0050] Clusters of points with a hardness coefficient greater than or equal to the hardness coefficient threshold are classified as hard agglomerates.

[0051] As a specific implementation method, the method for calculating the physical resistance characteristic weight of the corresponding point cloud voxel based on the hard agglomerate material physical characteristics of each point cloud voxel in the point cloud voxel set is as follows: in, For the first The physical resistance feature weights of each point cloud voxel The hardness coefficient is preset based on the type of hard agglomerated material. These are the eigenvalues ​​representing the extent of extension along the longest principal direction of the point cluster. , eigenvalues ​​representing the extent of extension of the secondary principal direction of a point cluster The eigenvalues ​​representing the extent of extension of the shortest principal direction of the point cluster. , For the first Terrain height gradient along each main direction, It is an exponential function.

[0052] Physical resistance features are a comprehensive indicator that characterizes the resistance exerted by materials (such as hard clumps of grain or loose grain surfaces in a grain silo scenario) on the robot's movement and breaking-up actions during operation. The core of this feature is the fusion of the inherent physical properties, spatial geometry, and terrain undulations of the hard clumps of material. Ultimately, this is determined by the weighting of physical resistance features. Achieving quantification is the core bridge connecting environmental perception with robot path planning and breaking down operational decision-making barriers.

[0053] yes The normalized weights normalize the feature values ​​of the three main directions to obtain the weight ratio of each direction. This allows the main direction with greater extension (the direction of the bulge) to contribute more to the resistance, which is consistent with physical intuition: the more bulging the bulge, the greater the resistance. This avoids misjudgment of features in a single direction, realizes the weighted quantification of geometric features, and accurately reflects the influence of the geometric shape of the point cluster on the resistance.

[0054] It is a nonlinear mapping of terrain gradients. For the height gradient in each principal direction, it performs a squared + logarithmic transformation. The squared transformation is used to amplify the influence of the gradient and highlight the blocky areas with obvious undulations. The logarithmic transformation is used to linearly amplify small gradients (slight blockages) and nonlinearly compress large gradients (extreme convexities). It takes into account the recognition of undulations across the entire range and avoids weight anomalies caused by extreme values. It solves the problem of "missing small undulations and overloading large undulations" in traditional gradient features and improves the robustness of terrain features.

[0055] It is a weighted fusion of the geometric features of point clusters and terrain features, which combines the geometric weights of each principal direction. Proportion and Topographic Gradient Features Multiplication followed by summation achieves a deep fusion of geometric shape and terrain undulation. This solves the misjudgment problem of single feature recognition: If only the geometric extension is large but the terrain is flat (non-clustered): the comprehensive feature is low, and it will not be misjudged as high resistance; if only the terrain undulation is large but the geometric extension is small (non-clustered): the comprehensive feature is low, and it will not be misjudged as high resistance; only when both geometry and terrain conform to clustering characteristics will a high comprehensive feature be obtained, significantly improving the accuracy of clustering recognition. By fusing the geometric features of the main direction of the point cluster with the corresponding terrain gradient, rather than the global gradient, it accurately matches the local shape of the cluster, improving the accuracy of resistance quantification.

[0056] The integrated features after fusion (i.e. )Do Exponential transformation maps the characteristics of the linear interval to the nonlinear interval, significantly widening the characteristic difference between high resistance (caking) and low resistance (grain surface), making the resistance distinction between caking and grain surface more obvious, facilitating subsequent threshold segmentation and caking identification, while ensuring... It is always positive, which conforms to the definition of physical resistance (resistance cannot be negative).

[0057] Multiply by the material hardness coefficient , It calibrates the resistance benchmark for different grain types by identifying specific grain types based on preset values ​​for grain varieties. Pre-built grain varieties and material hardness coefficients are used. The parameter library, targeting common grain varieties in grain warehouses, calibrates the material hardness coefficient through offline agglomeration resistance tests. A one-to-one parameter repository is established; administrators input natural language task instructions with grain varieties, and a multimodal large language model performs cross-modal parsing of the instructions to extract grain variety information and automatically retrieve the corresponding material hardness coefficient from the parameter repository. When there are no relevant task instructions, the robot reuses multimodal sensors such as millimeter-wave radar and depth cameras to extract features such as dielectric properties and visual texture of the grain. After identifying the grain variety through feature fusion, it matches the benchmark μ value from the parameter library.

[0058] As a specific implementation method, the multimodal recognition module uses a cross-modal self-attention mechanism to filter out a set of point cloud voxels related to the robot's current task instruction from the global coordinate system point cloud. The global coordinate system point cloud is the original coordinate data. It is necessary to extract the structured physical features that can be understood by the cross-membrane large-scale model, i.e., visual tokens, which specifically include: (1) geometric structure features: 3D structural information extracted from the point cluster, including: the height, normal vector, and curvature of the point cloud; (2) the local geometric tensor matrix H of the point cluster, , , Hardness coefficient (3) Spatial location, size, and shape of the point cluster; (4) Millimeter-wave radar reflection intensity: the RCS (radar cross-section) value of the millimeter-wave radar point cloud. The reflection intensity of the hard structure of the agglomerate is much higher than that of the loose grain surface, which is the key physical feature for distinguishing agglomerates; (5) Semantic label features: label the point clusters with semantic labels such as "normal grain surface", "hard agglomerate", "warehouse wall", and "obstacle", and transform the geometric features into semantic features that can be understood by the large model. The multi-dimensional features extracted above are the specific contents of "grain surface undulation", "hard agglomerate", and "environmental structure" in the visual token, which are calculated entirely from the physical properties of the point cloud. The extracted multi-dimensional physical features need to be converted into visual tokens that can be processed by the cross-modal large model built into the robot. Only the format is converted, and the feature content is not modified: each point cluster is used as a basic unit. The multi-dimensional features of each point cluster are encoded into a high-dimensional feature vector of fixed length. The feature vectors of all point clusters are arranged in spatial order to form a visual token sequence.

[0059] The current task instruction is calculated using the cross-modal self-attention mechanism of the robot's built-in multimodal large language model. Correlation weight matrix with point cloud voxel physical features The formula is as follows: in, This is a sequence of voxel visual tokens extracted from a point cloud in a global coordinate system. Each voxel visual token in the sequence contains the physical features of the corresponding voxel. It is a query matrix. It is a key matrix. It is a scaling factor. It is a normalized exponential function; The correlation weight matrix Visual tokens for each point cloud voxel in the visual token sequence Association weight The associated weights All point cloud voxels that are greater than or equal to the association weight threshold are merged into a set of point cloud voxels that are related to the robot's current task instruction.

[0060] The cross-modal large model can employ point cloud-text cross-modal pre-trained models such as PointCLIP (Point Cloud Visual Language Contrast Pre-trained Model) and Point-BERT (Point Cloud Masking Point Modeling Pre-trained Transformer Model). In this embodiment, the robot has Point-BERT built-in. Relying on Point-BERT's built-in cross-modal self-attention operator, it completes the global association calculation between the task instruction text and the visual features of point cloud voxels, outputs the required association weight matrix, and realizes intelligent selection of target voxels.

[0061] The training process of Point-BERT is as follows: The collected global point cloud data of the grain warehouse operation scene and the corresponding natural language task instruction text are preprocessed and cross-modal feature alignment is performed to obtain a cross-modal feature alignment dataset composed of point cloud voxel-task instruction pairs. This cross-modal feature alignment dataset is used as the training data for the Point-BERT model. In each point cloud voxel-task instruction pair, the true value of the association weight between the point cloud voxel and the task instruction is calculated by cross-modal feature similarity calibration algorithm after the task-related target voxels are labeled by professional annotators in the fields of robot path planning and computer vision. A training set and a validation set are then divided from the training data of the Point-BERT model.

[0062] Point cloud data preprocessing includes point cloud denoising, voxelization, local geometric tensor feature extraction, and random mask generation conforming to the Point-BERT pre-training mechanism; text data preprocessing includes task instruction word segmentation, tokenization encoding, and text semantic embedding vector extraction; cross-modal feature alignment maps point cloud voxel visual features and text semantic features to the same feature space, completing bidirectional alignment of spatial dimension and feature dimension.

[0063] Using the training set as input, Bayesian optimization is employed to optimize the hyperparameters of the Point-BERT model. The optimization objective is to minimize the mean squared error of the association weight prediction on the validation set, thereby obtaining the optimal combination of hyperparameters for the Point-BERT model. The hyperparameters to be optimized include point cloud masking rate, number of Transformer encoder layers, number of self-attention heads, feature embedding dimension, batch size, initial learning rate, and weight decay coefficient.

[0064] During iterative training, in forward propagation, the Point-BERT model takes the training set as input and extracts local geometric features and global scene semantic features based on the masked point cloud reconstruction task. Simultaneously, it fits and outputs the predicted association weights of each point cloud voxel with the current task instruction. In backpropagation, the error backpropagation algorithm iteratively calculates the gradient of the Point-BERT model parameters, and the Adam optimizer updates the Point-BERT model parameters. The sum of mean squared error and L2 regularization is used as the first loss function, which measures the difference between the predicted association weights output by the Point-BERT model based on the training set in the current iteration and the true values ​​of the association weights. The point cloud mask reconstruction loss is also incorporated as an auxiliary constraint. With the goal of minimizing the first loss function, the Point-BERT model is trained using cross-modal feature alignment and mask reconstruction dual-task simultaneous learning based on the training set. After updating the Point-BERT model parameters on the training set in each iteration, the loss is calculated on the validation set. The iteration terminates when the preset number of training rounds is reached.

[0065] As a specific implementation method, the method for calculating the comprehensive cost value of each point cloud voxel in the point cloud voxel set based on the physical resistance feature weights in the multimodal recognition module is as follows: in, For the first The comprehensive value of a point cloud voxel For the first The comprehensive value of a point cloud voxel As the first balance factor, As the second balance factor, As the third balance factor, For point cloud voxels spatial coordinates, For the robot's spatial coordinates, for and The straight-line distance between them It is the corresponding point cloud voxel height The compensation function, It is a compensation coefficient, which increases with altitude. It grows in a quadratic manner.

[0066] Using the initial physical resistance characteristic weights of voxels As the core input, multiplied by The quantified resistance of the voxel is the dominant factor in the overall cost value. Integrating the customized resistance quantification index of this invention into the cost value achieves a deep binding between physical resistance and path, solving the engineering pain point of existing technologies that only use distance to plan paths and cannot adapt to clogged scenarios. This is the core innovation of this formula.

[0067] Starting from the robot's current position, calculate the straight-line distance to the voxel, and multiply it by... The penalty path length is used to prevent the robot from taking extremely long routes to avoid clumps, ensuring path rationality and operational efficiency. The influence of resistance is balanced to achieve an optimal balance between minimum resistance and shortest distance, avoiding unnecessary detours.

[0068] For voxel vertical height Through the compensation function Quantify the impact of terrain undulation on traffic.

[0069] In summary, the construction method of the 3D semantic cost map is referenced. Figure 3 The process involves: generating a global coordinate system point cloud through multimodal sensor fusion; obtaining point clusters through spatial clustering and calculating hardness coefficients to identify hard agglomerates; dividing the global point cloud into standardized point cloud voxels; and selecting task-relevant voxel sets through a cross-modal self-attention mechanism. Based on the voxel's geometric features, terrain features, and material properties, the physical resistance feature weights are calculated. Then, by integrating parameters such as voxel spatial distance and height compensation, a comprehensive cost value is obtained. Finally, using point cloud voxels as the smallest unit, a discretized 3D semantic cost map containing semantic attributes, physical resistance, and passage costs is formed. Based on this, to address the issues of cost gaps between voxels in the discrete cost map and its inability to adapt to non-uniform grain surfaces, this invention employs a three-step method: voxel cost interpolation with physical resistance feature weights, continuous surface fitting of the non-uniform grain surface, and gradient vectorization of the cost field. This method reconstructs the non-uniform plane (non-uniform undulating grain surface) into a continuous, computable vector field, achieving the transformation from a discrete cost map to a continuous vector field. For any location in global space, a fusion method is used... The weighted inverse distance interpolation method, based on neighborhood voxels The continuous comprehensive cost value is calculated by extending the discrete voxel cost value to a global continuous scalar field, while enhancing the cost value contribution of high-resistance clusters; the moving least squares method is used to perform surface fitting on the global point cloud, with the shortest principal direction of the point cluster ( Using the normal constraint, a globally continuous parameterized surface of the grain surface is generated, providing a bearing plane for the vector field. Three-dimensional gradient calculations are performed on the continuous cost scalar field, and the gradient vector is projected onto the tangent plane of the grain surface, forming a continuous computable vector field within the grain surface. This provides directional guidance and continuous cost input for subsequent heuristic path search algorithm planning. An example diagram of the three-dimensional semantic cost map is shown below. Figure 2 As shown.

[0070] As a specific implementation method, in the environment reconstruction decision module, the specific method for calculating the set of points with the minimum comprehensive cost from the robot's current position to the hard nodule region among all points contained in the point cloud voxel set using a heuristic path search algorithm is as follows: Employing heuristic pathfinding algorithms (such as the A* algorithm), with a cost function To optimize the objective and minimize the total cost, the path is solved. Cost function As shown below: in, For path Cost function For the first Time of the first A point cloud voxel, path It is the first Time of the first A point cloud voxel The set, For smoothness weights, For the first Time of the first A point cloud voxel instantaneous speed, That is the total time.

[0071] In this embodiment, the heuristic pathfinding algorithm uses the A* algorithm, and the cost evaluation formula for the A* algorithm is as follows: , It is a search node, which in this embodiment represents a single point cloud voxel. It is the total assessment cost. The actual cost has already been paid. It's a heuristic cost estimation, using the A* algorithm. Give the cost function Scoring, finding the cost function The shortest path.

[0072] The total resistance cost is the core optimization objective. Integrating the continuous comprehensive cost over all positions along the entire trajectory yields the total physical resistance cost of the trajectory, characterizing the total difficulty for the robot to traverse the path. If the path passes through a hard, agglomerated region, If the path is too high, the total cost increases significantly, and the algorithm will automatically avoid that path; if the path follows a loose grain surface, The path with the lowest total cost is the one the algorithm prioritizes. On a continuous cost vector field, the resistance of discrete point cloud voxels is quantized and expanded into the total cost of the continuous trajectory, ensuring the global optimality of the path.

[0073] This is a trajectory smoothness penalty term. It is calculated by integrating the square of the instantaneous velocity magnitude over the entire trajectory, penalizing the intensity of the trajectory's motion (rapid acceleration, sudden braking, sharp turns, and zigzag bends). This is a smoothness weight used to adjust the penalty intensity. Traditional discrete A* paths are polygonal lines with large speed abrupt changes. During robot execution, frequent starts, stops, and turns are required, easily leading to chassis slippage and trajectory deviation. This trajectory smoothness penalty term forces trajectory smoothness by penalizing the square of the speed, allowing the robot to move along a continuous, gentle curve, adapting to the actual kinematic requirements of grain silo robots. It completely solves the engineering pain point of non-smooth discrete A* paths, eliminating the need for additional post-processing smoothing and directly generating executable continuous trajectories while ensuring path optimization.

[0074] Smoothness weight This is the core parameter that balances the two sub-items mentioned above, enabling task adaptation. For example, when the task is rapid inspection or breaking up clumps, the smoothness weight... Prioritize minimizing total resistance cost by selecting the smallest value, and ensure the path follows low-resistance areas to guarantee operational efficiency; when the task is smooth inspection or grain silo turning, the smoothness weight is adjusted. Choose the larger value to ensure smooth trajectory, avoid frequent robot starts and stops, and improve operational stability.

[0075] As a specific implementation method, the method for setting the agglomeration breaking driving parameter sequence based on the physical resistance characteristic weights of the hard agglomeration region in the environmental reconfiguration decision module is as follows: Set the control vector for block breaking ,in, For a moment The angular velocity of the drill bit rotation, axial depth axial feed pressure, For a moment The robot's movement speed; in, It is the lower limit of the drill bit's rotational angular velocity. Upper limit of drill bit rotational angular velocity It is the acceleration coefficient of the drill bit's rotational angular velocity. It is a natural constant; in, It is the axial depth to which the drill bit penetrates the agglomerate. It is a proportionality coefficient. It is the expected preset torque. It is a damping adjustment factor. It is the torque collected in real time by the drill bit motor.

[0076] in, The robot's maximum forward speed on a flat, unobstructed surface. This is the attenuation coefficient, used to control the sensitivity of speed as resistance increases.

[0077] Based on the physical resistance characteristic weights of the corresponding point cloud voxels in the hard agglomerate region. A preset sequence of driving parameters is generated through a dynamic mapping operator. The dynamic mapping operator uses the physical drag characteristic weights of point cloud voxels. A family of linear mapping functions for core input variables, whose mathematical definition is This operator converts the environmental stiffness characteristics into the robot's dynamic control boundaries in real time, specifically mapping and generating the following preset drive parameter sequence: Rotation speed range mapping: The operator dynamically adjusts the intensity range of the drill bit's operation, and the operator linearly increases the lower limit of the rotation speed. When the physical resistance characteristic weight When the speed is increased, the mapping operator linearly increases the upper limit of the rotational speed. To provide higher cutting speeds to break hard constraints, and The calculation formula is as follows: Lower limit of drill bit rotational angular velocity To ensure the drill bit has a minimum cutting capacity under this resistance, the upper limit of the drill bit's rotational angular velocity is... To prevent motor overload or excessive material crushing due to excessive rotation speed, and These are the preset lower and upper limits of the basic rotational speed, representing the standard operating speed of the robot under no-load or extremely soft material conditions, respectively. The actual bulk density of the material in the area to be worked on is given by sensor feedback or preset parameters. It serves as a standard reference density for materials, acting as a normalization benchmark to eliminate differences in the fundamental physical properties of different types of materials. , These are the first gain adjustment coefficient and the second gain adjustment coefficient, used to balance the physical resistance characteristic weights. The sensitivity to the effect of rotational speed is set according to the actual engineering test.

[0078] Desired Torque Mapping: Establishing a positive correlation between the desired preset torque and the weights of physical resistance characteristics, i.e. .in To enhance the coefficient, Based on the base idling torque, ensure that the harder the clump, the greater the expected preset torque, and the greater the breaking power.

[0079] As a specific implementation method, in the reverse optimization gain module, based on the torque data during the process of breaking up hard agglomerates, a rule-based state identification algorithm using time-domain feature analysis is employed to determine the critical point of the transition from a hard constrained state to a granular state of the hard agglomerate. in, for The status indicator shows the breaking mechanism in various states: 2 indicates the breaking mechanism is under high load and requires intervention; 1 indicates the breaking mechanism is in progress and operating normally; and 0 indicates the breaking mechanism has completed breaking, meaning the hard agglomerate has been broken up. High load torque threshold The rate of change of torque. The peak torque within the time window. The threshold for the rate of change of torque. For logical NOT operator, The residual torque threshold, The torque variance within the time window. This is the torque variance threshold.

[0080] for , This indicates that the real-time torque is much higher than the high-load torque threshold, suggesting that the current agglomerate hardness is far greater than expected. This indicates a very small rate of torque change, suggesting a stable high-load state rather than instantaneous fluctuations. When encountering extremely hard clumps (such as ultra-hard clumps formed over long-term storage), the drill bit continuously experiences high resistance. The robot recognizes this as a high load and automatically triggers intervention, such as increasing feed pressure, adjusting rotational speed, or performing emergency avoidance maneuvers to prevent damage to the mechanism.

[0081] for , This indicates that the current torque is continuously decreasing, which means that the clumps are being broken up and the resistance is gradually decreasing; The continuous decrease in peak torque within the time window indicates that the maximum resistance of the agglomeration is also continuously decreasing, and the agglomeration as a whole is in the process of breaking up. This indicates that high-load conditions are excluded to ensure mutual exclusion of states. The robot normally breaks up the clumps, the torque continues to decrease, the clumps gradually break up, the state machine outputs 1, and controls the breaking mechanism to continue working according to the current parameters.

[0082] for , This indicates that the current torque is less than the residual torque threshold, meaning that the agglomeration resistance has been reduced to the baseline level of the loose grain surface, and the agglomerates have been broken up. If the torque variance is less than the threshold within the time window, it indicates that the torque fluctuation is minimal and the material has been completely converted into a bulk particle state. and This indicates the exclusion of high load and breaking states, ensuring state mutual exclusion. The agglomerates have been completely broken up, reaching the critical point where the hard constraint becomes a loose particle. The state machine outputs 0, triggering the reverse optimization gain module: calculating the environmental reverse action gain signal based on torque release efficiency, and correcting the physical resistance characteristic weights of the corresponding voxels. This provides input for subsequent dynamic trajectory reconstruction.

[0083] By binding torque feedback with state determination, a closed-loop process of operation, feedback, correction, and reconstruction is achieved, solving the pain point of lagging trajectory reconstruction in existing technologies. Layered priority and mutual exclusion logic avoid misjudgment and omission of state, adapting to the complex clumping scenario of grain warehouses. Based on rule-based state recognition, no complex model is required, and the real-time performance is high, adapting to the real-time operation requirements of robots.

[0084] The effectiveness of the three-state operation state machine and its environmental evolution effect can be assessed through... Figure 4 verify: Figure 4The upper part shows the RViz visualization results: the left image "RViz static (constraints persist)" corresponds to the state before the operation, and the red hot area is the high-resistance clump that has not been broken (constraints persist); the right image "RViz evolution (obstacle reconstruction)" corresponds to the state after the operation, and the red clump area is transformed into the green low-resistance safe area (obstacles are broken up and rebuilt).

[0085] Figure 4 The lower half shows the torque comparison curves: the left figure "no evolving torque" represents the effect of not enabling the reverse optimization algorithm of this invention, with the torque maintaining a high load of about 120 N·m for a long time; the right figure "with evolving torque (empowered)" represents the torque dropping sharply to a low load of 30 N·m in about 1 second (critical point, corresponding to State=0) after enabling the algorithm, which is completely synchronized with the environmental evolution state of RViz, intuitively verifying the effectiveness of the correlation between torque feedback and environmental state, as well as the accurate capture of the agglomeration breaking critical point by the three-state machine.

[0086] To ensure the robot can cut into the block in the optimal posture, an endpoint constraint is adopted: The trajectory's endpoint tangent vector is required to align with the normal vector of the agglomerate surface, thus creating the physical conditions for the vertical drilling of the breaking mechanism. Among these, Let be the unit normal vector of the agglomerate surface. It is the drill bit tangential vector extracted from the terminal state and normalized. It is the trajectory terminal state vector, the trajectory at the terminal moment. The complete state vector (i.e., the moment when the robot reaches the block cutting position) contains two core components: the spatial coordinates of the trajectory endpoint (block cutting position); and the tangential motion vector of the trajectory at the endpoint (i.e., the instantaneous cutting direction of the drill bit). It is the attitude deviation constraint function, based on the terminal state. The input attitude deviation quantization function's core function is to calculate the degree of deviation between the trajectory endpoint tangent vector and the nodule surface normal vector: the greater the deviation, The larger the value, the smaller the deviation (0). =0, corresponding to the tangent vector and normal vector being completely parallel. A forced attitude deviation of 0, meaning the trajectory endpoint tangent vector is completely consistent with (parallel to) the block surface normal vector, is the condition for the constraint to take effect. The block surface normal vector is the shortest principal direction obtained by the eigenvalue decomposition of the local geometric tensor matrix of the point cluster. The corresponding eigenvector, which is perpendicular to the surface of the agglomerate (i.e., the vertical protrusion direction perpendicular to the grain surface), is the inherent geometric normal of the agglomerate.

[0087] As a specific implementation method, in the reverse optimization gain module, the environmental reverse action gain signal is calculated using a nonlinear function based on torque release efficiency. The specific method for correcting the physical resistance feature weights of the corresponding point cloud voxels based on the environmental reverse action gain signal is as follows: in, This is the environmental reverse-action gain signal. This is the first gain adjustment coefficient. This is the maximum torque during the breaking process. This is the second gain adjustment coefficient. To break through the drilling depth of the mechanism, Thickness of hard nodules; This represents the torque release efficiency of agglomerates based on the relative decrease ratio of maximum torque to residual torque, characterizing the degree of transformation of agglomerates from hard to loose. Logarithmic transformation optimizes feature discrimination and avoids numerical anomalies. The harder the agglomerate and the more thoroughly it is broken up, the greater the torque decrease ratio, and the higher this component value. The larger.

[0088] This represents the percentage of drilling depth relative to the total thickness of the agglomerate, quantifying the progress of physical agglomerate removal and characterizing the degree to which the agglomerate is physically broken up. The deeper the drill bit penetrates and the higher the percentage, the higher this value. The larger.

[0089] pass , Two adjustable coefficients balance the contributions of torque and depth characteristics: if only torque decreases but penetration is not achieved ( (small), then Low torque will not be mistaken for complete breakage; if only the drill bit penetrates but the torque does not decrease ( (small), then Low penetration will not be mistaken for complete destruction; high penetration will only be achieved when the torque is fully released and the drilling depth is sufficient. It accurately quantifies the degree of block breaking and avoids misjudgment based on a single feature.

[0090] in, For the first Physical resistance feature weights corrected for each point cloud voxel To optimize the adjustment coefficient.

[0091] by As input, through the exponential function , will be higher Corrected to a lower . Used for nonlinear amplification correction, significantly increasing the open junction region (high). ) and non-caking areas (low The resistance difference is such that the weight is always positive, which conforms to the definition of physical resistance. The larger the exponent (the more thoroughly the clumping is broken), the more effective the exponent term. The smaller, The lower the value, the better it aligns with the physical intuition that resistance decreases after agglomerates are broken up. The torque / depth data from the task feedback is converted into resistance feature corrections for point cloud voxels, enabling dynamic updates of the discrete point cloud from "initial perception" to "actual state after the task," providing precise input for subsequent dynamic trajectory reconstruction. Through the above formula, a deep integration of agglomerate breaking task feedback and point cloud resistance quantification is achieved: dual-feature fusion accurately quantifies the degree of breaking up, avoiding misjudgments and omissions due to single features, improving the accuracy of agglomerate state determination; exponential nonlinear correction widens the resistance feature gap, making the corrected physical resistance feature weights more closely match the actual task state, improving path planning accuracy; and a closed-loop end-to-end system enables dynamic trajectory reconstruction, solving the engineering pain points of lagging trajectory planning and inability to adapt to post-task environmental changes in existing technologies, significantly improving the robot's operational efficiency and intelligence level.

[0092] The effect of the inverse optimization gain operator on the cost map correction is as follows: Figure 5 As shown: Figure 5 The "Initial Cost Map (Before Empowerment)" on the left corresponds to the 3D semantic cost map before correction. The original clumped areas (dark areas in the image) exhibit high costs (corresponding to physical resistance feature weights). The "Updated Cost Map (After Empowerment)" on the right corresponds to the revised cost map. The cost of the original clumped areas has been significantly reduced (light-colored areas in the map), corresponding to the revised physical resistance feature weights. As can be seen from the comparison of the two figures, the reverse optimization operator, through the environmental reverse action gain signal Δη, successfully reduced the cost of the high-resistance agglomeration area to a level close to that of the loose grain surface, forming a "logic hole" (high traffic priority area), which provides accurate cost input for subsequent trajectory reconstruction and solves the pain point that traditional cost maps cannot adapt to environmental changes after operation.

[0093] As a specific implementation method, the topology trajectory optimization module, based on the corrected physical resistance feature weights of the corresponding point cloud voxels, uses a heuristic path search algorithm to calculate the point cloud combination with the minimum comprehensive cost from the robot's current position to the target location among all point clouds contained in the point cloud voxel set. The specific method is as follows: in, The comprehensive cost value is obtained by recalculating the physical resistance feature weights based on the corresponding point cloud voxels. The offset distance is the distance relative to the preset working axis. The preset working axis refers to the main working path pre-planned by the robot when entering a hard, agglomerated area (usually a fixed straight line / curve perpendicular to the grain pile cross-section or along the grain silo inspection route). The offset distance refers to the Euclidean distance between the actual path generated by the robot and this preset axis. Add to This means that the robot not only seeks to minimize physical resistance, but also to minimize deviation from the axis. In order to break up the clumps, the robot may directly rush towards the center of the clumps (at which point...). The size is relatively large (because it deviates from the preset axis). After reconstructing the path, when the block is broken up, As the speed decreases, the robot will prioritize choosing the path that returns to the axis of regression. As the agglomerates shrink, the system forces the robot to automatically return to the preset main operating channel after breaking up the clumps to continue performing subsequent tasks (such as hopper turnover and material discharge), instead of lingering around the agglomerates. For areas that are not broken up, where the terrain is uneven... The value is very high (high penalty, the robot tries to avoid it). For the already broken-up area, the terrain has been flattened by the drill. When the value is very low (low reward, the robot can pass quickly), the robot will tend to choose areas that have been broken up and become flat when planning its path. Small, it will punish those places that, although with low resistance, are still uneven ( The large size of the grain storage area forces the robot to follow the path on the leveled grain surface. This achieves a closed loop of breaking up clumps, verifying the levelness, and reconstructing the path to follow the leveled surface, ensuring that the grain storage environment is truly cleaned and leveled, rather than just avoiding clumps.

[0094] During the dismantling process Continuously updated, subsequent block breaking control vector according to Recalculate.

[0095] because Compared to The significant decrease in cost field creates a substantial logical void at the original block coordinates. From the navigation algorithm's perspective, what was previously a detour obstacle has now become a high-priority passage. The planner then triggers a secondary optimization mechanism, using the aforementioned A* algorithm to regenerate the optimal subsequent trajectory through the hard block region. Since the overall cost value of the original hard block region has converged to an extremely low level, the generated subsequent trajectory appears as a straight line or smooth curve that directly traverses the original obstacle region with a very small rate of curvature change.

[0096] The path iteration search process of the topology trajectory optimization module is as follows: Figure 6 As shown: Figure 6 The core logic of bidirectional optimization is presented: the first step is initial path generation (based on the initial cost map of the multimodal recognition module), with the path bypassing high-resistance cluster areas; the second step is cost update (based on the corrected cost map from the reverse optimization module). The first step is a sharp drop in cost in the original blocky area; the second step is a secondary optimization trigger (the planner detects that the original obstacle area has become a low-cost area); the third step is the optimal trajectory output (generating a smooth curve that traverses the original blocky area). This process fully embodies the two-way optimization mechanism of initial planning, operation feedback, cost update, and trajectory reconstruction, ensuring that the robot can quickly adapt to environmental changes after operation and generate the most efficient subsequent trajectory.

[0097] Finally, the robot control system executes the subsequent optimal work trajectory, guiding the robot body to quickly pass through the reconstructed grain surface area.

[0098] Example 2 A method for dynamically reconstructing robot operation trajectories under large language model task interaction includes: A global coordinate system point cloud is constructed based on sensor data. Spatial clustering is performed on the global coordinate system point cloud to obtain multiple point clusters. For each point cluster, a local geometric tensor matrix is ​​constructed. The hardness coefficient of each point cluster is calculated based on the local geometric tensor matrix. The hardness nodule region corresponding to the point cluster is determined based on the hardness coefficient. The global coordinate system point cloud is divided into multiple point cloud voxels. A set of point cloud voxels related to the robot's current task instruction is selected from the global coordinate system point cloud through a cross-modal self-attention mechanism. The physical resistance feature weight of the corresponding point cloud voxel is calculated based on the physical characteristics of the hardness nodule material of each point cloud voxel in the set of point cloud voxels. The comprehensive cost value of each point cloud voxel in the set of point cloud voxels is calculated based on the physical resistance feature weight. A heuristic path search algorithm is used to calculate the set of points with the minimum comprehensive cost from the robot's current position to the hard nodule region among all points contained in the point cloud voxel set. This set is used as the robot's environment reconstruction path. The breaking parameter sequence is set according to the physical resistance characteristic weight of the hard nodule region. The robot is controlled to reach the hard nodule region according to the environment reconstruction path. The breaking mechanism of the robot is driven to break the hard nodule according to the breaking parameter sequence. Based on the torque data during the process of breaking up hard agglomerates, a rule-based state identification algorithm based on time-domain feature analysis is used to determine the critical point at which the hard agglomerates transform from a hard constrained state to a loose particle state. When the critical point is reached, the environmental reverse action gain signal is calculated using a nonlinear function based on torque release efficiency, and the physical resistance feature weights of the corresponding point cloud voxels are corrected based on the environmental reverse action gain signal. Based on the physical resistance feature weights corrected for the corresponding point cloud voxels, a heuristic path search algorithm is used to calculate the point cloud combination with the minimum comprehensive cost from the robot's current position to the target location among all point clouds contained in the point cloud voxel set, which is then used as the robot's path.

[0099] Example 3 A computer program product includes a computer program / instruction, which, when executed by a processor, implements the method for dynamic reconstruction of robot operation trajectory under large language model task interaction in Embodiment 2.

[0100] The contents not described in detail in this specification are prior art known to those skilled in the art. Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0101] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0102] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0103] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0104] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit its scope of protection. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that after reading the present invention, they can still make various changes, modifications or equivalent substitutions to the specific implementation of the invention, but these changes, modifications or equivalent substitutions are all within the scope of protection of the pending claims of the invention.

Claims

1. A dynamic reconstruction system for robot operation trajectory under large language model task interaction, characterized in that, include: The multimodal recognition module is used to construct a global coordinate system point cloud based on sensor data, perform spatial clustering on the global coordinate system point cloud to obtain multiple point clusters, construct a local geometric tensor matrix for each point cluster, calculate the hardness coefficient of each point cluster based on the local geometric tensor matrix, determine the hard agglomeration region corresponding to the point cluster based on the hardness coefficient, divide the global coordinate system point cloud into multiple point cloud voxels, and filter out the set of point cloud voxels related to the robot's current task instruction from the global coordinate system point cloud through a cross-modal self-attention mechanism. Based on the physical characteristics of the hard agglomeration material of each point cloud voxel in the set of point cloud voxels, calculate the physical resistance feature weight of the corresponding point cloud voxel, thereby obtaining the physical resistance feature weight of the hard agglomeration region, and calculate the comprehensive cost value of each point cloud voxel in the set of point cloud voxels based on the physical resistance feature weight. The environmental reconstruction decision module is used to calculate the set of points with the minimum comprehensive cost from the robot's current position to the hard nodule area among all points contained in the point cloud voxel set using a heuristic path search algorithm. This set of points serves as the robot's environmental reconstruction path. The module sets a breaking parameter sequence based on the physical resistance characteristic weight of the hard nodule area, controls the robot to reach the hard nodule area according to the environmental reconstruction path, and drives the robot's breaking mechanism to break the hard nodule according to the breaking parameter sequence. The reverse optimization gain module is used to determine the critical point of the hard nodule's transformation from a hard constraint state to a loose particle state based on the torque data during the process of breaking the hard nodule. When the critical point is reached, the environmental reverse action gain signal is calculated using a nonlinear function based on torque release efficiency, and the physical resistance feature weight of the corresponding point cloud voxel is corrected based on the environmental reverse action gain signal. The topology trajectory optimization module is used to calculate the combination of point clouds that minimizes the overall cost of the robot's journey from its current position to the target location among all point clouds contained in the point cloud voxel set, based on the physical resistance feature weights corrected for the corresponding point cloud voxels, using a heuristic path search algorithm. This combination serves as the robot's path.

2. The robot trajectory dynamic reconstruction system under large language model task interaction as described in claim 1, characterized in that, In the multimodal recognition module, a local geometric tensor matrix is ​​constructed for each point cluster. The hardness coefficient of each point cluster is calculated based on the local geometric tensor matrix. The specific method for determining the hard nodal region corresponding to the point cluster based on the hardness coefficient is as follows: For each point cluster, construct a local geometric tensor matrix. As shown below: in, This represents the number of points contained in the current cluster after spatial clustering. For the first point in the cluster The coordinate vector of a point, Let the point cluster be the centroid. It is the transpose symbol; For local geometric tensor matrix Perform eigenvalue decomposition to obtain eigenvalues ​​representing the extent of extension of the longest principal direction of the point cluster. , eigenvalues ​​representing the extent of extension of the secondary principal direction of a point cluster The eigenvalues ​​representing the extent of extension of the shortest principal direction of the point cluster. ; The formula for calculating the hardness coefficient is as follows: in, The hardness coefficient, , , These are the first-direction weights, the second-direction weights, and the third-direction weights, respectively. Hardness coefficient Clusters of points that are greater than or equal to the hardness coefficient threshold are identified as hard agglomerates.

3. The robot trajectory dynamic reconstruction system under large language model task interaction as described in claim 2, characterized in that, In the multimodal recognition module, the specific method for calculating the physical resistance characteristic weight of the corresponding point cloud voxel based on the physical characteristics of the hard agglomerate material of each point cloud voxel in the point cloud voxel set is as follows: in, For the first The physical resistance feature weights of each point cloud voxel The hardness coefficient is preset based on the type of hard agglomerated material. These are the eigenvalues ​​representing the extent of extension along the longest principal direction of the point cluster. , eigenvalues ​​representing the extent of extension of the secondary principal direction of a point cluster The eigenvalues ​​representing the extent of extension of the shortest principal direction of the point cluster. , For the first Terrain height gradient along each main direction, It is an exponential function.

4. The robot trajectory dynamic reconstruction system under large language model task interaction according to claim 3, characterized in that, In the multimodal recognition module, the specific method for calculating the comprehensive cost value of each point cloud voxel in the point cloud voxel set based on the physical resistance feature weights is as follows: in, For the first The comprehensive value of a point cloud voxel For the first The comprehensive value of a point cloud voxel As the first balance factor, As the second balance factor, As the third balance factor, For point cloud voxels spatial coordinates, For the robot's spatial coordinates, for and The straight-line distance between them It is the corresponding point cloud voxel height The compensation function, , It is the compensation coefficient.

5. The robot trajectory dynamic reconstruction system under large language model task interaction according to claim 1, characterized in that, In the environmental reconstruction decision module, the specific method for calculating the set of points with the minimum comprehensive cost from the robot's current position to the hard agglomerate region among all points contained in the point cloud voxel set using a heuristic path search algorithm is as follows: A heuristic pathfinding algorithm is employed, with a cost function To optimize the objective and minimize the total cost, the path is solved. Cost function As shown below: in, For path Cost function For the first Time of the first A point cloud voxel, path It is the first Time of the first A point cloud voxel The set, For smoothness weights, For the first Time of the first A point cloud voxel instantaneous speed, That is the total time.

6. The robot trajectory dynamic reconstruction system under large language model task interaction according to claim 1, characterized in that, In the environmental reconfiguration decision module, the specific method for setting the agglomeration breaking driving parameter sequence based on the physical resistance characteristic weights of the hard agglomeration region is as follows: Based on the physical resistance characteristic weights of the corresponding point cloud voxels in the hard agglomerate region, a preset driving parameter sequence is generated through a dynamic mapping operator. The dynamic mapping operator is based on the physical resistance feature weights of point cloud voxels. A family of linear mapping functions for input variables; According to the preset drive parameter sequence Set the control vector for block breaking ,in, For a moment The angular velocity of the drill bit rotation, axial depth axial feed pressure, For a moment The robot's movement speed; in, It is the lower limit of the drill bit's rotational angular velocity. Upper limit of drill bit rotational angular velocity It is the acceleration coefficient of the drill bit's rotational angular velocity. It is a natural constant; in, It is the axial depth to which the drill bit penetrates the agglomerate. It is a proportionality coefficient. It is the expected preset torque. It is a damping adjustment factor. It is the torque collected in real time by the drill bit motor.

7. The robot trajectory dynamic reconstruction system under large language model task interaction according to claim 1, characterized in that, In the reverse optimization gain module, based on the torque data during the process of breaking up hard agglomerates, a rule-based state identification algorithm using time-domain feature analysis is employed to determine the critical point at which the hard agglomerate transitions from a hard-constrained state to a granular state. The specific method is as follows: in, for The status indicator shows the breaking mechanism in various states: 2 indicates the breaking mechanism is under high load and requires intervention; 1 indicates the breaking mechanism is in progress and operating normally; and 0 indicates the breaking mechanism has completed breaking, meaning the hard agglomerate has been broken up. High load torque threshold The rate of change of torque. The peak torque within the time window. The threshold for the rate of change of torque. For logical NOT operator, The residual torque threshold, The torque variance within the time window. This is the torque variance threshold.

8. The robot trajectory dynamic reconstruction system under large language model task interaction according to claim 1, characterized in that, In the reverse optimization gain module, the environmental reverse action gain signal is calculated using a nonlinear function based on torque release efficiency. The specific method for correcting the physical resistance feature weights of the corresponding point cloud voxels based on the environmental reverse action gain signal is as follows: in, This is the environmental reverse-action gain signal. This is the first gain adjustment coefficient. This is the maximum torque during the breaking process. This is the second gain adjustment coefficient. To break through the drilling depth of the mechanism, Thickness of hard nodules; in, For the first Physical resistance feature weights corrected for each point cloud voxel To optimize the adjustment coefficient.

9. A method for dynamically reconstructing robot operation trajectories under large language model task interaction, characterized in that, include: A global coordinate system point cloud is constructed based on sensor data. Spatial clustering is performed on the global coordinate system point cloud to obtain multiple point clusters. For each point cluster, a local geometric tensor matrix is ​​constructed. The hardness coefficient of each point cluster is calculated based on the local geometric tensor matrix. The hardness agglomeration region corresponding to the point cluster is determined based on the hardness coefficient. The global coordinate system point cloud is divided into multiple point cloud voxels. A set of point cloud voxels related to the robot's current task instruction is selected from the global coordinate system point cloud through a cross-modal self-attention mechanism. The physical resistance feature weight of the corresponding point cloud voxel is calculated based on the physical characteristics of the hard agglomeration material of each point cloud voxel in the set of point cloud voxels, thereby obtaining the physical resistance feature weight of the hard agglomeration region. The comprehensive cost value of each point cloud voxel in the set of point cloud voxels is calculated based on the physical resistance feature weight. A heuristic path search algorithm is used to calculate the set of points with the minimum comprehensive cost from the robot's current position to the hard nodule region among all points contained in the point cloud voxel set. This set is used as the robot's environment reconstruction path. The breaking parameter sequence is set according to the physical resistance characteristic weight of the hard nodule region. The robot is controlled to reach the hard nodule region according to the environment reconstruction path. The breaking mechanism of the robot is driven to break the hard nodule according to the breaking parameter sequence. Based on the torque data during the process of breaking up hard agglomerates, a rule-based state identification algorithm based on time-domain feature analysis is used to determine the critical point at which the hard agglomerates transform from a hard constrained state to a loose particle state. When the critical point is reached, the environmental reverse action gain signal is calculated using a nonlinear function based on torque release efficiency, and the physical resistance feature weights of the corresponding point cloud voxels are corrected based on the environmental reverse action gain signal. Based on the physical resistance feature weights corrected for the corresponding point cloud voxels, a heuristic path search algorithm is used to calculate the point cloud combination with the minimum comprehensive cost from the robot's current position to the target location among all point clouds contained in the point cloud voxel set, which is then used as the robot's path.

10. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instruction is executed by the processor, it implements the method for dynamic reconstruction of robot operation trajectory under large language model task interaction as described in claim 9.

Images