A multimodal intelligent control method and system for bulk cargo sampling robots

Abstract

This invention relates to the field of industrial automation control technology and discloses a multimodal intelligent control method and system for bulk cargo sampling robots. The method includes: synchronously acquiring visual, point cloud, and posture data, fusing and processing them to generate a dynamic digital twin scene model; performing two-level intelligent decision-making based on this model, with the upper layer planning a global operation path based on task objectives and the lower layer generating motion and operation commands based on real-time feedback; controlling the robot to move stably and adaptively to the terrain, while the end effector autonomously switches sampling modes based on real-time force interaction information with the materials; and continuously optimizing the model and strategy through data closed-loop. The system consists of perception fusion, intelligent decision-making, collaborative execution, and data management modules. This invention achieves deep semantic understanding of the bulk cargo environment, scientific and autonomous sampling planning, compliant motion control in complex terrain, and continuous learning and optimization of the system, significantly improving the representativeness, adaptability, and intelligence level of sampling.

Description

Technical Field

[0001] This invention relates to the field of industrial automation control technology, specifically to a multimodal intelligent control method and system for a bulk cargo sampling robot. Background Technology

[0002] Sampling of bulk cargo is a crucial step in industrial production and quality inspection. Currently, the technical means in this field are mainly divided into two categories: manual sampling and automated sampling. Manual sampling relies on operators using hand tools, which has problems such as low efficiency, poor safety, and results being easily affected by subjective factors. Existing automated sampling technologies mainly include robotic arm sampling equipment based on fixed tracks or preset programs, as well as mobile robot platforms equipped with simple sensors. These devices perform tasks through pre-programmed paths or basic remote control operations, which to some extent replaces human labor and improves operational efficiency.

[0003] However, it still has some shortcomings in practical applications. For example, existing automated equipment mostly relies on lidar or vision sensors for environmental geometry modeling. The generated models only contain spatial contour information and cannot identify and distinguish the key physical properties such as the type, hardness, and density of different materials in the stockpile, resulting in a lack of targeted basis for subsequent sampling decisions.

[0004] On the other hand, when faced with loose and uneven material pile terrain, existing mobile platforms mostly adopt control algorithms based on fixed parameters. Their motion strategies are rigid and cannot be adaptively adjusted according to the real-time perception of terrain flexibility and material flowability. This can easily lead to problems such as sinking, slipping, and machine instability, and even damage to the material pile structure.

[0005] To address the above issues, we propose a multimodal intelligent control method and system for bulk cargo sampling robots. Summary of the Invention

[0006] This invention provides a multimodal intelligent control method and system for bulk cargo sampling robots, which helps to solve the problems mentioned in the background art.

[0007] On one hand, the present invention provides the following technical solution: a multimodal intelligent control method for a bulk cargo sampling robot, comprising:

[0008] S1. Simultaneously collect visual image data, three-dimensional point cloud data, and robot body posture data of the target bulk cargo yard, and perform spatiotemporal fusion processing on the visual image data and three-dimensional point cloud data to generate a dynamic digital twin scene model containing semantic information of material distribution.

[0009] S2. Based on the dynamic digital twin scene model, a two-level decision is made. The first-level decision plans the global operation sequence and spatial path according to the task objective. The second-level decision generates motion control instructions for the robot body and operation instructions for the end effector based on the global planning and real-time environmental feedback.

[0010] S3. Control the robot to move on complex terrain and maintain a stable working posture according to the motion control instructions. At the same time, control the end effector to adaptively select and switch sampling modes to complete sampling according to the operation instructions and real-time interaction information with the material.

[0011] S4. Feed back the multi-source data generated during the operation to the dynamic digital twin scenario model and decision-making process to optimize the model accuracy and control strategy.

[0012] By adopting the above technical solution, a complete intelligent control closed loop of perception-decision-execution-optimization is established, which transforms the bulk cargo sampling robot from an automated device that relies on fixed programs into an intelligent agent with environmental understanding, autonomous decision-making and continuous learning capabilities. This solves the fundamental problems of poor adaptability and high interruption rate of traditional methods in complex dynamic environments.

[0013] Furthermore, the data acquisition in S1 is achieved through the following methods:

[0014] The system uses a high-resolution industrial camera and a multi-line lidar for fusion sensing. The multi-line lidar outputs raw 3D point cloud data at the first frequency, while the industrial camera performs exposure sampling to acquire visual image data during the laser pulse intervals of the multi-line lidar.

[0015] The robot's attitude data is collected by an inertial measurement unit integrated on the robot body, and the robot's three-dimensional angular velocity, acceleration and tilt angle are fed back in real time.

[0016] The visual image data, 3D point cloud data, and robot body posture data are synchronized through hardware trigger signals, and motion compensation is performed on the visual image data and 3D point cloud data using angular velocity and acceleration data collected by the inertial measurement unit and an extended Kalman filter algorithm.

[0017] By adopting the above technical solution, specifying multi-sensor hardware synchronization and motion compensation algorithm based on extended Kalman filter, the precise spatiotemporal alignment of visual, point cloud, and pose data is ensured, providing a reliable data foundation for generating accurate dynamic digital twin scene models, and effectively overcoming the problems of blurred and misaligned perception data caused by on-site vibration and robot movement.

[0018] Furthermore, the generation of the dynamic digital twin scene model in S1 specifically includes:

[0019] The visual image data is semantically segmented to identify material texture and type; the identification results are mapped pixel-to-point cloud to form a material attribute semantic layer that includes density and hardness estimation; the material attribute semantic layer is fused with the three-dimensional geometric model constructed based on the point cloud to generate the dynamic digital twin scene model.

[0020] By adopting the above technical solution, a specific algorithm flow for generating dynamic digital twin scene models rich in semantic information is defined. By deeply integrating the material type of visual recognition with the geometric and physical attributes of point clouds, the model can not only reflect the shape of the environment, but also understand the essence of the material, providing key information support for subsequent intelligent decision-making based on material characteristics.

[0021] Furthermore, the first-level decision in S2 includes:

[0022] Access the large language model interface, parse the representative requirements in the task instructions, and map them as a quantitative coverage index of the semantic layer of material attributes in the dynamic digital twin scenario model.

[0023] Based on the quantitative coverage index, an objective function is constructed and optimized at the material attribute semantic layer in the dynamic digital twin scenario model to generate a sampling point sequence and travel path that can simultaneously satisfy spatial coverage and attribute coverage.

[0024] By adopting the above technical solution, a large language model interface is introduced to transform abstract task instructions into quantifiable optimization indicators. Global path planning is then performed at the semantic layer of the dynamic digital twin scenario model. This ensures the scientific placement of sampling points at the algorithm level, thereby improving the statistical representativeness of the samples and the rationality of the work plan from the source.

[0025] Furthermore, the second-level decision-making in S2 is implemented using a policy network trained by reinforcement learning;

[0026] The input state space of the policy network includes: local terrain geometric features and material property features around the robot's current position extracted from the dynamic digital twin scene model;

[0027] The reward function of the policy network is set to maximize fuselage stability and minimize disturbance to the stockpile structure while achieving the sub-target position.

[0028] By adopting the above technical solution and using a policy network based on reinforcement learning for second-level real-time decision-making, the robot can learn the best strategy for moving in complex terrain. While achieving its goal, it can autonomously optimize its body stability and the disturbance to the material pile, demonstrating the adaptability and intelligence of the underlying control.

[0029] Furthermore, in step S3, the sampling mode is adaptively selected and switched based on the operation instructions and real-time interaction information with the material, specifically including:

[0030] Contact force information is obtained by force sensors on the end effector. Based on a preset material property-mechanical response model, the hardness and adhesion of the material are judged in real time. The vacuum adsorption, high-frequency vibration crushing or mechanical gripping mode is dynamically switched according to the judgment result, and the operating parameters are adjusted.

[0031] By adopting the above technical solution, and through real-time interaction between force sensing and a pre-set mechanical model, the robot achieves online identification of the characteristics of the contact material and autonomous switching of the sampling mode. This enables the robot to flexibly adapt to different materials such as powder, lumps, or sticky and wet materials, just like a human hand, thus solving the problem of sample distortion or acquisition failure caused by traditional single-function sampling heads.

[0032] Furthermore, S4 specifically includes:

[0033] Multi-source data from the operation process are stored as training samples in an experience database. The training samples include environmental conditions, generated control commands, and corresponding operation result feedback.

[0034] Based on the aforementioned experience database, the decision algorithm model for generating the motion control commands and operation commands is iteratively optimized.

[0035] By adopting the above technical solutions, a closed-loop learning mechanism of data collection and model optimization was established, enabling the system's decision model to continuously iterate and improve performance using historical operation data. This endows the system with the ability to evolve itself and become more intelligent with use, ensuring the reliability and efficiency of long-term operation.

[0036] On the other hand, a multimodal intelligent control system for a bulk cargo sampling robot, used to implement the aforementioned multimodal intelligent control method for a bulk cargo sampling robot, includes:

[0037] The perception fusion module integrates visual sensors, LiDAR, and inertial measurement units to collect multi-source heterogeneous data of the environment and the subject, and generates a dynamic digital twin scene model through the built-in fusion algorithm processor.

[0038] The intelligent decision-making center, which is communicatively connected to the perception fusion module, includes:

[0039] The task planning unit is used to receive task instructions and access the dynamic digital twin scene model, and output a global task sequence and path planning.

[0040] The tactical control unit is used to calculate and generate underlying robot motion control commands and end-effector operation commands based on the output of the task planning unit and real-time sensor feedback.

[0041] The collaborative execution module includes:

[0042] The terrain-adaptive mobile platform is equipped with a servo drive system, which receives the motion control commands to achieve stable movement and precise docking on complex terrain.

[0043] A multimodal end effector integrates multiple different sampling mechanisms, receives the end operation command, and adaptively completes the sampling operation based on contact force sensing;

[0044] The data closed-loop and management module is used to record the system's full-link operation data and provide training and optimization data support for the model in the intelligent decision-making center.

[0045] Furthermore, the terrain-adaptive mobile platform has a multi-legged structure, and its servo drive system incorporates a model prediction controller based on a whole-body dynamics model.

[0046] The whole-body dynamics model includes an estimate of the non-rigid contact mechanics between the foot and the bulk material pile;

[0047] The whole-body dynamics model models the contact force between the robot and the ground as a function of foot displacement. Its stiffness coefficient and damping coefficient are obtained by using a preset mapping table through the density and humidity information provided by the material property semantic layer of the current foot contact area in the dynamic digital twin scene model.

[0048] By adopting the above technical solution, the multi-legged mobile platform is specifically designed to use model predictive control that combines non-rigid contact mechanics estimation, and the material properties in the digital twin are directly mapped to control parameters. This enables the robot's motion control to sense the mechanical properties of the material under its feet, achieving dynamic balance and stable walking on easily deformable bulk material piles.

[0049] The technical effects and advantages of this invention are as follows:

[0050] 1. This invention achieves a fundamental leap in understanding bulk cargo yard environments from traditional single geometric modeling to multi-dimensional physical properties such as material type, density, and hardness by performing pixel-to-point cloud association mapping between the semantic segmentation results of high-resolution visual images and the geometric and physical properties of three-dimensional laser point clouds, and then fusing them to generate a dynamic digital twin scene model containing a semantic layer of material properties. This provides a precise data foundation for solving the problem of blind sampling decisions caused by the lack of environmental semantic information.

[0051] 2. This invention parses natural language task instructions by accessing a large language model interface and quantifies abstract requirements such as representative sampling into mathematical coverage indicators of the semantic layer of material attributes in a dynamic digital twin scenario model. Then, it performs global optimization and solution, realizing a fundamental shift from fixed sampling based on human experience to artificial intelligence-driven, real-time environment model-based autonomous generation of scientifically optimal sampling schemes. This solves the core technical problems of low intelligence and insufficient scientific rigor in sampling planning.

[0052] 3. This invention employs a policy network trained based on reinforcement learning as the underlying decision-maker, enabling it to autonomously learn motion strategies for stable movement and minimizing disturbances on complex material piles based on the local terrain and material characteristics provided by the dynamic digital twin scene model. This achieves a breakthrough in the robot's movement on unstructured and easily deformable bulk cargo surfaces, moving from programmed movement based on fixed rules to intelligent and agile movement with autonomous adaptation and optimization capabilities. It solves the control problem of balancing movement stability and efficiency in complex terrain.

[0053] 4. By establishing a data closed loop that runs through the entire chain of perception, decision-making, and execution, this invention stores multi-source data of the operation process as experience samples and uses them to continuously iterate and optimize the decision model. This enables the entire intelligent control system to evolve from a static program that is deployed only once to a living system that can continuously learn from actual operations and evolve its performance iteratively. This provides a fundamental solution to the long-term operation and maintenance problem of intelligent systems that require frequent manual re-debugging due to environmental changes or task iterations. Attached Figure Description

[0054] Figure 1 This is a schematic diagram of the method flow of the present invention.

[0055] Figure 2 This is a schematic diagram of the system flow of the present invention.

[0056] In the diagram: 1. Perception fusion module; 2. Intelligent decision-making center; 3. Collaborative execution module; 4. Data closed loop and management module. Detailed Implementation

[0057] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Example

[0058] Reference Figure 1 A multimodal intelligent control method for a bulk cargo sampling robot includes the following steps:

[0059] S1. Simultaneously collect visual image data, 3D point cloud data and robot body posture data of the target bulk cargo yard, and perform spatiotemporal fusion processing on the visual image data and 3D point cloud data to generate a dynamic digital twin scene model containing semantic information of material distribution.

[0060] It should be further explained that the data collection in S1 is achieved through the following methods:

[0061] The system uses a high-resolution industrial camera and a multi-line lidar for fusion sensing. The multi-line lidar outputs raw 3D point cloud data at the first frequency, while the industrial camera performs exposure sampling to acquire visual image data during the laser pulse intervals of the multi-line lidar.

[0062] The robot's attitude data is collected by an inertial measurement unit integrated on the robot body, and the robot's three-dimensional angular velocity, acceleration and tilt angle are fed back in real time.

[0063] Visual image data, 3D point cloud data, and robot body posture data are synchronized through hardware trigger signals. Angular velocity and acceleration data collected by the inertial measurement unit are used to perform motion compensation on the visual image data and 3D point cloud data through an extended Kalman filter algorithm.

[0064] Specifically, the robot is equipped with a high-resolution camera, a multi-line LiDAR, and an inertial measurement unit (IMU). The system uses the LiDAR's scanning cycle as a time reference and drives the camera to expose during the intervals between laser pulses via hardware trigger signals, achieving millisecond-level time-synchronized acquisition and obtaining visual image frames. and the corresponding original point cloud frame ;

[0065] Because the robot collects data while moving, the data is subject to motion distortion. For the k-th synchronous acquisition time, For the previous data fusion moment, the transformation matrix provided by the IMU for body pose transformation. It indicates that from the last fusion moment arrive The body pose transformation is as follows: ;

[0066] in, Let be a rotation matrix. The translation vector is used; the angular velocity and acceleration data of the IMU are integrated and filtered by an extended Kalman filter to predict the value in real time. and This estimated pose is used to perform motion compensation on the original point cloud: ;

[0067] in, Indicates in The raw point cloud collection acquired in real time, compared with the synchronously acquired images. Digital image stabilization is performed using the estimated camera motion to obtain a corrected image with the blur removed. .

[0068] It should be further explained that the generation of dynamic digital twin scene models in S1 specifically includes:

[0069] Semantic segmentation is performed on visual image data to identify material texture and type; the identification results are mapped to 3D point cloud data pixel-to-point cloud association to form a material attribute semantic layer that includes density and hardness estimation; the material attribute semantic layer is fused with the 3D geometric model built based on point cloud to generate a dynamic digital twin scene model.

[0070] Specifically, firstly, a semantic segmentation network pre-trained on a large number of bulk cargo images is used to... Process the data and output the probability distribution map of the material category for each pixel. Each pixel position outputs a vector representing the probability of a preset category such as powder, small particles, block material, or background.

[0071] Next, pixel-point cloud semantic association is performed using a pre-calibrated coordinate transformation matrix between the camera and the LiDAR. Motion-compensated point cloud Projected onto semantic image Above, through this projection, for each three-dimensional point Assign a semantic label l to the pixel where its projection point is located and a confidence level c for that label, where x, y, and z are the values ​​of the pixel in the projection point. Coordinates in the coordinate system; simultaneously, calculate the local point cloud features of point p: normal vector n, curvature σ, and local point cloud density. Thus, the enhanced feature vector for each point is formed: ;

[0072] Then, point cloud clustering and attribute generation based on semantic constraints are performed, using an improved Euclidean clustering algorithm and setting a spatial distance threshold. And semantic consistency threshold; when merging neighboring points, the algorithm not only requires that the Euclidean distance between the two points be less than a certain threshold, but also... It also requires that their semantic labels be the same or compatible; this aggregates point clouds into multiple physically contiguous and semantically consistent point cloud clusters. For each cluster Calculate its aggregate attributes and main semantic categories. Average spatial density Average reflection intensity and the size of the cluster's bounding box as a spatial scale. The geometric center of the cluster is denoted as This generates a material attribute semantic layer, which is a set of... A descriptive description of the environment;

[0073] Finally, a dynamic digital twin scene model is constructed and updated. The system maintains a global 3D voxel model, where each voxel stores two types of information: occupancy probability and attribute probability distribution. For each point cloud cluster obtained in the current frame... By analyzing the properties and their influence on the voxel set, we not only update the geometric occupancy information but also use Bayesian update rules to update the semantic attribute probability distribution stored in each voxel v. For example, updating a semantic attribute with category j:

[0074]

[0075] in, It is the probability that voxel v belongs to category j before the update;

[0076] This is the updated probability;

[0077] It is an indicator function, when cluster Main categories The value is 1 when it equals j, and 0 otherwise.

[0078] It is the cumulative weight of historical data, reflecting the confidence level of old information;

[0079] Current observation The weights can be set based on factors such as the cluster's confidence level c and its distance from the sensor;

[0080] Physical properties are updated using a weighted average within the corresponding voxel. This continuously updated voxel model, containing both geometric and attribute information, is the dynamic digital twin scene model. .

[0081] S2. Two-level decision-making is carried out based on the dynamic digital twin scene model. The first-level decision-making plans the global operation sequence and spatial path according to the task objectives. The second-level decision-making generates motion control instructions for the robot body and operation instructions for the end effector based on the global planning and real-time environmental feedback.

[0082] It should be further explained that the first-level decision in S2 includes:

[0083] By accessing the large language model interface, the representative requirements in the task instructions are parsed and mapped to quantitative coverage indicators of the semantic layer of material attributes in the dynamic digital twin scenario model.

[0084] An objective function is constructed based on a quantitative coverage index, and optimized at the semantic layer of material attributes in a dynamic digital twin scenario model to generate a sampling point sequence and travel path that can simultaneously satisfy spatial coverage and attribute coverage.

[0085] Specifically, the output of the first-level decision is the task instruction, and the output is the global sampling point sequence. And the coarse path R, by transforming fuzzy representativeness into a model Mathematical optimization of the semantic layer of material attributes;

[0086] The system integrates a large language model module with fine-tuned instructions. The LLM parses natural language instructions into structured task description objects (Tasks); for example, defining a multi-attribute coverage maximization objective function: ;

[0087] in, It is a set of attributes of interest, which can be directly obtained from... Define and query in the semantic layer;

[0088] It is the weight coefficient of attribute A. ,all The sum can be 1, and the weights can be determined by the implicit requirements of the task instructions or by the preset domain knowledge.

[0089] It is the sampling point set G for all voxel regions with attribute A. The coverage utility function is quantified using a distance-attenuation-based model: ;

[0090] v is One of the voxels;

[0091] and Let v and g represent the three-dimensional coordinates of voxel v and the candidate sampling point position g, respectively.

[0092] Represents Euclidean distance;

[0093] The attenuation coefficient controls the effective coverage radius of a single sampling point. The larger the size, the wider the coverage area;

[0094] Operation means that the utility of a voxel v is determined by its nearest sampling point g;

[0095] Subsequently, a combinatorial optimization algorithm was used in Search within feasible areas Finally, based on Dahua's point set G*, Given geometric information, perform kinematically feasible path planning between points G* and output the global path R.

[0096] Specifically, the system integrates an open-source large language model that has been fine-tuned with instructions as a natural language understanding module. The input of this module is natural language instructions, and the model parses and maps them through the following steps:

[0097] The model first decomposes the instructions into structured fields, including target area, core requirements, and attributes of concern;

[0098] The system has a pre-built requirement-index mapping knowledge base. For representativeness, the mapping knowledge base is defined as being uniform in spatial distribution and proportionally covering the distribution of the material attributes of interest. Therefore, the model will output a structured task description object, which contains quantitative indicators: spatial uniformity weight = 0.5, attribute coverage weight = 0.5, and list of attributes of interest = [block material].

[0099] This Task object is passed to the path planning algorithm, where the attribute coverage weights and the list of attributes of interest are directly used to generate the weight coefficients of the objective function.

[0100] It should be further explained that the second-level decision in S2 is implemented using a policy network trained based on reinforcement learning;

[0101] The input state space of the policy network includes: local terrain geometric features and material property features around the robot's current position extracted from the dynamic digital twin scene model;

[0102] The reward function of the policy network is set to maximize fuselage stability and minimize disturbance to the stockpile structure while achieving the sub-target position.

[0103] Specifically, the policy network adopts an Actor-Critic architecture, where both the Actor and Critic networks are fully connected neural networks with three hidden layers, each containing 256, 128, and 64 neurons, respectively, and using ReLU as the activation function; their input states... At time t, including:

[0104] from The cropped shape, centered on the robot's current position, with a side length of... The cubic region voxel model is encoded as a multi-channel tensor; joint angle vector, joint velocity vector, Euler angle, angular velocity; coordinate offset of the current target point relative to the body, and current task stage flag;

[0105] The output actions of the policy network directly correspond to the set values ​​of the robot's underlying controller;

[0106] The policy network was trained in a simulation environment that modeled bulk material pile terrain with varying slopes, densities, and particle sizes. Training employed a near-end policy optimization algorithm, collecting empirical data from 4096 time steps per iteration, and using the Adam optimizer for multiple rounds of mini-batch gradient updates. Its reward function... for:

[0107] ;

[0108] in:

[0109] , , , These are the weighting coefficients for each reward, used to balance different objectives;

[0110] This means the negative distance from the current point to the target point, encouraging people to move closer to the target.

[0111] Punishment for aircraft roll angle Pitch angle Excessive and angular velocity Exceeding the standard This is the angular velocity penalty coefficient;

[0112] As a penalty term for pile disturbance, the work W done by the force exerted by the foot on the pile particles or the resulting total particle displacement D is calculated by the physics engine in the simulation. or , This is the perturbation penalty coefficient, which guides the agent to learn a gentle gait;

[0113] A large positive reward constant is given for successfully reaching a point or completing sampling.

[0114] After the policy converges in the simulation environment, the network parameters are deployed to the physical robot. To compensate for the difference between simulation and reality, the system adopts an online adaptive module: during the initial operation phase of the physical robot, a small amount of actual state-action data is collected in real time, and the output layer of the policy network is fine-tuned based on this data to make it quickly adapt to the dynamic characteristics of the real world.

[0115] S3. Control the robot to move on complex terrain and maintain a stable working posture according to motion control instructions. At the same time, control the end effector to adaptively select and switch sampling modes to complete sampling based on operation instructions and real-time interaction information with materials.

[0116] It should be further explained that S3 adaptively selects and switches sampling modes based on operation commands and real-time interaction information with materials, specifically including:

[0117] Contact force information is obtained by force sensors on the end effector. Based on a preset material property-mechanical response model, the hardness and adhesion of the material are judged in real time. The vacuum adsorption, high-frequency vibration crushing or mechanical gripping mode is dynamically switched according to the judgment result, and the operating parameters are adjusted.

[0118] Specifically, to achieve smooth and stable movement, model predictive control is used as the underlying tracking controller, and the dynamic model used is... In the middle, the contact force between the foot and the ground The model is as follows: ;

[0119] in:

[0120] It refers to the depth of foot penetration, that is, the depth to which the foot sinks into the ground;

[0121] It is the rate of change of penetration depth;

[0122] K and B are the equivalent grounding stiffness coefficient and damping coefficient, respectively;

[0123] and They are The estimated material density and humidity of the foot-contact area obtained from the query;

[0124] and This indicates that stiffness and damping are functions of density and humidity, and the mapping relationship is obtained through pre-calibration or online learning. The MPC controller uses this dynamic model to optimize joint torque.

[0125] The end-effector six-dimensional force sensor reads the contact force and torque vector in real time. Online contact characteristic classifier for force signals at the initial contact instant Perform analysis; for example, estimate the equivalent contact stiffness by calculating the slope of the force rise edge. Where v is the terminal approach velocity, The magnitude of the force signal is directly used to determine the material hardness, and combined with the spectral characteristics of the force signal to determine the material cohesion, thereby adjusting the sampling mode and parameters in real time.

[0126] Specifically, the pre-set material property-mechanical response model is a classifier based on support vector machines. This classifier is trained in a laboratory environment by collecting the initial contact force curves of various standard material samples with known properties when the end effector contacts them at a constant speed. The characteristics of the force curves include: the maximum slope of the force rise at the moment of contact, the frequency of force fluctuations after reaching a stable value, and the steady-state force value.

[0127] In real-time operation, when the end effector comes into contact with an unknown material, the control system synchronously collects six-dimensional force sensing data within 100 milliseconds after the start of contact, extracts the same force curve features in real time, and inputs them into a pre-trained classifier. The classifier outputs a joint judgment result on the material hardness and adhesion.

[0128] The system has a pre-set characteristic-mode mapping table; for example, if the determination is {hardness: soft, adhesion: high}, it is determined to be wet and sticky powder, and the vacuum adsorption mode is activated, while the adsorption pressure is adjusted to the highest level; if the determination is {hardness: hard, adhesion: low}, it is determined to be dry block material, and the high-frequency vibration crushing and mechanical claw gripping combination mode is activated.

[0129] S4. Feed back the multi-source data generated during the operation to the dynamic digital twin scenario model and decision-making process to optimize model accuracy and control strategy.

[0130] It should be further explained that S4 specifically refers to:

[0131] Multi-source data from the operation process are stored as training samples in the experience database. The training samples include environmental conditions, generated control commands, and corresponding operation result feedback.

[0132] Based on an empirical database, the decision algorithm model for generating motion control commands and operation commands is iteratively optimized.

[0133] Specifically, the system records the complete operation trajectory. During offline optimization, the objective function is minimized using the data in the experience database D through an offline reinforcement learning algorithm. To fine-tune the strategy network parameters ,in To estimate the action value function, and simultaneously, to calibrate the contact mapping function and semantic segmentation network using regression methods with real data, the system performance is iteratively improved.

[0134] Specifically, the experience database uses a circular buffer structure for storage, with each experience record being a tuple. After every 100 complete sampling operations, the system automatically triggers an offline optimization task. The optimization process mainly includes two parallel tasks:

[0135] Policy network fine-tuning: A batch of data is sampled from the experience database, and the parameters of the policy network are fine-tuned using an offline reinforcement learning algorithm. The optimization goal is to improve the long-term reward under actual working conditions without deviating from the learned safety policy.

[0136] Model calibration: By using the material characteristic judgment results recorded from successful sampling points in actual operations and the final sampling results, the classification boundary and confidence of the material characteristic-mechanical response model are recalibrated to improve its online judgment accuracy.

[0137] After the optimized new model parameters are verified, they are seamlessly deployed to the robot's main controller via hot updates to replace the old model. The system records the version, data batch, and performance improvement indicators for each optimization, thereby achieving continuous and traceable autonomous performance evolution.

[0138] Example 2

[0139] Reference Figure 2 A multimodal intelligent control system for a bulk cargo sampling robot, used to implement the aforementioned multimodal intelligent control method for a bulk cargo sampling robot, includes:

[0140] The perception fusion module 1 integrates a visual sensor, a lidar and an inertial measurement unit to collect multi-source heterogeneous data of the environment and the subject, and generates a dynamic digital twin scene model through the built-in fusion algorithm processor.

[0141] Intelligent decision-making center 2, which is communicatively connected to perception fusion module 1, includes:

[0142] The task planning unit is used to receive task instructions and access the dynamic digital twin scene model, and output the global task sequence and path planning.

[0143] The tactical control unit is used to calculate and generate low-level robot motion control commands and end-effector operation commands based on the output of the mission planning unit and real-time sensor feedback.

[0144] Collaborative execution module 3 includes:

[0145] The terrain-adaptive mobile platform is equipped with a servo drive system, receives motion control commands, and achieves stable movement and precise docking on complex terrain.

[0146] The multimodal end effector integrates multiple different sampling mechanisms, receives end operation commands, and adaptively completes sampling operations based on contact force sensing.

[0147] The data closed-loop and management module 4 is used to record the system's full-link operation data and provide training and optimization data support for the model in the intelligent decision-making center 2.

[0148] It should be further explained that the terrain-adaptive mobile platform has a multi-legged structure, and its servo drive system has a built-in model prediction controller based on the whole-body dynamics model.

[0149] The whole-body dynamics model includes an estimate of the non-rigid contact mechanics between the foot and the bulk material pile;

[0150] The whole-body dynamics model models the contact force between the robot and the ground as a function of foot displacement. Its stiffness coefficient and damping coefficient are obtained by using a preset mapping table through the density and humidity information provided by the material property semantic layer of the current foot contact area in the dynamic digital twin scene model.

[0151] Specifically, the control and computing platform of this system adopts a hierarchical and heterogeneous embedded architecture; the main computing unit adopts an embedded high-performance computing module with a graphics processing unit, which is responsible for running computationally intensive tasks such as environmental perception fusion, dynamic digital twin scene model construction and updating, global path planning, and forward inference of reinforcement learning policy network. This module provides sufficient parallel computing capabilities and memory bandwidth to meet the real-time requirements of semantic segmentation, point cloud processing, and deep learning model inference.

[0152] The real-time control unit adopts a real-time control subsystem based on a high-performance microcontroller. Its core is a processor core running a real-time operating system. This unit is responsible for receiving high-level instructions and executing high-frequency closed-loop servo control, model predictive controller calculation, and data acquisition and filtering by the inertial measurement unit and force sensor.

[0153] The sensing sensor suite includes a high-resolution industrial camera, a multi-line LiDAR, an inertial measurement unit, and a six-dimensional force / torque sensor integrated into the end effector. These sensors are connected to the main computing unit and real-time control unit via a high-speed serial bus or synchronous trigger interface to ensure synchronous data acquisition and low-latency transmission.

[0154] The computing units communicate with each other through a middleware based on a publish-subscribe model. This middleware supports deterministic data transmission and node scheduling. The system adopts a hybrid scheduling strategy that combines time-triggered and event-triggered scheduling to allocate deterministic computing resources and time slices for tasks with different real-time requirements, such as perception, planning, and control.

[0155] Secondly: The accompanying drawings of the embodiments disclosed in this invention only involve the structures involved in the embodiments disclosed in this invention. Other structures can refer to the general design. In the absence of conflict, the same embodiment and different embodiments of this invention can be combined with each other.

[0156] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A multimodal intelligent control method for a bulk cargo sampling robot, characterized in that, include: S1. Simultaneously collect visual image data, three-dimensional point cloud data, and robot body posture data of the target bulk cargo yard, and perform spatiotemporal fusion processing on the visual image data and three-dimensional point cloud data to generate a dynamic digital twin scene model containing semantic information of material distribution. S2. Based on the dynamic digital twin scene model, a two-level decision is made. The first-level decision plans the global operation sequence and spatial path according to the task objective. The second-level decision generates motion control instructions for the robot body and operation instructions for the end effector based on the global planning and real-time environmental feedback. S3. Control the robot to move on complex terrain and maintain a stable working posture according to the motion control instructions. At the same time, control the end effector to adaptively select and switch sampling modes to complete sampling according to the operation instructions and real-time interaction information with the material. S4. Feed back the multi-source data generated during the operation to the dynamic digital twin scene model and decision-making process to optimize the model accuracy and control strategy; S1 generates dynamic digital twin scene models, specifically including: The visual image data is semantically segmented to identify material texture and type; the identification results are mapped to the three-dimensional point cloud data using pixel-to-point cloud association to form a material attribute semantic layer that includes density and hardness estimation; the material attribute semantic layer is fused with the three-dimensional geometric model constructed based on the point cloud to generate the dynamic digital twin scene model. The first-level decision in S2 includes: Access the large language model interface, parse the representative requirements in the task instructions, and map them as a quantitative coverage index of the semantic layer of material attributes in the dynamic digital twin scenario model. Based on the quantitative coverage index, an objective function is constructed and optimized at the material attribute semantic layer in the dynamic digital twin scenario model to generate a sampling point sequence and travel path that can simultaneously satisfy spatial coverage and attribute coverage. S3 adaptively selects and switches sampling modes based on the operation instructions and real-time interaction information with the materials, specifically including: Contact force information is obtained by force sensors on the end effector. Based on a preset material property-mechanical response model, the hardness and adhesion of the material are judged in real time. The vacuum adsorption, high-frequency vibration crushing or mechanical gripping mode is dynamically switched according to the judgment result, and the operating parameters are adjusted.

2. The multimodal intelligent control method for a bulk cargo sampling robot according to claim 1, characterized in that, The data collection in S1 is achieved through the following methods: The system uses a high-resolution industrial camera and a multi-line lidar for fusion sensing. The multi-line lidar outputs raw 3D point cloud data at the first frequency, while the industrial camera performs exposure sampling to acquire visual image data during the laser pulse intervals of the multi-line lidar. The robot's attitude data is collected by an inertial measurement unit integrated on the robot body, and the robot's three-dimensional angular velocity, acceleration and tilt angle are fed back in real time. The visual image data, 3D point cloud data, and robot body posture data are synchronized through hardware trigger signals, and motion compensation is performed on the visual image data and 3D point cloud data using angular velocity and acceleration data collected by the inertial measurement unit and an extended Kalman filter algorithm.

3. The multimodal intelligent control method for a bulk cargo sampling robot according to claim 1, characterized in that, The second-level decision-making in S2 is implemented using a policy network trained by reinforcement learning; The input state space of the policy network includes: local terrain geometric features and material property features around the robot's current position extracted from the dynamic digital twin scene model; The reward function of the policy network is set to maximize fuselage stability and minimize disturbance to the stockpile structure while achieving the sub-target position.

4. The multimodal intelligent control method for a bulk cargo sampling robot according to claim 1, characterized in that, Specifically, S4 is: Multi-source data from the operation process are stored as training samples in an experience database. The training samples include environmental conditions, generated control commands, and corresponding operation result feedback. Based on the aforementioned experience database, the decision algorithm model for generating the motion control commands and operation commands is iteratively optimized.

5. A multimodal intelligent control system for a bulk cargo sampling robot, used to implement the multimodal intelligent control method for a bulk cargo sampling robot as described in any one of claims 1-4, characterized in that, include: The perception fusion module (1) integrates a visual sensor, a lidar and an inertial measurement unit to collect multi-source heterogeneous data of the environment and the body, and generates a dynamic digital twin scene model through the built-in fusion algorithm processor. The intelligent decision-making center (2), which is communicatively connected to the perception fusion module (1), includes: The task planning unit is used to receive task instructions and access the dynamic digital twin scene model, and output a global task sequence and path planning. The tactical control unit is used to calculate and generate underlying robot motion control commands and end-effector operation commands based on the output of the task planning unit and real-time sensor feedback. The collaborative execution module (3) includes: The terrain-adaptive mobile platform is equipped with a servo drive system, which receives the motion control commands to achieve stable movement and precise docking on complex terrain. A multimodal end effector integrates multiple different sampling mechanisms, receives the end operation command, and adaptively completes the sampling operation based on contact force sensing; The data loop and management module (4) is used to record the system's full-link operation data and provide training and optimization data support for the model in the intelligent decision-making center (2).

6. The multimodal intelligent control system for a bulk cargo sampling robot according to claim 5, characterized in that, The terrain-adaptive mobile platform has a multi-legged structure, and its servo drive system incorporates a model prediction controller based on a whole-body dynamics model. The whole-body dynamics model includes an estimate of the non-rigid contact mechanics between the foot and the bulk material pile; The whole-body dynamics model models the contact force between the robot and the ground as a function of foot displacement. Its stiffness coefficient and damping coefficient are obtained by using a preset mapping table through the density and humidity information provided by the material property semantic layer of the current foot contact area in the dynamic digital twin scene model.

Images