A multi-sensor fusion robot grasping path planning method and device

Abstract

The application provides a multi-sensor fusion robot grasping path planning method and device, and relates to the field of automation. The method comprises the following steps: constructing a first time correction relationship based on multi-source sensor observation parameters; inputting hardware timestamps synchronized to a master clock source into an extended state estimation model based on the first time correction relationship; iteratively updating the multi-source sensor observation parameters based on the extended state estimation model, and constructing a second time correction relationship according to the updated multi-source sensor observation parameters; constructing a pose continuous time trajectory function corresponding to a target robot based on the second time correction relationship; taking the continuous time trajectory function and the multi-source sensor observation parameters as inputs, and outputting a grasping path corresponding to the target robot through a joint optimization model with a time sequence consistency constraint; and controlling the target robot to perform a grasping operation based on the grasping path. The application solves the problem of low precision of the robot grasping path in the prior art.

Description

Technical Field

[0001] This application relates to the field of automation, and in particular to a method and apparatus for robot grasping path planning using multi-sensor fusion. Background Technology

[0002] In automotive manufacturing and parts assembly, robots are widely used for the automated grasping and positioning of body structural components. To ensure operational accuracy under conditions of complex curved surfaces, reflective materials, and interference from tooling fixtures, existing technologies generally employ a multi-source sensor architecture, including industrial cameras, LiDAR, inertial measurement units, and force sensors, to achieve accurate identification, spatial modeling, and grasping control of parts.

[0003] Existing technologies mostly employ multi-sensor fusion methods based on extended Kalman filtering for robot grasping path planning. This method requires recursively estimating the robot's pose under unified state transition equations and observation equations. Because this method demands that data from different sensors be input simultaneously and maintain temporal consistency, any misalignment in timing will directly affect the filtering results, making it highly sensitive to time synchronization.

[0004] In automotive body component assembly scenarios, industrial cameras, LiDAR, inertial measurement units, and force sensors all have different sampling frequencies, inherent delays in the triggering links, and network jitter during data transmission. This results in observations that are physically occurring at the same moment actually corresponding to inconsistent robot postures. When processing such asynchronous inputs, the extended Kalman filter misinterprets this inconsistency as a genuine change in pose, leading to spurious motion and hysteresis estimations, thereby reducing the accuracy of the robot's grasping path.

[0005] Therefore, there is an urgent need for a method and device for robot grasping path planning that integrates multiple sensors. Summary of the Invention

[0006] This application provides a multi-sensor fusion robot grasping path planning method and apparatus, which solves the problem that in automotive body parts assembly scenarios, the sampling frequencies of industrial cameras, LiDAR, inertial measurement units, and force sensors are different, triggering links have inherent delays, and data transmission is accompanied by network jitter. Existing robot grasping path planning methods misjudge this inconsistency as a real change in robot pose when processing such asynchronous inputs, resulting in pseudo-motion and hysteresis estimation, thereby reducing the accuracy of robot grasping path.

[0007] The first aspect of this application provides a multi-sensor fusion-based robot grasping path planning method, comprising: acquiring observation parameters from multiple sources of sensors, and correcting the local clocks of each sensor based on the observation parameters to construct a first time correction relationship; configuring hardware timestamps at the sampling front-end of each sensor based on the first time correction relationship, and synchronizing the hardware timestamps to a master clock source; inputting the hardware timestamps synchronized to the master clock source into an extended state estimation model; iteratively updating the observation parameters from multiple sources of sensors based on the extended state estimation model, and re-correcting the local clocks of each sensor based on the updated observation parameters to construct a second time correction relationship; constructing a pose continuous-time trajectory function corresponding to the target robot based on the second time correction relationship; using the continuous-time trajectory function and the observation parameters from multiple sources of sensors as input, outputting a grasping path corresponding to the target robot through a joint optimization model with temporal consistency constraints; and controlling the target robot to perform a grasping operation based on the grasping path.

[0008] Optionally, the observation parameters of the multi-source sensors are obtained, specifically including: obtaining the observation parameters of the multi-source sensors based on the multi-sensor system, wherein the multi-sensor system includes a vision sensor, a force sensor and a lidar sensor, and the observation parameters of the multi-source sensors include frequency offset parameters, phase offset parameters and fixed link delay parameters.

[0009] Optionally, the local clocks of each sensor are calibrated based on the observation parameters of the multi-source sensors to establish a first time calibration relationship. Specifically, this includes: acquiring the local clock readings of each sensor and proportionally correcting the local clock readings according to the frequency offset parameter; superimposing the phase offset parameter on the proportional correction result to perform phase compensation; introducing a fixed link delay parameter to perform delay compensation; and mapping the time results after proportional correction, phase compensation, and delay compensation to the master clock source to establish the first time calibration relationship.

[0010] Optionally, an extended state estimation model is constructed, specifically including: constructing a target correspondence and generating a time matching factor based on the target correspondence, wherein the target correspondence is the correspondence between the hardware time synchronized to the master clock source and the reference time determined by the first time correction relationship; using multi-source sensor observation parameters as graph node variables to construct a parameter node set; applying factor constraints to the parameter node set based on the time matching factor to form a factor relationship describing the observation error; writing the observation error into the factor graph in the form of a residual function based on the factor relationship; applying prior constraints to the parameter nodes in the factor graph, wherein the prior constraints are used to limit the slowly varying characteristics of the multi-source sensor observation parameters; and selecting a nonlinear least squares algorithm or an incremental inference algorithm to perform integrated operations on the factor constraints, prior constraints, and parameter node set to construct the extended state estimation model.

[0011] Optionally, a pose-continuous time trajectory function corresponding to the target robot is constructed based on the second time correction relationship. Specifically, this includes: extracting pose data points of the target robot at different time points based on the second time correction relationship; constructing a discrete pose sequence based on the pose data points, and performing continuous fitting on the discrete pose sequence to generate a pose function expression that can be queried in the entire time domain; performing constraint verification on the pose function expression and proposing function solutions that do not meet the constraint verification, including smoothness constraint verification and physical reachability constraint verification; and using the pose function expression after removing function solutions as the pose-continuous time trajectory function.

[0012] Optionally, before outputting the grasping path corresponding to the target robot through a joint optimization model with temporal consistency constraints, using the continuous-time trajectory function and multi-source sensor observation parameters as inputs, it is necessary to construct a joint optimization model. Specifically, this includes: using the continuous-time trajectory function as the first input and the multi-source sensor observation parameters as the second input to generate a joint optimization input set; introducing temporal consistency constraints into the input set, which are used to ensure that the observation time of each sensor is consistent with the time parameters of the continuous-time trajectory function; constructing an objective function based on the input set with introduced temporal consistency constraints, the objective function including path smoothing terms, obstacle avoidance terms, and attitude constraint terms; configuring path constraints for the objective function, including approach path constraints, alignment micro-motion path constraints, closed grasping path constraints, and withdrawal path constraints; and constructing a joint optimization model based on the configured objective function.

[0013] Optionally, controlling the target robot to perform grasping operations based on the grasping path specifically includes: decomposing the grasping path into one or more control command sequences including an approach path, an alignment micro-motion path, a closed grasping path, and a withdrawal path; and controlling the target robot to perform grasping operations based on one or more control command sequences.

[0014] A second aspect of this application provides a multi-sensor fusion robot grasping path planning device, the device including an acquisition module and a processing module, wherein,

[0015] The acquisition module is used to acquire observation parameters from multiple sensors and to correct the local clocks of each sensor based on the observation parameters to construct a first time correction relationship. Based on the first time correction relationship, hardware timestamps are configured at the sampling front end of each sensor and synchronized to the master clock source. The hardware timestamps synchronized to the master clock source are input into the extended state estimation model.

[0016] The processing module is used to iteratively update the observation parameters of the multi-source sensors based on the extended state estimation model, and recalibrate the local clocks of each sensor according to the updated multi-source sensor observation parameters to construct a second time calibration relationship; construct the pose continuous-time trajectory function corresponding to the target robot based on the second time calibration relationship; take the continuous-time trajectory function and the multi-source sensor observation parameters as input, and output the grasping path corresponding to the target robot through a joint optimization model with temporal consistency constraints; and control the target robot to perform grasping operations based on the grasping path.

[0017] A third aspect of this application provides an electronic device including a processor, a memory, a user interface, and a network interface. The memory is used to store instructions, the user interface and the network interface are used to communicate with other devices, and the processor is used to execute the instructions stored in the memory to cause the electronic device to perform the method as described above.

[0018] A fourth aspect of this application provides a computer-readable storage medium storing a computer program, which is executed by a processor using the method described in any of the foregoing descriptions.

[0019] One or more technical solutions provided in the embodiments of this application have at least the following technical effects or advantages:

[0020] 1. A first time correction relationship is constructed based on the observation parameters of multiple sensor sources. Based on the first time correction relationship, the hardware timestamp synchronized to the master clock source is input into the extended state estimation model. The observation parameters of multiple sensor sources are iteratively updated based on the extended state estimation model, and a second time correction relationship is constructed based on the updated observation parameters. Based on the second time correction relationship, the pose continuous time trajectory function corresponding to the target robot is constructed. Using the continuous time trajectory function and the observation parameters of multiple sensor sources as input, the grasping path corresponding to the target robot is output through a joint optimization model with temporal consistency constraints. The target robot is controlled to perform grasping operations based on the grasping path. This solves the problem in the existing technology where the observation data cannot be strictly aligned in the time dimension due to inconsistent sampling frequencies, trigger delays, and network jitter of industrial cameras, LiDAR, inertial measurement units, and force sensors. It avoids the situation where the filter misjudges asynchronous data as real pose changes, resulting in pseudo-motion and hysteresis estimation. It achieves consistency of multi-source sensor fusion and improves the accuracy of the robot's grasping path.

[0021] 2. Acquire the local clock readings of each sensor and perform proportional correction on the local clock readings according to the frequency offset parameter; superimpose the phase offset parameter on the proportional correction result to perform phase compensation; introduce a fixed link delay parameter to perform delay compensation; map the time results after proportional correction, phase compensation and delay compensation to the master clock source to establish the first time correction relationship, thereby eliminating the time inconsistency caused by frequency drift, initial phase difference and link delay of various sensors, ensuring that the multi-source sensors output synchronously aligned observation results under a unified time base, and providing a reliable time reference for subsequent fusion and path planning.

[0022] 3. Using the continuous-time trajectory function as the first input and the observation parameters from multiple sensors as the second input, a joint optimization input set is generated. Temporal consistency constraints are introduced into the input set to ensure that the observation time of each sensor remains consistent with the time parameters of the continuous-time trajectory function. An objective function is constructed based on the input set with introduced temporal consistency constraints. The objective function includes path smoothing, obstacle avoidance, and attitude constraints. Path constraints are configured for the objective function, including approach path constraints, alignment micro-motion path constraints, closed grasping path constraints, and withdrawal path constraints. A joint optimization model is constructed based on the configured objective function to ensure that the observation results from multiple sensors and trajectory planning work synergistically under a unified time base, avoiding pseudo-motion and lag caused by time mismatch. This ensures that the generated grasping path simultaneously meets the constraints in terms of smoothness, obstacle avoidance, and attitude feasibility, thereby improving the accuracy and stability of the target robot's grasping operation. Attached Figure Description

[0023] Figure 1 This is a flowchart illustrating a multi-sensor fusion-based robot grasping path planning method provided in an embodiment of this application;

[0024] Figure 2 This is a schematic diagram of a multi-sensor fusion robot grasping path planning device provided in an embodiment of this application;

[0025] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0026] Explanation of reference numerals in the attached figures: 21. Acquisition module; 22. Processing module; 301. Processor; 302. Communication bus; 303. User interface; 304. Network interface; 305. Memory. Detailed Implementation

[0027] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0028] The terminology used in the following embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. As used in the specification of this application, the singular expressions “a,” “an,” “the,” “the,” “the,” and “this” are intended to include the plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this application refers to and includes any or all possible combinations of one or more of the listed items.

[0029] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature, and in the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more.

[0030] To enable those skilled in the art to better understand the technical solution of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings.

[0031] Please refer to Figure 1 The diagram illustrates a flowchart of a multi-sensor fusion robot grasping path planning method provided in this application embodiment. The flowchart mainly includes the following steps: S101 to S104.

[0032] Step S101: Obtain the observation parameters of the multi-source sensors, and correct the local clock of each sensor based on the observation parameters of the multi-source sensors to establish the first time correction relationship.

[0033] Specifically, this application applies to target robot grasping control scenarios driven by multi-sensor fusion, typically applied to automated grasping operations of large or complex parts in industrial production lines. In this application scenario, the main technical problem lies in the observation time mismatch caused by the inconsistency of local clocks among multiple sensors. Specifically, this manifests in two ways: firstly, slight differences in clock frequencies between different sensors cause long-term accumulated time drift; secondly, phase differences and fixed link delays exist in the triggering links of various sensors, making it impossible for observation data obtained at the same actual moment to accurately correspond. This mismatch leads to pseudo-motion in target robot posture estimation, timing disorder in the grasping path, and lag in the response of the end effector. To address these problems, step S101 obtains observation parameters from multiple sensors and corrects the local clocks of each sensor based on these parameters to establish a first-time correction relationship. In this application, the target robot is a collective term for multiple controllable execution units, which may include multi-degree-of-freedom robotic arms, mobile platforms, or collaborative robot units. The aforementioned controllable execution units can work collaboratively under unified control commands. A single robotic arm can independently complete a grasping operation, or a combination of a robotic arm and a mobile platform can perform grasping operations over a wider spatial range. Multiple collaborative robotic arms can also perform grasping tasks in parallel within the same working area. This application does not limit the specific type or number of target robots; as long as they are controllable and can perform corresponding operations under grasping path constraints, they can fall within the scope of protection of this application.

[0034] In one possible implementation, step S101 further includes: acquiring multi-source sensor observation parameters based on a multi-sensor system, wherein the multi-sensor system includes a vision sensor, a force sensor, and a lidar sensor, and the multi-source sensor observation parameters include frequency offset parameters, phase offset parameters, and fixed link delay parameters.

[0035] Specifically, the arrangement of the multi-sensor system can be configured according to the target robot's structure, operating scenario, and observation requirements, and this application does not impose any limitations. For example, the arrangement of the multi-sensor system can be as follows: a vision sensor is installed on the end effector of the target robot to collect the appearance features and spatial position of the target object; a force sensor is embedded in the gripping mechanism of the target robot to perceive the contact force and friction force during the gripping process in real time; and a lidar sensor is fixed outside the working unit of the target robot to perform spatial scanning and contour extraction of the target object and obstacles in the working area.

[0036] Multi-source sensor observation parameters include frequency offset parameters, phase offset parameters, and fixed link delay parameters. Among them, the frequency offset parameter is used to characterize the difference in operating frequency between the local clock of different sensors and the master clock, reflecting whether the timing speed is too fast or too slow; the phase offset parameter is used to characterize the difference in the starting time point between the local clock of different sensors and the master clock, reflecting the shift in timing reference; the fixed link delay parameter is used to characterize the fixed delay generated during signal processing inside the sensor and transmission to the outside.

[0037] In one possible implementation, step S101 further includes: acquiring the local clock readings of each sensor and performing a proportional correction on the local clock readings according to the frequency offset parameter; superimposing the phase offset parameter on the proportional correction result to perform phase compensation; introducing a fixed link delay parameter to perform delay compensation; and mapping the time result after proportional correction, phase compensation and delay compensation to the master clock source to construct a first time correction relationship.

[0038] Specifically, the local clock readings of each sensor are used as the input for proportional correction, and each sampling point is assigned a serial number. Identify local clock readings This process creates a sequence of local clock readings arranged chronologically, maintaining a one-to-one correspondence with the corresponding observation data, for subsequent continuous processing and mapping to the master clock source. In the scaling correction, the local clock readings are linearly scaled using a frequency offset parameter to obtain a scaling correction result sequence. The formula is as follows:

[0039]

[0040] in, This represents the frequency offset parameter, whose value is used to characterize whether the local timing speed is faster or slower relative to the master clock source. Indicates the first Local clock readings at each sampling point; This represents the time result after scaling. This operation eliminates cumulative "fast" and "slow" errors through slope scaling, ensuring the scaled result maintains a timing rate consistent with the master clock source over a long time span. In phase compensation, the phase offset parameter is superimposed on the scaled result to obtain the phase compensation result sequence. The formula is as follows:

[0041]

[0042] in, This represents the phase offset parameter, whose value is used to characterize the difference in "early or late" time between the local clock and the master clock source at the starting reference point; This represents the time result after phase compensation. This operation aligns the intercept to zero time, thus correcting the reference starting point without changing the timing speed. In delay compensation, a fixed link delay parameter is introduced to shift the phase compensation result, resulting in a delay compensation result sequence. The formula is as follows:

[0043]

[0044] in, This represents a fixed link delay parameter, whose value is used to characterize the fixed delay generated by the triggering, sampling, processing, and transmission links. This represents the time result after delay compensation. This operation removes the constant lag introduced by assembly and communication links from the time stamp, making the time stamp more closely match the physical sampling instant. During the mapping to the master clock source, the time result after scaling, phase compensation, and delay compensation is determined as the timestamp sequence mapped to the master clock source. Based on this, a first-time correction relationship is constructed, as shown in the following formula:

[0045]

[0046] in, Indicates the corresponding number Each sampling point is mapped to the timestamp of the master clock source; , , These are the frequency offset parameter, phase offset parameter, and fixed link delay parameter, respectively. This equation converts the local clock reading into a timestamp under the master clock source in the form of a monotonically affine mapping, and outputs it in sequence as a unified timestamp sequence corresponding to the first time correction relationship, which is used as the time reference for subsequent steps.

[0047] Step S102: Based on the first time correction relationship, configure hardware timestamps at the sampling front end of each sensor and synchronize the hardware timestamps to the master clock source.

[0048] Specifically, using the first-time correction relationship as input, a physical definition of the time stamp point is set at the sampling front end of each sensor, so that the hardware timestamp is latched at the exposure start edge, the laser emission pulse edge, or the analog-to-digital conversion sampling edge, thereby establishing a clock synchronization path from the master clock source to the sampling front end. This path includes master clock source output and distribution, and sampling front end phase-locking and holding. A PPS plus reference frequency or IEEE 1588 hardware timestamp scheme is used to ensure that the time base of the sampling front end is consistent with the master clock source. The propagation delay and clock deviation between the master and slave are calibrated through a round-trip measurement, and the calculated relationship is as follows:

[0049]

[0050]

[0051] in, This indicates the time when the master clock source sends the synchronization message to the sampling front end. This indicates the reception time of the message received by the sampling front end. This indicates the time at which the sampling front-end sends a delay request message to the master clock source. This indicates the time at which the master clock source receives the delay request message; This represents an estimate of the one-way propagation delay between the master and slave. This represents the estimated clock skew relative to the master clock source. The hardware capture unit of the sampling front end is enabled to generate hardware timestamps at the sampling edges. This data is then bound one-to-one with sensor observation data from the same time point. To eliminate jitter in the drive and interrupt paths, the timestamp is directly latched by the front-end capture register or time-to-digital converter and written to the buffer via direct memory access, avoiding software stack delays in entering the timestamp. Subsequently, based on the initial affine mapping provided by the first time correction relationship, the data is processed... Perform a one-time alignment correction to obtain a timestamp synchronized to the master clock source. The calculation relationship is as follows:

[0052]

[0053] in, This represents the frequency offset parameter, used to correct for deviations in the timing speed of the sampling front end. This represents the phase offset parameter, used to correct the initial reference difference between the sampling front-end and the main clock source. This represents a fixed link delay parameter used to compensate for the constant delay between sampling triggering and time latching; This represents the synchronization timestamp mapped to the master clock source. Then, using the timestamp-bound observation pairs synchronized to the master clock source as input, the synchronization quality is constrained and verified, and the synchronization stability is evaluated using a jitter synthesis upper bound.

[0054]

[0055] in, This represents an upper bound estimate of the overall jitter of timestamps after synchronization. This represents the jitter component caused by the short-term stability of the master clock source. This represents the jitter component introduced by the allocation and transmission path. This represents the jitter component introduced by the sampling front-end capture and latching process. If If the value is less than the preset threshold, the synchronization is confirmed to be valid, and the timestamp binding observation pair of the synchronization is output. If the threshold is not met, the clock path self-check and resynchronization process is triggered until the threshold is met.

[0056] Step S103: Input the hardware timestamp synchronized to the master clock source into the extended state estimation model.

[0057] Specifically, after completing the correction based on the first time correction relationship and synchronizing the hardware timestamp with the master clock source, the synchronization result is used as input data for the extended state estimation model, so that the model can be used to iteratively update the frequency offset parameter, phase offset parameter and fixed link delay parameter under a unified time base.

[0058] In one possible implementation, step S103 further includes: constructing a target correspondence and generating a time matching factor based on the target correspondence, wherein the target correspondence is the correspondence between the hardware time synchronized to the master clock source and the reference time determined by the first time correction relationship; using multi-source sensor observation parameters as graph node variables to construct a parameter node set; applying factor constraints to the parameter node set based on the time matching factor to form a factor relationship describing the observation error; writing the observation error into the factor graph in the form of a residual function based on the factor relationship; applying prior constraints to the parameter nodes in the factor graph, wherein the prior constraints are used to limit the slowly varying characteristics of the multi-source sensor observation parameters; and selecting a nonlinear least squares algorithm or an incremental inference algorithm to perform integrated operations on the factor constraints, prior constraints, and parameter node set to construct an extended state estimation model.

[0059] Specifically, using the hardware timestamp synchronized to the master clock source and the reference time determined by the first time correction relationship as input, a target correspondence is constructed, and each sampling index is... Synchronization hardware time Corresponding reference time Establish a one-to-one correspondence and output the target correspondence set. ,in, Indicates sensor identification, This indicates the sensor's first One sampling point, This represents the hardware timestamp after synchronization in step S102. This represents the reference time determined by the first time correction relationship, used as a benchmark for subsequent factor construction under a unified time base.

[0060] With target correspondence set As input, generate a set of time matching factors. and for each The observation residual function is defined as follows:

[0061]

[0062] in, The residuals representing the time matching factor, The weighting coefficient is derived from the synchronization quality assessment result of step S102 and is used to reflect the reliability of the time pair. This represents the frequency offset parameter. This represents the phase offset parameter. This represents the fixed link delay parameter; together, these three parameters constitute the multi-source sensor observation parameters in the sensor. The parameter triplet on; The corresponding affine time that maps the synchronous hardware time to the master clock reference under the current parameter estimation. This serves as the reference time. Therefore, the output set of time matching factors is determined. Its residual definition is used to apply factor constraints.

[0063] With time matching factor set As input, multi-source sensor observation parameters Used as graph node variables to construct the parameter node set and based on right Imposing factor constraints to form factor relationships In this factor constraint, the residuals are written with robust loss, and the objective function is defined as follows (for model definition only, not for solution execution):

[0064]

[0065] in, The standard function of the observed item is... This represents the robust loss function, used to suppress long-tail errors caused by outlier matching; typically, it takes the Huber or Cauchy form. It outputs the relationship between observed factors. Definition of the target function of the observed item .

[0066] With parameter node set Relationship with observed factors As input, construct a set of prior constraints. To constrain the slowly varying characteristics, the prior residual is defined as follows:

[0067]

[0068]

[0069]

[0070] in, , , These represent the prior residuals of the frequency offset parameter, the phase offset parameter, and the fixed link delay parameter, respectively. This represents the time interval between adjacent matching pairs under the master clock reference. The corresponding prior object function is defined as follows:

[0071]

[0072] in, These represent the standard deviations of the prior constraints, used to control the intensity of gradual variation. Output the set of prior constraints. With the definition of the prior objective function .

[0073] Based on the above observed factor relationships With the set of prior constraints As input, construct a factor graph. Furthermore, linearization and solution interfaces are defined at the model layer, and the normal equation form for local linearization is given below (the solution interface is used for model construction and does not perform the solution):

[0074]

[0075] in, Represents the Hessian approximation matrix. Represents the gradient term. Represents the set of parameter nodes Jacobian matrix, This represents the weighted matrix consisting of the weight coefficients and the covariance. This represents the concatenated residual vector. This represents the parameter increment variable. The output includes a model description containing the factor graph structure and the linearization interface. .

[0076] Describing by model As input, a nonlinear least squares algorithm or an incremental inference algorithm is selected, and a solution strategy and update rules are defined (such as the step size and damping interface of Gauss-Newton or Livenberg-Marquardt, or the node update order of an incremental Bayes tree). The factor constraints, prior constraints, and parameter node set are integrated and configured. Without performing parameter iteration, the extended state estimation model is constructed, and an instance of the extended state estimation model is output. This information is used in subsequent steps to iteratively update the observation parameters of the multi-source sensors.

[0077] Step S104: Iteratively update the observation parameters of the multi-source sensors based on the extended state estimation model, and recalibrate the local clocks of each sensor according to the updated multi-source sensor observation parameters to construct a second time calibration relationship.

[0078] Specifically, the constructed extended state estimation model is used as the operating environment. Taking the hardware timestamp synchronized to the master clock source and the reference time determined by the first time correction relationship as input, multiple rounds of iterative updates are performed on the frequency offset parameters, phase offset parameters, and fixed link delay parameters to obtain estimates that better reflect the actual operating state. After the update is completed, the corrected parameters are used to remap and correct the local clocks of each sensor, forming a unified time reference aligned with the master clock source, and thereby constructing the second time correction relationship.

[0079] Step S105: Construct the pose continuous time trajectory function corresponding to the target robot based on the second time correction relationship.

[0080] In one possible implementation, step S104 further includes: extracting pose data points of the target robot at different time points based on the second time correction relationship; constructing a discrete pose sequence based on the pose data points, and performing continuous fitting on the discrete pose sequence to generate a pose function expression that can be queried in the entire time domain; performing constraint verification on the pose function expression, and proposing function solutions that do not meet the constraint verification, the constraint verification including smoothness constraint verification and physical reachability constraint verification; and using the pose function expression after removing the function solutions as the pose continuous time trajectory function.

[0081] Specifically, the second time correction relationship is applied to the timestamps corresponding to each sensor to obtain the observation sequence under a unified time base. The observation sequence under the unified time base is then input into the pose solving routine, which outputs the set of pose data points of the target robot at different time points time by time. ,in Indicates the time of the target robot's end effector The pose at that location serves as the input for subsequently constructing a discrete pose sequence.

[0082] Using a set of pose data points as input, a discrete pose sequence is constructed in chronological order, and a continuous fit is performed on the discrete pose sequence to generate a pose function expression that can be queried in the entire time domain. To ensure consistent fitting between position and orientation, weighted fitting and smoothing regularization terms are established for position and rotation, respectively. The objective function is as follows:

[0083]

[0084] in, This indicates the position trajectory of the target robot's end effector. This indicates the position within the pose data point. The weight matrix representing the location observations. Indicates the positional smoothing regularization weights. The third derivative of position with respect to time is given. Indicates attitude trajectory, This represents the pose of the pose data points. Represent the logarithmic mapping from Lie groups to Lie algebras to measure rotation error. The weight matrix represents the attitude observation. Indicates the pose smoothing regularization weights. Indicates angular velocity. This represents angular acceleration. The above objectives constrain the fitting accuracy with observational consistency terms and the dynamic realizability of the trajectory with smoothing regularization terms, through... and Solve using either spline or Gaussian process parameterization, and output the pose function expression. .

[0085] Using the pose function expression as input, constraint verification is performed, and function solutions that do not meet the constraint verification requirements are eliminated. Constraint verification includes two categories: smoothness constraint verification and physical reachability constraint verification. The criteria are as follows:

[0086]

[0087]

[0088] in, This indicates the maximum allowable linear acceleration of the target robot's end effector. Indicates the maximum permissible linear jerk. Indicates the maximum permissible angular velocity. and Let represent the second and third time derivatives of the position, respectively. Indicates at the discrete verification time Through inverse kinematics The obtained joint position, and Indicates the upper and lower limits of joint position. and These represent the upper limits of joint velocity and joint acceleration, respectively. and This corresponds to the derivative. The first line of the criterion above corresponds to the smoothness constraint verification, used to limit dynamic spikes in both linear and angular directions, ensuring... exist The second criterion is the executability in a meaningful sense; the physical reachability constraint verification is performed by verifying that the pose function expression can be realized within the mechanism and drive boundary through discrete-time inverse kinematics and boundary checks.

[0089] Step S106: Using the continuous time trajectory function and multi-source sensor observation parameters as input, the joint optimization model with temporal consistency constraints outputs the grasping path corresponding to the target robot.

[0090] Specifically, the pose-continuous-time trajectory function and multi-source sensor observation parameters are jointly input into the constructed joint optimization model. A temporal consistency constraint is introduced under a unified time base to ensure a strict correspondence between the trajectory function and the multi-source observation data in the time dimension. Based on this input, the joint optimization model performs weighted solutions for multiple objectives, including path smoothness, obstacle avoidance, and attitude constraints. Under the constraints of approach path, alignment micro-motion path, closed grasping path, and withdrawal path, it generates a grasping path that satisfies both physical executability and dynamic reachability.

[0091] In one possible implementation, step S106 further includes: using a continuous-time trajectory function as a first input and multi-source sensor observation parameters as a second input to generate a joint optimization input set; introducing temporal consistency constraints into the input set, which are used to limit the observation time of each sensor to be consistent with the time parameters of the continuous-time trajectory function; constructing an objective function based on the input set with introduced temporal consistency constraints, the objective function including path smoothing terms, obstacle avoidance terms, and attitude constraint terms; configuring path constraints for the objective function, the path constraints including approach path constraints, alignment micro-motion path constraints, closed grasping path constraints, and evacuation path constraints; and constructing a joint optimization model based on the configured objective function.

[0092] Specifically, the pose continuous time trajectory function obtained in step S104 is used as the first input, and the multi-source sensor observation parameters determined in steps S101 to S104 are used as the second input to form an initial input set. The initial input set contains both trajectory parameters and sensor observation parameters in the time dimension.

[0093] Using the initial input set as input, a temporal consistency constraint is introduced to ensure that the observation timestamp of each sensor strictly corresponds to the time parameter in the pose continuous-time trajectory function, outputting an input set with temporal consistency constraints. This constraint ensures that multi-source data and the trajectory function are aligned under a unified time base, avoiding pseudo-motion or path drift caused by sensor outputs of different frequencies and phases.

[0094] Using an input set with temporal consistency constraints as input, an objective function is constructed, consisting of a path smoothing term, an obstacle avoidance term, and an attitude constraint term. The path smoothing term limits the higher-order derivatives of the continuous-time trajectory function, ensuring the continuity of the trajectory in terms of velocity and acceleration. The obstacle avoidance term, based on spatial information from the LiDAR sensor, constrains the minimum safe distance between the trajectory and obstacles. The attitude constraint term ensures that the robot's end effector maintains the same attitude as the surface normal of the target object during the grasping process, thereby ensuring grasping accuracy and stability.

[0095] Using the objective function as input, path constraints are configured, including approach path constraints, alignment micro-motion path constraints, closed grasping path constraints, and withdrawal path constraints. The approach path constraint ensures that the target robot gradually approaches the target object along a smooth, collision-free trajectory; the alignment micro-motion path constraint ensures that the minute adjustments to the trajectory near the grasping point meet the sensor's positioning accuracy requirements; the closed grasping path constraint ensures that the end effector completes a stable closing action at the grasping point; and the withdrawal path constraint ensures that the target robot smoothly withdraws after grasping and avoids obstacle areas.

[0096] By using the input set with temporal consistency constraints, the objective function, and path constraints as components, a joint optimization model is constructed with the aim of solving the objective function. This model can simultaneously satisfy trajectory smoothness, obstacle avoidance, and posture feasibility under the condition of temporal alignment, and outputs a grasping path consisting of an approach path, an alignment micro-motion path, a closed grasping path, and an escape path, which serves as the basis for the target robot to perform the grasping operation.

[0097] Step S107: Control the target robot to perform a grasping operation based on the grasping path.

[0098] Specifically, the grasping path is input to the target robot controller and parsed into control commands for four stages: approach path, alignment micro-motion path, closed grasping path, and withdrawal path. During the approach path stage, the controller drives the target robot's end effector closer to the target object time-by-time according to the pose continuous-time trajectory function. During the alignment micro-motion path stage, the end effector's pose is dynamically corrected based on multi-source sensor observation parameters to ensure the grasping direction remains consistent with the surface normal of the target object. During the closed grasping path stage, a closing command is issued to the gripping mechanism, and stable gripping is maintained after the force sensor feedback meets a preset threshold. During the withdrawal path stage, the controller drives the target robot's end effector away from the target object area along a predetermined trajectory, thus completing the entire grasping operation.

[0099] In one possible implementation, step S107 further includes: decomposing the grasping path into one or more control command sequences including an approach path, an alignment micro-motion path, a closed grasping path, and a withdrawal path; and controlling the target robot to perform a grasping operation based on one or more control command sequences.

[0100] Specifically, the grasping path is decomposed into one or more control command sequences, including an approach path, an alignment micro-motion path, a closed grasping path, and an withdrawal path. When the grasping path is decomposed into an approach path control command sequence, the controller outputs pose commands generated by a continuous time trajectory function at each moment, driving the target robot's end effector to approach the target object along a smooth trajectory, while maintaining obstacle avoidance conditions during the approach. When the grasping path is decomposed into an alignment micro-motion path control command sequence, the controller dynamically corrects the pose of the end effector by combining observation parameters from multiple sources of sensors, adjusting the tool posture to be consistent with the normal of the grasping point in a small space, ensuring that the grasping direction remains aligned with the surface of the target object. When the grasping path is decomposed into a closing grasping path control command sequence, the controller issues a closing command to the gripping mechanism and monitors force sensor feedback. When the contact force or gripping force reaches a preset threshold, it maintains a stable gripping state to prevent slippage or damage to the target object. When the grasping path is decomposed into a withdrawal path control command sequence, the controller drives the end effector to move away from the target object area along a withdrawal trajectory, while ensuring that it does not collide with surrounding obstacles during the withdrawal process, and after the path is completed, it moves the target object to a predetermined position or delivers it to a subsequent process. It should be noted that the target robot in this application performs grasping operations based on one or more control command sequences. Specifically, in practice, the target robot can complete simple grasping operations using only an approach path control command sequence and a closed grasping path control command sequence, or it can perform precise grasping operations in complex environments by combining a complete control command sequence of approach path, alignment micro-motion path, closed grasping path, and withdrawal path. This application does not limit the combination of control command sequences invoked by the target robot; as long as the grasping operation is completed under constraints of smoothness, stability, and reachability, it falls within the scope of protection of this application.

[0101] Please refer to Figure 2 This illustration shows a schematic diagram of a multi-sensor fusion robot grasping path planning device according to an embodiment of this application. The device includes an acquisition module 21 and a processing module 22, wherein...

[0102] The acquisition module 21 is used to acquire observation parameters from multiple sensors and to correct the local clock of each sensor based on the observation parameters to construct a first time correction relationship. Based on the first time correction relationship, a hardware timestamp is configured at the sampling front end of each sensor and synchronized to the master clock source. The hardware timestamp synchronized to the master clock source is input into the extended state estimation model.

[0103] The processing module 22 is used to iteratively update the observation parameters of the multi-source sensors based on the extended state estimation model, and recalibrate the local clock of each sensor according to the updated multi-source sensor observation parameters to construct a second time calibration relationship; construct the pose continuous time trajectory function corresponding to the target robot based on the second time calibration relationship; take the continuous time trajectory function and the multi-source sensor observation parameters as input, and output the grasping path corresponding to the target robot through a joint optimization model with time consistency constraints; and control the target robot to perform grasping operations based on the grasping path.

[0104] In one possible implementation, the acquisition module 21 is used to acquire multi-source sensor observation parameters, specifically including: acquiring multi-source sensor observation parameters based on a multi-sensor system, wherein the multi-sensor system includes a vision sensor, a force sensor, and a lidar sensor, and the multi-source sensor observation parameters include frequency offset parameters, phase offset parameters, and fixed link delay parameters.

[0105] In one possible implementation, the acquisition module 21 is used to correct the local clock of each sensor based on the observation parameters of the multi-source sensors to construct a first time correction relationship. Specifically, it includes: acquiring the local clock readings of each sensor and proportionally correcting the local clock readings according to the frequency offset parameter; superimposing the phase offset parameter on the proportional correction result to perform phase compensation; introducing a fixed link delay parameter to perform delay compensation; and mapping the time result after proportional correction, phase compensation and delay compensation to the master clock source to construct the first time correction relationship.

[0106] In one possible implementation, the acquisition module 21 is used to construct an extended state estimation model, specifically including: constructing a target correspondence and generating a time matching factor based on the target correspondence, wherein the target correspondence is the correspondence between the hardware time synchronized to the master clock source and the reference time determined by the first time correction relationship; using multi-source sensor observation parameters as graph node variables to construct a parameter node set; applying factor constraints to the parameter node set based on the time matching factor to form a factor relationship describing the observation error; writing the observation error into the factor graph in the form of a residual function based on the factor relationship; applying prior constraints to the parameter nodes in the factor graph, wherein the prior constraints are used to limit the slowly varying characteristics of the multi-source sensor observation parameters; and selecting a nonlinear least squares algorithm or an incremental inference algorithm to perform integrated operations on the factor constraints, prior constraints, and parameter node set to construct the extended state estimation model.

[0107] In one possible implementation, the processing module 22 is used to construct a pose continuous-time trajectory function corresponding to the target robot based on a second time correction relationship. Specifically, this includes: extracting pose data points of the target robot at different time points based on the second time correction relationship; constructing a discrete pose sequence based on the pose data points, and performing continuous fitting on the discrete pose sequence to generate a pose function expression that can be queried in the entire time domain; performing constraint verification on the pose function expression and proposing function solutions that do not meet the constraint verification, the constraint verification including smoothness constraint verification and physical reachability constraint verification; and using the pose function expression after removing function solutions as the pose continuous-time trajectory function.

[0108] In one possible implementation, the processing module 22 is used to construct a joint optimization model before outputting the grasping path corresponding to the target robot through a joint optimization model with temporal consistency constraints, using the continuous-time trajectory function and multi-source sensor observation parameters as inputs. Specifically, this includes: generating a joint optimization input set by taking the continuous-time trajectory function as the first input and the multi-source sensor observation parameters as the second input; introducing temporal consistency constraints into the input set, which are used to limit the observation time of each sensor to be consistent with the time parameters of the continuous-time trajectory function; constructing an objective function based on the input set with introduced temporal consistency constraints, the objective function including path smoothing terms, obstacle avoidance terms, and attitude constraint terms; configuring path constraints for the objective function, including approach path constraints, alignment micro-motion path constraints, closed grasping path constraints, and withdrawal path constraints; and constructing a joint optimization model based on the configured objective function.

[0109] In one possible implementation, the processing module 22 is used to control the target robot to perform a grasping operation based on the grasping path, specifically including: decomposing the grasping path into one or more control command sequences including an approach path, an alignment micro-motion path, a closed grasping path, and a withdrawal path; and controlling the target robot to perform a grasping operation based on one or more control command sequences.

[0110] It should be noted that the above embodiments of the apparatus are only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0111] This application also provides an electronic device. (See reference...) Figure 3 , Figure 3This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device may include: at least one processor 301, at least one communication bus 302, a user interface 303, at least one network interface 304, and a memory 305.

[0112] The communication bus 302 is used to enable communication between these components.

[0113] The user interface 303 may include a display screen and a camera. Optionally, the user interface 303 may also include a standard wired interface and a wireless interface.

[0114] The network interface 304 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface).

[0115] The processor 301 may include one or more processing cores. The processor 301 connects to various parts of the server using various interfaces and lines, and performs various server functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in memory 305, and by calling data stored in memory 305. Optionally, the processor 301 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 301 may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content required for display; and the modem handles wireless communication. It is understood that the modem may also not be integrated into the processor 301 and may be implemented as a separate chip.

[0116] The memory 305 may include random access memory (RAM) or read-only memory. Optionally, the memory 305 may include a non-transitory computer-readable storage medium. The memory 305 may be used to store instructions, programs, code, code sets, or instruction sets. The memory 305 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the above-described method embodiments, etc.; the data storage area may store data involved in the above-described method embodiments, etc. Optionally, the memory 305 may also be at least one storage device located remotely from the aforementioned processor 301. (Refer to...) Figure 3 The memory 305, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and a multi-sensor fusion robot grasping path planning application.

[0117] exist Figure 3 In the illustrated electronic device, the user interface 303 is primarily used to provide an input interface for the user and acquire user input data; while the processor 301 can be used to call the robot grasping path planning application stored in the memory 305, which incorporates multiple sensor fusion. When executed by one or more processors 301, the electronic device performs one or more of the methods described in the above embodiments. It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps can be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0118] This application also provides a computer-readable storage medium storing instructions. When executed by one or more processors, these instructions cause an electronic device to perform one or more of the methods described in the above embodiments.

[0119] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0120] In the various embodiments provided in this application, it should be understood that the disclosed apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some service interface; the indirect coupling or communication connection between apparatuses or units may be electrical or other forms.

[0121] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0122] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0123] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as USB flash drives, portable hard drives, magnetic disks, or optical disks.

[0124] The above description is merely an exemplary embodiment disclosed in this application and should not be construed as limiting the scope of this application. Any equivalent changes and modifications made in accordance with the teachings of this application shall still fall within the scope of this application.

[0125] This application is intended to cover any variations, uses, or adaptations disclosed herein that follow the general principles disclosed herein and include common knowledge or customary technical means in the art that are not described in this application.

Claims

1. A multi-sensor fusion-based robot grasping path planning method, characterized in that, The method comprises: obtaining multi-source sensor observation parameters based on a multi-sensor system, the multi-sensor system comprising a visual sensor, a force sensor and a laser radar sensor, the multi-source sensor observation parameters comprising a frequency offset parameter, a phase offset parameter and a fixed link time delay parameter; and correcting local clocks of each sensor based on the multi-source sensor observation parameters to construct a first time correction relationship, specifically comprising: obtaining local clock readings of each sensor, and proportionally correcting the local clock readings according to the frequency offset parameter; superimposing the phase offset parameter based on the proportionally corrected result to perform phase compensation; introducing the fixed link time delay parameter to perform time delay compensation; and mapping time results after the proportionally correction, the phase compensation and the time delay compensation to a master clock source to construct the first time correction relationship; based on the first time correction relationship, configuring a hardware timestamp at a sampling front end of each sensor, and synchronizing the hardware timestamp to the master clock source; inputting the hardware timestamp synchronized to the master clock source into an extended state estimation model; iteratively updating the multi-source sensor observation parameters based on the extended state estimation model, and re-correcting the local clocks of each sensor based on the updated multi-source sensor observation parameters to construct a second time correction relationship; based on the second time correction relationship, constructing a pose continuous time trajectory function corresponding to a target robot, specifically comprising: extracting pose data points corresponding to the target robot at different time points based on the second time correction relationship; constructing a discrete pose sequence based on the pose data points, and continuously fitting the discrete pose sequence to generate a pose function expression that can be queried in a full time domain; performing constraint verification on the pose function expression, and proposing a function solution that does not satisfy the constraint verification, the constraint verification comprising smoothness constraint verification and physical reachability constraint verification; taking the pose function expression after eliminating the function solution as the pose continuous time trajectory function; inputting the continuous time trajectory function and the multi-source sensor observation parameters, and outputting a grasping path corresponding to the target robot through a joint optimization model of time sequence consistency constraints; controlling the target robot to perform a grasping operation based on the grasping path.

2. The method of claim 1, wherein, constructing the extended state estimation model, specifically comprising: constructing a target correspondence relationship, and generating a time matching factor based on the target correspondence relationship, the target correspondence relationship being a correspondence relationship between the hardware time synchronized to the master clock source and a reference time determined by the first time correction relationship; taking the multi-source sensor observation parameters as graph node variables to construct a parameter node set; applying a factor constraint to the parameter node set based on the time matching factor to form a factor relationship describing an observation error; based on the factor relationship, writing the observation error in the form of a residual function into a factor graph; applying a priori constraint to the parameter nodes in the factor graph, the a priori constraint being used to limit the slowly varying characteristics of the multi-source sensor observation parameters; The nonlinear least squares algorithm or the incremental inference algorithm is selected to integrate the factor constraints, the prior constraints and the parameter node set to construct the extended state estimation model.

3. The method of claim 1, wherein, Before the joint optimization model is output with the continuous time trajectory function and the multi-source sensor observation parameter as input, the joint optimization model is constructed, and the joint optimization model comprises the following steps: The continuous time trajectory function is taken as a first input, and the multi-source sensor observation parameter is taken as a second input to generate an input set of joint optimization; The time sequence consistency constraint is introduced in the input set, and the time sequence consistency constraint is used to limit the observation time of each sensor to be consistent with the time parameter of the continuous time trajectory function; A target function is constructed based on the input set in which the time sequence consistency constraint is introduced, and the target function comprises a path smoothing term, an obstacle avoidance term and a pose constraint term; The target function is configured with a path constraint condition, and the path constraint condition comprises an approach path constraint, an alignment micro-motion path constraint, a closed grasp path constraint and a retreat path constraint; The joint optimization model is constructed based on the configured target function.

4. The method of claim 1, wherein, The target robot is controlled to perform a grasping operation based on the grasping path, and the grasping operation comprises the following steps: The grasping path is decomposed into one or more control instruction sequences comprising an approach path, an alignment micro-motion path, a closed grasp path and a retreat path; The target robot is controlled to perform a grasping operation based on one or more control instruction sequences. 5.A multi-sensor fusion robot grasping path planning device, characterized by, The device comprises an acquisition module and a processing module, wherein The acquisition module is configured to acquire multi-source sensor observation parameters based on a multi-sensor system, the multi-sensor system comprising a vision sensor, a force sensor and a laser radar sensor, the multi-source sensor observation parameters comprising a frequency offset parameter, a phase offset parameter and a fixed link time delay parameter, and correct local clocks of each sensor based on the multi-source sensor observation parameters to construct a first time correction relationship, specifically comprising: acquiring local clock readings of each sensor, and proportionally correcting the local clock readings according to the frequency offset parameter; superimposing the phase offset parameter based on the proportionally corrected result to perform phase compensation; introducing the fixed link time delay parameter to perform time delay compensation; mapping time results after the proportionally correction, the phase compensation and the time delay compensation to a master clock source to construct the first time correction relationship; based on the first time correction relationship, configuring a hardware time stamp at a sampling front end of each sensor, and synchronizing the hardware time stamp to the master clock source; inputting the hardware time stamp synchronized to the master clock source into an extended state estimation model; The processing module is configured to iteratively update the multi-source sensor observation parameters based on the extended state estimation model, and recalibrate local clocks of each sensor according to the updated multi-source sensor observation parameters to construct a second time correction relationship; based on the second time correction relationship, a pose continuous-time trajectory function corresponding to the target robot is constructed, specifically including: based on the second time correction relationship, pose data points corresponding to the target robot at different time points are extracted; based on the pose data points, a discrete pose sequence is constructed, and the discrete pose sequence is continuously fitted to generate a pose function expression that can be queried in the entire time domain; the pose function expression is subjected to constraint verification, and a function solution that does not satisfy the constraint verification is proposed, the constraint verification including smoothness constraint verification and physical reachability constraint verification; the pose function expression after the function solution is eliminated is taken as the pose continuous-time trajectory function; the continuous-time trajectory function and the multi-source sensor observation parameters are taken as inputs, and a joint optimization model of time sequence consistency constraint is used to output a grasping path corresponding to the target robot; and the target robot is controlled to perform a grasping operation based on the grasping path.

6. An electronic device, comprising: The electronic device includes a processor, a communication bus, a user interface, a network interface, and a memory, the memory is configured to store instructions, the user interface and the network interface are configured to communicate with other devices, and the processor is configured to execute the instructions stored in the memory to enable the electronic device to perform the method of any one of claims 1 to 4.

7. A computer readable storage medium characterized by The computer-readable storage medium stores instructions that, when executed, perform the method of any one of claims 1 to 4.

Images