A robot viewpoint intelligent planning method and system of deformation-viewpoint dynamic coupling

Abstract

A deformation-viewpoint dynamic coupling robot viewpoint intelligent planning method and system, the method comprising: constructing a deformation-view mapping model and training; constructing a space-time graph convolution network and training; constructing a multi-objective hierarchical decoupling optimization network, inputting the multi-modal information in the actual into the trained deformation-view mapping model, generating a coverage prediction value, generating a compensated view according to the deformation compensation layer; the motion optimization layer uses a deep reinforcement learning strategy network and combines an adaptive importance sampling strategy to screen the compensated view and generate a screened candidate viewpoint set; generating a space-time safety corridor according to the trained space-time graph convolution network, scoring the collision risk of the screened candidate viewpoint set according to the safety constraint layer, constructing a multi-objective fitness function and solving, obtaining the optimal viewpoint set. The present application is suitable for high-precision visual detection scene, which significantly improves the real-time performance and robustness of robot viewpoint planning.

Description

A Deformation-View Dynamic Coupling Intelligent Robot Viewpoint Planning Method and System Technical Field

[0001] This invention relates to the field of robot vision inspection and autonomous motion control technology, and in particular to a robot viewpoint intelligent planning method and system with deformation-viewpoint dynamic coupling. Background Technology

[0002] In the field of robot vision inspection, traditional viewpoint planning methods are based on static environment assumptions, completing target measurement through preset paths or fixed perspectives. However, in dynamic assembly scenarios (such as parts welding and composite material assembly), the following problems arise between the robot end effector and the assembly target due to factors such as material deformation and motion disturbances:

[0003] Viewpoint shift leads to measurement failure: The deformation gradient of the assembly causes dynamic changes in the geometric features of the target surface, and static viewpoint planning cannot compensate for the attenuation of field coverage caused by deformation in real time.

[0004] High-dimensional solution space optimization problem: Viewpoint planning needs to simultaneously satisfy multiple objective constraints such as deformation compensation, motion smoothness, and obstacle avoidance. Traditional optimization methods are difficult to solve efficiently in dynamic and time-varying scenarios.

[0005] Dynamic obstacle interference: The random motion patterns of obstacles such as mobile devices and personnel in the assembly environment threaten the safety of robot movement, and existing methods lack the ability to predict nonlinear trajectories.

[0006] In existing technologies, viewpoint planning methods based on fixed deformation compensation models or single-objective optimization suffer from poor real-time performance and insufficient robustness, making it difficult to meet the high-precision inspection requirements of dynamic assembly scenarios. Summary of the Invention

[0007] This invention provides a robot viewpoint intelligent planning method and system with deformation-viewpoint dynamic coupling to solve the technical problems mentioned in the background art.

[0008] To achieve the above objectives, the technical solution of the present invention is implemented as follows:

[0009] This invention provides a robot viewpoint intelligent planning method based on deformation-viewpoint dynamic coupling, comprising the following steps:

[0010] S1. Collect multimodal information, including three-dimensional deformation gradient tensor, true value of field of view coverage, environmental parameter vector and corresponding optimal observation attitude;

[0011] S2. Construct a deformation-viewpoint mapping model, which includes a multimodal input layer, a residual feature extraction module, a multi-head attention mechanism layer, and a viewpoint prediction output layer connected in sequence; train the deformation-viewpoint mapping model using multimodal data;

[0012] S3. Acquire multimodal sensing data and construct a spatiotemporal graph convolutional network. Train the spatiotemporal graph convolutional network using the multimodal sensing data.

[0013] S4. Construct a multi-objective hierarchical decoupling optimization network, which includes a deformation compensation layer, a motion optimization layer, and a safety constraint layer connected in sequence; deploy the trained deformation-viewpoint mapping model, the trained spatiotemporal graph convolutional network, and the multi-objective hierarchical decoupling optimization network onto the robot system.

[0014] S5. Scan the real 3D scene to obtain a scanned point cloud model, extract surface voxel features from the scanned point cloud model, input the real-world multimodal information into the trained deformation-viewpoint mapping model to generate coverage prediction values, and then input them together with the surface voxel features into the deformation compensation layer to generate compensated viewpoints; the motion optimization layer uses a deep reinforcement learning policy network combined with an adaptive importance sampling strategy to filter the compensated viewpoints and generate a set of candidate viewpoints; a spatiotemporal safety corridor is generated based on the trained spatiotemporal graph convolutional network, and the spatiotemporal safety corridor is input into the safety constraint layer to perform collision risk scoring on the selected candidate viewpoint set, output a safe and feasible subset of viewpoints, and use the viewpoint subset to construct a multi-objective fitness function and solve it to select the Pareto optimal set of viewpoints;

[0015] S6. Combine the Pareto optimal viewpoint set to output the optimal end-effector pose parameters that meet the requirements, and adjust the robot system to the specified viewpoint based on the optimal end-effector pose parameters.

[0016] Furthermore, the multimodal input layer is used to receive multimodal data and perform standardization processing on the multimodal data;

[0017] The residual feature extraction module consists of four residual connection layers. Each residual connection layer contains a three-dimensional convolutional layer, a batch normalization layer, and a ReLU activation function connected in sequence. The residual feature extraction module is used to extract features from the three-dimensional deformation gradient tensor, the ground truth of the field of view coverage, and the environment parameter vector to obtain multiple different feature vectors.

[0018] The multi-head attention mechanism layer includes four parallel attention heads, which are used to dynamically assign weights to different deformation vectors so that the deformation-view mapping model focuses on the region whose influence on view coverage reaches a set threshold, and generates a weighted feature vector.

[0019] The view prediction output layer is used to map the weighted feature vector through a fully connected layer (FC) to obtain the view sensitivity index and coverage prediction value.

[0020] Furthermore, the multimodal sensing data includes lidar point clouds, visual images, and force sensor signals.

[0021] Furthermore, the spatiotemporal graph convolutional network includes a spatiotemporal graph convolutional network, a bidirectional gated recurrent unit, and an attention mechanism connected in sequence;

[0022] The spatiotemporal graph convolutional network is used to capture the temporal correlation of the velocity, acceleration, and direction changes of the obstacle's motion trajectory through spatiotemporal graph convolutional layers, thereby obtaining temporal features;

[0023] The spatiotemporal graph convolutional network is used to fuse historical trajectory data and multimodal perception data, calculate the probability of obstacle presence at different time steps, and generate a spatiotemporal safety corridor.

[0024] Furthermore, the spatiotemporal graph convolutional network trained in S3 is expressed as follows:

[0025] ;

[0026] in, This represents the prediction of future state sequences by a spatiotemporal graph convolutional network, i.e., a spatiotemporal safety corridor. Represents a sequence of historical states; subscript Indicates time step The subscript T indicates the length of the historical observation window; the subscript H indicates the length of the predicted horizon. This represents the spatiotemporal graph convolutional network after training. This represents Gaussian noise.

[0027] Furthermore, step S5 specifically includes the following steps:

[0028] S51. Scan the real 3D scene to obtain a scanned point cloud model, and extract surface voxel features from the scanned point cloud model.

[0029] S52. The data acquisition module of the robot system obtains multimodal information in reality and inputs it into the trained deformation-viewpoint mapping model to generate coverage prediction values.

[0030] S53. Input the coverage prediction value into the deformation compensation layer, and generate viewpoint compensation parameters based on the surface voxel features. Then, based on the viewpoint compensation parameters From the current perspective Compensation is performed to obtain the compensated perspective. ;

[0031] S54. The motion optimization layer uses a deep reinforcement learning policy network to optimize the viewpoint in the pose space after compensation. Gaussian noise is added to generate a candidate viewpoint set; then, an adaptive importance sampling strategy is used to filter the candidate viewpoint set, resulting in the filtered candidate viewpoint set.

[0032] S55. Construct the optimization objective function for the filtered candidate viewpoint set, as follows:

[0033] ;

[0034] in, Indicates the optimization objective. Indicates the viewpoint. , As weight, and , ; The visibility coverage of key feature regions, The cost of sports;

[0035] S56. Obtain the optimized candidate viewpoints by solving the objective function. Verify the optimized candidate viewpoints The cumulative collision probability of the corresponding trajectory. If the cumulative collision probability is greater than the set threshold, return S53; otherwise, output the verified candidate viewpoint.

[0036] S57. Call the robot kinematics model and convert the verified candidate viewpoints into joint space coordinates; the robot system moves according to the joint space coordinates, and during the movement, the data acquisition module on the robot system collects the pose data of the dynamic obstacles in real time, and then uses the Kalman filter algorithm to control the measurement error of the pose data of the dynamic obstacles within the set range.

[0037] The pose data of dynamic obstacles is transmitted to a spatiotemporal graph convolutional network trained in an industrial control computer to calculate the probability of obstacle existence at different time steps and generate a spatiotemporal safety corridor.

[0038] S58. Input the spatiotemporal safety corridor into the safety constraint layer, perform collision risk scoring on the selected candidate viewpoint set, and output a safe and feasible subset of viewpoints. During this process, if the collision risk score is lower than the set collision threshold, return to S54 and regenerate the candidate viewpoint set.

[0039] S59. Establish a multi-objective fitness function using a subset of viewpoints, solve the multi-objective fitness function using a non-dominated sorting genetic algorithm, and select the Pareto optimal viewpoint set.

[0040] Furthermore, the probability expression for the existence of obstacles at different time steps in S57 is as follows:

[0041] ;

[0042] in, This represents the probability of an obstacle existing at different time steps, or the probability of a possible collision. This represents the position of the robot system at time step t; k represents the k-th Gaussian component; K represents the total number of Gaussian components. Indicates scale; Indicated by The mean is Gaussian density on the scale of .

[0043] Furthermore, the expression for the spatiotemporal security corridor in S57 is as follows:

[0044] ;

[0045] in, This represents a set of spatial points that meet safety conditions, i.e., a spacetime safety corridor. Indicates spatial location, Represents the set of real numbers; This represents the collision probability at time step t at control position q, where 0.12 is the maximum allowed collision probability set manually.

[0046] Furthermore, the multi-objective fitness function in S59 is specifically as follows:

[0047] ;

[0048] in, For the Jacobian matrix of the robotic arm; Indicates the visibility rate of feature points; A normalized measure of collision risk; This represents a pose variable, which indicates the position and orientation of a rigid body in three-dimensional space. The form in which the pose variable P is expressed; This represents the F-norm.

[0049] In another aspect, the present invention provides a robot viewpoint intelligent planning system, including a robot system configured to perform a robot viewpoint intelligent planning method.

[0050] The beneficial effects of this invention are:

[0051] 1. This invention discloses an intelligent robot viewpoint planning method based on deformation-viewpoint dynamic coupling. By constructing a deep learning-driven deformation-viewpoint mapping model, it achieves intelligent matching of dynamic deformation and viewpoint parameters. Simultaneously, this invention employs a multi-objective hierarchical decoupling optimization network, improving computational efficiency while ensuring planning accuracy. Furthermore, it integrates a spatiotemporal graph convolutional network and a deep reinforcement learning policy network to enhance dynamic obstacle prediction and safe obstacle avoidance capabilities. In addition, this invention is applicable to high-precision visual inspection scenarios, significantly improving the real-time performance and robustness of robot viewpoint planning.

[0052] 2. This invention reveals the nonlinear relationship between deformation gradient and view coverage through deep learning, breaking through the view offset bottleneck of static planning and improving the measurement accuracy in dynamic scenes.

[0053] 3. This invention employs a multi-objective hierarchical decoupling optimization network to decompose the high-dimensional solution space into a three-layer hierarchical structure, namely, a deformation compensation layer, a motion optimization layer, and a safety constraint layer. The three-layer hierarchical structure significantly improves computational efficiency through dimensionality reduction mapping and parallel solution, thus meeting the requirements of real-time planning.

[0054] 4. This invention effectively predicts the trajectory of nonlinear moving obstacles by integrating spatiotemporal graph convolutional networks and attention mechanisms, generating probabilistic safe corridors and enhancing the safety and reliability of planning results. Attached Figure Description

[0055] Figure 1 is a structural block diagram of the deformation-viewpoint mapping model in this invention;

[0056] Figure 2 is a flowchart of the overall process of the robot viewpoint intelligent planning method in this invention. Detailed Implementation

[0057] To facilitate understanding of the present invention, a more complete description will be given below with reference to the accompanying drawings. Preferred embodiments of the invention are shown in the drawings. However, the invention can be implemented in many other different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to provide a thorough and complete understanding of the disclosure of the invention.

[0058] Referring to Figures 1 and 2, this application provides a deformation-viewpoint dynamic coupling intelligent planning method for robot viewpoints, including the following steps:

[0059] S1. Acquire multimodal information, including a three-dimensional deformation gradient tensor (including radial stress, shear stress, and bending curvature components), ground truth of field of view coverage, environmental parameter vectors, and the corresponding optimal observation attitude (including azimuth, pitch, and translation vectors); the dimension of the three-dimensional deformation gradient tensor is... ; Let represent the set of real numbers; where the corresponding optimal observation pose serves as the training label.

[0060] S2. Construct a deformation-viewpoint mapping model using the PyTorch (deep learning) framework. The deformation-viewpoint mapping model includes a multimodal input layer, a residual feature extraction module, a multi-head attention mechanism layer, and a viewpoint prediction output layer connected in sequence. Train the deformation-viewpoint mapping model using multimodal data.

[0061] Specifically, the deformation-view mapping model is trained end-to-end. The training data covers a large-scale experimental dataset of various materials such as metals, composites, and elastomers, covering multiple stages such as part assembly, welding, and inspection. With the goal of minimizing the view coverage prediction error, the network is trained through supervised learning to improve its generalization ability across working conditions.

[0062] S3. Acquire multimodal sensing data and construct a spatiotemporal graph convolutional network. Train the spatiotemporal graph convolutional network using the multimodal sensing data.

[0063] S4. Construct a multi-objective hierarchical decoupling optimization network, which includes a deformation compensation layer, a motion optimization layer, and a safety constraint layer connected in sequence; deploy the trained deformation-view mapping model, the trained spatiotemporal graph convolutional network, and the multi-objective hierarchical decoupling optimization network onto the industrial control computer (such as an Intel i7-12800HX) of the robot system.

[0064] S5. Scan the real 3D scene to obtain a scanned point cloud model, extract surface voxel features from the scanned point cloud model, input the real-world multimodal information into the trained deformation-viewpoint mapping model to generate coverage prediction values, and then input them together with the surface voxel features into the deformation compensation layer to generate compensated viewpoints; the motion optimization layer uses a deep reinforcement learning policy network combined with an adaptive importance sampling strategy to filter the compensated viewpoints and generate a set of candidate viewpoints; a spatiotemporal safety corridor is generated based on the trained spatiotemporal graph convolutional network, and the spatiotemporal safety corridor is input into the safety constraint layer to perform collision risk scoring on the selected candidate viewpoint set, output a safe and feasible subset of viewpoints, and use the viewpoint subset to construct a multi-objective fitness function and solve it to select the Pareto optimal set of viewpoints;

[0065] S6. Combine the Pareto optimal viewpoint set to output the optimal end-effector pose parameters that meet the requirements of real-time performance and robustness, and adjust the robot system to the specified viewpoint based on the optimal end-effector pose parameters.

[0066] This invention discloses an intelligent robot viewpoint planning method based on deformation-viewpoint dynamic coupling. By constructing a deep learning-driven deformation-viewpoint mapping model, it achieves intelligent matching of dynamic deformation and viewpoint parameters. Simultaneously, this invention employs a multi-objective hierarchical decoupling optimization network, improving computational efficiency while ensuring planning accuracy. Furthermore, integrating a spatiotemporal graph convolutional network and a deep reinforcement learning policy network enhances dynamic obstacle prediction and safe obstacle avoidance capabilities. In addition, this invention is applicable to high-precision visual inspection scenarios, significantly improving the real-time performance and robustness of robot viewpoint planning.

[0067] Furthermore, this invention reveals the nonlinear relationship between deformation gradient and view coverage through deep learning, breaking through the view offset bottleneck of static planning and improving measurement accuracy in dynamic scenes.

[0068] In some embodiments, S1 specifically includes the following steps:

[0069] High-precision laser scanners (such as Keyence LJ-V7000) and distributed fiber optic strain sensors (such as Omron E32-L14) are deployed on the dynamic assembly line to synchronously acquire three-dimensional deformation field data and surface strain distribution data of the target workpiece at 100Hz. The acquisition accuracy of the three-dimensional deformation field data is ±0.05mm, and the resolution of the surface strain distribution data is 1με.

[0070] A dataset for collecting multimodal information and constructing a deformation-attitude mapping model using this information includes:

[0071] Data related to the three-dimensional deformation gradient tensor: covering the dynamic deformation fields of five typical materials (aluminum alloy, carbon fiber, rubber, polyurethane, and steel) under three assembly stages (welding thermal deformation, bolt preload deformation, and detection contact deformation). Through the fusion and calibration of finite element simulation and measured data, the three-dimensional deformation gradient tensor and strain energy density distribution are generated.

[0072] Under a unified clock, a multi-camera array reconstructs the time-stamped end-body trajectory. The camera pose is obtained through hand-eye calibration. ,in This represents the rigid body transformation matrix from the camera coordinate system C to the end effector coordinate system E. Based on... Using the set A of key feature regions labeled in the point cloud model of the target workpiece, visibility is determined at each time step, and the visible area is accumulated to obtain the visibility coverage rate. ;in This represents the area of ​​the visible region. This indicates the total area.

[0073] Environmental parameter vector: Record the vibration spectrum of the assembly line (recording range 0-500Hz), the ambient light intensity (recording range 0-100klx), and obtain the spatial coordinates of dynamic obstacles through a ToF (Time of Flight) sensor for subsequent anti-interference modeling.

[0074] Ultimately, 100,000 samples were generated, each containing a three-dimensional deformation gradient tensor F∈ The true value of field coverage CR∈[0,1], and the environmental parameter vector. And the corresponding optimal observation pose (calibrated offline using the optimal coverage algorithm).

[0075] In some embodiments, the deformation-viewpoint mapping model uses a 3D-ResNet50 network. The 3D-ResNet50 network is a three-dimensional convolutional neural network architecture that extends from a two-dimensional residual network.

[0076] In some embodiments, the training process in S2 includes the following steps:

[0077] First, the dataset was divided into training, validation, and test sets, with the three sets accounting for 80% of the total dataset, 10% of the validation set, and 10% of the test set, respectively. During training, the deformation-pose mapping model was trained using the training set. Furthermore, mixed precision (e.g., FP16) and data augmentation techniques (e.g., random rotation, scaling) were enabled during training, and a single NVIDIA RTX A6000 GPU was used for training, which took 48 hours. During training, the validation and test sets were used for validation and testing. Validation metrics: field of view coverage prediction error ≤5%, cross-material generalization error ≤8%.

[0078] In some embodiments, the multimodal input layer is used to receive multimodal data and perform standardization processing on the multimodal data;

[0079] The residual feature extraction module consists of four residual connection layers. Each residual connection layer contains a 3D convolutional layer (3D Conv), a batch normalization layer (BN), and a ReLU activation function connected in sequence. The residual feature extraction module is used to extract features from the 3D deformation gradient tensor, the ground truth of the field of view coverage, and the environmental parameter vector, respectively, to obtain multiple different feature vectors. The optimal observation pose is used as a label and is not used as the input to the deformation-view mapping model.

[0080] The multi-head attention mechanism layer includes four parallel attention heads, used to dynamically assign weights to different deformation vectors, so that the deformation-viewpoint mapping model focuses on regions whose influence on viewpoint coverage reaches a set threshold (e.g., significant influence), generating a weighted feature vector; the dimension of the weighted feature vector is... ;

[0081] The viewpoint prediction output layer is used to map the weighted feature vector through a fully connected (FC) layer to obtain the viewpoint sensitivity index (dimension). The measurement parameters are: 1) the viewpoint sensitivity index and the coverage prediction value (scalar ∈ [0,1]). The viewpoint sensitivity index is used to evaluate the performance of the viewpoint, including visibility, feature change rate, geometric sensitivity, etc.

[0082] In some embodiments, the multimodal sensing data includes lidar point clouds, visual images, and force sensor signals.

[0083] In some embodiments, the spatiotemporal graph convolutional network includes a spatiotemporal graph convolutional network, a bidirectional gated recurrent unit, and an attention mechanism connected in sequence;

[0084] The spatiotemporal graph convolutional network is used to capture the temporal correlation of the speed, acceleration, and direction changes of obstacle motion trajectories through spatiotemporal graph convolutional layers to obtain temporal features; the bidirectional gated recurrent unit is used to process long-term motion trends; and the attention mechanism is used to identify short-term sudden behaviors.

[0085] The spatiotemporal graph convolutional network is used to fuse historical trajectory data and multimodal perception data, calculate the probability of obstacle presence at different time steps, and generate a spatiotemporal safety corridor.

[0086] In some embodiments, the spatiotemporal graph convolutional network trained in S3 is expressed as follows:

[0087] ;

[0088] in, This represents the prediction of future state sequences by a spatiotemporal graph convolutional network, i.e., a spatiotemporal safety corridor. Represents a sequence of historical states; subscript Indicates time step The subscript T indicates the length of the historical observation window; the subscript H indicates the length of the predicted horizon. This represents the spatiotemporal graph convolutional network after training. This represents Gaussian noise.

[0089] In some embodiments, S5 specifically includes the following steps:

[0090] S51. Scan the real 3D scene to obtain a scanned point cloud model, and extract surface voxel features from the scanned point cloud model.

[0091] S52. The data acquisition module of the robot system obtains multimodal information in reality and inputs it into the trained deformation-viewpoint mapping model to generate coverage prediction values.

[0092] S53. Input the coverage prediction value into the deformation compensation layer, and generate viewpoint compensation parameters based on the surface voxel features. Viewpoint compensation parameters ,in, These represent the azimuth and elevation angles, respectively. This represents the difference in time steps, then it is calculated based on the viewpoint compensation parameters. From the current perspective Compensation is performed to obtain the compensated perspective. ; ;

[0093] S54. The motion optimization layer uses a deep reinforcement learning policy network to optimize the viewpoint in the pose space after compensation. Gaussian noise is added to generate a candidate viewpoint set; then, an adaptive importance sampling strategy is used to filter the candidate viewpoint set, resulting in the filtered candidate viewpoint set.

[0094] S55. Construct the optimization objective function for the filtered candidate viewpoint set, as follows:

[0095] ;

[0096] in, Indicates the optimization objective. Indicates the viewpoint. , As weight, and , ; The visibility coverage of key feature regions, The cost of sports;

[0097] S56. Obtain the optimized candidate viewpoints by solving the objective function. Verify the optimized candidate viewpoints The cumulative collision probability of the corresponding trajectory. If the cumulative collision probability is greater than the set threshold (e.g., 0.0.3), return S53; otherwise, output the verified candidate viewpoint.

[0098] S57. Call the robot kinematics model (such as the UR5e robotic arm) to convert the verified candidate viewpoints into joint space coordinates; the robot system moves according to the joint space coordinates, and during the movement, the data acquisition module on the robot system collects the pose data of the dynamic obstacles in real time, and then uses the Kalman filter algorithm to control the measurement error of the pose data of the dynamic obstacles within the set range.

[0099] The pose data of dynamic obstacles is transmitted to a spatiotemporal graph convolutional network trained in an industrial control computer to calculate the probability of obstacle existence at different time steps and generate a spatiotemporal safety corridor.

[0100] S58. Input the spatiotemporal safety corridor into the safety constraint layer, perform collision risk scoring on the selected candidate viewpoint set, and output a safe and feasible subset of viewpoints. During this process, if the collision risk score is lower than the set collision threshold, return to S54 and regenerate the candidate viewpoint set.

[0101] S59. Establish the target fitness function using the viewpoint subset, solve the multi-target fitness function using the Non-dominated Sorting Genetic Algorithm-II (NSGA-II), and select the Pareto optimal viewpoint set.

[0102] In some embodiments, the probability expression for the existence of obstacles at different time steps in S57 is as follows:

[0103] ;

[0104] in, This represents the probability of an obstacle existing at different time steps, or the probability of a possible collision. This represents the position of the robot system at time step t; k represents the k-th Gaussian component; K represents the total number of Gaussian components. Indicates scale; Indicated by The mean is Gaussian density on the scale of ;

[0105] In some embodiments, the expression for the spatiotemporal security corridor in S57 is as follows:

[0106] ;

[0107] in, This represents a set of spatial points that meet safety conditions, i.e., a spacetime safety corridor. Indicates spatial location, Represents the set of real numbers; This represents the collision probability at time step t at control position q, where 0.12 is the maximum allowed collision probability set manually.

[0108] In some embodiments, the multi-objective fitness function in S59 is specifically as follows:

[0109] ;

[0110] in, For the Jacobian matrix of the robotic arm; Indicates the visibility rate of feature points; A normalized measure of collision risk; This represents a pose variable, which indicates the position and orientation of a rigid body in three-dimensional space. The form in which the pose variable P is expressed; This represents the F-norm.

[0111] This invention employs a multi-objective hierarchical decoupling optimization network to decompose the high-dimensional solution space into a three-layer hierarchical structure: a deformation compensation layer, a motion optimization layer, and a safety constraint layer. This three-layer hierarchical structure significantly improves computational efficiency through dimensionality reduction mapping and parallel solution, thus meeting the requirements of real-time planning.

[0112] In another aspect, the present invention provides a robot viewpoint intelligent planning system, including a robot system configured to perform a robot viewpoint intelligent planning method.

[0113] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Furthermore, the technical solutions of the various embodiments of the present invention can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A deformation-viewpoint dynamic coupling intelligent planning method for robot viewpoints, characterized in that, The process includes the following steps: S1. Acquire multimodal information, including 3D deformation gradient tensor, ground truth of field of view coverage, environmental parameter vector, and corresponding optimal observation pose; S2. Construct a deformation-viewpoint mapping model, which includes a multimodal input layer, a residual feature extraction module, a multi-head attention mechanism layer, and a viewpoint prediction output layer connected in sequence; train the deformation-viewpoint mapping model using multimodal data; S3. Acquire multimodal perception data and construct a spatiotemporal graph convolutional network, train the spatiotemporal graph convolutional network using multimodal perception data; S4. Construct a multi-objective hierarchical decoupling optimization network, which includes a deformation compensation layer, a motion optimization layer, and a safety constraint layer connected in sequence. Deploy the trained deformation-viewpoint mapping model, the trained spatiotemporal graph convolutional network, and the multi-objective hierarchical decoupling optimization network onto the robot system. S5. Scan the real 3D scene to obtain a scanned point cloud model, extract surface voxel features from the scanned point cloud model, input the real-world multimodal information into the trained deformation-viewpoint mapping model to generate coverage prediction values, and then input them together with the surface voxel features into the deformation compensation layer to generate the compensated viewpoint. The motion optimization layer utilizes a deep reinforcement learning policy network combined with an adaptive importance sampling strategy to filter the compensated viewpoints and generate a set of candidate viewpoints. A spatiotemporal safety corridor is generated based on the trained spatiotemporal graph convolutional network. This corridor is then input into the safety constraint layer to perform collision risk scoring on the selected candidate viewpoints, outputting a safe and feasible subset of viewpoints. A multi-objective fitness function is constructed and solved using this subset to select the Pareto optimal viewpoint set. Finally, the optimal end-effector pose parameters that meet the requirements are output based on the Pareto optimal viewpoint set. The robot system is then adjusted to the specified viewpoint based on these optimal end-effector pose parameters.

2. The robot viewpoint intelligent planning method based on deformation-viewpoint dynamic coupling according to claim 1, characterized in that, The multimodal input layer receives multimodal data and performs standardization processing on it. The residual feature extraction module consists of four residual connection layers, each containing a sequentially connected 3D convolutional layer, a batch normalization layer, and a ReLU activation function. The residual feature extraction module extracts features from the 3D deformation gradient tensor, the ground truth of the field of view coverage, and the environmental parameter vector, respectively, to obtain multiple different feature vectors. The multi-head attention mechanism layer includes four parallel attention heads, which dynamically allocate weights to different deformation vectors so that the deformation-viewpoint mapping model focuses on the region whose influence on viewpoint coverage reaches a set threshold, generating a weighted feature vector. The viewpoint prediction output layer maps the weighted feature vector through a fully connected (FC) layer to obtain the viewpoint sensitivity index and the coverage prediction value.

3. The robot viewpoint intelligent planning method based on deformation-viewpoint dynamic coupling according to claim 1, characterized in that, The multimodal sensing data includes lidar point clouds, visual images, and force sensor signals.

4. The robot viewpoint intelligent planning method based on deformation-viewpoint dynamic coupling according to claim 1, characterized in that, The spatiotemporal graph convolutional network includes a spatiotemporal graph convolutional network, a bidirectional gated recurrent unit, and an attention mechanism connected in sequence; the spatiotemporal graph convolutional network is used to capture the temporal correlation of the velocity, acceleration, and direction changes of the obstacle's motion trajectory through the spatiotemporal graph convolutional layer to obtain temporal features; The spatiotemporal graph convolutional network is used to fuse historical trajectory data and multimodal perception data, calculate the probability of obstacle presence at different time steps, and generate a spatiotemporal safety corridor.

5. The robot viewpoint intelligent planning method based on deformation-viewpoint dynamic coupling according to claim 1, characterized in that, The spatiotemporal graph convolutional network trained in S3 is expressed as follows: ;in, This represents the prediction of future state sequences by a spatiotemporal graph convolutional network, i.e., a spatiotemporal safety corridor. Represents a sequence of historical states; subscript Indicates time step The subscript T indicates the length of the historical observation window; the subscript H indicates the length of the predicted horizon. This represents the spatiotemporal graph convolutional network after training. This represents Gaussian noise.

6. The robot viewpoint intelligent planning method with deformation-viewpoint dynamic coupling according to claim 5, characterized in that, S5 specifically includes the following steps: S51, scanning the real 3D scene to obtain a scanned point cloud model, and extracting surface voxel features from the scanned point cloud model; S52, collecting multimodal information from the data acquisition module of the robot system and inputting it into the trained deformation-viewpoint mapping model to generate coverage prediction values; S53, inputting the coverage prediction values ​​into the deformation compensation layer, and generating viewpoint compensation parameters based on the surface voxel features. Then, based on the viewpoint compensation parameters From the current perspective Compensation is performed to obtain the compensated perspective. S54, the motion optimization layer uses a deep reinforcement learning policy network to optimize the viewpoint in the pose space after compensation. Gaussian noise is added to generate a candidate viewpoint set; then, an adaptive importance sampling strategy is used to filter the candidate viewpoint set, resulting in the filtered candidate viewpoint set; S55, the optimization objective function for the filtered candidate viewpoint set is constructed as follows: ;in, Indicates the optimization objective. Indicates the viewpoint. 、 As weight, and ， ； The visibility coverage of key feature regions, For motion cost; S56, use the optimization objective function to obtain the optimized candidate viewpoints. Verify the optimized candidate viewpoints The cumulative collision probability of the corresponding trajectory is calculated. If the cumulative collision probability is greater than a set threshold, the process returns to S53; otherwise, the verified candidate viewpoints are output. S57: The robot kinematics model is called to convert the verified candidate viewpoints into joint space coordinates. The robot system moves according to the joint space coordinates, and during the movement, the data acquisition module on the robot system collects the pose data of the dynamic obstacles in real time. Then, the Kalman filter algorithm is used to control the measurement error of the dynamic obstacle pose data within a set range. The pose data of the dynamic obstacles is transmitted to the spatiotemporal graph convolutional network trained in the industrial control computer to calculate the probability of obstacle existence at different time steps and generate a spatiotemporal safety corridor. S58: The spatiotemporal safety corridor is input into the safety constraint layer. The collision risk score of the selected candidate viewpoint set is calculated, and a safe and feasible viewpoint subset is output. During this process, if the collision risk score is lower than the set collision threshold, the process returns to S54 to regenerate the candidate viewpoint set. S59: A multi-objective fitness function is established using the viewpoint subset. The multi-objective fitness function is solved using a non-dominated sorting genetic algorithm to select the Pareto optimal viewpoint set.

7. The robot viewpoint intelligent planning method with deformation-viewpoint dynamic coupling according to claim 6, characterized in that, The probability expression for the existence of obstacles at different time steps in S57 is as follows: ;in, This represents the probability of an obstacle existing at different time steps, or the probability of a possible collision. This represents the position of the robot system at time step t; k represents the k-th Gaussian component; K represents the total number of Gaussian components. Indicates scale; Indicated by The mean is Gaussian density on the scale of .

8. The robot viewpoint intelligent planning method with deformation-viewpoint dynamic coupling according to claim 7, characterized in that, The expression for the spatiotemporal safety corridor in S57 is as follows: ;in, This represents a set of spatial points that meet safety conditions, i.e., a spacetime safety corridor. Indicates spatial location, Represents the set of real numbers; This represents the collision probability at time step t at control position q, where 0.12 is the maximum allowed collision probability set manually.

9. The robot viewpoint intelligent planning method based on deformation-viewpoint dynamic coupling according to claim 8, characterized in that, The multi-objective fitness function in S59 is as follows: ;in, For the Jacobian matrix of the robotic arm; Indicates the visibility rate of feature points; A normalized measure of collision risk; This represents a pose variable, which indicates the position and orientation of a rigid body in three-dimensional space. The form in which the pose variable P is expressed; This represents the F-norm.

10. A robot viewpoint intelligent planning system, characterized in that, Includes a robot system configured to perform the robot viewpoint intelligent planning method according to any one of claims 1 to 9.

Images