Robot control method and system based on computer vision

Abstract

The invention discloses a robot control method and system based on computer vision, and belongs to the technical field of computer vision, and the method specifically comprises the steps: building a unified space model containing a robot and an external environment through obtaining robot joint angle data and scene point cloud data collected by a depth camera; the surface contour of a target operation object is recognized in the model, a deformation sequence of the target operation object in the grabbing process is predicted, and then a corresponding grabbing posture change sequence is generated. And the grabbing posture sequence serves as a target path, a joint movement track needed by the robot for tracking the path is calculated, and a first control instruction is sent to an executing mechanism. Continuously acquiring new sensor data in the execution process, and dynamically updating a subsequent motion track; according to the invention, the robot can adaptively grab the flexible object in a complex environment.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and more specifically to a robot control method and system based on computer vision. Background Technology

[0002] In the fields of industrial automation and intelligent manufacturing, computer vision-based robot control technology is the core for achieving flexible grasping and autonomous operation. As application scenarios continue to expand, the objects that robots need to handle are no longer limited to rigid workpieces with fixed shapes, but increasingly include non-rigid objects such as food, sponges, and wire harnesses that are easily deformed under external forces. This requires robot systems not only to perceive the static geometric information of the environment, but also to have the ability to understand and respond to the dynamic physical characteristics of the target object in order to achieve stable and reliable grasping operations.

[0003] Currently, most existing robotic grasping methods are based on static environment assumptions. Specifically, these methods typically acquire 3D point cloud data of the target object using visual sensors, then calculate one or more optimal grasping poses on the static point cloud model, and plan a motion trajectory from the starting point to the fixed grasping point. Some more advanced methods introduce simple deformation models or use online visual servoing to fine-tune the robot's motion to handle slight displacements or deformations of the object. These technologies provide effective solutions for the automated grasping of rigid objects and have been successfully applied in specific scenarios.

[0004] However, in real-world operating environments, the state of the target object is not static. Its spatial position during robot grasping may shift due to external disturbances, and its surface contours and internal structures may undergo continuous morphological evolution due to contact pressure. Faced with this dynamic change, existing technologies exhibit significant limitations. First, grasping planning methods based on static point clouds cannot proactively predict the continuous deformation process of the target object under contact pressure, causing the planned grasping pose to quickly fail after physical interaction begins, failing to maintain stable force closure. Second, trajectory execution strategies based on fixed paths or relying solely on end-effector position feedback lack real-time tracking capabilities for the dynamically changing grasping posture caused by object deformation. This leads to a disconnect between the robot's grasping actions and the actual morphological evolution of the object, ultimately resulting in grasping failure. These limitations constrain the further development of robots in high-end applications such as packaging, biological tissue manipulation, and complex assembly. Summary of the Invention

[0005] The purpose of this invention is to provide a robot control method and system based on computer vision, and to solve the following technical problems: Existing grasping planning methods based on static environment assumptions fail to predict the continuous deformation of the target object during the grasping process, causing the pre-calculated grasping pose to fail during the physical contact phase. At the same time, existing trajectory execution strategies lack a synchronous tracking mechanism for the dynamic deformation process, causing the robot's grasping action to become disconnected from the actual object's shape evolution.

[0006] The objective of this invention can be achieved through the following technical solutions: A computer vision-based robot control method includes the following steps: S1. Acquire sensor data, including the current angle data of the robot joints and the scene point cloud data of the target object collected by the depth camera; S2. Integrate scene point cloud data with robot kinematics model to construct a unified spatial model that includes the space occupied by the robot itself and external environmental obstacles; S3. In the unified spatial model, identify the surface contour of the target object and predict the sequence of surface contour changes of the target object during the execution of the robot's grasping action. S4. Based on the predicted surface contour change sequence, reverse the generation of the corresponding robot grasping posture change sequence. S5. Using the sequence of grasping posture changes as the target path, in the unified space model, the robot calculates a series of joint angle changes required to track the target path from the current joint angle, forming a forward-looking joint motion trajectory. S6. Send the joint angle command of the first time node of the forward joint motion trajectory to the robot joint controller, and while executing the command, repeat S1-S5 based on the new sensor data to update the subsequent forward joint motion trajectory.

[0007] As a further aspect of the present invention: in S2, the process of constructing the unified spatial model is as follows: The robot analyzes the link structure data in its kinematic model and generates a set of tightly packed cubic bounding boxes for each link. The size of the cubic bounding boxes encloses the surface contour of the link structure. The obstacle point cloud is read from the scene point cloud data, and the coordinates of each 3D point in the obstacle point cloud are converted into voxel indices in the 3D voxel mesh world. After obtaining the surface contour change sequence of the target object, the 3D point coordinates of the surface point cloud in each frame of the sequence are converted into voxel indices in the 3D voxel mesh world. In the three-dimensional voxel mesh world, the voxels occupied by the robot link cube bounding box, the voxels where the obstacle point cloud index is located, and the voxels where the target object surface point cloud index is located are all marked as the first state; the unoccupied voxels are marked as the second state. The unified spatial model is composed of the three-dimensional voxel mesh world marked as the first state and the second state.

[0008] As a further aspect of the present invention: in step S3, the process of predicting the surface contour change sequence is as follows: The target object point cloud in the scene point cloud data is segmented, and the surface curvature distribution features of the target object point cloud are extracted using a region growing algorithm. The surface curvature distribution features include the principal curvature values ​​and principal directions of each region on the point cloud surface. A viscoelastic mechanical model of the target object is established. The viscoelastic mechanical model includes a combination structure of spring and damping units. The elastic and viscous parameters of the viscoelastic mechanical model are initialized based on the principal curvature values ​​in the surface curvature distribution features. The dynamic deformation process of stress transmission and energy dissipation within the viscoelastic mechanical model is simulated under the contact pressure applied by the robot's end effector. The dynamic deformation process is discretized into multiple consecutive time frames, and the three-dimensional point set of the target object surface corresponding to each time frame is output to form the surface contour change sequence.

[0009] As a further aspect of the present invention: the parameter adjustment process of the viscoelastic mechanical model is as follows: After assigning values ​​to the viscoelastic mechanics model, the principal curvature values ​​and principal directions in the surface curvature distribution features are input into a pre-trained parameter mapping network. The parameter mapping network contains three fully connected layers and one normalized output layer. The parameter mapping network outputs the adjustment coefficients of the relaxation time parameter and the steady-state elastic modulus parameter in the viscoelastic mechanics model. The relaxation time parameter and the steady-state elastic modulus parameter are multiplied by the corresponding adjustment coefficients to update the parameters of the viscoelastic mechanical model. The updated viscoelastic mechanical model is then used to predict the surface profile change sequence.

[0010] As a further aspect of the present invention: in step S4, the process of reverse-generating the robot's grasping posture change sequence is as follows: The initial grasping posture is calculated at the start frame of the surface contour change sequence, and the final grasping posture is calculated at the end frame. Both the initial and final grasping postures include the spatial position and orientation of the end effector and the gripper opening width. Based on the initial and final grasping postures, interpolation calculations are performed between adjacent time frames. Based on the displacement vector of the centroid coordinates of the corresponding regions of the three-dimensional point sets on the surface of the preceding and following frames, the spatial position of the end effector is linearly interpolated; based on the change of the principal normal vector of the surface region of the preceding and following frames, the spatial orientation of the end effector is linearly interpolated; based on the change of the bounding box size of the surface region of the preceding and following frames, the opening width of the gripper is linearly adjusted; and the robot grasping posture change sequence is generated through the above interpolation calculation.

[0011] As a further aspect of the present invention: in S5, the process of forming a forward-looking joint motion trajectory is as follows: Each grasping posture in the grasping posture change sequence is mapped to multiple candidate joint angle combinations through inverse kinematics of the robot. These candidate joint angle combinations are connected in time order to form an initial joint space path. The time interval between path points on the initial joint space path is consistent with the reciprocal of the robot control cycle. In the unified space model, it is checked whether the robot link profile at each path point on the initial joint space path intersects with the point cloud in the environmental obstacle or the predicted surface profile change sequence. When the voxel intersection is detected, the initial joint space path is locally smoothed and adjusted in the joint space to generate a prospective joint motion trajectory.

[0012] As a further aspect of the present invention: the process of local smoothing adjustment is as follows: Traverse the initial joint space path, locate the path points where voxel intersections occur and the corresponding intersection links, and record the timestamps of the voxel intersection path points and the identifiers of the intersection links; define a dynamic adjustment time window before and after the timestamps of the intersection path points; within the dynamic adjustment time window, construct a sinusoidal correction curve for a main joint that drives the movement of the intersection links; the amplitude of the sinusoidal correction curve is increased iteratively until the voxel intersections no longer exist, and the iteration step size decreases as the number of iterations increases; At the start and end of the dynamically adjusted time window, the joint angle values ​​of the sine wave correction curve are consistent with the joint angle values ​​of the initial joint space path; the sine wave correction curve is superimposed on the main joint angle sequence of the initial joint space path to form a forward-looking joint motion trajectory after local smoothing adjustment.

[0013] As a further aspect of the present invention: in step S6, the process of updating the subsequent forward-looking joint motion trajectory is as follows: After executing the first joint motion trajectory, the latest collected scene point cloud data and robot joint angle data are acquired; the latest collected scene point cloud data is compared with the corresponding time frame point cloud in the predicted surface contour change sequence, and the average deviation of the point cloud position is calculated. When the average deviation exceeds the set threshold, the latest collected scene point cloud data and robot joint angle data are used as input to re-execute S2 to S5, generating a new forward-looking joint motion trajectory starting from the next control cycle; the newly generated forward-looking joint motion trajectory replaces the previously generated but not executed joint motion trajectory.

[0014] The present invention also includes a computer vision-based robot control system for implementing the above-described computer vision-based robot control method, comprising: The data perception module is used to acquire sensor data, including the current angle data of the robot joints and the scene point cloud data of the target manipulator collected by the depth camera. The spatial modeling module is used to fuse scene point cloud data with robot kinematics models to construct a unified spatial model that includes the space occupied by the robot itself and external environmental obstacles. The deformation prediction module is used to identify the surface contour of the target object in a unified spatial model and predict the sequence of surface contour changes of the target object during the execution of the robot's grasping action. The posture planning module is used to inversely generate the corresponding robot grasping posture change sequence based on the predicted surface contour change sequence. The trajectory generation module is used to take the grasping posture change sequence as the target path. In the unified space model, it calculates the series of joint angle changes required for the robot to track the target path from the current joint angle, forming a forward-looking joint motion trajectory. The closed-loop control module is used to send the joint angle command of the first time node of the forward joint motion trajectory to the robot joint controller. While executing the command, the module operation is repeated based on the new sensor data to update the subsequent forward joint motion trajectory.

[0015] The beneficial effects of this invention are: This invention effectively overcomes the shortcomings of existing technologies in terms of adaptability to grasping non-rigid objects. By constructing a closed-loop system of dynamic deformation prediction and robot motion control, it achieves proactive response to changes in the shape of the target object. Specifically, based on a viscoelastic mechanics model, it performs physically accurate prediction of the surface contour change sequence of the target object during the grasping process, enabling the robot to proactively generate a grasping posture change sequence synchronized with the deformation process, thus solving the problem of grasping posture failure caused by deformation. Furthermore, by calculating the joint motion trajectory of the robot tracking the dynamic grasping posture within a unified spatial model and employing a local smoothing adjustment method based on a sine wave correction curve to achieve real-time obstacle avoidance, it ensures continuous alignment between the robot's actual motion and the object's shape evolution. Finally, by integrating the latest perception data into the prediction and planning loop through a rolling update mechanism, it forms a continuous adaptability to dynamically changing environments, thereby achieving stable and reliable grasping of non-rigid objects in real-world operating environments. Attached Figure Description

[0016] The invention will now be further described with reference to the accompanying drawings.

[0017] Figure 1 This is a flowchart illustrating a computer vision-based robot control method according to the present invention. Figure 2 This is a schematic diagram of a robot control system based on computer vision according to the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] Please see Figure 1 As shown, this invention is a robot control method based on computer vision, comprising the following steps: S1. Acquire sensor data, including the current angle data of the robot joints and the scene point cloud data of the target object collected by the depth camera; S2. Integrate scene point cloud data with robot kinematics model to construct a unified spatial model that includes the space occupied by the robot itself and external environmental obstacles; S3. In the unified spatial model, identify the surface contour of the target object and predict the sequence of surface contour changes of the target object during the execution of the robot's grasping action. S4. Based on the predicted surface contour change sequence, reverse the generation of the corresponding robot grasping posture change sequence. S5. Using the sequence of grasping posture changes as the target path, in the unified space model, the robot calculates a series of joint angle changes required to track the target path from the current joint angle, forming a forward-looking joint motion trajectory. S6. Send the joint angle command of the first time node of the forward joint motion trajectory to the robot joint controller, and while executing the command, repeat S1-S5 based on the new sensor data to update the subsequent forward joint motion trajectory.

[0020] In a preferred embodiment of the present invention, the process of constructing a unified spatial model in step S2 is as follows: Read the robot's DH parameter table and extract the geometry, joint type, and range of motion for each link. Taking a typical six-axis industrial robot as an example, process the six links from the base to the end effector sequentially. The geometric data of each link includes length, diameter, and outline parameters.

[0021] The process of generating a cubic bounding box for each link employs a hierarchical bounding box algorithm. First, the minimum bounding cube of the link is calculated based on its actual geometry. For example, for a cylindrical link, the minimum cube size that can completely enclose the cylinder is calculated. Assuming a link is 0.3 meters long and 0.05 meters in diameter, a cubic bounding box of 0.3 meters by 0.05 meters by 0.05 meters is generated. To improve accuracy, this main bounding box is subdivided into smaller cubic units, each with a side length of 0.02 meters. These small cubes are closely packed together to completely enclose the surface contour of the link.

[0022] The processing of scene point cloud data is performed concurrently. Raw point cloud data, containing millions of 3D spatial points, is acquired from a depth camera or LiDAR. Each point is represented by floating-point coordinates; for example, the coordinates of an obstacle point might be (1.25, 0.83, 0.47). These floating-point coordinates are converted to voxel grid indices using a formula that divides each coordinate value by the voxel size of 0.01 meters and rounds down. Following this rule, the point (1.25, 0.83, 0.47) is converted to the voxel index (125, 83, 47). The same conversion logic is used for processing the surface contour change sequence of the target object. Time-series point cloud data is received from a vision sensor, with each frame containing thousands of points on the surface of the target object. For each point in each frame, the same coordinate-to-voxel index conversion is performed. For example, in a frame of data at timestamp t0, the coordinates of a surface point (2.15, 1.02, 0.33) are converted to the voxel index (215, 102, 33). Record the changes of this voxel index over time.

[0023] In the construction phase of the 3D voxel mesh world, a sufficiently large 3D array is initialized to represent the entire workspace. Assuming the workspace size is 3 meters by 3 meters by 2 meters, and the voxel size is 0.01 meters, then the dimensions of this 3D array are 300 by 300 by 200. Each element in the array corresponds to a voxel, initially set to 0, representing the second state, i.e., the unoccupied state. The process of marking occupied voxels is divided into three parallel sub-processes. The first sub-process handles the bounding boxes of the robot links. It iterates through all the small cubes within the bounding boxes, calculating the voxel index range corresponding to each small cube. For example, a bounding box of cubes from (100, 50, 30) to (130, 80, 60) will mark all voxels within it as 1, representing the first state, i.e., the occupied state. The second sub-process handles the obstacle point cloud. It iterates through all the voxel indices obtained from the obstacle point cloud, setting the 3D array element corresponding to each index to 1. For example, the array element corresponding to the voxel index (125, 83, 47) is marked as 1. The third subprocess processes the point cloud on the surface of the target object. It not only marks the voxel index of the current frame but also maintains a time-series occupancy record. For example, a voxel index (215, 102, 33) might be marked as 1 across multiple consecutive frames, recording the duration for which this voxel is occupied.

[0024] After all marking processes are completed, a consistency check is performed to ensure that the same voxel is not marked repeatedly. If conflicting markings are found, the conflicts are resolved according to the priority order of robot links, obstacles, and target manipulated objects. The final unified spatial model is a three-dimensional mesh world containing hundreds of thousands of voxels, each with a clear state label, providing accurate spatial occupancy information for subsequent path planning and collision detection. The entire construction process adopts an incremental update mechanism. When an environmental change is detected, only the voxel state of the affected area is updated, rather than rebuilding the entire model, which greatly improves the real-time performance of the process. A full update is performed every 50 milliseconds to ensure that the unified spatial model can reflect the latest state of the working environment in a timely manner.

[0025] In another preferred embodiment of the present invention, the process of predicting the surface contour change sequence in S3 is as follows: First, the target object point cloud in the scene point cloud data is segmented using a region growing algorithm based on normal vectors. This algorithm starts with a random seed point and searches for neighboring points with similar normal vector directions in the point cloud, forming continuous surface regions. Each region generates a surface patch, and the principal curvature value and principal direction of the surface patch are calculated. For example, when processing a cube-shaped target object, six planar regions may be identified, each with a principal curvature value close to zero, and principal directions corresponding to the normal directions of the six faces of the cube.

[0026] When establishing the viscoelastic mechanical model of the target manipulated object, a Maxwell model with parallel spring and damping elements is used. Each spring element represents the elastic properties of the material, and each damping element represents the viscous properties. During model initialization, elastic and viscous parameters are set according to the principal curvature values ​​in the surface curvature distribution characteristics. For regions with large curvature, higher elastic parameter values ​​are set; for example, in a region with a curvature of 0.5 mm to the power of -1, the elastic parameter is set to 2.3 MPa. For regions with small curvature, lower elastic parameter values ​​are set; for example, in a region with a curvature of 0.1 mm to the power of -1, the elastic parameter is set to 1.2 MPa.

[0027] When simulating the deformation process under contact pressure, a pressure load is applied at the point of contact between the robot's end effector and the target object. The magnitude of the pressure load is calculated based on the motion parameters of the end effector; for example, when the end effector approaches the target object at a speed of 0.2 m / s, the calculated initial contact pressure is 15 N. The viscoelastic mechanical model internally calculates stress transmission using the finite element method, with each element calculating the stress-strain relationship based on its elastic and viscous parameters. The energy dissipation process is implemented using damped elements, with a damping coefficient set to 0.05 N / s per millimeter.

[0028] The dynamic deformation process is discretized into 200 consecutive time frames, each corresponding to a simulation time of 0.01 seconds. Within each time frame, the displacement of all nodes in the viscoelastic mechanical model is calculated. For example, in time frame 50, the displacement of a certain surface node is 0.3 mm. The three-dimensional point set of the target object surface corresponding to each time frame is output, and these point sets constitute a complete sequence of surface profile changes. Each point in the sequence contains three-dimensional coordinate information and a timestamp; for example, the coordinates of point number 1001 in time frame 30 are (102.5, 86.3, 45.2).

[0029] In the parameter tuning process of the viscoelastic mechanics model, the principal curvature values ​​in the surface curvature distribution features are first mapped to the pre-trained parameter mapping network along with the principal directions. This network contains three fully connected layers, each with 128, 64, and 32 neurons, respectively. The first layer uses the ReLU activation function, the second layer uses the Sigmoid activation function, and the third layer uses the linear activation function. The normalized output layer restricts the network output values ​​to the range of 0 to 1.

[0030] The parameter mapping network outputs two adjustment coefficients: one for the relaxation time parameter and one for the steady-state elastic modulus parameter. For example, when the input principal curvature value is 0.3 mm⁻¹ and the principal direction vector is (0.87, 0.35, 0.15), the network might output a relaxation time adjustment coefficient of 0.85 and a steady-state elastic modulus adjustment coefficient of 1.15. These adjustment coefficients are calculated using the network's weight matrix and bias vector, which are optimized during the training phase using a backpropagation algorithm.

[0031] The original relaxation time parameter of the viscoelastic mechanics model is multiplied by the corresponding adjustment factor. For example, if the original relaxation time parameter is 0.5 seconds, multiplying it by the adjustment factor of 0.85 updates it to 0.425 seconds. Simultaneously, the steady-state elastic modulus parameter is multiplied by the corresponding adjustment factor; for example, if the original steady-state elastic modulus parameter is 1.8 MPa, multiplying it by the adjustment factor of 1.15 updates it to 2.07 MPa. This parameter update process is performed within each simulation time step to ensure that the model parameters can adapt to spatial variations in surface curvature.

[0032] When predicting surface profile change sequences using the updated viscoelastic model, the deformation process under contact pressure is recalculated. Because the model parameters have been adjusted according to the surface curvature characteristics, the deformation prediction results are more accurate. For example, in high curvature regions, the degree of deformation may be reduced by 15% compared to the initial prediction, while in low curvature regions, the degree of deformation may be increased by 10%. These adjustments make the predicted surface profile change sequences more consistent with the actual mechanical properties of the target object.

[0033] The entire prediction process employs an iterative optimization approach. After each time frame, the output value of the parameter mapping network is fine-tuned based on the difference between the predicted result and the measured data. For example, when the error between the predicted deformation and the actual deformation exceeds 0.1 mm, the bias vector in the network is automatically adjusted to reduce the error in the next prediction. This iterative optimization ensures that the prediction accuracy of the surface profile change sequence continuously improves as the simulation progresses.

[0034] The prediction process also considers the influence of environmental factors, such as the effects of ambient temperature and humidity on the viscoelastic mechanical model parameters. When the ambient temperature rises from 25 degrees Celsius to 30 degrees Celsius, the relaxation time parameter is automatically multiplied by a temperature correction factor of 0.95, and the steady-state elastic modulus parameter is multiplied by a temperature correction factor of 0.98. These correction factors are determined through prior material experiments to ensure that the prediction results remain accurate under different environmental conditions.

[0035] The final output sequence of surface profile changes contains complete data across hundreds of time frames, each containing the 3D coordinates of approximately 5000 surface points. This data is stored in binary format, preserving the normal vector and curvature information for each point, providing complete surface deformation prediction data for subsequent robot motion planning. The sequence data also includes timestamp information and simulation parameter records for easy analysis and verification.

[0036] In another preferred embodiment of the present invention, the process of reverse-generating the robot grasping posture change sequence in step S4 is as follows: First, the initial grasping attitude is calculated in the starting frame of the surface contour change sequence. This process extracts surface features of the target object from the 3D point set of the starting frame and divides the surface point cloud into several continuous regions using a region segmentation algorithm. The centroid coordinates of each region are calculated; for example, the centroid coordinates of a certain region are X=120.5 mm, Y=80.3 mm, and Z=50.2 mm. Simultaneously, the principal normal vector of each region is calculated; for example, the principal normal vector of a certain region is (0.87, 0.35, 0.15). Based on these feature data and the geometric parameters of the end effector, the initial grasping attitude is determined. The initial grasping attitude includes the spatial position of the end effector X=125.3 mm, Y=82.1 mm, Z=55.6 mm, the spatial orientation expressed in Euler angles as a yaw angle of 30 degrees, a pitch angle of 15 degrees, a roll angle of 10 degrees, and a gripper opening width set to 45 mm.

[0037] The same calculation process is performed at the end frame of the surface contour change sequence to obtain the final gripping posture. Because the surface contour of the target object changes during deformation, the centroid coordinates of the surface region in the end frame may become X = 135.2 mm, Y = 85.6 mm, Z = 52.8 mm. The principal normal vector may become (0.82, 0.42, 0.18). Based on these changed feature data, the final gripping posture is determined as follows: spatial position X = 140.1 mm, Y = 87.9 mm, Z = 58.3 mm; spatial orientation: yaw angle 35 degrees, pitch angle 18 degrees, roll angle 12 degrees; gripper opening width adjusted to 48 mm.

[0038] After obtaining the initial and final grasping poses, interpolation calculations begin. Interpolation calculations are performed between adjacent time frames; the entire surface contour change sequence comprises 200 time frames, therefore requiring 199 interpolation calculations. Each interpolation calculation consists of three independent processes.

[0039] The spatial interpolation calculation is based on the displacement vector of the centroid coordinates of the corresponding regions of the three-dimensional point set on the surface in the preceding and following frames. For example, from time frame 50 to time frame 51, the centroid coordinates of a certain region change from X=128.3 mm, Y=83.5 mm, Z=53.1 mm to X=128.9 mm, Y=83.8 mm, Z=53.3 mm, with a displacement vector of (0.6, 0.3, 0.2) mm. Based on this displacement vector, the spatial position of the end effector is linearly interpolated to keep the position change of the end effector synchronized with the change of the surface centroid coordinates.

[0040] The spatial orientation interpolation is based on the change in the principal normal vector of the surface region between frames. The principal normal vector changes from (0.85, 0.38, 0.16) in time frame 50 to (0.84, 0.39, 0.17) in time frame 51. The change in rotation angle is obtained by calculating the rotation matrix between the two normal vectors, and then the spatial orientation of the end effector is linearly interpolated. For example, the yaw angle is interpolated from 32.5 degrees to 32.8 degrees, the pitch angle from 16.2 degrees to 16.5 degrees, and the roll angle from 11.3 degrees to 11.6 degrees.

[0041] The gripper opening width is adjusted based on the change in the surface area bounding box dimensions between the preceding and following frames. The surface area bounding box measures 46.2 mm in length, 42.5 mm in width, and 38.8 mm in height at time frame 50, and changes to 46.5 mm in length, 42.7 mm in width, and 39.1 mm in height at time frame 51. The gripper opening width is linearly adjusted from 46.5 mm to 46.8 mm according to the amount of change in the bounding box dimensions.

[0042] Through the three interpolation calculation processes described above, a corresponding grasping posture is generated for each time frame. Each grasping posture data includes three parts: spatial position coordinates, spatial orientation Euler angles, and gripper opening width. These data are arranged in chronological order to form a complete sequence of robot grasping posture changes.

[0043] In another preferred embodiment of the present invention, the process of forming a forward joint motion trajectory in S5 is as follows: First, each grasping posture in the grasping posture change sequence is mapped to multiple candidate joint angle combinations using inverse kinematics. Taking a six-DOF robot as an example, each grasping posture typically corresponds to up to eight different joint angle combinations. These joint angle combinations are arranged at 0.1-second intervals according to the robot control cycle, forming an initial joint space path containing 200 path points. Each path point contains the angle values ​​of six joints; for example, at time t=1.0 seconds, the angle values ​​of the six joints are 30.5 degrees, 45.2 degrees, 60.1 degrees, 15.3 degrees, 75.6 degrees, and 25.4 degrees, respectively.

[0044] When verifying the initial joint space path in the unified space model, a voxel intersection detection algorithm is used. This algorithm traverses each path point on the path and calculates the voxel index range corresponding to the bounding box of each robot link in the 3D voxel mesh. For example, at time t=2.5 seconds, the bounding box of the third link may cover a voxel region from voxel index (150,120,80) to (180,150,110). These voxels are compared with voxels marked as obstacles in the environment, and simultaneously compared with voxels marked as target manipulators in the predicted surface profile change sequence. When an overlap is found between the bounding box voxel of a link and an obstacle voxel or a target manipulator voxel, the timestamp of the path point where the voxel intersection occurs and the specific link identifier are recorded.

[0045] The local smoothing adjustment process begins by locating the problem area. Assume that at time t=3.2 seconds, a voxel intersection between the fourth link and an obstacle is detected, and the system records the timestamp of this path point as 3.2 seconds and the link identifier as 4. Centered on this time point, a dynamic adjustment time window lasting 0.6 seconds is defined, extending from t=2.9 seconds to t=3.5 seconds. Within this time window, the main joint driving the fourth link's movement—i.e., the robot's fourth joint—is selected, and a sinusoidal correction curve is constructed for it.

[0046] The sine wave correction curve is constructed using a parametric method. The initial amplitude of the curve is set to 0.5 degrees, and the phase is aligned with the starting point of the time window. In the first iteration, a sine wave with an amplitude of 0.5 degrees is superimposed on the fourth joint angle sequence of the initial joint space path. The voxel intersection of the adjusted path within the time window is re-examined. If intersection still exists, the amplitude is increased to 1.0 degrees for the second iteration. As the number of iterations increases, the iteration step size is gradually reduced: the second iteration step size is reduced to 0.3 degrees, the third iteration step size is reduced to 0.2 degrees, and so on, until the minimum effective amplitude for eliminating voxel intersection is found.

[0047] At the start and end points of the dynamic adjustment time window (t=2.9 seconds and t=3.5 seconds), ensure that the joint angle values ​​of the sine wave correction curve are completely consistent with the joint angle values ​​of the initial joint space path, maintaining path continuity. For example, at t=2.9 seconds, the initial angle of the fourth joint is 42.3 degrees, and the correction curve value at that point is exactly 0 degrees; the angle remains unchanged after superposition. The continuity of angle values ​​is also maintained at t=3.5 seconds.

[0048] The optimized sinusoidal correction curve is superimposed onto the master joint angle sequence of the initial joint space path to form a locally smoothed, forward-looking joint motion trajectory. This adjustment process may require processing multiple path points with voxel intersections sequentially. For example, after processing the collision of the fourth link at t=3.2 seconds, a potential collision is found in the second link at t=4.1 seconds, requiring the same smoothing adjustment process to be repeated.

[0049] Throughout the adjustment process, it is essential to ensure that the robot's joint motion limits are not violated. Each joint has a specific angular range; for example, the first joint's range of motion is from -160 degrees to +160 degrees, and the second joint's range of motion is from -125 degrees to +125 degrees. When the adjusted joint angles approach these limits, the system automatically introduces additional constraints to ensure that the generated trajectory remains within the robot's physical capabilities.

[0050] After completing all necessary local adjustments, a smoothness check is performed on the entire joint motion trajectory. The smoothness of the trajectory is ensured by calculating the first and second derivatives of the rate of change of joint angles. For example, the angular acceleration of each joint is checked to see if it exceeds a preset threshold of 100 degrees per second squared. If excessive angular acceleration is found in certain segments, additional smoothing is applied to these segments, typically using a fifth-order polynomial interpolation method for optimization.

[0051] The final generated forward-looking joint motion trajectory contains 2000 control points, spanning 20 seconds with a control point interval of 0.01 seconds. Each control point records the angle, angular velocity, and angular acceleration values ​​of six joints. This data is stored in a specific binary format, comprising a header, data body, and checksum, and can be directly transmitted to the robot controller for execution. The trajectory data also includes timestamp information and version identifiers to ensure synchronization with the surface contour change sequence.

[0052] In another preferred embodiment of the present invention, in step S6, the process of updating the subsequent forward joint motion trajectory is as follows: The process of updating subsequent forward-looking joint motion trajectories is achieved through a closed-loop feedback mechanism. This process begins after the initial joint motion trajectory is executed, at which point the latest operational data is acquired from the robot's control system. This data includes the latest scene point cloud data captured by a depth camera installed in the robot's workspace, and real-time joint angle data read from the encoders of each joint. The latest scene point cloud data contains approximately 500,000 3D points, each with millimeter-level precision; for example, the coordinates of a point might be (1025.3, 658.7, 215.4). The joint angle data contains the actual angle values ​​of the six joints; for example, at time t=3.5 seconds, the angles of the six joints are (35.2, 48.7, 62.3, 18.6, 72.9, 28.1), in degrees.

[0053] When comparing the newly acquired scene point cloud data with the corresponding time frame point clouds in the predicted surface contour change sequence, a point cloud registration algorithm is employed. First, the two point clouds are preprocessed, including downsampling and noise filtering, to ensure they have the same density. Then, an iterative nearest-point algorithm is used for precise registration, calculating the positional deviations of corresponding points between the two point clouds. For example, at time frame 35, the positional deviations of 5000 corresponding points are calculated, and these deviation values ​​constitute a deviation sequence.

[0054] The average deviation of the point cloud position is calculated using a weighted average method. A weight is assigned to the deviation of each point, depending on the importance of that point's position on the target object's surface. Areas closer to the grasping point are given higher weights, for example, 0.8; more distant areas are given lower weights, for example, 0.2. The weighted average of all point deviations is calculated to obtain a comprehensive average deviation value. For example, in a certain calculation, the average deviation obtained was 0.63 mm.

[0055] The threshold is determined based on the physical characteristics of the target object and the task requirements. For precision assembly tasks, the threshold is set to 0.5 mm; for general gripping tasks, the threshold is set to 1.0 mm. When the calculated average deviation of 0.63 mm exceeds the set threshold of 0.5 mm, the trajectory update process is triggered.

[0056] Using the latest acquired scene point cloud data and robot joint angle data as input, processes S2 to S5 are re-executed. First, a unified spatial model is reconstructed based on the new point cloud data, and the voxel labels of obstacles and target manipulated objects are updated. Then, the robot's current state is corrected based on the actual joint angle data, and the surface contour change sequence is re-predicted. Next, based on the updated data, the grasping posture change sequence is recalculated, and finally, a new forward-looking joint motion trajectory is generated.

[0057] The newly generated forward-looking joint motion trajectory begins in the next control cycle, covering the motion range for the next 3 seconds and containing 300 control points. This trajectory completely replaces the previously generated but unexecuted joint motion trajectory, ensuring that the robot's subsequent movements remain consistent with the latest environmental state. The trajectory update process is completed within 50 milliseconds, guaranteeing real-time control. Each trajectory update records the update reason, timestamp, and key parameters for subsequent analysis and optimization.

[0058] Please see Figure 2 As shown, the present invention also includes a computer vision-based robot control system for implementing the above-described computer vision-based robot control method, comprising: The data perception module is used to acquire sensor data, including the current angle data of the robot joints and the scene point cloud data of the target manipulator collected by the depth camera. The spatial modeling module is used to fuse scene point cloud data with robot kinematics models to construct a unified spatial model that includes the space occupied by the robot itself and external environmental obstacles. The deformation prediction module is used to identify the surface contour of the target object in a unified spatial model and predict the sequence of surface contour changes of the target object during the execution of the robot's grasping action. The posture planning module is used to inversely generate the corresponding robot grasping posture change sequence based on the predicted surface contour change sequence. The trajectory generation module is used to take the grasping posture change sequence as the target path. In the unified space model, it calculates the series of joint angle changes required for the robot to track the target path from the current joint angle, forming a forward-looking joint motion trajectory. The closed-loop control module is used to send the joint angle command of the first time node of the forward joint motion trajectory to the robot joint controller. While executing the command, the module operation is repeated based on the new sensor data to update the subsequent forward joint motion trajectory.

[0059] The foregoing has provided a detailed description of one embodiment of the present invention, but this description is merely a preferred embodiment and should not be construed as limiting the scope of the invention. All equivalent variations and modifications made within the scope of the claims of this invention should still fall within the patent coverage of this invention.

Claims

1. A robot control method based on computer vision, characterized in that, Includes the following steps: S1. Acquire sensor data, including the current angle data of the robot joints and the scene point cloud data of the target object collected by the depth camera; S2. Integrate scene point cloud data with robot kinematics model to construct a unified spatial model that includes the space occupied by the robot itself and external environmental obstacles; S3. In the unified spatial model, identify the surface contour of the target object and predict the sequence of surface contour changes of the target object during the execution of the robot's grasping action. S4. Based on the predicted surface contour change sequence, reverse the generation of the corresponding robot grasping posture change sequence. S5. Using the sequence of grasping posture changes as the target path, in the unified space model, the robot calculates a series of joint angle changes required to track the target path from the current joint angle, forming a forward-looking joint motion trajectory. S6. Send the joint angle command of the first time node of the forward joint motion trajectory to the robot joint controller, and while executing the command, repeat S1-S5 based on the new sensor data to update the subsequent forward joint motion trajectory.

2. The robot control method based on computer vision according to claim 1, characterized in that, In S2, the process of constructing the unified spatial model is as follows: The robot analyzes the link structure data in its kinematic model and generates a set of tightly packed cubic bounding boxes for each link. The size of the cubic bounding boxes encloses the surface contour of the link structure. The obstacle point cloud is read from the scene point cloud data, and the coordinates of each 3D point in the obstacle point cloud are converted into voxel indices in the 3D voxel mesh world. After obtaining the surface contour change sequence of the target object, the 3D point coordinates of the surface point cloud in each frame of the sequence are converted into voxel indices in the 3D voxel mesh world. In the three-dimensional voxel mesh world, the voxels occupied by the robot link cube bounding box, the voxels where the obstacle point cloud index is located, and the voxels where the target object surface point cloud index is located are all marked as the first state; the unoccupied voxels are marked as the second state. The unified spatial model is composed of the three-dimensional voxel mesh world marked as the first state and the second state.

3. The robot control method based on computer vision according to claim 1, characterized in that, In S3, the process of predicting the surface contour change sequence is as follows: The target object point cloud in the scene point cloud data is segmented, and the surface curvature distribution features of the target object point cloud are extracted using a region growing algorithm. The surface curvature distribution features include the principal curvature values ​​and principal directions of each region on the point cloud surface. A viscoelastic mechanical model of the target object is established. The viscoelastic mechanical model includes a combination structure of spring and damping units. The elastic and viscous parameters of the viscoelastic mechanical model are initialized based on the principal curvature values ​​in the surface curvature distribution features. The dynamic deformation process of stress transmission and energy dissipation within the viscoelastic mechanical model was simulated under the contact pressure applied by the robot's end effector. The dynamic deformation process is discretized into multiple consecutive time frames, and the three-dimensional point set of the target object surface corresponding to each time frame is output to form the surface contour change sequence.

4. The robot control method based on computer vision according to claim 3, characterized in that, The parameter adjustment process of the viscoelastic mechanical model is as follows: After assigning values ​​to the viscoelastic mechanics model, the principal curvature values ​​and principal directions in the surface curvature distribution features are input into a pre-trained parameter mapping network. The parameter mapping network contains three fully connected layers and one normalized output layer. The parameter mapping network outputs the adjustment coefficients of the relaxation time parameter and the steady-state elastic modulus parameter in the viscoelastic mechanics model. The relaxation time parameter and the steady-state elastic modulus parameter are multiplied by the corresponding adjustment coefficients to update the parameters of the viscoelastic mechanical model. The updated viscoelastic mechanical model is then used to predict the surface profile change sequence.

5. The robot control method based on computer vision according to claim 1, characterized in that, In step S4, the process of reverse-generating the robot's grasping posture change sequence is as follows: The initial grasping posture is calculated at the start frame of the surface contour change sequence, and the final grasping posture is calculated at the end frame. Both the initial and final grasping postures include the spatial position and orientation of the end effector and the gripper opening width. Based on the initial and final grasping postures, interpolation calculations are performed between adjacent time frames. Based on the displacement vector of the centroid coordinates of the corresponding regions of the three-dimensional point sets on the surface of the preceding and following frames, the spatial position of the end effector is linearly interpolated; based on the change of the principal normal vector of the surface region of the preceding and following frames, the spatial orientation of the end effector is linearly interpolated; based on the change of the bounding box size of the surface region of the preceding and following frames, the opening width of the gripper is linearly adjusted; and the robot grasping posture change sequence is generated through the above interpolation calculation.

6. The robot control method based on computer vision according to claim 1, characterized in that, In S5, the process of forming a forward-looking joint motion trajectory is as follows: Each grasping posture in the grasping posture change sequence is mapped to multiple candidate joint angle combinations through inverse kinematics of the robot. These candidate joint angle combinations are connected in time order to form an initial joint space path. The time interval between path points on the initial joint space path is consistent with the reciprocal of the robot control cycle. In the unified space model, it is checked whether the robot link profile at each path point on the initial joint space path intersects with the point cloud in the environmental obstacle or the predicted surface profile change sequence. When the voxel intersection is detected, the initial joint space path is locally smoothed and adjusted in the joint space to generate a prospective joint motion trajectory.

7. The robot control method based on computer vision according to claim 6, characterized in that, The process of local smoothing adjustment is as follows: Traverse the initial joint space path, locate the path points where voxel intersections occur and the corresponding intersection links, and record the timestamps of the voxel intersection path points and the identifiers of the intersection links; define a dynamic adjustment time window before and after the timestamps of the intersection path points; within the dynamic adjustment time window, construct a sinusoidal correction curve for a main joint that drives the movement of the intersection links; the amplitude of the sinusoidal correction curve is increased iteratively until the voxel intersections no longer exist, and the iteration step size decreases as the number of iterations increases; At the start and end of the dynamically adjusted time window, the joint angle values ​​of the sine wave correction curve are consistent with the joint angle values ​​of the initial joint space path; the sine wave correction curve is superimposed on the main joint angle sequence of the initial joint space path to form a forward-looking joint motion trajectory after local smoothing adjustment.

8. The robot control method based on computer vision according to claim 1, characterized in that, In step S6, the process of updating the subsequent forward-looking joint motion trajectory is as follows: After executing the first joint motion trajectory, the latest collected scene point cloud data and robot joint angle data are acquired; the latest collected scene point cloud data is compared with the corresponding time frame point cloud in the predicted surface contour change sequence, and the average deviation of the point cloud position is calculated. When the average deviation exceeds the set threshold, the latest collected scene point cloud data and robot joint angle data are used as input to re-execute S2 to S5, generating a new forward-looking joint motion trajectory starting from the next control cycle; the newly generated forward-looking joint motion trajectory replaces the previously generated but not executed joint motion trajectory.

9. A computer vision-based robot control system, used to implement the computer vision-based robot control method according to any one of claims 1-8, characterized in that, include: The data perception module is used to acquire sensor data, including the current angle data of the robot joints and the scene point cloud data of the target manipulator collected by the depth camera. The spatial modeling module is used to fuse scene point cloud data with robot kinematics models to construct a unified spatial model that includes the space occupied by the robot itself and external environmental obstacles. The deformation prediction module is used to identify the surface contour of the target object in a unified spatial model and predict the sequence of surface contour changes of the target object during the execution of the robot's grasping action. The posture planning module is used to inversely generate the corresponding robot grasping posture change sequence based on the predicted surface contour change sequence. The trajectory generation module is used to take the grasping posture change sequence as the target path. In the unified space model, it calculates the series of joint angle changes required for the robot to track the target path from the current joint angle, forming a forward-looking joint motion trajectory. The closed-loop control module is used to send the joint angle command of the first time node of the forward joint motion trajectory to the robot joint controller. While executing the command, the module operation is repeated based on the new sensor data to update the subsequent forward joint motion trajectory.

Images