Robot autonomous obstacle avoidance method and system based on dynamic image analysis

Abstract

The present application belongs to the technical field of digital image data processing and image analysis, and relates to a robot autonomous obstacle avoidance method and system based on dynamic image analysis. The method binds the time sequence lag of visual image sampling, arm claw pose sampling, chassis displacement sampling and space calibration parameters when a composite mobile robot carries a fixture, to obtain a vision-arm correlation frame sequence. The method performs a continuous frame set operation on the vision-arm correlation frame sequence to generate a self-occlusion envelope. The method performs intersection, set and difference set operations on the self-occlusion envelope to generate an occlusion criterion set, extracts exposed edge features to generate a to-be-determined residual contour, performs rigid inverse transformation on the to-be-determined residual contour according to the chassis displacement sampling to generate trajectory displacement parameters, performs outer edge coordinate difference calculation based on the self-occlusion envelope to generate boundary motion parameters, compares the trajectory displacement parameters and the boundary motion parameters to obtain a real obstacle feature set, and extracts a safe connected domain from the chassis displacement sampling, the real obstacle feature set and the occlusion criterion set to generate an obstacle avoidance boundary.

Description

Technical Field

[0001] This invention relates to the field of digital image data processing and image analysis technology, and in particular to a robot autonomous obstacle avoidance method and system based on dynamic image analysis. Background Technology

[0002] With the increasing demand for flexible handling, collaborative machining units, and unmanned turnaround in industrial manufacturing sites, composite mobile robots are gradually being used to move fixtures or workpieces between multiple machining units, buffer stations, and clamping stations. These composite mobile robots typically have a mobile chassis, a vehicle-mounted robotic arm, a clamp, and a vision perception device, enabling them to perform operations such as fixture handling, station approach, path avoidance, and end-effector posture adjustment during chassis movement.

[0003] For composite mobile robots with robotic arms, existing technologies estimate the projection area of ​​the robotic arm itself in the camera's field of view using a robotic arm model and subtract this projection area as an occlusion area caused by the robot's own structure. This can, to some extent, avoid misidentifying the robotic arm itself as an external obstacle, thereby reducing interference from the robot's own structure in obstacle recognition. However, during the process of a composite mobile robot handling a fixture, the onboard robotic arm, gripper, and the fixture's extended parts periodically enter the forward field of view with folding or extending movements, forming an occlusion boundary in the field of view that changes with its own movement. When the robot chassis moves simultaneously, the suspected edge near this occlusion boundary in the forward field of view will be affected by both the chassis movement and the robotic arm itself. The combined effects of motion and changes in the spatial position of external obstacles cause these suspected edges to exhibit similar continuous changes in consecutive frames. In other words, the edges of real external obstacles may appear continuous after chassis motion compensation, while the outer edges of robotic arms, grippers, or jigs may appear continuous due to the synchronous movement of random robotic arm occlusion boundaries. These may all appear as edge objects that continue across frames at the image level, making it difficult to determine whether the continuity of the edge comes from real obstacles in external space or from occlusion boundary residues caused by the robot's own structural motion. This leads to unreliable obstacle avoidance boundary attribution in subsequent obstacle avoidance data, causing the robot to deviate during obstacle avoidance.

[0004] In summary, existing robot obstacle avoidance methods suffer from the problem of failing to accurately define obstacle boundaries, leading to deviations during obstacle avoidance. Summary of the Invention

[0005] Therefore, the technical problem to be solved by the present invention is to overcome the problem that the existing robot obstacle avoidance methods cannot accurately define the obstacle boundary, which causes the robot to deviate during obstacle avoidance.

[0006] To address the aforementioned technical problems, this invention provides a robot autonomous obstacle avoidance method based on dynamic image analysis, comprising: The visual image sampling, arm and gripper pose sampling, chassis displacement sampling and spatial calibration parameters of the composite mobile robot are time-lag bound to obtain the visual arm associated frame sequence. Based on arm and gripper pose sampling and spatial calibration parameters, continuous frame union operation is performed on the visual arm associated frame sequence to generate a self-occlusion envelope for each frame image in the visual image sampling, which is used to characterize the area occupied by the vehicle-mounted robotic arm, gripper and fixture in the frame image. The intersection, union, and difference operations are performed on the self-occlusion envelopes of adjacent frames to generate an occlusion criterion set for each frame, thereby extracting the exposed edge features of each frame and generating the undetermined residual contour of each frame. Based on the chassis displacement corresponding to each frame of the chassis displacement sampling, the undetermined residual contour of each frame of the image is subjected to rigid body inverse transformation to generate trajectory displacement parameters containing the trajectory continuation direction and trajectory continuation displacement amount in each frame of the image. The outer edge coordinate difference is calculated based on the self-occlusion envelope of adjacent frame images to generate boundary motion parameters in each frame image, which include the contour offset direction and the contour offset displacement. The trajectory displacement parameters and boundary motion parameters of each frame image are compared. When the trajectory continuation direction is not parallel to the contour offset direction, or the trajectory continuation displacement amount and the contour offset displacement amount do not increase or decrease proportionally, the real obstacle feature set is obtained based on the undetermined residual contour of that frame image. For each frame of image, safe connected components are extracted from the chassis displacement, real obstacle feature set, and occlusion criterion set to generate the obstacle avoidance boundary for each frame of image.

[0007] Preferably, visual image sampling , This represents the image in frame t. This indicates the number of forward image frames included in a single continuous processing task when a composite mobile robot handles a jig. Arm pose sampling , This represents the angle vector of each joint of the vehicle-mounted robotic arm at the acquisition time of the t-th frame image; Chassis displacement sampling , , This indicates the displacement of the moving chassis at the acquisition time of the t-th frame image. This represents the displacement of the chassis at the moment the t-th frame of the image was acquired. This represents the angle of rotation of the moving chassis at the moment the t-th frame of the image is acquired; Spatial calibration parameters It includes at least the camera intrinsic parameter matrix, the camera extrinsic parameter projection matrix, the base mounting bias matrix, and the extrinsic parameter transformation matrix from the camera to the chassis.

[0008] Preferably, the visual image sampling, arm pose sampling, chassis displacement sampling, and spatial calibration parameters of the composite mobile robot during fixture handling are time-lag bound, including: Using the exposure midpoint of the visual image sampling as the reference time axis, the arm and gripper pose sampling and chassis displacement sampling are time-shifted according to the nominal transmission lag. The translated arm and claw pose samples and chassis displacement samples are bound to each frame of the visual image samples using the nearest neighbor timestamp method.

[0009] Preferably, a continuous frame union operation is performed on the view arm-related frame sequence to generate a self-occlusion envelope for each frame in the visual image sampling, representing the area occupied by the vehicle-mounted robotic arm, gripper, and fixture in that frame image, including: Extract the base mounting offset matrix and camera extrinsic projection matrix from the spatial calibration parameters; The forward kinematics solution is performed on the chassis displacement corresponding to the t-th frame image in the chassis displacement sampling to obtain the three-dimensional spatial outer contour equation of the robotic arm. The three-dimensional spatial outer contour equation is transformed by multiplying it sequentially with the base mounting offset matrix and the camera extrinsic projection matrix to obtain the two-dimensional projected polygon of the t-th frame image in the image coordinate system. After rasterizing the two-dimensional projected polygons of the t-th frame image, the (t-1)-th frame image, and the (t-2)-th frame image to the same pixel grid, a Boolean union operation is performed to obtain the self-occlusion envelope of the t-th frame image.

[0010] Preferably, the intersection, union, and difference operations are performed on the self-occlusion envelopes of adjacent frames to generate an occlusion criterion set for each frame, including: Construct a full-frame image mask to cover the entire image based on the maximum number of pixel rows and columns in the t-th frame; The core occlusion region of the image at frame t is obtained by performing an intersection operation on the overlapping region of the self-occlusion envelope of the image at frame t-1 and the self-occlusion envelope of the image at frame t-1. Perform a region union operation on the self-occlusion envelope of the t-th frame image and the self-occlusion envelope of the (t-1)-th frame image to obtain the merged polygon of the t-th frame image; subtract the core occlusion region of the t-th frame image from the merged polygon of the t-th frame image to obtain the boundary transition region of the t-th frame image. Perform a Boolean difference operation on the merged polygon of the full-frame image mask and the t-th frame image to obtain the outer adjacent region of the t-th frame image; The core occlusion region, boundary transition region, and external adjacent region of the t-th frame image are combined to form the occlusion criterion set of the t-th frame image.

[0011] Preferably, the exposed edge features of each frame image are extracted to generate the undetermined residual contour of each frame image, including: Extract the pixel gradient of the t-th frame image and perform spatial clustering on the pixels whose pixel gradient is greater than a preset gradient threshold to obtain multiple continuous edge point sets of the t-th frame image; The coverage matrix of the image t is obtained by comparing each set of consecutive edge points in the image t with the occlusion criterion set of the image t, and obtaining the proportion of pixels in each set of consecutive edge points that fall into the core occlusion area, the boundary transition area, and the external adjacent area to the total number of pixels in that set. Each row of the coverage matrix corresponds to a set of consecutive edge points, and the three columns correspond to the proportion of pixels in the set of consecutive edge points that fall into the core occlusion area, the boundary transition area, and the external adjacent area, respectively. Based on the coverage matrix of the t-th frame image, the set of continuous edge points that fall into the core occlusion area with a pixel ratio greater than or equal to a preset removal threshold is removed to obtain the set of remaining continuous edge points of the t-th frame image. Calculate the Hausdorff matrix of the remaining continuous edge point set of the t-th frame image and the remaining continuous edge point set of the (t-1)-th frame image to obtain the feature distance of the t-th frame image; Based on the chassis displacement corresponding to the t-th frame image and the (t-1)-th frame image in the chassis displacement sampling, the inter-frame translation increment and inter-frame rotation increment of the t-th frame image are calculated. Based on the spatial calibration projection matrix, the inter-frame translation increment and inter-frame rotation increment are mapped to the image pixel coordinate system. Thus, the motion reprojection calculation is performed on the remaining continuous edge points in the set of remaining continuous edge points to obtain the theoretical inter-frame pixel deviation of the t-th frame image caused by the rigid body motion of the chassis. The inter-frame pixel deviation of the t-th frame image is obtained by superimposing a preset pixel noise margin on the theoretical inter-frame pixel deviation of the t-th frame image. The feature distance of the t-th frame image is compared with the inter-frame pixel deviation of the t-th frame image. When the feature distance is less than the inter-frame pixel deviation, the remaining continuous edge point set of the t-th frame image is used as the residual contour. When the feature distance is greater than or equal to the inter-frame pixel deviation, the undetermined residual contour of the t-th frame image is not output.

[0012] Preferably, based on the chassis displacement corresponding to each frame of the chassis displacement sampling, an inverse rigid body transformation is performed on the undetermined residual contour of each frame of the image to generate trajectory displacement parameters in each frame of the image, including the trajectory continuity direction and the amount of trajectory continuity displacement, including: Multiply each pixel coordinate point in the undetermined residual contour of the t-th frame image by the inverse matrix of the camera intrinsic matrix in the spatial calibration parameters to obtain the three-dimensional observation ray vector; rotate and translate the three-dimensional observation ray vector using the camera extrinsic transformation matrix in the spatial calibration parameters to obtain the global ray equation of the t-th frame image. The vertical axis translation component of the camera extrinsic transformation matrix is ​​extracted to obtain the ground height reference plane. The spatial geometric intersection of the ground height reference plane and the global ray equation of the t-th frame image is calculated, thereby obtaining the spatial geometric intersection of each pixel in the undetermined residual contour of the t-th frame image. Based on the arrangement order of pixels in the undetermined residual contour, the spatial geometric intersection points are arrayed and stitched to obtain the chassis base point sequence of the t-th frame image; Based on the integral of the chassis linear velocity in the chassis displacement over the time interval between the t-th frame image and the (t-1)-th frame image, the translation inverse compensation vector of the t-th frame image is obtained; based on the integral of the chassis angular velocity in the chassis displacement over the time interval between the t-th frame image and the (t-1)-th frame image, the rotation inverse transformation matrix of the t-th frame image is obtained. Based on the rotation inverse transformation matrix, translation inverse compensation vector, and chassis base point sequence of the t-1 frame image, the rigid body compensation point sequence of the t-1 frame image is obtained. Calculate the mean coordinates of the rigid body compensation point sequence of the (t-1)th frame image and the chassis foundation point sequence of the tth frame image to obtain the preceding points of the (t-1)th frame image and the points of the tth frame image. The difference calculation result of the t-th frame image is obtained based on the difference between the points of the t-th frame image and the preceding points of the (t-1)-th frame image. Calculate the projection angle of the difference calculation result of the t-th frame image onto the horizontal bottom plane, and generate the trajectory continuation direction of the t-th frame image; calculate the Euclidean distance modulus of the difference calculation result of the t-th frame image, and generate the trajectory continuation displacement of the t-th frame image. Based on the trajectory continuation direction and trajectory continuation displacement of the t-th frame image, the trajectory displacement parameters of the t-th frame image are obtained.

[0013] Preferably, the outer edge coordinate difference is calculated based on the self-occlusion envelope of adjacent frame images to generate boundary motion parameters in each frame image, including the contour offset direction and contour offset displacement, including: Boundary contours are extracted from the self-occlusion envelope of the t-th frame image and the self-occlusion envelope of the (t-1)-th frame image to obtain the outer edge of the t-th frame image and the preceding outer edge of the (t-1)-th frame image. The second derivative of each pixel on the outer edge of the t-th frame image is calculated to obtain the curvature extremum points. The intersection of straight lines along the normal direction of each curvature extremum point on the preceding outer edge of the (t-1)-th frame image is solved to obtain the spatial corresponding points of each curvature extremum point. Obtain the image pixel coordinates of each curvature extremum point in the t-th frame image and the ground space mapping relationship corresponding to each curvature extremum point. Then, back-project each curvature extremum point and its corresponding spatial point to the ground space. Perform coordinate subtraction on the curvature extremum points back-projected to the ground space and their corresponding spatial points to obtain the physical space offset of the t-th frame image. Extract the direction angle of the physical space offset of the t-th frame image to obtain the contour offset direction of the t-th frame image; calculate the Euclidean distance modulus of the physical space offset of the t-th frame image to obtain the contour offset displacement of the t-th frame image. Based on the contour offset direction and contour offset displacement of the t-th frame image, the boundary motion parameters of the t-th frame image are obtained.

[0014] Preferably, safe connected components are extracted from the chassis displacement, real obstacle feature set, and occlusion criterion set corresponding to each frame of image to generate the obstacle avoidance boundary for each frame of image, including: By performing a double integral in the time dimension on the chassis linear velocity in the chassis displacement corresponding to the t-th frame image in the chassis displacement sampling, the cumulative displacement range of the moving chassis corresponding to the t-th frame image is obtained, thereby dividing the t-th frame image into a two-dimensional initial empty grid array with the grid as the smallest unit. The real obstacle feature set of the t-th frame image is mapped to a two-dimensional initial empty grid array through spatial calibration parameters to obtain the obstacle feature points of the t-th frame image; the core occlusion region and boundary transition region in the occlusion criterion set of the t-th frame image are mapped to a two-dimensional initial empty grid array through spatial calibration parameters to obtain the visual occlusion points of the t-th frame image. The elements of the initial two-dimensional empty grid array of the t-th frame image are traversed. Grids containing obstacle feature points are marked as grids occupied by entity collisions, and grids containing visual occlusion points but not obstacle feature points are marked as unobservable blind zone grids. The obstacle avoidance boundary of the t-th frame image is obtained by subtracting the grid cells occupied by entity collisions and the grid cells of unobservable blind zones from the two-dimensional initial empty grid array of the t-th frame image.

[0015] This invention also provides a robot autonomous obstacle avoidance system based on dynamic image analysis, comprising: The data acquisition and sampling timing binding module is used to perform timing lag binding on visual image sampling, arm and gripper pose sampling, chassis displacement sampling and spatial calibration parameters when the composite mobile robot is handling the jig, so as to obtain the visual arm associated frame sequence. The self-occlusion envelope acquisition module is used to perform continuous frame union operation on the visual arm associated frame sequence based on arm pose sampling and spatial calibration parameters, and generate a self-occlusion envelope for each frame image in the visual image sampling to characterize the area occupied by the vehicle-mounted robotic arm, gripper and fixture in the frame image. The residual contour extraction module is used to perform intersection, union and difference operations on the self-occlusion envelopes of adjacent frame images to generate an occlusion criterion set for each frame image, thereby extracting the exposed edge features of each frame image and generating the undetermined residual contour of each frame image. The first motion parameter generation module is used to perform rigid body inverse transformation on the undetermined residual contour of each frame image based on the chassis displacement corresponding to each frame image in the chassis displacement sampling, and generate trajectory displacement parameters in each frame image containing the trajectory continuation direction and trajectory continuation displacement amount. The second motion parameter generation module is used to calculate the outer edge coordinate difference based on the self-occlusion envelope of adjacent frame images, and generate boundary motion parameters in each frame image containing the contour offset direction and contour offset displacement. The residual contour determination and attribution module is used to compare the trajectory displacement parameters and boundary motion parameters of each frame image. When the trajectory continuation direction is not parallel to the contour offset direction, or the trajectory continuation displacement amount and the contour offset displacement amount do not increase or decrease proportionally, the real obstacle feature set is obtained based on the undetermined residual contour of the frame image. The obstacle avoidance boundary generation module is used to extract safe connected components from the chassis displacement, real obstacle feature set and occlusion criterion set corresponding to each frame of image, and generate the obstacle avoidance boundary of each frame of image.

[0016] The robot autonomous obstacle avoidance method based on dynamic image analysis provided in this application has the following beneficial effects: The self-occlusion region, which changes with the movement of the robotic arm, is transformed into a two-dimensional projected polygon that can be calculated frame by frame in the image coordinate system through forward kinematics calculation and coordinate projection. This polygon is then expanded outwards by combining three adjacent frames to form a self-occlusion envelope covering the robotic arm's edge movement. By applying intersection, union, and difference operations to the self-occlusion envelopes of adjacent frames and combining them with a full-frame image mask, the image is divided into occlusion criterion sets. These three types of partitions are used to partition and remove continuous edge point sets. Then, the comparison between Hausdorff distance and inter-frame pixel deviation is used as a cross-frame continuation criterion. Residual points in areas with high incidence of self-edge residue and cross-frame continuation are clustered into undetermined residual contours. Finally, based on chassis displacement... The sampling process involves performing an inverse rigid body transformation on the residual profile to unify the residual profiles of adjacent frames under the same chassis coordinate reference. This yields the trajectory continuity direction and trajectory continuity displacement, thereby eliminating the profile drift caused by the chassis's own motion and ensuring that the inter-frame displacement of the residual profile reflects its true continuity state relative to the environment. Furthermore, by extracting the self-occlusion envelope of adjacent frames and calculating the outer edge coordinate difference, the profile offset direction and profile offset displacement are obtained, thus quantifying the synchronous sweeping motion of the robot arm's self-occlusion boundary. The two are compared, and when the directions are not parallel or the displacement is not proportionally increased or decreased, they are written into the real obstacle feature set. This eliminates pseudo-residuals that move synchronously with the self-occlusion boundary, reducing the probability that the robot arm's boundary residue is misjudged as an external obstacle. By extracting pixel gradients and spatially clustering them to form a continuous set of edge points, discrete noisy edges can be constrained into traceable contours. The distribution ratio of these points in three types of occlusion areas is statistically analyzed to form a coverage matrix, which can quantify the relationship between edges and occlusion areas. Point sets that fall completely into the core occlusion area are removed to eliminate self-occluded edges of the robotic arm. The Hausdorff distance between the remaining point set and the previous frame can be calculated to determine the continuity of the contour across frames. The inter-frame pixel deviation is generated from the chassis displacement and calibration parameters to compensate for the robot's self-motion. When the feature distance is lower than this deviation, an undetermined residual contour is output, which can preserve exposed edges with temporal consistency. Attached Figure Description

[0017] To make the content of this invention easier to understand, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings, wherein: Figure 1 A flowchart of the robot autonomous obstacle avoidance method based on dynamic image analysis provided in this application; Figure 2 A block diagram illustrating the principle of robot autonomous obstacle avoidance based on dynamic image analysis provided in this application; Figure 3 The flowchart for generating a self-occluding envelope provided in this application; Figure 4 A flowchart for generating the profile of the undetermined residual provided in this application; Figure 5 A flowchart for generating trajectory displacement parameters provided in this application; Figure 6 A schematic diagram of the structure of the robot autonomous obstacle avoidance system based on dynamic image analysis provided in this application. Detailed Implementation

[0018] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments described are not intended to limit the present invention.

[0019] Based on the above issues, please refer to the following: Figure 1 and Figure 2 , Figure 1 The diagram shown is a flowchart of the robot autonomous obstacle avoidance method based on dynamic image analysis provided in this application. Figure 2 The block diagram of the robot autonomous obstacle avoidance principle based on dynamic image analysis provided in this application is applicable to the self-occlusion obstacle avoidance scenario of the robotic arm when the composite mobile robot is carrying a jig. The composite mobile robot usually has a mobile chassis, a vehicle-mounted robotic arm, a gripper and a visual perception device. It can complete operations such as jig carrying, workstation approach, path avoidance and end-effector posture adjustment during the movement of the mobile chassis.

[0020] It should be noted that the method in this embodiment does not rely on dedicated circuit structures or external positioning markers. Its processing targets are visual image samples obtained by the visual perception device, and simultaneously acquired arm and gripper pose samples, chassis displacement samples, and spatial calibration parameters. Obstacle boundary attribution is primarily determined through self-occlusion envelope generation, occlusion criterion set division, exposed edge feature extraction, dual-path motion parameter comparison, and secure connected domain image segmentation. Furthermore, since the onboard robotic arm, gripper, and extended parts of the fixture periodically enter the forward field of view during the handling process of the composite mobile robot, forming occlusion boundaries that change with its own movement, and the movement of the mobile chassis causes continuous displacement of the actual external obstacles in the forward field of view, some embodiments of this application set reusable parameter constraints and anomaly handling rules at each processing stage.

[0021] Specifically, the robot autonomous obstacle avoidance method based on dynamic image analysis provided in this application includes: S10~S70.

[0022] S10: Perform time-lag binding on visual image sampling, arm and gripper pose sampling, chassis displacement sampling, and spatial calibration parameters when the composite mobile robot is handling the fixture to obtain the visual arm associated frame sequence.

[0023] Specifically, visual image sampling , Represents the image of frame t. This indicates the number of forward image frames included in a single continuous processing task when the composite mobile robot is handling a jig. (Arm / gripper pose sampling) , This represents the angle vectors of each joint of the vehicle-mounted robotic arm at the acquisition time of the t-th frame image. (Chassis displacement sampling) , , This indicates the displacement of the moving chassis at the acquisition time of the t-th frame image. This represents the displacement of the chassis at the moment the t-th frame of the image was acquired. This represents the rotation angle of the moving chassis at the acquisition time of the t-th frame image, and the spatial calibration parameters. It includes at least the camera intrinsic parameter matrix, the camera extrinsic projection matrix, the base mounting bias matrix, and the camera-to-chassis extrinsic transformation matrix. It is worth noting that the spatial calibration parameters are a set of parameters acquired during the calibration phase and remain unchanged during the execution of a single continuous task.

[0024] It should be noted that, in the embodiments of this application, chassis displacement sampling and arm pose sampling are only used as auxiliary parameters for dynamic image registration, rigid body inverse compensation and occlusion mask projection. Their function is to correct the pixel positions of edges, contours and region labels in the image, and they are not used to generate the driving trajectory, speed control amount or execute actions of the moving chassis.

[0025] Specifically, since the self-occlusion boundary is determined by the actual pose of the vehicle-mounted robotic arm and the image displacement of the external obstacle is determined by the actual movement of the mobile chassis, the two must be separated under a unified time reference and coordinate reference. Therefore, the embodiments of this application simultaneously acquire visual image sampling, arm pose sampling, chassis displacement sampling and spatial calibration parameters, and incorporate these four types of parameters into the same processing chain.

[0026] Furthermore, since visual image sampling, arm pose sampling, and chassis displacement sampling come from three types of sensors—camera, robotic arm joint encoder, and chassis odometer—the reporting of these three types of data has inherent transmission and exposure delays. If they are directly aligned according to the order of arrival, the image content and robotic arm pose within the same frame will be mismatched in time, leading to a shift between the subsequently generated self-occlusion envelope and the actual position of the robotic arm in the image. Therefore, this application performs time-lag binding on the visual image sampling, arm pose sampling, chassis displacement sampling, and spatial calibration parameters when the composite mobile robot is handling the jig. Specifically, this includes: S100~S101.

[0027] S100: Using the exposure midpoint of the visual image sampling as the reference time axis, the arm and gripper pose sampling and chassis displacement sampling are time-shifted according to the nominal transmission lag.

[0028] S101: Bind the translated arm and claw pose samples and chassis displacement samples to each frame of the visual image samples using the nearest neighbor timestamp method.

[0029] Optionally, to address potential missing or abnormal situations in continuous data input, this application embodiment also sets data boundary processing rules: when a certain type of sampling experiences a single-frame loss in a certain frame and the number of consecutive lost frames is less than the tolerable number of frames... When the number of consecutive lost frames is greater than or equal to the number of lost frames, the frame is bridged using the linear interpolation results of its preceding and following valid frames, so that the sequence of related frames of the view arm remains continuous in time; If this period is marked as undecidable, the current determination is interrupted, and the view arm associated frame sequence is re-established after the end of this period. Since cross-frame comparison of the self-occlusion envelope depends on the continuity of adjacent frames, a tolerance frame number is set. Its value ranges from 2 to 5 frames, with a typical value of 3 frames.

[0030] It is worth noting that the view arm associated frame sequence generated by the time-delay binding simultaneously carries three types of information: image content, robot arm pose, and chassis motion. The robot arm pose is used to generate the self-occlusion envelope, and the chassis motion is used to perform rigid body inverse transformation on the determined residual contour. It is precisely because the three types of information are bound to the same frame under the same time reference that the boundary motion parameters from the robot arm and the trajectory displacement parameters from the chassis can be applied to the same object respectively, and the two can be compared. This is the prerequisite for this application to distinguish between self-occlusion boundaries and real obstacles.

[0031] S20: Perform continuous frame union operation on the visual arm associated frame sequence based on arm and gripper pose sampling and spatial calibration parameters to generate a self-occlusion envelope for each frame of the visual image sampling, which represents the area occupied by the vehicle-mounted robotic arm, gripper, and fixture in that frame.

[0032] Furthermore, such as Figure 3 As shown, step S20 includes: S200~S203.

[0033] S200: Extracting the base mounting offset matrix from spatial calibration parameters and camera extrinsic projection matrix .

[0034] Specifically, the base mounts the offset matrix. This represents the fixed installation pose of the robotic arm base coordinate system relative to the moving chassis coordinate system, and the camera extrinsic projection matrix. This represents the projection relationship from the moving chassis coordinate system to the camera image coordinate system.

[0035] S201: The chassis displacement sampled in the image corresponding to the t-th frame. By performing forward kinematics calculations, the three-dimensional outer contour equation of the robotic arm is obtained. .

[0036] Specifically, the forward kinematics solution calculates the three-dimensional spatial position of each link surface in the coordinate system of the robot arm base by multiplying the link transformation matrix link by link, based on the geometric dimensions and joint angles of each link of the robot arm. The envelope of each link surface is then fitted to the three-dimensional outer contour equation. .

[0037] For example, let the joint angle vector of frame t be... =( , ..., The homogeneous transformation from the robot arm base to the coordinate system of the i-th link is as follows: , , in, Let represent the homogeneous transformation matrix from the robot arm base coordinate system to the i-th link coordinate system. , and Let these be the calibration structural parameters for the k-th link. and Using radians, , The translation components of the homogeneous transformation matrix use the same unit of length; the rotation submatrix is ​​dimensionless; Let represent the DH homogeneous transformation matrix from the coordinate system of the -1st link to the coordinate system of the kth link; Represents the rotation matrix about the z-axis; This represents the translation matrix along the z-axis; Represents the translation matrix along the x-axis; Represents the rotation matrix about the x-axis; For the homogeneous vertex in the local model of the i-th link Its transformations in the moving chassis coordinate system and the image coordinate system satisfy: , , in, Represents the homogeneous vertex in the local model of the i-th link. Transform to homogeneous three-dimensional coordinates in the moving chassis coordinate system. This represents the transformation of a local vertex of the link to a homogeneous three-dimensional coordinate point in the coordinate system of the robot arm's base. Represents the image pixels in homogeneous form. For the camera intrinsic parameter matrix, and These represent the rotation and translation components from the moving chassis coordinate system to the camera coordinate system. The homogeneous scale factor is used to divide the projection point by the third homogeneous component to obtain the two-dimensional coordinates in pixels. Then, the union of the projection contours of each link, fixture, and jig model is obtained to get the two-dimensional projection polygon of the t-th frame.

[0038] It should be noted that this application sets out configuration boundary treatment rules for the geometric configuration assumptions upon which the forward kinematics solution depends: the three-dimensional spatial outer contour equation. The default fitting method assumes that the outer surface of each link of the robotic arm is approximately enclosed by a finite number of convex polyhedra. When the robotic arm is in a special pose where its links overlap, causing the convex polyhedron envelope to be significantly larger than the actual occlusion range, an alternative method is used to generate two-dimensional projected polygons by projecting each link segment separately and then finding the union of the segments, in order to avoid excessive expansion of the self-occlusion envelope.

[0039] S202: Equation of the outer contour in three-dimensional space Offset matrices are installed sequentially with the base. and camera extrinsic projection matrix Perform a multiplication transformation to obtain the two-dimensional projection polygon of the t-th frame image in the image coordinate system. .

[0040] For example, a single-frame two-dimensional projected polygon The calculation expression is: .

[0041] S203: Project the two-dimensional polygon of the t-th frame image. The two-dimensional projection polygon of the (t-1)th frame image and the two-dimensional projection polygon of the image in frame t-2. After rasterizing to the same pixel grid, a Boolean union operation is performed to obtain the self-occlusion envelope of the t-th frame image. .

[0042] It should be noted that the reason why the Boolean union operation is performed on the two-dimensional projected polygons of the current frame and the two adjacent frames in this embodiment, instead of just the current frame, is because the robotic arm moves continuously between adjacent frames. Within the exposure and timing binding error range, the edge of the robotic arm will swing slightly between adjacent frames. Taking only the two-dimensional projected polygon of the current frame will make the edge of the self-occlusion envelope slightly smaller than the actual occlusion range of the robotic arm, so that the self-edge of the robotic arm that is close to the edge of the robotic arm will remain exposed outside the self-occlusion envelope. Taking the union of three frames can make the self-occlusion envelope expand appropriately in the direction of the robotic arm movement, covering the swing range of the edge of the robotic arm. The time window corresponding to the union of three frames is the three adjacent frames. The basis is that the displacement of the robotic arm in a single frame usually does not exceed a few pixels within this time window. Three frames are sufficient to cover without excessively expanding and encroaching on the real obstacle area.

[0043] In addition, the arm pose sampling and spatial calibration parameters on which the self-occlusion envelope is generated are all from the view arm associated frame sequence generated by temporal lag binding. The arm pose sampling is aligned with the corresponding image frame at the exposure time in the temporal lag binding, thereby ensuring that the self-occlusion envelope obtained by arm pose sampling corresponds to the actual position of the robot arm in the image at the pixel level. This correspondence is the prerequisite for the accuracy of subsequent division of the occlusion criterion set based on the self-occlusion envelope and calculation of boundary motion parameters based on the outer edge of the self-occlusion envelope.

[0044] Specifically, the core concept of the self-occlusion envelope generation scheme in this application is to transform the occlusion area of ​​its own structure, which changes with the movement of the robotic arm, into a two-dimensional projected polygon that can be calculated frame by frame in the image coordinate system through forward kinematics calculation and coordinate projection. The polygon is then expanded outward to cover the swing of the robotic arm edge by combining three adjacent frames. The difference between this application and the existing technology, which only uses the projection area of ​​a single frame robotic arm model as the occlusion area, is that the self-occlusion envelope obtained in this application covers the continuous movement of the robotic arm edge in the time dimension. It can replace the existing single-frame projection subtraction technology and the occlusion mask technology based on a fixed shape template.

[0045] S30: Perform intersection, union, and difference operations on the self-occlusion envelopes of adjacent frame images to generate an occlusion criterion set for each frame image, thereby extracting the exposed edge features of each frame image and generating the undetermined residual contour of each frame image.

[0046] Furthermore, this application constructs a region in the time dimension of the cross-frame swing of the cover robotic arm edge, thereby generating a self-occlusion envelope which is used as the geometric input for dividing the core occlusion region, the boundary transition region and the external adjacent region in the occlusion criterion set division process. Specifically, step S30 includes: S300~S311.

[0047] S300: Construct a full-frame image mask to cover the entire image based on the maximum number of pixel rows and columns in the t-th frame. .

[0048] Specifically, full-frame image mask It is a two-dimensional region of the same size as the visual image sample, with all pixels set to valid, used to represent the entire pixel range that the current frame image can cover.

[0049] S301: Self-occlusion envelope of the t-th frame image Self-occlusion envelope of the (t-1)th frame image Perform an intersection operation on the overlapping regions to obtain the core occlusion region of the t-th frame image. .

[0050] Specifically, the core occlusion region refers to the image area that is occluded by the robotic arm in both adjacent frames. It is a stable occlusion region that is always occluded by the robotic arm and where external obstacles are difficult to appear.

[0051] S302: Self-occlusion envelope of the t-th frame image Self-occlusion envelope of the (t-1)th frame image Perform a region union operation to obtain the merged polygon of the t-th frame image; subtract the core occlusion region of the t-th frame image from the merged polygon of the t-th frame image to obtain the boundary transition region of the t-th frame image. .

[0052] Specifically, the boundary transition zone refers to the image area that is occluded by the robotic arm in only one of two adjacent frames, that is, the area that is newly entered or newly exited from occlusion as the robotic arm moves, and is the area where self-edge residue is most likely to occur.

[0053] S303: Full-frame image mask Perform a Boolean difference operation with the merged polygon of the t-th frame image to obtain the outer adjacent region of the t-th frame image. .

[0054] Specifically, the external adjacency region refers to the image area that is not obscured by the robotic arm in two adjacent frames, and is the area where real external obstacles can be fully observed.

[0055] S304: Occlude the core occlusion region of the t-th frame image. Boundary transition zone and external adjacent areas The combination serves as the occlusion criterion set for the t-th frame image.

[0056] Specifically, the occlusion criterion set is a partitioning map that assigns each pixel of the current frame image to one of the core occlusion region, the boundary transition region, or the external adjacent region, and is used to subsequently determine the occlusion state of exposed edge features.

[0057] It should be noted that the occlusion criterion set divides images into three categories: core occlusion region, boundary transition region, and external adjacent region. This is based on the three different levels of confidence in exposed edge features corresponding to these regions: edge points falling into the core occlusion region are stably occluded by the robotic arm and are unlikely to be real external obstacles, thus they should be removed; edge points falling into the boundary transition region are in areas with a high incidence of self-edge residue and require further comparison of dual-path motion parameters to determine their classification; edge points falling into the external adjacent region can be observed completely and are most likely real external obstacles. These three regions together constitute the criteria for subsequent residual contour extraction and classification determination. Their values ​​are the pixel sets covered by each of the three regions, and the update condition is to recalculate once for each new frame of self-occlusion envelope.

[0058] This application divides the image into a core occlusion region, a boundary transition region, and an external adjacent region by applying intersection, union, and difference operations to the self-occlusion envelopes of adjacent frames and combining them with a full-frame image mask. Unlike the conventional processing method that treats the occluded region as a single binary mask for overall subtraction, this embodiment classifies the image in terms of the stability and transition of the occlusion state. The resulting occlusion criterion set is used as a partitioning criterion for removing occluded edges and locating high-frequency residual regions in the subsequent exposed edge feature extraction stage.

[0059] Furthermore, such as Figure 4 The diagram shown is a flowchart for generating undetermined residual profiles provided in this application.

[0060] S305: Extract the pixel gradient of the t-th frame image and perform spatial clustering on pixels with pixel gradients greater than a preset gradient threshold to obtain multiple continuous edge point sets of the t-th frame image. .

[0061] Specifically, a continuous edge point set is a set of edge pixels in an image that have significant gray-level jumps and extend continuously in space.

[0062] It should be noted that the preset gradient threshold of the gradient magnitude is used to distinguish between real edges and image noise. Its value ranges from 10% to 25% of the dynamic range of the image gradient magnitude, with a typical value of 15%.

[0063] S306: Compare each set of consecutive edge points in the t-th frame image with the occlusion criterion set of the t-th frame image, and obtain the proportion of pixels in each set of consecutive edge points that fall into the core occlusion region, the boundary transition region, and the external adjacent region to the total number of pixels in that set, thus obtaining the coverage matrix of the t-th frame image. In the coverage matrix, each row corresponds to a continuous set of edge points, and the three columns correspond to the percentage of pixels in the core occlusion area, the boundary transition area, and the external adjacent area of ​​the continuous edge point set.

[0064] Specifically, the sum of the percentages of pixels falling into the core occlusion area, the boundary transition area, and the external adjacent area is 1.

[0065] S307: Based on the coverage matrix of the t-th frame image, remove the set of continuous edge points whose pixel proportion falling into the core occlusion area is greater than or equal to a preset removal threshold to obtain the remaining set of continuous edge points in the t-th frame image. .

[0066] Specifically, the basis for excluding continuous edge point sets in this application is that edge points that fall completely or almost completely into the core occlusion area are located in the stable occlusion area of ​​the robotic arm and cannot originate from real external obstacles.

[0067] It should be noted that the preset culling threshold for falling into the core occlusion area is used to cullate edge points that are stably occluded. Its value ranges from 0.8 to 0.95, with a typical value of 0.9. That is, when more than 90% of the pixels of a continuous set of edge points fall into the core occlusion area, they are culled.

[0068] S308: Calculate the set of remaining continuous edge points in the t-th frame image. Hausdorff matrix of the remaining continuous edge point set of the (t-1)th frame image The feature distance of the t-th frame image is obtained. .

[0069] Specifically, the feature distance measures the maximum mismatch between the remaining point set of the current frame and the remaining point set of the previous frame on the image plane, and its specific calculation formula is as follows: , in, , These are the pixels in the two remaining point sets, The Euclidean distance between two pixels. , These are the supremum and infimum, respectively.

[0070] S309: Based on the chassis displacement corresponding to the t-th frame image and the (t-1)-th frame image in the chassis displacement sampling, calculate the inter-frame translation increment and inter-frame rotation increment of the t-th frame image. Based on the spatial calibration projection matrix, map the inter-frame translation increment and inter-frame rotation increment to the image pixel coordinate system, thereby performing motion reprojection calculation on the remaining continuous edge points in the set of remaining continuous edge points to obtain the theoretical inter-frame pixel deviation of the t-th frame image caused by the rigid body motion of the chassis.

[0071] Specifically, the spatial calibration projection matrix refers to the comprehensive matrix that transforms 3D spatial points in the world coordinate system of the chassis / base to 2D image points in the camera image pixel coordinate system. This comprehensive matrix is ​​obtained from the camera intrinsic parameter matrix, the camera extrinsic parameter projection matrix, the base mounting offset matrix, and the extrinsic parameter transformation matrix from the camera to the chassis.

[0072] Specifically, motion reprojection calculation is performed on the remaining continuous edge points in the set of remaining continuous edge points to obtain the theoretically predicted pixel coordinates of the remaining continuous edge points on the t-th frame image; the actual observed pixel coordinates of each remaining continuous edge point on the t-th frame image are compared with the corresponding theoretically predicted pixel coordinates of the rigid body motion to obtain the inter-frame theoretical pixel deviation of the t-th frame image caused by the rigid body motion of the chassis.

[0073] It should be noted that if the number of remaining continuous edge points is 1, the theoretical inter-frame pixel deviation of the image in frame t caused by the rigid body motion is obtained by directly calculating the difference between the actual observed pixel coordinates of the remaining edge point in frame t and the corresponding theoretically predicted pixel coordinates of the rigid body motion. If the number of remaining continuous edge points is greater than 1, the average of the difference between the actual observed pixel coordinates of the remaining edge points in frame t and the corresponding theoretically predicted pixel coordinates of the rigid body motion is calculated to obtain the theoretical inter-frame pixel deviation of the image in frame t caused by the rigid body motion of the chassis.

[0074] S310: Add a preset pixel noise margin to the theoretical inter-frame pixel deviation of the t-th frame image to obtain the inter-frame pixel deviation amount of the t-th frame image. .

[0075] Specifically, the inter-frame pixel deviation represents the expected pixel displacement of edge points on the image plane caused solely by the movement of the moving chassis between adjacent frames.

[0076] S311: Compare the feature distance of the t-th frame image with the inter-frame pixel deviation of the t-th frame image. When the feature distance is less than the inter-frame pixel deviation, use the remaining continuous edge point set of the t-th frame image as the residual contour. When the feature distance is greater than or equal to the inter-frame pixel deviation, do not output the undetermined residual contour of the t-th frame image.

[0077] Specifically, using the feature distance being lower than the inter-frame pixel deviation as the criterion for the continuation of the remaining point set means that: if the maximum mismatch between the remaining point set and adjacent frames does not exceed the expected pixel displacement caused only by chassis movement, then the remaining point set is considered a continuation of the same object between adjacent frames and can be used as a pending residual contour for subsequent assignment determination; conversely, if the feature distance is greater than or equal to the inter-frame pixel deviation, then the remaining point sets of adjacent frames cannot be identified as the same object, the current frame does not output a pending residual contour, and it is left for rematching in subsequent frames.

[0078] It should be noted that the generation of the undetermined residual contours reuses both the occlusion criterion set and chassis displacement sampling: the occlusion criterion set is derived from the intersection, union, and difference operations of the self-occlusion envelopes of adjacent frames, used to remove edge points falling into the core occlusion area and retain edge points in the boundary transition area and the external adjacent area; the chassis displacement sampling is transformed by multiplying the projection components from physical space to image pixels in the spatial calibration parameters to obtain the inter-frame pixel deviation, which serves as the comparison benchmark for the cross-frame continuation of the remaining point set. Because the expected pixel displacement caused solely by chassis movement is used as the continuation criterion, this embodiment of the application eliminates stray edges whose displacement amplitude significantly exceeds the interpretable range of chassis movement during the generation of the undetermined residual contours, making the quantity and quality of the undetermined residual contours entering the subsequent dual-path comparison more controllable.

[0079] The core concept of this application's scheme for generating undetermined residual contours lies in first partitioning and eliminating continuous edge point sets using three types of partitions in the occlusion criterion set, and then using the comparison between Hausdorff distance and inter-frame pixel deviation as a cross-frame continuation criterion to aggregate the remaining points in the high-incidence area of ​​self-edge residues and cross-frame continuation as undetermined residual contours. The difference between this scheme and existing technologies that directly perform continuous tracking of the edges of the entire image after deducting its own projection is that this scheme has already performed dual screening of edge points by combining occlusion state partitioning and chassis motion before entering the attribution determination. This can replace existing full-image edge continuous tracking technology and technology that only uses a fixed threshold to determine whether the edge is continuous.

[0080] It should be noted that this application also sets applicable condition boundary processing rules for the applicable conditions on which the cross-frame continuation judgment depends: the calculation of Hausdorff distance and the comparison of inter-frame pixel deviation are based on the premise that adjacent frames correspond to the same handling condition and the robotic arm has not undergone a pose change. When the arm and gripper pose sampling jumps beyond the pose change threshold between adjacent frames, the remaining point set of the current frame and the remaining point set of the previous frame no longer have a comparable correspondence. At this time, the continuation judgment of the current undetermined residual contour is terminated and the starting frame for cross-frame matching is re-established with the remaining point set of the current frame.

[0081] Furthermore, this application differs from the conventional method of directly tracking the edges of the entire image after deducting its own projection by performing occlusion partitioning and culling on the continuous edge point set using the coverage matrix and comparing the Hausdorff distance with the inter-frame pixel deviation to filter the remaining point set that continues across frames. Subsequently, it performs pre-screening on two levels: the occlusion status of the edge points and the interpretability of the chassis motion. The resulting undetermined residual contour is used as a unified input object for applying trajectory displacement parameters and boundary motion parameters in the subsequent dual-path motion parameter generation stage.

[0082] From the overall logical framework of the process, the self-occlusion envelope describes the image region occupied by the vehicle-mounted robotic arm, gripper, and fixture in the current frame. The occlusion criterion set divides the image into different occlusion state partitions based on the self-occlusion envelopes of adjacent frames. The undetermined residual contour is extracted from visual image sampling, and its attribution—whether it originates from real external obstacles or from a cluster of continuous edge points remaining on the body's own edges—is yet to be determined. The three are interconnected: the self-occlusion envelope is the input to the occlusion criterion set, and the occlusion criterion set is the basis for filtering exposed edge features and generating the undetermined residual contour.

[0083] Furthermore, the reason this application preserves and tracks the self-occlusion envelope across frames, rather than discarding it after subtracting the robot arm's projected area, is that the occlusion boundary of the vehicle-mounted robot arm continuously moves with folding or extending movements during the jig handling process. The residual self-edges that are close to this occlusion boundary and not completely subtracted will move synchronously with the occlusion boundary. If the self-occlusion envelope is not preserved, it is difficult to obtain the motion reference that these residual self-edges should follow. This design solves the problem of residual self-edges on the outer edge of the robot arm, gripper, or jig being misjudged as continuously existing real obstacles because they move synchronously with the occlusion boundary. This is because it transforms the self-occlusion area, which would otherwise be discarded once, into a geometric reference that can be compared with the obstacle's movement, rather than a natural extension of the existing technology's approach of simply subtracting the projected area.

[0084] It should also be noted that the undetermined residual contour is the basic analysis unit of this application. Unlike the determination object which is a single edge or a single pixel in a single frame image, the undetermined residual contour aggregates a cluster of continuous edge points that fall within a specific partition of the occlusion criterion set and show continuity between adjacent frames into a whole determination object. Subsequent attribution determination is no longer based on isolated edges, but is uniformly performed on the whole object. This upgrades the object of monitoring and determination from a single frame edge to a residual contour with temporal continuity, thereby enabling it to carry the comparison of motion parameters from dual paths.

[0085] S40: Based on the chassis displacement corresponding to each frame of the chassis displacement sampling, perform inverse rigid body transformation on the undetermined residual contour of each frame of the image to generate trajectory displacement parameters containing the trajectory continuation direction and trajectory continuation displacement amount in each frame of the image.

[0086] Specifically, step S40 includes: S400~S408.

[0087] S400: Compare the coordinates of each pixel in the undetermined residual contour of the t-th frame image with the camera intrinsic parameter matrix in the spatial calibration parameters. inverse matrix Multiply to obtain the three-dimensional observation ray vector; use the camera extrinsic transformation matrix in the empty calibration parameters. By rotating and translating the 3D observation ray vector, the global ray equation of the t-th frame image is obtained.

[0088] Specifically, the camera intrinsic parameter matrix Describes imaging parameters such as camera focal length and principal point, and the extrinsic transformation matrix from camera to chassis. This describes the rotation and translation relationship between the camera coordinate system and the moving chassis coordinate system. The global ray equation represents the spatial equation of a ray originating from the camera optical center and passing through a pixel in the undetermined residual contour in the moving chassis coordinate system.

[0089] For example, the expression for calculating the ray direction vector in the global ray equation is: , in, This refers to the ray direction vector in the moving chassis coordinate system. The extrinsic transformation matrix from camera to chassis The rotational component in Let be the homogeneous pixel coordinates of a certain pixel point in the undetermined residual contour.

[0090] S401: Extract the vertical axis translation component of the camera extrinsic transformation matrix to obtain the ground height reference plane, calculate the spatial geometric intersection point of the global ray equation of the ground height reference plane and the t-th frame image, and thus obtain the spatial geometric intersection point corresponding to each pixel point in the undetermined residual contour of the t-th frame image.

[0091] Specifically, the ground height reference plane is a horizontal plane that coincides with the driving ground of the mobile chassis, determined by the installation height of the camera relative to the ground. The spatial geometric intersection is the three-dimensional position of each edge point in the coordinate system of the mobile chassis when the undetermined residual profile is assumed to be on the ground.

[0092] It should be noted that, regarding the configuration assumptions upon which the ground height reference plane depends, configuration boundary processing rules are set: the spatial geometric intersection point is obtained based on the assumption that the undetermined residual profile is located on the ground where the mobile chassis is traveling. When the actual obstacle corresponding to the undetermined residual profile is significantly higher than the ground, causing a systematic deviation between the spatial geometric intersection point obtained based on the ground assumption and the actual position, this deviation is applied in the same direction to the chassis base point series of adjacent frames. This deviation is canceled out when the trajectory displacement parameters are subsequently calculated using the difference between adjacent frames, so it does not affect the determination of the trajectory continuity direction. If a more accurate height is required, triangulation can be used instead of the ground assumption to obtain the spatial geometric intersection point when multi-view observation is available.

[0093] S402: Based on the arrangement order of pixels in the undetermined residual contour, the spatial geometric intersection points are arrayed and stitched to obtain the chassis base point sequence of the t-th frame image. .

[0094] Specifically, the chassis foundation point sequence is a three-dimensional point sequence of the undetermined residual profile in the moving chassis coordinate system, and is the object of subsequent rigid body inverse transformation.

[0095] In addition, the chassis base point sequence transforms the undetermined residual contour from two-dimensional pixels in the image coordinate system to a three-dimensional point sequence in the moving chassis coordinate system. This transformation enables the subsequent rigid body inverse transformation of the undetermined residual contour based on the actual motion of the moving chassis. It is precisely because the chassis base point sequence has been unified to the moving chassis coordinate system that the chassis displacement sampling given by the chassis odometer can be directly applied to it, thereby restoring the positional changes at the image level to the real spatial motion relative to the moving chassis.

[0096] This application maps the undetermined residual contour from the image coordinate system to a series of chassis base points in the moving chassis coordinate system by back-projecting each pixel of the undetermined residual contour through the inverse matrix of the camera intrinsic parameters and finding its intersection with the ground height reference plane. Unlike the conventional processing method that only measures edge displacement in the image plane, this application establishes the spatial position of the residual contour at the unified three-dimensional coordinate system of the moving chassis. The resulting series of chassis base points is then used as the object of rigid body inverse transformation in the subsequent trajectory displacement parameter generation stage.

[0097] Furthermore, such as Figure 5 As shown, based on the chassis displacement sampling, a rigid body inverse transformation is performed on the determined residual profile to generate trajectory displacement parameters: S403: Based on the integral of the chassis linear velocity in the chassis displacement over the time interval between the t-th frame and the (t-1)-th frame, obtain the translation inverse compensation vector for the t-th frame. Based on the integral of the chassis angular velocity in the chassis displacement over the time interval between the t-th frame and the (t-1)-th frame, the inverse rotation transformation matrix of the t-th frame is obtained. .

[0098] Specifically, the translation inverse compensation vector and the rotation inverse transformation matrix represent the inverse transformations required to cancel out the translation and rotation of the moving chassis between adjacent frames.

[0099] S404: Inverse rotation transformation matrix based on the t-th frame image Translation inverse compensation vector Chassis base point sequence of the image in frame t-1 The rigid body compensation point sequence of the (t-1)th frame image is obtained. .

[0100] Specifically, the rigid body compensation point list represents the result of transforming the chassis base point list of the previous frame to the moving chassis coordinate system of the current frame, that is, the position where the contour should appear in the current frame if it is assumed to be a spatially stationary object.

[0101] It should be noted that the translation inverse compensation vector cannot be directly multiplied with the point sequence. If a non-homogeneous coordinate form is used, the rigid body compensation point sequence of the (t-1)th frame image... This involves first performing matrix multiplication on the chassis foundation point sequence using the inverse rotation transformation matrix, and then superimposing the inverse translation compensation vector. If 4... To achieve integrated rotation and translation operations using a homogeneous transformation matrix, it is necessary to first expand each point in the chassis foundation point sequence to homogeneous coordinates, and then combine the inverse rotation transformation matrix and the inverse translation compensation vector to form a 4-dimensional matrix. 4. Multiply the homogeneous transformation matrix with the chassis foundation point sequence after expanding homogeneous coordinates to obtain the rigid body compensation point sequence.

[0102] S405: Rigid body compensation point list for the (t-1)th frame image Chassis base point sequence with frame t Calculate the mean of the coordinates to obtain the preceding points of the (t-1)th frame image. and the points of the t-th frame image .

[0103] Specifically, the previous point represents the centroid position of the contour in the previous frame after rigid body compensation, and the current point represents the centroid position of the contour in the current frame.

[0104] S406: Points based on the image of frame t and the preceding points of the (t-1)th frame image The difference is used to obtain the difference calculation result of the image in frame t. .

[0105] Specifically, the difference calculation result is the residual spatial displacement vector of the undetermined residual profile relative to the moving chassis coordinate system between adjacent frames after deducting the motion of the moving chassis.

[0106] S407: Calculate the difference result of the t-th frame image. The projection angle on the horizontal bottom plane determines the trajectory continuation direction for generating the t-th frame image. Calculate the Euclidean distance modulus of the difference calculation result of the t-th frame image, and generate the trajectory continuation displacement of the t-th frame image. .

[0107] S408: Based on the trajectory continuation direction and trajectory continuation displacement of the t-th frame image, obtain the trajectory displacement parameters of the t-th frame image.

[0108] It should be noted that the generation of trajectory displacement parameters reuses the chassis base point sequence from the previous frame. Both the chassis base point sequence from the previous frame and the chassis base point sequence from the current frame are derived from the mapping results of the intersection of the undetermined residual profile with the ground height reference plane. It is precisely because the rigid body inverse transformation obtained by integrating the chassis linear velocity and chassis angular velocity cancels out the motion of the moving chassis that the difference calculation result only retains the real spatial motion of the undetermined residual profile relative to the moving chassis. This allows the trajectory displacement parameters to characterize the actual motion of the profile in external space and serve as the motion that should be satisfied under the assumption of real external obstacles in the subsequent comparison with the boundary motion parameters.

[0109] The core concept of this application's scheme for generating trajectory displacement parameters lies in applying a rigid body inverse transformation to the chassis base point sequence of the previous frame using the rotation inverse transformation matrix obtained by integrating the chassis linear velocity and chassis angular velocity, and the translation inverse compensation vector. Then, the residual motion of the undetermined residual contour relative to the moving chassis is extracted using the difference between the previous point and the current point after compensation. The difference between this scheme and existing technologies that only observe whether the edges are continuous in the image plane is that this scheme explicitly subtracts the motion contribution of the moving chassis, which can replace existing image plane optical flow continuity judgment technology and edge tracking technology that does not distinguish the source of motion.

[0110] S50: Calculate the outer edge coordinate difference based on the self-occlusion envelope of adjacent frame images to generate boundary motion parameters in each frame image, including the contour offset direction and contour offset displacement.

[0111] Further, step S50 includes: S500~S504.

[0112] S500: Self-occlusion envelope of the t-th frame image Self-occlusion envelope of the (t-1)th frame image Boundary contours are extracted separately to obtain the outer edge of the t-th frame image and the preceding outer edge of the (t-1)-th frame image.

[0113] Specifically, the current outer edge and the previous outer edge are the boundary pixel sequences of the self-occlusion envelope of the current frame and the self-occlusion envelope of the previous frame, respectively, which are the outlines of the occlusion area of ​​the robotic arm.

[0114] S501: Calculate the second derivative of each pixel on the outer edge of the t-th frame image to obtain the curvature extrema. Solve for the intersection of straight lines along the normal direction of each curvature extrema on the preceding outer edge of the (t-1)-th frame image to obtain the spatial corresponding point of each curvature extrema.

[0115] Specifically, the curvature extremum point is a feature point on the current outer edge where the curvature is locally maximum or minimum, and the spatial corresponding point is a point on the previous outer edge that corresponds to the curvature extremum point in the normal direction. The two constitute the point correspondence between the outer edges of the self-occlusion envelope of adjacent frames.

[0116] It should be noted that, regarding the configuration assumptions upon which solving for corresponding points in space along the normal direction depends, configuration boundary processing rules are set: assuming that the current outer edge is smooth near the curvature extremum and its normal direction can stably point to the previous outer edge, when the current outer edge has a sharp corner or break at a certain curvature extremum, making the normal direction non-unique, the corresponding point in space is determined on the previous outer edge in the neighborhood of that curvature extremum using a nearest-point search method, to ensure the computability of the outer edge coordinate difference.

[0117] S502: Obtain the image pixel coordinates of each curvature extremum point in the t-th frame image and the corresponding ground space mapping relationship of each curvature extremum point. Then, backproject each curvature extremum point and its corresponding spatial point to the ground space. Perform coordinate subtraction on the curvature extremum points and their corresponding spatial points backprojected to the ground space to obtain the physical space offset of the t-th frame image. .

[0118] It should be noted that when the number of curvature extrema is 1, the physical space offset of the t-th frame image can be obtained directly by performing coordinate subtraction based on the curvature extrema projected onto the ground space and their corresponding spatial points. When the number of curvature extrema is greater than 1, coordinate subtraction is performed on each curvature extrema and its corresponding spatial point back-projected onto the ground space, and the mean of the results is used as the physical space offset of the t-th frame image. .

[0119] For example, the expression for calculating the physical space offset vector is: , in, Let be the coordinates of the spatial point corresponding to the curvature extremum point projected onto the ground space. These are the coordinates of the curvature extrema points projected back onto the ground space.

[0120] S503: Extract the physical space offset of the t-th frame image. The direction angle is used to obtain the contour offset direction of the image in frame t. Calculate the Euclidean distance modulus of the physical space offset of the t-th frame image to obtain the contour offset displacement of the t-th frame image. .

[0121] S504: Based on the contour offset direction and contour offset displacement of the t-th frame image, obtain the boundary motion parameters of the t-th frame image.

[0122] Specifically, the core concept of the boundary motion parameter generation scheme in this application lies in obtaining the corresponding spatial point on the preceding outer edge using the normal direction of the current outer edge curvature extreme point, and then converting the outer edge coordinate difference into a physical space offset vector through spatial calibration parameters, thereby characterizing the motion of the self-occluding envelope outer edge between adjacent frames. The difference from the prior art is that it does not track the motion of its own occlusion boundary. This application quantifies the self-occlusion boundary motion discarded by the prior art into a physical quantity that can be compared with the obstacle motion, which can replace the existing technology that only deducts the occlusion area without utilizing its motion information.

[0123] This application establishes the point correspondence of the outer edge of the self-occlusion envelope of adjacent frames by using the normal direction of the curvature extreme point and converts it into a physical space offset vector through spatial calibration parameters. Unlike the conventional processing method that removes the self-occlusion area and then no longer uses its boundary motion, this application extracts parameters at the physical space motion level of the outer edge of the self-occlusion envelope. The resulting boundary motion parameters are used in the subsequent attribution determination stage as the other end to compare the direction parallelism and displacement proportionality with the trajectory displacement parameters.

[0124] This application obtains trajectory displacement parameters and boundary motion parameters by unfolding two independent motion parameter generation paths in parallel on the same undetermined residual contour. The first path performs an inverse rigid body transformation on the undetermined residual contour based on chassis displacement sampling to obtain trajectory displacement parameters. The physical meaning of this is: if the undetermined residual contour corresponds to a stationary real external obstacle in space, its image position change between adjacent frames should be entirely explained by the motion of the moving chassis. After inverse compensation based on chassis motion, the corresponding positions in adjacent frames should coincide, and the residual position difference reflects the true spatial motion of the contour relative to the moving chassis coordinate system. The second path extracts the outer edge coordinate difference of the self-occlusion envelope between adjacent frames to obtain boundary motion parameters. The physical meaning of this is: if the undetermined residual contour corresponds to a self-edge residue closely attached to the occlusion boundary, its image position change between adjacent frames should be consistent with the movement of the outer edge of the self-occlusion envelope. The specific solution steps and formulas for the two paths will be explained in the subsequent corresponding descriptions. Here, we only give the parallel relationship between the two in the overall process: the trajectory displacement parameters and the boundary motion parameters both take the same undetermined residual profile as input and output a direction quantity and a displacement quantity respectively. The two together serve as the comparison objects for the next step of the attribution determination.

[0125] It should be further explained that the trajectory displacement parameter describes the real spatial motion of the undetermined residual contour relative to the moving chassis, while the boundary motion parameter describes the image motion of the outer edge of the self-occlusion envelope. The two paths correspond to the real external obstacle hypothesis and the self-edge residue hypothesis, respectively. This application generates two sets of motion parameters in parallel instead of relying on only one set because the edges of real external obstacles and self-edge residues may both exhibit cross-frame continuity at the image level, and it is difficult to distinguish between the two based on a single path. Only by simultaneously obtaining the motion that the same object should follow under both hypotheses and comparing them can its true attribution be determined.

[0126] S60: Compare the trajectory displacement parameters and boundary motion parameters of each frame image. When the trajectory continuation direction is not parallel to the contour offset direction, or the trajectory continuation displacement and the contour offset displacement do not increase or decrease proportionally, obtain the real obstacle feature set based on the undetermined residual contour of that frame image.

[0127] When the trajectory continuation direction is not parallel to the contour offset direction, or when the trajectory continuation displacement and the contour offset displacement do not increase or decrease proportionally, the undetermined residual contour is written into the real obstacle feature set. It should be noted that the attribution determination process is as follows: the trajectory continuity direction in the trajectory displacement parameters is denoted as... The direction of the contour offset is denoted as The displacement of the trajectory continuity is denoted as The amount of contour offset displacement is denoted as First, compare the angle between the two directions. When the angle between the two directions is greater than the parallelism tolerance, the result is considered positive. At that time, it is determined that the trajectory continuation direction is not parallel to the contour offset direction; secondly, the normalized distance difference between the two displacements is compared. Since the self-edge residue moves synchronously with the outer edge of the self-occlusion envelope, its trajectory continuation displacement should be equal to the contour offset displacement. Therefore, this proportional benchmark is denoted as... and take When the normalized distance difference exceeds the proportional tolerance When the trajectory continuity displacement and the contour offset displacement do not increase or decrease proportionally, the undetermined residual contour is written into the real obstacle feature set as long as either of the above two conditions is met.

[0128] For example, the criterion expression for attribution determination is: , in, For the attribution determination result, For indicator functions; Indicates the lower limit of distance protection, used for avoidance. Division by zero exception when the value is 0; This represents the normalized distance difference.

[0129] The output of the criterion expression When the value is 1, it means that the current undetermined residual profile is written into the real obstacle feature set, which serves as the source of obstacle feature points in the subsequent safe connected component extraction.

[0130] Because the residual image motion of the self-occlusion boundary after inverse compensation by chassis motion should move along the direction of the outer edge of the self-occlusion envelope and change proportionally with the scaling of the outer edge, while the residual image motion direction and amplitude of the real external obstacle after inverse compensation by chassis motion are determined by the obstacle's own spatial position and have no intrinsic relationship with the movement of the outer edge of the self-occlusion envelope, non-parallelism or unequal increase / decrease in displacement is used as the criterion for real obstacles; directional parallelism tolerance. Used to absorb angle quantization errors and sensor noise, its value ranges from 3 to 10 degrees, with a typical value of 5 degrees; proportional tolerance. It is used to absorb displacement quantization error, and its value ranges from 0.1 to 0.3, with a typical value of 0.2.

[0131] In addition, take the proportional benchmark. The premise is the displacement of the trajectory continuity. With contour offset displacement Taken from corresponding adjacent locations on the residual edge of the autologous tissue: The residual displacement is the centroid of the residual profile after rigid body inverse compensation. The normal displacement of the self-occluding envelope outer edge, which closely follows the residual contour, at the curvature extremum point, corresponds approximately to the same local motion when the self-residue is close to the occlusion boundary; therefore, their displacements are approximately equal. When the vehicle-mounted robotic arm moves in a folding or extending manner, and the self-occluding envelope outer edge undergoes significant scaling or rotation between adjacent frames, the displacement amplitudes differ at points at different distances from the instantaneous center of motion. and It may deviate from strict equality, in which case This is only an approximate benchmark.

[0132] It should also be noted that there are critical handling issues regarding the ratio of the two displacements: when the contour offset displacement... When the ratio is close to zero, that is, when the outer edge of the self-occlusion envelope hardly moves between adjacent frames, the ratio is... It no longer possesses numerical stability; therefore, the proportionality condition is no longer used as the criterion, but rather the displacement of the trajectory is used directly. Is it greater than the minimum resolvable displacement? As a supplementary criterion: if If the undetermined residual contour has a real spatial motion that cannot be explained by the self-occluding boundary motion, it is added to the real obstacle feature set; if If the undetermined residual profile is not written into the real obstacle feature set in the current frame, it will be left for observation in subsequent frames. Minimum resolvable displacement The value ranges from 0.01 meters to 0.05 meters, with a typical value of 0.02 meters.

[0133] Regarding trajectory continuity displacement Critical handling approaching zero: When the residual spatial displacement of a stationary real external obstacle in space corresponding to the undetermined residual profile approaches zero after rigid body inverse compensation, the trajectory continuity displacement... The trajectory continuation direction extracted from the near-zero vector is close to zero. When the numerical values ​​are unstable, the direction condition is no longer used as the criterion; instead, the contour offset displacement is used. As an auxiliary judgment: if the contour offset displacement amount Greater than the minimum resolvable displacement And trajectory continuity displacement If the value is close to zero, it indicates that the contour does not move synchronously with the outer edge of the self-occluding envelope, and it is written into the set of real obstacle features; if the trajectory continues the displacement... With contour offset displacement If both values ​​are close to zero, then the contour shows no significant movement relative to either reference point between adjacent frames, and its classification is temporarily uncertain. Therefore, it will not be written into the real obstacle feature set in the current frame and will be observed in subsequent frames.

[0134] S70: Extract safe connected components from the chassis displacement, real obstacle feature set, and occlusion criterion set corresponding to each frame of image to generate the obstacle avoidance boundary of each frame of image.

[0135] Further, step S70 includes: S700~S703.

[0136] S700: Perform a double integral in the time dimension on the chassis linear velocity in the chassis displacement corresponding to the t-th frame image during chassis displacement sampling to obtain the cumulative displacement range of the moving chassis corresponding to the t-th frame image, thereby dividing the t-th frame image into a two-dimensional initial empty grid array with the grid as the smallest unit. .

[0137] Specifically, the two-dimensional initial empty grid array is a planar grid map that covers the reachable area of ​​the mobile chassis and initially marks all grids as empty.

[0138] S701: Map the real obstacle feature set of the image in frame t to a two-dimensional initial empty grid array through spatial calibration parameters to obtain the obstacle feature points of the image in frame t; map the core occlusion area and boundary transition area in the occlusion criterion set of the image in frame t to a two-dimensional initial empty grid array through spatial calibration parameters to obtain the visual occlusion points of the image in frame t.

[0139] Specifically, obstacle feature points are the positions of real external obstacles in the grid array, and visual occlusion points are the positions of image areas that are difficult to observe due to self-occlusion by the robotic arm in the grid array.

[0140] S702: Perform element traversal on the initial two-dimensional empty grid array of the t-th frame image, mark the grid containing obstacle feature points as entity collision occupied grid, and mark the grid containing visual occlusion points but not obstacle feature points as unobservable blind zone grid.

[0141] Specifically, the entity collision-occupied grid represents a grid where there is a real obstacle and the robot cannot pass through, while the unobservable blind zone grid represents a grid that is not observed due to self-occlusion and the robot should not pass through it.

[0142] It should be noted that marking the grid containing visual occlusion points but no obstacle feature points separately as unobservable blind zone grids is a reuse of the core occlusion area and the boundary transition area: the core occlusion area and the boundary transition area are derived from the intersection, union, and difference operations of the self-occlusion envelopes of adjacent frames, corresponding to areas in the image where it is difficult to confirm the existence of obstacles due to robotic arm occlusion. Because this part of the area is explicitly marked as an unobservable blind zone grid rather than an empty grid, this application avoids the robot mistakenly entering unobserved self-occlusion areas as passable empty areas, thereby improving the safety of passage in self-occlusion scenarios.

[0143] S703: Obtain the obstacle avoidance boundary of the t-th frame image by subtracting the grid cells occupied by entity collisions and the grid cells of unobservable blind zones from the two-dimensional initial empty grid array of the t-th frame image.

[0144] Specifically, the core concept of the obstacle avoidance image analysis result generation scheme in this application lies in constructing a two-dimensional initial empty grid array using the double integral of chassis displacement, and then mapping the real obstacle feature set and occlusion criterion set into entity collision occupied grid and unobservable blind zone grid respectively, and then subtracting them from the grid array. The difference between this application and the existing technology, which only marks the identified obstacles as impassable and treats self-occupied areas as passable, is that this application treats the unobserved areas caused by self-occlusion as an independent blind zone category for avoidance. This can replace the existing binary occupied grid mapping technology and the obstacle avoidance planning technology that does not distinguish between unobserved areas and empty areas.

[0145] This application maps the real obstacle feature set and occlusion criterion set into entity collision occupied grids and unobservable blind zone grids respectively, and subtracts them from the two-dimensional initial empty grid array. Unlike the conventional processing method that only subtracts the binary occupied grids of identified obstacles, this application distinguishes the grids at the level of three states: passable, impassable, and unobservable. The resulting obstacle avoidance image analysis results are used as the remaining empty areas to be screened in the subsequent connected component extraction and dilation safety processing stages.

[0146] Specifically, the expansion security processing and connected component filtering include steps 1 to 4.

[0147] Step 1: Extract the chassis outline dimension matrix from the spatial calibration parameters and calculate its minimum circumscribed circle radius. Combine this with a preset safety margin to obtain the physical safety radius. Convert the physical safety radius into the number of raster cells according to the raster map resolution, and generate a morphological expansion kernel based on this number of raster cells. Output morphological expansion kernel .

[0148] Specifically, the chassis outline dimension matrix describes the external dimensions of the mobile chassis projected onto the ground. The minimum circumscribed circle radius is the smallest circle radius that can enclose the shape. The physical safety radius is obtained by superimposing a preset safety margin on the minimum circumscribed circle radius. The grid map resolution characterizes the actual physical length corresponding to a single grid cell. The number of grid cells is obtained by dividing the physical safety radius by the grid map resolution and rounding up. The morphological expansion kernel is a structural element with this radius as its size.

[0149] For example, physical safety radius ,in, , These represent the length and width of the mobile chassis, respectively. Indicates the preset safety margin; number of grid cells. ,in, Indicates the resolution of the raster map; This represents the function for rounding up.

[0150] It should be noted that the size of the morphological expansion kernel is determined by the minimum circumscribed circle radius of the moving chassis outline size matrix. This is based on the fact that the moving chassis is regarded as a circle with a radius equal to the minimum circumscribed circle radius to simplify collision detection. This ensures that as long as the center of the moving chassis is within the remaining empty grid matrix, it will not collide with the grid occupied by the entity collision. The minimum circumscribed circle radius is taken as half the length of the diagonal of the moving chassis's planar projection. The typical value depends on the size of the chassis. For example, a chassis with a diagonal length of 0.8 meters corresponds to a minimum circumscribed circle radius of 0.4 meters.

[0151] Step 2: Perform morphological dilation filtering calculation on the collision-occupied grid of the entity based on the morphological dilation kernel, and output the dilated safety boundary grid.

[0152] Specifically, the expanding safety boundary grid is a grid area that expands outward from the collision-occupied grid of the moving chassis by the radius of the circumcircle of the moving chassis. It is used to ensure that the moving chassis, as a rigid body with volume, does not collide with real obstacles when it passes through.

[0153] Step 3: Initially empty two-dimensional grid array The expansion safety boundary grid and the unobservable blind zone grid are subjected to matrix subtraction operation to remove the expansion safety boundary grid and the unobservable blind zone grid from the initial two-dimensional empty grid array, and the remaining empty grid matrix is ​​output.

[0154] Specifically, the remaining empty grid matrix is ​​the set of grids that are still marked as empty after excluding the expanded obstacle safety boundary and the self-occlusion blind zone.

[0155] Step 4: Input the remaining empty raster matrix, perform connected component labeling on the raster cells with empty values ​​to obtain several connected grid sets, count the number of raster cells in each connected grid set, and output the connected grid set with the largest number of raster cells as the obstacle avoidance image analysis result.

[0156] It is worth noting that regarding the possible empty set situation in connected component filtering: when there are no empty grid cells in the remaining empty grid matrix, that is, when the current reachable range of the mobile chassis is completely covered by the expanded safety boundary of the grid occupied by the entity collision and the unobservable blind zone grid, the obstacle avoidance image analysis result output is empty. At this time, the robot stops moving forward and triggers in-situ observation or backward replanning. After obtaining a new view arm associated frame sequence and the unobservable blind zone grid shrinks due to the robot arm retraction, the safe connected component extraction is re-executed.

[0157] Through the above technical solution, this application performs morphological expansion on the collision-occupied grid of entities using the radius of the chassis's circumcircle as the expansion core, and takes the largest connected grid set after deducting the expansion safety boundary grid and the unobservable blind zone grid. This differs from the conventional approach of taking all empty areas in the grid map as feasible regions without connectivity screening. This application screens feasible regions by considering the expansion safety boundary of the chassis volume and grid connectivity, thereby outputting the connected grid set with the largest number of grids as the final obstacle avoidance image analysis result, which can be directly called by the robot motion planning stage.

[0158] In this application, chassis displacement sampling is used to determine the range of the grid array in space and the current position of the robot. The real obstacle feature set is used to mark the occupancy of impassable physical obstacles in the grid array. The occlusion criterion set is used to mark the blind spots that are difficult to observe due to self-occlusion in the grid array. Finally, the remaining connected region after deducting the physical obstacle occupancy and blind spots from the grid array is the obstacle avoidance image analysis result. Unlike existing technologies that estimate the image projection area of ​​the robotic arm and then subtract it as a one-time occlusion area, relying solely on the cross-frame continuity of edges in the subtracted image as the obstacle criterion, the core difference of this application lies not in the use of inputs such as robotic arm pose, chassis odometer, or camera calibration, but in how to organize the relationship between these features: first, the self-occlusion envelope is used as the geometric reference for cross-frame preservation; then, continuous edge points falling within a specific partition of the occlusion criterion set are clustered into the basic analysis unit of the undetermined residual contour; then, trajectory displacement parameters determined by chassis motion and boundary motion parameters determined by the outer edge of the self-occlusion envelope are applied to the same undetermined residual contour; finally, whether the directions of the two are parallel and whether the displacement increases or decreases proportionally are used as the attribution criteria. The effect of this organization is that it can distinguish between real obstacle edges that both appear as cross-frame continuity at the image level and self-edge residues. This effect is difficult to obtain by parameter tuning of a single continuity threshold or by simply combining multiple edge detection operators, because the latter always works under the assumption of a single motion source and cannot separate two simultaneously existing motion causes. Through the above technical solution, this application decouples self-occlusion residues from real external obstacles by using self-occlusion envelope as a cross-frame reference, applying dual-path motion parameters to the same undetermined residual contour, and using directional parallelism and displacement proportionality as criteria. This differs from the conventional processing method that only judges obstacles based on a single continuity after subtracting its own projection. The judgment is made at the motion source attribution level rather than the edge continuity level, thereby generating a reliable set of real obstacle features. This set of real obstacle features is used as the marker source of the entity collision-occupied grid in the subsequent safe connected domain extraction stage.

[0159] Based on the robot autonomous obstacle avoidance method based on dynamic image analysis provided in the above embodiments, this application also provides a robot autonomous obstacle avoidance system based on dynamic image analysis, such as... Figure 6 As shown, the system includes: The data acquisition and sampling timing binding module 10 is used to perform timing lag binding on visual image sampling, arm and claw pose sampling, chassis displacement sampling and spatial calibration parameters when the composite mobile robot is handling the fixture, so as to obtain the visual arm associated frame sequence.

[0160] The self-occlusion envelope acquisition module 20 is used to perform continuous frame union operation on the visual arm associated frame sequence based on arm pose sampling and spatial calibration parameters, and generate a self-occlusion envelope for each frame of the visual image sampling to characterize the area occupied by the vehicle-mounted robotic arm, gripper and fixture in that frame of the image.

[0161] The residual contour extraction module 30 is used to perform intersection, union and difference operations on the self-occlusion envelopes of adjacent frame images to generate an occlusion criterion set for each frame image, thereby extracting the exposed edge features of each frame image and generating the undetermined residual contour of each frame image.

[0162] The first motion parameter generation module 40 is used to perform rigid body inverse transformation on the undetermined residual contour of each frame image based on the chassis displacement corresponding to each frame image in the chassis displacement sampling, and generate trajectory displacement parameters in each frame image containing the trajectory continuation direction and trajectory continuation displacement amount.

[0163] The second motion parameter generation module 50 is used to calculate the outer edge coordinate difference based on the self-occlusion envelope of adjacent frame images, and generate boundary motion parameters in each frame image containing the contour offset direction and contour offset displacement.

[0164] The residual contour determination and attribution module 60 is used to compare the trajectory displacement parameters and boundary motion parameters of each frame image. When the trajectory continuation direction is not parallel to the contour offset direction, or the trajectory continuation displacement amount and the contour offset displacement amount do not increase or decrease proportionally, the real obstacle feature set is obtained based on the undetermined residual contour of the frame image.

[0165] The obstacle avoidance boundary generation module 70 is used to extract the safe connected components of the chassis displacement, real obstacle feature set and occlusion criterion set corresponding to each frame of image, and generate the obstacle avoidance boundary of each frame of image.

[0166] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0167] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0168] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0169] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0170] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A robot autonomous obstacle avoidance method based on dynamic image analysis, characterized in that, include: The visual image sampling, arm and gripper pose sampling, chassis displacement sampling and spatial calibration parameters of the composite mobile robot are time-lag bound to obtain the visual arm associated frame sequence. Based on arm and gripper pose sampling and spatial calibration parameters, continuous frame union operation is performed on the visual arm associated frame sequence to generate a self-occlusion envelope for each frame image in the visual image sampling, which is used to characterize the area occupied by the vehicle-mounted robotic arm, gripper and fixture in the frame image. The intersection, union, and difference operations are performed on the self-occlusion envelopes of adjacent frames to generate an occlusion criterion set for each frame, thereby extracting the exposed edge features of each frame and generating the undetermined residual contour of each frame. Based on the chassis displacement corresponding to each frame of the chassis displacement sampling, the undetermined residual contour of each frame of the image is subjected to rigid body inverse transformation to generate trajectory displacement parameters containing the trajectory continuation direction and trajectory continuation displacement amount in each frame of the image. The outer edge coordinate difference is calculated based on the self-occlusion envelope of adjacent frame images to generate boundary motion parameters in each frame image, which include the contour offset direction and the contour offset displacement. The trajectory displacement parameters and boundary motion parameters of each frame image are compared. When the trajectory continuation direction is not parallel to the contour offset direction, or the trajectory continuation displacement amount and the contour offset displacement amount do not increase or decrease proportionally, the real obstacle feature set is obtained based on the undetermined residual contour of that frame image. For each frame of image, safe connected components are extracted from the chassis displacement, real obstacle feature set, and occlusion criterion set to generate the obstacle avoidance boundary for each frame of image.

2. The robot autonomous obstacle avoidance method based on dynamic image analysis according to claim 1, characterized in that, Visual image sampling , This represents the image in frame t. This indicates the number of forward image frames included in a single continuous processing task when a composite mobile robot handles a jig. Arm and claw pose sampling , This represents the angle vector of each joint of the vehicle-mounted robotic arm at the acquisition time of the t-th frame image; Chassis displacement sampling , , This indicates the displacement of the moving chassis at the acquisition time of the t-th frame image. This represents the displacement of the chassis at the moment the t-th frame of the image was acquired. This represents the angle of rotation of the moving chassis at the moment the t-th frame of the image is acquired; Spatial calibration parameters It includes at least the camera intrinsic parameter matrix, the camera extrinsic parameter projection matrix, the base mounting bias matrix, and the extrinsic parameter transformation matrix from the camera to the chassis.

3. The robot autonomous obstacle avoidance method based on dynamic image analysis according to claim 2, characterized in that, The visual image sampling, arm pose sampling, chassis displacement sampling, and spatial calibration parameters of the composite mobile robot during fixture handling are time-lag bound, including: Using the exposure midpoint of the visual image sampling as the reference time axis, the arm and gripper pose sampling and chassis displacement sampling are time-shifted according to the nominal transmission lag. The translated arm and claw pose samples and chassis displacement samples are bound to each frame of the visual image samples using the nearest neighbor timestamp method.

4. The robot autonomous obstacle avoidance method based on dynamic image analysis according to claim 1, characterized in that, Perform a continuous frame union operation on the view arm-related frame sequence to generate a self-occlusion envelope for each frame in the visual image sampling, representing the area occupied by the vehicle-mounted robotic arm, gripper, and fixture in that frame image, including: Extract the base mounting offset matrix and camera extrinsic projection matrix from the spatial calibration parameters; The forward kinematics solution is performed on the chassis displacement corresponding to the t-th frame image in the chassis displacement sampling to obtain the three-dimensional spatial outer contour equation of the robotic arm. The three-dimensional spatial outer contour equation is transformed by multiplying it sequentially with the base mounting offset matrix and the camera extrinsic projection matrix to obtain the two-dimensional projected polygon of the t-th frame image in the image coordinate system. After rasterizing the two-dimensional projected polygons of the t-th frame image, the (t-1)-th frame image, and the (t-2)-th frame image to the same pixel grid, a Boolean union operation is performed to obtain the self-occlusion envelope of the t-th frame image.

5. The robot autonomous obstacle avoidance method based on dynamic image analysis according to claim 1, characterized in that, Perform intersection, union, and difference operations on the self-occlusion envelopes of adjacent frames to generate an occlusion criterion set for each frame, including: Construct a full-frame image mask to cover the entire image based on the maximum number of pixel rows and columns in the t-th frame; The core occlusion region of the image at frame t is obtained by performing an intersection operation on the overlapping region of the self-occlusion envelope of the image at frame t-1 and the self-occlusion envelope of the image at frame t-1. Perform a region union operation on the self-occlusion envelope of the t-th frame image and the self-occlusion envelope of the (t-1)-th frame image to obtain the merged polygon of the t-th frame image; subtract the core occlusion region of the t-th frame image from the merged polygon of the t-th frame image to obtain the boundary transition region of the t-th frame image. Perform a Boolean difference operation on the merged polygon of the full-frame image mask and the t-th frame image to obtain the outer adjacent region of the t-th frame image; The core occlusion region, boundary transition region, and external adjacent region of the t-th frame image are combined to form the occlusion criterion set of the t-th frame image.

6. The robot autonomous obstacle avoidance method based on dynamic image analysis according to claim 5, characterized in that, Extract the exposed edge features of each frame image to generate the undetermined residual contour of each frame image, including: Extract the pixel gradient of the t-th frame image and perform spatial clustering on the pixels whose pixel gradient is greater than a preset gradient threshold to obtain multiple continuous edge point sets of the t-th frame image; The coverage matrix of the image t is obtained by comparing each set of consecutive edge points in the image t with the occlusion criterion set of the image t, and obtaining the proportion of pixels in each set of consecutive edge points that fall into the core occlusion area, the boundary transition area, and the external adjacent area to the total number of pixels in that set. Each row of the coverage matrix corresponds to a set of consecutive edge points, and the three columns correspond to the proportion of pixels in the set of consecutive edge points that fall into the core occlusion area, the boundary transition area, and the external adjacent area, respectively. Based on the coverage matrix of the t-th frame image, the set of continuous edge points that fall into the core occlusion area with a pixel ratio greater than or equal to a preset removal threshold is removed to obtain the set of remaining continuous edge points of the t-th frame image. Calculate the Hausdorff matrix of the remaining continuous edge point set of the t-th frame image and the remaining continuous edge point set of the (t-1)-th frame image to obtain the feature distance of the t-th frame image; Based on the chassis displacement corresponding to the t-th frame image and the (t-1)-th frame image in the chassis displacement sampling, the inter-frame translation increment and inter-frame rotation increment of the t-th frame image are calculated. Based on the spatial calibration projection matrix, the inter-frame translation increment and inter-frame rotation increment are mapped to the image pixel coordinate system. Thus, the motion reprojection calculation is performed on the remaining continuous edge points in the set of remaining continuous edge points to obtain the theoretical inter-frame pixel deviation of the t-th frame image caused by the rigid body motion of the chassis. The inter-frame pixel deviation of the t-th frame image is obtained by superimposing a preset pixel noise margin on the theoretical inter-frame pixel deviation of the t-th frame image. The feature distance of the t-th frame image is compared with the inter-frame pixel deviation of the t-th frame image. When the feature distance is less than the inter-frame pixel deviation, the remaining continuous edge point set of the t-th frame image is used as the residual contour. When the feature distance is greater than or equal to the inter-frame pixel deviation, the undetermined residual contour of the t-th frame image is not output.

7. The robot autonomous obstacle avoidance method based on dynamic image analysis according to claim 1, characterized in that, Based on the chassis displacement corresponding to each frame of the chassis displacement sampling, an inverse rigid body transformation is performed on the undetermined residual contour of each frame to generate trajectory displacement parameters for each frame, including the trajectory continuity direction and the amount of trajectory continuity displacement. Multiply each pixel coordinate point in the undetermined residual contour of the t-th frame image by the inverse matrix of the camera intrinsic matrix in the spatial calibration parameters to obtain the three-dimensional observation ray vector; rotate and translate the three-dimensional observation ray vector using the camera extrinsic transformation matrix in the spatial calibration parameters to obtain the global ray equation of the t-th frame image. The vertical axis translation component of the camera extrinsic transformation matrix is ​​extracted to obtain the ground height reference plane. The spatial geometric intersection of the ground height reference plane and the global ray equation of the t-th frame image is calculated, thereby obtaining the spatial geometric intersection of each pixel in the undetermined residual contour of the t-th frame image. Based on the arrangement order of pixels in the undetermined residual contour, the spatial geometric intersection points are arrayed and stitched to obtain the chassis base point sequence of the t-th frame image; Based on the integral of the chassis linear velocity in the chassis displacement over the time interval between the t-th frame image and the (t-1)-th frame image, the translation inverse compensation vector of the t-th frame image is obtained; based on the integral of the chassis angular velocity in the chassis displacement over the time interval between the t-th frame image and the (t-1)-th frame image, the rotation inverse transformation matrix of the t-th frame image is obtained. Based on the rotation inverse transformation matrix, translation inverse compensation vector, and chassis base point sequence of the t-1 frame image, the rigid body compensation point sequence of the t-1 frame image is obtained. Calculate the mean coordinates of the rigid body compensation point sequence of the (t-1)th frame image and the chassis foundation point sequence of the tth frame image to obtain the preceding points of the (t-1)th frame image and the points of the tth frame image. The difference calculation result of the t-th frame image is obtained based on the difference between the points of the t-th frame image and the preceding points of the (t-1)-th frame image. Calculate the projection angle of the difference calculation result of the t-th frame image onto the horizontal bottom plane, and generate the trajectory continuation direction of the t-th frame image; calculate the Euclidean distance modulus of the difference calculation result of the t-th frame image, and generate the trajectory continuation displacement of the t-th frame image. Based on the trajectory continuation direction and trajectory continuation displacement of the t-th frame image, the trajectory displacement parameters of the t-th frame image are obtained.

8. The robot autonomous obstacle avoidance method based on dynamic image analysis according to claim 1, characterized in that, The outer edge coordinate difference is calculated based on the self-occlusion envelope of adjacent frames, generating boundary motion parameters for each frame that include the contour offset direction and the contour offset displacement, including: Boundary contours are extracted from the self-occlusion envelope of the t-th frame image and the self-occlusion envelope of the (t-1)-th frame image to obtain the outer edge of the t-th frame image and the preceding outer edge of the (t-1)-th frame image. The second derivative of each pixel on the outer edge of the t-th frame image is calculated to obtain the curvature extremum points. The intersection of straight lines along the normal direction of each curvature extremum point on the preceding outer edge of the (t-1)-th frame image is solved to obtain the spatial corresponding points of each curvature extremum point. Obtain the image pixel coordinates of each curvature extremum point in the t-th frame image and the ground space mapping relationship corresponding to each curvature extremum point. Then, back-project each curvature extremum point and its corresponding spatial point to the ground space. Perform coordinate subtraction on the curvature extremum points back-projected to the ground space and their corresponding spatial points to obtain the physical space offset of the t-th frame image. Extract the direction angle of the physical space offset of the t-th frame image to obtain the contour offset direction of the t-th frame image; calculate the Euclidean distance modulus of the physical space offset of the t-th frame image to obtain the contour offset displacement of the t-th frame image. Based on the contour offset direction and contour offset displacement of the t-th frame image, the boundary motion parameters of the t-th frame image are obtained.

9. The robot autonomous obstacle avoidance method based on dynamic image analysis according to claim 1, characterized in that, For each frame of image, safe connected components are extracted from the chassis displacement, real obstacle feature set, and occlusion criterion set to generate the obstacle avoidance boundary for each frame, including: By performing a double integral in the time dimension on the chassis linear velocity in the chassis displacement corresponding to the t-th frame image in the chassis displacement sampling, the cumulative displacement range of the moving chassis corresponding to the t-th frame image is obtained, thereby dividing the t-th frame image into a two-dimensional initial empty grid array with the grid as the smallest unit. The real obstacle feature set of the t-th frame image is mapped to a two-dimensional initial empty grid array through spatial calibration parameters to obtain the obstacle feature points of the t-th frame image; the core occlusion region and boundary transition region in the occlusion criterion set of the t-th frame image are mapped to a two-dimensional initial empty grid array through spatial calibration parameters to obtain the visual occlusion points of the t-th frame image. The elements of the initial two-dimensional empty grid array of the t-th frame image are traversed. Grids containing obstacle feature points are marked as grids occupied by entity collisions, and grids containing visual occlusion points but not obstacle feature points are marked as unobservable blind zone grids. The obstacle avoidance boundary of the t-th frame image is obtained by subtracting the grid cells occupied by entity collisions and the grid cells of unobservable blind zones from the two-dimensional initial empty grid array of the t-th frame image.

10. A robot autonomous obstacle avoidance system based on dynamic image analysis, characterized in that, include: The data acquisition and sampling timing binding module is used to perform timing lag binding on visual image sampling, arm and gripper pose sampling, chassis displacement sampling and spatial calibration parameters when the composite mobile robot is handling the jig, so as to obtain the visual arm associated frame sequence. The self-occlusion envelope acquisition module is used to perform continuous frame union operation on the visual arm associated frame sequence based on arm pose sampling and spatial calibration parameters, and generate a self-occlusion envelope for each frame image in the visual image sampling to characterize the area occupied by the vehicle-mounted robotic arm, gripper and fixture in the frame image. The residual contour extraction module is used to perform intersection, union and difference operations on the self-occlusion envelopes of adjacent frame images to generate an occlusion criterion set for each frame image, thereby extracting the exposed edge features of each frame image and generating the undetermined residual contour of each frame image. The first motion parameter generation module is used to perform rigid body inverse transformation on the undetermined residual contour of each frame image based on the chassis displacement corresponding to each frame image in the chassis displacement sampling, and generate trajectory displacement parameters in each frame image containing the trajectory continuation direction and trajectory continuation displacement amount. The second motion parameter generation module is used to calculate the outer edge coordinate difference based on the self-occlusion envelope of adjacent frame images, and generate boundary motion parameters in each frame image containing the contour offset direction and contour offset displacement. The residual contour determination and attribution module is used to compare the trajectory displacement parameters and boundary motion parameters of each frame image. When the trajectory continuation direction is not parallel to the contour offset direction, or the trajectory continuation displacement amount and the contour offset displacement amount do not increase or decrease proportionally, the real obstacle feature set is obtained based on the undetermined residual contour of the frame image. The obstacle avoidance boundary generation module is used to extract safe connected components from the chassis displacement, real obstacle feature set and occlusion criterion set corresponding to each frame of image, and generate the obstacle avoidance boundary of each frame of image.

Images