Method for correcting deviation of tightening trajectory, computer storage medium and terminal device

Abstract

The application discloses a tightening trajectory deviation correction method, a computer storage medium and a terminal device, relates to the technical field of image recognition, and the application acquires a template feature set of a feature to be tightened in a standard device first, then performs two-dimensional image positioning on a region of interest on a current operation device in sequence, then intercepts a three-dimensional point cloud based on the region of interest and fits out a measured feature set, then matches and aligns the template feature set and the measured feature set, calculates a space conversion matrix corresponding to the template feature set and the measured feature set, finally generates a deviation correction amount according to the space conversion matrix and outputs the deviation correction amount to a robot, so that the tightening trajectory of the robot is corrected, dynamic self-adaptive correction of the tightening trajectory is realized, and the deviation correction efficiency of the tightening trajectory is improved.

Description

Technical Field

[0001] This application relates to the field of image recognition technology, and in particular to a method for correcting a tightening trajectory, a computer storage medium, and a terminal device. Background Technology

[0002] Currently, automated tightening solutions involve robots carrying tightening guns that operate along pre-set hard trajectories. Typically, a robot carries the tightening gun and its studs to a designated location on the vehicle body to screw the studs into the bolt holes. This method relies on the stopping accuracy of the conveyor line and the repeatability of the tooling fixtures; the robot's movement trajectory is fixed once taught.

[0003] However, due to wear and tear on the fixtures and fluctuations in the incoming car body material, there is an uncontrollable random deviation between the actual position of the studs or screw holes on the car body and the theoretical position during teaching each time it stops at the workstation. This deviation causes the axis of the tightening gun to be unable to precisely align with the axis of the studs or screw holes when the robot moves along a fixed trajectory, resulting in insufficient tightening torque and damage to the screw threads. At this point, manual inspection and correction of the stud or screw hole position are required. In actual operation, due to the uncertainty of trajectory deviation, a large amount of manpower is needed to check each one individually, resulting in low efficiency of the relevant solution in correcting the tightening trajectory. Summary of the Invention

[0004] To address the aforementioned technical problems, this application provides a method for correcting tightening trajectories, a computer storage medium, and a terminal device, comprising: Obtain the three-dimensional pose information of the feature to be tightened in the standard equipment, and save the three-dimensional pose information as a template feature set; Acquire a two-dimensional image of the current operating equipment, and determine the region of interest corresponding to the feature to be tightened in the two-dimensional image through a preset matching mode; Based on the region of interest and the three-dimensional point cloud corresponding to the two-dimensional image, the target three-dimensional information corresponding to the feature to be tightened is extracted from the three-dimensional point cloud, and a measured feature set is generated. The template feature set is matched and aligned with the measured feature set, and the spatial transformation matrix corresponding to the template feature set and the measured feature set is calculated. A correction value is generated based on the spatial transformation matrix, and the correction value is sent to the robot so that the robot can correct the tightening trajectory according to the correction value.

[0005] In one embodiment, the step of extracting the target 3D information corresponding to the feature to be tightened from the 3D point cloud based on the region of interest and the 3D point cloud corresponding to the 2D image, and generating a measured feature set, includes: Obtain a three-dimensional point cloud corresponding to the two-dimensional image, and extract a local point cloud corresponding to the feature to be tightened from the three-dimensional point cloud according to the region of interest; The local point cloud is preprocessed to generate a point cloud to be processed, and the three-dimensional coordinates of the center point of the feature to be tightened are determined in the point cloud to be processed by geometric fitting. The three-dimensional coordinates are used as the target three-dimensional information corresponding to the feature to be tightened, and a measured feature set is generated.

[0006] In one embodiment, the step of matching and aligning the template feature set with the measured feature set, and calculating the spatial transformation matrix corresponding to the template feature set and the measured feature set, includes: The template feature set is matched and aligned with the measured feature set; Calculate the two-dimensional centroid of the aligned template feature set and the measured feature set on the working plane, as well as their average height in the direction perpendicular to the working plane; A two-dimensional covariance matrix is ​​constructed based on the two-dimensional centroid, and singular value decomposition is performed on the two-dimensional covariance matrix to calculate the rotation matrix in the working plane. The rotation matrix is ​​embedded into the corresponding position of the three-dimensional spatial rotation matrix, and the difference in the average height in the vertical direction is used as the vertical translation amount to generate the spatial transformation matrix.

[0007] In one embodiment, the step of matching and aligning the template feature set with the measured feature set includes: Obtain the pre-stored template feature set, which contains the center point coordinates of at least two of the features to be tightened in the standard device; The center points of the measured feature set are matched with the center points of the template feature set in sequence, and the positional deviation between each center point is calculated. By removing outliers whose positional deviations exceed a preset deviation range, the aligned template feature set and the measured feature set are obtained.

[0008] In one embodiment, the step of determining the region of interest corresponding to the feature to be tightened in the two-dimensional image by means of a preset matching mode further includes: The region of interest in the two-dimensional image is determined by the preset matching mode, wherein the preset matching mode includes a first matching mode and a second matching mode; When the preset matching mode is detected to be the first matching mode, a preset template image is called through the template matching algorithm to search for a target region in the two-dimensional image that meets the preset conditions, and the target region is determined as the region of interest; Otherwise, the two-dimensional image is detected by a deep neural network in the second matching mode, and the output feature region is determined as the region of interest.

[0009] In one embodiment, after the step of detecting the two-dimensional image through a deep neural network in the second matching mode and determining the output feature region as the region of interest, the method further includes: Determine the confidence level of the deep neural network output; When the confidence level is lower than a preset threshold, the system switches to the first matching mode to identify and output the region of interest. Obtain the region of interest output by the first matching pattern; The region of interest is cropped and labeled, and a training dataset is generated based on the labeled information, so as to update the network parameters of the deep neural network corresponding to the second matching pattern through the training dataset.

[0010] In one embodiment, the step of calling a preset template image using a template matching algorithm, searching for a target region in the two-dimensional image whose matching degree meets a preset condition, and determining the target region as the region of interest further includes: A template library is generated by extracting a first image of the features to be tightened from a standard device as an initial template and combining it with a second image acquired under at least one angle condition and / or at least one lighting condition. Using a template matching algorithm based on edge gradient and / or grayscale gradient, a matching calculation is performed between the template image in the template library and the two-dimensional image to determine the target region in the two-dimensional image whose matching degree meets the preset conditions, and the target region is used as the output region of interest.

[0011] In one embodiment, the step of generating a correction value based on the spatial transformation matrix and sending the correction value to the robot so that the robot corrects the tightening trajectory based on the correction value further includes: The spatial transformation matrix is ​​decomposed into translation along the three coordinate axes and rotation perpendicular to the coordinate axes of the working plane; The translation and rotation amounts are combined to form the correction amount, and a pose offset command is generated based on the correction amount. The pose offset command is sent to the robot to control the robot to correct the tightening trajectory and perform the tightening action through the tightening tool.

[0012] In addition, to achieve the above objectives, this application also provides a computer storage medium storing executable program code; the executable program code is used to execute the tightening trajectory correction method as described above.

[0013] In addition, to achieve the above objectives, this application also provides a terminal device, including a memory and a processor; the memory stores program code that can be executed by the processor; the program code is used to execute the tightening trajectory correction method as described above.

[0014] This application provides a method for correcting tightening trajectories, a computer storage medium, and a terminal device. The method involves first acquiring a template feature set of the features to be tightened from a standard device, then sequentially performing two-dimensional image localization of the region of interest on the current working device, followed by extracting a three-dimensional point cloud based on the region of interest and fitting a measured feature set. Next, the template feature set and the measured feature set are matched and aligned, and the spatial transformation matrix corresponding to the template feature set and the measured feature set is calculated. Finally, a correction amount is generated based on the spatial transformation matrix and output to the robot to correct the robot's tightening trajectory. This achieves dynamic adaptive correction of the tightening trajectory, thereby improving the efficiency of tightening trajectory correction. Attached Figure Description

[0015] Figure 1 This is a flowchart illustrating an embodiment of the tightening trajectory correction method of this application; Figure 2 This is a schematic diagram of a two-dimensional image involved in an embodiment of this application; Figure 3 This is a schematic diagram of the region of interest involved in the embodiments of this application; Figure 4 This is a schematic diagram of the depth map involved in the solution of this embodiment; Figure 5 This is a schematic diagram of the point cloud of the ROI region involved in the embodiments of this application; Figure 6 This is a schematic diagram of the feature point matching results involved in this implementation plan; Figure 7 This is a schematic diagram of the hardware operating environment of the terminal device involved in the embodiments of this application. Detailed Implementation

[0016] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0017] It should be noted that if the embodiments of this application involve directional indicators, such as up, down, left, right, front, back, etc., these directional indicators are only used to explain the relative positional relationship and movement of the components in a specific posture. If the specific posture changes, the directional indicators will also change accordingly. Furthermore, if the embodiments of this application involve descriptions such as "first," "second," "S1," "S2," "step one," "step two," etc., these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance, or implicitly indicating the number of technical features indicated or the order of method execution. Those skilled in the art will understand that anything that does not violate the inventive concept should be included within the scope of protection of this application.

[0018] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the tightening trajectory correction method of this application, as shown below. Figure 1 As shown, this application provides a method for correcting the tightening trajectory, including: Step S10: Obtain the three-dimensional pose information of the feature to be tightened in the standard equipment, and save the three-dimensional pose information as a template feature set.

[0019] In this embodiment, the standard equipment refers to a reference vehicle whose initial equipment dimensions, studs, or screw hole positions meet the theoretical values ​​in the design drawings. This is typically a vehicle that has passed three-coordinate measurement during the production line debugging phase. It should be understood that the standard equipment can also be any device with studs or screw holes that need tightening, or any device with other features to be tightened. The feature to be tightened refers to the target object that the robot needs to operate with a tightening gun, including but not limited to the center of the stud surface or the center point of the screw hole. Three-dimensional pose information refers to the three-dimensional coordinates of this feature in space, including the X-coordinate along the front-rear direction of the vehicle body, the Y-coordinate along the left-right direction, and the Z-coordinate along the up-down direction, expressed as three-dimensional rectangular coordinate values. The template feature set refers to a data set that organizes the three-dimensional pose information of multiple features to be tightened on the standard equipment in a predetermined order. For example, the center point coordinates of the upper and lower studs on the left side door are stored sequentially in the template feature set.

[0020] Specifically, the processing actions are executed by a correction control system, which communicates with the robot. The correction control system first reads a pre-stored standard equipment digital model file, or controls the robot to scan and photograph the standard equipment using a 3D camera. Then, it directly extracts the theoretical coordinates of the feature points to be tightened in the design coordinate system from the digital model file, or acquires the 3D point cloud of the standard equipment through the camera, performs circle fitting on the studs or screw holes in the point cloud, and calculates the center coordinates of the circle as 3D pose information. The extracted 3D coordinates of multiple feature points are arranged in a fixed logical order to form an ordered template feature set, which is then stored in a database as a reference for subsequent real-time correction.

[0021] Optionally, before acquiring the 3D pose information of the feature to be tightened in the standard equipment and saving the 3D pose information as a template feature set, it is also necessary to perform hand-eye calibration to obtain the transformation matrix between the camera and the robot, and establish the user's vehicle body coordinate system through CAD modeling or actual measurement. The cascaded transformation matrix of camera coordinate system → flange coordinate system → robot base coordinate system is obtained through hand-eye calibration; all feature points are ultimately mapped to the robot base coordinate system.

[0022] As an alternative implementation method for obtaining the transformation matrix between the camera and the robot in hand-eye calibration, the robot is first controlled to move the camera above a preset calibration board, ensuring the calibration board is fully visible within the camera's field of view. The robot is then controlled to change multiple poses, with the camera acquiring an image of the calibration board in each pose, while simultaneously recording the pose data of the robot's end effector flange in the robot's base coordinate system. For each acquired pose, image processing algorithms are used to extract the coordinates of feature points on the calibration board in the camera coordinate system. Simultaneously, based on the actual physical dimensions and known layout of the calibration board, the coordinates of these feature points in the calibration board's own coordinate system are calculated. The transformation relationship between the calibration board's coordinate system and the robot's base coordinate system is considered unknown, but the flange pose and calibration board pose in each pose can be correlated using the robot's kinematic model. Then, a nonlinear equation system is constructed, based on the calibration method or the classic AX=XB hand-eye calibration model (where A represents the robot's end effector motion and B represents the camera motion), to solve for the rotation matrix and translation vector of the camera coordinate system relative to the robot's tool coordinate system. After the solution is obtained, the transformation matrix is ​​stored in memory for use in subsequent real-time correction. In actual operation, after the camera detects the coordinates of the stator in the camera coordinate system, it is transformed to the robot tool coordinate system using a preset formula. Then, combined with the robot's current pose, it can be transformed to the robot base coordinate system or the vehicle body coordinate system.

[0023] Furthermore, the process of creating a digital model using CAD includes: First, reading the CAD model file of the vehicle body and parsing the predefined user coordinate system definition. Typically, the CAD model contains a reference system named "Body_Coordinate_System," from which the origin coordinates and the direction vectors of the three coordinate axes are extracted. Simultaneously, the theoretical coordinate values ​​of the center points of all studs or screw holes to be tightened in the model are extracted in the user coordinate system, and these values ​​are directly used as the initial data for the template feature set. Finally, the actual positions of the fixed reference points on the vehicle body are measured on-site using a robot, and the transformation matrix from the user coordinate system to the robot's base coordinate system is calculated.

[0024] The process of establishing a system based on actual measurements involves the following steps: When the CAD model is unavailable or requires on-site correction, first select at least three non-collinear datum features on the vehicle body, such as the two main locating pin holes at the front of the vehicle and an auxiliary datum surface at the center of the rear axle. Control the robot, equipped with a 3D camera or contact probe, to measure the actual spatial coordinates of these datum features in the robot's base coordinate system. Obtain the theoretical coordinates of these datum features in the user coordinate system. Using a three-point or multi-point fitting algorithm (such as least squares or SVD registration), calculate the optimal spatial transformation matrix to transform the theoretical coordinates to the measured coordinates. This matrix is ​​the transformation matrix from the user coordinate system to the robot's base coordinate system.

[0025] Step S20: Acquire a two-dimensional image of the current working equipment, and determine the region of interest corresponding to the feature to be tightened in the two-dimensional image through a preset matching mode.

[0026] In this embodiment, a camera module deployed on the robot can acquire two-dimensional images of the current working equipment. The camera module is a three-dimensional camera installed at the robot's end effector or a fixed position. This camera can simultaneously output two-dimensional grayscale or color images and a three-dimensional point cloud corresponding to each pixel of the two-dimensional image. The preset matching mode refers to a pre-configured two-dimensional image recognition strategy, including a first matching mode (a traditional algorithm based on template matching) and a second matching mode (a learning algorithm based on a deep neural network). The Region of Interest (ROI) refers to a specific area in an image that needs to be analyzed, processed, or extracted. It refers to the smallest rectangular or circular area occupied by the feature to be tightened in the two-dimensional image. The extent of this area is represented by pixel coordinates; for example, a rectangular area is determined by the pixel coordinates of the upper left and lower right corners.

[0027] Specifically, the correction control system first sends a trigger signal to the camera module, which then takes a picture of the currently operating equipment parked at the workstation, acquiring a two-dimensional image containing the features to be tightened, and sends it back to the correction control system. Then, based on the system's preset operating mode, it selects to enter either the first matching mode or the second matching mode. If entering the first matching mode, a pre-stored standard template image is retrieved, and a template matching algorithm based on edge gradients or grayscale gradients is used to slide and search the entire two-dimensional image, calculating the similarity between each sliding window and the template, and determining the location of the window with the highest similarity as the region of interest. If entering the second matching mode, the two-dimensional image is input into a pre-trained lightweight object detection network, which outputs the position and category of a predicted bounding box, which is then determined as the region of interest. When outputting the region of interest, the boundary coordinates of that region in the pixel coordinate system are also recorded.

[0028] For example, such as Figure 2 As shown, Figure 2 This is a schematic diagram of a two-dimensional image involved in an embodiment of this application. The two-dimensional image of the current operating equipment is acquired by the camera module, resulting in the image shown below. Figure 2 The two-dimensional image shown. The region of interest is then determined using a preset matching mode, resulting in the image shown. Figure 3 The region of interest shown Figure 3 This is a schematic diagram of the region of interest (ROI) involved in the embodiment of this application. Finally, the ROI is passed to the 3D detection region.

[0029] Optionally, in this embodiment, the step of determining the region of interest corresponding to the feature to be tightened in the two-dimensional image by means of a preset matching mode includes: Step S21: Determine the region of interest in the two-dimensional image through a preset matching mode, wherein the preset matching mode includes a first matching mode and a second matching mode; Step S22: When the preset matching mode is detected to be the first matching mode, the preset template image is called through the template matching algorithm to search for the target region in the two-dimensional image that meets the preset conditions, and the target region is determined as the region of interest; Step S23: Otherwise, detect the two-dimensional image through the deep neural network in the second matching mode, and determine the output feature region as the region of interest.

[0030] Specifically, first, a preset template image is loaded from memory. This image is typically a grayscale image with a size of tens to hundreds of pixels square. Then, the two-dimensional image of the current operating device, obtained in step S20, is used. The entire two-dimensional image is traversed, and a sliding window is created from top to bottom and left to right according to the size of the template image. At each sliding position, the pixel matrix within the window is extracted and its similarity is calculated with the pixel matrix of the template image. For example, a normalized cross-correlation algorithm is used to calculate the dot product of the two matrices divided by the product of their respective moduli, yielding a correlation coefficient. After calculations are completed for all sliding positions, the maximum value of the correlation coefficient and its corresponding window position are identified. This maximum value is compared with a preset threshold. If the maximum value is greater than or equal to the threshold, the coordinate range of the window is identified as the region of interest; if the maximum value is less than the threshold, the matching is considered a failure, and a retry strategy (such as adjusting image preprocessing parameters and re-matching) or an exception is reported. Finally, when outputting the region of interest, the center coordinates and boundaries of the region are recorded simultaneously.

[0031] For example, the first matching mode can be set to mode A, and the second matching mode can be set to mode B. Mode A is guided recognition based on fast template matching: This method utilizes OpenCV's high-performance template matching algorithm, based on edge or grayscale gradient matching. In the initial production phase, only the stud / hole area of ​​a standard vehicle needs to be manually or automatically extracted as the initial template. During the production verification phase (e.g., the first 5-10 vehicles), feature maps under different angles and lighting conditions are dynamically added to the template library, improving the initial recognition success rate through parallel matching of multiple templates. This addresses the pain points of deep learning in the early stages of projects, such as lack of on-site labeled data, long model training cycles, and the inability to achieve "plug and play." Furthermore, in practical applications, this method can achieve a recognition success rate of over 99% with only about 10 template images.

[0032] Pattern B is based on deep neural network recognition: Lightweight object detection networks, such as the YOLO series optimized for industrial CPUs, are employed. Offline training is performed using image data collected from actual production environments (including scenarios with varying levels of reflection, oil contamination, and occlusion). The algorithm leverages the high geometric similarity between studs and bolt holes on different vehicle models to construct a pre-trained model, enabling rapid transfer across projects. This addresses the issue of traditional vision algorithms becoming inaccurate under drastic lighting changes, dirty workpiece surfaces, or significant pose shifts.

[0033] Furthermore, the first matching mode employs a template matching algorithm, which can be quickly deployed without training and is suitable for early-stage projects or scenarios with stable lighting. The second matching mode uses a deep neural network, which can adapt to complex environments such as changes in lighting and oil smudges, exhibiting stronger robustness. The control system selects which mode to execute based on configuration and real-time feedback, achieving a flexible balance between "rapid deployment" and "long-term robustness." Combined with this embodiment, the solution can adaptively switch recognition strategies under different production stages and operating conditions, effectively avoiding the failure risk of a single recognition method in specific scenarios, significantly improving the success rate and stability of 2D positioning, thus providing reliable input for subsequent 3D point cloud local extraction and correction calculations.

[0034] Furthermore, in this embodiment, after the step of detecting the two-dimensional image through the deep neural network in the second matching mode and determining the output feature region as the region of interest, the method further includes: Determine the confidence level of the deep neural network output; when the confidence level is lower than a preset threshold, switch to the first matching mode for recognition and output the region of interest; obtain the region of interest output by the first matching mode; crop the region of interest and generate annotation information, and generate a training dataset based on the annotation information, so as to update the network parameters of the deep neural network corresponding to the second matching mode through the training dataset.

[0035] Specifically, the system prioritizes running Mode B; if the confidence level of Mode B falls below a threshold, it automatically switches to Mode A for assisted localization. In the initial stages of system operation, the system automatically crops and labels samples in the current environment using the successfully matched ROI regions and their precise coordinates from Mode A, serving as the training dataset for the Mode B deep learning model. This training dataset is then used to update the network parameters of the deep neural network corresponding to Mode B.

[0036] Step S30: Based on the region of interest and the corresponding 3D point cloud of the 2D image, extract the target 3D information corresponding to the feature to be tightened from the 3D point cloud and generate the measured feature set.

[0037] In this embodiment, a 3D point cloud refers to a set of 3D spatial data aligned with the coordinates of a 2D image. Each point in the point cloud contains X, Y, and Z coordinate values, where X and Y represent the horizontal position in space, and Z represents the depth value. Target 3D information refers to the 3D coordinates of the center point of the feature to be tightened in the current working equipment within the camera coordinate system. The measured feature set is a data set that organizes the 3D coordinates of at least one feature to be tightened in the current working equipment in sequence.

[0038] Optionally, in this embodiment, the step of extracting the target 3D information corresponding to the feature to be tightened from the 3D point cloud based on the region of interest and the 3D point cloud corresponding to the 2D image, and generating a measured feature set, includes: Step S31: Obtain the three-dimensional point cloud corresponding to the two-dimensional image, and extract the local point cloud corresponding to the feature to be tightened from the three-dimensional point cloud according to the region of interest; Step S32: Preprocess the local point cloud to generate a point cloud to be processed, and determine the three-dimensional coordinates of the center point of the feature to be tightened in the point cloud to be processed by geometric fitting. Step S33: Use the three-dimensional coordinates as the target three-dimensional information corresponding to the feature to be tightened, and generate a measured feature set.

[0039] Specifically, firstly, the two-dimensional image obtained from step S20 is acquired from the camera module, and then the two-dimensional image is converted into a corresponding depth map, such as... Figure 4 As shown, Figure 4 This is a schematic diagram of the depth map involved in this embodiment, from which 3D point cloud data is extracted. Then, based on the pixel coordinate range of the region of interest, all points falling within that pixel range are selected from the 3D point cloud to form a local point cloud, such as... Figure 5 As shown, Figure 5 This is a schematic diagram of the point cloud of the ROI region involved in the embodiments of this application. To eliminate background noise and outliers, the local point cloud is preprocessed. Optionally, the preprocessing method can include actual methods for eliminating background noise and outliers. For example, a depth threshold can be set to remove points whose depth values ​​significantly deviate from the normal range of the stud surface; or statistical filtering can be used to remove isolated points with few surrounding points. The point cloud within the ROI contains clutter interference (such as points near the edge of the screw hole). Point cloud cleaning aims to remove non-cylindrical main points, ensuring that only pure stud cylindrical point clouds are retained. Furthermore, Euclidean clustering can also be used to remove interfering point clouds in the local point cloud.

[0040] Next, geometric fitting is performed on the preprocessed point cloud to be processed. For stud features, the cylindrical circular surface is fitted to obtain the center pose; for screw hole features, a circle fitting algorithm is used to extract the center coordinates. Finally, the three-dimensional coordinates of the fitted center point are used as the target three-dimensional information of the feature. Following the order of the template feature set, the above process is repeated for each feature to be tightened in the current working equipment, arranging all the extracted three-dimensional coordinates in the same order to generate the measured feature set.

[0041] For example, such as Figure 6 As shown, Figure 6 This is a schematic diagram of the feature point matching results involved in this implementation plan. Figure 6This is the final feature cloud obtained after feature point matching. Red represents the point cloud after matching the template feature set, and white represents the point cloud after matching the measured feature set. Normally there are 4 points, but due to the abnormal missing data of the measured vehicles, the final matching data is 3-3. This part not only ensures the one-to-one matching of the template point set and the measured point set, but also plays a role in filtering out special cases, ensuring the robustness of the algorithm.

[0042] Step S40: Match and align the template feature set with the measured feature set, and calculate the spatial transformation matrix corresponding to the template feature set and the measured feature set.

[0043] In this embodiment, each feature point in the measured feature set is paired with the corresponding feature points in the template feature set, and invalid pairings caused by missed detections or outliers are removed. The spatial transformation matrix consists of a rotation part and a translation part, and is used to describe the spatial position and orientation differences of the entire measured feature set relative to the template feature set.

[0044] Specifically, the correction control system first checks whether the measured feature set and the template feature set have the same number of feature points. If the number is different, invalid points at corresponding positions in the template feature set are removed according to the preset missed detection rules, so that the two feature sets achieve a one-to-one correspondence.

[0045] Optionally, in this embodiment, the steps of matching and aligning the template feature set with the measured feature set, and calculating the spatial transformation matrix corresponding to the template feature set and the measured feature set include: Step S41: Match and align the template feature set with the measured feature set; Step S42: Calculate the two-dimensional centroids of the aligned template feature set and the measured feature set on the working plane, as well as their average heights in the direction perpendicular to the working plane; Step S43: Construct a two-dimensional covariance matrix based on the two-dimensional centroid, and perform singular value decomposition on the two-dimensional covariance matrix to calculate the rotation matrix in the working plane; Step S44: Embed the rotation matrix into the corresponding position of the three-dimensional space rotation matrix, and use the difference in the average height in the vertical direction as the vertical translation amount to generate a spatial transformation matrix.

[0046] Specifically, after obtaining the aligned template point set and the measured point set, the two-dimensional centroid coordinates of the two point sets on the working plane (i.e., the XY plane) and the mean height in the vertical direction (Z direction) are calculated respectively. A 2×2 covariance matrix is ​​constructed using the transpose of the two-dimensional centroid-free coordinates of the measured point set and the template point set. Singular value decomposition is performed on this matrix to obtain a two-dimensional rotation matrix. This two-dimensional rotation matrix is ​​embedded in the upper left corner of a 3×3 identity matrix to form a complete spatial rotation matrix. Regarding the translation, the XY direction translation is calculated by rotating the two-dimensional centroid of the measured point set back to the template point set coordinate system. The Z direction translation is directly given by the difference in the mean heights of the two point sets. The rotation matrix and translation vectors are combined into a 4×4 spatial transformation matrix.

[0047] As an optional implementation, the above-mentioned spatial transformation matrix generation employs a modified SVD registration algorithm. Targeting the common vertical tightening scenarios in automotive assembly, it constrains the rotation dimension, calculating only specific rotations such as Rz and displacements. This ensures a stable correction matrix is ​​output even with only two feature points (such as the left and right studs on the tailgate). This approach solves the problem that general 3D SVD algorithms may experience rotation axis calculation collapse due to collinearity of point clouds when there are only two feature points.

[0048] This algorithm is used to calculate template feature sets. With measured feature set The transformation matrix between them. Unlike traditional 6-DOF registration, a registration strategy with restricted rotation dimensions is adopted to address the characteristic that the car body is perpendicular to the ground in the car tightening scenario: (1) Dimensional reduction centroid extraction: Calculate the two-dimensional centroids of the two point sets in the XY plane. and Then, independently calculate the average height of the two point sets in the Z direction. ,and Eliminate noise interference in non-operational directions.

[0049] (2) Construct the two-dimensional covariance matrix : Construct using only the centroid coordinates of the point set in the XY plane covariance matrix .

[0050] (3) SVD-based Solving by rotation: right Singular value decomposition (SVD) of a matrix yields a rotation matrix. This ensures that rotation calculations remain robust even when there are few feature points.

[0051] (4) Spatial transformation matrix synthesis: Two-dimensional rotation matrix Embedded Construct a complete spatial rotation matrix from the top left corner of the identity matrix. ,at this time The angle is always 0. As for the calculation of the translation... Translation of direction is based on Calculation, and The directional translation is directly calculated from the difference in the mean height.

[0052] (5) Output results: The final output includes Translation and Rotating Homogeneous transformation matrix, also known as spatial transformation matrix.

[0053] Furthermore, in this embodiment, the step of matching and aligning the template feature set with the measured feature set includes: Obtain the pre-stored template feature set, which contains the center point coordinates of at least two of the features to be tightened in the standard device; match the center points of the measured feature set with the center points of the template feature set in sequence, and calculate the positional deviation between each center point; remove abnormal points whose positional deviation exceeds the preset deviation range to obtain the aligned template feature set and the measured feature set.

[0054] Specifically, the template feature set contains the center point coordinates of at least two features to be tightened in the standard equipment. Similarly, the measured feature set contains the center point coordinates of at least two features to be tightened in the current operating equipment. Furthermore, for the common vertical tightening scenarios in automotive assembly, the rotation dimension is constrained, calculating only specific rotations such as Rz and displacements to ensure a stable correction matrix is ​​output even with only two feature points (such as the left and right studs on the tailgate). This approach solves the problem that the general 3D SVD algorithm may experience rotation axis calculation collapse due to collinearity of point clouds when there are only two feature points.

[0055] At the work site, due to oil stains, reflections, or obstructions, the point cloud fitting result of a feature to be tightened may occasionally deviate significantly from its actual position. These outliers can severely interfere with subsequent point cloud registration algorithms, causing distortion of the entire transformation matrix. By setting a reasonable deviation range, such as two or three times the standard deviation of the statistically obtained average deviation, outliers with excessive deviations are removed, retaining only points with deviations within the normal range for registration. Simultaneously, to ensure that the two point sets remain equal in size, points at corresponding positions in the template feature set are also removed. After removal, a pair of "clean" point sets are obtained, called the aligned template feature set and the aligned measured feature set, improving the robustness of the correction algorithm and preventing a single outlier from dragging down the entire alignment process.

[0056] First, the pre-stored template feature set file is read, containing the number of feature points, the X, Y, and Z coordinates of each feature point, and the label corresponding to each feature point. The feature point coordinates are loaded sequentially into a list or array structure in memory according to the order they were saved, forming the template feature set P_t. Simultaneously, the number of feature points N and the label information for each point are recorded.

[0057] The measured feature set should contain the same number of points as the template feature set. First, compare the lengths of the measured feature sets P_c and P_t: if the lengths are equal, pair P_c[i] with P_t[i] according to index order (i from 0 to N-1); if the lengths are unequal, reorder them according to label information or the nearest neighbor principle to align them. It should be understood that this embodiment assumes the case where the lengths are equal and the order is consistent. For each pair of paired points, calculate the three-dimensional Euclidean distance deviation. Store all calculated deviation values ​​(distances) in an array. The unit of the deviation values ​​is consistent with the coordinate unit, and the statistical characteristics of the deviation values, such as the mean, standard deviation, maximum, and minimum values, are calculated. Based on all calculated deviation values, determine the upper limit of a preset deviation threshold range. This threshold can be a preset fixed value, such as 5mm, because the deviation of the vehicle stud position usually does not exceed 5mm, or it can be a dynamically calculated statistical value, such as the mean plus twice the standard deviation. Then, iterate through each index i and check whether distances[i] is greater than the threshold. If the value is greater than the value, the point is considered an outlier; if the value is less than or equal to the value, the point is considered a normal point.

[0058] Create two new empty lists, filtered_Pt and filtered_Pc. For each index i, if distances[i] <= threshold, add P_t[i] to filtered_Pt and P_c[i] to filtered_Pc; if distances[i] > threshold, skip that index (i.e., remove both template points and measured points). After traversal, obtain the aligned template feature set (filtered_Pt) and the aligned measured feature set (filtered_Pc). If the number of remaining points after removal is less than 2, the detection is considered a failure, and an alarm can be issued or the correction amount from the previous frame can be used.

[0059] First, the system matches and calculates positional deviations sequentially, then removes outliers that exceed the deviation limit. This effectively filters out erroneous feature points caused by common industrial interferences such as obstruction, oil stains, and reflections, preventing these outliers from contaminating subsequent point cloud registration and correction calculations. By removing outliers, the correction algorithm estimates the true positional deviation more accurately, significantly improving overall robustness. Even in extreme cases where some feature points fail, the system can still utilize the remaining valid points for downgraded correction (e.g., translation without rotation), thereby maximizing the continuous operation capability of the production line and reducing downtime caused by detected anomalies.

[0060] Step S50: Generate a correction value based on the spatial transformation matrix and send the correction value to the robot so that the robot can correct the tightening trajectory according to the correction value.

[0061] In this embodiment, the correction amount refers to the motion compensation value required by the robot's end effector tightening gun. It is usually expressed as the translation along the X, Y, and Z axes and the rotation around each axis, i.e., a 1×6 pose offset vector. Correcting the tightening trajectory involves superimposing the correction amount on each target point in the original taught trajectory, so that the actual working position is precisely aligned with the stud or screw hole position of the current working equipment.

[0062] The spatial transformation matrix calculated in step S40 is parsed into a pose offset format that the robot can understand. Specifically, the translation amounts in the X, Y, and Z directions are extracted from the fourth column of the matrix, and the rotation angle around the Z-axis is extracted from the 3×3 rotation matrix in the upper left corner of the matrix. These six values ​​(ΔX, ΔY, ΔZ, 0, 0, ΔRz) are packaged into a pose offset vector according to the data format specified by the robot manufacturer. This offset vector is sent to the robot. After receiving the offset, the robot adds it to the taught trajectory of the current task: for example, if the originally taught tightening position is (X0, Y0, Z0, Rx0, Ry0, Rz0), the robot's actual target position becomes (X0+ΔX, Y0+ΔY, Z0+ΔZ, Rx0, Ry0, Rz0+ΔRz). The robot moves according to the corrected trajectory, precisely aligning the axis of the tightening gun with the axis of the stud or screw hole on the current work equipment, and then performs the tightening action.

[0063] Optionally, in this embodiment, the step of generating a correction value based on the spatial transformation matrix and sending the correction value to the robot so that the robot can correct the tightening trajectory according to the correction value further includes: The spatial transformation matrix is ​​decomposed into translation along three coordinate axes and rotation perpendicular to the coordinate axes of the working plane; the translation and rotation are combined into a correction value, and a pose offset command is generated based on the correction value; the pose offset command is sent to the robot to control the robot to correct the tightening trajectory and perform the tightening action through the tightening tool.

[0064] Specifically, the spatial transformation matrix calculated in step S40 is a 4×4 homogeneous matrix, containing rotation and translation components. The rotation component is a 3×3 matrix, theoretically potentially including rotations around the X, Y, and Z axes. However, in the body-in-white stud tightening scenario, the body plane is essentially perpendicular to the direction of gravity, and the tightening operation requires the tightening gun to be perpendicular to the body surface. Therefore, only the rotation angle perpendicular to the working plane (i.e., the Z-axis) needs to be compensated, while the tilt rotations around the X and Y axes are usually negligible. This step decomposes the matrix into three translational quantities (ΔX, ΔY, ΔZ) and one rotational quantity (ΔRz) around the Z-axis. The purpose is to convert the matrix form into an independent parameter form that is easier for the robot controller to understand and execute, facilitating subsequent combination into correction and pose offset commands.

[0065] Based on the robot's model and pre-configured communication protocol, the obtained (ΔX, ΔY, ΔZ, ΔRz) values ​​are combined into a complete correction value. Since compensation is limited to rotation around the Z-axis, rotations around the X and Y axes are set to 0. Then, a 1×6 pose offset vector is constructed: (ΔX, ΔY, ΔZ, 0, 0, ΔRz). The first three values ​​represent translation (millimeters), and the last three represent rotation (degrees). Next, pose offset instructions are generated according to the robot's instruction format. For robots that support direct transmission of offset vectors, data is packaged according to its specified byte order and data type. The offset coordinate system is set to the robot's current tool coordinate system, and the six values ​​are sequentially filled into the instruction buffer. For other robots that need to send JSON or custom binary protocols via TCP / IP, corresponding data packets need to be constructed. Furthermore, checksums or sequence numbers can be added to the instructions to ensure communication reliability.

[0066] Finally, the generated pose offset command is sent to the robot's controller via the communication interface. Upon receiving the command, the robot controller parses the offset and stores it in an internal variable. The robot moves to the target position according to the corrected trajectory and then activates the end effector's tightening gun to tighten the studs or screw holes. After tightening, the robot can return a completion status code to the controller. The controller can optionally receive confirmation information and record the result of this correction operation.

[0067] In the technical solution provided in this embodiment, by first acquiring the template feature set of a standard vehicle, then sequentially performing two-dimensional image localization of the region of interest (ROI) on the current vehicle, extracting a three-dimensional point cloud based on the ROI and fitting a measured feature set, and finally calculating the spatial transformation matrix and outputting the correction amount to the robot, dynamic adaptive correction of the tightening trajectory is achieved. Compared with the traditional fixed trajectory solution, this embodiment can effectively overcome the random pose deviation caused by the conveyor line and tooling fixtures, controlling the alignment accuracy of the tightening gun and the stud to within 0.5 mm, significantly reducing the risk of screw damage and vehicle collision. In addition, the correction efficiency of this embodiment is significantly improved compared with manual inspection and correction of the trajectory.

[0068] On the other hand, this application also provides a computer storage medium storing executable program code; the executable program code is used to execute the above-mentioned correction method for any tightening trajectory.

[0069] On the other hand, this application also provides a terminal device, including a memory and a processor; the memory stores program code that can be executed by the processor; the program code is used to execute the above-mentioned correction method for any tightening trajectory.

[0070] For example, the program code can be divided into one or more modules / units, which are stored in the memory and executed by the processor to complete this application. The one or more modules / units can be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the program code in a terminal device.

[0071] The terminal device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The terminal device may include, but is not limited to, a processor and memory. Those skilled in the art will understand that the terminal device may also include input / output devices, network access devices, buses, etc.

[0072] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0073] The memory can be an internal storage unit of the terminal device, such as a hard drive or RAM. The memory can also be an external storage device of the terminal device, such as a plug-in hard drive, SmartMedia Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, the memory can include both internal and external storage units of the terminal device. The memory is used to store the program code and other programs and data required by the terminal device. The memory can also be used to temporarily store data that has been output or will be output.

[0074] The aforementioned terminal device is created based on the aforementioned method for correcting the tightening trajectory. Its technical function and beneficial effects will not be elaborated here. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0075] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for correcting a tightening trajectory, characterized in that, include: Obtain the three-dimensional pose information of the feature to be tightened in the standard equipment, and save the three-dimensional pose information as a template feature set; Acquire a two-dimensional image of the current operating equipment, and determine the region of interest corresponding to the feature to be tightened in the two-dimensional image through a preset matching mode; Based on the region of interest and the three-dimensional point cloud corresponding to the two-dimensional image, the target three-dimensional information corresponding to the feature to be tightened is extracted from the three-dimensional point cloud, and a measured feature set is generated. The template feature set is matched and aligned with the measured feature set, and the spatial transformation matrix corresponding to the template feature set and the measured feature set is calculated. A correction value is generated based on the spatial transformation matrix, and the correction value is sent to the robot so that the robot can correct the tightening trajectory according to the correction value.

2. The method for correcting the tightening trajectory according to claim 1, characterized in that, The step of extracting the target 3D information corresponding to the feature to be tightened from the 3D point cloud based on the region of interest and the 3D point cloud corresponding to the 2D image, and generating a measured feature set, includes: Obtain a three-dimensional point cloud corresponding to the two-dimensional image, and extract a local point cloud corresponding to the feature to be tightened from the three-dimensional point cloud according to the region of interest; The local point cloud is preprocessed to generate a point cloud to be processed, and the three-dimensional coordinates of the center point of the feature to be tightened are determined in the point cloud to be processed by geometric fitting. The three-dimensional coordinates are used as the target three-dimensional information corresponding to the feature to be tightened, and a measured feature set is generated.

3. The method for correcting the tightening trajectory according to claim 1, characterized in that, The step of matching and aligning the template feature set with the measured feature set, and calculating the spatial transformation matrix corresponding to the template feature set and the measured feature set, includes: The template feature set is matched and aligned with the measured feature set; Calculate the two-dimensional centroid of the aligned template feature set and the measured feature set on the working plane, as well as their average height in the direction perpendicular to the working plane; A two-dimensional covariance matrix is ​​constructed based on the two-dimensional centroid, and singular value decomposition is performed on the two-dimensional covariance matrix to calculate the rotation matrix in the working plane. The rotation matrix is ​​embedded into the corresponding position of the three-dimensional spatial rotation matrix, and the difference in the average height in the vertical direction is used as the vertical translation amount to generate the spatial transformation matrix.

4. The method for correcting the tightening trajectory according to claim 3, characterized in that, The step of matching and aligning the template feature set with the measured feature set includes: Obtain the pre-stored template feature set, which contains the center point coordinates of at least two of the features to be tightened in the standard device; The center points of the measured feature set are matched with the center points of the template feature set in sequence, and the positional deviation between each center point is calculated. By removing outliers whose positional deviations exceed a preset deviation range, the aligned template feature set and the measured feature set are obtained.

5. The method for correcting the tightening trajectory according to claim 1, characterized in that, The step of determining the region of interest corresponding to the feature to be tightened in the two-dimensional image by means of a preset matching mode further includes: The region of interest in the two-dimensional image is determined by the preset matching mode, wherein the preset matching mode includes a first matching mode and a second matching mode; When the preset matching mode is detected to be the first matching mode, a preset template image is called through the template matching algorithm to search for a target region in the two-dimensional image that meets the preset conditions, and the target region is determined as the region of interest; Otherwise, the two-dimensional image is detected by a deep neural network in the second matching mode, and the output feature region is determined as the region of interest.

6. The method for correcting the tightening trajectory according to claim 5, characterized in that, After the step of detecting the two-dimensional image through a deep neural network in the second matching mode and determining the output feature region as the region of interest, the method further includes: Determine the confidence level of the deep neural network output; When the confidence level is lower than a preset threshold, the system switches to the first matching mode to identify and output the region of interest. Obtain the region of interest output by the first matching pattern; The region of interest is cropped and labeled, and a training dataset is generated based on the labeled information, so as to update the network parameters of the deep neural network corresponding to the second matching pattern through the training dataset.

7. The method for correcting the tightening trajectory according to claim 5, characterized in that, The step of calling a preset template image through a template matching algorithm, searching for a target region in the two-dimensional image that meets a preset matching condition, and determining the target region as the region of interest further includes: A template library is generated by extracting a first image of the features to be tightened from a standard device as an initial template and combining it with a second image acquired under at least one angle condition and / or at least one lighting condition. Using a template matching algorithm based on edge gradient and / or grayscale gradient, a matching calculation is performed between the template image in the template library and the two-dimensional image to determine the target region in the two-dimensional image whose matching degree meets the preset conditions, and the target region is used as the output region of interest.

8. The method for correcting the tightening trajectory according to claim 1, characterized in that, The step of generating a correction value based on the spatial transformation matrix and sending the correction value to the robot so that the robot corrects the tightening trajectory based on the correction value further includes: The spatial transformation matrix is ​​decomposed into translation along the three coordinate axes and rotation perpendicular to the coordinate axes of the working plane; The translation and rotation amounts are combined to form the correction amount, and a pose offset command is generated based on the correction amount. The pose offset command is sent to the robot to control the robot to correct the tightening trajectory and perform the tightening action through the tightening tool.

9. A computer storage medium, characterized in that, It stores executable program code; the executable program code is used to execute the method for correcting the tightening trajectory according to any one of claims 1 to 8.

10. A terminal device, characterized in that, It includes a memory and a processor; the memory stores program code that can be executed by the processor; the program code is used to execute the method for correcting the tightening trajectory according to any one of claims 1 to 8.

Images