Image recognition-based auxiliary positioning method and system for automobile screw installation

Abstract

This invention discloses an image recognition-based auxiliary positioning method and system for automotive screw installation, belonging to the field of image recognition technology. The method includes the following steps: deploying and calibrating a multimodal sensor array at the assembly station; based on the acquired multimodal images, performing specular reflection detection and compensation on the metal surface using a reflection compensation strategy to obtain a reflection-compensated screw image; performing screw target detection and attitude estimation to obtain the screw head center coordinates, axial direction, and deflection angle; performing coordinate transformation and initial trajectory planning on the identified screw pose and tightening tool pose; and having the robot perform insertion actions based on visual coarse positioning. During insertion and tightening, a pose-controlled particle swarm optimization algorithm is used to fuse torque sensor data and visual micro-measurements in real time for state determination and tightening tool control. This invention can automatically compensate for abnormal engagement, improving the accuracy of tightening judgment and assembly safety.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, specifically to an image recognition-based auxiliary positioning method and system for automotive screw installation. Background Technology

[0002] With the increasing automation of automobile assembly, intelligent screw recognition and installation assistance based on machine vision have become key technologies for improving assembly accuracy and efficiency. However, in actual production environments, screw positioning and reliable tightening still face the problem that screws and other fasteners are small-scale targets that are easily obscured or confused with the background / fixture. Traditional target detection methods cannot simultaneously meet sub-millimeter-level assembly tolerances in terms of recall and positioning accuracy.

[0003] To address the aforementioned issues, research and engineering practice typically employ multimodal sensing and joint control strategies to improve robustness. At the sensing end, high-resolution cameras and other sensors are introduced to compensate for the limitations of single-channel sensing. At the control end, visual coarse positioning combined with force / torque feedback-based compliant control or torque-angle closed-loop controllers are commonly used to complete the final insertion and tightening determination. These methods alleviate the problems caused by single sensing or single control to some extent, improving assembly pass rates and equipment safety.

[0004] However, for automotive screws, firstly, the high gloss and specular reflection of the metal surface cause polarization imaging. While multi-exposure and dichroic reflection models can physically separate the specular component, simple thresholding or static models struggle to adapt to complex incident angles, batch material refractive index variations, and local saturation (pixel saturation). Active depth often exhibits virtual depth / no depth phenomena in specular areas. Secondly, visual perception alone cannot reliably determine the tightening status. While vision can provide presence and geometric pose information, whether tightening is complete is a mechanical process that relies on feedback from torque / force sensors or torque tools. A single signal may indicate misalignment or false engagement, and misjudgment can lead to safety hazards (loosening) or over-tightening causing screw breakage.

[0005] Therefore, the present invention provides an auxiliary positioning method and system for automotive screw installation based on image recognition. Summary of the Invention

[0006] The purpose of this invention is to provide an image recognition-based auxiliary positioning method and system for automotive screw installation, in order to solve the existing problems mentioned in the background art.

[0007] To achieve the above objectives, the present invention provides the following technical solution: an image recognition-based auxiliary positioning method for automotive screw installation, comprising the following steps: S1. Deploy and calibrate a multimodal sensor array at the assembly station; S2. Based on the multimodal acquired image, a reflection compensation strategy is used to perform specular reflection detection and reflection compensation on the metal surface to obtain a reflection-compensated screw image. S3. Based on the reflection-compensated screw image, perform screw target detection and attitude estimation to obtain the screw head center coordinates, axial direction and deflection angle; S4. Perform coordinate transformation and initial trajectory planning on the identified screw pose and tightening tool pose, and the robot performs the insertion action according to the visual coarse positioning. S5. During the insertion and tightening process, the torque sensor data and visual micro-measurement are fused in real time through the pose control particle swarm algorithm to determine the state and control the tightening tool. S6. Record key events in the tightening process and generate quality judgment results.

[0008] A further improvement of this invention is that the reflection compensation strategy specifically includes the following steps: S21. Acquire a set of images of the same field of view at different polarization angles and multiple exposure levels, denoted as follows: ,in Indicates the first One polarization angle, Indicates the first One exposure level; S22. Calculate pixel-level linear polarization degree and polarization difference by using the polarization component intensities parallel and perpendicular to the mirror reflection direction. : in and These represent the intensities of the polarization components parallel and perpendicular to the direction of specular reflection, respectively. This represents a hyperparameter for preventing division by zero. S23. Obtain the brightness threshold through the specular reflection distortion index. Then, the pixels are divided into a set of highly reflective pixels and a set of low reflective pixels to construct a specular mask. ; S24. Estimating the mirror aspect ratio using polarization information. Thus, diffuse reflection estimation is obtained. The diffuse reflection image is then used to replace the pixels corresponding to the high reflectivity pixel set in the original RGB image to obtain the reflection-compensated screw image.

[0009] A further improvement of this invention is that the specular distortion index is obtained by extracting the region of interest (ROI) of the screw head in the image; firstly, the area of ​​pixels with gray values ​​greater than the saturation threshold within the ROI is counted, and the proportion of these pixels to the total area of ​​the ROI is calculated to obtain the saturation ratio; then, the gray-level histogram information entropy of the ROI region is calculated, and edge gradient features are extracted using the Laplacian operator; finally, the specular distortion index is generated based on the weighted sum of the saturation ratio, information entropy, and edge gradient features. ; Then the brightness threshold The regulation rules are ,in This indicates the calculation of the specular distortion index on the ROI. Indicates the baseline threshold. This represents the scaling factor.

[0010] A further improvement of this invention is that the specific steps of screw target detection and pose estimation in step S3 include: S31. Diffuse reflection image after reflection compensation The coarse localization bounding box is obtained by using a multi-scale small target detection network. S32. Within the coarse positioning bounding box, sub-pixel edge extraction and least-squares polygon fitting are used to extract the geometric features of the screw head, and sub-pixel interpolation is used to obtain the screw center. ; S33. Using the reflection-compensated screw image and the screw head geometric model, establish a 2D–3D corresponding point set and solve for the initial pose. ; S34. Optimize the initial pose using contour-based least squares. The objective function is to minimize the sum of pixel reprojection error and depth registration error: in Represents the three-dimensional points of the template. Indicates the corresponding pixel coordinates. Represents the camera projection function. This represents the weighting coefficient.

[0011] A further improvement of this invention is that the pose control particle swarm algorithm is based on the real-time specular distortion index. With axial impedance stability index An adaptive particle swarm optimization-driven PID parameter optimization framework is established, wherein the pose-controlled particle swarm algorithm includes: S51. Set state determination rules, and determine whether to proceed to step S52 based on the state determination rules; S52. A nonlinear cooperative modulation mechanism is constructed based on a coupling adjustment function with dual feature inputs. The function outputs the cooperative risk entropy by calculating the nonlinear product term of the specular reflection distortion index and the axial impedance stability. ; S53. Configure an adaptive particle swarm optimization engine by deploying the particle swarm optimization algorithm as the core controller, and simultaneously establish a parameter adaptive channel to output the collaborative risk entropy from step S52. Inertia weights mapped to the particle swarm optimization algorithm and mutation probability ; S54. Represent the PID parameter set as particles, perform PSO updates based on online evaluation indicators, output the pose compensation command of the robotic arm and the real-time closed-loop tightening control of the PID control parameters, and perform parameter rollback or enter force-dominated compliant control when unstable control or over-limit is detected.

[0012] A further improvement of this invention is that the state determination rule includes: acquiring a torque signal. Its derivative And visual measurements including axial gaps and micro-deformation measurement ; When satisfied When the engagement state is determined, the engagement state is entered. Indicates the threshold for sudden change in contact stiffness. This indicates the preset thread engagement depth threshold. When satisfied When the tightening is stopped, it is determined that the tightening has ended. Indicates the target torque value. Indicates the maximum permissible rotation angle. Indicates the tool's angular velocity; If the tightening process meets the mechanical termination condition but the visual inspection system simultaneously reports the presence of residual gaps, it is considered as follows: If the risk of misalignment is detected, S52 will be executed.

[0013] A further improvement of this invention is that the adaptive particle swarm optimization engine is configured as follows: establishing inertial weights and cooperative risk entropy. Positive correlation mapping Simultaneously establish mutation probability Positive correlation mapping with collaborative risk entropy When the collaborative risk entropy exceeds its preset upper limit due to the dual deterioration of the input specular reflection distortion index and axial impedance stability, it is determined to be a high-risk state; the inertial weight and the mutation probability are simultaneously locked to their respective preset upper limits.

[0014] A further improvement of the present invention is that the axial impedance stability index The axial resistance stability is obtained by acquiring the torque data sequence and Z-axis depth data sequence within the current time window. First, the differential value of torque with respect to Z-axis depth is calculated to obtain the instantaneous contact stiffness. Second, the absolute value of the deviation between the instantaneous contact stiffness and the preset standard thread engagement stiffness curve is calculated. Finally, the variance of the torque data sequence is calculated, and this variance is used to normalize the absolute value of the deviation to obtain the axial resistance stability. .

[0015] A further improvement of this invention is that the construction formula of the nonlinear cooperative modulation mechanism is expressed as: in, Indicates the single-factor coefficient. Represents the coupling coefficient. This indicates the exponential enhancement factor.

[0016] On the other hand, the present invention provides an image recognition-based auxiliary positioning system for automotive screw installation, comprising: The multi-modal data acquisition module is configured to deploy and calibrate a combination of multi-modal sensors at the assembly station. The reflection compensation module is configured to perform specular reflection detection and reflection compensation on the metal surface based on the multimodal acquired image, and obtain a reflection-compensated screw image by using a reflection compensation strategy. The pose estimation module is configured to perform screw target detection and pose estimation based on the reflection-compensated screw image to obtain the screw head center coordinates, axial direction and deflection angle. The visual coarse positioning module is configured to perform coordinate transformation and initial trajectory planning on the identified screw pose and tightening tool pose, and the robot performs the insertion action according to the visual coarse positioning. The pose control module is configured to use a pose control particle swarm algorithm to fuse torque sensor data and visual microscopic measurements in real time during insertion and tightening to determine the state and control the tightening tool. The data logging module is configured to record key events in the tightening process and generate quality assessment results.

[0017] Compared with the prior art, the beneficial effects of the present invention are: 1. This invention first introduces a polarization-based mirror separation and reflection compensation process at the sensing end, and replaces high-reflectivity pixels with diffuse reflection estimation to realize texture and depth recovery of areas contaminated by high light. This solves the problem of RGB saturation and active depth failure caused by high light / specular reflection on metal surfaces, thereby significantly improving point cloud availability and reducing positioning failure rate and assembly rework rate.

[0018] 2. By constructing a mixing adaptive control model based on interference observation, and using a particle swarm optimization engine with nonlinear cooperative function adjustment based on real-time calculated visual distortion index and axial impedance stability index, the PID control parameters are searched and adaptively adjusted online. This achieves real-time fusion and state determination of visual micro-measurement and torque signal, solving the problem that visual alone cannot reliably determine whether the torque meets the standard. This enables automatic compensation when there is abnormal engagement, improving the accuracy of tightening determination and assembly safety. Attached Figure Description

[0019] Figure 1 This is a flowchart of the image recognition-based auxiliary positioning method for automotive screw installation according to the present invention.

[0020] Figure 2 This is a framework diagram of the image recognition-based automotive screw installation auxiliary positioning system of the present invention. Detailed Implementation

[0021] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present invention and the specific features in the embodiments are detailed descriptions of the technical solution of the present invention, rather than limitations thereof. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments can be combined with each other.

[0022] The term "and / or" is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three cases: A exists alone, A and B exist simultaneously, and B exists alone.

[0023] Example 1 Figure 1 The flowchart of the image recognition-based auxiliary positioning method for automotive screw installation disclosed in this embodiment is shown, and the steps are as follows: S1. Deploy and calibrate a multimodal sensor combination at the assembly station. The multimodal sensor combination includes, but is not limited to: a high-resolution RGB camera under ring diffuse reflection illumination, a switchable polarization angle camera with a polarizer, a short-wave infrared (SWIR) or near-infrared camera, an active depth camera (structured light or ToF) and a laser profile sensor, and a torque / force sensing unit mounted on a tightening tool. S2. Based on the multimodal acquired image, a reflection compensation strategy is used to perform specular reflection detection and reflection compensation on the metal surface to obtain a reflection-compensated screw image. The specific steps of the reflection compensation strategy include: S21. Acquire a set of images of the same field of view at different polarization angles (at least two angles, preferably three angles) and multiple exposure levels, and record them as follows: ,in Indicates the first One polarization angle, Indicates the first One exposure level; S22. Calculate the pixel-level Degree of Linear Polarization (DOLP) and polarization difference using the polarization component intensities parallel and perpendicular to the mirror reflection direction. : in and These represent the polarization component intensities parallel and perpendicular to the direction of specular reflection, respectively. To prevent division by zero hyperparameters; a larger DOLP indicates a more significant mirror component; In the polarization component, specular reflection is strongly polarized, while diffuse reflection is relatively unpolarized. A normalized degree of polarization (values ​​∈ (−1, 1)) is obtained by dividing the difference between the parallel / perpendicular components by the sum, and then... Preventing division by zero. Quantizing the specular component at the pixel level allows for precise location of highly reflective pixels, thereby reducing the risk of mistaking highlights for real textures.

[0024] S23. Obtain the brightness threshold through the specular reflection distortion index. Then, the pixels are divided into a set of highly reflective pixels and a set of low reflective pixels to construct a specular mask. ; The specular distortion index is obtained by extracting the region of interest (ROI) of the screw head in the image. First, the area of ​​pixels within the ROI with gray values ​​greater than the saturation threshold is counted, and their proportion of the total ROI area is calculated to obtain the saturation percentage. Then, the gray-level histogram information entropy of the ROI region is calculated, and edge gradient features are extracted using the Laplacian operator. Finally, the specular distortion index is generated based on the weighted sum of the saturation percentage, information entropy, and edge gradient features. The calculation formula is expressed as follows: ,in, Quantify the area of ​​saturated highlights; The method uses Laplacian to detect brightness abrupt changes, followed by a second-order detection of the grayscale histogram entropy to capture evidence of texture loss and local flattening. A higher specular distortion index indicates more severe texture loss due to specular reflection.

[0025] Then the brightness threshold The regulation rules are ,in This indicates the calculation of the specular distortion index on the ROI. Indicates the baseline threshold. This represents the scaling factor. Thus, when the ROI exhibits severe specular distortion ( When the threshold is large, it is automatically increased to avoid misjudging diffuse reflection as specular component.

[0026] This embodiment aggregates multiple observations into an interpretable distortion index, facilitating regional adaptive decision-making; and dynamically adjusts the mirror determination threshold based on the ROI. This improves the targeting and robustness of mirror detection, and avoids the failure of a globally fixed threshold under different materials / lighting conditions.

[0027] S24. Estimating the mirror aspect ratio using polarization information. Thus, diffuse reflection estimation is obtained. The diffuse image is then used to replace the pixels corresponding to the high reflectivity pixel set in the original RGB image to obtain the reflection-compensated screw image. Based on the dichromatic reflection model, the total intensity of the pixel is... Expressed as the sum of the diffuse reflection term and the specular term: And estimate the mirror ratio using polarization information. Approximate calculation of mirror components: in For camera geometry (incident angle) ) and material refractive index The maximum observable degree of polarization (approximately calculated by the Fresnel equation or obtained through field calibration); S25. In this embodiment, the image of the reflection compensation screw can also be processed. Introducing pixel-level confidence weights ( To adjust the parameters, a confidence-weighted deep repair method was used: in The depth is estimated by monocular depth estimation or laser contour reconstruction based on RGB; and the depth is obtained by... Perform weighted bilateral filtering or global optimization based on conditional random fields (CRF) to remove noise.

[0028] Deep weighted fusion based on pixel confidence It can reliably recover depth using monocular / contour data at points where active depth fails, reducing 6-DoF pose error.

[0029] S3. Based on the reflection-compensated screw image, perform screw target detection and sub-pixel level pose estimation to obtain the screw head center coordinates, axial direction, and deflection angle (6-DoF pose); specific steps include: S31. Diffuse reflection image after reflection compensation The coarse localization bounding box is obtained by using a multi-scale small target detection network (such as an FPN-based detector); S32. Within the coarse positioning bounding box, sub-pixel edge extraction and least-squares polygon fitting are used to extract the geometric features of the screw head (such as the center of the internal hexagonal hole and the outline of the external hexagonal hole), and the screw center is obtained by sub-pixel interpolation. ; S33. Establish a 2D–3D corresponding point set using the reflection-compensated screw image and the screw head geometric model, and solve for the initial pose using PnP (Perspective-n-Point). ; S34. Optimize the initial pose using contour-based least squares. The objective function is to minimize the sum of pixel reprojection error and depth registration error: in Represents the three-dimensional points of the template. Indicates the corresponding pixel coordinates. Represents the camera projection function. This represents the weighting coefficient.

[0030] By combining diffuse reflection-enhanced 2D features with the repaired depth map, both detection robustness and sub-pixel pose accuracy are ensured.

[0031] S4. Perform coordinate transformation and initial trajectory planning on the identified screw pose and tightening tool pose, and the robot performs the insertion action according to the visual coarse positioning. S41, the initial approach pose From the filtered pose According to the formula Confirmed, among which This embodiment uses a unit vector along the screw axis and reserves a safety distance. Within the range of 3-15 mm; S42, The insertion velocity profile adopts a three-segment velocity curve. ,in The acceleration-steady speed-deceleration segmented curve is as follows: This represents the maximum allowable insertion speed at the workstation. This is a velocity decay factor used to assess real-time pose error. Adaptive deceleration; S43. During the insertion process, the specular distortion index, an image quality metric, is used. With depth confidence Implement control mode switching: When and When using a vision-dominated mode, image-based visual servoing (IBVS) or position-based visual servoing (PBVS) is preferred; when or Switch to force-driven compliant control, with torque / force primarily triggering engagement and tightening actions; S44. The compliance control adopts an admission controller, the model of which is: in For equivalent mass, damping, and stiffness parameters, For the desired contact force, To measure the contact force; by adjusting A strategy for achieving different levels of compliance to ensure safe insertion force under visual failure or high reflectivity conditions; S45. Set reentry / retreat conditions: When the real-time pose error... (For example, when using M6 screws) or angle error (For example When ), pause insertion and revert to To readjust, and when continuous If the tightening status is still not determined after the second retry, manual intervention or a downgrade process will be triggered.

[0032] S5. During the insertion and tightening process, the torque sensor data and visual micro-measurements (such as gaps, drops, and micro-deformations) are fused in real time through the pose control particle swarm algorithm to determine the state and control the tightening tool. The pose control particle swarm algorithm is based on the real-time specular distortion index. With axial impedance stability index An adaptive particle swarm optimization-driven PID parameter optimization framework is established, wherein the axial impedance stability index... The axial resistance stability is obtained by acquiring the torque data sequence and Z-axis depth data sequence within the current time window. First, the differential value of torque with respect to Z-axis depth is calculated to obtain the instantaneous contact stiffness. Second, the absolute value of the deviation between the instantaneous contact stiffness and the preset standard thread engagement stiffness curve is calculated. Finally, the variance of the torque data sequence is calculated, and this variance is used to normalize the absolute value of the deviation to obtain the axial resistance stability. The calculation formula is expressed as follows: , Indicates instantaneous stiffness and theoretical engagement stiffness. Deviations reveal contact anomalies; divided by (Variance) can suppress false alarms under high-noise operating conditions, and use Avoid division by zero; It is sensitive to real contact instability and robust in noisy conditions, thus providing a reliable mechanical input for PSO.

[0033] The larger the axial impedance stability value, the more unstable the contact state is during the screw-in process.

[0034] This embodiment normalizes the instantaneous torque by combining the derivative of the torque with depth (approximate contact stiffness) with the torque variance, enabling it to distinguish between true stiffness deviation and noise fluctuations in dynamic noise environments; after normalization... It can be safely and comparablely used as PSO input, with Using units of the same dimension ensures the stability of coupled logic.

[0035] The pose control particle swarm algorithm includes: S51. Set state determination rules, and determine whether to proceed to step S52 based on the state determination rules; the state determination rules include: acquiring torque signal. Its derivative And visual measurements including axial gaps (Axial distance measured from sub-pixel edges / contours) and micro-deformation metrics (Measured by sub-pixel level template difference or structured light deformation of adjacent frames); When satisfied When the engagement state is determined, the engagement state is entered. Indicates the threshold for sudden change in contact stiffness. This indicates the preset thread engagement depth threshold. When satisfied When the tightening is stopped, it is determined that the tightening has ended. Indicates the target torque value. Indicates the maximum permissible rotation angle. Indicates the tool's angular velocity; If the tightening process meets the mechanical termination condition ( However, the visual inspection system also reported the presence of residual gaps, indicating that... If the risk of misalignment is detected, S52 will be executed.

[0036] By combining instantaneous torque derivative, visual gap, and micro-deformation into a judgment rule, key events such as engagement, compliance, and risk of malocclusion can be detected in a timely and reliable manner. When mechanical / visual inconsistencies occur, PSO is triggered in a timely manner to readjust or compensate parameters, avoiding misjudgments and damage caused by purely visual or purely force-based control.

[0037] S52. A nonlinear cooperative modulation mechanism is constructed based on a coupling adjustment function with dual feature inputs. The function outputs the cooperative risk entropy by calculating the nonlinear product term of the specular reflection distortion index and the axial impedance stability. This characterizes the combined risk level of simultaneous failure of vision and force perception under the current working conditions; the construction formula of the nonlinear cooperative modulation mechanism is expressed as: in, Indicates the single-factor coefficient. Represents the coupling coefficient. Indicates the exponential enhancement factor; when and When both are in the high value range, the product coupling term makes It exhibits non-linear exponential growth, thus triggering the algorithm's escape response; and when When it approaches 0, regardless of The numerical value of the product coupling term is suppressed to prevent visual noise from falsely triggering high-risk responses.

[0038] By using nonlinear and exponential coupling, visual and mechanical anomalies are amplified into a single risk quantity Ψ, enabling the system to quickly enter a high exploration / safety mode when both deteriorate, avoiding misjudgment of any single feature; exponential coupling and product terms give priority to situations where both indicators deteriorate simultaneously, improving the system's response speed and reliability to real complex faults.

[0039] S53. Configure an adaptive particle swarm optimization engine by deploying the particle swarm optimization algorithm as the core controller, and simultaneously establish a parameter adaptive channel to output the collaborative risk entropy from step S52. Inertia weights mapped to the particle swarm optimization algorithm and mutation probability This enables dynamic control over the algorithm's exploration and convergence capabilities; the adaptive particle swarm optimization engine is configured by establishing inertial weights and collaborative risk entropy. Positive correlation mapping Simultaneously establish mutation probability Positive correlation mapping with collaborative risk entropy When the collaborative risk entropy exceeds its preset upper limit (e.g., 0.8) due to the dual deterioration of the input specular reflection distortion index and axial impedance stability, it is determined to be a high-risk state; the inertial weight and mutation probability are simultaneously locked to their respective preset upper limits. When the inertial weight and mutation probability deteriorate and approach 1, the inertial weight and mutation probability are driven to reach the preset upper limit, forcing the particle swarm optimization engine to perform a global large-scale search in order to escape the local optimum trap.

[0040] By linking the algorithm's exploration and development capabilities with real-time operating conditions (Ψ), optimization becomes more biased towards global search under high-risk conditions, enabling it to escape from noise / local optima; converges faster under low-risk conditions to improve efficiency; and the upper limit locking mechanism ensures that the algorithm adopts the maximum exploration strategy under extreme deterioration, thereby enhancing its robust escape capability under complex and abnormal operating conditions.

[0041] S54. Represent the PID parameter set as particles, based on online evaluation indicators (including installation error). With installation time scalarization target ) Execute PSO update, output the pose compensation command of the robotic arm and the PID control parameter configuration for real-time closed-loop tightening control, and execute parameter rollback or enter force-dominated compliant control when unstable control or over-limit is detected.

[0042] By coupling online PSO with visual / mechanical features, control parameters can be adaptively adjusted in complex, time-varying and noisy environments, balancing accuracy (E) and efficiency (T). When visual interference or mechanical instability occurs, the global search capability is increased to escape local optima, thereby reducing stripping / jamming or rework caused by local erroneous parameters.

[0043] S6. Record key events in the tightening process (such as thread engagement, target torque, and springback detection) and generate quality judgment results.

[0044] The threshold and weight settings can be set by default according to the present invention, or they can be set by those skilled in the art.

[0045] Example 2 Figure 2 The diagram illustrates the framework of the image recognition-based auxiliary positioning system for automotive screw installation according to the present invention. Based on the same inventive concept as Embodiment 1, the present invention provides an image recognition-based auxiliary positioning system for automotive screw installation, the system comprising: The multi-modal data acquisition module is configured to deploy and calibrate a combination of multi-modal sensors at the assembly station. The reflection compensation module is configured to perform specular reflection detection and reflection compensation on the metal surface based on the multimodal acquired image, and obtain a reflection-compensated screw image by using a reflection compensation strategy. The pose estimation module is configured to perform screw target detection and pose estimation based on the reflection-compensated screw image to obtain the screw head center coordinates, axial direction and deflection angle. The visual coarse positioning module is configured to perform coordinate transformation and initial trajectory planning on the identified screw pose and tightening tool pose, and the robot performs the insertion action according to the visual coarse positioning. The pose control module is configured to use a pose control particle swarm algorithm to fuse torque sensor data and visual microscopic measurements in real time during insertion and tightening to determine the state and control the tightening tool. The data logging module is configured to record key events in the tightening process and generate quality assessment results.

[0046] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0047] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0048] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0049] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0050] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims. All of these forms are within the protection scope of the present invention.

Claims

1. An image recognition-based auxiliary positioning method for automotive screw installation, characterized in that: Includes the following steps: S1. Deploy and calibrate a multimodal sensor array at the assembly station; S2. Based on the multimodal acquired image, a reflection compensation strategy is used to perform specular reflection detection and reflection compensation on the metal surface to obtain a reflection-compensated screw image. S3. Based on the reflection-compensated screw image, perform screw target detection and attitude estimation to obtain the screw head center coordinates, axial direction and deflection angle; S4. Perform coordinate transformation and initial trajectory planning on the identified screw pose and tightening tool pose, and the robot performs the insertion action according to the visual coarse positioning. S5. During the insertion and tightening process, the torque sensor data and visual micro-measurement are fused in real time through the pose control particle swarm algorithm to determine the state and control the tightening tool. S6. Record key events in the tightening process and generate quality judgment results.

2. The image recognition-based auxiliary positioning method for automotive screw installation according to claim 1, characterized in that: The specific steps of the reflection compensation strategy include: S21. Acquire a set of images of the same field of view at different polarization angles and multiple exposure levels, denoted as follows: ,in Indicates the first One polarization angle, Indicates the first One exposure level; S22. Calculate pixel-level linear polarization degree and polarization difference by using the polarization component intensities parallel and perpendicular to the mirror reflection direction. : in and These represent the intensities of the polarization components parallel and perpendicular to the direction of specular reflection, respectively. This represents a hyperparameter for preventing division by zero. S23. Obtain the brightness threshold through the specular reflection distortion index. Then, the pixels are divided into a set of highly reflective pixels and a set of low reflective pixels to construct a specular mask. ; S24. Estimating the mirror aspect ratio using polarization information. Thus, diffuse reflection estimation is obtained. The diffuse reflection image is then used to replace the pixels corresponding to the high reflectivity pixel set in the original RGB image to obtain the reflection-compensated screw image.

3. The image recognition-based auxiliary positioning method for automotive screw installation according to claim 2, characterized in that: The specular distortion index is obtained by extracting the region of interest (ROI) of the screw head in the image. First, the area of ​​pixels within the ROI with gray values ​​greater than the saturation threshold is counted, and their proportion of the total ROI area is calculated to obtain the saturation percentage. Then, the gray-level histogram information entropy of the ROI region is calculated, and edge gradient features are extracted using the Laplacian operator. Finally, the specular distortion index is generated based on the weighted sum of the saturation percentage, information entropy, and edge gradient features. ; Then the brightness threshold The regulation rules are ,in This indicates the calculation of the specular distortion index on the ROI. Indicates the baseline threshold. This represents the scaling factor.

4. The image recognition-based auxiliary positioning method for automotive screw installation according to claim 3, characterized in that: The specific steps for screw target detection and pose estimation in step S3 include: S31. Diffuse reflection image after reflection compensation The coarse localization bounding box is obtained by using a multi-scale small target detection network. S32. Within the coarse positioning bounding box, sub-pixel edge extraction and least-squares polygon fitting are used to extract the geometric features of the screw head, and sub-pixel interpolation is used to obtain the screw center. ; S33. Using the reflection-compensated screw image and the screw head geometric model, establish a 2D–3D corresponding point set and solve for the initial pose. ; S34. Optimize the initial pose using contour-based least squares. The objective function is to minimize the sum of pixel reprojection error and depth registration error: in Represents the three-dimensional points of the template. Indicates the corresponding pixel coordinates. Represents the camera projection function. This represents the weighting coefficient.

5. The image recognition-based auxiliary positioning method for automotive screw installation according to claim 4, characterized in that: The pose control particle swarm algorithm is based on the real-time specular distortion index. With axial impedance stability index An adaptive particle swarm optimization-driven PID parameter optimization framework is established, wherein the pose-controlled particle swarm algorithm includes: S51. Set state determination rules, and determine whether to proceed to step S52 based on the state determination rules; S52. A nonlinear cooperative modulation mechanism is constructed based on a coupling adjustment function with dual feature inputs. The function outputs the cooperative risk entropy by calculating the nonlinear product term of the specular reflection distortion index and the axial impedance stability. ; S53. Configure an adaptive particle swarm optimization engine by deploying the particle swarm optimization algorithm as the core controller, and simultaneously establish a parameter adaptive channel to output the collaborative risk entropy from step S52. Inertia weights mapped to the particle swarm optimization algorithm and mutation probability ; S54. Represent the PID parameter set as particles, perform PSO updates based on online evaluation indicators, output the pose compensation command of the robotic arm and the real-time closed-loop tightening control of the PID control parameters, and perform parameter rollback or enter force-dominated compliant control when unstable control or over-limit is detected.

6. The image recognition-based auxiliary positioning method for automotive screw installation according to claim 5, characterized in that: The state determination rule includes: acquiring torque signal. Its derivative And visual measurements including axial gap and micro-deformation measurement ; When satisfied When the engagement state is determined, the engagement state is entered. Indicates the threshold for sudden change in contact stiffness. This indicates the preset thread engagement depth threshold. When satisfied When the tightening is stopped, it is determined that the tightening has ended. Indicates the target torque value. Indicates the maximum permissible rotation angle. Indicates the tool's angular velocity; If the tightening process meets the mechanical termination condition but the visual inspection system simultaneously reports the presence of residual gaps, it is considered as follows: If the risk of misalignment is detected, S52 will be executed.

7. The image recognition-based auxiliary positioning method for automotive screw installation according to claim 5, characterized in that: The adaptive particle swarm optimization engine is configured as follows: an inertial weight and a cooperative risk entropy are established. Positive correlation mapping Simultaneously establish mutation probability Positive correlation mapping with collaborative risk entropy ; When the collaborative risk entropy exceeds its preset upper limit due to the dual deterioration of the input specular reflection distortion index and axial impedance stability, it is determined to be a high-risk state; the inertial weight and the mutation probability are simultaneously locked to their respective preset upper limits.

8. The image recognition-based auxiliary positioning method for automotive screw installation according to claim 5, characterized in that: The axial impedance stability index The axial resistance stability is obtained by acquiring the torque data sequence and Z-axis depth data sequence within the current time window. First, the differential value of torque with respect to Z-axis depth is calculated to obtain the instantaneous contact stiffness. Second, the absolute value of the deviation between the instantaneous contact stiffness and the preset standard thread engagement stiffness curve is calculated. Finally, the variance of the torque data sequence is calculated, and this variance is used to normalize the absolute value of the deviation to obtain the axial resistance stability. .

9. The image recognition-based auxiliary positioning method for automotive screw installation according to claim 5, characterized in that: The construction formula for the nonlinear cooperative modulation mechanism is expressed as follows: in, Indicates the single-factor coefficient. Represents the coupling coefficient. This indicates the exponential enhancement factor.

10. An image recognition-based auxiliary positioning system for automotive screw installation, used to execute the image recognition-based auxiliary positioning method for automotive screw installation as described in any one of claims 1-9, characterized in that: The system includes: The multi-modal data acquisition module is configured to deploy and calibrate a combination of multi-modal sensors at the assembly station. The reflection compensation module is configured to perform specular reflection detection and reflection compensation on the metal surface based on the multimodal acquired image, and obtain a reflection-compensated screw image by using a reflection compensation strategy. The pose estimation module is configured to perform screw target detection and pose estimation based on the reflection-compensated screw image to obtain the screw head center coordinates, axial direction and deflection angle. The visual coarse positioning module is configured to perform coordinate transformation and initial trajectory planning on the identified screw pose and tightening tool pose, and the robot performs the insertion action according to the visual coarse positioning. The pose control module is configured to use a pose control particle swarm algorithm to fuse torque sensor data and visual microscopic measurements in real time during insertion and tightening to determine the state and control the tightening tool. The data logging module is configured to record key events in the tightening process and generate quality assessment results.

Images