3D curved surface adaptive dispensing guide method and system, storage medium and device

Abstract

The application discloses a 3D curved surface adaptive dispensing guiding method and system, a storage medium and a device. The method comprises the following steps: acquiring an acquisition image under different color light sources, wherein the acquisition image comprises a white light image, a blue light image and a red light image; extracting an image feature of the white light image, and matching the image feature with a preset CAD model rendering view feature to obtain a coarse positioning result; performing adaptive structured light three-dimensional reconstruction, point cloud registration and trajectory correction based on a preset algorithm and the coarse positioning result to obtain an execution trajectory; and guiding a dispensing operation of an industrial robot based on the execution trajectory, and performing closed-loop control on process parameters. The application realizes full-process adaptive guiding of dispensing on a complex curved surface, improves three-dimensional reconstruction accuracy and stability through a multi-modal vision architecture of 'global positioning + follow-up tracking', realizes dynamic compensation of a three-dimensional trajectory and visual closed-loop control of process parameters, significantly improves glue strip consistency and quality, and is suitable for small-batch and multi-variety intelligent manufacturing.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically, to a 3D curved surface adaptive dispensing guidance method, system, storage medium, and device. Background Technology

[0002] In modern manufacturing, dispensing is a crucial step in electronic packaging, product assembly, and sealing, and its quality directly affects product reliability, waterproof and dustproof performance, and mechanical strength. As product designs become increasingly complex, dispensing paths are no longer limited to simple two-dimensional planes but frequently appear on three-dimensional freeform surfaces, such as the bonding of smartphone frames to glass, the sealing of automotive headlight lenses, and component enclosures on PCBs.

[0003] Traditional dispensing methods using mechanical templates or offline programming can no longer meet the demands of highly flexible and precise production. This is mainly reflected in the following aspects: First, incoming workpieces have dimensional tolerances and assembly positioning errors, and fixed trajectories cannot adaptively compensate for these deviations, leading to glue leakage or overflow; second, offline programming of complex curved surface trajectories is time-consuming and labor-intensive, and cannot cope with real-time changes in the posture of workpieces on the production line; finally, the adhesive itself has rheological properties, and its accumulation pattern is affected by multiple factors such as dispensing speed, trajectory curvature, and workpiece surface tilt angle when moving in three-dimensional space, making it difficult to guarantee the consistency and quality of the adhesive strip with fixed process parameters. Summary of the Invention

[0004] The purpose of this invention is to provide a 3D curved surface adaptive dispensing guidance method, system, storage medium, and device to solve key problems in the prior art, such as the inability of static guidance schemes to cope with online deformation and colloidal rheological interference, the weak global perception capability of dynamic tracking schemes and their reliance on high-precision preset trajectories, poor imaging stability for multiple materials and highly reflective surfaces, and the lack of closed-loop control due to the disconnect between process parameters and visual perception.

[0005] The first aspect of this invention provides a 3D curved surface adaptive dispensing guidance method, comprising the following steps:

[0006] Acquire images under different color light sources, including white light images, blue light images, and red light images;

[0007] The image features of the white light image are extracted and matched with the preset CAD model rendering view features to obtain a coarse positioning result;

[0008] Based on the preset algorithm and the coarse localization results, adaptive structured light 3D reconstruction, point cloud registration, and trajectory correction are performed to obtain the execution trajectory.

[0009] The industrial robot's dispensing operation is guided in real time based on the execution trajectory, and the process parameters are controlled in a closed loop.

[0010] In this solution, the method further includes offline processing to generate the CAD model rendering view features, specifically including:

[0011] The calibration mapping relationship includes hand-eye calibration, global camera calibration, and laser contour sensor calibration, among which,

[0012] The hand-eye calibration is used to calibrate a preset laser contour sensor. and global shutter camera Industrial robot flange coordinate system Transformation relationship between ;

[0013] The global camera calibration is used to calibrate the global camera coordinate system. With world coordinate system Transformation relationship between and camera intrinsic parameter matrix and distortion coefficient;

[0014] The laser contour sensor is used to calibrate the pixel coordinates of the laser line in the global shutter camera image. The corresponding three-dimensional point coordinates in the laser sensor coordinate system Based on the following relationship, a pixel-3D mapping table is obtained. ;

[0015] Import the workpiece CAD model, plan the theoretical dispensing trajectory on the model surface, and associate each point on the trajectory with the initial set of process parameters to obtain the rendering view features of the CAD model.

[0016] In this solution, acquiring images under different color light sources, including white light images, blue light images, and red light images, specifically includes:

[0017] The preset ring light source is controlled to emit red light, blue light, and white light in sequence;

[0018] The system then sequentially acquires three images based on a pre-set color area scan camera, specifically including a white light image. Blue light images and red light image .

[0019] In this solution, the step of extracting the image features of the white light image and matching them with preset CAD model rendering view features to obtain a coarse localization result specifically includes:

[0020] The image features are obtained by extracting features from the white light image based on scale-invariant feature transformation or ORB feature transformation.

[0021] The image features are matched with the features of the rendered view of the CAD model, and the approximate pose of the workpiece is estimated using a perspective n-point algorithm. This serves as the coarse positioning result.

[0022] In this scheme, the step of obtaining the execution trajectory by performing adaptive structured light 3D reconstruction, point cloud registration, and trajectory correction based on a preset algorithm and the coarse localization results specifically includes:

[0023] Based on the coarse positioning results and workpiece material, the grating intensity and encoding strategy are adjusted, and a preset multi-frequency phase unwrapping optimization algorithm is executed to calculate the absolute phase to obtain the workpiece surface point cloud. ;

[0024] Point cloud of the workpiece surface Feature-based iterative nearest-point registration is performed with the CAD model point cloud to solve for the pose transformation matrix. ;

[0025] pose transformation matrix Applied to theoretical dispensing trajectory Obtain the globally corrected trajectory This serves as the execution trajectory.

[0026] In this solution, the dispensing operation of the industrial robot, guided in real-time by the execution trajectory, and the closed-loop control of the process parameters, specifically include:

[0027] Before the dispensing nozzle expels the adhesive, a laser contour sensor continuously scans the workpiece surface at a preset position to obtain a real-time three-dimensional contour line. ;

[0028] Based on the three-dimensional contour line Combined with the execution trajectory The trajectory is matched with the theoretical contour line to obtain real-time trajectory fine-tuning instructions through dynamic correction.

[0029] Based on the laser line image captured by the global shutter camera, the adhesive strip is segmented and its three-dimensional size is restored to obtain the real-time size of the adhesive strip;

[0030] Based on the real-time dimensions of the adhesive strip and a preset rule base, the process parameters are adaptively adjusted to obtain the adjusted process parameter instructions.

[0031] A second aspect of the present invention also provides a 3D curved surface adaptive dispensing guidance system, including a memory and a processor. The memory includes a 3D curved surface adaptive dispensing guidance method program, which, when executed by the processor, performs the following steps:

[0032] Acquire images under different color light sources, including white light images, blue light images, and red light images;

[0033] The image features of the white light image are extracted and matched with the preset CAD model rendering view features to obtain a coarse positioning result;

[0034] Based on the preset algorithm and the coarse localization results, adaptive structured light 3D reconstruction, point cloud registration, and trajectory correction are performed to obtain the execution trajectory.

[0035] The industrial robot's dispensing operation is guided in real time based on the execution trajectory, and the process parameters are controlled in a closed loop.

[0036] In this solution, the method further includes offline processing to generate the CAD model rendering view features, specifically including:

[0037] The calibration mapping relationship includes hand-eye calibration, global camera calibration, and laser contour sensor calibration, among which,

[0038] The hand-eye calibration is used to calibrate a preset laser contour sensor. and global shutter camera Industrial robot flange coordinate system Transformation relationship between ;

[0039] The global camera calibration is used to calibrate the global camera coordinate system. With world coordinate system Transformation relationship between and camera intrinsic parameter matrix and distortion coefficient;

[0040] The laser contour sensor is used to calibrate the pixel coordinates of the laser line in the global shutter camera image. The corresponding three-dimensional point coordinates in the laser sensor coordinate system Based on the following relationship, a pixel-3D mapping table is obtained. ;

[0041] Import the workpiece CAD model, plan the theoretical dispensing trajectory on the model surface, and associate each point on the trajectory with the initial set of process parameters to obtain the rendering view features of the CAD model.

[0042] In this solution, acquiring images under different color light sources, including white light images, blue light images, and red light images, specifically includes:

[0043] The preset ring light source is controlled to emit red light, blue light, and white light in sequence;

[0044] The system then sequentially acquires three images based on a pre-set color area scan camera, specifically including a white light image. Blue light images and red light image .

[0045] In this solution, the step of extracting the image features of the white light image and matching them with preset CAD model rendering view features to obtain a coarse localization result specifically includes:

[0046] The image features are obtained by extracting features from the white light image based on scale-invariant feature transformation or ORB feature transformation.

[0047] The image features are matched with the features of the rendered view of the CAD model, and the approximate pose of the workpiece is estimated using a perspective n-point algorithm. This serves as the coarse positioning result.

[0048] In this scheme, the step of obtaining the execution trajectory by performing adaptive structured light 3D reconstruction, point cloud registration, and trajectory correction based on a preset algorithm and the coarse localization results specifically includes:

[0049] Based on the coarse positioning results and workpiece material, the grating intensity and encoding strategy are adjusted, and a preset multi-frequency phase unwrapping optimization algorithm is executed to calculate the absolute phase to obtain the workpiece surface point cloud. ;

[0050] Point cloud of the workpiece surface Feature-based iterative nearest-point registration is performed with the CAD model point cloud to solve for the pose transformation matrix. ;

[0051] pose transformation matrix Applied to theoretical dispensing trajectory Obtain the globally corrected trajectory This serves as the execution trajectory.

[0052] In this solution, the dispensing operation of the industrial robot, guided in real-time by the execution trajectory, and the closed-loop control of the process parameters, specifically include:

[0053] Before the dispensing nozzle expels the adhesive, a laser contour sensor continuously scans the workpiece surface at a preset position to obtain a real-time three-dimensional contour line. ;

[0054] Based on the three-dimensional contour line Combined with the execution trajectory The trajectory is matched with the theoretical contour line to obtain real-time trajectory fine-tuning instructions through dynamic correction.

[0055] Based on the laser line image captured by the global shutter camera, the adhesive strip is segmented and its three-dimensional size is restored to obtain the real-time size of the adhesive strip;

[0056] Based on the real-time dimensions of the adhesive strip and a preset rule base, the process parameters are adaptively adjusted to obtain the adjusted process parameter instructions.

[0057] A third aspect of the present invention provides a computer-readable storage medium comprising a machine program for a 3D surface adaptive dispensing guidance method, wherein when executed by a processor, the 3D surface adaptive dispensing guidance method program implements the steps of a 3D surface adaptive dispensing guidance method as described in any of the preceding claims.

[0058] A fourth aspect of the present invention provides a 3D curved surface adaptive dispensing guide device, comprising:

[0059] The module consists of an optical imaging module, a control module, and an execution module.

[0060] The optical imaging module includes a color area array camera and a ring light source. The color area array camera is used to acquire workpiece images, and the ring light source is used to provide illumination.

[0061] The control module includes an industrial computer, used to implement the steps of a 3D curved surface adaptive dispensing guidance method;

[0062] The execution module includes an industrial robot, as well as a dispensing valve and a follow-up vision unit installed at the end of the industrial robot. The follow-up vision unit includes a laser profile sensor, a global shutter camera, and a coaxial point light source.

[0063] In this solution, the laser line plane of the laser contour sensor is perpendicular to the movement direction of the industrial robot, and is used to scan the three-dimensional contour line of the workpiece surface at the valve nozzle of the dispensing valve in real time; the optical axis of the global shutter camera forms a preset angle with the laser line plane, and is used to acquire laser line images; the coaxial point light source provides supplementary illumination for the global shutter camera.

[0064] The 3D curved surface adaptive dispensing guidance method, system, storage medium, and device disclosed in this invention have the following beneficial effects:

[0065] 1. Achieved full-dimensional, end-to-end visual guidance: Through a multimodal architecture of "global vision (positioning) + follow-up vision (tracking + detection)," it overcomes the limitations of a single vision mode. Global vision solves the overall workpiece positioning problem and compensates for the shortcomings of pure laser tracking. The static solution overcomes the weakness of planar perception; the dynamic vision system enables dynamic compensation and online quality inspection, making up for the inability of the static solution to cope with process changes. The combination of the two achieves seamless, multi-dimensional guidance from "before the start" to "during the process".

[0066] 2. Improved robustness and accuracy of 3D reconstruction of complex surfaces: The proposed optimization algorithm based on multispectral illumination selection and multi-frequency phase unwrapping can adaptively handle difficult-to-measure surfaces such as high reflectivity and light absorption, effectively suppress phase jumps and noise, and obtain more complete and accurate initial point cloud data, laying a solid foundation for high-precision trajectory correction.

[0067] 3. Enhanced intelligence and stability of dynamic trajectory tracking: Look-ahead line laser scanning combined with a curvature-feature-based contour matching algorithm not only provides high-precision tracking but also... It can compensate for the trajectory in the horizontal direction and effectively sense and compensate for the trajectory in the horizontal direction. It achieves true three-dimensional dynamic tracking by minimizing deviations in direction, which has a fundamental advantage over traditional one-dimensional laser tracking;

[0068] 4. A new paradigm of vision-based closed-loop control for adhesive strip quality has been pioneered: The major innovation of this invention is the combination of online visual measurement of adhesive strip dimensions with real-time adjustment of process parameters (speed, pressure / time) in a closed loop. By mapping two-dimensional image information back to a known three-dimensional surface, non-contact online estimation of the three-dimensional dimensions of the adhesive strip is achieved. Based on this, the fuzzy PID control can actively adapt to changes in the flow state of the adhesive caused by changes in the curvature and tilt angle of the surface, fundamentally ensuring the consistency of the adhesive strip molding quality, upgrading from "correct trajectory" to "high-quality result".

[0069] 5. Improved overall system flexibility and application range: It has a strong adaptive capability to workpiece positioning errors, incoming material deformation, surface material, and even minor disturbances in the production process. It can handle more types of products without hardware modification, shortening the changeover and machine adjustment time. It is particularly suitable for flexible manufacturing scenarios with small batches and multiple varieties. Attached Figure Description

[0070] Figure 1 This diagram illustrates the steps of a 3D curved surface adaptive dispensing guidance method according to the present invention.

[0071] Figure 2 A schematic diagram illustrating the dynamic trajectory correction principle of a 3D curved surface adaptive dispensing guidance method according to the present invention is shown.

[0072] Figure 3 A schematic diagram of the visual closed-loop control logic for process parameters of a 3D curved surface adaptive dispensing guidance method according to the present invention is shown.

[0073] Figure 4 A block diagram of a 3D curved surface adaptive dispensing guidance system according to the present invention is shown. Detailed Implementation

[0074] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments of the present invention and the features thereof can be combined with each other.

[0075] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0076] Among these, using machine vision for online guidance and adjustment has become an inevitable choice for achieving high-precision 3D dispensing. Machine vision systems acquire the position, orientation, and three-dimensional shape information of the workpiece in real time through non-contact measurement, and dynamically adjust the robot's motion trajectory and dispensing process parameters accordingly. This compensates for manufacturing errors and changes in working conditions. Currently, the most similar existing solutions mainly fall into two categories: static 3D guidance solutions based on monocular structured light and dynamic tracking solutions based on laser displacement sensors.

[0077] Specifically, 1. A static 3D guidance scheme based on monocular structured light: This system typically consists of a high-resolution industrial camera, a digital light processing (DLP) or liquid crystal display (LCD) projector, and a precision motion axis system (or robot). Its workflow is as follows: First, the system performs initial calibration on the tray or fixture without a workpiece, establishing a world coordinate system. Then, after the workpiece to be dispensed is placed in position, the projector projects a series of specifically coded grating patterns (such as Gray code or phase-shifted fringes) onto the workpiece surface, while the camera simultaneously acquires grating images deformed by the height modulation of the workpiece surface. Through phase resolution and 3D reconstruction algorithms, the dense 3D point cloud data of the workpiece surface is recovered. Subsequently, the reconstructed point cloud is matched with a preset workpiece CAD model (such as using an iterative nearest-point algorithm) to calculate the deviation between the actual and ideal positions of the workpiece (i.e., the six-degree-of-freedom pose transformation matrix). Finally, the transformation matrix is ​​applied to the preset ideal dispensing trajectory to generate a corrected trajectory program that the robot can execute, guiding the robot to complete the dispensing operation in one go.

[0078] The core of the mathematical model for this scheme is the phase-height mapping relationship. For the phase-shifting method, projection... Amplitude-phase difference is The sinusoidal fringes are represented by the light intensity distribution captured by the camera as follows:

[0079] ;

[0080] in, Background light intensity, For modulation intensity, Given the absolute phase to be solved, the wrapping phase can be solved using the least squares method:

[0081] ;

[0082] Then, phase expansion is performed using Gray code or other methods to obtain the absolute phase. The phase is mapped to three-dimensional coordinates using parameters obtained from system calibration. :

[0083] ;

[0084] in, Image pixel coordinates, It is a projection matrix that includes camera intrinsic parameters, extrinsic parameters, and phase-height transformation relationships.

[0085] 2. Dynamic Tracking Scheme Based on Laser Displacement Sensors: This system is typically integrated into the robot's end effector (dispensing valve), and its core is one or more laser displacement sensors (often employing the triangulation principle). During the dispensing process, the robot moves along a preset approximate trajectory, and the laser sensor continuously measures the distance to the workpiece surface directly below its probe. By comparing the real-time measured distance value with the coordinates of the center of the robot's end flange... Combined, and the fixed transformation of the sensor and flange is known. It can calculate the three-dimensional coordinates of the workpiece surface where the sensor spot is located in real time. :

[0086] ;

[0087] These real-time acquired surface points are fitted into a spatial curve and compared with a preset theoretical curve. The controller then adjusts the curve based on the deviation. The robot's position in the normal direction is dynamically adjusted using control algorithms such as PID control. The position of the dispensing nozzle (or the tangential speed) can be adjusted to ensure that the nozzle always maintains a constant stand-off distance from the curved surface, thereby achieving uniform adhesion of the adhesive strip on the curved surface.

[0088] Through in-depth analysis of the aforementioned existing technical solutions, although these existing technologies have been applied to a certain extent, they still have the following inherent drawbacks, which limit their application in more demanding scenarios:

[0089] Disadvantage 1: Static guidance schemes cannot handle "online" deformation and colloidal rheological interference. Monocular structured light-based schemes operate on a "measure first, execute later" static model. While it completes 3D reconstruction and trajectory planning before dispensing begins, it cannot address dynamic issues that arise during the dispensing process. For example, for flexible or thin-walled workpieces (such as FPC flexible circuit boards), the contact of the dispensing nozzle or the heat of the colloidal material can cause minute deformations; the colloidal material's own rheological properties (such as thixotropy and shear thinning) can lead to discrepancies between the actual adhesive strip cross-section and the expected result. Static schemes are "blind" to such process changes, potentially causing quality issues.

[0090] Disadvantage 2: The dynamic tracking scheme has weak global perception capability and relies on a high-precision preset trajectory. Laser tracking is essentially a "one-dimensional" measurement; it can only acquire distance information within a very small area where the sensor spot is located, lacking visual perception of the entire workpiece features and the global trajectory. Therefore, it heavily depends on the initial accuracy of the robot's preset trajectory. If the initial trajectory deviates entirely due to workpiece positioning errors, the laser sensor can only perform local height tracking on its scanning path and cannot correct the trajectory on the horizontal plane. The overall offset within the adhesive strip may cause the entire adhesive strip to deviate from the target bonding surface.

[0091] Disadvantage 3: Poor imaging stability on multi-material, highly reflective surfaces. Existing solutions lack robustness when dealing with complex surfaces. Structured light solutions are prone to phase information saturation or attenuation on highly reflective (e.g., metallic luster) or light-absorbing (e.g., black rubber) surfaces, leading to missing point clouds or significant noise. Laser triangulation is prone to specular reflection and signal loss at smooth slopes or edges. This limits its application in production lines for automotive, consumer electronics, and other industries with diverse material workpieces.

[0092] Disadvantage 4: The process parameters are disconnected from visual perception, lacking closed-loop control. Existing solutions mainly focus on correcting the "trajectory," while neglecting to address dispensing process parameters (such as dispensing pressure). Valve opening time, movement speed This is considered an independent, pre-defined constant. In reality, on a three-dimensional curved surface, to ensure the cross-sectional area of ​​the adhesive strip... While the trajectory curvature or surface tilt angle remains constant, dynamic adjustments are required. and The existing system lacks real-time visual feedback-based measurement of the adhesive strip quality (width). ,high Online detection and closed-loop control mechanism for process parameters.

[0093] To address these shortcomings, the present invention aims to overcome the deficiencies of the prior art and provide an intelligent dispensing guidance system and method that combines global three-dimensional perception with local dynamic tracking capabilities, and enables visual closed-loop feedback control of process parameters. Specific objectives include:

[0094] 1. Achieve high-precision, robust, full-field 3D positioning and reconstruction of the workpiece before dispensing;

[0095] 2. Achieve real-time, full-dimensional (six degrees of freedom) dynamic tracking and compensation of the adhesive strip trajectory during the dispensing process;

[0096] 3. Achieve self-adaptation to the surface material and reflective properties of the workpiece, thereby improving the stability of the system under complex working conditions;

[0097] 4. Achieve online visual inspection of the adhesive strip forming quality and dynamically adjust dispensing process parameters accordingly to form a closed-loop quality control system. The core innovation and key protection points are:

[0098] 1. Multimodal vision system architecture: Protects a hardware architecture for a 3D dispensing guidance system consisting of a fixed-mount global vision module (for initial 3D positioning of the workpiece) and a composite vision module (integrating a line laser sensor for look-ahead 3D contour tracking and a miniature camera for adhesive strip quality inspection) mounted on the robot end effector.

[0099] 2. Adaptive Multi-Frequency Phase Unwrapping Optimization Method: This paper protects a phase solution method for structured light 3D reconstruction that optimizes phase resolution by introducing random phase perturbation and multi-frequency weighted fusion, combined with total variational regularization. Specifically, its mathematical optimization model is described below.

[0100] ;

[0101] in, For regularization terms (such as the TV norm). For adaptive weights based on image quality, The core of this method lies in improving the robustness of reconstruction of complex optical surfaces.

[0102] 3. A dynamic trajectory correction method based on look-ahead contour and curvature matching: This method utilizes a line laser sensor to acquire the workpiece contour during dispensing. By matching the geometric features (especially curvature features) of the measured and predicted contours, it calculates the robot end effector's trajectory in multiple degrees of freedom (at least including...). An algorithm for real-time compensation of translational distances. Its core lies in expanding one-dimensional distance information into a three-dimensional spatial curve matching problem;

[0103] 4. Online Visual Measurement and Closed-Loop Control Method for 3D Dimensions of Adhesive Strips: This invention protects a method that combines 2D image information of adhesive strips with a known 3D surface model of the workpiece, and estimates the actual width and height of the adhesive strips through projection back-calculation and geometric modeling. Further, it protects a closed-loop control system and method based on this measurement result, which uses intelligent control algorithms (such as fuzzy PID) to adjust the robot's movement speed and dispensing parameters in real time.

[0104] 5. Integrated workflow of "global positioning - follow-up tracking - quality closed loop": Protects the complete workflow and control logic implemented by the above hardware and methods, including the three stages of offline preparation, online global positioning, real-time follow-up guidance and closed-loop control.

[0105] Specifically, Figure 1 The diagram illustrates the steps of a 3D curved surface adaptive dispensing guidance method according to the present invention.

[0106] like Figure 1 As shown, this invention discloses a 3D curved surface adaptive dispensing guidance method, comprising the following steps:

[0107] S102, acquire images under different color light sources, including white light images, blue light images and red light images;

[0108] S104, Extract the image features of the white light image and match them with the preset CAD model rendering view features to obtain a coarse positioning result;

[0109] S106, Based on the preset algorithm and the coarse positioning results, adaptive structured light 3D reconstruction, point cloud registration and trajectory correction are performed to obtain the execution trajectory;

[0110] S108, based on the execution trajectory, guide the industrial robot's dispensing operation in real time and perform closed-loop control of process parameters.

[0111] It should be noted that, in this embodiment, a benchmark is established through system calibration and theoretical trajectory generation in the offline preparation stage. Then, in the online global positioning stage, high-precision positioning of the workpiece is achieved by using multispectral images and adaptive structured light 3D reconstruction, and the theoretical trajectory is corrected by point cloud registration. In the real-time follow-up guidance and closed-loop control stage, dynamic trajectory compensation is achieved by looking-ahead laser contour scanning and curvature matching. At the same time, based on visual inspection of adhesive strip images and closed-loop control of process parameters, the dispensing speed and dispensing parameters are adjusted in real time, thereby achieving adaptive guidance and quality control of the dispensing process on complex curved surfaces throughout the entire process.

[0112] According to an embodiment of the present invention, the method further includes offline processing to generate the CAD model rendering view features, specifically including:

[0113] The calibration mapping relationship includes hand-eye calibration, global camera calibration, and laser contour sensor calibration, among which,

[0114] The hand-eye calibration is used to calibrate a preset laser contour sensor. and global shutter camera Industrial robot flange coordinate system Transformation relationship between ;

[0115] The global camera calibration is used to calibrate the global camera coordinate system. With world coordinate system Transformation relationship between and camera intrinsic parameter matrix and distortion coefficient;

[0116] The laser contour sensor is used to calibrate the pixel coordinates of the laser line in the global shutter camera image. The corresponding three-dimensional point coordinates in the laser sensor coordinate system Based on the following relationship, a pixel-3D mapping table is obtained. ;

[0117] Import the workpiece CAD model, plan the theoretical dispensing trajectory on the model surface, and associate each point on the trajectory with the initial set of process parameters to obtain the rendering view features of the CAD model.

[0118] It should be noted that, in this embodiment, the offline preparation phase aims to establish all necessary models and mapping relationships, wherein the hand-eye calibration specifically calibrates the laser profile sensor. and global shutter camera Coordinate system with robot end flange Transformation relationship between Specifically, this can be accomplished using the standard hand-eye calibration board method; the global camera calibration is used to calibrate the global camera coordinate system. With world coordinate system (usually with the robot's base coordinate system) Transformation relationship between alignment and camera intrinsic parameter matrix and distortion coefficients; the laser contour sensor is used to calibrate the pixel coordinates of the laser line in the global shutter camera image. The corresponding three-dimensional point coordinates in the laser sensor coordinate system Based on the following relationship, a pixel-3D mapping table is obtained. The laser profile sensor is calibrated by scanning a standard step block to calibrate the global shutter camera. Laser line pixel coordinates in the image Its corresponding 3D point coordinates in the laser sensor coordinate system The relationship below, that is, to obtain its inherent "pixel-3D" mapping table. .

[0119] Furthermore, in this embodiment, a workpiece CAD model is imported, and a theoretical dispensing trajectory is planned on the model surface. Each point contains three-dimensional coordinates and the surface normal vector at that point. Associating an initial set of process parameters with each point on the trajectory. Based on the above, the rendering view features of the CAD model are obtained.

[0120] According to an embodiment of the present invention, the acquisition of images under different color light sources, including white light images, blue light images, and red light images, specifically includes:

[0121] The preset ring light source is controlled to emit red light, blue light, and white light in sequence;

[0122] The system then sequentially acquires three images based on a pre-set color area scan camera, specifically including a white light image. Blue light images and red light image .

[0123] It should be noted that, in this embodiment, in practical application, the preset ring LED light source is controlled to emit red light, blue light, and white light sequentially, and three images are simultaneously acquired based on a preset color area array camera, which correspond to the white light image. Blue light images and red light image .

[0124] According to an embodiment of the present invention, the step of extracting image features from the white light image and matching them with preset CAD model rendering view features to obtain a coarse localization result specifically includes:

[0125] The image features are obtained by extracting features from the white light image based on scale-invariant feature transformation or ORB feature transformation.

[0126] The image features are matched with the features of the rendered view of the CAD model, and the approximate pose of the workpiece is estimated using a perspective n-point algorithm. This serves as the coarse positioning result.

[0127] It should be noted that, in this embodiment, the image features are obtained by extracting features from the white light image based on scale-invariant feature transform or ORB feature transform. ORB corresponds to "FAST keypoints with orientation + rotation-invariant BRIEF descriptor," which is used in this embodiment. Then, the image features are matched with the features of the CAD model rendering view, and the approximate pose of the workpiece is estimated using a perspective n-point algorithm. This serves as the coarse positioning result.

[0128] According to an embodiment of the present invention, the step of obtaining the execution trajectory by adaptive structured light 3D reconstruction, point cloud registration, and trajectory correction based on a preset algorithm and the coarse localization results specifically includes:

[0129] Based on the coarse positioning results and workpiece material, the grating intensity and encoding strategy are adjusted, and a preset multi-frequency phase unwrapping optimization algorithm is executed to calculate the absolute phase to obtain the workpiece surface point cloud. ;

[0130] Point cloud of the workpiece surface Feature-based iterative nearest-point registration is performed with the CAD model point cloud to solve for the pose transformation matrix. ;

[0131] pose transformation matrix Applied to theoretical dispensing trajectory Obtain the globally corrected trajectory This serves as the execution trajectory.

[0132] It should be noted that, in this embodiment, the intensity and encoding strategy of the projection grating are intelligently adjusted based on the coarse positioning results and known workpiece material information (which can be read from the material system). For example, for highly reflective areas, a low-intensity, multi-spectral speckle pattern is used instead of standard sinusoidal fringes to avoid overexposure. An improved multi-frequency phase unwrapping algorithm is executed. Traditional methods use two frequencies; this invention introduces a third intermediate frequency to form a frequency sequence. ,in, Provides high precision, Provides a wide range of unambiguous definitions of the total phase. ,in, For height, To combat random phase perturbations introduced by highlights, a more robust absolute phase is obtained by solving the following optimization problem. : ,in, For the wrapper operator, For weights that are adaptive based on the image signal-to-noise ratio (SNR), This is a total variation regularization term used to smooth noise while preserving edges. The regularization coefficient is determined using calibration parameters. Convert to a complete workpiece surface point cloud .

[0133] Furthermore, in this embodiment, Feature-based iterative nearest-neighbor registration is performed between the point cloud of the CAD model and the model. Instead of using all points, only points located near a preset dispensing trajectory are selected and given higher weights. Simultaneously, the 2D texture feature points extracted in the first stage are used as corresponding constraints to jointly solve for a high-precision pose transformation matrix. and will Applied to theoretical trajectories Generate a globally corrected robot execution trajectory And distribute it to the robot.

[0134] According to an embodiment of the present invention, the dispensing operation of the industrial robot based on the real-time follow-up guidance of the execution trajectory, and the closed-loop control of the process parameters, specifically includes:

[0135] Before the dispensing nozzle expels the adhesive, a laser contour sensor continuously scans the workpiece surface at a preset position to obtain a real-time three-dimensional contour line. ;

[0136] Based on the three-dimensional contour line Combined with the execution trajectory The trajectory is matched with the theoretical contour line to obtain real-time trajectory fine-tuning instructions through dynamic correction.

[0137] Based on the laser line image captured by the global shutter camera, the adhesive strip is segmented and its three-dimensional size is restored to obtain the real-time size of the adhesive strip;

[0138] Based on the real-time dimensions of the adhesive strip and a preset rule base, the process parameters are adaptively adjusted to obtain the adjusted process parameter instructions.

[0139] It should be noted that, in this embodiment, the robot performs... Simultaneously, millisecond-level dynamic compensation and process adjustments are performed, such as... Figure 2 As shown, this is a schematic diagram of the dynamic trajectory correction principle. Before the dispensing valve nozzle expels the adhesive, the laser contour sensor continuously scans the workpiece surface about 10mm in front of it, forming a real-time three-dimensional contour line. Compare this measured contour line with the one based on The theoretical contour line predicted by the workpiece model The matching algorithm considers the curvature features of the contour and defines the set of points on the contour line as follows: Its curvature is The fine-tuning of the robot in the three translational degrees of freedom at the current moment is solved by minimizing the following objective function. ,in, For the reason and estimated minute rotation amount Transformation of composition For curvature matching weights, the solution is... The high-speed (>1kHz) real-time feedback to the robot enables dynamic trajectory correction, solving the problem that laser tracking solutions can only perform one-dimensional height compensation.

[0140] Furthermore, in this embodiment, the real-time size of the adhesive strip is obtained by segmenting and restoring its three-dimensional dimensions based on the laser line image captured by the global shutter camera. Specifically, while capturing the laser line, the global shutter camera also captures an image of the adhesive strip that has been extruded and formed behind the dispensing valve nozzle at another exposure time. The method combines deep learning and image processing to analyze Adhesive Strip Segmentation: Real-time segmentation of adhesive strip regions in images using a lightweight U-Net network. 3D Dimension Restoration: Given the known 3D coordinates of the workpiece surface in the area where the adhesive strip is located (from laser scanning), the 2D image information of the adhesive strip is "projected" back onto the 3D curved surface using the perspective projection model of the global shutter camera. Specifically, for the segmentation mask... Each pixel in By interpolating the nearest neighbor point found on the laser contour line, the three-dimensional coordinates of the surface corresponding to that pixel can be obtained. And the normal vector. Assuming the cross-section of the adhesive strip is approximately semi-circular, the pixel width perpendicular to the trajectory direction is analyzed. And combined with the calibration scale factor at that location (Related to camera angle and surface normal), the actual width of the adhesive strip can be calculated. and estimated height : ,in, This is the aspect ratio coefficient calibrated through prior experiments.

[0141] Furthermore, in this embodiment, the process parameters are adaptively adjusted based on the real-time size of the adhesive strip and a preset rule base to obtain the adjusted process parameter instructions. Specifically, as follows: Figure 3 The diagram shown illustrates the visual closed-loop control logic for process parameters, where the system presets the target size of the adhesive strip. A dynamic response model of the adhesive strip size and process parameters is established, and the error is defined. A fuzzy PID controller is used to adjust the opening and closing time of the dispensing valve. and the robot's movement speed Example rule base: IF e_W isNegative_Large AND Δe_W is Zero, THEN ΔT_{open} is Positive_Large AND Δvis Negative_Small; After the control law output is smoothed and filtered, it is sent to the dispensing controller and robot speed loop in real time, forming a real-time closed loop of "visual inspection - size calculation - parameter adjustment" to ensure the uniformity of the adhesive strip on the curved surface.

[0142] Figure 4 A block diagram of a 3D curved surface adaptive dispensing guidance system according to the present invention is shown.

[0143] like Figure 4 As shown, this invention discloses a 3D curved surface adaptive dispensing guidance system, including a memory and a processor. The memory includes a 3D curved surface adaptive dispensing guidance method program. When the 3D curved surface adaptive dispensing guidance method program is executed by the processor, it performs the following steps:

[0144] Acquire images under different color light sources, including white light images, blue light images, and red light images;

[0145] The image features of the white light image are extracted and matched with the preset CAD model rendering view features to obtain a coarse positioning result;

[0146] Based on the preset algorithm and the coarse localization results, adaptive structured light 3D reconstruction, point cloud registration, and trajectory correction are performed to obtain the execution trajectory.

[0147] The industrial robot's dispensing operation is guided in real time based on the execution trajectory, and the process parameters are controlled in a closed loop.

[0148] It should be noted that when the 3D curved surface adaptive dispensing guidance system disclosed in this application is applied, the specific process corresponds to the 3D curved surface adaptive dispensing guidance method described in the above embodiments. Since the specific implementation details of the system application are consistent with the content of the above 3D curved surface adaptive dispensing guidance method, no further details will be provided in this embodiment.

[0149] A third aspect of the present invention provides a computer-readable storage medium comprising a 3D surface adaptive dispensing guidance method program, wherein when the 3D surface adaptive dispensing guidance method program is executed by a processor, it implements the steps of a 3D surface adaptive dispensing guidance method as described in any of the preceding claims.

[0150] A fourth aspect of the present invention provides a 3D curved surface adaptive dispensing guide device, comprising:

[0151] The module consists of an optical imaging module, a control module, and an execution module.

[0152] The optical imaging module includes a color area array camera and a ring light source. The color area array camera is used to acquire workpiece images, and the ring light source is used to provide illumination.

[0153] The control module includes an industrial computer, used to implement the steps of the 3D curved surface adaptive dispensing guidance method described above.

[0154] The execution module includes an industrial robot, as well as a dispensing valve and a follow-up vision unit installed at the end of the industrial robot. The follow-up vision unit includes a laser profile sensor, a global shutter camera, and a coaxial point light source.

[0155] It should be noted that, in this embodiment, the laser line plane of the laser contour sensor is perpendicular to the movement direction of the industrial robot, and is used to scan the three-dimensional contour line of the workpiece surface at the valve nozzle of the dispensing valve in real time; the optical axis of the global shutter camera forms a preset angle with the laser line plane, and is used to acquire laser line images; the coaxial point light source provides supplementary illumination for the global shutter camera.

[0156] It should be noted that the entire workflow is sequential and collaborative. Before the dispensing operation begins, the global vision module (color area array camera and ring light source) is activated to complete the workpiece identification, reconstruction, and rough and fine positioning, generating... Once the robot begins to move, the global vision module enters standby mode, the follow-up vision unit starts working, the laser contour sensor scans continuously in front like a "searchlight" to provide the robot with real-time path correction signals, and the global shutter camera monitors the adhesive strip from behind like a "quality inspector" to provide feedback signals for the dispensing process parameters. The robot controller receives dual feedback, adjusting its own trajectory on one hand and the motion parameters of the end tool (dispensing valve) on the other, forming two parallel closed-loop control systems.

[0157] This invention discloses a 3D curved surface adaptive dispensing guidance method, system, storage medium, and device, which realizes full-process adaptive guidance for dispensing on complex curved surfaces. Through a multimodal vision architecture of "global positioning + follow-up tracking", it improves the accuracy and stability of 3D reconstruction, realizes dynamic compensation of 3D trajectory and visual closed-loop control of process parameters, significantly improves the consistency and quality of adhesive strips, and is suitable for small-batch, multi-variety intelligent manufacturing.

[0158] In the several embodiments provided by this invention, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0159] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0160] In addition, in the various embodiments of the present invention, each functional unit can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0161] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0162] Alternatively, if the integrated units of this invention are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

Claims

1. A 3D curved surface adaptive dispensing guidance method, characterized in that, Includes the following steps: Acquire images under different color light sources, including white light images, blue light images, and red light images; The image features of the white light image are extracted and matched with the preset CAD model rendering view features to obtain a coarse positioning result; Based on the preset algorithm and the coarse localization results, adaptive structured light 3D reconstruction, point cloud registration, and trajectory correction are performed to obtain the execution trajectory. The industrial robot's dispensing operation is guided in real time based on the execution trajectory, and the process parameters are controlled in a closed loop.

2. The 3D curved surface adaptive dispensing guiding method according to claim 1, wherein, The method further includes offline processing to generate the rendered view features of the CAD model, specifically including: The calibration mapping relationship includes hand-eye calibration, global camera calibration, and laser contour sensor calibration, among which, The hand-eye calibration is used for calibrating a preset laser profile sensor and global shutter cameras with an industrial robot flange coordinate system between the two ; The global camera calibration is used to calibrate the global camera coordinate system. With world coordinate system Transformation relationship between and camera intrinsic parameter matrix and distortion coefficient; The laser contour sensor is used to calibrate the pixel coordinates of the laser line in the global shutter camera image. The corresponding three-dimensional point coordinates in the laser sensor coordinate system Based on the following relationship, a pixel-3D mapping table is obtained. ; Import the workpiece CAD model, plan the theoretical dispensing trajectory on the model surface, and associate each point on the trajectory with the initial set of process parameters to obtain the rendering view features of the CAD model.

3. The 3D curved surface adaptive dispensing guidance method according to claim 2, characterized in that, The acquisition of images under different color light sources, including white light images, blue light images, and red light images, specifically includes: The preset ring light source is controlled to emit red light, blue light, and white light in sequence; The system then sequentially acquires three images based on a pre-set color area scan camera, specifically including a white light image. Blue light images and red light image .

4. The 3D curved surface adaptive dispensing guidance method according to claim 3, characterized in that, The step of extracting image features from the white light image and matching them with preset CAD model rendering view features to obtain coarse localization results specifically includes: The image features are obtained by extracting features from the white light image based on scale-invariant feature transformation or ORB feature transformation. The image features are matched with the features of the rendered view of the CAD model, and the approximate pose of the workpiece is estimated using a perspective n-point algorithm. This serves as the coarse positioning result.

5. The 3D curved surface adaptive dispensing guidance method according to claim 4, characterized in that, The process of obtaining the execution trajectory by performing adaptive structured light 3D reconstruction, point cloud registration, and trajectory correction based on a preset algorithm and the coarse localization results specifically includes: Based on the coarse positioning results and workpiece material, the grating intensity and encoding strategy are adjusted, and a preset multi-frequency phase unwrapping optimization algorithm is executed to calculate the absolute phase to obtain the workpiece surface point cloud. ; Point cloud of the workpiece surface Feature-based iterative nearest-point registration is performed with the CAD model point cloud to solve for the pose transformation matrix. ; pose transformation matrix Applied to theoretical dispensing trajectory Obtain the globally corrected trajectory This serves as the execution trajectory.

6. The 3D curved surface adaptive dispensing guidance method according to claim 5, characterized in that, The dispensing operation of the industrial robot, guided in real-time by the execution trajectory, and the closed-loop control of process parameters, specifically includes: Before the dispensing nozzle expels the adhesive, a laser contour sensor continuously scans the workpiece surface at a preset position to obtain a real-time three-dimensional contour line. ; Based on the three-dimensional contour line Combined with the execution trajectory The trajectory is matched with the theoretical contour line to obtain real-time trajectory fine-tuning instructions through dynamic correction. Based on the laser line image captured by the global shutter camera, the adhesive strip is segmented and its three-dimensional size is restored to obtain the real-time size of the adhesive strip; Based on the real-time dimensions of the adhesive strip and a preset rule base, the process parameters are adaptively adjusted to obtain the adjusted process parameter instructions.

7. A 3D curved surface adaptive dispensing guidance system, characterized in that, The system includes a memory and a processor. The memory contains a 3D surface adaptive dispensing guidance method program, which, when executed by the processor, performs the following steps: Acquire images under different color light sources, including white light images, blue light images, and red light images; The image features of the white light image are extracted and matched with the preset CAD model rendering view features to obtain a coarse positioning result; Based on the preset algorithm and the coarse localization results, adaptive structured light 3D reconstruction, point cloud registration, and trajectory correction are performed to obtain the execution trajectory. The industrial robot's dispensing operation is guided in real time based on the execution trajectory, and the process parameters are controlled in a closed loop.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a 3D surface adaptive dispensing guidance method program, which, when executed by a processor, implements the steps of a 3D surface adaptive dispensing guidance method as described in any one of claims 1 to 6.

9. A 3D curved surface adaptive dispensing guide device, characterized in that, include: The module consists of an optical imaging module, a control module, and an execution module. The optical imaging module includes a color area array camera and a ring light source. The color area array camera is used to acquire images of the workpiece, and the ring light source is used to provide illumination. The control module includes an industrial computer, used to implement the steps of the 3D curved surface adaptive dispensing guidance method as described in any one of claims 1 to 6; The execution module includes an industrial robot, as well as a dispensing valve and a follow-up vision unit installed at the end of the industrial robot. The follow-up vision unit includes a laser profile sensor, a global shutter camera, and a coaxial point light source.

10. A 3D curved surface adaptive dispensing guide device according to claim 9, characterized in that, The laser line plane of the laser contour sensor is perpendicular to the direction of movement of the industrial robot, and is used to scan the three-dimensional contour line of the workpiece surface at the valve nozzle of the dispensing valve in real time; the optical axis of the global shutter camera forms a preset angle with the laser line plane, and is used to acquire laser line images; the coaxial point light source provides supplementary illumination for the global shutter camera.

Images