A method and system for detecting defects in curved paint spraying using a robotic arm with normal motion.
By using robotic arm normal tracking and 3D point cloud compensation technology, the problem of misjudgment of reference surface in the defect detection of curved paint spraying surfaces was solved, and high-precision defect identification and automated processing were achieved.
Patent Information
- Application Number
- CN202611481645.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-22
- Publication Date
- 2026-11-17
AI Technical Summary
In existing methods for detecting defects in curved paint surfaces, the reference surface is affected by curvature fluctuations, workpiece deviations, and large-scale defects, leading to problems of missed defects and false alarms.
By controlling the projector's optical axis to coincide with the local normal through the normal follow of the robotic arm, and combining the 3D model and the deformed stripe image acquired by the camera, triangulation and point cloud processing are performed to eliminate surface undulations and workpiece deviations. Defect areas are identified by fitting local reference surfaces and connecting clustering.
It enables accurate identification of defects in curved surface painting, reduces missed detections and false alarms, and the output results can be directly used for automatic sanding and touch-up painting, improving detection accuracy and efficiency.
Smart Images

Figure CN123118601A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of machine vision inspection, and in particular to a method and system for detecting defects in curved paint spraying using a robotic arm with normal motion. Background Technology
[0002] After painting, the paint surface of automobile bodies, rail vehicle sidewalls, and large irregularly shaped sheet metal parts needs to be inspected for appearance quality on each piece. Among them, paint runs, particles, and scratches are the three most common defects. Paint runs are strip-shaped protrusions formed by paint falling on a vertical surface due to gravity. Particles are dot-shaped protrusions formed by dust or paint residue mixed into the paint film. Scratches are linear indentations formed by bumps and knocks during transportation. The color of the three is basically the same as the surrounding paint surface, only differing from the normal paint surface in height by tens to hundreds of micrometers. They appear in large quantities on vertically placed curved surfaces such as car doors and fenders. They are difficult to distinguish by grayscale or color imaging alone. It is necessary to obtain the three-dimensional shape of the paint surface for reliable identification.
[0003] To address this, one existing approach is to install a structured light sensor at the end of a robotic arm. The robotic arm moves the sensor to various shooting positions on the workpiece surface according to the taught or offline planned pose. The sensor projects coded stripes onto the painted surface, and the camera captures the deformed stripes. The three-dimensional point cloud within the field of view is obtained through phase solving and triangulation. Then, using the reference surface fitted within the field of view or the theoretical surface registered with the three-dimensional model as a reference, the deviation of each point in the point cloud relative to the reference surface is calculated. Areas with deviations exceeding a threshold are identified as defects.
[0004] In their analysis of the above methods, technicians discovered that the accuracy of such solutions depends on whether the reference surface truly represents the normal paint surface. However, this premise often does not hold true in curved surface painting scenarios. When using a reference plane within the field of view, the undulations of the curved surface itself within a range of tens of millimeters can reach a fraction of a millimeter, far exceeding the defect height. The curvature fluctuations are misjudged as defects across the entire surface. When using the theoretical curved surface of the 3D model as a reference, the clamping deviation and stamping deviation of the workpiece are also far greater than the defect height, and the real defects are submerged in the deviations. If the fitting range is reduced to suppress the influence of curvature, the fitting range is often smaller than the paint flow extending in a vertical direction. The paint flow points occupy the majority in a local area, and when the reference surface obtained by robust fitting is applied to the paint flow surface, the deviation of the paint flow area is close to zero.
[0005] This results in some or all of the defects, such as paint runs, on curved surfaces being missed, while normal areas with large curvature are falsely reported as defects. The test results need to be manually verified and are difficult to use directly to guide subsequent automatic sanding and paint touch-up. Summary of the Invention
[0006] One of the objectives of this invention is to provide a method and system for detecting defects in curved paint spraying using a robotic arm with normal motion, in order to solve the problem that in the existing three-dimensional detection of defects in curved paint spraying, the reference surface is affected by curvature fluctuations, workpiece deviations and sheet defects, causing it to deviate from the real paint surface, resulting in missed defects and false alarms.
[0007] This invention is achieved through the following technical solution: a method for detecting defects in painted surfaces using a robotic arm with normal tracking, comprising the following steps: planning a sequence of pose nodes based on the three-dimensional model of the workpiece to be tested, and acquiring the local normal and local surface of the surface to be tested at each pose node; controlling the robotic arm carrying a binocular structured light sensor to move sequentially to each pose node, so that the optical axis of the projector in the binocular structured light sensor coincides with the local normal, and maintains a preset working distance with the surface to be tested; when the robotic arm is stationary, the projector projects a stripe pattern sequence, and triggers the first and second cameras to simultaneously acquire deformed stripe images with the synchronization signal output by the projector, while simultaneously recording the pose of the robotic arm end relative to the robot base; solving the absolute phase based on the deformed stripe images, and based on the absolute phase... Matching of corresponding points is performed, and a 3D point cloud in the camera coordinate system is obtained through triangulation. Based on the recorded pose and the pre-calibrated hand-eye transformation relationship, the local surface is transformed to the camera coordinate system, and the 3D point cloud is compensated according to the undulation of the local surface relative to the tangent plane at the pose node to obtain a compensated point cloud. The compensated point cloud is divided into multiple local neighborhoods, and local points are determined in each local neighborhood by removing points with abrupt height changes. The local reference surface is fitted by the local points. The signed normal distance of each point relative to the local reference surface is calculated, and points with an absolute value of normal distance greater than a detection threshold are clustered to obtain the paint defect region. The position of the paint defect region is mapped to the robot base coordinate system according to the pose and the hand-eye transformation relationship and then output.
[0008] Further, the planning of the pose node sequence and acquisition of local normals and local surfaces, as well as aligning the optical axis of the projector with the local normals, includes: discretizing the 3D model into a triangular mesh, acquiring the normal and area of each triangular facet; arranging the pose nodes along a serpentine trajectory, the spacing between adjacent pose nodes being determined by the field-of-view size of the binocular structured light sensor at the working distance and a preset overlap rate; combining the facet normals of each triangular facet within the field of view corresponding to each pose node by area weighting to obtain the local normal at that pose node, and obtaining the two principal curvatures and corresponding principal directions at that pose node from the 3D model as the local surface; using the opposite direction of the local normal as the third axis direction of the tool coordinate system, and using the projection direction of the scanning forward direction in the tangent plane of the pose node as the first axis direction of the tool coordinate system to obtain the target pose at that pose node, the origin of the tool coordinate system being located on the optical axis of the projector and at a distance from the projector from the working distance.
[0009] Furthermore, the step of projecting a stripe pattern sequence and synchronously acquiring deformed stripe images when the robotic arm is stationary includes: sending a hardware trigger pulse to the projector when the encoder feedback speed of each joint of the robotic arm is zero; after being triggered, the projector sequentially projects multiple Gray code patterns and multiple phase-shifted sinusoidal stripe patterns, the stripe direction of each pattern being perpendicular to the line connecting the first camera and the second camera; the projector outputs the synchronization signal when projecting each pattern, and the synchronization signal is connected to the hardware trigger terminals of the first camera and the second camera.
[0010] Further, the process of solving for the absolute phase and obtaining the three-dimensional point cloud through triangulation includes: solving for the wrapping phase, background light intensity, and modulation degree of each pixel from the deformed fringe image corresponding to the phase-shifted sinusoidal fringe pattern, and marking pixels with a modulation degree lower than a preset modulation degree threshold as invalid pixels; using the background light intensity as a binarization reference, binarizing and decoding the deformed fringe image corresponding to the Gray code pattern to obtain the fringe level of each pixel, and obtaining the absolute phase from the fringe level and the wrapping phase; performing epipolar correction on the images of the first camera and the second camera, searching for positions with equal absolute phase on the same epipolar line in the second camera image for each valid pixel in the first camera image, obtaining sub-pixel matching positions by interpolation of adjacent pixels, and performing triangulation based on the disparity obtained from the matching.
[0011] Further, the compensation of the 3D point cloud includes: converting the position of the pose node, the local normal, and the two principal directions to the camera coordinate system through the recorded pose and hand-eye transformation relationship to obtain the reference point, reference normal, and reference principal directions; projecting the position vector of each point in the 3D point cloud relative to the reference point to the two reference principal directions to obtain the tangential coordinates of each point; calculating the undulation of the local surface at that point relative to the tangential plane using the two principal curvatures and the tangential coordinates, and translating the point along the reference normal by the undulation to obtain the compensated point cloud.
[0012] Further, the step of determining local points and fitting a local reference surface within each local neighborhood includes: the side length of the local neighborhood is not greater than an upper limit value, the upper limit value being determined by a preset lower tolerance limit and the allowable deviation of the actual curvature of the workpiece under test relative to the principal curvature; performing a first round of fitting using the upper tolerance limit as the distance tolerance, determining the distance tolerance of the local neighborhood based on the dispersion of the normal distances of each point obtained in the first round of fitting, and ensuring that the distance tolerance is between the lower tolerance limit and the upper tolerance limit; performing a second round of fitting using the distance tolerance, and refitting the local points obtained in the second round of fitting using the least squares method to obtain the local reference surface; the positive direction of the signed normal distance is in the same direction as the reference normal, and the detection threshold is not less than the upper tolerance limit.
[0013] Further, the connected clustering and mapping to the robot base coordinate system includes: organizing the three-dimensional point cloud according to the pixel arrangement of the first camera; performing connected clustering on points whose absolute value of normal distance is greater than the detection threshold according to eight neighborhoods; determining them as convex defects or concave defects according to the sign of the normal distance in each connected region; and removing connected regions whose projected area is less than a preset area threshold and whose bounding box volume is less than a preset volume threshold; using the average coordinate of each point in the paint defect region in the three-dimensional point cloud as the defect centroid; sequentially transforming the defect centroid to the robot base coordinate system through the hand-eye transformation relationship and the recorded pose; and outputting the normal of the local reference surface corresponding to the paint defect region to the robot base coordinate system through the same rotation transformation, and outputting it together with the coordinate of the defect centroid.
[0014] Another aspect of the present invention provides a curved surface paint defect detection system with robotic arm normal motion, including a robotic arm, a binocular structured light sensor, and a host computer; the binocular structured light sensor is installed at the end of the robotic arm and includes a protective housing and a projector, a first camera, and a second camera disposed within the protective housing, the first camera and the second camera being symmetrically disposed on both sides of the projector; the host computer is communicatively connected to the robotic arm and the binocular structured light sensor, and the host computer is configured to execute a curved surface paint defect detection method with robotic arm normal motion as described above.
[0015] Furthermore, the projector has a first linear polarizer at its light output port, and both the first and second cameras have a second linear polarizer in front of their lenses. The transmission direction of the second linear polarizer is perpendicular to the transmission direction of the first linear polarizer.
[0016] Furthermore, the optical axes of the first camera and the second camera are both tilted toward the optical axis of the projector, and both intersect the optical axis of the projector at the same point.
[0017] Compared with the prior art, the present invention has the following advantages and beneficial effects: 1. This invention uses the normal follow-up of the robotic arm to make the optical axis of the projector coincide with the local normal at each pose node, and uses the pose recorded during acquisition to transform the local curved surface of the 3D model to the camera coordinate system. By deducting the theoretical fluctuations of the curved surface relative to the tangent plane, the compensated normal paint surface is close to a plane in the local neighborhood, which overcomes the problem in the prior art that the curvature fluctuations are misjudged as defects when the reference plane in the field of view is used as a reference.
[0018] 2. This invention only subtracts the surface shape of the 3D model from the point cloud, while the clamping deviation and forming deviation of the workpiece are absorbed by the local reference surface fitted in each local neighborhood, which overcomes the problem of workpiece deviation submerging defects when the theoretical surface is directly used as a reference in the prior art.
[0019] 3. After deducting surface undulations, the size of the local neighborhood is only limited by the actual curvature deviation, which can be significantly larger than the size of sheet defects such as paint flow. The defect points are few in number in the local neighborhood, and the local reference surface falls on the normal paint surface. This overcomes the problem in the prior art that the reference surface is attached to the paint flow surface after the fitting range is reduced, which leads to missed detection of paint flow.
[0020] 4. When the robotic arm is stationary, the present invention directly triggers two cameras with a projection synchronization signal and maps the centroid of the defect and the reference plane normal at the defect to the robot base coordinate system. The output result can be directly used as the target position and approach direction of the grinding or painting robot. Attached Figure Description
[0021] The accompanying drawings, which are included to provide a further understanding of embodiments of the invention and form part of this application, do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart of the method provided in Embodiment 1 of the present invention; Figure 2 This is a schematic diagram of stripe pattern projection and three-dimensional point cloud reconstruction in Embodiment 1 of the present invention; Figure 3 This is a schematic diagram of the signed normal distance within a local neighborhood in Embodiment 1 of the present invention; Figure 4 This is a flowchart illustrating the transformation of defect coordinates from pixel coordinates to the robot base coordinate system in Embodiment 1 of the present invention; Figure 5 This is a point cloud height distribution map before and after deducting model fluctuations in Embodiment 3 of the present invention; Figure 6 This is a comparison chart of the defect detection results of the four schemes in Embodiment 3 of the present invention; Figure 7 This is a schematic diagram of the overall structure of the detection system provided in Embodiment 4 of the present invention; Figure 8 This is a schematic diagram of the internal arrangement of the binocular structured light sensor in Embodiment 4 of the present invention.
[0022] The attached diagram shows the markings and corresponding component names: 1. Robotic arm; 2. Binocular structured light sensor; 21. Protective housing; 22. Projector; 23. First camera; 24. Second camera; 3. Workpiece under test; 31. Defect location. Detailed Implementation
[0023] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments.
[0024] It should be noted that the directional indications in the embodiments of the present invention are only used to explain the relative positional relationship between the components in a specific posture. When the specific posture changes, the directional indications also change accordingly. The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.
[0025] Example 1:
[0026] Understandably, in existing 3D inspection of curved paint surfaces, the sensor's shooting posture is determined by teaching or experience. The paint surface within the field of view is tilted and curved relative to the sensor. Technical personnel, in analyzing existing technologies, have found that the reference surface relied upon for defect determination, under these conditions, will both treat curvature fluctuations as defects and be attracted and deflected by paint flows extending in vertical directions. This embodiment addresses the workpiece 3, such as painted body panels and sidewalls, whose surfaces are mostly vertically placed, with continuously changing curvature and concave areas, exhibiting high-gloss paint. The defects to be detected are paint flows, particles, and scratches whose height differs from normal paint surfaces by tens to hundreds of micrometers and whose color is essentially the same as the normal paint surface.
[0027] Therefore, this embodiment discloses a method for detecting defects in curved surface paint spraying using a robotic arm with normal motion. Figure 1 The flowchart of the method in this embodiment is shown, which specifically includes the following steps: Step S1: Based on the three-dimensional model of the workpiece 3, plan the pose node sequence and obtain the local normal and local surface of the surface to be measured at each pose node; control the robotic arm 1 to carry the binocular structured light sensor 2 to move to each pose node in sequence, so that the optical axis of the projector 22 coincides with the local normal and maintains a preset working distance with the surface to be measured.
[0028] Among them, pose nodes refer to the shooting positions that the robotic arm 1 sequentially reaches during the detection process and the end effector posture at those positions. Each pose node corresponds to a field of view on the measured surface. Local normals refer to the average outward normal direction of the measured surface within the field of view. Local surfaces refer to the surface shape described by the two principal curvatures and their principal directions at the pose node. Both are calculated from the 3D model. Working distance refers to the distance between the projector 22 and the measured surface along the optical axis of the projector 22. For example, the working distance... A focal length of 300mm can be selected, which corresponds to the focusing distance of the binocular structured light sensor 2.
[0029] In this embodiment, to ensure that the binocular structured light sensor 2 is directly facing the surface being measured at each pose node, step S1 specifically includes: S11. Discretize the 3D model into a triangular mesh and obtain the normal and area of each triangular facet. Specifically, import the 3D model of the workpiece 3 to be measured (e.g., a file in STEP or IGES format), and use triangulation to discretize the curved surfaces in the model into a mesh composed of triangular facets. The vertices of each triangular facet are arranged in a counterclockwise order when viewed from the outside, so that the calculated facet normals point to the outside of the workpiece 3 to be measured. It should be noted that the workpiece 3 to be measured is positioned by the fixture, and the 3D model has been converted to the robot base coordinate system according to the positioning reference of the fixture before importing. Therefore, all the following vectors are represented in the robot base coordinate system. S12. Arrange pose nodes along the serpentine trajectory; Specifically, arrange pose nodes row by row on the surface being measured. Pose nodes in the same row are arranged sequentially along the scanning direction, and the directions of adjacent rows are opposite, thus forming a reciprocating serpentine trajectory. The spacing between adjacent pose nodes is determined by the field size and overlap rate. The reason for retaining overlap is to ensure that the edge parts of adjacent fields of view are repeatedly covered, and to avoid defects at the field of view boundary being truncated due to falling on the edge. S13. For each pose node, the normals of each triangular facet within the field of view corresponding to the pose node are weighted by area to obtain the local normal at the pose node. Here, the triangular facet within the field of view refers to the triangular facet whose centroid is no more than half the field of view size. At the same time, the two principal curvatures and the two corresponding principal directions at the pose node are obtained from the surface equation of the 3D model and used as the local surface at the pose node. They are stored together with the local normal. S14. Obtain the target pose at the pose node based on the local normal and the scanning direction, and control the robotic arm 1 to move to the target pose.
[0030] For example, the aforementioned local normal and target pose can be calculated using the following steps: First, regarding the first A triangular facet, the normal and area of which are:
[0031]
[0032] Secondly, the local normal at the pose node is:
[0033] Furthermore, the spacing between adjacent pose nodes is:
[0034] Finally, the opposite direction of the local normal is taken as the third axis of the tool coordinate system, the projection of the scan forward direction onto the tangent plane is taken as the first axis, and the second axis is determined by the right-hand rule, that is:
[0035]
[0036] in, The number of the triangular facet is used. The summation in equation (3) is performed on all triangular facets within the field of view determined in step S13. , , For the first The coordinate vectors of the three vertices of each triangular facet in the robot base coordinate system are derived from the triangular mesh in step S11. For the first The normal of each triangular facet points to the outside of the workpiece 3 being measured. For the first The area of a triangular facet is equal to half the magnitude of the cross product of the two edge vectors; Here, represents the local normal at the pose node, and represents the unit vector in the robot's base coordinate system. This is the spacing between adjacent pose nodes; nodes within the same row and between two adjacent rows are arranged according to this spacing. For the binocular structured light sensor 2 at the working distance The field of view size at a given location is the smaller of the longer and shorter sides of the field of view, and is determined by the selection of the camera and lens. For example, the preset overlap rate can be 20% to 30%; The unit vector of the scanning forward direction of the row where the pose node is located is determined by the serpentine trajectory in step S12; , , These are the unit direction vectors of the first, second, and third axes of the tool coordinate system in the robot base coordinate system; The target rotation matrix that the end effector of robotic arm 1 needs to perform at this pose node is given by... , , It consists of three columns, in equation (5) This is the default direction of the projector's optical axis 22 in the tool coordinate system. This formula represents the direction along which the optical axis 22 is positioned. After rotation, the optical axis of the projector 22 points to That is, it is injected into the measured surface along the local normal in the opposite direction; The coordinates of the pose node on the measured surface are the position coordinates of the pose node, that is, the position of the center of the field of view corresponding to the pose node on the surface of the three-dimensional model. Let be the target homogeneous transformation matrix of the tool coordinate system relative to the robot base coordinate system at this pose node; The homogeneous transformation matrix of the tool coordinate system relative to the end-effector coordinate system of the robotic arm 1 is obtained from the system calibration in Example 4; The target pose of the end effector of robotic arm 1 at this pose node is calculated by the controller of robotic arm 1 into the rotation angle of each joint.
[0037] It should be noted that the two principal curvatures obtained in step S13 are denoted as... , The corresponding main direction of the unit is denoted as , Both are located in the tangent plane of the pose node and are perpendicular to each other. The sign convention for the principal curvature is that it is positive when the surface bends away from the sensor (convex) in this direction, and negative when it bends towards the sensor (concave). For surfaces represented by non-uniform rational B-splines in the 3D model, the principal curvature and principal direction can be directly obtained from the first and second basic forms of the surface, which are not the focus of this application, so they will not be elaborated further.
[0038] It is understandable that the origin of the tool coordinate system is set on the optical axis of the projector 22, at a working distance from the projector 22. The position is determined, and its third axis is aligned with the optical axis of the projector 22 so that when the tool coordinate system reaches the target pose according to equation (6), its origin falls exactly on the point on the measured surface. At that location, the optical axis of the projector 22 is along The projector 22 is directed towards the surface being measured, coinciding with the local normal, and the distance between the projector 22 and the surface being measured is exactly [missing information]. The requirements of normal alignment and distance fixation are met by the same target pose at once. Based on this, Equation (5) constrains the first axis of the tool coordinate system with the scanning forward direction, so that the rotation angle of the binocular structured light sensor 2 around the optical axis remains consistent in the same row, the line connecting the first camera 23 and the second camera 24 is always approximately along the scanning forward direction, and the splicing direction of adjacent fields of view does not twist.
[0039] It should be noted that the local normal is the area-weighted value of the normals of each facet within the field of view, rather than the normal of a single facet. This is because the size of a single triangular facet is much smaller than the field of view, and its normal only reflects the local orientation of the discrete grid. If aligned in this way, the sensor might actually be tilted relative to the entire field of view. By using the area-weighted value, the optical axis of the projector 22 is consistent with the average orientation of the entire field of view, and the tilt at various points within the field of view is minimized. This also ensures that when compensating for the point cloud with local curved surfaces in step S4, the sensor's observation direction is consistent with the normal of the local curved surface. If normal alignment is omitted and a fixed angle is used for shooting, the tilt of the paint surface relative to the sensor within the field of view changes with position, the planar approximation in the local neighborhood no longer holds, and the concave region will also show stripe occlusion due to oblique illumination.
[0040] Step S2: When the robotic arm 1 is stationary, the projector 22 projects a stripe pattern sequence, and the first camera 23 and the second camera 24 are triggered by the synchronization signal output by the projector 22 to synchronously acquire the deformed stripe image, and at the same time record the pose of the end of the robotic arm 1 relative to the robot base.
[0041] Among them, the static state refers to the state in which the encoder feedback speed of each joint of the robotic arm 1 is zero; the deformed stripe image refers to the image obtained by the camera after the stripe pattern is projected onto the surface being measured and modulated by the undulation of the paint surface; the synchronization signal refers to the frame synchronization electrical signal output by the projector 22 when it starts to project each pattern. For example, the field synchronization signal of the digital light processing projector can be used.
[0042] In this embodiment, to ensure that each image acquired at the same pose node strictly corresponds to the same static scene, step S2 specifically includes: S21. The host computer reads the encoder feedback speed of each joint of the robotic arm 1 in real time. When the feedback speed of each joint is zero, it sends a high-level hardware trigger pulse to the projector 22 through the input / output interface. S22 and projector 22, after being triggered, will sequentially project within a preset time limit (for example, within 0.5 seconds). Gray code pattern and A phase-shifting sinusoidal fringe pattern, where the fringe direction of each pattern is perpendicular to the line connecting the first camera 23 and the second camera 24, is used when the first camera 23 and the second camera 24 are arranged laterally, i.e., as shown in the image. Figure 2 The image shows vertical stripes; S23 and projector 22 output a synchronization signal when projecting each pattern. This synchronization signal is directly connected to the hardware trigger terminals of the first camera 23 and the second camera 24. The two cameras are exposed at the same time and each captures the image. Aspect-distorted stripe image; S24. Simultaneously with issuing the hardware trigger pulse, the host computer reads the end-effector pose fed back by the robotic arm 1 controller via network communication, and records it as... The image of the pose node is stored together with the image of the pose node; after the acquisition is completed, the robotic arm 1 immediately moves to the next pose node.
[0043] For example, the first camera 23 or the second camera 24 captures the first... The grayscale distribution of the amplitude-phase shift deformable fringe image can be represented as:
[0044] in, These are the pixel coordinates in the camera image. Along the line connecting the first camera 23 and the second camera 24, Perpendicular to it; The sequence number of the phase-shifting pattern. ; The number of phase shift steps can be 4 or 6 in this embodiment; For the first Amplitude phase shift deformable stripe image in pixels The grayscale value at that location is obtained from step S23. Background light intensity, i.e., pixels The average brightness of the area not affected by stripe modulation; The modulation index, or the amplitude of the variation in the brightness of the stripes, reflects whether the stripes at that pixel are clear and usable. The phase is wrapped, carrying information about the paint surface undulations at that pixel, and is the quantity to be solved in step S3; For the first The phase shift of the first pattern relative to the 0th pattern is predetermined by the pattern projected by the projector 22 and is a known quantity.
[0045] It is understandable that each pixel in equation (7) has , , With three unknowns, a phase shift of at least 3 steps is sufficient to solve the problem. Choosing 4 or 6 steps is to suppress random noise with redundant equations. The fringe direction is perpendicular to the line connecting the two cameras, ensuring the wrapped phase follows... The direction changes, and after the polar correction in step S3, the polar line also changes direction. The direction, therefore the phase monotonically increases along the epipolar line, which is a prerequisite for corresponding points to be uniquely matched according to their phase values. It should be noted that the reason for requiring a stationary state and direct triggering of the camera by the synchronization signal of projector 22 is because the phase-shift method requires… If each image corresponds to the same scene, and the robotic arm 1 is still micro-moving, the same pixel will correspond to different positions on the paint surface in different images, and the obtained phase will introduce system errors. If the projection and shooting are triggered by software in sequence, there will be millisecond-level timing jitter between the two, and there will be a situation where the projector 22 has not yet switched to the corresponding pattern when a certain image is exposed. In a specific example, after using the above hardware synchronous trigger, the time deviation between projection and acquisition can be controlled within 1ms.
[0046] Step S3: Solve for the absolute phase based on the deformed fringe image, perform corresponding point matching based on the absolute phase, and obtain the 3D point cloud in the camera coordinate system through triangulation.
[0047] Among them, absolute phase refers to the elimination of phase. The phase is the phase that changes continuously within the entire image after periodic truncation; corresponding points refer to a pair of pixels on the same object point on the measured surface in the images of the first camera 23 and the second camera 24; the camera coordinate system refers to a coordinate system with the optical center of the first camera 23 after epipolar correction as the origin and its optical axis as the third axis. The compensation and fitting in step S4 are performed in this coordinate system.
[0048] In this embodiment, in order to obtain a complete and reliable 3D point cloud from the stripe image, step S3 specifically includes: S31, by The phase-shift deformable fringe image is solved pixel by pixel for the wrapping phase, background light intensity and modulation degree, and pixels with modulation degree lower than the preset modulation degree threshold are marked as invalid pixels; S32. Using background light intensity as the binarization reference, The Gray code deformed fringe image is binarized and decoded pixel by pixel to obtain the fringe order of each pixel, and then the absolute phase is obtained from the fringe order and the wrapping phase. S33. Using the intrinsic and extrinsic parameters of the first camera 23 and the second camera 24 obtained from calibration, perform epipolar correction on the absolute phase maps of the two cameras so that corresponding points are located on the same row of the two images. S34. For each valid pixel in the image of the first camera 23, search for positions with equal absolute phase in the same row of the image of the second camera 24, obtain sub-pixel matching positions by linear interpolation of the phases of two adjacent pixels, and then perform triangulation based on parallax.
[0049] For example, the above quantities can be calculated using the following steps: First, using the least squares principle, the phase, background intensity, and modulation are defined as follows:
[0050]
[0051] Secondly, the binarization, decoding, and absolute phase of Gray code are as follows:
[0052]
[0053]
[0054] Secondly, the subpixel matching position is:
[0055] Finally, the coordinates of the point obtained by triangulation are:
[0056] in, To wrap the phase, the arctangent in equation (8) is taken as the quadrant value according to the signs of the numerator and denominator, and the result is added. An integer multiple of 0, such that it is not less than 0 and is less than 0. ; , , The meaning is the same as that of equation (7); Background light intensity is equal to The average gray level of the phase-shifted image at that pixel; In order to adjust the system, when If the gray level of a pixel is lower than the preset modulation threshold, or if the gray level of any phase-shifted image at that pixel reaches the camera's saturation value, the pixel is marked as an invalid pixel and will not participate in subsequent matching. The sequence number of the Gray code pattern. The smaller the serial number, the wider the corresponding stripe; The number of Gray code patterns that satisfy... Not less than the number of periods of the phase-shifted sinusoidal fringes within the field of view; For the first Gray code distorted stripe image in pixels The grayscale value at that location is obtained from step S23. For the first The bit Gray code value is obtained by binarizing the background light intensity of Equation (9) as a reference. The pixel coordinates are omitted in Equation (11). ; The first bit is obtained by bitwise XOR transformation of Gray code. Bit binary code value, Represents the XOR operation; The fringe order is the index of the sinusoidal fringe period in which the pixel is located. It is an absolute phase; , These are the absolute phase diagrams of the first camera 23 and the second camera 24 after epipolar correction, respectively. Correct the effective pixels in the image of the first camera 23; Correcting the image for the second camera 24th Satisfaction in the line Integer pixel column number; To correct subpixel matching positions in the images from the second camera 24; The focal length of the corrected camera is expressed in pixels. The baseline distance between the optical centers of the first camera 23 and the second camera 24 after correction; The coordinates of the principal point of the first camera 23 after correction; , , The coordinates of the object point are in the camera coordinate system.
[0057] It is understandable that in equation (8), the numerator and denominator are the sums of the gray levels of each frame, weighted by sine and cosine, respectively, and are respectively related to... , The reflectivity is proportional to the background light intensity and is independent of the background light intensity, so the local difference in the reflectivity of the paint surface will not directly enter the phase; the adjacent code values of the Gray code are only one bit different, so even if one bit is misjudged at the edge of the stripe, the stripe level will only be one level wrong and will not jump to a distant position. Furthermore, with the background light intensity of Equation (9) as the binarization reference, the binarization threshold of the highlight and dark areas changes adaptively with the pixel, and there is no need to project full bright and full dark patterns separately; since the corresponding points are searched with the absolute phase value, and the absolute phase is monotonically along the corrected row direction, each effective pixel of the first camera 23 has only one matching position on the same row of the second camera 24, and Equation (14) can directly give the depth. It should be noted that the three-dimensional point cloud in this embodiment is obtained by parallax triangulation between the two cameras. The projector 22 only provides phase encoding and does not participate in triangulation, so there is no need to calibrate the intrinsic parameters of the projector 22, and the distortion of the lens of the projector 22 will not be transmitted to the point cloud; the marking of invalid pixels avoids the generation of outliers by incorrect phase in the highlight saturation or shadow.
[0058] Step S4: Based on the recorded pose and the pre-calibrated hand-eye transformation relationship, transform the local surface to the camera coordinate system, and compensate the 3D point cloud according to the undulation of the local surface relative to the tangent plane at the pose node to obtain the compensated point cloud; divide the compensated point cloud into multiple local neighborhoods, determine the inliers in each local neighborhood by removing points with abrupt changes in height, and fit the local reference plane with the inliers; calculate the signed normal distance of each point relative to the local reference plane, and perform connected clustering on the points whose absolute value of the normal distance is greater than the detection threshold to obtain the paint defect area.
[0059] Among them, undulation refers to the distance by which a local surface of a 3D model deviates from the tangential plane of the pose node at a certain tangential position. It only reflects the design shape of the measured surface and does not include the clamping deviation and forming deviation of the workpiece. Compensated point cloud refers to the point cloud obtained by translating each point in the 3D point cloud along the reference normal by its undulation. The normal paint surface is close to a plane in the compensated point cloud. Local neighborhood refers to the square area divided in the compensated point cloud according to the first camera's 23-pixel arrangement. Local reference plane refers to the plane that represents the normal paint surface in the local neighborhood. Signed normal distance refers to the vertical distance from the point to the local reference plane, with the side pointed to by the reference normal as positive.
[0060] In this embodiment, to ensure that the local reference surface is neither affected by curvature fluctuations and workpiece deviations, nor is it deflected by sheet defects, step S4 specifically includes: S41. Utilize the end-effector pose recorded in step S24 With hand-eye transformation matrix The position, local normal, and two principal directions of the pose node are transformed to the camera coordinate system to obtain the reference position, reference normal, and reference principal direction. S42. Project the position vectors of each point in the 3D point cloud relative to the reference point onto the two reference principal directions to obtain the tangential coordinates of each point. Calculate the undulation at each point using the two principal curvatures and the tangential coordinates, and translate each point along the reference normal by the undulation to obtain the compensated point cloud. S43. Determine the side length of the local neighborhood so that it is no greater than the upper limit determined by the allowable deviation between the lower tolerance limit and the actual curvature, and is significantly greater than the expected size of a single flow of paint. Divide the compensated point cloud into multiple local neighborhoods according to this side length. S44. In each local neighborhood, the upper limit of the tolerance is used as the distance tolerance for the first round of fitting: each time, three non-collinear points are randomly selected from the local neighborhood to determine a candidate reference surface. The absolute value of the normal distance is not greater than the number of local points in the distance tolerance. After the maximum number of iterations is reached, the candidate reference surface with the most local points is retained. S45. Determine the distance tolerance of the local neighborhood based on the dispersion of the normal distance of each point obtained in the first round of fitting, and limit it between the lower limit and the upper limit of the tolerance. Then, perform the second round of fitting according to the distance tolerance in step S44. Refit the plane with the least squares method for all the local points obtained in the second round of fitting, and use it as the local reference plane of the local neighborhood. S46. Calculate the signed normal distance of each point in the local neighborhood relative to the local reference plane. Mark the points whose absolute value of the normal distance is greater than the detection threshold as candidate defect points. Perform connected clustering on the candidate defect points according to the eight neighborhood to obtain the paint defect area.
[0061] For example, the above quantities can be calculated using the following steps: First, the reference point, reference normal, and reference principal direction are as follows:
[0062] Secondly, the The tangential coordinates, undulation amount, and compensated coordinates of each point are as follows:
[0063]
[0064] Furthermore, the upper bound of the local neighborhood edge length and the maximum number of iterations are:
[0065]
[0066] Finally, the unit normal of the local reference plane, the signed normal distance, and the distance tolerance used in the second round of fitting are:
[0067]
[0068]
[0069] in, The coordinates of the pose node on the measured surface in step S1. Its coordinates in the camera coordinate system, i.e., the reference point; The end pose recorded in step S24, Let be the rotation matrix; The hand-eye transformation matrix, The rotation matrix is obtained from the hand-eye calibration in Example 4; This is the rotation matrix from the camera coordinate system to the robot base coordinate system; The local normal obtained in step S13 Its representation in the camera coordinate system, i.e., the reference normal; , For the two unit principal directions obtained in step S13, , Its representation in the camera coordinate system, i.e., the reference principal direction; The index of a point in a 3D point cloud; For the first The coordinates of the points in the camera coordinate system are obtained by equation (14); , For the first Tangential coordinates of a point along two principal reference directions; , The two principal curvatures obtained in step S13 are defined by the notation conventions in step S1. For the local surface in the first The undulation at the tangential position of a point; For the first The coordinates after point compensation, all This constitutes a compensated point cloud; Let be the side length of the local neighborhood on the measured surface; This is the lower limit of the tolerance. As the upper limit of the tolerance, for example, the two can be 0.02mm and 0.05mm respectively; The allowable relative deviation of the actual curvature of the workpiece 3 relative to the principal curvature of the model is determined by the curvature control level of the workpiece forming process. For example, for stamped car body panels, it can be estimated at 5%. The larger of the two principal curvatures; The result is rounded up to the nearest integer representing the maximum number of iterations. The expected success probability of fitting is the probability that at least one sampling will yield three points that are all normal paint points, and its value is close to 1. This represents the proportion of normal paint surface points within the local neighborhood. The initial value can be given empirically, and it is updated during iteration based on the proportion of local points on the current optimal candidate reference surface. , , , Plane equations for candidate datum planes or local datum planes The coefficients, after all four coefficients change sign simultaneously, do not change the plane. In this embodiment, we take... A group; The unit normal to the candidate datum plane or local datum plane; For the first Signed normal distance of each point This indicates that the point is above the local reference plane. This indicates that the point is below the local reference plane; The candidate reference surface retained for the first round of fitting Signed normal distance of each point; This represents taking the median of all points within a local neighborhood; To estimate the dispersion of the normal distance based on the absolute deviation of the median, a constant of 1.4826 is used to make it comparable to the standard deviation under a normal distribution; This is the tolerance factor; for example, it can be taken as 3 according to the three-standard-deviation criterion. The distance tolerance used for the second round of fitting.
[0070] It is understandable that in equation (17) It is precisely the quadratic surface in tangential coordinates The deviation from the tangent plane is such that a point on the convex surface is located below the tangent plane and translated upwards along the reference normal. Then it returns to the vicinity of the tangent plane, the concave surface If the value is negative, the translation direction is reversed. The compensation only uses the surface shape of the three-dimensional model. The reference point and the reference main direction are converted from the planned value. Even if there is a clamping deviation in the workpiece, the offset of the reference point along the normal will not enter the tangential coordinate of equation (16). The remaining overall offset and tilt after compensation is exactly a plane, which is absorbed by the local reference plane of each local neighborhood. Thus, the defects of the two reference methods in the prior art are handled separately: the design shape of the surface is subtracted by the three-dimensional model, and the deviation of the workpiece relative to the design is absorbed by local fitting.
[0071] It should be noted that the undulations remaining in the local neighborhood after compensation only come from the deviation between the actual curvature and the principal curvature of the model, and their magnitude is approximately Setting it to be no greater than the lower tolerance limit yields equation (18); compared to not compensating and directly defining the neighborhood by the radius of curvature, equation (18) allows for a larger side length. times, when When the value is 5%, it is approximately 4.5 times. After the neighborhood side length is expanded, the large-scale defects of paint flow extending more than ten millimeters vertically are only a minority in the local neighborhood. The random sampling in step S44 is mainly based on normal paint surface points, and the local reference plane will not be attached to the paint flow surface. If compensation is omitted and the smaller neighborhood is still used for fitting, the paint flow can cover most of the neighborhood, and the local reference plane will be raised accordingly. The normal distance of the paint flow area is close to zero and will be missed. Under the premise of satisfying formula (18), the side length of the local neighborhood should be larger, for example, more than 1.5 times the expected length of a single paint flow.
[0072] Based on the above embodiments, as a further optional implementation, Equations (21) and (22) make the distance tolerance adaptively change with the micro-undulations of the local paint surface; in areas with obvious orange peel texture, the normal distance distribution of the normal paint surface points is wider, and the distance tolerance is correspondingly widened to avoid the normal texture being driven out of the local points and making the reference surface unstable; in smooth areas, the distance tolerance is correspondingly tightened, and the reference surface is closer to the real paint surface; after the second round of fitting, the local points are refitted in the least squares manner because the candidate reference surface obtained by random sampling is determined by only three points, and there can be a certain offset and tilt within the distance tolerance range. Direct use will cause the actual position of the detection threshold to drift. After refitting with all local points, the local reference surface falls at the median position of the normal paint surface; the absolute deviation of the median is not sensitive to outliers, and the defect points remaining in the first round of fitting will not raise the dispersion estimate.
[0073] For example, please refer to Figure 3 After obtaining the local reference plane, the signed normal distance is calculated point by point using equation (20), and the detection threshold is determined. For example, 0.05 mm can be taken, and make Not less than the upper limit of tolerance This ensures that the local points involved in the fitting do not overlap with the candidate defect points; since the 3D point cloud is organized according to the pixel arrangement of the first camera 23, adjacent pixels correspond to adjacent object points, thus... Candidate defect points are directly clustered into eight-neighborhood clusters on the pixel grid, with each connected region being a paint defect region. Subsequently, the projected area and bounding box volume of each paint defect region are calculated. The projected area is the convex hull area projected by each point in the region onto the local reference plane. The bounding box is centered on the normal of the local reference plane. When the projected area is less than a preset area threshold and the bounding box volume is less than a preset volume threshold, the region is considered a normal fluctuation in paint roughness and is removed. For the retained paint defect regions, those with predominantly positive normal distances are classified as protrusion defects, and those with predominantly negative normal distances are classified as depression defects.
[0074] In some embodiments, the long axis direction of the paint defect area can be further... Transform to the robot base coordinate system and compare with the vertical direction: those with a major axis close to the vertical and convex are classified as paint flow; those with a major axis close to the minor axis and convex are classified as particles; and those with a thin, elongated shape and concave are classified as scratches. In some embodiments, for areas with drastic curvature changes and insufficient approximation of the quadratic surface, the distance from the 3D model mesh to the tangent plane along the reference normal can be directly calculated as the undulation amount, replacing the undulation in equation (17). In some embodiments, when the clamping deviation of the workpiece 3 is large, and the reference point position and reference principal direction obtained by the end pose and hand-eye transformation matrix are insufficient to reflect the actual position of the workpiece, the three-dimensional point cloud and the corresponding area of the three-dimensional model can be iteratively coarsely registered and finely registered to the nearest point to determine the actual pose of the camera coordinate system in the coordinate system of the three-dimensional model. Then, the pose obtained by registration can be used to replace the pose in equation (15). The reference point position, reference normal, and reference principal direction are converted, while the remaining steps remain unchanged; any minor offsets and tilts remaining after registration are still absorbed by the local reference surfaces of each local neighborhood. In some embodiments, candidate reference surfaces with an angle greater than a preset angle threshold between the normal and the reference normal can be discarded during fitting. This angle threshold should be greater than the normal deviation caused by the positioning error of the robotic arm 1 and the clamping deviation of the workpiece 3, and is used to exclude candidate reference surfaces with obvious tilt determined by points on the steep slope of the defect edge.
[0075] Step S5: Based on the recorded pose and hand-eye transformation relationship, map the position of the paint defect area to the robot base coordinate system and output it.
[0076] The robot base coordinate system refers to the coordinate system used by the subsequent grinding robot or programmable logic controller to perform repair operations.
[0077] In this embodiment, to ensure that the detection results can directly guide subsequent repairs, please refer to... Figure 4 Step S5 specifically includes: S51. The average coordinates of all points within the paint defect area in the 3D point cloud are used as the centroid of the defect; it should be noted that the coordinates before compensation are used here. The compensation only serves to fit the reference surface and does not change the actual spatial location of the defect; S52. Transform the centroid of the defect into the robot base coordinate system by sequentially using the hand-eye transformation matrix and the end pose recorded by the pose node; S53. The normal of the local reference plane corresponding to the paint defect area is transformed to the robot base coordinate system by the same rotation, and output to the programmable logic controller or the control cabinet of the grinding robot along with the coordinates of the defect centroid and the defect type.
[0078] For example, the above quantities can be calculated using the following steps: First, the homogeneous coordinates of the defect centroid are:
[0079] Secondly, the coordinates of the defect centroid in the robot base coordinate system are:
[0080]
[0081] Finally, the direction of the reference plane normal at the defect location in the robot base coordinate system is:
[0082] in, This represents the total number of points within the paint defect area. For the first in this region The coordinates of each point before compensation have the same meaning as in equation (16). same; , , The coordinates of the defect centroid in the camera coordinate system; The homogeneous coordinates of the defect centroid in the camera coordinate system; Let be the homogeneous coordinates of the defect centroid in the robot base coordinate system; , They are respectively and A 3×1 translation vector; It is a zero vector of size 1×3; , , The meaning is the same as that of equation (15); The unit normal of the local reference plane in the local neighborhood where the paint defect area is located is obtained by equation (20); Let be the unit vector of the reference plane normal at the defect location in the robot's base coordinate system.
[0083] It is understandable that equation (24) is first derived from... Transform the centroid of the defect from the camera coordinate system to the end coordinate system, and then... Transform to the robot base coordinate system, and The value recorded during the acquisition of the pose node is taken, so the actual positioning error of the robotic arm 1 at each pose node is accurately included; Equation (26) only involves the rotation part, and the output is Pointing outwards from the painted surface, this direction can be directly used as the approach direction for sanding tools or touch-up paint spray guns. It should be noted that adjacent fields of view overlap, and the same defect may be detected repeatedly at two pose nodes. Therefore, before outputting, paint defect areas with a centroid distance less than the preset merging distance and of the same category in the robot base coordinate system are merged into one to avoid duplicate repairs.
[0084] Example 2:
[0085] This embodiment uses the paint inspection of the side panel of a car body after spraying as the application scenario, and further explains the method of Embodiment 1 as follows: In actual operation, after the car body enters the inspection station along the conveyor line, it is fixed by the positioning pin and the clamping mechanism. The outer door panel and the fender are approximately vertical at this station. Its surface has both large surfaces with gentle curvature and areas with larger curvature such as waistline and wheel arch, as well as concave areas such as door handle recess. Before inspection, the host computer imports the three-dimensional model of the car model and plans the serpentine trajectory and each pose node of the side surface according to steps S11 to S14 of Example 1. For each pose node, the local normal, principal curvature and principal direction, target pose and the local neighborhood edge length determined by equation (18) are calculated in advance. These data are reused in the continuous inspection of the same car model.
[0086] During detection, robotic arm 1, carrying binocular structured light sensor 2, moves sequentially through each pose node. At each pose node, step S2 is performed to acquire and record the end-effector pose while the arm is stationary. The arm then moves to the next pose node. Simultaneously, the host computer reconstructs the point cloud from the acquired image using step S3. Acquisition and calculation are performed in parallel. After all pose nodes have been acquired (in... Figure 1 (The judgment of "whether it is the last pose node" is shown in the middle). The host computer completes the model undulation compensation and extracts the paint defect area in each field of view according to step S4. Then, according to step S5, its centroid, normal and category are unified to the robot base coordinate system. After merging and repeated detection, a defect list is generated.
[0087] In this scenario, the curvature of the waistline and wheel arches is relatively large, and the side length of the local neighborhood obtained by equation (18) is correspondingly reduced. However, the curvature of the large surface of the car door outer panel is relatively small, and the local neighborhood can be taken as tens of millimeters. The paint flow distributed in patches along the direction of gravity on the vertical large surface accounts for only a minority in the local neighborhood. After the defect list is issued to the control cabinet of the subsequent sanding and painting station, the repair robot uses As the target location, with The opposite direction is used as the tool feed direction for grinding and painting, eliminating the need for manual remarking of defect locations between inspection and repair.
[0088] Example 3:
[0089] To facilitate understanding of the role of model fluctuation compensation and local fitting in step S4, this embodiment is illustrated with specific examples. A field of view measurement data is constructed based on the actual working conditions of the car door outer panel at the inspection station using numerical simulation. Defect extraction is performed using the method of this embodiment and three comparative schemes.
[0090] In the example, the field of view is 60mm×60mm with a dot pitch of 0.1mm, which is comparable to the sampling density of the binocular structured light sensor 2 at a working distance of 300mm. The horizontal principal curvature radius of the measured surface at the center of the field of view is 1200mm and the vertical principal curvature radius is 5000mm, which is on the curvature scale of a large surface of a car door panel. The differences between the actual workpiece and the 3D model are set as follows: there is a 5% deviation in the principal curvature, a 0.15° tilt and a 0.10mm normal offset caused by clamping, and a long-waveform deviation with an amplitude of 0.03mm and a wavelength of 250mm is superimposed; an orange peel texture with a root mean square of about 2μm and a correlation length of about 1.5mm is superimposed on the paint surface, and the root mean square of the measured noise is taken as 5μm; five defects are set in the field of view: three particles with heights of 0.08mm, 0.12mm and 0.06mm respectively; one scratch with a depth of 0.07mm and a length of about 16mm; and one paint flow with a width of 8mm and a vertical length of 18mm, the thickness of which increases from zero to 0.15mm from top to bottom and then falls back at the lower edge.
[0091] The above quantities are all selected according to the orders of magnitude commonly found in spray coating inspection, with the defect height covering a range from near the detection threshold to several times the detection threshold.
[0092] All four schemes use the same detection threshold of 0.05mm, the same eight-neighbor connectivity clustering, and the same small region removal conditions. The only difference lies in the method of determining the reference plane: Scheme 1 uses the least squares plane of the entire field of view as the reference; Scheme 2 uses the theoretical surface of the 3D model under the nominal pose as the reference; Scheme 3 does not perform model undulation compensation and limits the neighborhood side length according to the radius of curvature, i.e., it takes... mm, within each neighborhood, a plane is fitted using conventional random sampling as a reference; in this embodiment, the model fluctuations are first deducted according to equations (15) to (17), and the upper limit of the side length is approximately 43.8 mm according to equation (18). The side length of the local neighborhood is taken as 30 mm, which is approximately 1.7 times the length of the paint flow. Then, two rounds of fitting and least squares refitting are performed according to steps S44 and S45. The region with an absolute value of the true value height not less than 0.05 mm is taken as the true value region of each defect. The proportion of defects detected in the true value region of each defect is counted, and the area outside the true value region that is judged as a defect is counted.
[0093] Further reading Figure 5 It shows the point cloud height distribution before and after model undulations are removed. It can be seen that before compensation, the undulations of the field of view edge relative to the center reached a few tenths of a millimeter, and the defects only formed subtle changes in brightness on it. After compensation, the surface undulations were basically eliminated, and what remained was the overall tilt and long-wave deviation caused by the clamping. Particles, scratches and paint runs were clearly visible.
[0094] And through Figure 6The results of four detection schemes are shown, where black represents areas identified as defects, and thin solid lines represent the true value regions of each defect. In contrast, Scheme 1's reference plane cannot follow the curved surface, resulting in large areas in the center and sides of the field of view being identified as defects. The misclassified area outside the true value region is approximately 2594.9 mm². 2 In the first scheme, over 70% of the field of view was occupied by particles with a height of 0.06mm, which were completely missed because the curved surface where they were located was lower than the reference plane. In contrast, the theoretical curved surface of the second scheme did not take into account clamping tilt and offset, and approximately 80% of the field of view was judged as a defect, with a misjudged area of approximately 2930.4mm. 2 The scratches, however, were not detected because they were in the opposite direction to the deviation; compared to Scheme 3, the false detection area was reduced to 0.2 mm. 2 However, the neighborhood edge length is only 9.8 mm, and the flow paint occupies the majority of the neighborhood. When the reference surface is attached to the surface of the flow paint, the detection rate of the true value region of the flow paint is only 10.3%, the detection rates of the three particles are 54.1%, 52.3% and 0, respectively, and the scratch is 24.2%. The method of this embodiment has no misjudgment outside the true value region. The detection rates of the three particles are 91.9%, 94.5% and 100%, respectively, the scratch is 93.4%, and the flow paint is 80.5%. The undetected part of the flow paint is concentrated in the transition zone with a thickness of just over 0.05 mm at the upper edge. This is a normal boundary effect near the threshold and does not affect the identification of the flow paint as a complete defect area.
[0095] Therefore, it can be seen that subtracting model fluctuations solves the problem of misjudged curvature in Comparison Scheme 1; absorbing workpiece deviations through local neighborhoods solves the problem of deviations overshadowing defects in Comparison Scheme 2; and the expanded local neighborhoods after compensation solve the problem of paint flow causing the reference surface to rise in Comparison Scheme 3. It should be noted that the values in the above examples vary depending on the simulation conditions. Their purpose is to illustrate the coordination relationship between each step. In actual testing, the detection threshold, tolerance limits, and neighborhood edge length should be determined according to the curvature, forming accuracy, and paint surface condition of the workpiece being tested.
[0096] Example 4:
[0097] Please refer to Figure 7 and Figure 8 This embodiment discloses a curved surface paint defect detection system with robotic arm normal motion, used to execute the method described in Embodiment 1, which includes: robotic arm 1, binocular surface structured light sensor 2 and host computer.
[0098] like Figure 7As shown, the robotic arm 1 is a six-axis articulated industrial robot. Its base is fixed next to the inspection station, and the flange at its end (sixth axis) is rigidly connected to the binocular structured light sensor 2, used to carry the binocular structured light sensor 2 to each pose node and complete normal tracking. The load and working radius of the robotic arm 1 are selected according to the weight of the binocular structured light sensor 2 and the size of the workpiece 3 being measured. These are not the focus of this application and will not be elaborated further. The workpiece 3 being measured is located within the working range of the robotic arm 1. Figure 7 The diagram at defect location 31 illustrates a paint defect on the tested curved surface. This refers to the working distance.
[0099] like Figure 8 The binocular structured light sensor 2 shown includes a protective housing 21 and a projector 22, a first camera 23, and a second camera 24 disposed within the protective housing 21. The three are arranged horizontally, i.e., a two-camera-one-projector layout. The projector 22 is located in the center of the protective housing 21. For example, a high frame rate digital light processing projector can be used to complete the projection of the entire Gray code pattern and phase-shifted sine stripe pattern within 0.5 seconds. The first camera 23 and the second camera 24 are symmetrically disposed on both sides of the projector 22. Both are industrial area array cameras that support hardware triggering, and their hardware trigger terminals are connected to the synchronization signal output terminal of the projector 22.
[0100] Understandably, in order to achieve the largest possible common field of view for both cameras at their working distance, in the above-described scheme, the optical axes of both the first camera 23 and the second camera 24 are tilted towards the optical axis of the projector 22, i.e., arranged in an inward tilt. Furthermore, in a specific embodiment, the optical axes of both the first camera 23 and the second camera 24 intersect the optical axis of the projector 22 at the same position, which is located on the optical axis of the projector 22 and at a working distance from the projector 22. That is, the origin of the tool coordinate system in Example 1; in Figure 8 As shown, the inclination angle between the optical axes of both cameras and the optical axis of the projector 22 is 20°. In other embodiments, the inclination angle may be determined based on the baseline distance and the working distance. The corresponding adjustments are made so that the three optical axes still intersect at the origin of the tool coordinate system. In the specific implementation of this scheme, the arrangement of the three optical axes intersecting at the same point maximizes the overlap of the projection area, the field of view of the first camera 23, and the field of view of the second camera 24 at the working distance. Due to the normal alignment in step S1, the points on the measured surface are... Located precisely at this intersection, the central area of the field of view is simultaneously illuminated by projector 22 and seen by both cameras. Clearly, based on the aforementioned structure, the sidewalls of the concave region stagger the occlusion of the two cameras, meaning that the area occluded by one camera is often still within the field of view of the other camera, thus reducing the point cloud loss in the concave region.
[0101] Based on the above embodiments, as a further optional implementation, a first linear polarizer is provided at the light output port of the projector 22, and a second linear polarizer is provided in front of the lenses of the first camera 23 and the second camera 24. The transmission direction of the second linear polarizer is perpendicular to the transmission direction of the first linear polarizer, thereby forming an orthogonal polarization suppression structure.
[0102] It is understandable that a high-gloss paint surface will produce strong specular reflection under stripe illumination. Once the specular reflected light enters the camera, local pixels will reach saturation and stripe information will be lost. In step S31, these pixels can only be marked as invalid pixels. During acquisition, the light emitted by the projector 22 becomes linearly polarized light after passing through the first linear polarizer. After illuminating the paint surface, the specular reflection component basically maintains its original polarization direction. When it reaches the camera, it is mostly blocked by the second linear polarizer with the transmission direction perpendicular to it. The polarization direction of the diffuse reflection component formed by multiple scattering inside the paint film tends to be random. The part that is consistent with the transmission direction of the second linear polarizer can pass through and form an image, so the stripes carrying the information of the paint surface undulation are preserved.
[0103] By employing this orthogonal polarization arrangement, the number of saturated pixels is significantly reduced, and the number of pixels marked as invalid in step S31 is also reduced, thus improving the integrity of the point cloud. In a specific example, the tested paint surface is a high-gloss clear coat. Without the linear polarizer, the proportion of overexposed saturated pixels in the specular highlight area within the field of view is approximately 35% to 40%, and these pixels are all marked as invalid pixels in step S31. After assembling the above-mentioned orthogonal polarization structure and keeping the projection brightness and exposure time constant, the proportion of overexposed saturated pixels drops to below 1.2%, and the signal-to-noise ratio of the reconstructed point cloud is correspondingly improved by approximately 10 dB to 15 dB.
[0104] It should also be noted that linear polarizers reduce the light intensity entering the camera. In actual use, this can be compensated by appropriately extending the exposure time or increasing the projection brightness, as long as large-area saturation is no longer observed. In some embodiments, the first and second linear polarizers can be laminated polarizers or polarizing filters mounted in front of the lens. Both can be adjusted around the optical axis at the window of the protective housing 21 so that the transmission directions are adjusted to be perpendicular to each other during installation.
[0105] Understandably, the host computer can be an industrial control computer, which communicates with the controller of the robotic arm 1 via Ethernet using the TCP / IP protocol, and is connected to the binocular structured light sensor 2 via a trigger interface and an image data interface. The host computer runs multi-threaded processing software, in which the first thread is responsible for sending motion commands for the pose nodes to the controller of the robotic arm 1, and reading the encoder feedback speed and end pose of each joint; the second thread is responsible for sending hardware trigger pulses to the projector 22 in a stationary state, receiving images from the two cameras, and performing calculations from steps S3 to S5; the two threads run separately, so that while the robotic arm 1 moves to the next pose node, the point cloud reconstruction and defect extraction of the previous pose node are already underway.
[0106] The system needs to be calibrated as follows before use: Camera calibration involves placing a checkerboard calibration plate in different poses in front of the binocular structured light sensor 2, simultaneously acquiring multiple images from the first camera 23 and the second camera 24. The Zhang Zhengyou calibration method is used to determine the intrinsic parameters, distortion coefficients, and extrinsic parameters between the two cameras, and the epipolar correction parameters are calculated accordingly. The parameters in equation (14) are... , , , This is how it came about.
[0107] Hand-eye calibration can be exemplarily performed using the nine-point method. At least nine feature points are arranged in the workspace of the robotic arm 1. The coordinates of each feature point in the camera coordinate system are measured by the binocular structured light sensor 2. Then, the tip tool attached to the end of the robotic arm 1 touches each feature point to obtain its coordinates in the robot base coordinate system. Combining the end pose recorded during each measurement, the hand-eye transformation matrix is solved by least squares according to the transformation relationship of equation (24). In some embodiments, imaging and solving of the calibration plate at multiple end poses can also be employed. Hand-eye calibration method for formal equations.
[0108] Tool coordinate system calibration, in the hand-eye transformation matrix Once determined, the direction and working distance of the projector's 22 optical axis in the camera coordinate system are used. Determine the position and orientation of the tool coordinate system origin and the third axis in the camera coordinate system, and then... The transformation matrix between the tool coordinate system and the end coordinate system is obtained by calculation. The direction of the optical axis of the projector 22 can be obtained by connecting the three-dimensional coordinates of the light spots formed by the central pixel of the projector 22 on the planar targets at different distances.
[0109] Example 5:
[0110] This embodiment further provides an electronic device, which includes a processor and a memory. The memory stores a computer program, and when the processor executes the computer program, it implements the various steps of the curved surface paint defect detection method with robotic arm normal following described in Embodiment 1.
[0111] For example, the electronic device may be the host computer in embodiment 4, and the processor may be a central processing unit or a combination of a central processing unit and a graphics processing unit. The pixel-by-pixel phase calculation in step S3 and the compensation of each point and the fitting of each local neighborhood in step S4 are independent of each other and are suitable to be executed in parallel by the graphics processing unit.
[0112] This embodiment also provides a computer-readable storage medium storing a computer program thereon. When executed by a processor, this computer program implements the various steps of the method for detecting defects in curved paint spraying using a robotic arm with normal motion as described in Embodiment 1. The storage medium can be a read-only memory, random access memory, disk, optical disk, or solid-state drive, or any other medium capable of storing program code.
[0113] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
[0114] It should be noted that the accompanying drawings of this invention are schematic drawings, and the proportions of some structures are exaggerated. They do not constitute a limitation on the actual size and proportions, but the positional relationships, connection methods and action coordination logic between the components shown in the drawings are clear and accurate.
Claims
1. A method for detecting defects in curved surface paint spraying using a robotic arm with normal motion, characterized in that, The detection method includes: Based on the three-dimensional model of the workpiece being tested, a sequence of pose nodes is planned, and the local normal and local surface of the surface being tested at each pose node are obtained; the robotic arm is controlled to carry the binocular structured light sensor and move to each pose node in sequence, so that the optical axis of the projector in the binocular structured light sensor coincides with the local normal and maintains a preset working distance from the surface being tested. When the robotic arm is stationary, the projector projects a stripe pattern sequence, and the synchronization signal output by the projector triggers the first and second cameras to synchronously acquire deformed stripe images, while simultaneously recording the pose of the robotic arm end relative to the robot base; The absolute phase is calculated from the deformed fringe image, corresponding point matching is performed based on the absolute phase, and a three-dimensional point cloud in the camera coordinate system is obtained through triangulation. Based on the recorded pose and the pre-calibrated hand-eye transformation relationship, the local surface is transformed to the camera coordinate system, and the 3D point cloud is compensated according to the undulation of the local surface relative to the tangent plane at the pose node to obtain the compensated point cloud. The compensated point cloud is divided into multiple local neighborhoods. In each local neighborhood, local points are determined by removing points with abrupt changes in height, and local reference surfaces are fitted by the local points. Calculate the signed normal distance of each point relative to the local reference plane, perform connected clustering on points whose absolute value of the normal distance is greater than the detection threshold to obtain the paint defect region, and output the position of the paint defect region to the robot base coordinate system according to the pose and the hand-eye transformation relationship.
2. The method for detecting defects in curved surface painting using a robotic arm with normal motion according to claim 1, characterized in that, The process of planning the pose node sequence, obtaining local normals and local surfaces, and aligning the projector's optical axis with the local normals includes: The three-dimensional model is discretized into a triangular mesh, and the normal and area of each triangular facet are obtained. The pose nodes are arranged along a serpentine trajectory, and the spacing between adjacent pose nodes is determined by the field of view size of the binocular structured light sensor at the working distance and the preset overlap rate. The normals of each triangular facet within the field of view corresponding to each pose node are weighted by area to obtain the local normal at that pose node. The two principal curvatures and the corresponding principal directions at that pose node are obtained from the three-dimensional model and used as the local surface. Using the opposite direction of the local normal as the third axis direction of the tool coordinate system, and the projection direction of the scanning forward direction in the tangent plane of the pose node as the first axis direction of the tool coordinate system, the target pose at the pose node is obtained. The origin of the tool coordinate system is located on the optical axis of the projector and is at a working distance from the projector.
3. The method for detecting defects in curved surface painting using a robotic arm with normal motion according to claim 1, characterized in that, The process of projecting a stripe pattern sequence while the robotic arm is stationary and simultaneously acquiring deformed stripe images includes: When the encoder feedback speed of each joint of the robotic arm is zero, a hardware trigger pulse is sent to the projector; When triggered, the projector sequentially projects multiple Gray code patterns and multiple phase-shifted sinusoidal fringe patterns, with the fringe direction of each pattern perpendicular to the line connecting the first camera and the second camera. The projector outputs the synchronization signal when projecting each pattern, and the synchronization signal is connected to the hardware trigger terminals of the first camera and the second camera.
4. The method for detecting defects in curved surface painting using a robotic arm with normal motion according to claim 3, characterized in that, The process of solving for the absolute phase and obtaining the three-dimensional point cloud through triangulation includes: The wrapping phase, background light intensity, and modulation degree of each pixel are solved from the deformed fringe image corresponding to the phase-shifted sinusoidal fringe pattern, and pixels with a modulation degree lower than a preset modulation degree threshold are marked as invalid pixels. Using the background light intensity as a binarization reference, the deformed stripe image corresponding to the Gray code pattern is binarized and decoded to obtain the stripe level of each pixel. The absolute phase is obtained from the stripe level and the wrapping phase. Epipolar correction is performed on the images of the first camera and the second camera. For each effective pixel in the first camera image, a position with absolute phase equality is searched on the same epipolar line in the second camera image. Subpixel matching positions are obtained by interpolation of adjacent pixels. Triangulation is performed based on the disparity obtained by matching.
5. The method for detecting defects in curved surface painting using a robotic arm with normal motion according to claim 2, characterized in that, The compensation of the 3D point cloud includes: The position of the pose node, the local normal, and the two principal directions are transformed into the camera coordinate system through the recorded pose and hand-eye transformation relationship to obtain the reference position, reference normal, and reference principal direction; The position vectors of each point in the three-dimensional point cloud relative to the reference point are projected onto the two reference principal directions to obtain the tangential coordinates of each point. The undulation of the local surface at that point relative to the tangential plane is calculated using the two principal curvatures and the tangential coordinates, and the point is translated along the reference normal by the undulation to obtain the compensated point cloud.
6. The method for detecting defects in curved surface painting using a robotic arm with normal motion according to claim 5, characterized in that, The step of determining local points within each local neighborhood and fitting a local reference surface includes: The side length of the local neighborhood is no greater than the upper limit value, which is determined by the preset tolerance lower limit and the allowable deviation of the actual curvature of the workpiece under test relative to the principal curvature. The first round of fitting is performed using the upper limit of the tolerance as the distance tolerance. The distance tolerance of the local neighborhood is determined based on the dispersion of the normal distance of each point obtained from the first round of fitting, and the distance tolerance is made to be between the lower limit of the tolerance and the upper limit of the tolerance. A second round of fitting is performed using this distance tolerance, and the local points obtained from the second round of fitting are refitted using the least squares method to obtain the local reference surface; The positive direction of the signed normal distance is the same as that of the reference normal, and the detection threshold is not less than the upper tolerance limit.
7. The method for detecting defects in curved surface painting using a robotic arm with normal motion according to claim 1, characterized in that, The connected clustering and mapping to the robot base coordinate system includes: The three-dimensional point cloud is organized according to the pixel arrangement of the first camera. Points with an absolute value of normal distance greater than the detection threshold are clustered by eight neighborhoods. Based on the sign of the normal distance in each connected region, they are determined to be protrusion defects or depression defects. Connected regions with a projected area smaller than a preset area threshold and a bounding box volume smaller than a preset volume threshold are removed. The average coordinates of each point in the three-dimensional point cloud within the paint defect area are used as the centroid of the defect. The centroid of the defect is then transformed sequentially to the robot base coordinate system through the hand-eye transformation relationship and the recorded pose. The normal of the local reference plane corresponding to the paint defect area is transformed to the robot base coordinate system by the same rotation, and output together with the coordinates of the defect centroid.
8. A curved surface paint defect detection system with robotic arm normal motion, characterized in that, The detection system includes: robotic arm; A binocular structured light sensor is installed at the end of the robotic arm and includes a protective housing and a projector, a first camera, and a second camera disposed within the protective housing. The first camera and the second camera are symmetrically disposed on both sides of the projector. A host computer is communicatively connected to the robotic arm and the binocular structured light sensor, and the host computer is configured to perform a method for detecting defects in curved paint spraying by normal motion of a robotic arm as described in any one of claims 1 to 7.
9. A curved surface paint defect detection system with robotic arm normal motion according to claim 8, characterized in that, The projector has a first linear polarizer at its light output port, and a second linear polarizer is provided in front of the lenses of both the first and second cameras. The transmission direction of the second linear polarizer is perpendicular to the transmission direction of the first linear polarizer.
10. A curved surface paint defect detection system with robotic arm normal motion according to claim 8, characterized in that, The optical axes of the first camera and the second camera are both tilted toward the optical axis of the projector, and both intersect the optical axis of the projector at the same point.
