Projection marking method based on visual sensing

Abstract

The invention relates to a projection marking method based on visual sensing, belongs to the technical field of projection marking, and solves the problem of low marking precision of a projection marking method in the prior art. Comprising the following steps: calibrating parameters of a projection marking system based on a calibration board; wherein the projection marking system comprises a camera and a projector; randomly and non-uniformly arranging a plurality of mark points on the measured workpiece, measuring to obtain a point cloud of the measured workpiece, and collecting an image of the measured workpiece through the projection marking system; wherein a color representing errors is rendered in the point cloud of the measured workpiece, and mark points are marked on the point cloud; based on the acquired image of the measured workpiece, the point cloud of the measured workpiece and the parameters of the projection marking system, the pose of the point cloud of the measured workpiece is adjusted, projection is carried out on the measured workpiece, and projection marking of an error distribution diagram is realized. And high-precision projection marking is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of projection marking, and in particular to a projection marking method based on visual sensing. Background Art

[0002] In the context of the digital wave in the global manufacturing industry, intelligent manufacturing has attracted widespread attention as a key force to promote industrial upgrading. Intelligent manufacturing is supported by advanced information technology. Through the deep integration of technologies such as the Internet of Things, big data, and artificial intelligence, it has realized the digitization, automation, and intelligence of the production process, bringing revolutionary changes to the manufacturing industry. Intelligent detection is an important part of intelligent manufacturing. It uses advanced sensor technology and data analysis algorithms to achieve efficient, accurate, and automated detection of workpieces. Current research often focuses on how to organically combine multi-source heterogeneous sensors, big data processing, and artificial intelligence algorithms to build a more intelligent, efficient, and adaptive detection system. However, the measurement results and error analysis reports obtained from the detection cannot be directly matched with the actual detected workpieces to analyze errors or guide subsequent processing equipment.

[0003] Projection marking technology is a type of spatial augmented reality (SAR). Compared with spatial augmented reality technology in the general sense, projection marking technology emphasizes display accuracy more and focuses on its integration with the production and manufacturing fields. When applied to industrial production and manufacturing, workers do not need to wear any equipment and can obtain the required digital information with their naked eyes, and it meets the requirements of multi-person collaboration and long-term work.

[0004] However, when the existing projection marking method solves the position information of the projector to the workpiece coordinates, the projector calibration accuracy is low and mismatching is prone to occur when local point cloud features match global features, resulting in low projection marking accuracy and difficulty in meeting the increasingly stringent requirements of workpiece manufacturing. Summary of the invention

[0005] In view of the above analysis, an embodiment of the present invention aims to provide a projection marking method based on visual sensing to solve the problem of low marking accuracy of the existing projection marking method.

[0006] The embodiment of the present invention provides a projection marking method based on visual sensing, comprising the following steps:

[0007] Calibrate the parameters of the projection marking system based on the calibration plate; wherein the projection marking system includes a camera and a projector;

[0008] A plurality of marking points are randomly and non-uniformly arranged on the workpiece to be measured, a point cloud of the workpiece to be measured is obtained by measurement, and an image of the workpiece to be measured is collected by the projection marking system; wherein the point cloud of the workpiece to be measured is rendered with a color representing an error and is marked with marking points;

[0009] Based on the acquired image of the workpiece, the point cloud of the workpiece and the parameters of the projection marking system, the position and posture of the point cloud of the workpiece is adjusted, and then projection is performed on the workpiece to realize the projection marking of the error distribution map.

[0010] Further, the parameters of the projection marking system include the internal and external parameters of the camera, the internal and external parameters of the projector, and the external parameters between the camera and the projector; the adjusting the position and posture of the point cloud of the measured workpiece based on the collected image of the measured workpiece, the point cloud of the measured workpiece and the parameters of the projection marking system includes:

[0011] Based on the acquired image of the workpiece being measured, the center coordinates of each mark point in the camera coordinate system and the projector coordinate system are obtained;

[0012] Based on the center coordinates of each marker point in the camera coordinate system and the projector coordinate system, as well as the intrinsic and extrinsic parameters of the camera and the intrinsic and extrinsic parameters of the projector, the three-dimensional coordinates of the center of each marker point in the camera coordinate system are obtained, and each three-dimensional marker point is reconstructed;

[0013] Based on the reconstructed three-dimensional landmark points and the landmark points in the point cloud of the measured workpiece, the point cloud of the measured workpiece in the camera coordinate system is obtained;

[0014] Based on the point cloud of the workpiece under test in the camera coordinate system and the external parameters between the camera and the projector, the point cloud of the workpiece under test in the projector coordinate system is obtained, and the position and posture adjustment of the point cloud of the workpiece under test is realized.

[0015] Furthermore, the step of obtaining the point cloud of the workpiece under the camera coordinate system based on the reconstructed three-dimensional marker points and the marker points in the point cloud of the workpiece under test includes:

[0016] Matching the three-dimensional landmark points reconstructed in the camera coordinate system with the landmark points in the point cloud of the workpiece to obtain matched landmark point pairs;

[0017] Based on the matched landmark point pairs, the transformation relation of the point cloud of the workpiece under test is obtained;

[0018] Based on the transformation relation of the point cloud of the workpiece to be measured, the point cloud of the workpiece to be measured in the camera coordinate system is obtained.

[0019] Furthermore, methods for matching the three-dimensional marker points reconstructed in the camera coordinate system with the marker points in the point cloud of the workpiece under test include manual matching, RANSAC algorithm, SIFT or brute force matching.

[0020] Furthermore, based on the matched landmark point pairs, the transformation relation of the point cloud of the workpiece under test is obtained by an iterative closest point algorithm.

[0021] Furthermore, the parameters of the projection marking system are calibrated based on the calibration plate in the following manner:

[0022] The calibration plate is set to multiple positions and postures to obtain the center coordinates of each feature graphic in one-to-one correspondence between the camera coordinate system and the projector coordinate system in each position;

[0023] Based on the one-to-one correspondence between the camera coordinate system and the projector coordinate system at each pose, the initial calibration values ​​of the projection marking system parameters are obtained by using the Zhang Zhengyou calibration method and the binocular stereo calibration method.

[0024] Based on the bundle adjustment optimization, the nonlinear optimization equations of the projection marking system parameters are constructed. Then, based on the initial calibration values ​​of the projection marking system parameters, the optimal values ​​of the projection marking system parameters are obtained through the LM iteration method to complete the calibration of the projection marking system parameters.

[0025] Furthermore, the one-to-one corresponding center coordinates of each feature graphic in the camera coordinate system and the projector coordinate system at each posture are obtained by the following method, including:

[0026] The projector of the projection marking system projects a grating stripe image and a pure white image onto the calibration plate of the current position, and the corresponding image is collected by the camera;

[0027] Based on the collected images, the feature pattern extraction algorithm and phase calculation are used to obtain the center coordinates of each feature pattern of the calibration plate in the camera coordinate system and the projector coordinate system;

[0028] The center coordinates of each characteristic pattern of the calibration plate in the camera coordinate system and the projector coordinate system are sorted and matched to obtain the one-to-one corresponding center coordinates of each characteristic pattern in the camera coordinate system and the projector coordinate system.

[0029] Furthermore, the center coordinates of each characteristic pattern of the calibration plate in the camera coordinate system and the projector coordinate system are obtained by the following method:

[0030] Based on the pure white image in the captured image, the contours of each feature pattern of the calibration plate are obtained through the feature pattern extraction algorithm, and then the center coordinates and sizes of each feature pattern in the camera coordinate system are obtained;

[0031] Based on the grating fringe images in the collected image, the central phase value and size of each characteristic pattern of the calibration plate are obtained through the multi-frequency heterodyne principle and the four-step phase shift algorithm;

[0032] Based on the central phase value of each characteristic pattern of the calibration plate and the projector image resolution, the central coordinates of each characteristic pattern of the calibration plate in the projector coordinate system are obtained.

[0033] Furthermore, the initial calibration values ​​of the projection marking system parameters are obtained based on the Zhang Zhengyou calibration method and the binocular stereo calibration method, including:

[0034] Construct a world coordinate system and obtain the center coordinates of each characteristic graphic of the calibration plate under the world coordinate system;

[0035] Based on the center coordinates of each feature graphic in the camera coordinate system and the corresponding center coordinates of the feature graphic in the world coordinate system, the camera's intrinsic and extrinsic parameters are obtained by Zhang Zhengyou's calibration method.

[0036] Based on the center coordinates of each feature graphic in the projector coordinate system and the corresponding center coordinates of the feature graphic in the world coordinate system, the internal and external parameters of the projector are obtained by Zhang Zhengyou calibration method;

[0037] Based on the camera extrinsic parameters and the projector extrinsic parameters, the extrinsic parameters between the camera and the projector are obtained through the binocular stereo calibration method.

[0038] Furthermore, the optimal values ​​of the projection marking system parameters are obtained by:

[0039] According to the initial calibration values ​​of the coordinates of the center points of each feature graphic and the projection marking system parameters in the camera coordinate system, the projector coordinate system and the world coordinate system, a reprojection error equation is constructed;

[0040] The cost function is constructed based on the reprojection error equation. The cost function is nonlinearly optimized based on the LM iteration method with the goal of minimizing the reprojection error to obtain the optimal values ​​of the projection marking system parameters.

[0041] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:

[0042] The present invention provides a projection marking method based on visual sensing. The method calibrates the parameters of the projection marking system based on a calibration plate, randomly and non-uniformly arranges multiple marking points on the workpiece to be measured, measures the point cloud of the workpiece to be measured, and collects the image of the workpiece to be measured through the projection marking system. Based on the collected image of the workpiece to be measured, the point cloud of the workpiece to be measured and the parameters of the projection marking system, the posture of the point cloud of the workpiece to be measured is adjusted, and then projection is performed on the workpiece to achieve projection marking of the error distribution map. The method combines the structured light measurement principle and the marking point feature matching method, reconstructs the three-dimensional coordinates of the marking points by actively projecting the grating stripe image, effectively reduces the number of marking points, and matches the features of the local high-precision marking points with the global marking points, thereby improving the matching efficiency and the projection marking accuracy. The workpiece to be measured is identified by a camera, and the posture information from the projector to the workpiece coordinates is solved, so that the error distribution map is directly displayed on the workpiece surface, providing the operator with clearer and more intuitive guidance, which is helpful to guide the subsequent processing and assembly, thereby improving product quality and production efficiency.

[0043] In the present invention, the above-mentioned technical solutions can also be combined with each other to achieve more preferred combination solutions. Other features and advantages of the present invention will be described in the subsequent description, and some advantages can become obvious from the description, or can be understood by practicing the present invention. The purpose and other advantages of the present invention can be achieved and obtained through the contents particularly pointed out in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The drawings are only for the purpose of illustrating particular embodiments and are not to be considered limiting of the present invention. Like reference symbols denote like components throughout the drawings.

[0045] Figure 1 A schematic flow chart of a projection marking method based on visual sensing provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0046] The preferred embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not used to limit the scope of the present invention.

[0047] A specific embodiment of the present invention discloses a projection marking method based on visual sensing, such as Figure 1 As shown, the following steps are included:

[0048] S1. Calibrate parameters of a projection marking system based on a calibration plate; wherein the projection marking system includes a camera and a projector.

[0049] Specifically, the camera in the projection marking system includes a lens, and the camera and the projector are fixed by a fixing device.

[0050] Specifically, the calibration plate is composed of multiple characteristic patterns, the intermediate spacing between adjacent characteristic patterns is the same, and the characteristic patterns are arranged in rows and columns; wherein the calibration plate contains characteristic patterns of two sizes; wherein the number of characteristic patterns of larger size is 5, and they are unevenly distributed in the calibration plate.

[0051] Preferably, the characteristic pattern is a circle.

[0052] Specifically, the parameters of the projection marking system include the intrinsic and extrinsic parameters of the camera, the intrinsic and extrinsic parameters of the projector, and the extrinsic parameters between the camera and the projector.

[0053] During implementation, the parameters of the projection marking system are calibrated based on the calibration plate in the following way:

[0054] S11, setting the calibration plate to multiple positions and postures, and obtaining the center coordinates of each characteristic graphic in a one-to-one correspondence between the camera coordinate system and the projector coordinate system in each position;

[0055] Specifically, different postures are obtained by moving the calibration plate so that it faces, approaches, moves away from, tilts, and rotates relative to the projection marking system.

[0056] Specifically, the calibration plate has at least four postures.

[0057] In specific implementation, the one-to-one corresponding center coordinates of each feature graphic in the camera coordinate system and the projector coordinate system at each posture are obtained by the following methods, including:

[0058] A1. Use the projector of the projection marking system to project the grating fringe images and pure white images of the set conditions onto the calibration plate of the current posture, and use the camera to collect the corresponding images;

[0059] Specifically, the calibration plate is moved to make the calibration plate image clear in the camera lens of the projection marking system, and then the projector projects various grating stripe images of set conditions and a pure white image into the calibration plate space, and the camera is controlled to collect each image; wherein the grating stripe image includes horizontal and vertical stripe images.

[0060] More specifically, each grating stripe image satisfies three-frequency four-step phase shift, that is, there are 12 horizontal and vertical stripe images, which are images of four phase shifts at three frequencies.

[0061] Preferably, in this embodiment, 0, π / 2. π and 3π / 2 are the phase shifts of the four-step phase-shifted grating fringe images, and the three frequencies are 1 / 114, 1 / 109, 1 / 103.

[0062] A2. Based on the acquired image, the feature pattern extraction algorithm and phase calculation are used to obtain the center coordinates of each feature pattern of the calibration plate in the camera coordinate system and the projector coordinate system; specifically:

[0063] A21. Based on the pure white image in the acquired image, the contours of each feature pattern of the calibration plate are obtained through the feature pattern extraction algorithm, and then the center coordinates and sizes of each feature pattern in the camera coordinate system are obtained;

[0064] A22. Based on the grating fringe images in the collected image, the central phase value and size of each characteristic pattern of the calibration plate are obtained through the multi-frequency heterodyne principle and the four-step phase shift algorithm;

[0065] A23. Based on the central phase value of each characteristic pattern of the calibration plate and the projector image resolution, the central coordinates of each characteristic pattern of the calibration plate in the projector coordinate system are obtained.

[0066] In this embodiment, taking the characteristic pattern on the calibration plate as a circle as an example, the center coordinates of each characteristic pattern on the calibration plate in the camera coordinate system and the projector coordinate system are obtained by the following method:

[0067] (1) Based on the pure white image in the acquired image, the contours of each characteristic circle of the calibration plate are obtained through the ellipse extraction algorithm, and then the center coordinates and diameters of each characteristic circle in the camera coordinate system are obtained:

[0068] The pure white image is binarized to obtain a binarized image; wherein the binarization process adopts the maximum inter-class variance method;

[0069] The binary image is corroded to obtain the corroded image; corrosion can eliminate noise, refine the structure, eliminate small objects, smooth the boundaries, and reduce holes;

[0070] The binary image is subtracted from the eroded image, and the contour is extracted using the morphological method;

[0071] Compare the length of each contour with the set threshold. If the contour length is less than the set threshold, the contour is determined to be a noise point and deleted.

[0072] Perform the least square ellipse fitting on each contour to obtain the initial coordinates of the center of each characteristic circle;

[0073] Based on the initial coordinates of each characteristic circle and the integer pixel contour, the grayscale weighted method is used to extract the sub-pixel coordinates of each contour, and then the least squares ellipse fitting is performed again to obtain the coordinates of the center of each characteristic circle;

[0074] All integer pixel contours are fitted with the least squares circle to obtain the diameter of each characteristic circle.

[0075] (2) Based on the grating fringe images in the collected image, the central phase value and size of each characteristic pattern of the calibration plate are obtained through the multi-frequency heterodyne principle and the four-step phase shift algorithm:

[0076] Based on the grating fringe images in the collected image, a four-step phase shift algorithm is used to obtain the wrapped phase of each pixel at each frequency;

[0077] Based on the wrapped phase of each pixel at each frequency, the triple-frequency heterodyne principle is used to unwrap the phase to obtain the absolute phase and obtain the phase map;

[0078] Based on the phase image, the ellipse extraction algorithm is used to obtain the center phase value and diameter of each characteristic circle.

[0079] It can be understood that, from the reversibility of the optical path, the projector projecting a pattern and the camera taking an image are opposite processes, so the projector can be calibrated as a reverse camera. In order to give the projector the ability to take pictures, the phase value is used in this embodiment to establish an accurate correspondence between the camera and the projector image. High-precision phase calculation is the basis for achieving high-precision stereo matching and projector calibration. The purpose of phase calculation is to make each pixel on the image unique so as to establish a correspondence with the pixel of another image. Compared with the digital image correlation algorithm that measures the similarity of grayscale values ​​between sub-regions by correlation coefficients, stereo matching based on phase values ​​has higher accuracy and stability, and is more resistant to the influence of ambient light and surface texture of the object being measured. In this embodiment, the phase shift principle is first used to extract the phase principal value corresponding to each pixel from a series of grating fringe images with a constant phase difference; after calculating the phase principal value, in order to obtain a continuous and monotonic absolute phase value, the multi-frequency heterodyne method is used to phase unfold the phase principal value, and the calculated continuous phase value will be used for subsequent projector calibration.

[0080] The phase shift principle is to calculate the main phase value of the surface of the object being measured by projecting and collecting multiple grating fringe images with constant phase difference. This embodiment uses a standard four-step phase shift algorithm to calculate the main phase value of the grating fringe image, and selects a three-frequency heterodyne with higher stability and accuracy for phase unwrapping to obtain a phase map.

[0081] (3) Based on the phase value and the projector image resolution, the center coordinates of each characteristic pattern of the calibration plate in the projector coordinate system are obtained.

[0082] A3. Sort and match the center coordinates of each characteristic pattern of the calibration plate in the camera coordinate system and the projector coordinate system to obtain the one-to-one corresponding center coordinates of each characteristic pattern in the camera coordinate system and the projector coordinate system.

[0083] Specifically, the center coordinates of each characteristic pattern of the calibration plate in the camera coordinate system or the projector coordinate system are sorted in the following manner:

[0084] Sort the characteristic patterns according to their sizes and determine five characteristic patterns with larger sizes;

[0085] According to the relative relationship between the actual positions of the five larger characteristic figures on the calibration plate and the relative relationship between the distances of the five larger characteristic figures on the calibration plate in the current coordinate system, the numbers of the characteristic figures in the corresponding coordinate system are obtained; among which, the actual numbers of the five larger characteristic figures on the calibration plate are known.

[0086] Specifically, by performing one-to-one matching on the numbers of the characteristic patterns in the camera coordinate system and the projector coordinate system, the one-to-one corresponding center coordinates of the characteristic patterns in the camera coordinate system and the projector coordinate system are obtained.

[0087] S12, based on the one-to-one correspondence between the center coordinates of each feature graphic in the camera coordinate system and the projector coordinate system in each posture, the Zhang Zhengyou calibration method and the binocular stereo calibration method are used to obtain the initial calibration values ​​of the projection marking system parameters; specifically:

[0088] B1. Construct a world coordinate system and obtain the center coordinates of each characteristic figure of the calibration plate under the world coordinate system;

[0089] B2. Based on the center coordinates of each feature graphic in the camera coordinate system and the corresponding center coordinates of the feature graphic in the world coordinate system, the initial calibration values ​​of the camera's intrinsic and extrinsic parameters are obtained by the Zhang Zhengyou calibration method;

[0090] B3. Based on the center coordinates of each feature graphic in the projector coordinate system and the corresponding center coordinates of the feature graphic in the world coordinate system, the initial calibration values ​​of the intrinsic and extrinsic parameters of the projector are obtained by the Zhang Zhengyou calibration method;

[0091] B4. Based on the camera extrinsic parameters and the projector extrinsic parameters, the initial calibration value of the extrinsic parameters between the camera and the projector is obtained through the binocular stereo calibration method.

[0092] Specifically, a world coordinate system is established with the center of the characteristic pattern at the upper left corner of the calibration plate as the origin and two directions parallel to the edges of the calibration plate as coordinate axes, and the coordinates of the center of each characteristic pattern are obtained based on the center distance of each characteristic pattern.

[0093] S13, based on the bundle method adjustment optimization, construct the nonlinear optimization equation of the projection marking system parameters, and then obtain the optimal value of the projection marking system parameters through the LM iteration method based on the initial calibration value of the projection marking system parameters, and complete the calibration of the projection marking system parameters; specifically:

[0094] C1. Construct the reprojection error equation according to the coordinates of the center points of each feature graphic and the projection marking system parameters in the camera coordinate system, projector coordinate system and world coordinate system;

[0095] C2. Construct a cost function based on the reprojection error equation, with the goal of minimizing the reprojection error. The initial calibration values ​​of the projection marking system parameters are the initial values. The cost function is nonlinearly optimized based on the LM iteration method to obtain the optimal values ​​of the projection marking system parameters.

[0096] Specifically, the reprojection error is the sum of the camera reprojection error, the projector reprojection error, and the camera-projector reprojection error; where,

[0097] The camera reprojection error is the difference between the coordinates of the center points of each feature graphic in the world coordinate system calculated based on the intrinsic and extrinsic parameters of the camera, converted to the coordinates of the center points of each feature graphic in the camera coordinate system, and the coordinates of the center points of each feature graphic in the camera coordinate system actually measured;

[0098] The projector reprojection error is the difference between the coordinates of the center points of each feature graphic in the world coordinate system calculated based on the internal and external parameters of the projector and converted to the coordinates of the center points of each feature graphic in the projector coordinate system and the coordinates of the center points of each feature graphic in the projector coordinate system actually measured;

[0099] The reprojection error between camera and projector is obtained by converting the coordinates of the center points of each feature graphic in the world coordinate system calculated based on the intrinsic and extrinsic parameters of the camera to the coordinates of the center points of each feature graphic in the camera coordinate system, and then calculating the difference between the coordinates of the center points of each feature graphic in the projector coordinate system and the actual measured coordinates of the center points of each feature graphic in the projector coordinate system through the extrinsic parameters between camera and projector.

[0100] Specifically, the cost function is the sum of squares of the reprojection errors.

[0101] S2. A plurality of marking points are randomly and non-uniformly arranged on the workpiece to be measured, a point cloud of the workpiece to be measured is obtained by measurement, and an image of the workpiece to be measured is collected through the projection marking system; wherein the point cloud of the workpiece to be measured is rendered with a color representing an error and is marked with marking points.

[0102] Specifically, the number of marking points arranged on the workpiece to be measured is at least 3. It should be noted that the marking points are arranged on the workpiece to be measured by sticking the marking points on the workpiece to be measured in the form of stickers.

[0103] Specifically, the shape of the marking point is the same as the shape of the characteristic pattern on the positioning plate, and the image of the workpiece to be measured is collected through the same process as step A1.

[0104] Specifically, the point cloud of the workpiece is obtained by the following method:

[0105] S21. Obtain the original point cloud of the workpiece being measured by a point cloud measurement method.

[0106] More specifically, point cloud measurement methods include three-coordinate measurement, optical measurement, and acoustic measurement.

[0107] S22, performing digital model analysis on the original point cloud of the workpiece being measured and the CAD model of the workpiece being measured to obtain a point cloud deviation.

[0108] S23, rendering the original point cloud of the workpiece to be measured in color based on the point cloud deviation to obtain the point cloud of the workpiece to be measured.

[0109] S3. Based on the acquired image of the workpiece, the point cloud of the workpiece and the parameters of the projection marking system, the position and posture of the point cloud of the workpiece is adjusted, and then projected onto the workpiece to realize the projection marking of the error distribution map.

[0110] During implementation, adjusting the position and posture of the point cloud of the workpiece to be measured based on the collected image of the workpiece to be measured, the point cloud of the workpiece to be measured and the parameters of the projection marking system includes:

[0111] S31. Based on the acquired image of the workpiece being measured, the center coordinates of each marking point in the camera coordinate system and the projector coordinate system are obtained.

[0112] Specifically, the shape of the marking point is the same as the shape of the characteristic pattern on the positioning plate, and the center coordinates of each marking point in the camera coordinate system and the projector coordinate system are obtained through the same process as step A2.

[0113] S32. Based on the center coordinates of each marker point in the camera coordinate system and the projector coordinate system, as well as the intrinsic and extrinsic parameters of the camera and the intrinsic and extrinsic parameters of the projector, the three-dimensional coordinates of the center of each marker point in the camera coordinate system are obtained, and each three-dimensional marker point is reconstructed.

[0114] Specifically, each three-dimensional landmark point is reconstructed according to the triangulation principle.

[0115] S33, based on the reconstructed three-dimensional marker points and the marker points in the point cloud of the measured workpiece, obtain the point cloud of the measured workpiece in the camera coordinate system. Specifically:

[0116] a1. Match the three-dimensional landmark points reconstructed in the camera coordinate system with the landmark points in the point cloud of the workpiece to obtain matched landmark point pairs.

[0117] Specifically, methods for matching each three-dimensional marker point reconstructed in the camera coordinate system with the marker points in the point cloud of the workpiece under test include manual matching, RANSAC algorithm, SIFT or brute force matching.

[0118] a2. Based on the matched landmark point pairs, the transformation relation of the point cloud of the workpiece is obtained.

[0119] Specifically, based on the matched landmark point pairs, the transformation relation of the point cloud of the workpiece to be measured is obtained by an iterative closest point algorithm.

[0120] More specifically, the transformation relation of the point cloud of the workpiece being measured is expressed as:

[0121] p i =Rp i ′+t

[0122] In the formula, p i 、p i ′ respectively represent the three-dimensional coordinates of the reconstructed three-dimensional marker point in the i-th marker point pair and the three-dimensional coordinates of the marker point in the point cloud of the measured workpiece, R represents the rotation deviation, and t represents the translation deviation.

[0123] It should be noted that, with the goal of minimizing the error between the marker points, the least square method is used to optimize and adjust the translation deviation and rotation deviation, so as to obtain the optimal translation deviation and rotation deviation;

[0124] The error E between the landmark points is expressed as:

[0125]

[0126] in,

[0127] R=UV T , t = p - Rp′;

[0128]

[0129] Where U and V T are the left singular vector matrix and the right singular vector matrix of the covariance matrix H, respectively. n represents the number of landmark pairs. || || 2 2 represents the 2-norm.

[0130] Furthermore, singular value decomposition (SVD) is performed on the covariance matrix W to obtain the left singular vector matrix U and the right singular vector matrix V T ; Among them, the covariance matrix W is expressed as:

[0131] W=U∑V T

[0132] in,

[0133]

[0134] q i =p i -p,q i ′=p i ′-p′

[0135] Where ∑ represents the singular value matrix of the covariance matrix W.

[0136] a3. Based on the transformation relation of the point cloud of the workpiece, the point cloud of the workpiece under the camera coordinate system is obtained.

[0137] S34. Based on the point cloud of the workpiece under test in the camera coordinate system and the external parameters between the camera and the projector, the point cloud of the workpiece under test in the projector coordinate system is obtained to achieve position and posture adjustment of the point cloud of the workpiece under test.

[0138] Compared with the prior art, this embodiment provides a projection marking method based on visual sensing, which calibrates the parameters of the projection marking system based on a calibration plate, randomly and non-uniformly arranges multiple marking points on the workpiece to be measured, measures the point cloud of the workpiece to be measured, and collects the image of the workpiece to be measured through the projection marking system. Based on the collected image of the workpiece to be measured, the point cloud of the workpiece to be measured and the parameters of the projection marking system, the posture of the point cloud of the workpiece to be measured is adjusted, and then projected on the workpiece to achieve projection marking of the error distribution map. This method combines the structured light measurement principle and the marking point feature matching method, reconstructs the three-dimensional coordinates of the marking points by actively projecting the grating fringe image, effectively reduces the number of marking points, and matches the features of the local high-precision marking points with the global marking points, thereby improving the matching efficiency and projection marking accuracy; the workpiece to be measured is identified by a camera, and the posture information from the projector to the workpiece coordinates is solved, so that the error distribution map is directly displayed on the workpiece surface, providing the operator with clearer and more intuitive guidance, which is helpful to guide subsequent processing and assembly, thereby improving product quality and production efficiency.

[0139] Those skilled in the art will appreciate that all or part of the processes of the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, wherein the computer-readable storage medium is a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0140] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. A projection marking method based on visual sensing, characterized in that: The following steps are involved: Calibrate the parameters of the projection marking system based on the calibration plate; wherein the projection marking system includes a camera and a projector; A plurality of marking points are randomly and non-uniformly arranged on the workpiece to be measured, a point cloud of the workpiece to be measured is obtained by measurement, and an image of the workpiece to be measured is collected by the projection marking system; wherein the point cloud of the workpiece to be measured is rendered with a color representing an error and is marked with marking points; Based on the acquired image of the workpiece, the point cloud of the workpiece and the parameters of the projection marking system, the position and posture of the point cloud of the workpiece is adjusted, and then projection is performed on the workpiece to realize the projection marking of the error distribution map.

2. The projection marking method based on visual sensing according to claim 1, characterized in that: The parameters of the projection marking system include the internal and external parameters of the camera, the internal and external parameters of the projector, and the external parameters between the camera and the projector; the posture of the point cloud of the measured workpiece is adjusted based on the collected image of the measured workpiece, the point cloud of the measured workpiece and the parameters of the projection marking system, including: Based on the acquired image of the workpiece being measured, the center coordinates of each mark point in the camera coordinate system and the projector coordinate system are obtained; Based on the center coordinates of each marker point in the camera coordinate system and the projector coordinate system, as well as the intrinsic and extrinsic parameters of the camera and the intrinsic and extrinsic parameters of the projector, the three-dimensional coordinates of the center of each marker point in the camera coordinate system are obtained, and each three-dimensional marker point is reconstructed; Based on the reconstructed three-dimensional landmark points and the landmark points in the point cloud of the measured workpiece, the point cloud of the measured workpiece in the camera coordinate system is obtained; Based on the point cloud of the workpiece under test in the camera coordinate system and the external parameters between the camera and the projector, the point cloud of the workpiece under test in the projector coordinate system is obtained, and the position and posture adjustment of the point cloud of the workpiece under test is realized.

3. The projection marking method based on visual sensing according to claim 2, characterized in that: The step of obtaining the point cloud of the workpiece under the camera coordinate system based on the reconstructed three-dimensional mark points and the mark points in the point cloud of the workpiece under test comprises: Matching the three-dimensional landmark points reconstructed in the camera coordinate system with the landmark points in the point cloud of the workpiece to obtain matched landmark point pairs; Based on the matched landmark point pairs, the transformation relation of the point cloud of the workpiece under test is obtained; Based on the transformation relation of the point cloud of the workpiece to be measured, the point cloud of the workpiece to be measured in the camera coordinate system is obtained.

4. The projection marking method based on visual sensing according to claim 3 is characterized in that: Methods for matching the three-dimensional landmark points reconstructed in the camera coordinate system with the landmark points in the point cloud of the workpiece under test include manual matching, RANSAC algorithm, SIFT or brute force matching.

5. The projection marking method based on visual sensing according to claim 3 is characterized in that: Based on the matched landmark points, the transformation relation of the point cloud of the workpiece is obtained through the iterative closest point algorithm.

6. The projection marking method based on visual sensing according to claim 2, characterized in that: The parameters of the projection marking system are calibrated based on the calibration plate in the following ways: The calibration plate is set to multiple positions and postures to obtain the center coordinates of each feature graphic in one-to-one correspondence between the camera coordinate system and the projector coordinate system in each position; Based on the one-to-one correspondence between the camera coordinate system and the projector coordinate system at each pose, the initial calibration values ​​of the projection marking system parameters are obtained by using the Zhang Zhengyou calibration method and the binocular stereo calibration method. Based on the bundle adjustment optimization, the nonlinear optimization equations of the projection marking system parameters are constructed. Then, based on the initial calibration values ​​of the projection marking system parameters, the optimal values ​​of the projection marking system parameters are obtained through the LM iteration method to complete the calibration of the projection marking system parameters.

7. The projection marking method based on visual sensing according to claim 6, characterized in that: The one-to-one corresponding center coordinates of each feature graphic in the camera coordinate system and the projector coordinate system at each pose are obtained by the following methods, including: The projector of the projection marking system projects a grating stripe image and a pure white image onto the calibration plate of the current position, and the corresponding image is collected by the camera; Based on the collected images, the feature pattern extraction algorithm and phase calculation are used to obtain the center coordinates of each feature pattern of the calibration plate in the camera coordinate system and the projector coordinate system; The center coordinates of each characteristic pattern of the calibration plate in the camera coordinate system and the projector coordinate system are sorted and matched to obtain the one-to-one corresponding center coordinates of each characteristic pattern in the camera coordinate system and the projector coordinate system.

8. The projection marking method based on visual sensing according to claim 7, characterized in that: The center coordinates of each feature graphic of the calibration plate in the camera coordinate system and the projector coordinate system are obtained by the following method: Based on the pure white image in the captured image, the contours of each feature pattern of the calibration plate are obtained through the feature pattern extraction algorithm, and then the center coordinates and sizes of each feature pattern in the camera coordinate system are obtained; Based on the grating fringe images in the collected image, the central phase value and size of each characteristic pattern of the calibration plate are obtained through the multi-frequency heterodyne principle and the four-step phase shift algorithm; Based on the central phase value of each characteristic pattern of the calibration plate and the projector image resolution, the central coordinates of each characteristic pattern of the calibration plate in the projector coordinate system are obtained.

9. The projection marking method based on visual sensing according to claim 6, characterized in that: The initial calibration values ​​of the projection marking system parameters obtained based on the Zhang Zhengyou calibration method and the binocular stereo calibration method include: Construct a world coordinate system and obtain the center coordinates of each characteristic graphic of the calibration plate under the world coordinate system; Based on the center coordinates of each feature graphic in the camera coordinate system and the corresponding center coordinates of the feature graphic in the world coordinate system, the camera's intrinsic and extrinsic parameters are obtained by Zhang Zhengyou's calibration method. Based on the center coordinates of each feature graphic in the projector coordinate system and the corresponding center coordinates of the feature graphic in the world coordinate system, the internal and external parameters of the projector are obtained by Zhang Zhengyou calibration method; Based on the camera extrinsic parameters and the projector extrinsic parameters, the extrinsic parameters between the camera and the projector are obtained through the binocular stereo calibration method.

10. The projection marking method based on visual sensing according to claim 9, characterized in that: The optimal values ​​of the projection marking system parameters are obtained by: According to the initial calibration values ​​of the coordinates of the center points of each feature graphic and the projection marking system parameters in the camera coordinate system, the projector coordinate system and the world coordinate system, a reprojection error equation is constructed; The cost function is constructed based on the reprojection error equation. The cost function is nonlinearly optimized based on the LM iteration method with the goal of minimizing the reprojection error to obtain the optimal values ​​of the projection marking system parameters.

Images