Curved surface normal measurement method and system based on monocular vision and multiple laser planes

Abstract

The invention discloses a curved surface normal measurement method and system based on monocular vision and multiple laser planes. The method comprises the following steps: calibrating laser planes generated by a monocular camera and at least two laser plane projection units; collecting an image of a laser strip formed by the laser plane on the surface of the workpiece through a monocular camera; processing the image, and extracting a central pixel position of each laser strip to obtain a pixel point sequence; back-projecting the pixel point sequence into a three-dimensional ray in a camera coordinate system based on camera model parameters; and performing intersection operation on the three-dimensional rays and the equations of the corresponding laser planes to obtain a plurality of space intersection point sets, establishing a local curved surface model of the workpiece surface, calculating and outputting the normal direction of the workpiece surface according to the local curved surface model, and converting the normal direction into a reference coordinate system for unified representation. According to the method, the stability and universality of normal estimation are improved, the structure of a measurement system is simplified, and the adaptive capacity in the fields of robot machining, automatic detection and industrial measurement is improved.

Description

Technical Field

[0001] This invention relates to the fields of industrial measurement and computer vision technology, and in particular to a method and system for measuring the normal of curved surfaces based on monocular vision and multiple laser planes. Background Technology

[0002] With the increasing precision requirements of industrial manufacturing, the orientation information of workpiece surfaces, especially the normal direction of curved surfaces, has become an indispensable key parameter in applications such as robotic machining, automated welding, drilling, and inspection and calibration. Existing workpiece normal measurement methods are mainly divided into two categories: contact mechanical measurement methods and vision-based non-contact measurement methods.

[0003] Traditional contact-based normal measurement devices typically contact the workpiece surface through multiple contacts, pressure feet, or mechanical indicating mechanisms, calculating the local normal direction of the workpiece using displacement or force feedback sensors. While these structures offer a certain level of measurement accuracy, they generally suffer from problems such as complex structure, large size, slow response speed, and contact interference with the workpiece surface. Furthermore, contact methods are difficult to apply to thin-walled parts, flexible parts, or workpieces with specific surface coatings, hindering the miniaturization of machining equipment and the integration of robotic end effectors.

[0004] To overcome the shortcomings of contact-based methods, an increasing number of studies are attempting to estimate normals using visual methods. Existing non-contact visual measurement methods mainly rely on binocular vision, structured light 3D reconstruction, or laser scanning to obtain local 3D point clouds of the workpiece surface, and then calculate the normal direction through plane or curved surface fitting. However, existing technologies still have the following limitations:

[0005] (1) Binocular vision has a complex structure and high calibration requirements: Binocular systems usually require strict maintenance of the geometric relationship between the two cameras. Their assembly, calibration and long-term stability are poor, which is not conducive to rapid deployment and maintenance in industrial environments.

[0006] (2) Single-line laser scanning has insufficient information in normal estimation: the intersection of a single laser plane and the workpiece surface can only form a spatial curve, which cannot independently determine the complete spatial gradient information of the local surface. Therefore, it is difficult to reliably calculate the normal direction of the workpiece, especially in areas with a certain curvature, where normal ambiguity is easy to occur.

[0007] (3) Multi-line laser or structured light solutions usually rely on complex projection devices or multi-view camera systems: Existing structured light systems often require the use of projectors or stripe projection devices, which are large in size and expensive, and are not suitable for installation at the end of a robot, nor are they convenient for portable or compact measuring devices.

[0008] (4) Traditional 3D point cloud fitting normal is sensitive to noise: The laser strip is affected by factors such as reflection, workpiece material, and ambient light. The reconstructed point cloud often contains noise or local anomalies. Using simple plane fitting or traditional PCA methods can easily lead to unstable normals, which cannot meet the requirements of high-precision processing scenarios.

[0009] In summary, existing technologies lack a unified technical solution that is simple in structure, easy to calibrate and deploy, and capable of obtaining sufficient geometric constraints under monocular vision conditions using multiple laser planes, thereby enabling accurate normal measurement of both workpiece planes and local quadratic surfaces. Therefore, a new method for measuring the normal of workpiece surfaces based on monocular vision and multiple laser planes is urgently needed to address the problems of bulky traditional contact-based structures and insufficient information or poor stability of non-contact methods. Summary of the Invention

[0010] The purpose of this invention is to provide a method and system for measuring the normal direction of curved surfaces based on monocular vision and multiple laser planes. By introducing at least two known laser planes under monocular vision conditions, the camera can obtain sufficient spatial geometric constraints from the intersection lines of multiple lasers and the workpiece surface. This enables accurate measurement of the normal direction of the workpiece plane and local quadratic surface without the need for complex binocular structures or contact mechanisms, improving the stability and versatility of normal estimation, simplifying the measurement system structure, and enhancing its adaptability in robotic processing, automatic inspection, and industrial measurement.

[0011] To achieve the above objectives, the present invention provides the following solution:

[0012] A method for measuring the normal of a curved surface based on monocular vision and multiple laser planes, the method comprising the following steps:

[0013] (1) System calibration: Perform intrinsic parameter calibration on the monocular camera to obtain camera model parameters; calibrate the laser planes generated by at least two laser plane projection units to obtain the equation parameters of each laser plane;

[0014] (2) Image acquisition: The monocular camera acquires an image containing laser stripes formed on the workpiece surface by at least two of the laser planes;

[0015] (3) Laser line extraction: The acquired image is processed to extract the center pixel position of each laser strip to obtain a pixel sequence;

[0016] (4) Three-dimensional ray back projection: Based on the camera model parameters obtained from the calibration, the extracted pixel sequence is back projected into a three-dimensional ray in the camera coordinate system;

[0017] (5) Three-dimensional point reconstruction: The three-dimensional rays are intersected with the equations of the corresponding laser planes to obtain multiple sets of spatial intersection points;

[0018] (6) Surface modeling and normal calculation: Based on the multiple spatial intersection point sets, establish a local surface model of the workpiece surface, calculate and output the normal direction of the workpiece surface according to the local surface model;

[0019] (7) Coordinate reference selection and coordinate transformation: After completing the three-dimensional point reconstruction and surface normal calculation, the installation position coordinates of the laser projection unit or the pre-set equipment reference coordinates are used as the reference coordinate system. Through the known spatial pose relationship or the coordinate transformation relationship obtained by calibration, the spatial point set of the workpiece surface and its corresponding normal direction are transformed from the camera coordinate system to the reference coordinate system for unified expression and output.

[0020] Furthermore, in step (1), the camera intrinsic parameter calibration adopts the standard imaging calibration method, and the obtained camera model parameters include focal length, principal point coordinates and distortion coefficients.

[0021] Furthermore, the specific process of laser plane calibration in step (1) is as follows:

[0022] Each laser plane is projected onto a calibration plate with a known geometry. The pixel positions of the laser lines in the image are extracted, and multiple spatial rays are obtained through pixel back projection. Then, the parameters of the laser plane equation are solved based on the known orientation of the calibration plate. The equation of the laser plane is expressed as:

[0023]

[0024] in, , , , Here are the parameters for the laser plane equation, where i is the laser plane number, and x, y, and z are the three-dimensional coordinates.

[0025] Further, the method for extracting the center pixel position of the laser strip in step (3) is one of the following: sub-pixel line center extraction method, brightness threshold method, and filtering / fitting method. The resulting pixel sequence is represented as follows:

[0026]

[0027] Where i is the laser plane number. , Let N be the x and y coordinates of the j-th pixel corresponding to the i-th laser plane. i This represents the total number of pixels corresponding to the i-th laser plane.

[0028] Furthermore, the expression for the three-dimensional ray in step (4) is:

[0029]

[0030] in, For the camera optical center, For normalized direction vectors, Let be the ray parameter corresponding to the intersection point of the j-th ray and the laser plane corresponding to the i-th laser plane, where i is the laser plane number and j is the pixel number.

[0031] Furthermore, the formula for calculating the spatial intersection point in step (5) is:

[0032]

[0033] in, Pij represents the ray parameter corresponding to the intersection point of the j-th ray corresponding to the i-th laser plane and the laser plane, where Pij is the coordinate of the intersection point, i is the laser plane number, and j is the pixel number;

[0034] Each laser plane corresponds to a set of spatial curve points:

[0035]

[0036] Two or more point sets together provide the local three-dimensional geometry of the workpiece.

[0037] Furthermore, the local surface model mentioned in step (6) is a planar model or a quadratic surface model;

[0038] When the local surface model is a planar model, the plane equation is fitted using the least squares method:

[0039]

[0040] Where A, B, C, and D are plane equation parameters, and the normal vector is: ;

[0041] When the local surface model is a quadratic surface model, the fitted local quadratic surface equation is:

[0042]

[0043] Among them, α, β, γ, δ, ζ are the parameters of the quadratic surface equation. The normal is calculated based on the gradient of the surface.

[0044]

[0045] Normalization is then performed to obtain the surface normal.

[0046] Furthermore, in step (6), the normal direction is calculated at the target location based on the selected surface model:

[0047]

[0048] in, , , These are the components of the normal vector along the x, y, and z axes, respectively, and the normal vector is represented in the camera coordinate system.

[0049] This invention also provides a surface normal measurement system based on monocular vision and multiple laser planes, applied to the aforementioned surface normal measurement method based on monocular vision and multiple laser planes, comprising:

[0050] A monocular camera is used to acquire images of workpieces containing laser stripes and obtain the pixel distribution of the laser line on the imaging plane; the monocular camera has calibrated intrinsic parameters for pixel back projection.

[0051] At least two laser plane projection units are used to project and form at least two laser planes, wherein the laser planes intersect with the workpiece surface to form the laser strip;

[0052] The calculation and control unit is used to process image data and finally output the local normal information of the workpiece, including: image processing module, calibration module, 3D reconstruction module, surface modeling module, normal calculation module and coordinate transformation module;

[0053] The image processing module is used to process the laser stripes in the image, extract the center pixel position of the laser stripes, and obtain a pixel sequence.

[0054] The calibration module is used to calibrate the intrinsic parameters of the monocular camera and obtain camera model parameters; it is also used to calibrate the laser plane and obtain the equation parameters of each laser plane.

[0055] The three-dimensional reconstruction module is used to backproject the pixel sequence into a three-dimensional ray based on the camera model parameters, and to perform intersection operations between the three-dimensional ray and the corresponding equation of the laser plane to obtain multiple spatial intersection point sets;

[0056] The surface modeling module is used to establish a local surface model of the workpiece surface based on the multiple spatial intersection point sets;

[0057] The normal calculation module is used to calculate and output the normal direction of the workpiece surface based on the local surface model;

[0058] The coordinate transformation module is used to transform the spatial point set and its corresponding normal direction of the workpiece surface from the camera coordinate system to the reference coordinate system after completing the 3D point reconstruction and surface normal calculation. The module uses the installation position coordinates of the laser projection unit or the pre-set equipment reference coordinates as the reference coordinate system and transforms the workpiece surface spatial point set and its corresponding normal direction from the camera coordinate system to the reference coordinate system through known spatial pose relationships or calibrated coordinate transformation relationships for unified expression and output.

[0059] Furthermore, the monocular camera is a traditional pinhole camera, a wide-angle camera, a global shutter industrial camera, or a high dynamic range camera;

[0060] The laser plane projection unit is a crosshair laser or a combination of multiple single-line lasers, and the crosshair laser can generate two light planes that are at a fixed angle to each other.

[0061] The laser projected by the laser planar projection unit is a red laser, a green laser, a blue laser, or a structured fiber laser.

[0062] According to specific embodiments of the present invention, the surface normal measurement method and system based on monocular vision and multiple laser planes provides an opportunity to obtain multi-directional geometric constraints that a single laser line cannot provide by introducing at least two known laser planes into the monocular vision system and constructing multiple spatial intersection lines between the laser and the workpiece surface. This enables accurate determination of the normal direction of the workpiece plane and local curved surfaces under monocular conditions. Compared with existing normal measurement methods that rely on contact devices, binocular imaging, or single-line laser scanning, the present invention has the following technical advantages:

[0063] (1) Significantly improves the stability and accuracy of normal direction calculation: Since the present invention uses at least two laser planes to form two or more spatial curves, it can constrain the local shape of the workpiece surface from different directions, so that the plane fitting or surface fitting has higher redundancy. In typical industrial scenarios, the normal direction stability obtained by the present invention is significantly better than that of the single-line laser scheme, and local small noise or reflection interference will not have a significant impact on the normal direction.

[0064] (2) Applicable to planes and quadratic surfaces with small curvature changes: This invention supports the use of planar models or local quadratic surface models to describe workpiece surfaces, and can be applied to planes, arcs, parabolic surfaces and other areas of general industrial parts. Through the geometric information provided by multiple laser curves, this invention can still maintain consistent and reliable normal estimation when processing surfaces with a certain curvature, and will not have problems with ambiguity or uncertainty of direction of single-curve normals.

[0065] (3) The system has a simple structure and is easy to integrate, reducing equipment costs and maintenance difficulty: This invention requires only a single camera and at least two laser projection units, without the need for binocular systems, projectors, or complex structured light modules, and does not rely on mechanical contact structures. Therefore, the system is small in size and easy to install on robot end effectors or inspection fixtures; calibration is simple and long-term operation is highly stable; the equipment cost is significantly lower than that of multi-view vision or projection-type structured light systems. This effect has obvious advantages for space-constrained and cost-sensitive industrial inspection equipment.

[0066] (4) Improve the efficiency of industrial field inspection and realize real-time normal measurement: All calculations in this invention are based on a single image and simple geometric intersection and surface fitting, which can be run in real time on a conventional industrial computing platform. Compared with traditional contact measurement, this invention does not require mechanical movement or contact action, which can significantly shorten the measurement cycle; compared with binocular or projection structured light methods, this scheme has fewer images and lower computational complexity, which is more in line with the real-time requirements of online inspection or robotic processing.

[0067] (5) Reduce the impact of contact measurement on the workpiece surface and improve measurement reliability: Traditional contact normal measurement may cause indentation or scratches on thin-walled parts, flexible parts or special coating surfaces, or even cause measurement errors due to local deformation of the workpiece. The present invention adopts a completely non-contact visual measurement method, which can avoid the impact on the workpiece and improve the repeatability and consistency of the measurement process.

[0068] (6) Enhanced system applicability and scalability: Since the core of this invention is "monocular back projection + multi-laser plane constraint + local surface modeling", the system can be flexibly expanded. The number of laser planes can be increased to improve the surface reconstruction density; the surface model can be extended to higher-order surfaces or parametric surfaces; the camera can be replaced with different focal lengths and different field-of-view models; the applicable scope covers multiple scenarios such as robot processing, assembly calibration, and online inspection, thus having good engineering portability and long-term adaptability.

[0069] (7) Overall improvement of industrial processing and inspection accuracy: This invention can provide reliable surface normal information for processes such as robot hole making, welding, grinding, spraying, and cutting, which helps to make the tool direction consistent with the workpiece normal, reduce the posture deviation caused by clamping error, improve processing quality and consistency, and reduce rework and scrap rate caused by normal deviation. It has practical value for improving the process stability and economic benefits of manufacturing enterprises. Attached Figure Description

[0070] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0071] Figure 1 This is a flowchart of the surface normal measurement method based on monocular vision and multiple laser planes in an embodiment of the present invention;

[0072] Figure 2 This is a schematic diagram of the surface normal measurement system based on monocular vision and multiple laser planes in an embodiment of the present invention;

[0073] Figure 3 This is a schematic diagram illustrating the spatial measurement constraint formed between the laser plane and the workpiece surface in an embodiment of the present invention.

[0074] Figure 4 This is a schematic diagram of the pixel distribution of the laser strip on the camera imaging plane in an embodiment of the present invention.

[0075] Figure 5 This is a schematic diagram of three-dimensional point reconstruction based on the intersection of pixel back projection and laser plane in an embodiment of the present invention;

[0076] Figure 6 This is a schematic diagram of local surface fitting based on multiple spatial point sets in an embodiment of the present invention;

[0077] Explanation of reference numerals in the attached figures: 101, Calculation and control unit; 102, Monocular camera; 103, Laser planar projection unit; 104, Workpiece;

[0078] P1, laser plane 1; P2, laser plane 2; L1, intersection line 1; L2, intersection line 2;

[0079] 301. Camera imaging plane; 302. Laser line image pixel trajectory one; 303. Laser line image pixel trajectory two;

[0080] 401. Ray; 402. Intersection point; 501. Spatial curve reconstructed from laser plane P1; 502. Spatial curve reconstructed from laser plane P2. Detailed Implementation

[0081] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0082] The purpose of this invention is to address the problems of complex structure, difficult calibration, limited measurement accuracy, and insufficient single-line laser information in existing workpiece normal measurement technologies, and to provide a surface normal measurement method and system based on monocular vision and multiple laser planes.

[0083] This invention introduces at least two known laser planes under monocular vision conditions, enabling the camera to obtain sufficient spatial geometric constraints from the intersection lines of multiple lasers with the workpiece surface. This allows for accurate measurement of the normal directions of workpiece planes and local quadratic surfaces without the need for complex binocular structures or contact mechanisms. The invention aims to improve the stability and versatility of normal estimation, simplify the measurement system structure, and enhance its adaptability in robotic machining, automated inspection, and industrial measurement.

[0084] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0085] Example 1

[0086] like Figure 1 As shown, the surface normal measurement method based on monocular vision and multiple laser planes provided by this invention includes the following steps:

[0087] (1) System calibration: Perform intrinsic parameter calibration on the monocular camera to obtain camera model parameters; calibrate the laser planes generated by at least two laser plane projection units to obtain the equation parameters of each laser plane;

[0088] (2) Image acquisition: The monocular camera acquires an image containing laser stripes formed on the workpiece surface by at least two of the laser planes;

[0089] (3) Laser line extraction: The acquired image is processed to extract the center pixel position of each laser strip to obtain a pixel sequence;

[0090] (4) Three-dimensional ray back projection: Based on the camera model parameters obtained from the calibration, the extracted pixel sequence is back projected into a three-dimensional ray in the camera coordinate system;

[0091] (5) Three-dimensional point reconstruction: The three-dimensional rays are intersected with the equations of the corresponding laser planes to obtain multiple sets of spatial intersection points;

[0092] (6) Surface modeling and normal calculation: Based on the multiple spatial intersection points, a local surface model of the workpiece surface is established, and the normal direction of the workpiece surface is calculated and output according to the local surface model.

[0093] (7) Coordinate reference selection and coordinate transformation: After completing the three-dimensional point reconstruction and surface normal calculation, the installation position coordinates of the laser projection unit or the pre-set equipment reference coordinates are used as the reference coordinate system. Through the known spatial pose relationship or the coordinate transformation relationship obtained by calibration, the spatial point set of the workpiece surface and its corresponding normal direction are transformed from the camera coordinate system to the reference coordinate system for unified expression and output.

[0094] The steps described above are explained in detail below:

[0095] (1) System calibration:

[0096] Camera intrinsic parameter calibration: Camera model parameters (including focal length, principal point coordinates, and distortion coefficients) are obtained using standard imaging calibration methods to enable back projection from pixels to 3D rays. This step forms the basis for subsequent 3D reconstruction in this invention.

[0097] Laser plane calibration: Each laser plane is projected onto a calibration plate with a known geometry. The pixel positions of the laser lines in the image are extracted, and multiple spatial rays are obtained through pixel back-projection. Then, the parameters of the laser plane equations are solved based on the known orientation of the calibration plate. Each laser plane can be represented as:

[0098]

[0099] in, , , , Here are the parameters for the laser plane equation, where i is the laser plane number, and x, y, and z are the three-dimensional coordinates.

[0100] The laser plane calibration results are used in the detection phase to calculate the spatial intersection point between the ray and the laser plane.

[0101] Taking two laser projection units as an example, two laser planes P1 and P2 with known orientations are projected respectively. After being projected onto the workpiece surface, laser planes P1 and P2 form two laser stripes. The parameters of laser planes P1 and P2 are obtained after calibration and used for 3D point intersection calculation during the detection stage.

[0102] (2) Image acquisition:

[0103] At least two laser planes illuminate the workpiece surface, intersecting to form two or more laser spatial curves. A camera captures images containing these laser stripes, which serve as the data source for subsequent processing.

[0104] like Figure 3As shown, two laser planes, P1 and P2, are projected onto the workpiece surface, intersecting with the workpiece surface to form two spatial curves, labeled L1 and L2. These two spatial curves provide geometric constraints in different directions for the calculation of the surface normal.

[0105] like Figure 4 As shown, the pixel trajectory of the laser stripe is captured in the camera's imaging plane 301. The two stripes are labeled as laser line image pixel trajectory one 302 and laser line image pixel trajectory two 303, respectively.

[0106] (3) Laser line extraction:

[0107] The center pixel positions of each laser strip are extracted using image processing methods, forming a point sequence arranged in pixel order:

[0108]

[0109] Where i is the laser plane number. , Let N be the x and y coordinates of the j-th pixel corresponding to the i-th laser plane. i This represents the total number of pixels corresponding to the i-th laser plane.

[0110] This invention does not limit the specific image processing method, and can employ subpixel line center extraction method, brightness thresholding method, filtering / fitting method, etc.

[0111] (4) Three-dimensional ray back projection:

[0112] like Figure 5 As shown, each pixel is back-projected into a three-dimensional ray Ray 402 passing through the camera optical center (Cameracenter 401) according to the camera model.

[0113] Using camera intrinsics, each pixel is back-projected into a 3D ray in the camera coordinate system:

[0114]

[0115] in, For the camera optical center, For normalized direction vectors, Let be the ray parameter corresponding to the intersection point of the j-th ray and the laser plane corresponding to the i-th laser plane, where i is the laser plane number and j is the pixel number.

[0116] (5) Three-dimensional point reconstruction:

[0117] Substituting the ray parameter equations into the laser plane equations, the intersection points are obtained:

[0118]

[0119] in, Pij represents the ray parameter corresponding to the intersection point of the j-th ray with the i-th laser plane and the laser plane. Pij is the coordinate of the intersection point, i is the laser plane number, and j is the pixel number. This point is the spatial point where the laser plane intersects with the workpiece surface.

[0120] Each laser plane corresponds to a set of spatial curve points:

[0121]

[0122] Two or more point sets together provide the local three-dimensional geometry of the workpiece.

[0123] Substituting the ray into the equation of the laser plane, the coordinates of the intersection point are obtained. The processing unit calculates the point sets corresponding to the two planes:

[0124] A collection of point clouds from P1;

[0125] A collection of point clouds from P2.

[0126] like Figure 5 As shown, the reconstructed intersection point is labeled 402.

[0127] (6) Surface modeling and normal calculation:

[0128] This invention utilizes multiple spatial curves generated by at least two laser planes to form sufficient geometric constraints on the local curved surface of the workpiece, thereby enabling normal solution in a monocular system.

[0129] like Figure 6 As shown, two spatial curves 501 and 502, reconstructed from laser planes P1 and P2, are distributed on the surface of the workpiece.

[0130] Depending on the application requirements, the present invention can select the following two types of surface models:

[0131] (6.1) Local planar model

[0132] This method is suitable for scenarios where the workpiece surface is approximately flat, and it fits the plane equation using the least squares method.

[0133]

[0134] Where A, B, C, and D are plane equation parameters, and the normal vector is: ;

[0135] (6.2) Local quadratic surface model

[0136] This method is applicable to surfaces with general curvature by fitting the equation of a local quadratic surface:

[0137]

[0138] Among them, α, β, γ, δ, ζ are the parameters of the quadratic surface equation. The normal is calculated based on the gradient of the surface.

[0139]

[0140] Normalization yields the surface normal. The normal can be obtained from the surface partial derivative:

[0141]

[0142] This invention does not limit the specific mathematical method for surface fitting, and can employ methods such as least squares method, weighted fitting, regularized fitting, or robust fitting.

[0143] Output normal vector:

[0144] Based on the selected surface model, the normal direction at the target location is calculated:

[0145]

[0146] in, , , These are the components of the normal vector in the x, y, and z axes, respectively. The output normal vector is represented in the camera coordinate system and can be used for tasks such as robot posture adjustment, workpiece clamping calibration, and machining path compensation.

[0147] like Figure 6 As shown, the final calculated surface normal is labeled as n in the form of an arrow.

[0148] The normal vector is represented in the camera coordinate system and can be used to adjust the robot's posture or calibrate the workpiece clamping.

[0149] In the above embodiments, the number of laser planes is expanded to three or more to enhance the stability of the normal solution; the arrangement of the lasers can be any spatial angle, and orthogonality is not required; the surface model can be expanded to B-spline surface, local polynomial surface, or other differentiable surface model; the camera can be replaced with imaging devices with different focal lengths, different resolutions, or with filtering functions; the three-dimensional reconstruction can adopt other forms of back projection model, such as pinhole model or wide-angle distortion model.

[0150] Example 2

[0151] like Figure 2 As shown, the present invention also provides a surface normal measurement system based on monocular vision and multiple laser planes, applied to the above-mentioned surface normal measurement method based on monocular vision and multiple laser planes, including:

[0152] A monocular camera is used to acquire images of workpieces containing laser stripes and obtain the pixel distribution of the laser line on the imaging plane; the monocular camera has calibrated intrinsic parameters for pixel back projection.

[0153] At least two laser plane projection units are used to project and form at least two laser planes, which intersect with the workpiece surface to form the laser strip; the laser planes may be composed of the following devices:

[0154] Crosshair laser (two light planes that are at a fixed angle to each other);

[0155] Multiple single-line lasers are combined to form at least two optical planes;

[0156] The laser plane parameters are obtained through calibration and provide geometric constraints in subsequent 3D intersection calculations;

[0157] The calculation and control unit is used to process image data and finally output the local normal information of the workpiece, including: image processing module, calibration module, 3D reconstruction module, surface modeling module, normal calculation module and coordinate transformation module;

[0158] The image processing module is used to process the laser stripes in the image, extract the center pixel position of the laser stripes, and obtain a pixel sequence.

[0159] The calibration module is used to calibrate the intrinsic parameters of the monocular camera and obtain camera model parameters; it is also used to calibrate the laser plane and obtain the equation parameters of each laser plane.

[0160] The three-dimensional reconstruction module is used to backproject the pixel sequence into a three-dimensional ray based on the camera model parameters, and to perform intersection operations between the three-dimensional ray and the corresponding equation of the laser plane to obtain multiple spatial intersection point sets;

[0161] The surface modeling module is used to establish a local surface model of the workpiece surface based on the multiple spatial intersection point sets;

[0162] The normal calculation module is used to calculate and output the normal direction of the workpiece surface based on the local surface model;

[0163] The coordinate transformation module is used to transform the spatial point set and its corresponding normal direction of the workpiece surface from the camera coordinate system to the reference coordinate system after completing the 3D point reconstruction and surface normal calculation. The module uses the installation position coordinates of the laser projection unit or the pre-set equipment reference coordinates as the reference coordinate system and transforms the workpiece surface spatial point set and its corresponding normal direction from the camera coordinate system to the reference coordinate system through known spatial pose relationships or calibrated coordinate transformation relationships for unified expression and output.

[0164] The above-mentioned components together constitute the overall system framework of the present invention. It is compact, easy to integrate, and can be used in robot end effectors or independent detection devices.

[0165] The video output of the monocular camera is connected to the calculation and control unit via a data cable. The laser projection unit maintains a stable geometric relationship with the monocular camera through a fixed structure. Laser planes P1 and P2 (spatial light planes generated by the laser projection unit) are projected onto the workpiece surface, forming two laser stripes. The workpiece is a planar or curved part to be measured.

[0166] For example, the monocular camera can be a traditional pinhole camera, a wide-angle camera, a global shutter industrial camera, or a high dynamic range (HDR) camera; if an ultra-wide field of view is required, an equidistant imaging model can be used, but this does not affect the back projection principle.

[0167] The laser projected by the laser plane projection unit is a red laser, green laser, blue laser, or structured fiber laser. The number of laser planes is two, three, or more. Before establishing the local surface model, outlier removal or weighting is performed on the spatial intersection set. Data processing methods include, but are not limited to: using RANSAC to filter outliers, using weighted least squares, using a sliding window to fit the local surface, performing dynamic threshold segmentation based on the reflectivity of the workpiece surface, and adding spatial smoothing constraints to improve robustness. The calculated workpiece normal is used for posture calibration during robot machining, welding, hole drilling, or grinding.

[0168] The end effector's posture is adjusted in real time based on the measured normal to achieve: welding torch angle compensation, grinding angle correction, drilling axis alignment, visual servoing, etc.

[0169] The layout of the laser projection unit can be varied:

[0170] The laser projection unit can be fixed together with the camera on a rigid frame; mounted on the robot's end effector; and employs an adjustable angle structure to adapt to different workpiece shapes.

[0171] The angle between the laser planes is not limited and can be optimized according to the application scenario to improve the stability of curved surface modeling.

[0172] The number of laser planes can be expanded:

[0173] This invention uses at least two laser planes, but can also be extended to three or four laser planes.

[0174] This provides more spatial constraints in more directions, further improving the accuracy of normal estimation for complex surfaces.

[0175] The present invention also provides an electronic device including one or more processors; a memory; and one or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the one or more processors, and the one or more application programs are configured to perform the surface normal measurement method based on monocular vision and multiple laser planes as described above.

[0176] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for measuring the normal of a curved surface based on monocular vision and multiple laser planes, characterized in that, Includes the following steps: (1) System calibration: Perform intrinsic parameter calibration on the monocular camera to obtain camera model parameters; calibrate the laser planes generated by at least two laser plane projection units to obtain the equation parameters of each laser plane; (2) Image acquisition: The monocular camera acquires an image containing laser stripes formed on the workpiece surface by at least two of the laser planes; (3) Laser line extraction: The acquired image is processed to extract the center pixel position of each laser strip to obtain a pixel sequence; (4) Three-dimensional ray back projection: Based on the camera model parameters obtained from the calibration, the extracted pixel sequence is back projected into a three-dimensional ray in the camera coordinate system; (5) Three-dimensional point reconstruction: The three-dimensional rays are intersected with the equations of the corresponding laser planes to obtain multiple sets of spatial intersection points; (6) Surface modeling and normal calculation: Based on the multiple spatial intersection point sets, establish a local surface model of the workpiece surface, calculate and output the normal direction of the workpiece surface according to the local surface model; (7) Coordinate reference selection and coordinate transformation: After completing the three-dimensional point reconstruction and surface normal calculation, the installation position coordinates of the laser projection unit or the pre-set equipment reference coordinates are used as the reference coordinate system. Through the known spatial pose relationship or the coordinate transformation relationship obtained by calibration, the spatial point set of the workpiece surface and its corresponding normal direction are transformed from the camera coordinate system to the reference coordinate system for unified expression and output.

2. The surface normal measurement method based on monocular vision and multiple laser planes according to claim 1, characterized in that, In step (1), the camera intrinsic parameter calibration adopts the standard imaging calibration method, and the obtained camera model parameters include focal length, principal point coordinates and distortion coefficients.

3. The surface normal measurement method based on monocular vision and multiple laser planes according to claim 1, characterized in that, The specific process of laser plane calibration in step (1) is as follows: Each laser plane is projected onto a calibration plate with a known geometry. The pixel positions of the laser lines in the image are extracted, and multiple spatial rays are obtained through pixel back projection. Then, the parameters of the laser plane equation are solved based on the known orientation of the calibration plate. The equation of the laser plane is expressed as: in, , , , Here are the parameters for the laser plane equation, where i is the laser plane number, and x, y, and z are the three-dimensional coordinates.

4. The surface normal measurement method based on monocular vision and multiple laser planes according to claim 1, characterized in that, The method for extracting the center pixel position of the laser strip in step (3) is one of the following: sub-pixel line center extraction method, brightness threshold method, and filtering / fitting method. The resulting pixel sequence is represented as follows: Where i is the laser plane number. , Let N be the x and y coordinates of the j-th pixel corresponding to the i-th laser plane. i This represents the total number of pixels corresponding to the i-th laser plane.

5. The surface normal measurement method based on monocular vision and multiple laser planes according to claim 1, characterized in that, The expression for the three-dimensional ray in step (4) is: in, For the camera optical center, For normalized direction vectors, Let be the ray parameter corresponding to the intersection point of the j-th ray and the laser plane corresponding to the i-th laser plane, where i is the laser plane number and j is the pixel number.

6. The surface normal measurement method based on monocular vision and multiple laser planes according to claim 1, characterized in that, The formula for calculating the spatial intersection point in step (5) is: in, Pij represents the ray parameter corresponding to the intersection point of the j-th ray corresponding to the i-th laser plane and the laser plane, where Pij is the coordinate of the intersection point, i is the laser plane number, and j is the pixel number; Each laser plane corresponds to a set of spatial curve points: Two or more point sets together provide the local three-dimensional geometry of the workpiece.

7. The surface normal measurement method based on monocular vision and multiple laser planes according to claim 1, characterized in that, The local surface model mentioned in step (6) is a planar model or a quadratic surface model; When the local surface model is a planar model, the plane equation is fitted using the least squares method: Where A, B, C, and D are plane equation parameters, and the normal vector is: ; When the local surface model is a quadratic surface model, the fitted local quadratic surface equation is: Among them, α, β, γ, δ, ζ are the parameters of the quadratic surface equation. The normal is calculated based on the gradient of the surface. Normalization is then performed to obtain the surface normal.

8. The method for measuring the normal of a curved surface based on monocular vision and multiple laser planes according to claim 1, characterized in that, In step (6), the normal direction is calculated at the target location based on the selected surface model: in, , , These are the components of the normal vector along the x, y, and z axes, respectively, and the normal vector is represented in the camera coordinate system.

9. A surface normal measurement system based on monocular vision and multiple laser planes, applied to the surface normal measurement method based on monocular vision and multiple laser planes as described in any one of claims 1-8, characterized in that, include: A monocular camera is used to acquire images of workpieces containing laser stripes and obtain the pixel distribution of the laser line on the imaging plane; the monocular camera has calibrated intrinsic parameters for pixel back projection. At least two laser plane projection units are used to project and form at least two laser planes, wherein the laser planes intersect with the workpiece surface to form the laser strip; The calculation and control unit is used to process image data and finally output the local normal information of the workpiece, including: image processing module, calibration module, 3D reconstruction module, surface modeling module, normal calculation module and coordinate transformation module; The image processing module is used to process the laser stripes in the image, extract the center pixel position of the laser stripes, and obtain a pixel sequence. The calibration module is used to calibrate the intrinsic parameters of the monocular camera and obtain camera model parameters; it is also used to calibrate the laser plane and obtain the equation parameters of each laser plane. The three-dimensional reconstruction module is used to backproject the pixel sequence into a three-dimensional ray based on the camera model parameters, and to perform intersection operations between the three-dimensional ray and the corresponding equation of the laser plane to obtain multiple spatial intersection point sets; The surface modeling module is used to establish a local surface model of the workpiece surface based on the multiple spatial intersection point sets; The normal calculation module is used to calculate and output the normal direction of the workpiece surface based on the local surface model; The coordinate transformation module is used to transform the spatial point set and its corresponding normal direction of the workpiece surface from the camera coordinate system to the reference coordinate system after completing the 3D point reconstruction and surface normal calculation. The module uses the installation position coordinates of the laser projection unit or the pre-set equipment reference coordinates as the reference coordinate system and transforms the workpiece surface spatial point set and its corresponding normal direction from the camera coordinate system to the reference coordinate system through known spatial pose relationships or calibrated coordinate transformation relationships for unified expression and output.

10. The surface normal measurement system based on monocular vision and multiple laser planes according to claim 9, characterized in that, The monocular camera is a traditional pinhole camera, a wide-angle camera, a global shutter industrial camera, or a high dynamic range camera. The laser plane projection unit is a crosshair laser or a combination of multiple single-line lasers, and the crosshair laser can generate two light planes that are at a fixed angle to each other. The laser projected by the laser planar projection unit is a red laser, a green laser, a blue laser, or a structured fiber laser.

Images