Gear three-dimensional color reconstruction method and system based on image and point cloud information fusion

Abstract

The invention discloses a gear three-dimensional color reconstruction method and system based on image and point cloud information fusion, and the method comprises the steps: collecting first images and first point cloud data of calibration blocks at different heights and different positions, and collecting a second image and second point cloud data of a gear; based on the plurality of first images, roughly identifying first rough two-dimensional angular point coordinates corresponding to each first image; determining a plurality of groups of first target two-dimensional angular point coordinates based on the first coarse two-dimensional angular point coordinates corresponding to each first image; determining multiple groups of target three-dimensional angular point coordinates based on the multiple groups of first point cloud data; calculating a multi-dimensional data conversion relationship between the plurality of groups of first target two-dimensional corner coordinates and the plurality of groups of target three-dimensional corner coordinates to obtain a conversion matrix; and reconstructing a three-dimensional color model of the gear based on the conversion matrix, the second image and the second point cloud data. According to the invention, high-precision gear three-dimensional color reconstruction is realized, so that the detection efficiency of the gear is improved, and the detection precision of the gear is improved.

Description

Technical Field

[0001] The present application relates to the technical field of three-dimensional reconstruction, and in particular to a method and system for three-dimensional color reconstruction of gears based on the fusion of image and point cloud information. Background Art

[0002] The design and manufacturing of aviation gears have made great progress in the past few decades, but aviation gear inspection is still mainly based on coordinate measuring machine point sampling and manual visual inspection, with a low level of automation and intelligence, which can no longer meet the current development needs of high-speed intelligent and digital aviation industry.

[0003] The content of aviation gear inspection is relatively complex and diverse, including tooth shape, tooth direction, surface defects and contact marks. Among them, the measurement of surface defects and contact marks is mainly completed by manual visual inspection. Specifically, the quality inspector observes the inner and outer surfaces of the gear to see if there are surface defects such as scratches, dents and rust, and then records the inspection results of the inspected products; manually sticks contact marks, measures contact mark parameters with calipers and records them. These manual methods not only have problems such as low efficiency, poor data storage, maintainability and large human subjective errors, but also cannot quantitatively measure the defect size, category and contact mark evaluation parameters with high precision, and cannot meet the requirements of aircraft engines for improving the performance and quality consistency of aviation gears. Summary of the invention

[0004] The present application aims to propose a gear three-dimensional color reconstruction method and system based on image and point cloud information fusion, which can improve the gear detection efficiency and accuracy by achieving high-precision gear three-dimensional color reconstruction, and lay an important foundation for multi-dimensional detection of gears.

[0005] In a first aspect, an embodiment of the present application provides a method for three-dimensional color reconstruction of a gear based on image and point cloud information fusion, the method comprising:

[0006] Collecting first images and first point cloud data of calibration blocks at different heights and different positions to obtain multiple first images and multiple groups of first point cloud data, and collecting second images and second point cloud data of the gear, wherein the first images and first point cloud data of the calibration blocks at the same height and the same position are collected simultaneously;

[0007] Based on the multiple first images, roughly identifying first rough two-dimensional corner point coordinates corresponding to each of the first images;

[0008] Determine multiple sets of first target two-dimensional corner point coordinates based on the first rough two-dimensional corner point coordinates corresponding to each of the first images;

[0009] Based on the multiple sets of first point cloud data, determining multiple sets of target three-dimensional corner point coordinates;

[0010] Calculating a multi-dimensional data conversion relationship between the multiple sets of first target two-dimensional corner point coordinates and the multiple sets of target three-dimensional corner point coordinates to obtain a conversion matrix;

[0011] A three-dimensional color model of a gear is reconstructed based on the transformation matrix, the second image and the second point cloud data.

[0012] Compared with the prior art, the first aspect of the present application has the following beneficial effects:

[0013] The method acquires a plurality of first images and a plurality of groups of first point cloud data by acquiring a first image and a first point cloud data of a calibration block at different heights and different positions, and acquires a second image and a second point cloud data of a gear, wherein the first image and the first point cloud data of the calibration block at the same height and the same position are acquired simultaneously; based on the plurality of first images, a first rough two-dimensional corner point coordinate corresponding to each first image is roughly identified; based on the first rough two-dimensional corner point coordinate corresponding to each first image, a plurality of groups of first target two-dimensional corner point coordinates are determined; based on the plurality of groups of first point cloud data, a plurality of groups of target three-dimensional corner point coordinates are determined; a multi-dimensional data conversion relationship between the plurality of groups of first target two-dimensional corner point coordinates and the plurality of groups of target three-dimensional corner point coordinates is calculated to acquire a conversion matrix; based on the conversion matrix, the second image and the second point cloud data, a three-dimensional color model of the gear is reconstructed. In this way, by using the first rough two-dimensional corner point coordinates to further determine the more accurate first target two-dimensional corner point coordinates, and then using the same method to determine the target three-dimensional corner point coordinates, and then according to the accurate first target two-dimensional corner point coordinates and the target three-dimensional corner point coordinates, calculate the accurate transformation matrix, and finally reconstruct the gear three-dimensional color model based on the accurate transformation matrix, it is possible to achieve high-precision gear three-dimensional color reconstruction, so as to improve the gear detection efficiency, improve the accuracy of gear detection, and lay an important foundation for multi-dimensional detection of gears.

[0014] In some implementations, the step of roughly identifying first rough two-dimensional corner point coordinates corresponding to each of the first images based on the plurality of first images includes:

[0015] Performing filtering and edge detection on the calibration blocks in the plurality of first images to obtain edge two-dimensional pixel points of the calibration block in each of the first images;

[0016] Downsampling edge two-dimensional pixel points of the calibration block in each of the first images to obtain multiple groups of edge pixel points of the sampled calibration blocks, wherein a group of edge pixel points of the sampled calibration blocks corresponds to one of the first images;

[0017] Sorting each group of edge pixel points of the sampled calibration block in a head-to-tail manner to obtain multiple groups of edge pixel points of the sorted calibration blocks;

[0018] Randomly select four sorted edge pixel points from each group of edge pixel points of the sorted calibration block, calculate the areas of the four sorted edge pixel points, and obtain a plurality of first areas;

[0019] The four edge two-dimensional pixel points with the largest areas among the multiple first areas are determined as rough two-dimensional corner point coordinates, and first rough two-dimensional corner point coordinates corresponding to each of the first images are obtained.

[0020] In some implementations, determining multiple sets of first target two-dimensional corner point coordinates based on the first rough two-dimensional corner point coordinates corresponding to each of the first images includes:

[0021] The other edge two-dimensional pixel points between two adjacent rough two-dimensional corner point coordinates in the first rough two-dimensional corner point coordinates corresponding to each of the first images are saved into a set to obtain four point sets corresponding to each of the first images;

[0022] Perform iterative straight line fitting on each of the point sets to obtain four straight lines;

[0023] Calculating the intersection coordinates of the four straight lines to obtain the coordinates of four intersection points corresponding to each of the first images;

[0024] The four intersection point coordinates are used as a set of target two-dimensional corner point coordinates to obtain multiple sets of first target two-dimensional corner point coordinates.

[0025] In some implementations, determining multiple sets of target three-dimensional corner point coordinates based on the multiple sets of first point cloud data includes:

[0026] Performing plane fitting on each group of the first point cloud data to obtain a fitting plane corresponding to each group of the first point cloud data, wherein the first point cloud data is point cloud data of the top plane of the calibration block collected by a line laser scanner;

[0027] Calculating a normal vector of a fitting plane corresponding to each group of the first point cloud data to obtain a plurality of normal vectors;

[0028] Calculating the angle between each normal vector and the Z axis in the line laser scanner coordinate system to obtain a plurality of angles;

[0029] According to the normal vector and the angle corresponding to each group of the first point cloud data, the point cloud data on the fitting plane corresponding to each group of the first point cloud data is projected onto a plane to obtain a plurality of projected planes;

[0030] Encoding each point on the projected plane to obtain multiple sets of encoded coordinate values;

[0031] Based on each group of the encoded coordinate values, roughly identifying the second rough two-dimensional corner point coordinates corresponding to each of the projected planes;

[0032] Determine multiple sets of second target two-dimensional corner point coordinates based on the second rough two-dimensional corner point coordinates corresponding to each of the projected planes;

[0033] The multiple sets of second target two-dimensional corner point coordinates are inversely transformed to obtain multiple sets of target three-dimensional corner point coordinates.

[0034] In some implementations, projecting the point cloud data on the fitting plane corresponding to each group of the first point cloud data onto a plane according to the normal vector and the angle corresponding to each group of the first point cloud data to obtain a plurality of projected planes includes:

[0035] Make the normal vector corresponding to each group of the first point cloud data parallel to the Z axis in the line laser scanner coordinate system, and make the fitting plane corresponding to each group of the first point cloud data parallel to the XOY plane in the line laser scanner coordinate system;

[0036] After the fitting plane corresponding to each group of the first point cloud data is parallel to the XOY plane in the line laser scanner coordinate system, the point cloud data on the fitting plane corresponding to each group of the first point cloud data is projected onto the XOY plane to obtain a plurality of projected planes.

[0037] In some implementations, encoding each point on the projected plane to obtain multiple sets of encoded coordinate values ​​includes:

[0038] Preset coding area interval;

[0039] Taking the coordinates of the plane point with the smallest X value and Y value in each projected plane as the coordinates of the origin of the encoding coordinate system;

[0040] The coordinates of all plane points in each of the projected planes are subtracted from the coordinates of the origin of the coding coordinate system and then divided by the coding area interval to obtain multiple groups of coding coordinate values.

[0041] In some embodiments, reconstructing the three-dimensional color model of the gear based on the transformation matrix, the second image and the second point cloud data includes:

[0042] According to the conversion matrix, the second point cloud data is projected onto the second image, and RGB information of a pixel value on the second image corresponding to each second point cloud data is recorded;

[0043] Mapping the RGB information to second point cloud data having a three-dimensional space through a conversion matrix to obtain color point cloud data;

[0044] Based on the color point cloud data, a three-dimensional color model of the gear is reconstructed.

[0045] In a second aspect, the embodiment of the present application further provides a gear three-dimensional color reconstruction system based on image and point cloud information fusion, the system comprising:

[0046] A data acquisition unit is used to acquire first images and first point cloud data of calibration blocks at different heights and different positions, to obtain multiple first images and multiple groups of first point cloud data, and to acquire second images and second point cloud data of the gear, wherein the first images and first point cloud data of the calibration blocks at the same height and the same position are acquired simultaneously;

[0047] a coarse identification unit, configured to coarsely identify first coarse two-dimensional corner point coordinates corresponding to each of the first images based on the multiple first images;

[0048] A first data determination unit, configured to determine a plurality of sets of first target two-dimensional corner point coordinates based on first rough two-dimensional corner point coordinates corresponding to each of the first images;

[0049] A second data determination unit, configured to determine a plurality of sets of target three-dimensional corner point coordinates based on the plurality of sets of first point cloud data;

[0050] A conversion matrix calculation unit, used to calculate the multi-dimensional data conversion relationship between the multiple sets of first target two-dimensional corner point coordinates and the multiple sets of target three-dimensional corner point coordinates to obtain a conversion matrix;

[0051] A color model reconstruction unit is used to reconstruct a three-dimensional color model of the gear based on the transformation matrix, the second image and the second point cloud data.

[0052] In the third aspect, an embodiment of the present application also provides an electronic device, comprising at least one control processor and a memory for communicating with the at least one control processor; the memory stores instructions that can be executed by the at least one control processor, and the instructions are executed by the at least one control processor so that the at least one control processor can execute the above-mentioned three-dimensional color reconstruction method of gears based on the fusion of image and point cloud information.

[0053] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the above-mentioned method for three-dimensional color reconstruction of gears based on the fusion of image and point cloud information.

[0054] It can be understood that the beneficial effects of the second to fourth aspects compared with the related art are the same as the beneficial effects of the first aspect compared with the related art. Please refer to the relevant description in the first aspect, and no further details will be given here. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0056] Figure 1 It is a flow chart of an embodiment of a method for three-dimensional color reconstruction of gears based on image and point cloud information fusion provided by the present application;

[0057] Figure 2 It is a schematic diagram of collected pictures and point cloud data in the best embodiment of the gear three-dimensional color reconstruction method based on image and point cloud information fusion provided by the present application;

[0058] Figure 3 It is a schematic diagram of the coarsely identified corner point results in the best embodiment of the gear three-dimensional color reconstruction method based on image and point cloud information fusion provided by the present application;

[0059] Figure 4 It is a schematic diagram of two-dimensional precise corner points in the best embodiment of the gear three-dimensional color reconstruction method based on image and point cloud information fusion provided by the present application;

[0060] Figure 5 It is a schematic diagram of converting a point cloud into a two-dimensional plane in the best embodiment of the gear three-dimensional color reconstruction method based on image and point cloud information fusion provided by the present application;

[0061] Figure 6 It is a schematic diagram of four precise corner points in the coded coordinate system in the best embodiment of the gear three-dimensional color reconstruction method based on image and point cloud information fusion provided by the present application;

[0062] Figure 7 It is a schematic diagram of the coordinate values ​​of three-dimensional corner points in the best embodiment of the gear three-dimensional color reconstruction method based on image and point cloud information fusion provided by the present application;

[0063] Figure 8 It is a three-dimensional color schematic diagram of a gear in the best embodiment of the three-dimensional color reconstruction method of a gear based on image and point cloud information fusion provided by the present application;

[0064] Fig. 9 It is a schematic diagram of various types of aviation gears in the best embodiment of the gear three-dimensional color reconstruction method based on image and point cloud information fusion provided by the present application;

[0065] Fig.10 It is a structural schematic diagram of an embodiment of a gear three-dimensional color reconstruction system based on image and point cloud information fusion provided by the present application;

[0066] Fig.11 It is a schematic diagram of the structure of an embodiment of the electronic device provided by the present application. DETAILED DESCRIPTION

[0067] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be understood as limiting the present application.

[0068] In the description of this application, if there is a description of first, second, etc., it is only for the purpose of distinguishing technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0069] In the description of the present application, it should be understood that the descriptions involving orientation, such as the orientation or positional relationship indicated as up, down, etc., are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present application.

[0070] In the description of this application, it should be noted that, unless otherwise clearly defined, terms such as setting, installing, connecting, etc. should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meaning of the above terms in this application based on the specific content of the technical solution.

[0071] The design and manufacturing of aviation gears have made great progress in the past few decades, but aviation gear inspection is still mainly based on coordinate measuring machine point sampling and manual visual inspection, with a low level of automation and intelligence, which can no longer meet the current development needs of high-speed intelligent and digital aviation industry.

[0072] The content of aviation gear inspection is relatively complex and diverse, including tooth shape, tooth direction, surface defects and contact marks. Among them, the measurement of surface defects and contact marks is mainly completed by manual visual inspection. Specifically, the quality inspector observes the inner and outer surfaces of the gear to see if there are surface defects such as scratches, dents and rust, and then records the inspection results of the inspected products; manually sticks contact marks, measures contact mark parameters with calipers and records them. These manual methods not only have problems such as low efficiency, poor data storage, maintainability and large human subjective errors, but also cannot quantitatively measure the defect size, category and contact mark evaluation parameters with high precision, and cannot meet the requirements of aircraft engines for improving the performance and quality consistency of aviation gears.

[0073] In order to solve the problem that the above-mentioned manual method has low detection accuracy and cannot meet the requirements of aircraft engines for improving the performance and quality consistency of aviation gears, this application proposes a three-dimensional color reconstruction method and system for gears based on the fusion of image and point cloud information.

[0074] Reference Figure 1 The embodiment of the present application provides a method for three-dimensional color reconstruction of gears based on image and point cloud information fusion, the method comprising the following steps:

[0075] Step S100, collecting first images and first point cloud data of calibration blocks at different heights and different positions to obtain multiple first images and multiple groups of first point cloud data, and collecting second images and second point cloud data of the gear, wherein the first images and first point cloud data of the calibration blocks at the same height and the same position are collected simultaneously;

[0076] Step S200: based on the plurality of first images, roughly identifying first rough two-dimensional corner point coordinates corresponding to each first image;

[0077] Step S300, determining multiple sets of first target two-dimensional corner point coordinates based on the first rough two-dimensional corner point coordinates corresponding to each first image;

[0078] Step S400, determining multiple sets of target three-dimensional corner point coordinates based on multiple sets of first point cloud data;

[0079] Step S500, calculating a multi-dimensional data conversion relationship between multiple sets of first target two-dimensional corner point coordinates and multiple sets of target three-dimensional corner point coordinates to obtain a conversion matrix;

[0080] Step S600: reconstructing a three-dimensional color model of the gear based on the transformation matrix, the second image and the second point cloud data.

[0081] In this embodiment, by collecting the first image and the first point cloud data of the calibration block at different heights and different positions, multiple first images and multiple groups of first point cloud data are obtained, and the second image and the second point cloud data of the gear are collected, wherein the first image and the first point cloud data of the calibration block at the same height and the same position are collected simultaneously; based on the multiple first images, the first rough two-dimensional corner point coordinates corresponding to each first image are roughly identified; based on the first rough two-dimensional corner point coordinates corresponding to each first image, multiple groups of first target two-dimensional corner point coordinates are determined; based on the multiple groups of first point cloud data, multiple groups of target three-dimensional corner point coordinates are determined; the multi-dimensional data conversion relationship between the multiple groups of first target two-dimensional corner point coordinates and the multiple groups of target three-dimensional corner point coordinates is calculated to obtain a conversion matrix; based on the conversion matrix, the second image and the second point cloud data, the three-dimensional color model of the gear is reconstructed. In this way, by using the first rough two-dimensional corner point coordinates to further determine the more accurate first target two-dimensional corner point coordinates, and then using the same method to determine the target three-dimensional corner point coordinates, and then according to the accurate first target two-dimensional corner point coordinates and the target three-dimensional corner point coordinates, calculate the accurate transformation matrix, and finally reconstruct the gear three-dimensional color model based on the accurate transformation matrix, it is possible to achieve high-precision gear three-dimensional color reconstruction, so as to improve the gear detection efficiency, improve the accuracy of gear detection, and lay an important foundation for multi-dimensional detection of gears.

[0082] The above-mentioned acquisition of the first image and the first point cloud data of the calibration block at different heights and positions can be to adjust the camera and the line laser scanner to suitable positions, align the center of the field of view of the camera and the line laser scanner with the calibration block, and allow the camera to be as close to the calibration block as possible outside the minimum working distance limit and ensure that the camera and the line laser scanner have a high field of view overlap rate, place the calibration block (i.e., the calibration block) at different heights and different positions, acquire the first image of the calibration block by the camera, and acquire the first point cloud data of the calibration block by the line laser scanner.

[0083] It should be noted that, in this embodiment, devices known to those skilled in the art may also be used to collect the first image and the first point cloud data, and this embodiment does not impose any specific limitation.

[0084] The above-mentioned acquisition of the second image and the second point cloud data of the gear can be achieved by adjusting the camera and the line laser scanner to suitable positions, aligning the centers of the fields of view of the camera and the line laser scanner with the gear, allowing the camera to be as close to the gear as possible outside the minimum working distance limit and ensuring that the camera and the line laser scanner have a high field of view overlap rate, acquiring the second image of the gear through the camera, and acquiring the second point cloud data of the gear through the line laser scanner.

[0085] The above-mentioned method of roughly identifying the first rough two-dimensional corner point coordinates corresponding to each first image based on the multiple first images may be to extract the edge two-dimensional pixel points of the first image using an edge detection method, and then roughly calculate the rough two-dimensional corner point coordinates corresponding to each first image based on the edge two-dimensional pixel points. The edge detection method may be a method known to those skilled in the art, such as the canny edge detection method, and this embodiment does not specifically limit this.

[0086] The above-mentioned method of determining multiple sets of first target two-dimensional corner point coordinates based on the first rough two-dimensional corner point coordinates corresponding to each first image can be to obtain a pixel value point set between two adjacent corner point coordinates in the four first rough two-dimensional corner point coordinates to obtain four point sets, and then iteratively calculate using a straight line fitting method based on the four point sets so that the fitted straight line is infinitely close to the four straight lines at the edge of the calibration block, and then obtain four straight lines after the iteration is completed, and then calculate the precise two-dimensional corner point coordinates based on the four straight lines, that is, obtain the first target two-dimensional corner point coordinates. The four first target two-dimensional corner point coordinates corresponding to each first image are taken as a group to obtain multiple sets of first target two-dimensional corner point coordinates.

[0087] Based on the multiple sets of first point cloud data, multiple sets of target three-dimensional corner point coordinates are determined. Since one set of first point cloud data can be used to calculate one set of target three-dimensional corner point coordinates, multiple sets of first point cloud data can be used to obtain multiple sets of target three-dimensional corner point coordinates.

[0088] The above-mentioned reconstruction of the gear three-dimensional color model based on the conversion matrix, the second image and the second point cloud data can be based on the conversion matrix, the color information in the second image is obtained, and then the color information is assigned to the second point cloud data through the conversion matrix, and then the gear three-dimensional color model is reconstructed according to the colored point cloud data. It should be noted that the reconstruction of the gear three-dimensional color model according to the point cloud data can be a method for three-dimensional reconstruction according to point clouds known to those skilled in the art, and this embodiment does not make specific limitations and descriptions.

[0089] In some implementations, based on the plurality of first images, roughly identifying first rough two-dimensional corner point coordinates corresponding to each first image includes:

[0090] Performing filtering and edge detection on the calibration blocks in the plurality of first images to obtain edge two-dimensional pixel points of the calibration block in each first image;

[0091] Downsampling edge two-dimensional pixel points of the calibration block in each first image to obtain multiple groups of edge pixel points of the sampled calibration blocks, wherein a group of edge pixel points of the sampled calibration blocks corresponds to one first image;

[0092] Sorting edge pixel points of each group of sampled calibration blocks in a head-to-tail manner to obtain multiple groups of sorted edge pixel points of the calibration blocks;

[0093] Randomly select four sorted edge pixel points from each group of sorted edge pixel points of the calibration block, calculate the areas of the four sorted edge pixel points, and obtain a plurality of first areas;

[0094] The four edge two-dimensional pixel points with the largest areas among the multiple first areas are determined as rough two-dimensional corner point coordinates, and the first rough two-dimensional corner point coordinates corresponding to each first image are obtained.

[0095] In this embodiment, by determining the four edge two-dimensional pixel points with the largest area among multiple first areas as rough two-dimensional corner point coordinates, the first rough two-dimensional corner point coordinates corresponding to each first image are obtained, which can lay a good data foundation for the subsequent calculation of precise two-dimensional corner point coordinates.

[0096] The above-mentioned sorting of each group of sampled edge pixel points in a head-to-tail manner may be sorting of each group of sampled edge pixel points in a head-to-tail manner according to the coordinates corresponding to each edge pixel point.

[0097] In some implementations, determining multiple sets of first target two-dimensional corner point coordinates based on the first rough two-dimensional corner point coordinates corresponding to each first image includes:

[0098] The other edge two-dimensional pixel points between two adjacent rough two-dimensional corner point coordinates in the first rough two-dimensional corner point coordinates corresponding to each first image are saved into a set to obtain four point sets corresponding to each first image;

[0099] Perform iterative straight line fitting on each point set to obtain four straight lines;

[0100] Calculate the intersection coordinates of the four straight lines to obtain the four intersection coordinates corresponding to each first image;

[0101] The four intersection point coordinates are used as a set of target two-dimensional corner point coordinates to obtain multiple sets of first target two-dimensional corner point coordinates.

[0102] In this embodiment, by saving the other edge two-dimensional pixel points between two adjacent rough two-dimensional corner point coordinates in the first rough two-dimensional corner point coordinates corresponding to each first image into a set, four point sets corresponding to each first image are obtained; iterative straight line fitting is performed on each point set to obtain four straight lines; the intersection coordinates of the four straight lines are calculated to obtain four intersection coordinates corresponding to each first image; the four intersection coordinates are used as a set of target two-dimensional corner point coordinates to obtain multiple sets of first target two-dimensional corner point coordinates. In this way, accurate two-dimensional corner point coordinates can be calculated, which lays a good data foundation for the subsequent calculation of accurate transformation matrix, so as to achieve high-precision three-dimensional color reconstruction of gears.

[0103] In the above, the other edge two-dimensional pixel points between two adjacent rough two-dimensional corner point coordinates in the first rough two-dimensional corner point coordinates corresponding to each first image are saved into a set. Since the edge pixel points are sorted according to the principle of head-to-tail connection, the pixel coordinates of all edge pixel points constitute a list, and each edge pixel point corresponds to a coordinate index. For example, the starting coordinate index of the list is 0, and the end coordinate index is the number of edge pixel points minus 1. In this way, the coordinate indexes corresponding to the four first rough two-dimensional corner point coordinates can be obtained, and then the other edge two-dimensional pixel points between the two adjacent rough two-dimensional corner point coordinates are obtained according to the coordinate indexes, and finally the other edge two-dimensional pixel points between the two adjacent rough two-dimensional corner point coordinates are saved into a set.

[0104] The above iterative straight line fitting for each point set to obtain four straight lines can be performed by iterative straight line fitting for each point set using a linear least squares method to obtain four straight lines. It should be noted that the linear least squares method is a prior art and is not described in detail in this embodiment.

[0105] In some implementations, determining multiple sets of target three-dimensional corner point coordinates based on multiple sets of first point cloud data includes:

[0106] Performing plane fitting on each set of first point cloud data to obtain a fitting plane corresponding to each set of first point cloud data, wherein the first point cloud data is point cloud data of the top plane of the calibration block collected by the line laser scanner;

[0107] Calculate the normal vector of the fitting plane corresponding to each group of first point cloud data to obtain multiple normal vectors;

[0108] Calculate the angle between each normal vector and the Z axis in the line laser scanner coordinate system to obtain multiple angles;

[0109] According to the normal vector and the angle corresponding to each group of first point cloud data, the point cloud data on the fitting plane corresponding to each group of first point cloud data is projected onto a plane to obtain a plurality of projected planes;

[0110] Encode each point on the projected plane to obtain multiple sets of encoded coordinate values;

[0111] Based on each set of encoded coordinate values, roughly identify the second rough two-dimensional corner point coordinates corresponding to each projected plane;

[0112] Determine multiple sets of second target two-dimensional corner point coordinates based on the second rough two-dimensional corner point coordinates corresponding to each projected plane;

[0113] The multiple sets of second target two-dimensional corner point coordinates are inversely transformed to obtain multiple sets of target three-dimensional corner point coordinates.

[0114] In this embodiment, a fitting plane corresponding to each group of first point cloud data is obtained by performing plane fitting on each group of first point cloud data, where the first point cloud data is the point cloud data of the top plane of the calibration block collected by the line laser scanner; the normal vector of the fitting plane corresponding to each group of first point cloud data is calculated to obtain multiple normal vectors; the angle between each normal vector and the Z axis in the coordinate system of the line laser scanner is calculated to obtain multiple angles; according to the normal vector and angle corresponding to each group of first point cloud data, the point cloud data on the fitting plane corresponding to each group of first point cloud data is projected to the plane to obtain multiple projected planes; each group of projected plane points is encoded to obtain multiple groups of encoded coordinate values; based on each group of encoded coordinate values, the second rough two-dimensional corner point coordinates corresponding to each fitting plane are roughly identified; based on the second rough two-dimensional corner point coordinates corresponding to each fitting plane, multiple groups of second target two-dimensional corner point coordinates are determined; the multiple groups of second target two-dimensional corner point coordinates are inversely transformed to obtain multiple groups of target three-dimensional corner point coordinates. In this way, by projecting the three-dimensional point cloud data into two-dimensional plane points, and then determining the rough two-dimensional corner point coordinates and the precise two-dimensional corner point coordinates (i.e., the target two-dimensional corner point coordinates), and then inversely transforming multiple sets of second target two-dimensional corner point coordinates, we can obtain multiple sets of precise target three-dimensional corner point coordinates, thus laying a good data foundation for the subsequent calculation of the precise transformation matrix, so as to achieve high-precision three-dimensional color reconstruction of gears.

[0115] The above-mentioned plane fitting is performed on each group of first point cloud data to obtain the fitting plane corresponding to each group of first point cloud data. This can be achieved by calculating the distance from all point cloud data on the calibration block to the fitting plane after each plane fitting, removing a certain ratio (for example, 5% or 10%) of the point cloud data, for example, removing some point cloud data farthest from the fitting plane, and after several rounds of iterations, a fitting plane that meets the requirements can be obtained, and the fitting plane is infinitely close to the real plane of the calibration block.

[0116] The above calculation of the normal vector of the fitting plane corresponding to each set of first point cloud data may be to use a linear least squares method to calculate the normal vector of the fitting plane corresponding to each set of first point cloud data. The linear least squares method is a prior art and will not be described in detail in this embodiment.

[0117] The above-mentioned calculation of the angle between the normal vector and the Z axis in the line laser scanner coordinate system can be performed by using a method known to those skilled in the art, which is not described in detail in this embodiment.

[0118] The above-mentioned method roughly identifies the second rough two-dimensional corner point coordinates corresponding to each projected plane based on each group of encoded coordinate values, and can filter and detect edges of each projected plane to obtain the edge two-dimensional pixel points of each projected plane; downsample the edge two-dimensional pixel points of each projected plane to obtain multiple groups of sampled edge pixel points; sort each group of sampled edge pixel points in a head-to-tail manner to obtain multiple groups of sorted edge pixel points; calculate the area of ​​four sorted edge pixel points randomly selected from each group of sorted edge pixel points each time to obtain multiple second areas; determine the four edge two-dimensional pixel points with the largest areas in the multiple second areas as the rough two-dimensional corner point coordinates to obtain the second rough two-dimensional corner point coordinates corresponding to each projected plane.

[0119] The above-mentioned method of determining multiple sets of second target two-dimensional corner point coordinates based on the second rough two-dimensional corner point coordinates corresponding to each projected plane can be that other edge two-dimensional pixel points between two adjacent rough two-dimensional corner point coordinates in the second rough two-dimensional corner point coordinates corresponding to each projected plane are saved into a set to obtain four point sets corresponding to each projected plane; iterative straight line fitting is performed on each point set to obtain four straight lines; intersection coordinates of the four straight lines are calculated to obtain four intersection coordinates corresponding to each projected plane; the four intersection coordinates are used as a set of target two-dimensional corner point coordinates to obtain multiple sets of second target two-dimensional corner point coordinates.

[0120] The above-mentioned inverse transformation of multiple sets of second target two-dimensional corner point coordinates to obtain multiple sets of target three-dimensional corner point coordinates can be performed by first converting the encoded coordinate values ​​of the four second target two-dimensional corner point coordinates back to the plane point coordinate values ​​after projection, and then adding the normal vector Z coordinate value of the fitted plane during the projection process, and then converting the calculated normal vector and the angle between the Z axis in the line laser scanner coordinate system back to the three-dimensional corner point coordinates at the original calibration block position (i.e., the target three-dimensional corner point coordinates).

[0121] In some embodiments, according to the normal vector and the angle corresponding to each set of first point cloud data, the point cloud data on the fitting plane corresponding to each set of first point cloud data is projected onto a plane to obtain a plurality of projected planes, including:

[0122] Make the normal vector corresponding to each set of first point cloud data parallel to the Z axis in the line laser scanner coordinate system, and make the fitting plane corresponding to each set of first point cloud data parallel to the XOY plane in the line laser scanner coordinate system;

[0123] After the fitting plane corresponding to each set of first point cloud data is parallel to the XOY plane in the line laser scanner coordinate system, the point cloud data on the fitting plane corresponding to each set of first point cloud data is projected onto the XOY plane to obtain a plurality of projected planes.

[0124] In this embodiment, the normal vector corresponding to each set of first point cloud data is parallel to the Z axis in the line laser scanner coordinate system, so that the fitting plane corresponding to each set of first point cloud data is parallel to the XOY plane in the line laser scanner coordinate system; after the fitting plane corresponding to each set of first point cloud data is parallel to the XOY plane in the line laser scanner coordinate system, the point cloud data on the fitting plane corresponding to each set of first point cloud data is projected onto the XOY plane to obtain multiple projected planes. In this way, the accurate two-dimensional corner point coordinates of the target can be calculated through multiple sets of projected plane points, laying a good data foundation for calculating the accurate three-dimensional corner point coordinates.

[0125] In some implementations, each point on the projected plane is encoded to obtain multiple sets of encoded coordinate values, including:

[0126] Preset coding area interval;

[0127] The coordinates of the plane point with the smallest X and Y values ​​in each projected plane are taken as the coordinates of the origin of the encoding coordinate system;

[0128] The coordinates of all plane points in each projected plane are subtracted from the coordinates of the origin of the coding coordinate system and then divided by the coding area interval to obtain multiple sets of coding coordinate values.

[0129] In this embodiment, by presetting the coding area interval, taking the coordinates of the plane point with the smallest X and Y values ​​in each projected plane as the coordinates of the origin of the coding coordinate system, and subtracting the coordinates of the origin of the coding coordinate system from the coordinates of all plane points in each projected plane and dividing by the coding area interval, multiple sets of coding coordinate values ​​are obtained. In this way, calculating multiple sets of coding coordinate values ​​can lay a good data foundation for calculating accurate three-dimensional corner point coordinates.

[0130] In some embodiments, reconstructing a three-dimensional color model of a gear based on the transformation matrix, the second image, and the second point cloud data includes:

[0131] According to the conversion matrix, the second point cloud data is projected onto the second image, and the RGB information of the pixel value on the second image corresponding to each second point cloud data is recorded;

[0132] Mapping the RGB information to the second point cloud data having a three-dimensional space through a conversion matrix to obtain color point cloud data;

[0133] Reconstruct the gear 3D color model based on the color point cloud data.

[0134] In this embodiment, the second point cloud data is projected onto the second image according to the conversion matrix, and the RGB information of the pixel value on the second image corresponding to each second point cloud data is recorded; the RGB information is mapped to the second point cloud data with three-dimensional space through the conversion matrix to obtain color point cloud data; based on the color point cloud data, the three-dimensional color model of the gear is reconstructed. In this way, the RGB information of the gear is converted to the point cloud data of the gear tooth surface through an accurate conversion matrix, so that high-precision three-dimensional color reconstruction of the gear can be achieved.

[0135] To facilitate understanding by those skilled in the art, a set of best embodiments is provided below:

[0136] With the rapid development of China's aviation industry, the domestic demand for aircraft is increasing day by day. Whether it is civil aviation, commercial aircraft or military aircraft, their technical level and production capacity are constantly improving. The core system of aircraft is the aircraft engine, and the core part is the aviation gear in the transmission system. The design and manufacturing of aviation gears have made great progress in the past few decades, but aviation gear detection is still mainly based on coordinate measuring machine point sampling and manual visual inspection. The level of automation and intelligence is low, which is no longer suitable for the current development needs of high-speed intelligent digitalization in the aviation industry.

[0137] The content of aviation gear inspection is relatively complex and diverse, including tooth shape, tooth direction, surface defects and contact marks. Among them, the measurement of surface defects and contact marks is mainly completed by manual visual inspection. Specifically, the quality inspector observes the inner and outer surfaces of the gear to see if there are surface defects such as scratches, dents and rust, and then records the inspection results of the inspected products; manually sticks contact marks, measures contact mark parameters with calipers and records them. These manual methods not only have problems such as low efficiency, poor data storage, maintainability and large human subjective errors, but also cannot quantitatively measure the defect size, category and contact mark evaluation parameters with high precision, and cannot meet the requirements of aircraft engines for improving the performance and quality consistency of aviation gears.

[0138] The main reason why deep learning and visual measurement technology cannot be used to complete defect detection and contact mark measurement tasks with high precision, quantitatively and qualitatively is that these goals of aviation gear detection involve multi-dimensional information, which requires not only the color texture information of the image, but also the coordinate space information of the point cloud. Therefore, studying a data fusion method of image and point cloud information, that is, a three-dimensional color reconstruction method of aviation gears based on image and point cloud information, has become one of the important topics in the current research on efficient, high-precision and intelligent detection of aviation gears. This has far-reaching engineering value for promoting the integrated process of innovative design, manufacturing and detection of current aviation engines and transmission systems, and ensuring the quality, safety and service life of aviation power system products.

[0139] In order to address the limitations of current manual defect detection and contact mark measurement of aviation gears, this embodiment proposes a high-precision multi-dimensional data fusion method. Based on the collected image and point cloud information of aviation gears, a low-cost calibration block is combined with a high-precision calibration block corner point extraction method to solve the conversion relationship between multi-dimensional data, and the image color texture information of the aviation gear is converted to the gear tooth surface point cloud, thereby achieving high-precision three-dimensional color reconstruction of aviation gears.

[0140] The technical solution of this embodiment fills the technical gap in the field of multi-dimensional surface defect detection and contact mark measurement of aviation gears in production practice, and provides important technical support for realizing multi-dimensional information fusion measurement technologies such as high-precision quantitative surface defect detection and contact mark measurement of aviation gears.

[0141] This embodiment proposes a method for 3D color reconstruction of aviation gears based on the fusion of image and point cloud information. It should be noted that in addition to the aviation gears of this embodiment, other gear detections can also adopt the technical solution of this embodiment, and this embodiment does not impose specific restrictions. The method of this embodiment mainly includes the following parts: high-precision and rapid identification of 2D corner points based on a calibration block with two-stage coarse-fine recognition; 3D corner point identification based on a high-density calibration block with projection coding and rectangular features; solving multi-dimensional data conversion relationships; and 3D color reconstruction of aviation gears. The specific contents are:

[0142] 1. First, adjust the camera and line laser scanner to the appropriate position, align the center of the field of view of the camera and line laser scanner with the calibration block, let the camera be as close to the calibration block as possible outside the minimum working distance limit, and ensure that the camera and line laser have a high field of view overlap rate. Place the calibration block (i.e., calibration block) at different heights and different positions to collect multiple sets of calibration data (including images and point cloud data). Generally, 5 to 8 sets of pictures and point cloud data are required, and at least two sets of pictures and point cloud data are required. The specific number of sets of pictures and point cloud data required can be changed according to actual conditions, and this embodiment does not impose specific restrictions. The schematic diagram of collecting pictures and point cloud data of the calibration block is shown in the figure below. Figure 2 shown.

[0143] For the captured image, the 2D corner points are quickly and accurately identified through the calibration block based on the two-stage coarse-fine recognition. The edge 2D pixel points of the calibration block image are obtained by image preprocessing, such as filtering and edge detection, and then the edge 2D pixel points of the obtained calibration block image are downsampled.

[0144] The downsampled edge pixels are sorted so that the order of these edge pixels is connected head to tail. Then, four edge pixels are randomly selected from these edge pixels as a set, and the area of ​​the quadrilateral formed by all sets connected head to tail is calculated. The four edge pixels with the largest area are the corner point results of the first stage of coarse recognition (i.e., the first coarse two-dimensional corner point coordinates), as shown in Figure 3 shown.

[0145] The four roughly identified corner points are placed in the unsampled edge pixel points (i.e., the obtained edge two-dimensional pixel points of the calibration block image), and the pixel coordinate indexes corresponding to the four corner points are obtained; and according to the coordinate index, the other pixel points between two adjacent points are saved in a set, so as to obtain four point sets containing adjacent coarse corner points. The pixel coordinate indexes corresponding to the four corner points can be obtained in the following way:

[0146] Since the edge pixels are sorted in the principle of connecting head to tail, the pixel coordinates of all points form a list, and each point corresponds to an index (that is, the starting coordinate index of the list is 0, and the end coordinate index is the number of edge pixels - 1). In Python, the known functions can be used to obtain the coordinate index of the target point corresponding to the list from the list, so it is possible to obtain the pixel coordinate index corresponding to the four corner points.

[0147] The edge pixel points in the above four point sets are respectively subjected to iterative straight line fitting. After each fitting, the distance from each edge pixel point in the point set to the straight line is calculated, and a certain ratio (such as 5% or 10%) of the points farthest from the straight line are removed. After several rounds of iteration, four straight lines that are infinitely close to the edge of the calibration block can be obtained.

[0148] The intersection coordinates of the four straight lines obtained by the above processing are calculated by the method of finding the intersection of two straight line equations, which are the 2D precise corner points of the identified calibration block (i.e. the coordinates of the first target two-dimensional corner points), as shown in Figure 4 shown.

[0149] 2. For 3D corner point recognition of the high-density calibration block based on projection coding and rectangular features, a plane is fitted on the top of the 3D calibration block point cloud obtained by the line laser scanner, and the normal vector of the plane is calculated. Then the angle θ between the normal vector and the Z axis in the line laser coordinate system is solved. Specifically:

[0150] When fitting the calibration block plane, an iterative fitting method is used. After each plane fitting, the distances from all points on the calibration block point cloud to the plane are calculated, and a certain ratio (such as 5% or 10%) of the points farthest from the fitting plane are removed. After several rounds of iteration, a fitting plane that is infinitely close to the true plane of the calibration block can be obtained.

[0151] The fitting plane is rotated to be parallel to the XOY plane of the line laser scanner coordinate system using the angle obtained above, that is, the normal vector of the fitting plane is parallel to the Z axis of the line laser scanner coordinate system. The point cloud on the rotated calibration block fitting plane is then projected onto the XOY plane of the line laser scanner coordinate system, that is, the Z coordinates of these point clouds are set to zero.

[0152] Encode the point cloud projected onto the XOY plane (i.e., the points on the projected plane) to obtain the coordinate values ​​of the encoded coordinate system (i.e., the encoded coordinate values). The specific operations are as follows:

[0153] The coding range is set according to the calibration block range scanned by the line laser scanner, and the interval of the coding area is set according to the accuracy of the line laser scanner. For example, assuming that the maximum interval of the point cloud scanned by the line laser is 10um, the coding area interval can be set to 10um, that is, within a rectangular range, the rectangular range is divided into grid areas according to the coding area interval, wherein the interval between adjacent small cells in the grid is 10um.

[0154] Then divide the points projected on the XOY plane into corresponding grids, and calculate the coordinate values ​​of the projected coded coordinate system. The specific operation is to define the smallest X, Y value of all points projected on the XOY plane as the origin of the coded coordinate system, and then subtract the origin from the coordinates of all points projected on the XOY plane and divide them by the coding area interval to obtain the coordinate value of the coded coordinate system corresponding to each point (i.e., the coded coordinate value).

[0155] Through the above encoding conversion process, the point cloud can be converted to a two-dimensional plane, such as Figure 5 shown.

[0156] The encoding coordinate system can be regarded as a pixel coordinate system. The coordinate points in the encoding coordinate system are preprocessed. The same method as the above-mentioned "2D corner point high-precision and fast recognition based on the calibration block of two-stage coarse-fine recognition" is used to calculate the four 2D corner points on the XOY plane, such as Figure 6 As shown, the specific process is:

[0157] right Figure 5 The image is preprocessed on the two-dimensional plane in the coded coordinate system, such as filtering and edge detection, to obtain edge two-dimensional pixel points of the two-dimensional plane, and then the obtained edge two-dimensional pixel points of the two-dimensional plane are downsampled.

[0158] The downsampled edge pixels are sorted so that they are connected end to end. Then, four edge pixels are randomly selected from these edge pixels as a set, and the area of ​​the quadrilateral formed by all the sets connected end to end is calculated. The four edge pixels with the largest area are the corner point results of the first stage of coarse recognition (i.e., the second coarse two-dimensional corner point coordinates).

[0159] The pixel coordinate indices corresponding to the four roughly identified corner points are obtained in the unsampled edge pixel points (i.e., the edge two-dimensional pixel points of the obtained two-dimensional plane); and according to the coordinate index, the other pixel points between two adjacent points are saved in a set, so as to obtain four point sets containing adjacent coarse corner points.

[0160] The edge pixel points in the above four point sets are respectively subjected to iterative straight line fitting. After each fitting, the distance from each edge pixel point in the point set to the straight line is calculated, and a certain ratio (such as 5% or 10%) of the points farthest from the straight line are removed. After several rounds of iteration, four straight lines that are infinitely close to the edge of the calibration block can be obtained.

[0161] The intersection coordinates of the four straight lines obtained by the above processing are calculated by the method of finding the intersection point of two straight line equations, which are the 2D precise corner points of the identified calibration block (ie the coordinates of the second target two-dimensional corner points).

[0162] Finally, the four 2D corner points in the coded coordinate system obtained through the above operation are inversely transformed to obtain the corresponding four 3D corner point coordinate values. The specific operation is: first convert the coded coordinate values ​​of the four 2D corner points back to the projection coordinate system values, and then add the Z coordinate value of the normal vector of the fitted plane during the projection process, as well as the Z-axis angle between the normal vector and the line laser coordinate system calculated at the beginning to convert back to the 3D corner point coordinate value (i.e., the target 3D corner point coordinate) at the original calibration block position, such as Figure 7 shown.

[0163] 3. To solve the multi-dimensional data conversion relationship, the 2D corner points and 3D corner points of the calibration block obtained by the above steps are matched. The principle is to make the positions of their corner points correspond one-to-one in the real space, and then solve the multi-dimensional data conversion relationship.

[0164] Specifically, the corner point matching of 2D corner points and 3D corner points is performed by calculating the center coordinates of the four identified corner points, and the four corner points identified in 2D and 3D space are numbered 1, 2, 3, 4 and A, B, C, D in clockwise or counterclockwise order, respectively, and the field of view correspondence between the camera and the line laser is determined according to the position of the sensor. For example, number 1 in 2D space corresponds to A in 3D space, number 2 in 2D space corresponds to B in 3D space, number 3 in 2D space corresponds to C in 3D space, and number 4 in 2D space corresponds to D in 3D space.

[0165] By solving multiple sets of calibration block images and point cloud data, multiple sets of 2D-3D corner point pairs can be obtained, that is, the multi-dimensional data conversion relationship can be solved (that is, the conversion matrix is ​​obtained), that is, the conversion relationship from the pixel coordinate system to the line laser coordinate system.

[0166] 4. Perform three-dimensional color reconstruction of the aviation gear through the obtained conversion relationship (i.e., conversion matrix). The specific operation is: collect the second image and second point cloud data of the aviation gear, project the point cloud (i.e., second point cloud data) onto the plane of the two-dimensional image (i.e., second image) through the conversion matrix, the point cloud corresponds to a certain pixel value and records the RGB information of the pixel value, and then restore the RGB information to the 3D space to obtain the three-dimensional color reconstruction model of the aviation gear, such as Figure 8 shown.

[0167] The three-dimensional color reconstruction of the aviation gear using the obtained conversion relationship (i.e., the conversion matrix) may further include the following steps:

[0168] The two-dimensional image data is converted to three-dimensional space using the conversion matrix between the two-dimensional corner point coordinates and the three-dimensional corner point coordinates, and the corresponding relationship between a single pixel in the two-dimensional image data and the point cloud data in the three-dimensional space is obtained;

[0169] If a single pixel in the two-dimensional image data corresponds to multiple point clouds in the point cloud data, the point cloud with the largest point in the Z direction among the multiple point clouds corresponding to the single pixel is retained to obtain the color point cloud data;

[0170] Reconstruct the gear 3D color model based on the color point cloud data.

[0171] It should be noted that when mapping two-dimensional image data to three-dimensional space, a single pixel may correspond to multiple point cloud data. At this time, in order to ensure detection accuracy, only the largest point cloud in the Z direction will be retained. That is, from a bird's-eye view, the color image acquisition device usually captures the color of the outermost surface of the gear. Therefore, using the outermost point cloud data can make the conversion correspondence more accurate.

[0172] The three-dimensional color reconstruction of the aviation gear is performed using the obtained conversion relationship (i.e., the conversion matrix), and the following steps may also be included:

[0173] The two-dimensional image data is converted to three-dimensional space using the conversion matrix between the two-dimensional corner point coordinates and the three-dimensional corner point coordinates, and the corresponding relationship between a single pixel in the two-dimensional image data and the point cloud data in the three-dimensional space is obtained;

[0174] If a single pixel in the two-dimensional image data corresponds to multiple point clouds in the point cloud data, the point cloud with the largest Z direction among the multiple point clouds corresponding to the single pixel is retained, and the point clouds with an average height exceeding the preset pixel allocation requirement are retained based on the clustering method to obtain color point cloud data;

[0175] Reconstruct the gear 3D color model based on the color point cloud data.

[0176] On the basis of retaining the largest point cloud in the Z direction, we can further consider that due to the large density of the point cloud, some points will be assigned to adjacent pixels, so we can use the clustering method to save the point cloud clusters with a larger average height.

[0177] Compared with the prior art, the technical solution of this embodiment has the following advantages:

[0178] The method for 3D color reconstruction of aviation gears based on the fusion of image and point cloud information solves the multi-dimensional data conversion relationship from image to point cloud through a high-precision rapid recognition method for 2D corner points of the calibration block based on two-stage coarse-fine recognition and a high-density calibration block 3D corner point recognition method based on projection coding and rectangular features; and realizes 3D color reconstruction of aviation gears. It provides important technical support for multi-dimensional information fusion measurement technologies such as high-precision quantitative surface defect detection and contact mark measurement of aviation gears. It can be applied to various models of aviation gears for 3D color reconstruction, such as Fig. 9 As shown, it can complete complex tasks such as multi-dimensional surface defect detection and imprint measurement, and has universality.

[0179] Reference Fig.10 The embodiment of the present application also provides a gear three-dimensional color reconstruction system based on image and point cloud information fusion, the system includes a data acquisition unit 100, a rough recognition unit 200, a first data determination unit 300, a second data determination unit 400, a conversion matrix calculation unit 500 and a color model reconstruction unit 600, wherein:

[0180] The data acquisition unit 100 is used to acquire first images and first point cloud data of calibration blocks at different heights and different positions, obtain multiple first images and multiple groups of first point cloud data, and acquire second images and second point cloud data of the gear, wherein the first images and first point cloud data of the calibration blocks at the same height and the same position are acquired simultaneously;

[0181] A coarse identification unit 200, configured to coarsely identify first coarse two-dimensional corner point coordinates corresponding to each first image based on the plurality of first images;

[0182] The first data determination unit 300 is used to determine multiple sets of first target two-dimensional corner point coordinates based on the first rough two-dimensional corner point coordinates corresponding to each first image;

[0183] The second data determination unit 400 is used to determine multiple sets of target three-dimensional corner point coordinates based on multiple sets of first point cloud data;

[0184] The conversion matrix calculation unit 500 is used to calculate the multi-dimensional data conversion relationship between multiple sets of first target two-dimensional corner point coordinates and multiple sets of target three-dimensional corner point coordinates to obtain a conversion matrix;

[0185] The color model reconstruction unit 600 is used to reconstruct a three-dimensional color model of the gear based on the conversion matrix, the second image and the second point cloud data.

[0186] It should be noted that since the gear three-dimensional color reconstruction system based on image and point cloud information fusion in this embodiment and the above-mentioned gear three-dimensional color reconstruction method based on image and point cloud information fusion are based on the same inventive concept, the corresponding contents in the method embodiment are also applicable to the system embodiment and will not be described in detail here.

[0187] Reference Fig.11 The embodiment of the present application further provides an electronic device, the electronic device comprising:

[0188] at least one memory;

[0189] at least one processor;

[0190] at least one program;

[0191] The program is stored in the memory, and the processor executes at least one program to implement the three-dimensional color reconstruction method of gears based on the fusion of image and point cloud information as described above in the present disclosure.

[0192] The electronic device may be any intelligent terminal including a mobile phone, a tablet computer, a personal digital assistant (PDA), a vehicle-mounted computer, etc.

[0193] The electronic device according to the embodiment of the present application is described in detail below.

[0194] The processor 1600 may be implemented by a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present disclosure;

[0195] The memory 1700 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1700 can store an operating system and other application programs. When the technical solution provided in the embodiments of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1700, and the processor 1600 calls and executes the gear 3D color reconstruction method based on image and point cloud information fusion in the embodiment of the present disclosure.

[0196] Input / output interface 1800, used to implement information input and output;

[0197] The communication interface 1900 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0198] Bus 2000 , which transmits information between various components of the device (e.g., processor 1600 , memory 1700 , input / output interface 1800 , and communication interface 1900 );

[0199] The processor 1600 , the memory 1700 , the input / output interface 1800 , and the communication interface 1900 are connected to each other in communication within the device via the bus 2000 .

[0200] The embodiment of the present disclosure also provides a storage medium, which is a computer-readable storage medium, and the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the above-mentioned gear three-dimensional color reconstruction method based on the fusion of image and point cloud information.

[0201] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0202] The embodiments described in the embodiments of the present disclosure are intended to more clearly illustrate the technical solutions of the embodiments of the present disclosure and do not constitute a limitation on the technical solutions provided by the embodiments of the present disclosure. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present disclosure are also applicable to similar technical problems.

[0203] Those skilled in the art will appreciate that the technical solutions shown in the figures do not limit the embodiments of the present disclosure and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0204] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0205] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0206] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0207] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0208] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0209] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0210] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0211] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable an electronic device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store programs. The above is a detailed description of the embodiments of the present application in conjunction with the accompanying drawings, but the present application is not limited to the above embodiments. Within the scope of knowledge possessed by ordinary technicians in the relevant technical field, various changes can be made without departing from the purpose of the present application.

[0212] The embodiments of the present application are described in detail above in conjunction with the accompanying drawings, but the present application is not limited to the above embodiments. Various changes can be made within the knowledge scope of ordinary technicians in the relevant technical field without departing from the purpose of the present application.

Claims

1. A gear 3D color reconstruction method based on image and point cloud information fusion, characterized in that: The method comprises: Collecting first images and first point cloud data of calibration blocks at different heights and different positions to obtain multiple first images and multiple groups of first point cloud data, and collecting second images and second point cloud data of the gear, wherein the first images and first point cloud data of the calibration blocks at the same height and the same position are collected simultaneously; Based on the multiple first images, roughly identifying first rough two-dimensional corner point coordinates corresponding to each of the first images; Determine multiple sets of first target two-dimensional corner point coordinates based on the first rough two-dimensional corner point coordinates corresponding to each of the first images; Based on the multiple sets of first point cloud data, determining multiple sets of target three-dimensional corner point coordinates; Calculating a multi-dimensional data conversion relationship between the multiple sets of first target two-dimensional corner point coordinates and the multiple sets of target three-dimensional corner point coordinates to obtain a conversion matrix; A three-dimensional color model of a gear is reconstructed based on the transformation matrix, the second image and the second point cloud data.

2. The gear 3D color reconstruction method based on image and point cloud information fusion according to claim 1 is characterized in that: The step of roughly identifying first rough two-dimensional corner point coordinates corresponding to each of the first images based on the plurality of first images includes: Performing filtering and edge detection on the calibration blocks in the plurality of first images to obtain edge two-dimensional pixel points of the calibration block in each of the first images; Downsampling edge two-dimensional pixel points of the calibration block in each of the first images to obtain multiple groups of edge pixel points of the sampled calibration blocks, wherein a group of edge pixel points of the sampled calibration blocks corresponds to one of the first images; Sorting each group of edge pixel points of the sampled calibration block in a head-to-tail manner to obtain multiple groups of edge pixel points of the sorted calibration blocks; Randomly select four sorted edge pixel points from each group of edge pixel points of the sorted calibration block, calculate the areas of the four sorted edge pixel points, and obtain a plurality of first areas; The four edge two-dimensional pixel points with the largest areas among the multiple first areas are determined as rough two-dimensional corner point coordinates, and first rough two-dimensional corner point coordinates corresponding to each of the first images are obtained.

3. The gear 3D color reconstruction method based on image and point cloud information fusion according to claim 1 is characterized in that: The determining of multiple sets of first target two-dimensional corner point coordinates based on the first rough two-dimensional corner point coordinates corresponding to each of the first images includes: The other edge two-dimensional pixel points between two adjacent rough two-dimensional corner point coordinates in the first rough two-dimensional corner point coordinates corresponding to each of the first images are saved into a set to obtain four point sets corresponding to each of the first images; Perform iterative straight line fitting on each of the point sets to obtain four straight lines; Calculating the intersection coordinates of the four straight lines to obtain the coordinates of four intersection points corresponding to each of the first images; The four intersection point coordinates are used as a set of target two-dimensional corner point coordinates to obtain multiple sets of first target two-dimensional corner point coordinates.

4. The gear 3D color reconstruction method based on image and point cloud information fusion according to claim 1 is characterized in that: The step of determining multiple sets of target three-dimensional corner point coordinates based on the multiple sets of first point cloud data includes: Performing plane fitting on each group of the first point cloud data to obtain a fitting plane corresponding to each group of the first point cloud data, wherein the first point cloud data is point cloud data of the top plane of the calibration block collected by a line laser scanner; Calculating a normal vector of a fitting plane corresponding to each group of the first point cloud data to obtain a plurality of normal vectors; Calculating the angle between each normal vector and the Z axis in the line laser scanner coordinate system to obtain a plurality of angles; According to the normal vector and the angle corresponding to each group of the first point cloud data, the point cloud data on the fitting plane corresponding to each group of the first point cloud data is projected onto a plane to obtain a plurality of projected planes; Encoding each point on the projected plane to obtain multiple sets of encoded coordinate values; Based on each group of the encoded coordinate values, roughly identifying the second rough two-dimensional corner point coordinates corresponding to each of the projected planes; Determine multiple sets of second target two-dimensional corner point coordinates based on the second rough two-dimensional corner point coordinates corresponding to each of the projected planes; The multiple sets of second target two-dimensional corner point coordinates are inversely transformed to obtain multiple sets of target three-dimensional corner point coordinates.

5. The gear 3D color reconstruction method based on image and point cloud information fusion according to claim 4 is characterized in that: The step of projecting the point cloud data on the fitting plane corresponding to each group of the first point cloud data onto a plane according to the normal vector and the angle corresponding to each group of the first point cloud data to obtain a plurality of projected planes includes: Make the normal vector corresponding to each group of the first point cloud data parallel to the Z axis in the line laser scanner coordinate system, and make the fitting plane corresponding to each group of the first point cloud data parallel to the XOY plane in the line laser scanner coordinate system; After the fitting plane corresponding to each group of the first point cloud data is parallel to the XOY plane in the line laser scanner coordinate system, the point cloud data on the fitting plane corresponding to each group of the first point cloud data is projected onto the XOY plane to obtain a plurality of projected planes.

6. The gear 3D color reconstruction method based on image and point cloud information fusion according to claim 4 is characterized in that: The step of encoding each point on the projected plane to obtain multiple sets of encoded coordinate values ​​includes: Preset coding area interval; Taking the coordinates of the plane point with the smallest X value and Y value in each projected plane as the coordinates of the origin of the encoding coordinate system; The coordinates of all plane points in each of the projected planes are subtracted from the coordinates of the origin of the coding coordinate system and then divided by the coding area interval to obtain multiple groups of coding coordinate values.

7. The gear 3D color reconstruction method based on image and point cloud information fusion according to claim 1 is characterized in that: The method of reconstructing a three-dimensional color model of a gear based on the transformation matrix, the second image and the second point cloud data comprises: According to the conversion matrix, the second point cloud data is projected onto the second image, and RGB information of a pixel value on the second image corresponding to each second point cloud data is recorded; Mapping the RGB information to second point cloud data having a three-dimensional space through a conversion matrix to obtain color point cloud data; Based on the color point cloud data, a three-dimensional color model of the gear is reconstructed.

8. A gear 3D color reconstruction system based on image and point cloud information fusion, characterized in that: The system comprises: A data acquisition unit is used to acquire first images and first point cloud data of calibration blocks at different heights and different positions, to obtain multiple first images and multiple groups of first point cloud data, and to acquire second images and second point cloud data of the gear, wherein the first images and first point cloud data of the calibration blocks at the same height and the same position are acquired simultaneously; a coarse identification unit, configured to coarsely identify first coarse two-dimensional corner point coordinates corresponding to each of the first images based on the multiple first images; A first data determination unit, configured to determine a plurality of sets of first target two-dimensional corner point coordinates based on first rough two-dimensional corner point coordinates corresponding to each of the first images; A second data determination unit, configured to determine a plurality of sets of target three-dimensional corner point coordinates based on the plurality of sets of first point cloud data; A conversion matrix calculation unit, used to calculate the multi-dimensional data conversion relationship between the multiple sets of first target two-dimensional corner point coordinates and the multiple sets of target three-dimensional corner point coordinates to obtain a conversion matrix; A color model reconstruction unit is used to reconstruct a three-dimensional color model of the gear based on the conversion matrix, the second image and the second point cloud data.

9. An electronic device, characterized in that: It includes at least one control processor and a memory for communicating with the at least one control processor; the memory stores instructions that can be executed by the at least one control processor, and the instructions are executed by the at least one control processor so that the at least one control processor can execute the three-dimensional color reconstruction method of gears based on image and point cloud information fusion as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the gear three-dimensional color reconstruction method based on image and point cloud information fusion as described in any one of claims 1 to 7.

Images