Gear 3D color reconstruction method and system based on image and point cloud information fusion

Abstract

This application discloses a method and system for three-dimensional color reconstruction of gears based on image and point cloud information fusion. The method collects first images and first point cloud data of calibration blocks at different heights and positions, and second images and second point cloud data of the gear; based on multiple first images, roughly identifies first coarse two-dimensional corner point coordinates corresponding to each first image; determines multiple sets of first target two-dimensional corner point coordinates based on the first coarse two-dimensional corner point coordinates corresponding to each first image; determines multiple sets of target three-dimensional corner point coordinates based on the multiple sets of first point cloud data; calculates the multi-dimensional data conversion relationship between the multiple sets of first target two-dimensional corner point coordinates and the multiple sets of target three-dimensional corner point coordinates to obtain a conversion matrix; and reconstructs a three-dimensional color model of the gear based on the conversion matrix, the second image, and the second point cloud data. This application achieves high-precision three-dimensional color reconstruction of the gear to improve the detection efficiency and accuracy of the gear.

Description

Technical Field

[0001] The present application relates to the field of three-dimensional reconstruction technology, and in particular to a method and system for three-dimensional color reconstruction of gears based on the fusion of image and point cloud information. Background Art

[0002] The design and manufacturing of aviation gears have made great progress in the past few decades. However, aviation gear inspection is still mainly based on coordinate measuring machine point sampling and manual visual inspection. The level of automation and intelligence is relatively low, which is no longer suitable for the current development needs of high-speed intelligent digitalization in the aviation industry.

[0003] Aviation gear inspections are complex and diverse, encompassing tooth profile, tooth orientation, surface defects, and contact marks. Surface defect and contact mark measurement primarily relies on manual visual inspection. Specifically, quality inspectors examine the inner and outer surfaces of the gears for surface defects such as scratches, dents, and rust, then record the inspection results. Contact marks are manually applied and contact mark parameters are measured and recorded using a caliper. These manual methods not only suffer from low efficiency, poor data storage and maintainability, and significant subjective errors, but also lack the ability to quantitatively and accurately measure defect size, type, and contact mark evaluation parameters, failing to meet the requirements of aircraft engines for improved performance and quality consistency in aviation gears. Summary of the Invention

[0004] This application aims to propose a gear three-dimensional color reconstruction method and system based on the fusion of image and point cloud information. By achieving high-precision gear three-dimensional color reconstruction, the efficiency of gear detection can be improved, the accuracy of gear detection can be improved, and an important foundation can be laid for multi-dimensional detection of gears.

[0005] In a first aspect, an embodiment of the present application provides a method for three-dimensional color reconstruction of a gear based on image and point cloud information fusion, the method comprising:

[0006] Acquire first images and first point cloud data of the calibration block at different heights and different positions to obtain multiple first images and multiple sets of first point cloud data, and acquire second images and second point cloud data of the gear, wherein the first images and first point cloud data of the calibration block at the same height and the same position are acquired simultaneously;

[0007] Based on the multiple first images, roughly identifying first rough two-dimensional corner point coordinates corresponding to each of the first images;

[0008] determining a plurality of sets of first target two-dimensional corner point coordinates based on the first rough two-dimensional corner point coordinates corresponding to each of the first images;

[0009] Determining multiple sets of target three-dimensional corner point coordinates based on the multiple sets of first point cloud data;

[0010] Calculating a multi-dimensional data conversion relationship between the multiple sets of first target two-dimensional corner point coordinates and the multiple sets of target three-dimensional corner point coordinates to obtain a conversion matrix;

[0011] A three-dimensional color model of the gear is reconstructed based on the transformation matrix, the second image and the second point cloud data.

[0012] Compared with the prior art, the first aspect of the present application has the following beneficial effects:

[0013] This method obtains multiple first images and multiple groups of first point cloud data by collecting first images and first point cloud data of calibration blocks at different heights and different positions, and collects second images and second point cloud data of gears, wherein the first images and first point cloud data of the calibration blocks at the same height and the same position are collected simultaneously; based on the multiple first images, the first rough two-dimensional corner point coordinates corresponding to each first image are roughly identified; based on the first rough two-dimensional corner point coordinates corresponding to each first image, multiple groups of first target two-dimensional corner point coordinates are determined; based on the multiple groups of first point cloud data, multiple groups of target three-dimensional corner point coordinates are determined; the multi-dimensional data conversion relationship between the multiple groups of first target two-dimensional corner point coordinates and the multiple groups of target three-dimensional corner point coordinates is calculated to obtain a conversion matrix; based on the conversion matrix, the second image and the second point cloud data, a three-dimensional color model of the gear is reconstructed. In this way, by using the first rough two-dimensional corner point coordinates to further determine the more accurate first target two-dimensional corner point coordinates, and then using the same method to determine the target three-dimensional corner point coordinates, and then according to the accurate first target two-dimensional corner point coordinates and the target three-dimensional corner point coordinates, calculate the accurate transformation matrix, and finally reconstruct the gear three-dimensional color model based on the accurate transformation matrix, high-precision gear three-dimensional color reconstruction can be achieved, so as to improve the detection efficiency of gears, improve the accuracy of gear detection, and lay an important foundation for multi-dimensional detection of gears.

[0014] In some embodiments, the step of roughly identifying first rough two-dimensional corner point coordinates corresponding to each of the plurality of first images includes:

[0015] Performing filtering and edge detection on the calibration blocks in the plurality of first images to obtain two-dimensional edge pixel points of the calibration block in each of the first images;

[0016] Downsampling edge two-dimensional pixel points of the calibration block in each of the first images to obtain multiple groups of edge pixel points of the sampled calibration blocks, wherein a group of edge pixel points of the sampled calibration blocks corresponds to one of the first images;

[0017] Sorting the edge pixel points of each group of the sampled calibration blocks in a head-to-tail manner to obtain multiple groups of edge pixel points of the sorted calibration blocks;

[0018] Randomly selecting four sorted edge pixel points from each group of edge pixel points of the sorted calibration block, and calculating the areas of the four sorted edge pixel points to obtain a plurality of first areas;

[0019] The four edge two-dimensional pixel points with the largest areas among the multiple first areas are determined as rough two-dimensional corner point coordinates to obtain first rough two-dimensional corner point coordinates corresponding to each of the first images.

[0020] In some embodiments, determining multiple sets of first target two-dimensional corner point coordinates based on the first rough two-dimensional corner point coordinates corresponding to each of the first images includes:

[0021] Saving other edge two-dimensional pixel points between two adjacent rough two-dimensional corner point coordinates in the first rough two-dimensional corner point coordinates corresponding to each of the first images into a set, to obtain four point sets corresponding to each of the first images;

[0022] Performing iterative straight line fitting on each of the point sets to obtain four straight lines;

[0023] Calculating the intersection coordinates of the four straight lines to obtain the coordinates of four intersection points corresponding to each of the first images;

[0024] The four intersection point coordinates are used as a set of target two-dimensional corner point coordinates to obtain multiple sets of first target two-dimensional corner point coordinates.

[0025] In some embodiments, determining multiple sets of target three-dimensional corner point coordinates based on the multiple sets of first point cloud data includes:

[0026] performing plane fitting on each set of the first point cloud data to obtain a fitting plane corresponding to each set of the first point cloud data, wherein the first point cloud data is point cloud data of the top plane of the calibration block acquired by a line laser scanner;

[0027] Calculating a normal vector of a fitting plane corresponding to each set of the first point cloud data to obtain a plurality of normal vectors;

[0028] Calculating the angle between each normal vector and the Z axis in the line laser scanner coordinate system to obtain a plurality of angles;

[0029] projecting the point cloud data on the fitting plane corresponding to each set of the first point cloud data onto a plane according to the normal vector and the angle corresponding to each set of the first point cloud data, to obtain a plurality of projected planes;

[0030] Encoding each point on the projected plane to obtain multiple sets of encoded coordinate values;

[0031] Based on each set of the encoded coordinate values, roughly identifying the second rough two-dimensional corner point coordinates corresponding to each of the projected planes;

[0032] Determining multiple sets of second target two-dimensional corner point coordinates based on the second rough two-dimensional corner point coordinates corresponding to each of the projected planes;

[0033] The multiple sets of second target two-dimensional corner point coordinates are inversely transformed to obtain multiple sets of target three-dimensional corner point coordinates.

[0034] In some embodiments, projecting the point cloud data on the fitting plane corresponding to each set of the first point cloud data onto a plane according to the normal vector and the angle corresponding to each set of the first point cloud data to obtain a plurality of projected planes includes:

[0035] Parallelizing the normal vector corresponding to each set of the first point cloud data to the Z axis in the line laser scanner coordinate system, and parallelizing the fitting plane corresponding to each set of the first point cloud data to the XOY plane in the line laser scanner coordinate system;

[0036] After the fitting plane corresponding to each group of the first point cloud data is parallel to the XOY plane in the line laser scanner coordinate system, the point cloud data on the fitting plane corresponding to each group of the first point cloud data is projected onto the XOY plane to obtain multiple projected planes.

[0037] In some embodiments, encoding each point on the projected plane to obtain multiple sets of encoded coordinate values ​​includes:

[0038] Preset coding area interval;

[0039] Taking the coordinates of the plane point with the smallest X value and Y value in each projected plane as the coordinates of the origin of the coding coordinate system;

[0040] The coordinates of all plane points in each projected plane are subtracted from the coordinates of the origin of the coding coordinate system and then divided by the coding area interval to obtain multiple groups of coding coordinate values.

[0041] In some embodiments, reconstructing a three-dimensional color model of a gear based on the transformation matrix, the second image, and the second point cloud data includes:

[0042] Projecting the second point cloud data onto the second image according to the conversion matrix, and recording RGB information of pixel values ​​on the second image corresponding to each second point cloud data;

[0043] Mapping the RGB information to second point cloud data having a three-dimensional space through a conversion matrix to obtain color point cloud data;

[0044] Based on the color point cloud data, a three-dimensional color model of the gear is reconstructed.

[0045] In a second aspect, an embodiment of the present application further provides a gear 3D color reconstruction system based on image and point cloud information fusion, the system comprising:

[0046] a data acquisition unit, configured to acquire first images and first point cloud data of the calibration block at different heights and positions, to obtain a plurality of first images and a plurality of sets of first point cloud data, and to acquire a second image and a second point cloud data of the gear, wherein the first image and the first point cloud data of the calibration block at the same height and position are acquired simultaneously;

[0047] a coarse identification unit, configured to coarsely identify first coarse two-dimensional corner point coordinates corresponding to each of the plurality of first images;

[0048] a first data determining unit, configured to determine a plurality of sets of first target two-dimensional corner point coordinates based on the first rough two-dimensional corner point coordinates corresponding to each of the first images;

[0049] a second data determining unit, configured to determine a plurality of sets of target three-dimensional corner point coordinates based on the plurality of sets of first point cloud data;

[0050] a conversion matrix calculation unit, configured to calculate a multi-dimensional data conversion relationship between the plurality of sets of first target two-dimensional corner point coordinates and the plurality of sets of target three-dimensional corner point coordinates to obtain a conversion matrix;

[0051] A color model reconstruction unit is used to reconstruct a three-dimensional color model of the gear based on the transformation matrix, the second image and the second point cloud data.

[0052] In a third aspect, an embodiment of the present application also provides an electronic device comprising at least one control processor and a memory for communicating with the at least one control processor; the memory stores instructions that can be executed by the at least one control processor, and the instructions are executed by the at least one control processor so that the at least one control processor can execute the above-mentioned method for three-dimensional color reconstruction of gears based on the fusion of image and point cloud information.

[0053] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the above-mentioned method for three-dimensional color reconstruction of gears based on the fusion of image and point cloud information.

[0054] It can be understood that the beneficial effects of the above-mentioned second to fourth aspects compared with the relevant technologies are the same as the beneficial effects of the above-mentioned first aspect compared with the relevant technologies. Please refer to the relevant description in the above-mentioned first aspect and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0056] Figure 1 This is a flow chart of an embodiment of a method for three-dimensional color reconstruction of a gear based on image and point cloud information fusion provided by the present application;

[0057] Figure 2 This is a schematic diagram of collected pictures and point cloud data in the best embodiment of the gear three-dimensional color reconstruction method based on image and point cloud information fusion provided by this application;

[0058] Figure 3 This is a schematic diagram of the coarsely identified corner point results in the best embodiment of the gear 3D color reconstruction method based on image and point cloud information fusion provided by the present application;

[0059] Figure 4 This is a schematic diagram of two-dimensional precise corner points in the best embodiment of the gear three-dimensional color reconstruction method based on image and point cloud information fusion provided by this application;

[0060] Figure 5 This is a schematic diagram of converting a point cloud into a two-dimensional plane in the best embodiment of the gear three-dimensional color reconstruction method based on image and point cloud information fusion provided by this application;

[0061] Figure 6 This is a schematic diagram of four precise corner points in a coded coordinate system in a preferred embodiment of a gear three-dimensional color reconstruction method based on image and point cloud information fusion provided by this application;

[0062] Figure 7 This is a schematic diagram of the coordinate values ​​of three-dimensional corner points in the best embodiment of the gear three-dimensional color reconstruction method based on image and point cloud information fusion provided by this application;

[0063] Figure 8 This is a three-dimensional color schematic diagram of a gear in a preferred embodiment of the method for three-dimensional color reconstruction of a gear based on image and point cloud information fusion provided by the present application;

[0064] Figure 9 Schematic diagrams of various types of aviation gears in a preferred embodiment of the method for three-dimensional color reconstruction of gears based on image and point cloud information fusion provided by this application;

[0065] Figure 10 1 is a schematic structural diagram of an embodiment of a gear 3D color reconstruction system based on image and point cloud information fusion provided by the present application;

[0066] Figure 11 It is a structural diagram of an embodiment of the electronic device provided by this application. DETAILED DESCRIPTION

[0067] The following describes in detail embodiments of the present application. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application and are not to be construed as limiting the present application.

[0068] In the description of this application, if there is a description of first, second, etc., it is only for the purpose of distinguishing technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0069] In the description of this application, it should be understood that descriptions involving orientation, such as the orientation or positional relationship indicated by up, down, etc., are based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing this application and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on this application.

[0070] In the description of this application, it should be noted that, unless otherwise clearly defined, terms such as setting, installing, and connecting should be understood in a broad sense, and technical personnel in the relevant technical field can reasonably determine the specific meaning of the above terms in this application based on the specific content of the technical solution.

[0071] The design and manufacturing of aviation gears have made great progress in the past few decades. However, aviation gear inspection is still mainly based on coordinate measuring machine point sampling and manual visual inspection. The level of automation and intelligence is relatively low, which is no longer suitable for the current development needs of high-speed intelligent digitalization in the aviation industry.

[0072] Aviation gear inspections are complex and diverse, encompassing tooth profile, tooth orientation, surface defects, and contact marks. Surface defect and contact mark measurement primarily relies on manual visual inspection. Specifically, quality inspectors examine the inner and outer surfaces of the gears for surface defects such as scratches, dents, and rust, then record the inspection results. Contact marks are manually applied and contact mark parameters are measured and recorded using a caliper. These manual methods not only suffer from low efficiency, poor data storage and maintainability, and significant subjective errors, but also lack the ability to quantitatively and accurately measure defect size, type, and contact mark evaluation parameters, failing to meet the requirements of aircraft engines for improved performance and quality consistency in aviation gears.

[0073] In order to solve the problem that the above-mentioned manual method has low detection accuracy and cannot meet the requirements of aircraft engines for improving the performance and quality consistency of aircraft gears, this application proposes a three-dimensional color reconstruction method and system for gears based on image and point cloud information fusion.

[0074] Reference Figure 1 The embodiment of the present application provides a method for three-dimensional color reconstruction of gears based on image and point cloud information fusion, the method comprising the following steps:

[0075] Step S100: collecting first images and first point cloud data of calibration blocks at different heights and positions to obtain multiple first images and multiple sets of first point cloud data, and collecting second images and second point cloud data of the gear, wherein the first images and first point cloud data of the calibration blocks at the same height and position are collected simultaneously;

[0076] Step S200: based on the plurality of first images, roughly identifying first rough two-dimensional corner point coordinates corresponding to each first image;

[0077] Step S300: determining multiple sets of first target two-dimensional corner point coordinates based on the first rough two-dimensional corner point coordinates corresponding to each first image;

[0078] Step S400: determining multiple sets of target three-dimensional corner point coordinates based on multiple sets of first point cloud data;

[0079] Step S500: Calculate a multi-dimensional data conversion relationship between multiple sets of first target two-dimensional corner point coordinates and multiple sets of target three-dimensional corner point coordinates to obtain a conversion matrix;

[0080] Step S600: reconstructing a three-dimensional color model of the gear based on the transformation matrix, the second image and the second point cloud data.

[0081] In this embodiment, by collecting first images and first point cloud data of calibration blocks at different heights and different positions, multiple first images and multiple groups of first point cloud data are obtained, and second images and second point cloud data of gears are collected, wherein the first images and first point cloud data of the calibration blocks at the same height and the same position are collected simultaneously; based on the multiple first images, the first rough two-dimensional corner point coordinates corresponding to each first image are roughly identified; based on the first rough two-dimensional corner point coordinates corresponding to each first image, multiple groups of first target two-dimensional corner point coordinates are determined; based on the multiple groups of first point cloud data, multiple groups of target three-dimensional corner point coordinates are determined; the multi-dimensional data conversion relationship between the multiple groups of first target two-dimensional corner point coordinates and the multiple groups of target three-dimensional corner point coordinates is calculated to obtain a conversion matrix; based on the conversion matrix, the second image and the second point cloud data, the three-dimensional color model of the gear is reconstructed. In this way, by using the first rough two-dimensional corner point coordinates to further determine the more accurate first target two-dimensional corner point coordinates, and then using the same method to determine the target three-dimensional corner point coordinates, and then according to the accurate first target two-dimensional corner point coordinates and the target three-dimensional corner point coordinates, calculate the accurate transformation matrix, and finally reconstruct the gear three-dimensional color model based on the accurate transformation matrix, high-precision gear three-dimensional color reconstruction can be achieved, so as to improve the detection efficiency of gears, improve the accuracy of gear detection, and lay an important foundation for multi-dimensional detection of gears.

[0082] The above-mentioned collection of the first image and the first point cloud data of the calibration block at different heights and different positions can be to adjust the camera and the line laser scanner to a suitable position so that the center of the field of view of the camera and the line laser scanner is aligned with the calibration block, and the camera is allowed to be as close to the calibration block as possible outside the minimum working distance limit and ensure that the camera and the line laser scanner have a high field of view overlap rate, place the calibration block (i.e., the calibration block) at different heights and different positions, collect the first image of the calibration block by the camera, and collect the first point cloud data of the calibration block by the line laser scanner.

[0083] It should be noted that, in this embodiment, the first image and the first point cloud data may also be collected using equipment well known to those skilled in the art, and this embodiment does not impose any specific limitation.

[0084] The above-mentioned acquisition of the second image and second point cloud data of the gear can be achieved by adjusting the camera and the line laser scanner to appropriate positions so that the centers of the fields of view of the camera and the line laser scanner are aligned with the gear, and allowing the camera to be as close to the gear as possible outside the minimum working distance limit and ensuring that the camera and the line laser scanner have a high field of view overlap rate. The second image of the gear is acquired by the camera, and the second point cloud data of the gear is acquired by the line laser scanner at the same time.

[0085] The above-described method of roughly identifying the first coarse two-dimensional corner coordinates corresponding to each first image based on the multiple first images may employ an edge detection method to extract the two-dimensional edge pixels of the first images, and then roughly calculate the coarse two-dimensional corner coordinates corresponding to each first image based on the two-dimensional edge pixels. The edge detection method may employ a method well known to those skilled in the art, such as the Canny edge detection method, and is not specifically limited in this embodiment.

[0086] The above-mentioned determination of multiple sets of first target two-dimensional corner point coordinates based on the first coarse two-dimensional corner point coordinates corresponding to each first image can be performed by obtaining a set of pixel value points between two adjacent corner point coordinates from the four first coarse two-dimensional corner point coordinates to obtain four point sets, and then iteratively calculating using a straight line fitting method based on these four point sets so that the fitted straight lines are infinitely close to the four straight lines at the edges of the calibration block. After the iteration is completed, four straight lines are obtained, and then the precise two-dimensional corner point coordinates are calculated based on the four straight lines, namely, the first target two-dimensional corner point coordinates. The four first target two-dimensional corner point coordinates corresponding to each first image are taken as a group to obtain multiple sets of first target two-dimensional corner point coordinates.

[0087] The above-mentioned multiple sets of target three-dimensional corner point coordinates are determined based on multiple sets of first point cloud data. Since one set of first point cloud data can be used to calculate one set of target three-dimensional corner point coordinates, multiple sets of first point cloud data can be used to obtain multiple sets of target three-dimensional corner point coordinates.

[0088] The above-described reconstruction of the three-dimensional color model of the gear based on the transformation matrix, the second image, and the second point cloud data can involve obtaining color information from the second image based on the transformation matrix, then assigning the color information to the second point cloud data using the transformation matrix, and then reconstructing the three-dimensional color model of the gear based on the colored point cloud data. It should be noted that the reconstruction of the three-dimensional color model of the gear based on the point cloud data can be performed using methods known to those skilled in the art for three-dimensional reconstruction based on point clouds, and this embodiment does not specifically limit or describe this.

[0089] In some embodiments, based on the plurality of first images, roughly identifying first rough two-dimensional corner point coordinates corresponding to each first image includes:

[0090] Performing filtering and edge detection on the calibration blocks in the plurality of first images to obtain two-dimensional edge pixel points of the calibration block in each first image;

[0091] Downsampling the edge two-dimensional pixel points of the calibration block in each first image to obtain multiple groups of edge pixel points of the sampled calibration blocks, wherein a group of edge pixel points of the sampled calibration blocks corresponds to one first image;

[0092] Sorting the edge pixel points of each group of sampled calibration blocks in a head-to-tail manner to obtain multiple groups of sorted edge pixel points of the calibration blocks;

[0093] Randomly selecting four sorted edge pixel points from each group of sorted edge pixel points of the calibration block, calculating the areas of the four sorted edge pixel points, and obtaining a plurality of first areas;

[0094] The four edge two-dimensional pixel points with the largest areas among the multiple first areas are determined as rough two-dimensional corner point coordinates to obtain first rough two-dimensional corner point coordinates corresponding to each first image.

[0095] In this embodiment, by determining the four edge two-dimensional pixel points with the largest area among multiple first areas as rough two-dimensional corner point coordinates, the first rough two-dimensional corner point coordinates corresponding to each first image are obtained, which can lay a good data foundation for the subsequent calculation of precise two-dimensional corner point coordinates.

[0096] The above-mentioned sorting of each group of sampled edge pixels in a head-to-tail manner may be sorting of each group of sampled edge pixels in a head-to-tail manner according to the coordinates corresponding to each edge pixel.

[0097] In some embodiments, determining multiple sets of first target two-dimensional corner point coordinates based on the first rough two-dimensional corner point coordinates corresponding to each first image includes:

[0098] Saving other edge two-dimensional pixel points between two adjacent rough two-dimensional corner point coordinates in the first rough two-dimensional corner point coordinates corresponding to each first image into a set, to obtain four point sets corresponding to each first image;

[0099] Perform iterative straight line fitting on each point set to obtain four straight lines;

[0100] Calculate the intersection coordinates of the four straight lines to obtain the four intersection coordinates corresponding to each first image;

[0101] The four intersection point coordinates are used as a set of target two-dimensional corner point coordinates to obtain multiple sets of first target two-dimensional corner point coordinates.

[0102] In this embodiment, four point sets corresponding to each first image are obtained by saving the remaining 2D edge pixels between two adjacent coarse 2D corner coordinates in each first image into a set. Iterative straight-line fitting is performed on each point set to obtain four straight lines. The intersection coordinates of the four straight lines are calculated to obtain four intersection coordinates corresponding to each first image. The four intersection coordinates are then used as a set of target 2D corner coordinates to obtain multiple sets of first target 2D corner coordinates. This allows for the calculation of precise 2D corner coordinates, laying a solid foundation for the subsequent calculation of an accurate transformation matrix, enabling high-precision 3D color reconstruction of gears.

[0103] In the above, the other edge two-dimensional pixel points between the coordinates of two adjacent coarse two-dimensional corner points in the first coarse two-dimensional corner point coordinates corresponding to each first image are saved into a set. Since the edge pixel points are sorted according to the principle of head-to-tail connection, the pixel coordinates of all edge pixel points constitute a list, and each edge pixel point corresponds to a coordinate index. For example, the starting coordinate index of the list is 0, and the end coordinate index is the number of edge pixel points minus 1. In this way, the coordinate indexes corresponding to the four first coarse two-dimensional corner point coordinates can be obtained, and then the other edge two-dimensional pixel points between the two adjacent coarse two-dimensional corner point coordinates are obtained according to the coordinate indexes. Finally, the other edge two-dimensional pixel points between the two adjacent coarse two-dimensional corner point coordinates are saved into a set.

[0104] The above-mentioned iterative linear fitting of each point set to obtain four straight lines can be performed by iterative linear fitting of each point set to obtain four straight lines. It should be noted that the linear least squares method is a prior art and will not be described in detail in this embodiment.

[0105] In some embodiments, determining multiple sets of target three-dimensional corner point coordinates based on multiple sets of first point cloud data includes:

[0106] Performing plane fitting on each set of first point cloud data to obtain a fitting plane corresponding to each set of first point cloud data, where the first point cloud data is point cloud data of the top plane of the calibration block acquired by the line laser scanner;

[0107] Calculating the normal vector of the fitting plane corresponding to each set of first point cloud data to obtain multiple normal vectors;

[0108] Calculate the angle between each normal vector and the Z axis in the line laser scanner coordinate system to obtain multiple angles;

[0109] projecting the point cloud data on the fitting plane corresponding to each set of first point cloud data onto a plane according to the normal vector and the angle corresponding to each set of first point cloud data, to obtain a plurality of projected planes;

[0110] Encode each point on the projected plane to obtain multiple sets of encoded coordinate values;

[0111] Based on each set of encoded coordinate values, roughly identifying the second coarse two-dimensional corner point coordinates corresponding to each projected plane;

[0112] Determine multiple sets of second target two-dimensional corner point coordinates based on the second rough two-dimensional corner point coordinates corresponding to each projected plane;

[0113] The multiple sets of second target two-dimensional corner point coordinates are inversely transformed to obtain multiple sets of target three-dimensional corner point coordinates.

[0114] In this embodiment, a fitting plane corresponding to each group of first point cloud data is obtained by performing plane fitting on each group of first point cloud data, where the first point cloud data is the point cloud data of the top plane of the calibration block collected by the line laser scanner; the normal vector of the fitting plane corresponding to each group of first point cloud data is calculated to obtain multiple normal vectors; the angle between each normal vector and the Z axis in the line laser scanner coordinate system is calculated to obtain multiple angles; based on the normal vector and angle corresponding to each group of first point cloud data, the point cloud data on the fitting plane corresponding to each group of first point cloud data is projected onto a plane to obtain multiple projected planes; each group of projected plane points is encoded to obtain multiple groups of encoded coordinate values; based on each group of encoded coordinate values, the second coarse two-dimensional corner point coordinates corresponding to each fitting plane are roughly identified; based on the second coarse two-dimensional corner point coordinates corresponding to each fitting plane, multiple groups of second target two-dimensional corner point coordinates are determined; the multiple groups of second target two-dimensional corner point coordinates are inversely transformed to obtain multiple groups of target three-dimensional corner point coordinates. In this way, by projecting the three-dimensional point cloud data into two-dimensional plane points, and then determining the rough two-dimensional corner point coordinates and the precise two-dimensional corner point coordinates (i.e., the target two-dimensional corner point coordinates), and then inversely transforming multiple sets of second target two-dimensional corner point coordinates, multiple sets of precise target three-dimensional corner point coordinates are obtained, laying a good data foundation for the subsequent calculation of the precise transformation matrix, so as to achieve high-precision three-dimensional color reconstruction of gears.

[0115] The above-mentioned plane fitting is performed on each group of first point cloud data to obtain the fitting plane corresponding to each group of first point cloud data. This can be achieved by calculating the distance from all point cloud data on the calibration block to the fitting plane after each plane fitting, removing a certain ratio (for example, 5% or 10%) of the point cloud data, for example, removing some point cloud data farthest from the fitting plane. After several rounds of iteration, a fitting plane that meets the requirements can be obtained, and the fitting plane is infinitely close to the real plane of the calibration block.

[0116] The above calculation of the normal vector of the fitting plane corresponding to each set of first point cloud data may be performed by using a linear least squares method. The linear least squares method is a prior art and will not be described in detail in this embodiment.

[0117] The above calculation of the angle between the normal vector and the Z axis in the line laser scanner coordinate system can be performed by using a method known to those skilled in the art, which is not described in detail in this embodiment.

[0118] The above-mentioned method roughly identifies the second coarse two-dimensional corner coordinates corresponding to each projected plane based on each group of encoded coordinate values. The method can be to filter and detect edges of each projected plane to obtain the edge two-dimensional pixel points of each projected plane; downsample the edge two-dimensional pixel points of each projected plane to obtain multiple groups of sampled edge pixel points; sort each group of sampled edge pixel points in a head-to-tail manner to obtain multiple groups of sorted edge pixel points; calculate the areas of four sorted edge pixel points randomly selected from each group of sorted edge pixel points each time to obtain multiple second areas; determine the four edge two-dimensional pixel points with the largest areas in the multiple second areas as coarse two-dimensional corner coordinates to obtain the second coarse two-dimensional corner coordinates corresponding to each projected plane.

[0119] The above method of determining multiple sets of second target two-dimensional corner point coordinates based on the second rough two-dimensional corner point coordinates corresponding to each projected plane can be to save other edge two-dimensional pixel points between two adjacent rough two-dimensional corner point coordinates in the second rough two-dimensional corner point coordinates corresponding to each projected plane into a set to obtain four point sets corresponding to each projected plane; perform iterative straight line fitting on each point set to obtain four straight lines; calculate the intersection coordinates of the four straight lines to obtain four intersection coordinates corresponding to each projected plane; and use the four intersection coordinates as a set of target two-dimensional corner point coordinates to obtain multiple sets of second target two-dimensional corner point coordinates.

[0120] The above-mentioned inverse transformation of multiple sets of second target two-dimensional corner point coordinates to obtain multiple sets of target three-dimensional corner point coordinates can be performed by first converting the encoded coordinate values ​​of the four second target two-dimensional corner point coordinates back to the plane point coordinate values ​​after projection, and then adding the normal vector Z coordinate value of the fitted plane during the projection process, and then converting the calculated normal vector and the angle between the Z axis in the line laser scanner coordinate system back to the three-dimensional corner point coordinates at the original calibration block position (i.e., the target three-dimensional corner point coordinates).

[0121] In some embodiments, based on the normal vector and angle corresponding to each set of first point cloud data, the point cloud data on the fitting plane corresponding to each set of first point cloud data is projected onto a plane to obtain a plurality of projected planes, including:

[0122] Make the normal vector corresponding to each set of first point cloud data parallel to the Z axis in the line laser scanner coordinate system, and make the fitting plane corresponding to each set of first point cloud data parallel to the XOY plane in the line laser scanner coordinate system;

[0123] After the fitting plane corresponding to each set of first point cloud data is parallel to the XOY plane in the line laser scanner coordinate system, the point cloud data on the fitting plane corresponding to each set of first point cloud data is projected onto the XOY plane to obtain multiple projected planes.

[0124] In this embodiment, the normal vector corresponding to each set of first point cloud data is aligned parallel to the Z axis of the line laser scanner coordinate system, and the fitting plane corresponding to each set of first point cloud data is aligned parallel to the XOY plane of the line laser scanner coordinate system. After the fitting plane corresponding to each set of first point cloud data is aligned parallel to the XOY plane of the line laser scanner coordinate system, the point cloud data on the fitting plane corresponding to each set of first point cloud data is projected onto the XOY plane to obtain multiple projected planes. In this way, the precise coordinates of the target two-dimensional corner points can be calculated from these multiple sets of projected plane points, laying a good data foundation for calculating the precise three-dimensional corner point coordinates.

[0125] In some embodiments, encoding each point on the projected plane to obtain multiple sets of encoded coordinate values ​​includes:

[0126] Preset coding area interval;

[0127] The coordinates of the plane point with the smallest X and Y values ​​in each projected plane are taken as the coordinates of the origin of the encoding coordinate system;

[0128] The coordinates of all plane points in each projected plane are subtracted from the coordinates of the origin of the coding coordinate system and then divided by the coding area interval to obtain multiple sets of coding coordinate values.

[0129] In this embodiment, by presetting the coding region interval, the coordinates of the plane point with the smallest X and Y values ​​in each projected plane are used as the coordinates of the origin of the coding coordinate system. The coordinates of all plane points in each projected plane are subtracted from the coordinates of the origin of the coding coordinate system and then divided by the coding region interval to obtain multiple sets of coding coordinate values. In this way, calculating multiple sets of coding coordinate values ​​can lay a good data foundation for calculating the accurate coordinates of three-dimensional corner points.

[0130] In some embodiments, reconstructing a three-dimensional color model of the gear based on the transformation matrix, the second image, and the second point cloud data includes:

[0131] Projecting the second point cloud data onto the second image according to the conversion matrix, and recording the RGB information of the pixel value on the second image corresponding to each second point cloud data;

[0132] Mapping the RGB information to the second point cloud data in three-dimensional space through a conversion matrix to obtain color point cloud data;

[0133] Reconstruct the gear 3D color model based on the color point cloud data.

[0134] In this embodiment, the second point cloud data is projected onto the second image according to a transformation matrix, and the RGB information corresponding to the pixel value on the second image for each second point cloud data item is recorded. This RGB information is then mapped onto the second point cloud data in a three-dimensional space using the transformation matrix to obtain color point cloud data. Based on this color point cloud data, a three-dimensional color model of the gear is reconstructed. In this way, by accurately transforming the RGB information of the gear into point cloud data of the gear tooth surface using a precise transformation matrix, high-precision three-dimensional color reconstruction of the gear can be achieved.

[0135] To facilitate understanding by those skilled in the art, a set of best embodiments is provided below:

[0136] With the rapid development of China's aviation industry, domestic demand for aircraft is increasing. Whether for civil, commercial, or military aircraft, the technology level and production capacity are constantly improving. The core system of an aircraft is the aircraft engine, and the most critical component of this system is the aircraft gears in the transmission system. While the design and manufacturing of aircraft gears have made significant progress over the past few decades, aircraft gear inspection still relies primarily on coordinate measuring machine point sampling and manual visual inspection, with a low level of automation and intelligence, which is no longer suitable for the current rapid development of intelligent and digital aviation in the aviation industry.

[0137] Aviation gear inspections are complex and diverse, encompassing tooth profile, tooth orientation, surface defects, and contact marks. Surface defect and contact mark measurement primarily relies on manual visual inspection. Specifically, quality inspectors examine the inner and outer surfaces of the gears for surface defects such as scratches, dents, and rust, then record the inspection results. Contact marks are manually applied and contact mark parameters are measured and recorded using a caliper. These manual methods not only suffer from low efficiency, poor data storage and maintainability, and significant subjective errors, but also lack the ability to quantitatively and accurately measure defect size, type, and contact mark evaluation parameters, failing to meet the requirements of aircraft engines for improved performance and quality consistency in aviation gears.

[0138] The inability to accurately and quantitatively and qualitatively detect defects and measure contact marks using deep learning and visual measurement techniques is primarily due to the multidimensional nature of these inspection objectives for aviation gears, requiring not only image color and texture information but also coordinate space information from point clouds. Therefore, developing a data fusion method for image and point cloud information—reconstructing three-dimensional color images of aviation gears based on these information—has become a key research topic in efficient, high-precision, and intelligent aviation gear inspection. This approach holds far-reaching engineering value for promoting the integrated design, manufacturing, and inspection of modern aerospace engines and transmission systems, while ensuring the quality, safety, and service life of aerospace powertrain products.

[0139] To address the limitations of current manual defect detection and contact mark measurement for aviation gears, this embodiment proposes a high-precision multi-dimensional data fusion method. Based on the collected images and point cloud information of aviation gears, a low-cost calibration block is combined with a high-precision calibration block corner point extraction method to solve the conversion relationship between multi-dimensional data, and the image color and texture information of the aviation gear is converted to the gear tooth surface point cloud, thereby achieving high-precision three-dimensional color reconstruction of aviation gears.

[0140] The technical solution of this embodiment fills the technical gap in the field of multi-dimensional surface defect detection and contact mark measurement of aviation gears in production practice, and provides important technical support for realizing multi-dimensional information fusion measurement technologies such as high-precision quantitative surface defect detection and contact mark measurement of aviation gears.

[0141] This embodiment proposes a method for 3D color reconstruction of aviation gears based on the fusion of image and point cloud information. It should be noted that in addition to the aviation gears of this embodiment, other gear detection can also adopt the technical solution of this embodiment, and this embodiment does not impose specific restrictions. The method of this embodiment mainly includes the following parts: high-precision and rapid identification of 2D corner points based on a calibration block with two-stage coarse-fine recognition; 3D corner point identification based on a high-density calibration block with projection coding and rectangular features; solving multi-dimensional data conversion relationships; and 3D color reconstruction of aviation gears. The specific contents are:

[0142] 1. First, adjust the camera and line laser scanner to the appropriate position, align the center of the camera and line laser scanner's field of view with the calibration block, let the camera be as close to the calibration block as possible outside the minimum working distance limit, and ensure that the camera and line laser have a high field of view overlap rate. Place the calibration block (i.e., calibration block) at different heights and positions to collect multiple sets of calibration data (including images and point cloud data). Generally, 5 to 8 sets of pictures and point cloud data are required, and at least two sets of pictures and point cloud data are required. The specific number of sets of pictures and point cloud data required can be changed according to actual conditions, and this embodiment does not impose specific restrictions. The schematic diagram of collecting pictures and point cloud data of the calibration block is as follows: Figure 2 shown.

[0143] The captured image is quickly and accurately identified using a calibration block based on a two-stage coarse-fine recognition process. The captured calibration block image is preprocessed, such as filtering and edge detection, to obtain the 2D edge pixels of the calibration block image. These 2D edge pixels are then downsampled.

[0144] Sort the downsampled edge pixels so that they are connected end to end. Then, select four edge pixels from these edge pixels as a set and calculate the area of ​​the quadrilateral formed by connecting all the sets end to end. The four edge pixels with the largest area are the corner points of the first stage of coarse recognition (i.e., the first coarse two-dimensional corner coordinates), as shown in the figure. Figure 3 shown.

[0145] The four coarsely identified corner points are placed in the unsampled edge pixels (i.e., the obtained two-dimensional edge pixels of the calibration block image) to obtain the pixel coordinate indexes corresponding to the four corner points; and based on the coordinate indexes, the other pixels between two adjacent points are saved in a set, thereby obtaining four point sets containing adjacent coarse corner points. The pixel coordinate indexes corresponding to the four corner points can be obtained as follows:

[0146] Since edge pixels are sorted end-to-end, the pixel coordinates of all points form a list, with each point corresponding to an index (i.e., the starting coordinate index of the list is 0, and the ending coordinate index is the number of edge pixels - 1). In Python, using known functions, we can get the coordinate index of the target point from the list, thus obtaining the pixel coordinate index corresponding to the four corner points.

[0147] The edge pixels in the above four point sets are subjected to iterative straight line fitting respectively. After each fitting, the distance from each edge pixel in the point set to the straight line is calculated, and a certain ratio (such as 5% or 10%) of the points farthest from the straight line are removed. After several rounds of iteration, four straight lines that are infinitely close to the edge of the calibration block can be obtained.

[0148] The intersection coordinates of the four straight lines obtained by the above processing are calculated by the method of finding the intersection of two straight line equations, which are the 2D precise corner points of the identified calibration block (i.e. the coordinates of the first target two-dimensional corner points), as shown in Figure 4 shown.

[0149] 2. For 3D corner point recognition of the high-density calibration block based on projection coding and rectangular features, a plane is fitted to the top of the 3D calibration block point cloud acquired using a line laser scanner, and the normal vector of the plane is calculated. The angle θ between the normal vector and the Z axis in the line laser coordinate system is then solved. Specifically:

[0150] When fitting the calibration block plane, an iterative plane fitting method is used. After each plane fitting, the distances from all points on the calibration block point cloud to the plane are calculated, and a certain ratio (such as 5% or 10%) of the points farthest from the fitting plane are removed. After several rounds of iteration, a fitting plane that is infinitely close to the true plane of the calibration block can be obtained.

[0151] Use the angle calculated above to rotate the fitting plane to be parallel to the XOY plane of the line laser scanner coordinate system. In other words, make the normal vector of the fitting plane parallel to the Z axis of the line laser scanner coordinate system. Then, project the point cloud on the rotated fitting plane of the calibration block onto the XOY plane of the line laser scanner coordinate system, setting the Z coordinate of these point clouds to zero.

[0152] Encode the point cloud projected onto the XOY plane (i.e., the points on the projected plane) to obtain the coordinate values ​​of the coded coordinate system (i.e., the coded coordinate values). The specific operations are as follows:

[0153] The coding range is set according to the calibration block range scanned by the line laser scanner, and the interval of the coding area is set according to the accuracy of the line laser scanner. For example, assuming that the maximum interval of the point cloud scanned by the line laser is 10um, the coding area interval can be set to 10um, that is, within a rectangular range, the rectangular range is divided into grid areas according to the coding area interval, where the interval between adjacent small cells in the grid is 10um.

[0154] The points projected on the XOY plane are then divided into corresponding grids, and the coordinate values ​​of the projected coded coordinate system are calculated. The specific operation is to define the smallest X, Y value among all the points projected on the XOY plane as the origin of the coded coordinate system, and then subtract the origin from the coordinates of all the points projected on the XOY plane and divide them by the coding area interval to obtain the coordinate value of the coded coordinate system corresponding to each point (i.e., the coded coordinate value).

[0155] Then, through the above encoding conversion process, the point cloud can be converted to a two-dimensional plane, such as Figure 5 shown.

[0156] The encoding coordinate system can be regarded as a pixel coordinate system. The coordinate points in the encoding coordinate system are preprocessed. The same method as the above-mentioned "2D corner point high-precision and fast recognition based on the calibration block of two-stage coarse-fine recognition" is used to calculate the four 2D corner points on the XOY plane, as shown in the following figure: Figure 6 As shown, the specific process is:

[0157] right Figure 5 The image preprocessing is performed on the two-dimensional plane in the encoding coordinate system, such as filtering and edge detection, to obtain the edge two-dimensional pixel points of the two-dimensional plane, and then the obtained edge two-dimensional pixel points of the two-dimensional plane are downsampled.

[0158] The downsampled edge pixels are sorted so that they are connected end to end. Then, four edge pixels are randomly selected from these edge pixels as a set, and the area of ​​the quadrilateral formed by the end-to-end connection of all sets is calculated. The four edge pixels with the largest area are the corner point results of the first stage coarse recognition (i.e., the second coarse two-dimensional corner point coordinates).

[0159] The four roughly identified corner points are placed in the unsampled edge pixel points (i.e., the edge two-dimensional pixel points of the obtained two-dimensional plane) to obtain the pixel point coordinate indexes corresponding to the four corner points; and based on the coordinate index, the other pixel points between two adjacent points are saved in a set, thereby obtaining four point sets containing adjacent coarse corner points.

[0160] The edge pixels in the above four point sets are subjected to iterative straight line fitting respectively. After each fitting, the distance from each edge pixel in the point set to the straight line is calculated, and a certain ratio (such as 5% or 10%) of the points farthest from the straight line are removed. After several rounds of iteration, four straight lines that are infinitely close to the edge of the calibration block can be obtained.

[0161] The intersection coordinates of the four straight lines obtained by the above processing are calculated by the method of finding the intersection point of two straight line equations, which are the 2D precise corner points of the identified calibration block (ie the coordinates of the second target two-dimensional corner points).

[0162] Finally, the four 2D corner points in the coded coordinate system obtained through the above operation are inversely transformed to obtain the corresponding four 3D corner point coordinate values. The specific operation is: first convert the coded coordinate values ​​of the four 2D corner points back to the projection coordinate system value, and then add the Z coordinate value of the normal vector of the fitting plane during the projection process, as well as the Z-axis angle between the normal vector calculated at the beginning and the line laser coordinate system to convert back to the 3D corner point coordinate value (i.e., the target three-dimensional corner point coordinate) at the original calibration block position, as shown in the figure. Figure 7 shown.

[0163] 3. To solve the multi-dimensional data conversion relationship, the 2D corner points and 3D corner points of the calibration block obtained in the above steps are matched. The principle is to make the positions of their corner points correspond one-to-one in real space, and then solve the multi-dimensional data conversion relationship.

[0164] Specifically, corner point matching for 2D and 3D corner points is performed by calculating the center coordinates of each of the four identified corner points. The four identified corner points in 2D and 3D space are numbered 1, 2, 3, 4, and A, B, C, D in clockwise or counterclockwise order, respectively. The corresponding field of view between the camera and the line laser is determined based on the position of the sensor. For example, number 1 in 2D space corresponds to A in 3D space, number 2 in 2D space corresponds to B in 3D space, number 3 in 2D space corresponds to C in 3D space, and number 4 in 2D space corresponds to D in 3D space.

[0165] By solving multiple sets of calibration block images and point cloud data, multiple sets of 2D-3D corner point pairs can be obtained, that is, the multi-dimensional data conversion relationship can be solved (that is, the conversion matrix is ​​obtained), that is, the conversion relationship from the pixel coordinate system to the line laser coordinate system.

[0166] 4. Perform three-dimensional color reconstruction of the aviation gear using the obtained conversion relationship (i.e., conversion matrix). The specific operation is: collect the second image and second point cloud data of the aviation gear, project the point cloud (i.e., second point cloud data) onto the two-dimensional image (i.e., second image) plane through the conversion matrix, the point cloud corresponds to a certain pixel value and records the RGB information of the pixel value, and then restore the RGB information to the 3D space to obtain the three-dimensional color reconstruction model of the aviation gear, such as Figure 8 shown.

[0167] The three-dimensional color reconstruction of the aviation gear using the obtained conversion relationship (i.e., the conversion matrix) may further include the following steps:

[0168] Using the conversion matrix between the 2D corner point coordinates and the 3D corner point coordinates, the 2D image data is converted to 3D space, and the corresponding relationship between the single pixel in the 2D image data and the point cloud data in 3D space is obtained;

[0169] If a single pixel in the two-dimensional image data corresponds to multiple point clouds in the point cloud data, retain the point cloud with the largest point in the Z direction among the multiple point clouds corresponding to the single pixel to obtain the color point cloud data;

[0170] Reconstruct the gear 3D color model based on the color point cloud data.

[0171] It should be noted that when mapping two-dimensional image data to three-dimensional space, a single pixel may correspond to multiple point cloud data. At this time, in order to ensure detection accuracy, only the point cloud with the largest size in the Z direction will be retained. That is, from a bird's-eye view, the color image acquisition device usually captures the color of the outermost surface of the gear. Therefore, using the outermost point cloud data can make the conversion correspondence more accurate.

[0172] Performing three-dimensional color reconstruction of the aviation gear using the obtained conversion relationship (i.e., conversion matrix) may further include the following steps:

[0173] Using the conversion matrix between the 2D corner point coordinates and the 3D corner point coordinates, the 2D image data is converted to 3D space, and the corresponding relationship between the single pixel in the 2D image data and the point cloud data in 3D space is obtained;

[0174] If a single pixel in the two-dimensional image data corresponds to multiple point clouds in the point cloud data, retain the point cloud with the largest Z direction among the multiple point clouds corresponding to the single pixel, and retain the point clouds whose average height exceeds the preset pixel allocation requirement based on the clustering method to obtain color point cloud data;

[0175] Reconstruct the gear 3D color model based on the color point cloud data.

[0176] On the basis of retaining the largest point cloud in the Z direction, we can further consider that due to the high density of the point cloud, some points will be assigned to adjacent pixels, so we can use the clustering method to save the point cloud clusters with a larger average height.

[0177] Compared with the existing technology, the technical solution of this embodiment has the following advantages:

[0178] The three-dimensional color reconstruction method of aviation gears based on the fusion of image and point cloud information solves the multi-dimensional data conversion relationship from image to point cloud through a high-precision rapid recognition method of 2D corner points of calibration blocks based on two-stage coarse-fine recognition and a high-density 3D corner point recognition method of calibration blocks based on projection coding and rectangular features; and realizes the three-dimensional color reconstruction of aviation gears. It provides important technical support for multi-dimensional information fusion measurement technologies such as high-precision quantitative surface defect detection and contact mark measurement of aviation gears. It can be applied to various models of aviation gears for three-dimensional color reconstruction, such as Figure 9 As shown, it can complete complex tasks such as multi-dimensional surface defect detection and imprint measurement, and has universality.

[0179] Reference Figure 10 The present application also provides a gear 3D color reconstruction system based on image and point cloud information fusion. The system includes a data acquisition unit 100, a coarse recognition unit 200, a first data determination unit 300, a second data determination unit 400, a transformation matrix calculation unit 500, and a color model reconstruction unit 600, wherein:

[0180] The data acquisition unit 100 is configured to acquire first images and first point cloud data of a calibration block at different heights and positions, thereby obtaining a plurality of first images and a plurality of sets of first point cloud data, and to acquire a second image and a second point cloud data of the gear, wherein the first image and the first point cloud data of the calibration block at the same height and position are acquired simultaneously;

[0181] A coarse identification unit 200 is configured to coarsely identify first coarse two-dimensional corner point coordinates corresponding to each first image based on the plurality of first images;

[0182] The first data determining unit 300 is configured to determine a plurality of sets of first target two-dimensional corner point coordinates based on the first rough two-dimensional corner point coordinates corresponding to each first image;

[0183] The second data determination unit 400 is configured to determine multiple sets of target three-dimensional corner point coordinates based on multiple sets of first point cloud data;

[0184] The conversion matrix calculation unit 500 is used to calculate the multi-dimensional data conversion relationship between multiple sets of first target two-dimensional corner point coordinates and multiple sets of target three-dimensional corner point coordinates to obtain a conversion matrix;

[0185] The color model reconstruction unit 600 is used to reconstruct a three-dimensional color model of the gear based on the transformation matrix, the second image and the second point cloud data.

[0186] It should be noted that since the gear three-dimensional color reconstruction system based on image and point cloud information fusion in this embodiment and the above-mentioned gear three-dimensional color reconstruction method based on image and point cloud information fusion are based on the same inventive concept, the corresponding contents in the method embodiment are also applicable to the system embodiment and will not be described in detail here.

[0187] Reference Figure 11 , an embodiment of the present application further provides an electronic device, the electronic device comprising:

[0188] at least one memory;

[0189] at least one processor;

[0190] at least one program;

[0191] The program is stored in the memory, and the processor executes at least one program to implement the three-dimensional color reconstruction method of the gear based on the fusion of image and point cloud information as described above.

[0192] The electronic device may be any intelligent terminal including a mobile phone, a tablet computer, a personal digital assistant (PDA), a car computer, etc.

[0193] The electronic device according to the embodiment of the present application is described in detail below.

[0194] The processor 1600 may be implemented as a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is configured to execute relevant programs to implement the technical solutions provided by the embodiments of the present disclosure.

[0195] Memory 1700 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). Memory 1700 can store an operating system and other application programs. When implementing the technical solutions provided in the embodiments of this specification through software or firmware, the relevant program code is stored in memory 1700 and is called by processor 1600 to execute the method for three-dimensional color reconstruction of gears based on image and point cloud information fusion in the embodiments of this disclosure.

[0196] Input / output interface 1800, used for information input and output;

[0197] Communication interface 1900, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WiFi, Bluetooth, etc.);

[0198] bus 2000 , which transmits information between various components of the device (e.g., processor 1600 , memory 1700 , input / output interface 1800 , and communication interface 1900 );

[0199] The processor 1600 , the memory 1700 , the input / output interface 1800 , and the communication interface 1900 are connected to each other in communication within the device via the bus 2000 .

[0200] An embodiment of the present disclosure also provides a storage medium, which is a computer-readable storage medium. The computer-readable storage medium stores computer-executable instructions, which are used to enable a computer to execute the above-mentioned three-dimensional color reconstruction method of gears based on the fusion of image and point cloud information.

[0201] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0202] The embodiments described in the embodiments of the present disclosure are intended to more clearly illustrate the technical solutions of the embodiments of the present disclosure and do not constitute a limitation on the technical solutions provided by the embodiments of the present disclosure. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present disclosure are also applicable to similar technical problems.

[0203] Those skilled in the art will understand that the technical solutions shown in the drawings do not constitute a limitation on the embodiments of the present disclosure, and may include more or fewer steps than shown in the drawings, or a combination of certain steps, or different steps.

[0204] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0205] Those skilled in the art will appreciate that all or some of the steps in the methods, systems, and functional modules / units in the devices disclosed above may be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0206] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0207] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0208] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0209] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0210] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0211] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions for enabling an electronic device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes various media that can store programs, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk. The embodiments of the present application are described in detail above in conjunction with the accompanying drawings, but the present application is not limited to the above embodiments. Various changes can be made within the scope of knowledge possessed by ordinary technicians in the relevant technical field without departing from the purpose of the present application.

[0212] The embodiments of the present application are described in detail above in conjunction with the accompanying drawings, but the present application is not limited to the above embodiments. Various changes can be made within the scope of knowledge possessed by ordinary technicians in the relevant technical field without departing from the purpose of the present application.

Claims

1. A gear 3D color reconstruction method based on image and point cloud information fusion, characterized in that: The method comprises: Acquire first images and first point cloud data of the calibration block at different heights and different positions to obtain multiple first images and multiple sets of first point cloud data, and acquire second images and second point cloud data of the gear, wherein the first images and first point cloud data of the calibration block at the same height and the same position are acquired simultaneously; Based on the plurality of first images, roughly identifying first rough two-dimensional corner point coordinates corresponding to each of the first images includes: Performing filtering and edge detection on the calibration blocks in the plurality of first images to obtain two-dimensional edge pixel points of the calibration block in each of the first images; Downsampling edge two-dimensional pixel points of the calibration block in each of the first images to obtain multiple groups of edge pixel points of the sampled calibration blocks, wherein a group of edge pixel points of the sampled calibration blocks corresponds to one of the first images; Sorting the edge pixel points of each group of the sampled calibration blocks in a head-to-tail manner to obtain multiple groups of edge pixel points of the sorted calibration blocks; Randomly selecting four sorted edge pixel points from each group of edge pixel points of the sorted calibration block, and calculating the areas of the four sorted edge pixel points to obtain a plurality of first areas; Determine the four edge two-dimensional pixel points with the largest areas among the multiple first areas as rough two-dimensional corner point coordinates, and obtain first rough two-dimensional corner point coordinates corresponding to each of the first images; determining a plurality of sets of first target two-dimensional corner point coordinates based on the first rough two-dimensional corner point coordinates corresponding to each of the first images; Determining multiple sets of target three-dimensional corner point coordinates based on the multiple sets of first point cloud data; Calculating a multi-dimensional data conversion relationship between the multiple sets of first target two-dimensional corner point coordinates and the multiple sets of target three-dimensional corner point coordinates to obtain a conversion matrix; A three-dimensional color model of the gear is reconstructed based on the transformation matrix, the second image and the second point cloud data.

2. The gear 3D color reconstruction method based on image and point cloud information fusion according to claim 1 is characterized in that: The determining of multiple sets of first target two-dimensional corner point coordinates based on the first rough two-dimensional corner point coordinates corresponding to each of the first images includes: Saving other edge two-dimensional pixel points between two adjacent rough two-dimensional corner point coordinates in the first rough two-dimensional corner point coordinates corresponding to each of the first images into a set, to obtain four point sets corresponding to each of the first images; Performing iterative straight line fitting on each of the point sets to obtain four straight lines; Calculating the intersection coordinates of the four straight lines to obtain the coordinates of four intersection points corresponding to each of the first images; The four intersection point coordinates are used as a set of target two-dimensional corner point coordinates to obtain multiple sets of first target two-dimensional corner point coordinates.

3. The gear 3D color reconstruction method based on image and point cloud information fusion according to claim 1 is characterized in that: The determining of multiple sets of target three-dimensional corner point coordinates based on the multiple sets of first point cloud data includes: performing plane fitting on each set of the first point cloud data to obtain a fitting plane corresponding to each set of the first point cloud data, wherein the first point cloud data is point cloud data of the top plane of the calibration block acquired by a line laser scanner; Calculating a normal vector of a fitting plane corresponding to each set of the first point cloud data to obtain a plurality of normal vectors; Calculating the angle between each normal vector and the Z axis in the line laser scanner coordinate system to obtain a plurality of angles; projecting the point cloud data on the fitting plane corresponding to each set of the first point cloud data onto a plane according to the normal vector and the angle corresponding to each set of the first point cloud data, to obtain a plurality of projected planes; Encoding each point on the projected plane to obtain multiple sets of encoded coordinate values; Based on each set of the encoded coordinate values, roughly identifying the second rough two-dimensional corner point coordinates corresponding to each of the projected planes; Determining multiple sets of second target two-dimensional corner point coordinates based on the second rough two-dimensional corner point coordinates corresponding to each of the projected planes; The multiple sets of second target two-dimensional corner point coordinates are inversely transformed to obtain multiple sets of target three-dimensional corner point coordinates.

4. The method for 3D color reconstruction of gears based on image and point cloud information fusion according to claim 3, characterized in that: The step of projecting the point cloud data on the fitting plane corresponding to each set of the first point cloud data onto a plane according to the normal vector and the angle corresponding to each set of the first point cloud data to obtain a plurality of projected planes includes: Parallelizing the normal vector corresponding to each set of the first point cloud data to the Z axis in the line laser scanner coordinate system, and parallelizing the fitting plane corresponding to each set of the first point cloud data to the XOY plane in the line laser scanner coordinate system; After the fitting plane corresponding to each group of the first point cloud data is parallel to the XOY plane in the line laser scanner coordinate system, the point cloud data on the fitting plane corresponding to each group of the first point cloud data is projected onto the XOY plane to obtain multiple projected planes.

5. The method for 3D color reconstruction of gears based on image and point cloud information fusion according to claim 3, characterized in that: The encoding of each point on the projected plane to obtain multiple sets of encoded coordinate values ​​includes: Preset coding area interval; Taking the coordinates of the plane point with the smallest X value and Y value in each projected plane as the coordinates of the origin of the coding coordinate system; The coordinates of all plane points in each projected plane are subtracted from the coordinates of the origin of the coding coordinate system and then divided by the coding area interval to obtain multiple groups of coding coordinate values.

6. The gear 3D color reconstruction method based on image and point cloud information fusion according to claim 1 is characterized in that: The step of reconstructing a three-dimensional color model of a gear based on the transformation matrix, the second image, and the second point cloud data includes: Projecting the second point cloud data onto the second image according to the conversion matrix, and recording RGB information of pixel values ​​on the second image corresponding to each second point cloud data; Mapping the RGB information to second point cloud data having a three-dimensional space through a conversion matrix to obtain color point cloud data; Based on the color point cloud data, a three-dimensional color model of the gear is reconstructed.

7. A gear 3D color reconstruction system based on image and point cloud information fusion, characterized in that: The system comprises: a data acquisition unit, configured to acquire first images and first point cloud data of the calibration block at different heights and positions, to obtain a plurality of first images and a plurality of sets of first point cloud data, and to acquire a second image and a second point cloud data of the gear, wherein the first image and the first point cloud data of the calibration block at the same height and position are acquired simultaneously; A coarse identification unit, configured to coarsely identify first coarse two-dimensional corner point coordinates corresponding to each of the plurality of first images based on the plurality of first images, comprising: Performing filtering and edge detection on the calibration blocks in the plurality of first images to obtain two-dimensional edge pixel points of the calibration block in each of the first images; Downsampling edge two-dimensional pixel points of the calibration block in each of the first images to obtain multiple groups of edge pixel points of the sampled calibration blocks, wherein a group of edge pixel points of the sampled calibration blocks corresponds to one of the first images; Sorting the edge pixel points of each group of the sampled calibration blocks in a head-to-tail manner to obtain multiple groups of edge pixel points of the sorted calibration blocks; Randomly selecting four sorted edge pixel points from each group of edge pixel points of the sorted calibration block, and calculating the areas of the four sorted edge pixel points to obtain a plurality of first areas; Determine the four edge two-dimensional pixel points with the largest areas among the multiple first areas as rough two-dimensional corner point coordinates, and obtain first rough two-dimensional corner point coordinates corresponding to each of the first images; a first data determining unit, configured to determine a plurality of sets of first target two-dimensional corner point coordinates based on the first rough two-dimensional corner point coordinates corresponding to each of the first images; a second data determining unit, configured to determine a plurality of sets of target three-dimensional corner point coordinates based on the plurality of sets of first point cloud data; a conversion matrix calculation unit, configured to calculate a multi-dimensional data conversion relationship between the plurality of sets of first target two-dimensional corner point coordinates and the plurality of sets of target three-dimensional corner point coordinates to obtain a conversion matrix; A color model reconstruction unit is used to reconstruct a three-dimensional color model of the gear based on the transformation matrix, the second image and the second point cloud data.

8. An electronic device, characterized in that: It includes at least one control processor and a memory for communicating with the at least one control processor; the memory stores instructions that can be executed by the at least one control processor, and the instructions are executed by the at least one control processor to enable the at least one control processor to execute the three-dimensional color reconstruction method of gears based on image and point cloud information fusion as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the gear three-dimensional color reconstruction method based on image and point cloud information fusion as described in any one of claims 1 to 6.

Images