Method, system, medium and product for detecting rivet quality of aircraft panel

Abstract

The application discloses a kind of detection methods, systems, media and products of aircraft panel rivet quality, the method includes: monocular motion camera is on multiple equidistant positions Acquisition multiple aircraft panel images of the aircraft section wallboard to be measured, each aircraft panel image is carried out distortion correction processing.Acquire each rivet point and each rivet hole in the first coordinate information and corresponding identification parameter in adjacent corrected image coordinate system, based on the objective function for realizing re-projection error minimization constructed by bundle adjustment method, the second coordinate information of each rivet point and each rivet hole is acquired to construct first point cloud set, with the second point cloud set generated by the aircraft panel digitization model pre-constructed Matching analysis, obtain the second abnormal coordinate information of abnormal rivet point and rivet hole after conversion into third coordinate information in three-dimensional projector coordinate system, and are projected by three-dimensional projector and marked.The scheme improves the detection efficiency and accuracy of abnormal rivet point and rivet hole on aircraft panel.

Description

Technical Field

[0001] This invention relates to the field of machine vision technology, and in particular to a method, system, medium, and product for detecting the quality of rivets on aircraft panels. Background Technology

[0002] Riveting is the core connection method for aircraft panels. The pressurization and depressurization cycles during aircraft flight can easily cause stress concentration in the riveted structure. Quality problems such as loose rivets and cracks at the edges of rivet holes directly affect the integrity of the airframe structure and flight safety, making it a key inspection target in aircraft manufacturing and maintenance.

[0003] Quality inspection of rivets on aircraft sections requires identifying issues such as missing rivets, multiple rivets, and incorrect rivet positions. Accurate 3D reconstruction modeling of rivets is key to achieving this inspection. Existing technologies mainly employ three methods: using a 3D laser scanner to scan the aircraft surface and segment the rivet positions to complete 3D modeling; using a 2D ultra-high-definition camera to acquire images and identify rivets; combining the correspondence between 2D images and the 3D digital model to obtain inspection results; and using a triangular camera to detect, locate, and stitch together rivets from different frames to construct a rivet distribution model.

[0004] Existing technical solutions have obvious drawbacks: 3D laser scanning is time-consuming and has a complicated operation process; the method of combining 2D cameras with 3D digital models cannot independently determine the correspondence between the two rivets, and this correspondence needs to be preset, which limits its practical application; the triangular camera stitching modeling scheme is difficult to quickly and effectively match the correspondence between rivet points obtained from different cameras, affecting modeling efficiency and accuracy. Summary of the Invention

[0005] This invention provides a method, system, medium, and product for detecting the quality of rivets on aircraft panels. The system acquires images of aircraft panels using a constructed rivet quality detection system, generates a three-dimensional structural model of the coordinates of rivet points and rivet holes on the aircraft panels, and quickly locates abnormal rivet points and rivet holes by matching them with the corresponding digital model.

[0006] According to one aspect of the present invention, a method for detecting the quality of aircraft panel rivets is provided. The method is executed by an aircraft panel rivet quality detection system, the system including a base, a linear sliding guide rail mounted on the base, a monocular motion camera mounted on the linear sliding guide rail, and a three-dimensional projector mounted on the base. The method includes:

[0007] By controlling a monocular motion camera to translate on a linear sliding guide rail, multiple images of the aircraft panel to be tested, placed below the detection system, are acquired at multiple equally spaced positions.

[0008] Based on the pre-built camera image radial distortion correction model and camera image tangential distortion correction model, distortion correction processing is performed on each aircraft panel image to obtain each corrected image;

[0009] In each corrected image, rivet points and rivet holes are identified. Based on the identification positions of each rivet point and rivet hole in adjacent corrected images, the first coordinate information of each rivet point and rivet hole in the current image coordinate system, as well as the corresponding identification parameters, are obtained.

[0010] Based on the first coordinate information, and using the objective function constructed by the bundle adjustment method to minimize the reprojection error, the second coordinate information of each rivet point and each rivet hole in the slide rail base coordinate system is obtained.

[0011] A first point cloud set is constructed based on each second coordinate information, and the first point cloud set is matched and analyzed with the second point cloud set generated by the distribution of rivet points and rivet holes on the pre-constructed digital model of the aircraft panel to obtain the second abnormal coordinate information of abnormal rivet points and abnormal rivet holes in the coordinate system of the slide rail base.

[0012] The second abnormal coordinate information is converted into the third coordinate information in the three-dimensional projector coordinate system, and the abnormal rivet points and abnormal rivet holes that match the third coordinate information are projected and marked on the current aircraft section wall panel by the three-dimensional projector.

[0013] According to another aspect of the present invention, an apparatus for detecting the quality of aircraft panel rivets is provided. This apparatus can be configured in an aircraft panel rivet quality detection system and includes:

[0014] The image acquisition module is used to control a monocular motion camera to move on a linear sliding guide rail and acquire multiple images of the aircraft panel to be tested, which are placed below the detection system, at multiple equally spaced positions.

[0015] The image correction module is used to perform distortion correction processing on each aircraft panel image based on the pre-built camera image radial distortion correction model and camera image tangential distortion correction model, so as to obtain each corrected image.

[0016] The image adjacency correction module is used to identify rivet points and rivet holes in each corrected image, and to obtain the first coordinate information of each rivet point and each rivet hole in the current image coordinate system, as well as the corresponding identification parameters, based on the identification position of each rivet point and each rivet hole in the adjacent corrected images.

[0017] The coordinate information acquisition module is used to acquire the second coordinate information of each rivet point and each rivet hole in the slide rail base coordinate system based on the first coordinate information and the objective function constructed by the bundle adjustment method to minimize the reprojection error.

[0018] The anomaly detection module is used to construct a first point cloud set based on each second coordinate information, and to perform matching analysis between the first point cloud set and the second point cloud set generated by the distribution of rivet points and rivet holes on the pre-constructed digital model of the aircraft panel, so as to obtain the second anomaly coordinate information of the abnormal rivet points and abnormal rivet holes in the coordinate system of the slide rail base.

[0019] The anomaly annotation module is used to convert the second anomaly coordinate information into the third coordinate information in the three-dimensional projector coordinate system, and to project and annotate the abnormal rivet points and abnormal rivet holes that match the third coordinate information onto the current aircraft section panel using the three-dimensional projector.

[0020] According to another aspect of the present invention, a system is provided, the system comprising:

[0021] A base, a linear sliding rail mounted on the base, a monocular motion camera mounted on the linear sliding rail, a 3D projector mounted on the base, and at least one processor; and

[0022] A memory communicatively connected to the at least one processor; wherein,

[0023] The memory stores a computer program that can be executed by the at least one processor, which enables the at least one processor to perform the aircraft panel rivet quality detection method according to any embodiment of the present invention.

[0024] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions, the computer instructions being configured to cause a processor to execute and implement the aircraft panel rivet quality detection method according to any embodiment of the present invention.

[0025] According to another aspect of the present invention, a computer program product is also provided, including a computer program that, when executed by a processor, implements the steps of the method as described in any embodiment of the present invention.

[0026] The technical solution of this invention uses a constructed aircraft panel rivet quality detection system to control a monocular motion camera to translate on a linear sliding guide rail. Based on accuracy requirements, it acquires aircraft panel images at multiple preset equidistant positions. After distortion correction, the aircraft panel images undergo adjacent correction, and simultaneously acquires the first coordinate information of each rivet point and each rivet hole in the current image coordinate system, along with corresponding identification parameters. Then, based on the above data, it constructs an objective function to minimize reprojection error using bundle adjustment. The optimal solution is then obtained to determine the values ​​of each rivet point and each rivet hole. The system uses the second coordinate information in the slide rail base coordinate system and constructs a corresponding first point cloud set based on this. A pre-constructed second point cloud set reflecting the distribution of rivet points and rivet holes on the aircraft panel's digital model serves as the detection reference standard. Abnormal coordinate information of abnormal rivet points and holes in the slide rail base coordinate system can be obtained through matching and analysis within the first point cloud set. After converting the obtained abnormal coordinate information into coordinate information in the 3D projector coordinate system, the 3D projector can project and mark the positions of abnormal rivet points and holes on the current aircraft panel section based on the coordinate information. This technical solution provides a system for detecting the quality of rivets on aircraft panels. It uses a monocular motion camera to acquire images of the aircraft panel, constructs a point cloud set of rivet point and rivet hole coordinate information, and matches and analyzes this point cloud set with the corresponding point cloud set in the aircraft panel's digital model to perform anomaly detection. This method can quickly locate and identify abnormal rivet points and rivet holes on the aircraft panel, improving detection efficiency and accuracy.

[0027] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0028] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 This is a flowchart of a method for detecting the quality of rivets on an aircraft panel according to Embodiment 1 of the present invention;

[0030] Figure 2 This is a flowchart of another method for detecting the quality of aircraft panel rivets according to Embodiment 2 of the present invention;

[0031] Figure 3 This is a schematic diagram of the spatial coordinate relationship of each coordinate system in a method for detecting the quality of rivets on an aircraft panel according to Embodiment 2 of the present invention.

[0032] Figure 4 This is a schematic diagram of a monocular motion camera acquiring images of an aircraft panel according to Embodiment 2 of the present invention;

[0033] Figure 5 This is a schematic diagram of the identification of rivet points and rivet holes in an aircraft panel image applicable to Embodiment 2 of the present invention;

[0034] Figure 6 This is a schematic diagram of adjacent matching of aircraft panel images to the current image according to Embodiment 2 of the present invention;

[0035] Figure 7 This is a schematic diagram of adjacent image matching of aircraft panel images applicable to Embodiment 2 of the present invention;

[0036] Figure 8 This is a schematic diagram of iterative nearest point matching of adjacent images of aircraft panels according to Embodiment 2 of the present invention;

[0037] Figure 9 This is a schematic diagram of a system for detecting the quality of rivets on aircraft panels according to Embodiment 3 of the present invention;

[0038] Figure 10 This is a schematic diagram of a device for detecting the quality of rivets on aircraft panels according to Embodiment 4 of the present invention. Detailed Implementation

[0039] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0040] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0041] Example 1

[0042] Figure 1 This is a flowchart of a method for detecting the quality of rivets on aircraft panels according to Embodiment 1 of the present invention. This embodiment is applicable to situations where a three-dimensional structural model is constructed by acquiring images of rivet points and rivet holes on aircraft panels using a monocular motion camera, allowing for rapid location of abnormal rivet points and rivet holes. This method can be executed by an aircraft panel rivet quality detection device, which can be implemented in hardware and / or software and is generally configured within an aircraft panel rivet quality detection system. Figure 1 As shown, the method includes:

[0043] S110. By controlling a monocular motion camera to move along a linear sliding guide rail, multiple images of the aircraft panel to be tested, placed below the detection system, are acquired at multiple equally spaced positions.

[0044] Understandably, acquiring images of the aircraft panel is necessary before inspecting the rivet points and holes. This can be done using a pre-built aircraft panel rivet quality inspection system. The system includes a base, a linear sliding rail mounted on the base, a monocular motion camera mounted on the linear sliding rail, and a 3D projector mounted on the base. To improve the accuracy of identifying rivet points and holes on the aircraft panel, the monocular motion camera on the linear sliding rail can be controlled to move at preset intervals. At multiple equidistant positions, the monocular motion camera can acquire multiple images of the aircraft panel to be inspected, placed in the image acquisition area below the inspection system.

[0045] S120. Based on the pre-constructed camera image radial distortion correction model and camera image tangential distortion correction model, distortion correction processing is performed on each aircraft panel image to obtain each corrected image.

[0046] Among them, the radial distortion correction model can refer to a mathematical model used to correct the stretching and twisting distortion of image pixels along the radial direction caused by the optical characteristics of the lens, and restore the true geometric shape of the scene. The tangential distortion correction model can refer to a mathematical model used to correct the offset distortion of image pixels along the tangential direction caused by lens mounting deviation and optical component misalignment, and restore the true geometric structure of the image.

[0047] It is understandable that the initial aircraft panel images captured by a monocular motion camera will suffer from radial and tangential distortion due to the optical characteristics of the camera lens or deviations in hardware parameters. Therefore, a pre-built camera image radial distortion correction model and a camera image tangential distortion correction model can be used to correct the initial aircraft panel images captured by the monocular motion camera. This will eliminate the error influence of radial and tangential distortion on the rivet point and rivet hole position information, thereby obtaining a corrected image that can accurately reflect the position information of each rivet point and rivet hole on the aircraft panel image.

[0048] S130. Identify rivet points and rivet holes in each corrected image, and obtain the first coordinate information of each rivet point and each rivet hole in the current image coordinate system, as well as the corresponding identification parameters, based on the identification position of each rivet point and each rivet hole in the adjacent corrected images.

[0049] Adjacent image correction refers to the preprocessing operation of uniformly correcting imaging distortion and aligning spatial positions with pixel coordinate systems for adjacent frames acquired by a monocular motion camera at fixed intervals, ensuring consistent matching between adjacent images. The current image coordinate system refers to the Cartesian coordinate system defined on the digital image plane, using image pixels as units, to describe the two-dimensional position of pixels in the currently processed aircraft panel image. The first coordinate information refers to the two-dimensional coordinate information of each rivet point and rivet hole on the aircraft panel image, determined by pixels in the current image coordinate system. The identifier parameter refers to the parameter used to identify whether the same rivet point or rivet hole on the aircraft panel exists simultaneously in both adjacent matched aircraft panel images after adjacent matching between the current aircraft panel image and the aircraft panel image acquired at the next adjacent interval. If present, it is identified as 1; otherwise, it is identified as 0.

[0050] Understandably, a monocular motion camera acquires multiple aircraft panel images at the same translation distance position on a linear sliding guide rail. The translation distance of the monocular motion camera is adjusted according to the recognition accuracy requirements. Two adjacent aircraft panel images may include the same rivet point and rivet hole on multiple aircraft panel images. By traversing the corrected aircraft panel images and the aircraft panel images acquired at their adjacent distance positions, the first coordinate information of each rivet point and each rivet hole on the aircraft panel image in the corresponding current image coordinate system can be determined. In addition, if the position information of the same rivet point or rivet hole on the aircraft panel can be identified in both adjacent comparison aircraft panel images, the identification parameter of the rivet point or rivet hole corresponding to the two adjacent comparison aircraft panel images is set to 1. For rivet points or rivet holes on aircraft panel images that are not identified in both adjacent comparison aircraft panel images, the identification parameters of the rivet point or rivet hole corresponding to the two adjacent comparison aircraft panel images are all set to 0.

[0051] S140. Based on the first coordinate information, and using the objective function constructed by the bundle adjustment method to minimize the reprojection error, obtain the second coordinate information of each rivet point and each rivet hole in the coordinate system of the slide rail base.

[0052] The slide rail base coordinate system can be a three-dimensional rectangular coordinate system with a reference point on the slide rail base as the origin and specific directions on the base as coordinate axes. It can serve as a unified spatial reference for describing the positions of rivet points and rivet holes on the aircraft panel. The bundle adjustment method refers to an adjustment method that optimizes the exterior orientation elements and spatial coordinates of the points to be determined in each image of the acquired aircraft panel using the least squares method, minimizing the projection residuals of all light beams, thereby achieving high-precision calculation of the target's three-dimensional information. The objective function can be a mathematical expression that quantifies the deviation between the theoretical and measured values ​​of the pixel coordinates projected onto the image from the spatial coordinates of the rivet points and rivet holes on the aircraft panel in the slide rail base coordinate system, and constructs the optimal solution using the least squares method. The second coordinate information refers to the three-dimensional spatial coordinate information of each rivet point and rivet hole on the aircraft panel in the slide rail base coordinate system.

[0053] Understandably, in order to obtain more accurate second coordinate information of each rivet point and each rivet hole on the aircraft panel in the coordinate system of the slide rail base, the first coordinate information of each rivet point and each rivet hole can be used to construct an objective function to minimize the reprojection error using the bundle adjustment method. Under the constraint of minimizing the reprojection error, the second coordinate information of each rivet point and each rivet hole can be obtained, that is, the optimal three-dimensional spatial coordinate information represented in the coordinate system of the slide rail base.

[0054] S150. Construct a first point cloud set based on each second coordinate information, and perform matching analysis between the first point cloud set and the second point cloud set generated by the distribution of rivet points and rivet holes on the pre-constructed digital model of the aircraft panel to obtain the second abnormal coordinate information of abnormal rivet points and abnormal rivet holes in the coordinate system of the slide rail base.

[0055] The first point cloud set can refer to a dataset in the slide rail base coordinate system composed of second coordinate information representing the spatial positions of rivet points and rivet holes on the aircraft panel surface. The second point cloud set can refer to a dataset composed of the coordinate information of rivet points and rivet holes on a pre-constructed digital model of the aircraft panel in the slide rail base coordinate system. The second abnormal coordinate information can refer to the second coordinate information of rivet points or rivet holes in the first point cloud set that show abnormalities, determined through matching analysis using the second point cloud set as the detection standard.

[0056] Understandably, before performing the riveting process on the aircraft panel, a digital model of the spatial distribution of each rivet point and rivet hole on the aircraft panel needs to be constructed according to process requirements. Based on the pre-constructed digital model, a second point cloud set can be generated in the coordinate system of the slide rail base, serving as a matching standard for detecting abnormal rivet points and rivet holes on the aircraft panel. After obtaining the second coordinate information of each rivet point and rivet hole on the aircraft panel in the coordinate system of the slide rail base using the bundle adjustment method, a first point cloud set representing the distribution of all rivet points and rivet holes currently being detected on the aircraft panel can be constructed. After matching and analyzing with the pre-generated second point cloud set, the coordinate information of rivet points or rivet holes present in the second point cloud set but not in the first point cloud set, as well as the coordinate information of the same rivet point or rivet hole in the first point cloud set, and the coordinate information whose error with the coordinate information in the second point cloud set exceeds a threshold range, can be used as the second abnormal coordinate information.

[0057] S160. Convert the second abnormal coordinate information into the third coordinate information in the three-dimensional projector coordinate system, and project and mark the abnormal rivet points and abnormal rivet holes that match the third coordinate information onto the current aircraft section wall panel using the three-dimensional projector.

[0058] The three-dimensional projector coordinate system can refer to a three-dimensional rectangular coordinate system with a reference point of the three-dimensional projector mounted on the slide rail base as the origin and specific directions of the three-dimensional projector as the coordinate axes. It can serve as a unified spatial reference for describing the positions of rivet points and rivet holes on the aircraft panel. The third coordinate information can refer to the three-dimensional spatial coordinate information of each rivet point and each rivet hole on the aircraft panel in the three-dimensional projector coordinate system.

[0059] Understandably, after obtaining the second abnormal coordinate information of abnormal rivet points and abnormal rivet holes in the slide rail base coordinate system through matching analysis, the second abnormal coordinate information can be converted into the third coordinate information in the three-dimensional projector coordinate system according to the coordinate transformation relationship between the slide rail base coordinate system and the three-dimensional projector coordinate system. Furthermore, in order to facilitate the identification of abnormal rivet points and abnormal rivet holes by staff, the positions of abnormal rivet points and abnormal rivet holes corresponding to the third coordinate information can be projected and marked on the surface of the currently inspected aircraft section panel using a three-dimensional projector.

[0060] The technical solution of this invention uses a constructed aircraft panel rivet quality detection system to control a monocular motion camera to translate on a linear sliding guide rail. Based on accuracy requirements, it acquires aircraft panel images at multiple preset equidistant positions. After distortion correction, the aircraft panel images undergo adjacent correction, and simultaneously acquires the first coordinate information of each rivet point and each rivet hole in the current image coordinate system, along with corresponding identification parameters. Then, based on the above data, it constructs an objective function to minimize reprojection error using bundle adjustment. The optimal solution is then obtained to determine the values ​​of each rivet point and each rivet hole. The system uses the second coordinate information in the slide rail base coordinate system and constructs a corresponding first point cloud set based on this. A pre-constructed second point cloud set reflecting the distribution of rivet points and rivet holes on the aircraft panel's digital model serves as the detection reference standard. Abnormal coordinate information of abnormal rivet points and holes in the slide rail base coordinate system can be obtained through matching and analysis within the first point cloud set. After converting the obtained abnormal coordinate information into coordinate information in the 3D projector coordinate system, the 3D projector can project and mark the positions of abnormal rivet points and holes on the current aircraft panel section based on the coordinate information. This technical solution provides a system for detecting the quality of rivets on aircraft panels. It uses a monocular motion camera to acquire images of the aircraft panel, constructs a point cloud set of rivet point and rivet hole coordinate information, and matches and analyzes this point cloud set with the corresponding point cloud set in the aircraft panel's digital model to perform anomaly detection. This method can quickly locate and identify abnormal rivet points and rivet holes on the aircraft panel, improving detection efficiency and accuracy.

[0061] Example 2

[0062] Figure 2 This is a flowchart of another method for detecting the quality of aircraft panel rivets according to Embodiment 2 of the present invention. This embodiment is a specific embodiment based on the above embodiments, including: a specific method for correcting image distortion of the aircraft panel, and a specific method for constructing an objective function to minimize reprojection error using the bundle adjustment method. Figure 2 As shown, the method includes:

[0063] S210: By controlling a monocular motion camera to move along a linear sliding guide rail, multiple images of the aircraft panel to be tested, placed below the detection system, are acquired at multiple equally spaced positions.

[0064] Optionally, before acquiring multiple images of the aircraft panel to be tested, placed below the detection system, at multiple equidistant positions by controlling a monocular motion camera to translate along a linear sliding guide rail, the process includes:

[0065] The positions of the slide rail base coordinate system, camera coordinate system, and 3D projector coordinate system are calibrated on the aircraft panel rivet quality inspection system;

[0066] Based on the external parameters of the monocular motion camera, determine the transformation relationship of the coordinate values ​​of each point in the rivet points and rivet holes of the aircraft panel in the coordinate system of the slide rail base, the camera coordinate system, and the coordinate system of the three-dimensional projector.

[0067] The camera coordinate system can be a three-dimensional Cartesian coordinate system with a reference point at the current position of the monocular motion camera on the slide rail as the origin and specific camera directions as coordinate axes. It can serve as a unified spatial reference for describing the positions of rivet points and rivet holes on aircraft panels. Camera extrinsic parameters refer to the set of parameters describing the spatial position and attitude of the camera in the slide rail base coordinate system, primarily including translation vectors and rotation-related parameters.

[0068] Specifically, based on the structural distribution of the aircraft panel rivet quality inspection system, the positions of the slide rail base coordinate system, camera coordinate system, and 3D projector coordinate system can be calibrated, such as... Figure 3 As shown, A is a linear sliding guide rail, B is a 3D laser projector, C is a large-field-of-view high-resolution camera system, and D is the end panel of the part to be inspected. Coordinate system S1 is the sliding rail base coordinate system, where the end point of the sliding rail can be selected as the origin, the direction parallel to the sliding rail as the x-axis, the direction perpendicular to the x-axis and parallel to the ground as the y-axis, and the vertically downward direction as the z-axis. Coordinate system S3 is the camera coordinate system, where the center point of the camera's current position on the sliding rail can be selected as the origin, the direction parallel to the sliding rail as the x-axis, the direction perpendicular to the x-axis and parallel to the ground as the y-axis, and the vertically downward direction as the z-axis. Coordinate system S2 is the 3D projector coordinate system, where the center point of the 3D projector at the installation position of the inspection system can be selected as the origin, the direction parallel to the sliding rail as the x-axis, the direction perpendicular to the x-axis and parallel to the ground as the y-axis, and the vertically downward direction as the z-axis. The transformation matrix for converting coordinate values ​​from the camera coordinate system to the sliding rail base coordinate system is determined based on the camera's external parameters. Where L represents the distance the camera coordinate system origin moves on the slide rail relative to the origin of the slide rail base coordinate system, in the camera extrinsic parameters, Let q represent the rotation matrix of the camera coordinate system relative to the slider base coordinate system. x q y q z These represent the constant parameters related to the rotation of the camera coordinate system relative to the slider base coordinate system along the x, y, and z axes, respectively. Additionally, the transformation coordinate matrix can be determined based on the calibrated relative positions of the slider base coordinate system and the 3D projector coordinate system. Thus, the transformation formulas for the coordinate values ​​of each point in the rivet points and rivet holes of the aircraft panel in the coordinate system of the slide rail base, the camera coordinate system, and the 3D projector coordinate system are obtained:

[0069]

[0070] S220. Based on the pre-constructed radial distortion correction model and tangential distortion correction model of the camera image, distortion correction processing is performed on each aircraft panel image to obtain each corrected image.

[0071] Optionally, based on pre-built camera image radial distortion correction models and camera image tangential distortion correction models, distortion correction processing is performed on each aircraft panel image to obtain corrected images, including:

[0072] The formulas for the pre-built camera image radial distortion correction model, camera image tangential distortion correction model, and image distortion coordinate information are confirmed as follows:

[0073]

[0074]

[0075]

[0076]

[0077] Where k1 is the first-order radial distortion coefficient, k2 is the second-order radial distortion coefficient, k3 is the third-order radial distortion coefficient, p1 is the first tangential distortion component, and p2 is the second tangential distortion component. The x-coordinate of the camera image after radial distortion. This represents the ordinate of the camera image after radial distortion. The x-coordinate of the camera image after tangential distortion. This represents the ordinate of the camera image after tangential distortion. The composite x-axis after camera image distortion. y is the composite ordinate of the camera image after distortion, x is the abscissa of the camera image after distortion correction, y is the ordinate of the camera image after distortion correction, and r is the distance of the target pixel coordinates from the origin of the pixel coordinate system after tangential distortion correction.

[0078] One image of the aircraft panel is acquired sequentially and used as the current distorted image of the camera;

[0079] Use the x and y coordinates of each pixel in the current distorted image of the camera as... and After calculating the x and y values ​​corresponding to each pixel using the above formulas, the corrected image corresponding to the current distorted image of the camera is obtained.

[0080] Return to the previous step and execute the operation of acquiring one aircraft panel image at a time as the current camera distortion image, until all aircraft panel images have been processed.

[0081] Here, the current camera distortion image can refer to the aircraft panel image currently undergoing distortion correction processing, which is one of multiple initial distorted aircraft panel images acquired by a monocular motion camera at equally spaced positions. The target pixel coordinates can refer to the coordinates of the pixels currently undergoing distortion correction, selected sequentially within the current camera distortion image.

[0082] Specifically, multiple initial aircraft panel images acquired by a monocular motion camera at equidistant positions exhibit radial and tangential distortion. When correcting the distortion of these initial aircraft panel images, one can sequentially acquire one aircraft panel image as the current camera distortion image, and then sequentially iterate through the pixels in the current camera distortion image as the current pixel, assigning the horizontal and vertical coordinates of the current pixel to... and , where the x-axis value Includes the x-coordinate after radial distortion of the camera image and the x-coordinate of the camera image after tangential distortion y-axis value Includes the ordinate of the camera image after radial distortion And the ordinate of the camera image after tangential distortion Using the pre-constructed camera image radial distortion correction model, camera image tangential distortion correction model, and related formulas for image distortion coordinate information, the abscissa (x) and ordinate (y) of the current pixel after distortion correction can be calculated. Following this logic, the abscissa and ordinate values ​​of all pixels in the current distorted camera image can be calculated. After completing the distortion correction operation on all aircraft panel images as the current distorted camera image, the corrected image corresponding to multiple initial aircraft panel images acquired by the monocular motion camera at equally spaced positions can be finally obtained.

[0083] S230. Identify rivet points and rivet holes in each corrected image, and obtain the first coordinate information of each rivet point and each rivet hole in the current image coordinate system, as well as the corresponding identification parameters, based on the identification position of each rivet point and each rivet hole in the adjacent corrected images.

[0084] Optionally, rivet points and rivet holes are identified in each corrected image, and based on the identification positions of each rivet point and rivet hole in adjacent corrected images, the first coordinate information of each rivet point and rivet hole in the current image coordinate system, as well as the corresponding identification parameters, are obtained, including:

[0085] The rivet points and rivet holes are identified in each corrected image to obtain their locations in each corrected image.

[0086] Based on the identification positions of the rivet points and rivet holes in each corrected image, and the acquisition position of the monocular motion camera when acquiring each aircraft panel image corresponding to each corrected image, the first coordinate information of each identified rivet point and rivet hole in the current image coordinate system is obtained.

[0087] According to the order of acquisition positions from front to back, one corrected image is acquired in sequence as the current processing image, and the adjacent processing image that matches the next acquisition position of the current processing image is acquired.

[0088] The rivet points and rivet holes identified in the current processed image and adjacent processed images will be matched, and the identification parameters of the successfully matched rivet points and rivet holes in the current processed image and adjacent processed images will be set to the target values.

[0089] Returning to the previous step, the system will sequentially acquire one corrected image as the current image for processing, following the order of acquisition positions from front to back, until all corrected images have been processed.

[0090] Specifically, when performing adjacent correction on rivet points and rivet holes in the distortion-corrected images of each aircraft panel, such as... Figure 5 As shown, the positions of rivet points and rivet holes can first be identified in each corrected aircraft panel image. Based on the acquisition position of the monocular motion camera when acquiring each aircraft panel image corresponding to each corrected image, the first coordinate information of each rivet point and rivet hole in the current image coordinate system within each corrected aircraft panel image can be determined. Then, according to the order in which the monocular motion camera acquired the aircraft panel images, the corrected aircraft panel images corresponding to each acquisition position can be sequentially acquired as the current processing image, and processed as follows... Figure 6 The image shows the identification of the positions of each rivet point and each rivet hole. The aircraft panel image corresponding to the next acquisition position adjacent to the current processed image is taken as the adjacent processed image, and so on. Figure 7 The diagram shows the identification of the positions of each rivet point and each rivet hole. Based on the translation interval during image acquisition by the monocular motion camera, the transformation relationship between the image coordinate systems of the currently processed image and adjacent processed images can be determined. This allows us to determine the coordinate correspondence of the same rivet point or rivet hole in the image coordinate systems of two adjacent, corrected aircraft panel images. Further, as shown... Figure 8As shown, the iterative nearest-point matching algorithm is applied for adjacent matching. It iterates through all rivet points and rivet holes covered by the current and adjacent processed images. The algorithm judges the matching based on the coordinates of the same rivet point or hole in the image coordinate systems of the current and adjacent processed images. If the distance between the coordinates of the same rivet point or hole in the two image coordinate systems is less than a preset threshold, then the rivet point or hole is determined to have successfully matched in the adjacent correction, and the corresponding identifier parameters of the current and adjacent processed images are both set to 1; otherwise, they are all set to 0. For example, for the j-th rivet point, the current processed image is the i-th corrected aircraft panel image, and the adjacent processed image is the (i+1)-th corrected aircraft panel image. After successful adjacent correction matching, the identifier parameters of the pre-processed image are... Corresponding identifier parameters of adjacent processed images All are set to 1. Following the above logic, adjacent matching is performed on all the aircraft panel images acquired by the monocular motion camera and their corresponding corrected images to obtain the first coordinate information of each rivet point and each rivet hole in the image coordinate system corresponding to each aircraft panel image, as well as the corresponding identification parameters.

[0091] S240, The movement interval distance for acquiring images of the aircraft panel based on the translational movement of the monocular motion camera on the slide rail. We obtain the objective function for minimizing reprojection error by constructing a bundle adjustment method.

[0092]

[0093] Where m is the total number of rivets and rivet holes, and n is the number of positions where the monocular motion camera acquires images on the slide rail. This indicates whether the j-th rivet point or rivet hole is present in the i-th corrected image. If it is, then... If it is 1, then... f is 0 x f is the focal length along the x-axis in the image coordinate system. y Let u0 be the focal length along the y-axis in the image coordinate system, and v0 be the coordinates of the center point of the image in the image coordinate system. ij and v ij Let x be the two-dimensional image coordinate of the j-th rivet point or rivet hole in the i-th corrected image. wj ,y wj ,z wj R1 represents the second coordinate information of the j-th rivet or rivet hole in the slide rail base coordinate system. R1 and T1 represent the rotation and translation matrices of the image coordinates in the slide rail base coordinate system to the camera coordinate system in the first image acquired by the camera.

[0094] Specifically, such as Figure 4The monocular motion camera shown is positioned on the slider at preset intervals. To obtain more accurate detection coordinate information of each rivet point and rivet hole on the aircraft panel in the coordinate system of the slide rail base, images of the aircraft panel are acquired by translation. This can be achieved by constructing and solving an objective function to minimize the reprojection error using the bundle adjustment method. The formula for the reprojection error e in the bundle adjustment method is as follows: In the above formula, x w ,y w ,z w Let u and v be the coordinates of the rivet point or rivet hole in the slide rail base coordinate system, and let P be the camera projection matrix. The expression for the camera projection matrix P is: ,in, Let P be the transformation matrix from the rail base coordinate system to the camera coordinate system. Since the monocular motion camera acquires images of the aircraft panel by translating at preset intervals, the camera coordinate system changes with the position of the image acquired by the monocular motion camera, and the aircraft projection matrix P also changes accordingly. We can define the aircraft projection matrix P1 corresponding to the position where the monocular motion camera first acquires the image of the aircraft panel as... The translation distance of the monocular motion camera is Then, the machine projection matrix P2 corresponding to the second image acquisition position of the monocular motion camera can be represented as:

[0095]

[0096] Thus, the machine projection matrix P corresponding to the position of the i-th acquired image from the monocular motion camera can be obtained. i Calculation formula: Formula for summing structural projection errors: Ultimately, we can obtain the objective function that minimizes the reprojection error:

[0097]

[0098] S250. Solve the constructed objective function to minimize the reprojection error, and calculate the second coordinate information of each rivet point and rivet hole on the aircraft panel image in the coordinate system of the slide rail base.

[0099] Understandably, when performing anomaly detection on the rivet points and rivet holes on the aircraft panel, it is necessary to obtain the most accurate coordinate information of each rivet point and rivet hole in the coordinate system of the slide rail base. Therefore, the aircraft panel images are acquired by a monocular motion camera at preset intervals. After distortion correction and adjacent matching, the initial aircraft panel images are comprehensively considered, taking into account the identification error of the monocular motion camera at each acquisition position for the same rivet point or rivet hole. After solving the objective function that minimizes the reprojection error constructed by the bundle adjustment method, more accurate second coordinate information of each rivet point and rivet hole on the aircraft panel in the coordinate system of the slide rail base can be obtained, which serves as the basis for the subsequent construction of the detection point cloud of each rivet point and rivet hole on the aircraft panel.

[0100] S260. Construct a first point cloud set based on each second coordinate information, and perform matching analysis between the first point cloud set and the second point cloud set generated by the distribution of rivet points and rivet holes on the pre-constructed digital model of the aircraft panel to obtain the second abnormal coordinate information of abnormal rivet points and abnormal rivet holes in the coordinate system of the slide rail base.

[0101] Optionally, a first point cloud set is constructed based on each second coordinate information, and the first point cloud set is matched and analyzed with the second point cloud set generated from the distribution of rivet points and rivet holes on the pre-constructed digital model of the aircraft panel to obtain the second abnormal coordinate information of abnormal rivet points and abnormal rivet holes in the coordinate system of the slide rail base, including:

[0102] In the first point cloud set constructed under the coordinate system of the slide rail base, a rivet point or rivet hole is obtained in sequence as the current identification object, and the current second coordinate information corresponding to the current identification object is obtained;

[0103] In the second point cloud set, after obtaining the standard second coordinate information corresponding to the current identified object, the current second coordinate information is matched and analyzed with the standard second coordinate information;

[0104] If the matching result between the current second coordinate information and the standard second coordinate information is determined to meet the preset anomaly filtering rules, then the current second coordinate information is determined to be the second abnormal coordinate information;

[0105] Among them, the abnormal screening rules include: missing hole screening rules, multiple hole screening rules, missing nail screening rules, and excessive hole and nail position deviation screening rules.

[0106] Return to the first point cloud set constructed in the coordinate system of the slide rail base, and sequentially obtain one rivet point or rivet hole as the current identification object for processing, until all rivet points and rivet holes have been processed.

[0107] Here, the currently identified object can refer to the rivet point or rivet hole that is being detected for anomalies in the point cloud set corresponding to the digital model of the aircraft panel to be referenced, which is currently obtained from the first point cloud set. The current second coordinate information can refer to the second coordinate information in the first point cloud set used to characterize the currently identified object in the coordinate system of the slide rail base. The standard second coordinate information can refer to the coordinate information of each rivet point and rivet hole in the coordinate system of the slide rail base, generated based on the distribution of rivet points and rivet holes on the digital model of the aircraft panel.

[0108] Specifically, based on the second coordinate information of each rivet point and rivet hole on the aircraft panel in the coordinate system of the slide rail base obtained in the above steps, a first point cloud set in the coordinate system of the slide rail base can be constructed. The second point cloud set is constructed based on the coordinate information of each rivet point and rivet hole on the digital model of the aircraft panel in the coordinate system of the slide rail base, which can be used as the standard second coordinate information for detecting abnormal rivet points or rivet holes on the aircraft panel. One rivet point or rivet hole is sequentially obtained from the first point cloud set as the current identification object. After obtaining the corresponding current second coordinate information, the standard second coordinate information corresponding to the current identification object can be obtained from the second point cloud set. By matching and analyzing the current second coordinate information with the standard second coordinate information, the abnormal rivet point or abnormal rivet hole corresponding to the current identification object can be determined according to the preset anomaly filtering rules, and the current second coordinate information is determined as the second abnormal coordinate information. Following the logic described above, each rivet point and rivet hole in the first point cloud set is traversed as the current identification object. Anomaly detection is performed using the multi-hole filtering rule and the excessive hole / rivet position deviation filtering rule. Then, the standard second coordinate information of rivet points or rivet holes in the second point cloud set that have not been matched and analyzed is used for anomaly detection using the missing hole filtering rule and the missing rivet filtering rule. Finally, all abnormal rivet points and abnormal rivet holes on the currently detected aircraft panel are obtained. Specifically, the missing hole filtering rule is used when the standard second coordinate information of a rivet hole exists in the second point cloud set, but the second coordinate information of that rivet hole within the error threshold range does not exist in the first point cloud set; in this case, the rivet hole is determined to be a missing hole in the first point cloud set. The multi-hole filtering rule is used when the second coordinate information of the rivet hole corresponding to the current identification object exists in the first point cloud set, but the standard second coordinate information of the rivet hole corresponding to the current identification object within the error threshold range does not exist in the second point cloud set; in this case, the rivet hole is determined to be a multi-hole in the first point cloud set. The "Missing Rivet" filtering rule is used when a rivet point has standard second coordinate information in the second point cloud, but the second coordinate information of that rivet point within the error threshold range does not exist in the first point cloud. In this case, the rivet point is identified as a missing rivet in the first point cloud. The "Excessive Hole / Rivet Position Deviation" filtering rule is used when a rivet point or rivet hole corresponding to the currently identified object has second coordinate information in the first point cloud, but the standard second coordinate information of the rivet point or rivet hole corresponding to the currently identified object within the preset precision error range does not exist in the second point cloud. In this case, the rivet point or rivet hole corresponding to the currently identified object is identified as a rivet point or rivet hole with excessive hole / rivet position deviation in the first point cloud.

[0109] S270. Convert the second abnormal coordinate information into the third coordinate information in the three-dimensional projector coordinate system, and project and mark the abnormal rivet points and abnormal rivet holes that match the third coordinate information on the current aircraft section wall panel using the three-dimensional projector.

[0110] Optionally, the second abnormal coordinate information is converted into third coordinate information in the three-dimensional projector coordinate system, and the abnormal rivet points and abnormal rivet holes matching the third coordinate information are projected and marked on the current aircraft section panel using a three-dimensional projector, including:

[0111] The second abnormal coordinate information of the rivet point or rivet hole in the coordinate system of the slide rail base is converted into the third coordinate information of the three-dimensional projector coordinate system according to the coordinate system transformation relationship;

[0112] The positions of abnormal rivet points and abnormal rivet holes are determined based on the third coordinate information in the three-dimensional projector coordinate system, and the corresponding positions are projected and marked on the current aircraft section panel using the three-dimensional projector.

[0113] Specifically, after determining the second abnormal coordinate information of the abnormal rivet points and abnormal rivet holes in the coordinate system of the slide rail base, the formula can be used to... By converting the second abnormal coordinate information corresponding to abnormal rivet points and abnormal rivet holes into the corresponding third coordinate information in the coordinate system of the three-dimensional projector, the three-dimensional projector in the aircraft panel rivet quality inspection system can determine the location of abnormal rivet points and abnormal rivet holes currently being inspected on the aircraft panel based on the third coordinate information corresponding to abnormal rivet points and abnormal rivet holes, and can mark the corresponding locations through projection, making it convenient for staff to identify and handle them.

[0114] The technical solution of this invention determines the coordinate values ​​of each rivet point and each rivet hole on the aircraft panel being inspected in different coordinate systems by calibrating the coordinate system of the slide rail base, the camera coordinate system, and the three-dimensional projector coordinate system on the aircraft panel rivet quality detection system. Based on the interval distance of the translation of the aircraft panel image acquired by the monocular motion camera and the camera extrinsic parameters, the transformation relationship of the coordinate values ​​of each rivet point and each rivet hole between the three coordinate systems is determined. By combining camera intrinsic parameters, camera image radial distortion correction model, camera image tangential distortion correction model, and image distortion coordinate information, distortion correction is performed on the acquired distorted aircraft panel images. By performing adjacent correction on the distorted aircraft panel images, relevant parameters are determined. Based on the obtained relevant parameters, an objective function for minimizing reprojection error based on the bundle adjustment method can be derived. Based on the solved second coordinate information of each rivet point and each rivet hole in the slide rail base coordinate system, a corresponding first point cloud set is constructed. The pre-constructed second point cloud set reflecting the distribution of rivet points and rivet holes on the digital model of the aircraft panel is used as a detection reference standard. According to the preset anomaly screening rules, the abnormal coordinate information of abnormal rivet points and abnormal rivet holes is obtained by matching and analyzing in the first point cloud set. After converting it into coordinate information in the three-dimensional projector coordinate system, the three-dimensional projector projects and marks the positions of abnormal rivet points and abnormal rivet holes on the current aircraft section panel. The above technical solution provides a method for constructing a point cloud set of rivet point and rivet hole coordinate information during the process of building the aircraft panel rivet quality detection system. It involves distorting the aircraft panel image and constructing an objective function based on the bundle adjustment method to minimize the reprojection error. This method is applicable to quality inspection scenarios of aircraft panels of different sizes, improves the efficiency and accuracy of abnormal rivet points and abnormal rivet holes in aircraft panels, and reduces the false detection rate and false negative rate.

[0115] Example 3

[0116] Figure 9 This is a schematic diagram of a system for detecting the quality of rivets on aircraft panels, provided in Embodiment 3 of the present invention. Figure 9 As shown, the aircraft panel rivet quality inspection system 900 includes:

[0117] The system includes a base 910, a linear sliding guide rail 920 mounted on the base, a monocular motion camera 930 mounted on the linear sliding guide rail, a three-dimensional projector 940 mounted on the base, and a controller 950, wherein the controller is used to execute a method for detecting the quality of aircraft panel rivets that implements any one of the embodiments of the present invention.

[0118] Example 4

[0119] Figure 10 This is a schematic diagram of a device for detecting the quality of rivets on aircraft panels, provided in Embodiment 4 of the present invention. Figure 10As shown, the device includes: an image acquisition module 1010, an image correction module 1020, an image adjacency correction module 1030, a coordinate information acquisition module 1040, an anomaly detection module 1050, and an anomaly annotation module 1060.

[0120] The image acquisition module 1010 is used to control a monocular motion camera to move on a linear sliding guide rail and acquire multiple images of the aircraft panel to be tested, which are placed below the detection system, at multiple equally spaced positions.

[0121] The image correction module 1020 is used to perform distortion correction processing on each aircraft panel image according to the pre-built camera image radial distortion correction model and camera image tangential distortion correction model to obtain each corrected image.

[0122] The image adjacency correction module 1030 is used to identify rivet points and rivet holes in each corrected image, and to obtain the first coordinate information of each rivet point and each rivet hole in the current image coordinate system, as well as the corresponding identification parameters, based on the identification position of each rivet point and each rivet hole in the adjacent corrected images.

[0123] The coordinate information acquisition module 1040 is used to acquire the second coordinate information of each rivet point and each rivet hole in the slide rail base coordinate system based on each first coordinate information and the objective function constructed by the bundle adjustment method to minimize the reprojection error.

[0124] The anomaly detection module 1050 is used to construct a first point cloud set based on each second coordinate information, and to perform matching analysis between the first point cloud set and the second point cloud set generated by the distribution of rivet points and rivet holes on the pre-constructed digital model of the aircraft panel, so as to obtain the second anomaly coordinate information of the abnormal rivet points and abnormal rivet holes in the coordinate system of the slide rail base.

[0125] The anomaly annotation module 1060 is used to convert the second anomaly coordinate information into the third coordinate information in the three-dimensional projector coordinate system, and to project and annotate the abnormal rivet points and abnormal rivet holes that match the third coordinate information onto the current aircraft section panel through the three-dimensional projector.

[0126] The technical solution of this invention uses a constructed aircraft panel rivet quality detection system to control a monocular motion camera to translate on a linear sliding guide rail. Based on accuracy requirements, it acquires aircraft panel images at multiple preset equidistant positions. After distortion correction, the aircraft panel images undergo adjacent correction, and simultaneously acquires the first coordinate information of each rivet point and each rivet hole in the current image coordinate system, along with corresponding identification parameters. Then, based on the above data, it constructs an objective function to minimize reprojection error using bundle adjustment. The optimal solution is then obtained to determine the values ​​of each rivet point and each rivet hole. The system uses the second coordinate information in the slide rail base coordinate system and constructs a corresponding first point cloud set based on this. A pre-constructed second point cloud set reflecting the distribution of rivet points and rivet holes on the aircraft panel's digital model serves as the detection reference standard. Abnormal coordinate information of abnormal rivet points and holes in the slide rail base coordinate system can be obtained through matching and analysis within the first point cloud set. After converting the obtained abnormal coordinate information into coordinate information in the 3D projector coordinate system, the 3D projector can project and mark the positions of abnormal rivet points and holes on the current aircraft panel section based on the coordinate information. This technical solution provides a system for detecting the quality of rivets on aircraft panels. It uses a monocular motion camera to acquire images of the aircraft panel, constructs a point cloud set of rivet point and rivet hole coordinate information, and matches and analyzes this point cloud set with the corresponding point cloud set in the aircraft panel's digital model to perform anomaly detection. This method can quickly locate and identify abnormal rivet points and rivet holes on the aircraft panel, improving detection efficiency and accuracy.

[0127] Optionally, it may also include a coordinate system calibration module for: calibrating the positions of the slide rail base coordinate system, camera coordinate system, and 3D projector coordinate system on the aircraft panel rivet quality detection system; and determining the transformation relationship of coordinate values ​​of each point in the aircraft panel rivet point and rivet hole in the slide rail base coordinate system, camera coordinate system, and 3D projector coordinate system based on the camera extrinsic parameters of the monocular motion camera.

[0128] Optionally, the image correction module 1020 can be specifically used to: confirm the relevant formulas for the pre-built camera image radial distortion correction model, camera image tangential distortion correction model, and image distortion coordinate information:

[0129]

[0130]

[0131]

[0132]

[0133] Where k1 is the first-order radial distortion coefficient, k2 is the second-order radial distortion coefficient, k3 is the third-order radial distortion coefficient, p1 is the first tangential distortion component, and p2 is the second tangential distortion component. The x-coordinate of the camera image after radial distortion. This represents the ordinate of the camera image after radial distortion. The x-coordinate of the camera image after tangential distortion. This represents the ordinate of the camera image after tangential distortion. The composite x-axis after camera image distortion. Let x be the composite ordinate of the distorted camera image, y be the abscissa of the distorted camera image, and r be the distance from the origin of the pixel coordinate system after tangential distortion correction. Sequentially acquire one image of the aircraft panel as the current distorted camera image, and use the abscissa and ordinate values ​​of each pixel in the current distorted camera image as... and After calculating the x and y values ​​corresponding to each pixel using the formulas above, the corrected image corresponding to the current camera distortion image is obtained. The process then returns to the previous step, sequentially acquiring one aircraft panel image as the current camera distortion image, until all aircraft panel images have been processed.

[0134] Optionally, the image adjacency correction module 1030 can be specifically used to: identify rivet points and rivet holes in each corrected image, and obtain the identification positions of rivet points and rivet holes in each corrected image. Based on the identification positions of rivet points and rivet holes in each corrected image, and the acquisition position of the monocular motion camera when acquiring images of each aircraft panel corresponding to each corrected image, obtain the first coordinate information of each identified rivet point and rivet hole in the current image coordinate system. Following the order of acquisition positions from front to back, acquire one corrected image as the current processing image, and acquire adjacent processing images that match the next acquisition position of the current processing image. Match the rivet points and rivet holes identified in the current processing image and adjacent processing images, and set the identification parameters of the successfully matched rivet points and rivet holes in the current processing image and adjacent processing images to the target value. Return to execute the operation of acquiring one corrected image as the current processing image in the order of acquisition positions from front to back, until the processing of all corrected images is completed.

[0135] Optionally, the coordinate information acquisition module 1040 can be specifically used to: measure the movement interval distance of the monocular motion camera as it moves along the slide rail to acquire images of the aircraft panel. Obtain the objective function for minimizing reprojection error using the bundle adjustment method:

[0136]

[0137] Where m is the total number of rivets and rivet holes, and n is the number of positions where the monocular motion camera acquires images on the slide rail. This indicates whether the j-th rivet point or rivet hole is present in the i-th corrected image. If it is, then... If it is 1, then... f is 0 x f is the focal length along the x-axis in the image coordinate system. y Let u0 be the focal length along the y-axis in the image coordinate system, and v0 be the coordinates of the center point of the image in the image coordinate system. ij and v ij Let x be the two-dimensional image coordinate of the j-th rivet point or rivet hole in the i-th corrected image. wj ,y wj ,z wj Let R1 and T1 represent the second coordinate information of the j-th rivet or rivet hole in the slide rail base coordinate system, respectively. R1 and T1 represent the rotation and translation matrices used to transform the image coordinates from the slide rail base coordinate system to the camera coordinate system in the first image acquired by the camera. The constructed objective function for minimizing reprojection error is solved to calculate the second coordinate information of each rivet point and rivet hole on the aircraft panel image in the slide rail base coordinate system.

[0138] Optionally, the anomaly detection module 1050 can be specifically used to: sequentially acquire a rivet point or rivet hole as the current identification object from the first point cloud set constructed in the slide rail base coordinate system, and acquire the current second coordinate information corresponding to the current identification object. In the second point cloud set, after acquiring the standard second coordinate information corresponding to the current identification object, perform matching analysis between the current second coordinate information and the standard second coordinate information. If it is determined that the matching result between the current second coordinate information and the standard second coordinate information conforms to the preset anomaly filtering rules, then the current second coordinate information is determined as the second anomaly coordinate information. The anomaly filtering rules include: missing hole filtering rules, multiple hole filtering rules, missing rivet filtering rules, and excessive hole / rivet position deviation filtering rules. The process of sequentially acquiring a rivet point or rivet hole as the current identification object from the first point cloud set constructed in the slide rail base coordinate system is returned to execution until all rivet points and rivet holes have been processed.

[0139] Optionally, the anomaly annotation module 1060 can be specifically used to: convert the second anomaly coordinate information of the rivet point or rivet hole in the slide rail base coordinate system into the third coordinate information in the three-dimensional projector coordinate system according to the coordinate system transformation relationship. The positions of the abnormal rivet points and abnormal rivet holes are determined in the three-dimensional projector coordinate system based on the third coordinate information, and the corresponding positions are projected and annotated on the current aircraft section panel using a three-dimensional projector.

[0140] The aircraft panel rivet quality detection device provided in this embodiment of the invention can execute the aircraft panel rivet quality detection method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method.

[0141] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0142] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0143] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0144] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for detecting the quality of rivets on aircraft panel, the method being executed by an aircraft panel rivet quality detection system, the system comprising a base, a linear sliding guide rail mounted on the base, a monocular motion camera mounted on the linear sliding guide rail, and a three-dimensional projector mounted on the base, the method comprising: By controlling a monocular motion camera to translate on a linear sliding guide rail, multiple images of the aircraft panel to be tested, placed below the detection system, are acquired at multiple equally spaced positions. Based on the pre-built camera image radial distortion correction model and camera image tangential distortion correction model, distortion correction processing is performed on each aircraft panel image to obtain each corrected image; In each corrected image, rivet points and rivet holes are identified. Based on the identification positions of each rivet point and rivet hole in adjacent corrected images, the first coordinate information of each rivet point and rivet hole in the current image coordinate system, as well as the corresponding identification parameters, are obtained. Based on the first coordinate information, and using the objective function constructed by the bundle adjustment method to minimize the reprojection error, the second coordinate information of each rivet point and each rivet hole in the slide rail base coordinate system is obtained. A first point cloud set is constructed based on each second coordinate information, and the first point cloud set is matched and analyzed with the second point cloud set generated by the distribution of rivet points and rivet holes on the pre-constructed digital model of the aircraft panel to obtain the second abnormal coordinate information of abnormal rivet points and abnormal rivet holes in the coordinate system of the slide rail base. The second abnormal coordinate information is converted into the third coordinate information in the three-dimensional projector coordinate system, and the abnormal rivet points and abnormal rivet holes that match the third coordinate information are projected and marked on the current aircraft section wall panel by the three-dimensional projector.

2. The method according to claim 1, characterized in that, Before acquiring multiple images of the aircraft panel to be tested, placed below the detection system, at multiple equidistant positions by controlling a monocular motion camera to translate along a linear sliding guide rail, the process includes: The positions of the slide rail base coordinate system, camera coordinate system, and 3D projector coordinate system are calibrated on the aircraft panel rivet quality inspection system; Based on the external parameters of the monocular motion camera, determine the transformation relationship of the coordinate values ​​of each point in the rivet points and rivet holes of the aircraft panel in the coordinate system of the slide rail base, the camera coordinate system, and the coordinate system of the three-dimensional projector.

3. The method according to claim 1, characterized in that, Based on pre-built camera image radial distortion correction models and camera image tangential distortion correction models, distortion correction processing is performed on each aircraft panel image to obtain corrected images, including: The formulas for the pre-built camera image radial distortion correction model, camera image tangential distortion correction model, and image distortion coordinate information are confirmed as follows: ； ； ； ； Where k1 is the first-order radial distortion coefficient, k2 is the second-order radial distortion coefficient, k3 is the third-order radial distortion coefficient, p1 is the first tangential distortion component, and p2 is the second tangential distortion component. The x-coordinate of the camera image after radial distortion. This represents the ordinate of the camera image after radial distortion. The x-coordinate of the camera image after tangential distortion. This represents the ordinate of the camera image after tangential distortion. The composite x-axis after camera image distortion. y is the composite ordinate of the camera image after distortion, x is the abscissa of the camera image after distortion correction, y is the ordinate of the camera image after distortion correction, and r is the distance of the target pixel coordinates from the origin of the pixel coordinate system after tangential distortion correction. One image of the aircraft panel is acquired sequentially and used as the current distorted image of the camera; The horizontal and vertical coordinates of each pixel in the current camera distortion image are used as... and After calculating x and y corresponding to each pixel using the above formulas, the corrected image corresponding to the current distorted image of the camera is obtained. Return to the previous step and execute the operation of acquiring one aircraft panel image at a time as the current camera distortion image, until all aircraft panel images have been processed.

4. The method according to claim 2, characterized in that, In each corrected image, rivet points and rivet holes are identified. Based on the identification positions of each rivet point and rivet hole in adjacent corrected images, the first coordinate information of each rivet point and rivet hole in the current image coordinate system, as well as the corresponding identification parameters, are obtained, including: The rivet points and rivet holes are identified in each corrected image to obtain their locations in each corrected image. Based on the identification positions of the rivet points and rivet holes in each corrected image, and the acquisition position of the monocular motion camera when acquiring each aircraft panel image corresponding to each corrected image, the first coordinate information of each identified rivet point and rivet hole in the current image coordinate system is obtained. According to the order of acquisition positions from front to back, one corrected image is acquired in sequence as the current processing image, and the adjacent processing image that matches the next acquisition position of the current processing image is acquired. The rivet points and rivet holes identified in the current processed image and adjacent processed images will be matched, and the identification parameters of the successfully matched rivet points and rivet holes in the current processed image and adjacent processed images will be set to the target values. Returning to the previous step, the system will sequentially acquire one corrected image as the current image for processing, following the order of acquisition positions from front to back, until all corrected images have been processed.

5. The method according to claim 4, characterized in that, Based on the objective function constructed using the bundle adjustment method to minimize reprojection error, the second coordinate information of each rivet point and each rivet hole in the slide rail base coordinate system is obtained, including: The movement interval distance of the monocular motion camera to acquire images of the aircraft panel by translating along the slide rail. Obtain the objective function for minimizing reprojection error using the bundle adjustment method: ； Where m is the total number of rivets and rivet holes, and n is the number of positions where the monocular motion camera acquires images on the slide rail. This indicates whether the j-th rivet point or rivet hole is present in the i-th corrected image. If it is, then... If it is 1, then... f is 0 x f is the focal length along the x-axis in the image coordinate system. y Let u0 be the focal length along the y-axis in the image coordinate system, and v0 be the coordinates of the center point of the image in the image coordinate system. ij and v ij Let x be the two-dimensional image coordinate of the j-th rivet point or rivet hole in the i-th corrected image. wj ,y wj ,z wj R1 represents the second coordinate information of the j-th rivet or rivet hole in the slide rail base coordinate system. R1 and T1 represent the rotation matrix and translation matrix of the image coordinate transformation from the slide rail base coordinate system to the camera coordinate system in the first image acquired by the camera. The objective function for minimizing reprojection error is solved to obtain the second coordinate information of each rivet point and rivet hole on the aircraft panel image in the coordinate system of the slide rail base.

6. The method according to any one of claims 1-5, characterized in that, A first point cloud set is constructed based on the second coordinate information, and then matched and analyzed with a second point cloud set generated from the distribution of rivet points and rivet holes on a pre-constructed digital model of the aircraft panel. This yields the second abnormal coordinate information of abnormal rivet points and abnormal rivet holes in the coordinate system of the slide rail base, including: In the first point cloud set constructed under the coordinate system of the slide rail base, a rivet point or rivet hole is obtained in sequence as the current identification object, and the current second coordinate information corresponding to the current identification object is obtained; In the second point cloud set, after obtaining the standard second coordinate information corresponding to the current identified object, the current second coordinate information is matched and analyzed with the standard second coordinate information; If the matching result between the current second coordinate information and the standard second coordinate information is determined to meet the preset anomaly filtering rules, then the current second coordinate information is determined to be the second abnormal coordinate information; The abnormal screening rules include: missing hole screening rules, multiple hole screening rules, missing nail screening rules, and excessive hole and nail position deviation screening rules. Return to the first point cloud set constructed in the coordinate system of the slide rail base, and sequentially obtain one rivet point or rivet hole as the current identification object for processing, until all rivet points and rivet holes have been processed.

7. The method according to claim 2, characterized in that, The second abnormal coordinate information is converted into third coordinate information in the three-dimensional projector coordinate system, and the abnormal rivet points and abnormal rivet holes matching the third coordinate information are projected and marked on the current aircraft section panel using a three-dimensional projector, including: The second abnormal coordinate information of the rivet point or rivet hole in the coordinate system of the slide rail base is converted into the third coordinate information of the three-dimensional projector coordinate system according to the coordinate system transformation relationship; The positions of abnormal rivet points and abnormal rivet holes are determined based on the third coordinate information in the three-dimensional projector coordinate system, and the corresponding positions are projected and marked on the current aircraft section panel using the three-dimensional projector.

8. A system for detecting the quality of rivets on aircraft panels, characterized in that, The system includes: The base, a linear sliding rail mounted on the base, a monocular motion camera mounted on the linear sliding rail, a 3D projector mounted on the base, and a controller; The controller is used to execute the method for detecting the quality of aircraft panel rivets according to any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that are used to cause the aircraft panel rivet quality detection system to implement the aircraft panel rivet quality detection method according to any one of claims 1-7 when executed.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by an aircraft panel rivet quality detection system, implements the aircraft panel rivet quality detection method according to any one of claims 1-7.

Images