Wallboard defect detection method and device based on visual positioning, equipment and medium

Abstract

The application discloses a wallboard defect detection method, device, equipment and medium based on visual positioning. The method comprises the following steps: collecting three-dimensional point cloud and two-dimensional image data of the wallboard through a visual positioning unit, extracting fastener mounting hole feature points in the point cloud, matching the feature points with corresponding hole position theoretical coordinates of a pre-stored CAD model, adjusting the three-dimensional point cloud according to the matching, and obtaining a three-dimensional point cloud space aligned with the CAD model space. Pre-calibration parameters are used to correct two-dimensional image distortion, the corrected two-dimensional image is mapped to the aligned three-dimensional point cloud space, three-dimensional detection data with enhanced texture are generated, and a pre-trained deep learning classification model is inputted for defect detection. Through automatic spatial positioning and multi-dimensional data fusion, the application solves the positioning difficulty and low precision problem of large component detection, accurately identifies defects such as missing holes and missing fasteners with the help of deep learning, and significantly improves the control efficiency and reliability of the assembly quality of large wallboards.

Description

Technical Field

[0001] This invention relates to the field of industrial defect detection technology, and in particular to a method, apparatus, equipment and medium for detecting wall panel defects based on visual positioning. Background Technology

[0002] Aircraft skin panels are a crucial component of an aircraft's external structure. Their primary functions include providing structural strength, maintaining the fuselage's aerodynamic characteristics, protecting internal systems, and serving as mounting bases for various equipment and systems. During aircraft manufacturing, the skin panels contain a vast number of holes, rivets, and bolts. Ensuring the correct installation of each hole, rivet, and bolt is paramount. Missing holes, rivets, or bolts can lead to insufficient structural strength, impacting the aircraft's safety and performance. While manual counting and inspection can identify these issues, the process is tedious, error-prone, and cannot guarantee efficiency and accuracy.

[0003] While existing automation technologies, such as 3D scanning, can acquire complete three-dimensional data, they suffer from inherent drawbacks such as low scanning efficiency, high hardware costs, and poor adaptability to complex curved surfaces. On the other hand, although 2D vision inspection improves efficiency, it requires manual annotation of each hole and fastener position, which is not only labor-intensive and prone to human error, but also requires repeating the annotation process when the digital model changes. Its flexibility and scalability are severely lacking, so there is an urgent need for an efficient, accurate, and highly adaptable automated inspection solution. Summary of the Invention

[0004] Based on this, the present invention provides a method, device, equipment and medium for detecting panel defects based on visual positioning, so as to solve the problems of high error rate and poor adaptability to design changes caused by the reliance on manual annotation in traditional 2D visual inspection methods, while overcoming the problems of high cost and insufficient efficiency of 3D scanning inspection.

[0005] In a first aspect, embodiments of the present invention provide a visual positioning-based method for detecting wall panel defects, applied in a scenario where fastener defects are detected in the wall panel under test by controlling a telescopic linkage on the roof of a guided vehicle and a visual positioning unit on the top of the telescopic linkage, including:

[0006] The three-dimensional point cloud data of the surface of the wall panel under test is collected by the lidar in the visual positioning unit under the target pose that meets the detection conditions, and the two-dimensional image data of the surface of the wall panel under test is collected by the visual sensor in the visual positioning unit.

[0007] Extract fastener mounting hole feature points from the three-dimensional point cloud data, and match the fastener mounting hole feature points with the theoretical coordinates of the corresponding hole positions in the pre-stored CAD model;

[0008] Based on the matching results, the spatial transformation matrix from the 3D point cloud to the CAD model is calculated, and the 3D point cloud is rotated and translated according to the spatial transformation matrix to obtain a 3D point cloud space aligned with the CAD model space.

[0009] Using a pre-calibrated visual sensor intrinsic parameter matrix and distortion coefficients, optical distortion correction is performed on the two-dimensional image data, and the corrected two-dimensional image data is mapped to an aligned three-dimensional point cloud space based on the spatial transformation matrix to generate texture-enhanced three-dimensional detection data.

[0010] The texture-enhanced 3D detection data is input into a pre-trained deep learning classification model to perform defect detection on the panel under test.

[0011] Secondly, embodiments of the present invention also provide a wall panel defect detection device based on visual positioning, comprising:

[0012] The visual acquisition unit is used to acquire three-dimensional point cloud data of the surface of the wall panel under test by the lidar in the visual positioning unit under the target pose that meets the detection conditions, and to acquire two-dimensional image data of the surface of the wall panel under test by the visual sensor in the visual positioning unit.

[0013] The feature point matching unit is used to extract fastener mounting hole feature points from the three-dimensional point cloud data and match the fastener mounting hole feature points with the theoretical coordinates of the corresponding hole positions in the pre-stored CAD model.

[0014] The spatial alignment unit is used to calculate the spatial transformation matrix from the 3D point cloud to the CAD model based on the matching result, and to rotate and adjust the 3D point cloud according to the spatial transformation matrix to obtain a 3D point cloud space aligned with the CAD model space.

[0015] The texture enhancement unit is used to perform optical distortion correction on the two-dimensional image data using a pre-calibrated visual sensor intrinsic parameter matrix and distortion coefficients, and to map the corrected two-dimensional image data to an aligned three-dimensional point cloud space based on the spatial transformation matrix to generate texture-enhanced three-dimensional detection data.

[0016] The defect detection unit is used to input texture-enhanced 3D detection data into a pre-trained deep learning classification model to perform defect detection on the panel under test.

[0017] Thirdly, embodiments of the present invention also provide an electronic device, the electronic device comprising:

[0018] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform a visual positioning-based panel defect detection method according to any embodiment of the present invention.

[0019] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing computer instructions, which are used to cause a processor to execute and implement a visual positioning-based panel defect detection method according to any embodiment of the present invention.

[0020] This invention, through the fusion of LiDAR 3D point clouds and visual sensor 2D images, combined with CAD digital models for precise spatial alignment, generates texture-enhanced 3D inspection data that simultaneously carries geometric accuracy and appearance feature information. This data is suitable for fastener defect detection in large components such as aerospace fuselage panels and high-speed rail carriage sidewalls. Its core advantage lies in its ability to address the characteristics of large panels—large size, dense fastener distribution, and high precision requirements—by replacing traditional manual visual inspection or single-sensor detection with automated spatial positioning and multi-dimensional data fusion. This solves the problems of difficult positioning and low accuracy in large component inspection, and leverages deep learning models to accurately identify defects such as missing holes and missing fasteners. In industrial assembly line environments, this enables efficient and high-precision automated inspection, significantly improving the efficiency and reliability of large panel assembly quality control.

[0021] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart of a wall panel defect detection method based on visual positioning according to Embodiment 1 of the present invention;

[0024] Figure 2 This is a schematic diagram illustrating the positional relationship between a guided vehicle, a linkage telescopic body, a visual positioning unit, and a wall panel under test, applicable to embodiments of the present invention.

[0025] Figure 3 This is a flowchart of another visual positioning-based panel defect detection method provided according to Embodiment 2 of the present invention;

[0026] Figure 4 This is a schematic diagram of a wall panel defect detection device based on visual positioning according to Embodiment 3 of the present invention;

[0027] Figure 5 This is a schematic diagram of the structure of an electronic device that implements a vision-based panel defect detection method according to an embodiment of the present invention. Detailed Implementation

[0028] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0029] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0030] Example 1

[0031] Figure 1 This is a flowchart of a vision-based panel defect detection method according to Embodiment 1 of the present invention. This embodiment is applicable to high-precision defect detection of fastener mounting holes and matching fasteners in large metal or composite material panel walls. The method can be executed by a vision-based panel defect detection device, which can be implemented in hardware and / or software. This device can be configured in an industrial software platform integrating a CAD numerical model management system. Figure 1 As shown, the method includes:

[0032] S110. The laser radar in the visual positioning unit under the target pose that meets the detection conditions collects the three-dimensional point cloud data of the surface of the wall panel to be tested, and the visual sensor in the visual positioning unit collects the two-dimensional image data of the surface of the wall panel to be tested.

[0033] A visual positioning unit is a device integrating LiDAR, a visual sensor, and a rotating structure. It is used to determine the position and orientation of the inspection equipment relative to the panel under test, providing a foundation for subsequent data acquisition and analysis. In practical applications, a guided vehicle moves to the vicinity of the panel under test. By adjusting the telescopic linkage on the vehicle's roof, the visual positioning unit is positioned in a suitable location and orientation (i.e., target pose). This orientation must ensure that both the LiDAR and the visual sensor can clearly and completely acquire information from the panel surface. For example, in aircraft panel inspection, the guided vehicle parks next to the fuselage, the telescopic linkage extends and adjusts its angle, so that the visual positioning unit faces the area of ​​the panel to be inspected. At this time, the LiDAR emits lasers to scan the panel surface, acquiring the spatial coordinates of each point like a 3D scanner, forming 3D point cloud data that reflects the panel's three-dimensional shape. The visual sensor, like a camera, takes pictures of the panel surface, obtaining 2D image data and recording information such as the panel's color and texture.

[0034] Optionally, before acquiring the 3D point cloud data of the surface of the panel under test using the lidar in the visual positioning unit under the target pose that meets the detection conditions, the following may also be included:

[0035] The system receives a detection command for the wall panel to be tested from the control terminal, controls the guide vehicle to travel to the target detection point corresponding to the position of the wall panel to be tested based on the detection command, and completes the attitude locking of the vehicle body level and azimuth angle through the attitude sensor of the guide vehicle.

[0036] While the guided vehicle remains in a locked position, the real-time three-dimensional coordinates of the visual positioning unit on the top of the linkage telescopic body are obtained;

[0037] Based on the real-time three-dimensional coordinates and the preset detection area coordinates of the wall panel to be tested, the laser radar first measures the real-time distance between the visual positioning unit and the wall panel to be tested, and then calculates the target extension amount that the linkage telescopic body needs to adjust.

[0038] Based on the target extension amount, the control link extension body performs extension and retraction actions until the real-time distance meets the distance detection conditions, and then the extension and retraction length is locked to fix the distance.

[0039] The rotating structure of the visual positioning unit is activated to adjust the installation angle of the visual sensor until the angle between the optical axis of the lens of the visual sensor and the surface of the wall panel to be measured meets the angle detection conditions. The adjustment action is then stopped, and the visual positioning unit in the target pose that meets the detection conditions is obtained.

[0040] In the actual inspection process, the operator sends inspection commands to the system via the control terminal. These commands include information such as the panel number and location to be inspected. Upon receiving the command, the guided vehicle travels to the target inspection point corresponding to that panel according to a preset path planning algorithm (for example, in the inspection of an aircraft fuselage panel, the guided vehicle needs to stop at a specific marked position on the side of the fuselage). Upon arrival, the attitude sensors on the guided vehicle detect whether the vehicle body is level and whether the front of the vehicle is facing the panel. The onboard control system then locks the current attitude (e.g., activating the outriggers to stabilize the vehicle body and adjusting the shock absorbers to maintain levelness), providing a stable platform for subsequent inspections. For ease of understanding... Figure 2 A schematic diagram illustrating the positional relationship between a guided vehicle, a linkage telescopic body, a visual positioning unit, and a panel under test is shown, wherein... Figure 2 The components of the visual positioning unit were not described in detail.

[0041] Once the guided vehicle's attitude stabilizes, the system uses the positioning module to acquire the current 3D coordinates of the visual positioning unit in real time. Since the 3D coordinate range of the preset detection area on the wall panel is pre-stored, the system calculates the required extension or reduction length of the linkage by comparing the real-time 3D coordinates of the visual positioning unit with the coordinates of that area, combined with the real-time distance measured by the lidar. The guided vehicle's control system drives the motor of the linkage to extend or retract based on the calculated target extension / retraction amount. During this process, the lidar continuously measures the distance to the wall panel and feeds it back to the system until the distance enters the preset optimal range. At this point, the system locks the length of the linkage to ensure that the distance between the visual positioning unit and the wall panel remains stable during the detection process, preventing data ambiguity or proportional distortion due to distance changes. After the distance is fixed, the shooting angle of the visual sensor needs to be adjusted. The system activates a rotating structure to rotate the visual sensor, while simultaneously monitoring the angle between the lens optical axis and the wall panel surface using an angle sensor. When the angle meets the requirements, the rotating structure stops and locks the angle. At this point, both the position (distance) and attitude (angle) of the visual positioning unit have reached the optimal detection state.

[0042] S120. Extract fastener mounting hole feature points from the three-dimensional point cloud data, and match the fastener mounting hole feature points with the theoretical coordinates of the corresponding hole positions in the pre-stored CAD model.

[0043] The 3D point cloud data contains a vast number of points on the panel surface, requiring the extraction of key information. Fastener mounting holes are a crucial feature on the panel; their accurate positioning directly impacts fastener installation. Using specific algorithms (such as edge detection), feature points of the fastener mounting holes are identified from the 3D point cloud data, including the hole's center and key points on the edge. The coordinates of these extracted feature points are then compared with the theoretical coordinates of the corresponding mounting holes in the pre-existing CAD model. For example, the theoretical coordinates of a mounting hole in the CAD model might be (X1, Y1, Z1), while the extracted feature point coordinates from the 3D point cloud are (X2, Y2, Z2). Matching these coordinates reveals the difference.

[0044] S130. Calculate the spatial transformation matrix from the 3D point cloud to the CAD model based on the matching result, and rotate and adjust the 3D point cloud according to the spatial transformation matrix to obtain a 3D point cloud space aligned with the CAD model space.

[0045] Due to factors such as the parking position of the guided vehicle and the adjustment accuracy of the linkage telescopic body, the 3D point cloud data collected by the lidar may deviate spatially from the pre-stored CAD model. For example, the 3D point cloud may be slightly shifted to the left or rotated by a certain angle. Based on the matching results of the fastener mounting hole feature points in the previous step, a spatial transformation matrix can be calculated. This matrix can be understood as a spatial calibration formula, containing the rotation angle and displacement required to adjust the 3D point cloud to be completely aligned with the CAD model. Finally, the spatial transformation matrix is ​​used to process the 3D point cloud, making it completely coincident with the CAD model in space, forming an aligned 3D point cloud space.

[0046] Furthermore, rotating and translating the 3D point cloud according to the spatial transformation matrix to obtain a 3D point cloud space aligned with the CAD digital model space may include:

[0047] The rotation and translation components of the spatial transformation matrix are applied to the coordinates of each point in the three-dimensional point cloud to obtain the transformed three-dimensional point cloud.

[0048] Calculate the distance deviation between the feature points of each fastener mounting hole after matching and the corresponding theoretical hole positions in the CAD model;

[0049] When the distance deviation of all fastener mounting hole feature points does not exceed the deviation threshold, a three-dimensional point cloud space aligned with the CAD model space is obtained.

[0050] In practice, the spatial transformation matrix contains two key components: rotation and translation. The rotation component describes the parameters of the 3D point cloud rotating around the origin of the coordinate system, correcting angular deviations in space. The translation component describes the parameters of the 3D point cloud translating along the coordinate axes, correcting positional deviations in the X, Y, and Z directions. After these two steps, the overall pose and position of the 3D point cloud will initially approximate the spatial state of the CAD model, laying the foundation for subsequent accuracy verification. After the initial transformation, alignment accuracy needs to be verified. Since fastener mounting holes are key positioning features on the panel, their feature points are selected as verification benchmarks. For example, the center points of 10 fastener mounting holes are extracted from the transformed 3D point cloud, and their straight-line distances to the theoretical center points of the corresponding holes in the CAD model are calculated (e.g., the actual coordinates of one hole are 0.3mm from their theoretical coordinates, and another hole is 0.4mm). These distance deviations directly reflect the actual accuracy of spatial alignment. In actual testing, a reasonable deviation threshold needs to be set. If the distance deviation of all detected fastener mounting hole feature points is within the threshold range, it indicates that the 3D point cloud has achieved high-precision spatial alignment with the CAD model and can be used for subsequent 2D image mapping and defect detection. If the deviation of a certain hole exceeds the threshold, the spatial transformation matrix needs to be re-optimized and the above steps repeated until the accuracy requirements are met.

[0051] S140. Using a pre-calibrated visual sensor intrinsic parameter matrix and distortion coefficients, optical distortion correction is performed on the two-dimensional image data, and the corrected two-dimensional image data is mapped to the aligned three-dimensional point cloud space based on the spatial transformation matrix to generate texture-enhanced three-dimensional detection data.

[0052] Due to optical characteristics, the lenses of vision sensors (such as cameras) may capture distorted 2D images, for example, a straight line might be captured as a curve. This can affect the accuracy of subsequent inspections. In practical applications, the vision sensor is pre-calibrated to obtain the intrinsic parameter matrix and distortion coefficients. During inspection, these parameters are used to process the 2D image data to eliminate distortion and make the image more realistic. Using the spatial transformation matrix obtained in the previous step, the corrected 2D image data is overlaid onto a 3D point cloud space aligned with the CAD model. The resulting texture-enhanced 3D inspection data contains both 3D geometric information and 2D texture details, providing a more comprehensive reflection of the panel's condition and improving the accuracy of defect detection.

[0053] S150. Input the texture-enhanced 3D detection data into the pre-trained deep learning classification model to perform defect detection on the panel under test.

[0054] The pre-trained deep learning classification model, trained on a large amount of sample data containing various fastener defects (such as loose fasteners, missing fasteners, deformed mounting holes, etc.), has learned to identify defect features. In actual inspection, the generated texture-enhanced 3D inspection data is input into the model, which analyzes the data to determine whether fasteners on the panel under test have defects, and what kind of defects exist (such as whether mounting holes are cracked, whether fasteners are missing, etc.). This achieves automated defect detection of the panel, improving inspection efficiency and accuracy.

[0055] This invention, through the fusion of LiDAR 3D point clouds and visual sensor 2D images, combined with CAD digital models for precise spatial alignment, generates texture-enhanced 3D inspection data that simultaneously carries geometric accuracy and appearance feature information. This makes it highly suitable for fastener defect detection in large components such as aerospace fuselage panels and high-speed rail carriage sidewalls. Its core advantage lies in its ability to address the characteristics of large panels—large size, dense fastener distribution, and high precision requirements—by replacing traditional manual visual inspection or single-sensor detection with automated spatial positioning and multi-dimensional data fusion. This solves the problems of difficult positioning and low accuracy in large component inspection, and leverages deep learning models to accurately identify defects such as missing holes and fasteners. In industrial assembly line environments, this enables efficient and high-precision automated inspection, significantly improving the efficiency and reliability of large panel assembly quality control.

[0056] Example 2

[0057] Figure 3 This is a flowchart of another vision-based panel defect detection method provided in Embodiment 2 of the present invention. This embodiment is a refinement based on Embodiment 1. Specifically, as follows... Figure 3 As shown, the method includes:

[0058] S310. The laser radar in the visual positioning unit under the target pose that meets the detection conditions collects the three-dimensional point cloud data of the surface of the wall panel to be tested, and the visual sensor in the visual positioning unit collects the two-dimensional image data of the surface of the wall panel to be tested.

[0059] S320. Extract fastener mounting hole feature points from the three-dimensional point cloud data, and match the fastener mounting hole feature points with the theoretical coordinates of the corresponding hole positions in the pre-stored CAD model.

[0060] S330. Calculate the spatial transformation matrix from the 3D point cloud to the CAD model based on the matching result, and rotate and adjust the 3D point cloud according to the spatial transformation matrix to obtain a 3D point cloud space aligned with the CAD model space.

[0061] S340. Use the pre-stored intrinsic parameter matrix to perform perspective transformation correction on the two-dimensional image data to obtain an intermediate corrected image.

[0062] When a vision sensor captures an image, if the lens is not perpendicular to the surface of the panel being measured, the acquired two-dimensional image will exhibit perspective distortion. In practical applications, a pre-stored intrinsic parameter matrix contains core parameters such as the lens's focal length and principal point coordinates. These parameters describe how the lens projects three-dimensional objects onto a two-dimensional image plane. By performing perspective transformation calculations on the two-dimensional image data using the intrinsic parameter matrix, the distortion caused by tilted shooting is "corrected," for example, correcting a trapezoidal hole to a standard square, ensuring that the geometric proportions of objects in the image match reality, thus obtaining an intermediate corrected image.

[0063] S350. The intermediate corrected image is compensated for radial deformation by radial distortion coefficient to obtain a radially corrected image, and the radially corrected image is compensated for tangential deformation by tangential distortion coefficient to obtain corrected two-dimensional image data.

[0064] Even after perspective transformation correction, the optical characteristics of the lens itself can still cause two types of distortion: ① Radial distortion: For example, the edges of fastener mounting holes at the edge of the image may bulge outwards or be concave inwards, turning the originally straight hole edges into curves. The system will call the pre-stored radial distortion coefficients to compensate for the position of each pixel in the intermediate corrected image, correcting the curved edges to straight lines, thus obtaining the radially corrected image. ② Tangential distortion: If the lens is not strictly parallel to the imaging plane during installation, objects in the image may appear tilted and stretched as a whole. In this case, the radially corrected image is further processed using tangential distortion coefficients. By adjusting the positional deviation of pixels, the tangential deformation caused by assembly errors is corrected, ultimately obtaining fully corrected two-dimensional image data.

[0065] S360. Using a pre-calibrated visual sensor intrinsic parameter matrix and distortion coefficients, optical distortion correction is performed on the two-dimensional image data, and the unique projection position of each pixel of the corrected two-dimensional image data in the aligned three-dimensional point cloud space is calculated through the spatial transformation matrix.

[0066] The corrected 2D image data has eliminated optical distortion, but it is still essentially a planar image. The 3D point cloud space, however, is a 3D coordinate system aligned with the CAD model. To "superimpose" 2D texture information onto the 3D geometry, a spatial correspondence between the two needs to be established. In practical applications, the spatial transformation matrix already contains the alignment parameters (rotation, displacement) between the 3D point cloud and the CAD model. Using this matrix, the system can convert the coordinates of each pixel in the 2D image into 3D coordinates in the 3D point cloud space, i.e., a "unique projected position."

[0067] S370. Select the target 3D point in the 3D point cloud space that is closest to the unique projection position of each pixel, and assign the color information of each pixel to the matching target 3D point. Merge all the target 3D points carrying color information corresponding to each pixel to generate texture-enhanced 3D monitoring data.

[0068] A 3D point cloud is composed of discrete 3D coordinate points. Therefore, the unique projection position of a pixel may not exactly fall on a certain point cloud. In this case, it is necessary to find the point (target 3D point) in the 3D point cloud that is closest to the projection position, as the carrier of color information. By repeating this operation on all pixels in the 2D image, the final 3D point cloud space contains not only geometric shape but also corresponding color information (texture details). After merging, texture-enhanced 3D detection data is formed. For example, the 3D point cloud can show whether a bolt is loose (geometric position offset), while the color information can show whether the bolt is rusted (color change). The combination of the two can more comprehensively reflect the defect characteristics.

[0069] S380. Extract the three-dimensional coordinates of the fastener mounting hole, the two-dimensional hole edge contour in the two-dimensional image data, and the pixel grayscale features of the fastener from the texture-enhanced three-dimensional detection data.

[0070] Texture-enhanced 3D inspection data integrates 3D geometric information and 2D texture information. In actual inspection, key features related to defects need to be extracted: ① 3D coordinates of fastener mounting holes: By analyzing the 3D point cloud, the actual spatial coordinates of each mounting hole are obtained. Comparing these coordinates with the theoretical coordinates of the hole in the CAD model directly reflects the hole's positional deviation. ② 2D hole edge contours: Image edge detection algorithms are used to extract the edge lines of the mounting holes from the 2D image. For example, the contour of a normal hole should be a continuous circle. If the contour is discontinuous or irregular, there may be a processing defect; if no contour is detected, it may be an unfinished hole. ③ Pixel grayscale features of fasteners: Fasteners differ from the wall panel surface in material and reflectivity, resulting in a unique grayscale distribution in the 2D image. By analyzing the grayscale mean and gradient changes in the mounting hole area, it can be determined whether a fastener exists (if this grayscale feature is absent, it may be missing).

[0071] S390. The three-dimensional coordinates, two-dimensional hole edge contours, and pixel grayscale features of the fastener mounting hole are fused into a feature vector, which is then input into a pre-trained deep learning classification model.

[0072] A single feature cannot independently and accurately determine defects. The three types of features are quantified into numerical values ​​and then integrated into a feature vector, which is used as the input to the pre-trained model. The model learns the feature association patterns to make judgments on complex scenarios.

[0073] S3100. Identify the defect type of the panel under test through the deep learning classification model. The defect type includes missing holes, missing fasteners, and fastener misalignment.

[0074] The pre-trained model has learned the feature patterns of various defects through massive samples. During actual detection, it judges defects according to the following logic: ① Missing Hole: If the two-dimensional hole edge contour is missing in the feature vector and there is no corresponding three-dimensional coordinate, the model determines it as a missing hole. For example, if a region is designed to have 10 holes, and the contours and coordinates of 9 holes are detected, the missing hole is a missing hole. ② Missing Fastener: If the two-dimensional hole edge contour is complete and the three-dimensional coordinates are normal, but the fastener pixel grayscale features are missing, the model determines it as a missing fastener. For example, the mounting hole contour is clear, but the grayscale inside the hole is consistent with the wall panel, and there is no metallic reflective feature. ③ Fastener Offset: If the two-dimensional hole edge contour is complete and the fastener grayscale features exist, but the three-dimensional coordinate deviation exceeds the threshold, it is determined as a fastener offset. For example, the bolt is present, but its center is too far off from the hole center.

[0075] Optionally, before outputting the defect type of the fastener through the deep learning classification model, the following may also be included:

[0076] Two types of wall panel samples were collected: those in normal condition and those in defective condition. The wall panel samples in defective condition included samples with missing holes, samples with missing fasteners, and samples with offset fasteners.

[0077] For wall panel samples in normal condition, extract the three-dimensional coordinates of standard hole positions, the outline of qualified fasteners, and the grayscale features of normal assembly. For wall panel samples in defective condition, extract the three-dimensional coordinate parameters, two-dimensional outline features, and grayscale features corresponding to the defects.

[0078] The extracted features are fused into feature vectors for the corresponding state samples and labeled as normal, missing hole, missing fastener, and fastener offset type labels.

[0079] The feature vector carrying the label is input into the preset deep learning model, and the error value between the predicted state and the label is calculated using the cross-entropy loss function. Based on this error value, the network parameters are adjusted through the backpropagation algorithm.

[0080] The process of continuously iterating through feature vector input, error calculation, and network parameter adjustment continues until the deep learning model achieves a preset threshold in its accuracy in recognizing normal states and three types of defects, thus obtaining a deep learning classification model.

[0081] Panel samples refer to panel instances used for model training, including physical or data samples in normal (defect-free) and defective (with specific defects) states. Standard hole position 3D coordinates are the 3D coordinate values ​​of fastener mounting holes in normal state samples that meet design requirements, serving as a benchmark for defect judgment. Qualified fastener contours are the standard 2D edge shapes of fasteners and mounting holes in normal state samples. Normal assembly grayscale features are the standard grayscale distribution of fasteners and panel surfaces in 2D images in normal state samples. In practical applications, the first step in model training is to collect sufficient sample data. These samples need to cover panel sizes and materials, as well as acquisition scenarios under different lighting and angles, ensuring the model's versatility. Samples can be physical objects or 3D point cloud and 2D image data collected through previous inspection processes.

[0082] After collecting samples, features need to be extracted. Specifically, for normal samples, extract the standard 3D coordinates of each mounting hole, the regular contour of the fastener and the hole, and the normal grayscale features of the fastener. For samples with missing holes, extract the coordinates of the hole-free area, the edge-free contour features, and the grayscale features of the fastener-free area. For samples with missing fasteners, extract the normal 3D coordinates of the mounting hole, the complete hole contour, and the grayscale features of the fastener-free area. For samples with offset fasteners, extract the normal 3D coordinates of the mounting hole, the complete hole contour, the offset coordinates of the fastener, and the normal grayscale features.

[0083] A single feature is insufficient for the model to understand what a defect is. Multiple features from the same sample need to be integrated into a single feature vector. For example, the feature vector of a normal sample might be [0 (coordinate deviation), 1 (contour regularity), 1 (normal grayscale)], labeled "normal"; the feature vector of a sample with a missing hole might be [1 (no hole coordinates), 0 (no contour), 0 (no grayscale)], labeled "missing hole". The label tells the model "what the actual state corresponding to the current feature vector is", which is used to judge the correctness of the prediction results during model training.

[0084] The pre-set deep learning model is an untrained initial network structure with feature learning and classification capabilities, but its parameters are not optimized. During actual training, labeled feature vectors are input into the model, which outputs a prediction. The error between the prediction and the true label is calculated using the cross-entropy loss function. Then, the backpropagation algorithm is used to adjust the parameters layer by layer from output to input, making the model more likely to predict correctly when encountering similar features again. Model training is an iterative optimization process. Each iteration uses different sample feature vectors, repeating the process of "input → prediction → error calculation → parameter tuning". When the accuracy reaches a threshold, training stops, resulting in a "pre-trained deep learning classification model" that can be used for actual detection.

[0085] This invention, through the construction of a complete technology chain—"precise image correction—deep data fusion—targeted feature extraction and model training"—forms a highly efficient solution adapted to large panel inspection scenarios. Its core value lies in ensuring the authenticity of texture information through graded processing of image distortion, achieving precise matching of geometric and appearance data by combining spatial alignment of 3D point clouds and CAD models, and fundamentally solving the misjudgment problems caused by data distortion, limited information, or poor recognition generalization in traditional inspection by relying on multi-dimensional feature fusion and model training focused on specific defects. Ultimately, it achieves high-precision, automated identification of key defects such as missing holes, missing fasteners, and misalignment, providing reliable and efficient technical support for the quality inspection of large panels.

[0086] Example 3

[0087] Figure 4 This is a schematic diagram of a wall panel defect detection device based on vision positioning provided in Embodiment 3 of the present invention. Figure 4 As shown, the device includes:

[0088] The visual acquisition module 410 is used to acquire three-dimensional point cloud data of the surface of the wall panel under test by the lidar in the visual positioning unit under the target pose that meets the detection conditions, and to acquire two-dimensional image data of the surface of the wall panel under test by the visual sensor in the visual positioning unit.

[0089] The feature point matching module 420 is used to extract fastener mounting hole feature points from the three-dimensional point cloud data and match the fastener mounting hole feature points with the theoretical coordinates of the corresponding hole positions in the pre-stored CAD model.

[0090] The spatial alignment module 430 is used to calculate the spatial transformation matrix from the 3D point cloud to the CAD model based on the matching result, and to rotate and adjust the 3D point cloud according to the spatial transformation matrix to obtain a 3D point cloud space aligned with the CAD model space.

[0091] The texture enhancement module 440 is used to perform optical distortion correction on the two-dimensional image data using a pre-calibrated visual sensor intrinsic parameter matrix and distortion coefficients, and to map the corrected two-dimensional image data to an aligned three-dimensional point cloud space based on the spatial transformation matrix to generate texture-enhanced three-dimensional detection data.

[0092] The defect detection module 450 is used to input texture-enhanced 3D detection data into a pre-trained deep learning classification model to perform defect detection on the panel under test.

[0093] This invention, through the fusion of LiDAR 3D point clouds and visual sensor 2D images, combined with CAD digital models for precise spatial alignment, generates texture-enhanced 3D inspection data that simultaneously carries geometric accuracy and appearance feature information. This data is suitable for fastener defect detection in large components such as aerospace fuselage panels and high-speed rail carriage sidewalls. Its core advantage lies in its ability to address the characteristics of large panels—large size, dense fastener distribution, and high precision requirements—by replacing traditional manual visual inspection or single-sensor detection with automated spatial positioning and multi-dimensional data fusion. This solves the problems of difficult positioning and low accuracy in large component inspection, and leverages deep learning models to accurately identify defects such as missing holes and fasteners. In industrial assembly line environments, this enables efficient and high-precision automated inspection, significantly improving the efficiency and reliability of large panel assembly quality control.

[0094] Optionally, based on the above embodiments, it may include: a pose pre-adjustment unit, used to receive a detection command for the wall panel to be tested issued by the control terminal before collecting three-dimensional point cloud data of the surface of the wall panel to be tested by the lidar in the visual positioning unit under the target pose that meets the detection conditions, control the guide vehicle to drive to the target detection point corresponding to the position of the wall panel to be tested based on the detection command, and complete the attitude locking of the vehicle body level and azimuth angle through the attitude sensor of the guide vehicle;

[0095] While the guided vehicle remains in a locked position, the real-time three-dimensional coordinates of the visual positioning unit on the top of the linkage telescopic body are obtained;

[0096] Based on the real-time three-dimensional coordinates and the preset detection area coordinates of the wall panel to be tested, the laser radar first measures the real-time distance between the visual positioning unit and the wall panel to be tested, and then calculates the target extension amount that the linkage telescopic body needs to adjust.

[0097] Based on the target extension amount, the control link extension body performs extension and retraction actions until the real-time distance meets the distance detection conditions, and then the extension and retraction length is locked to fix the distance.

[0098] The rotating structure of the visual positioning unit is activated to adjust the installation angle of the visual sensor until the angle between the optical axis of the lens of the visual sensor and the surface of the wall panel to be measured meets the angle detection conditions. The adjustment action is then stopped, and the visual positioning unit in the target pose that meets the detection conditions is obtained.

[0099] Optionally, based on the above embodiments, the spatial alignment module 430 may include:

[0100] The point cloud transformation unit is used to apply the rotation and translation components of the spatial transformation matrix to the coordinates of each point in the three-dimensional point cloud to obtain the transformed three-dimensional point cloud.

[0101] Deviation calculation unit: The user calculates the distance deviation between the feature points of each fastener mounting hole after matching and the corresponding theoretical hole positions in the CAD model;

[0102] The spatial alignment condition determination unit is used to determine the 3D point cloud space aligned with the CAD model space when the distance deviation of all fastener mounting hole feature points does not exceed the deviation threshold.

[0103] Optionally, based on the above embodiments, the texture enhancement module 440 may include:

[0104] A perspective transformation unit is used to perform perspective transformation correction on the two-dimensional image data using a pre-stored intrinsic parameter matrix to obtain an intermediate corrected image;

[0105] The distortion compensation unit is used to compensate for radial deformation of the intermediate corrected image by using a radial distortion coefficient to obtain a radially corrected image, and to compensate for tangential deformation of the radially corrected image by using a tangential distortion coefficient to obtain corrected two-dimensional image data.

[0106] Optionally, based on the above embodiments, the texture enhancement module 440 may further include:

[0107] The projection position calculation unit is used to calculate the unique projection position of each pixel of the corrected two-dimensional image data in the aligned three-dimensional point cloud space through the spatial transformation matrix.

[0108] The 3D point color assignment unit is used to select the target 3D point in the 3D point cloud space that is closest to the unique projection position of each pixel, and assign the color information of each pixel to the matching target 3D point. All target 3D points carrying color information corresponding to each pixel are merged to generate texture-enhanced 3D monitoring data.

[0109] Optionally, based on the above embodiments, the defect detection module 450 may include:

[0110] The data extraction unit is used to extract the three-dimensional coordinates of the fastener mounting hole, the two-dimensional hole edge contour in the two-dimensional image data, and the pixel grayscale features of the fastener from the texture-enhanced three-dimensional detection data.

[0111] The feature vector input unit is used to fuse the three-dimensional coordinates, two-dimensional hole edge contours and pixel grayscale features of the fastener mounting hole into a feature vector, and input it into a pre-trained deep learning classification model.

[0112] The defect type identification unit is used to identify the defect type of the panel under test through the deep learning classification model. The defect types include missing holes, missing fasteners, and fastener misalignment.

[0113] Optionally, based on the above embodiments, it may also include: a deep learning classification model training unit, used to collect two types of wall panel samples, namely, normal state and defect state, before outputting the defect type of fastener through the deep learning classification model, wherein the wall panel samples of the defect state include samples of missing holes, samples of missing fasteners and samples of fastener offset.

[0114] For wall panel samples in normal condition, extract the three-dimensional coordinates of standard hole positions, the outline of qualified fasteners, and the grayscale features of normal assembly. For wall panel samples in defective condition, extract the three-dimensional coordinate parameters, two-dimensional outline features, and grayscale features corresponding to the defects.

[0115] The extracted features are fused into feature vectors for the corresponding state samples and labeled as normal, missing hole, missing fastener, and fastener offset type labels.

[0116] The feature vector carrying the label is input into the preset deep learning model, and the error value between the predicted state and the label is calculated using the cross-entropy loss function. Based on this error value, the network parameters are adjusted through the backpropagation algorithm.

[0117] The process of continuously iterating through feature vector input, error calculation, and network parameter adjustment continues until the deep learning model achieves a preset threshold in its accuracy in recognizing normal states and three types of defects, thus obtaining a deep learning classification model.

[0118] The visual positioning-based wall panel defect detection device provided in this embodiment of the invention can execute the visual positioning-based wall panel defect detection method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method.

[0119] Example 4

[0120] Figure 5 A schematic diagram of an electronic device 10 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0121] like Figure 5As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0122] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0123] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as a vision-based panel defect detection method.

[0124] That is: the lidar in the visual positioning unit under the target pose that meets the detection conditions collects the three-dimensional point cloud data of the surface of the wall panel to be tested, and the visual sensor in the visual positioning unit collects the two-dimensional image data of the surface of the wall panel to be tested.

[0125] Extract fastener mounting hole feature points from the three-dimensional point cloud data, and match the fastener mounting hole feature points with the theoretical coordinates of the corresponding hole positions in the pre-stored CAD model;

[0126] Based on the matching results, the spatial transformation matrix from the 3D point cloud to the CAD model is calculated, and the 3D point cloud is rotated and translated according to the spatial transformation matrix to obtain a 3D point cloud space aligned with the CAD model space.

[0127] Using a pre-calibrated visual sensor intrinsic parameter matrix and distortion coefficients, optical distortion correction is performed on the two-dimensional image data, and the corrected two-dimensional image data is mapped to an aligned three-dimensional point cloud space based on the spatial transformation matrix to generate texture-enhanced three-dimensional detection data.

[0128] The texture-enhanced 3D detection data is input into a pre-trained deep learning classification model to perform defect detection on the panel under test.

[0129] In some embodiments, a vision-based panel defect detection method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the vision-based panel defect detection method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform a vision-based panel defect detection method by any other suitable means (e.g., by means of firmware).

[0130] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0131] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0132] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0133] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0134] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0135] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0136] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0137] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A vision-based method for detecting panel defects, applied in a scenario where fastener defects are detected in the panel under test by controlling a telescopic linkage on the roof of a guided vehicle and a vision positioning unit on top of the telescopic linkage, characterized in that... The method includes: The three-dimensional point cloud data of the surface of the wall panel under test is collected by the lidar in the visual positioning unit under the target pose that meets the detection conditions, and the two-dimensional image data of the surface of the wall panel under test is collected by the visual sensor in the visual positioning unit. Extract fastener mounting hole feature points from the three-dimensional point cloud data, and match the fastener mounting hole feature points with the theoretical coordinates of the corresponding hole positions in the pre-stored CAD model; Based on the matching results, the spatial transformation matrix from the 3D point cloud to the CAD model is calculated, and the 3D point cloud is rotated and translated according to the spatial transformation matrix to obtain a 3D point cloud space aligned with the CAD model space. Using a pre-calibrated visual sensor intrinsic parameter matrix and distortion coefficients, optical distortion correction is performed on the two-dimensional image data, and the corrected two-dimensional image data is mapped to an aligned three-dimensional point cloud space based on the spatial transformation matrix to generate texture-enhanced three-dimensional detection data. Texture-enhanced 3D detection data is input into a pre-trained deep learning classification model to perform defect detection on the panel under test. Before acquiring the 3D point cloud data of the surface of the panel under test using the lidar in the visual positioning unit under the target pose that meets the detection conditions, the following steps are also included: The system receives a detection command for the wall panel to be tested from the control terminal, controls the guide vehicle to travel to the target detection point corresponding to the position of the wall panel to be tested based on the detection command, and completes the attitude locking of the vehicle body level and azimuth angle through the attitude sensor of the guide vehicle. While the guided vehicle remains in a locked position, the real-time three-dimensional coordinates of the visual positioning unit on the top of the linkage telescopic body are obtained; Based on the real-time three-dimensional coordinates and the preset detection area coordinates of the wall panel to be tested, the laser radar first measures the real-time distance between the visual positioning unit and the wall panel to be tested, and then calculates the target extension amount that the linkage telescopic body needs to adjust. Based on the target extension amount, the control link extension body performs extension and retraction actions until the real-time distance meets the distance detection conditions, and then the extension and retraction length is locked to fix the distance. The rotating structure of the visual positioning unit is activated to adjust the installation angle of the visual sensor until the angle between the optical axis of the lens of the visual sensor and the surface of the wall panel to be measured meets the angle detection conditions. The adjustment action is then stopped, and the visual positioning unit in the target pose that meets the detection conditions is obtained.

2. The method according to claim 1, characterized in that, The 3D point cloud is rotated and translated according to the spatial transformation matrix to obtain a 3D point cloud space aligned with the CAD digital model space, including: The rotation and translation components of the spatial transformation matrix are applied to the coordinates of each point in the three-dimensional point cloud to obtain the transformed three-dimensional point cloud. Calculate the distance deviation between the feature points of each fastener mounting hole after matching and the corresponding theoretical hole positions in the CAD model; When the distance deviation of all fastener mounting hole feature points does not exceed the deviation threshold, a three-dimensional point cloud space aligned with the CAD model space is obtained.

3. The method according to claim 1, characterized in that, Optical distortion correction is performed on the two-dimensional image data using a pre-calibrated intrinsic parameter matrix and distortion coefficients of the visual sensor, including: The two-dimensional image data is subjected to perspective transformation correction using a pre-stored intrinsic parameter matrix to obtain an intermediate corrected image; The intermediate corrected image is compensated for radial deformation by radial distortion coefficient to obtain a radially corrected image, and the radially corrected image is compensated for tangential deformation by tangential distortion coefficient to obtain corrected two-dimensional image data.

4. The method according to claim 3, characterized in that, The corrected 2D image data is mapped to the aligned 3D point cloud space based on the spatial transformation matrix to generate texture-enhanced 3D detection data, including: The spatial transformation matrix is ​​used to calculate the unique projection position of each pixel in the aligned 3D point cloud space of the corrected 2D image data. Select the target 3D point in the 3D point cloud space that is closest to the unique projection position of each pixel, and assign the color information of each pixel to the matching target 3D point. Then merge all the target 3D points carrying color information corresponding to each pixel to generate texture-enhanced 3D monitoring data.

5. The method according to claim 1, characterized in that, Texture-enhanced 3D detection data is input into a pre-trained deep learning classification model to perform defect detection on the panel under test, including: The three-dimensional coordinates of the fastener mounting hole, the two-dimensional hole edge contour in the two-dimensional image data, and the pixel grayscale features of the fastener are extracted from the texture-enhanced three-dimensional detection data. The three-dimensional coordinates, two-dimensional hole edge contours, and pixel grayscale features of the fastener mounting hole are fused into a feature vector, which is then input into a pre-trained deep learning classification model. The deep learning classification model identifies the defect types of the panel under test, including missing holes, missing fasteners, and fastener misalignment.

6. The method according to claim 4, characterized in that, Before outputting the defect type of the fastener through the deep learning classification model, the method further includes: Two types of wall panel samples were collected: those in normal condition and those in defective condition. The wall panel samples in defective condition included samples with missing holes, samples with missing fasteners, and samples with offset fasteners. For wall panel samples in normal condition, extract the three-dimensional coordinates of standard hole positions, the outline of qualified fasteners, and the grayscale features of normal assembly. For wall panel samples in defective condition, extract the three-dimensional coordinate parameters, two-dimensional outline features, and grayscale features corresponding to the defects. The extracted features are fused into feature vectors for the corresponding state samples and labeled as normal, missing hole, missing fastener, and fastener offset type labels. The feature vector carrying the label is input into the preset deep learning model, and the error value between the predicted state and the label is calculated using the cross-entropy loss function. Based on this error value, the network parameters are adjusted through the backpropagation algorithm. The process of continuously iterating through feature vector input, error calculation, and network parameter adjustment continues until the deep learning model achieves a preset threshold in its accuracy in recognizing normal states and three types of defects, thus obtaining a deep learning classification model.

7. A vision-based panel defect detection device, applied in a scenario where fastener defects are detected in the panel under test by controlling a telescopic linkage on the roof of a guided vehicle and a vision positioning unit on top of the telescopic linkage, characterized in that... The device includes: The visual acquisition module is used to acquire three-dimensional point cloud data of the surface of the wall panel under test by the lidar in the visual positioning unit under the target pose that meets the detection conditions, and to acquire two-dimensional image data of the surface of the wall panel under test by the visual sensor in the visual positioning unit. The feature point matching module is used to extract fastener mounting hole feature points from the three-dimensional point cloud data and match the fastener mounting hole feature points with the theoretical coordinates of the corresponding hole positions in the pre-stored CAD model. The spatial alignment module is used to calculate the spatial transformation matrix from the 3D point cloud to the CAD model based on the matching result, and to rotate and translate the 3D point cloud according to the spatial transformation matrix to obtain a 3D point cloud space aligned with the CAD model space. The texture enhancement module is used to perform optical distortion correction on the two-dimensional image data using a pre-calibrated visual sensor intrinsic parameter matrix and distortion coefficients, and to map the corrected two-dimensional image data to an aligned three-dimensional point cloud space based on the spatial transformation matrix to generate texture-enhanced three-dimensional detection data. The defect detection module is used to input texture-enhanced 3D detection data into a pre-trained deep learning classification model to perform defect detection on the panel under test. It also includes: a pose pre-adjustment unit, which is used to receive a detection command for the wall panel to be tested issued by the control terminal before collecting three-dimensional point cloud data of the surface of the wall panel to be tested by the lidar in the visual positioning unit under the target pose that meets the detection conditions, and control the guide vehicle to drive to the target detection point corresponding to the position of the wall panel to be tested based on the detection command, and complete the attitude locking of the vehicle body level and azimuth angle through the attitude sensor of the guide vehicle. While the guided vehicle remains in a locked position, the real-time three-dimensional coordinates of the visual positioning unit on the top of the linkage telescopic body are obtained; Based on the real-time three-dimensional coordinates and the preset detection area coordinates of the wall panel to be tested, the laser radar first measures the real-time distance between the visual positioning unit and the wall panel to be tested, and then calculates the target extension amount that the linkage telescopic body needs to adjust. Based on the target extension amount, the control link extension body performs extension and retraction actions until the real-time distance meets the distance detection conditions, and then the extension and retraction length is locked to fix the distance. The rotating structure of the visual positioning unit is activated to adjust the installation angle of the visual sensor until the angle between the optical axis of the lens of the visual sensor and the surface of the wall panel to be measured meets the angle detection conditions. The adjustment action is then stopped, and the visual positioning unit in the target pose that meets the detection conditions is obtained.

8. An electronic device, characterized in that, The electronic device includes: At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform a visual positioning-based panel defect detection method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute and implement the visual positioning-based panel defect detection method according to any one of claims 1-6.

Images