Robot automatic fiber placement defect online detection method and system based on 3D vision

Abstract

This invention provides a method and system for online automatic wire layup defect detection using a robot based on 3D vision. The method includes: constructing and calibrating a 3D vision imaging system based on a Sham lens and a line laser; synchronously moving with a wire layup robotic arm to acquire two-dimensional images of the layup and reconstruct three-dimensional point cloud data; fusing and analyzing the two-dimensional images and three-dimensional point cloud data; using a lightweight deep learning network to perform real-time defect detection on the two-dimensional images; simultaneously locating and segmenting defect regions based on point cloud normal vectors and curvature information; fusing two-dimensional texture features and three-dimensional geometric features through a collaborative representation model to achieve accurate defect identification and location; transforming the defect information in a unified coordinate system; and controlling multiple laser projection devices to project onto the layup surface in real-time using a segmented relay method. This invention achieves high-quality imaging under low reflectivity, large depth of field, and short-range conditions by fusing Sham lens imaging and line laser scanning; achieves a recognition rate of over 95% in high-speed wire layup by combining dual-modal fusion and lightweight few-sample learning; and utilizes multi-device collaborative calibration and relay projection to achieve real-time defect feedback.

Description

Technical Field

[0001] This invention relates to automated filament placement inspection technology, and in particular to a robotic automated filament placement defect online detection method and automated measurement system based on 3D vision. Background Technology

[0002] In recent years, composite materials have shown significant performance improvements compared to traditional materials in industries such as aerospace, shipbuilding, and wind power generation, leading to their increasingly widespread application. To improve the manufacturing efficiency of composite materials, Automatic Fiber Placement (AFP) technology has emerged. AFP technology significantly reduces the manufacturing cost and time of composite material parts. However, due to the complexity of the AFP process, some defects inevitably occur during manufacturing, such as gaps, overlaps, bubbles, wrinkles, and twisting.

[0003] Studies have shown that defects in fiber layup, such as gaps, overlaps, and wrinkles, weaken the strength and stiffness of composite laminates, leading to deterioration of load conditions and reduced safety in material applications. Traditional manual inspection methods rely on the experience and skills of operators and involve long downtime for inspection, reducing actual production efficiency. Therefore, developing online detection technology for AFP layup defects is of great significance: 1) ensuring the layup quality of composite materials and significantly improving material safety; 2) enabling real-time automatic defect detection of fiber layup while maintaining production efficiency, thus reducing manufacturing costs.

[0004] AFP defect detection technology is very active both domestically and internationally. Existing mature technologies can achieve single-modal defect information acquisition and processing, but cannot adapt to complex defect detection scenarios. Specific technologies include: 1) Visible light cameras effectively acquire the geometric shape, grayscale data, and texture structure of the inspection image to achieve optical feature defect detection on the composite material surface; 2) Laser profilometers effectively acquire the depth, flatness, and cross-sectional contour data between points on the composite material surface by reconstructing a three-dimensional laser point cloud to achieve three-dimensional morphological defect detection; 3) Thermal imaging cameras can acquire the heat capacity or thermal conductivity of the composite material surface / subsurface to determine defects such as porosity, delamination, and inclusions in the depth direction of the composite material; 4) Eddy current sensors acquire the impedance data of the composite material surface to achieve defects such as fiber breakage, wrinkles, and delamination depth. The application of the above detection methods can effectively shorten the manual inspection time and improve the efficiency of defect detection. In engineering sites, there are often more than one type of defect. It is obviously impossible to meet the detection needs by using only one modality of data. Therefore, it is urgent to effectively integrate multimodal detection data to form a comprehensive evaluation of detected defects.

[0005] In recent years, with the continuous advancement of AFP defect detection technology, complex defect data can be preliminarily detected through multimodal information fusion. However, existing multimodal information fusion methods for defect detection have limited defect types and low accuracy. In 2021, the team led by Ke Yinglin at Zhejiang University proposed a detection method based on infrared and visible light image fusion, which can detect defects such as bubbles, missing parts, foreign objects, twists, and wrinkles, achieving an average precision of 0.825, an average recall of 0.880, and a detection speed of 0.9 m / s. Following this, in 2022, Tang from Zhejiang University proposed an online AFP detection method based on laser profilometer point cloud data, a PointNet point cloud segmentation neural network, and a defect classification strategy. This method can detect defects such as gaps, wrinkles, overlaps, foreign objects, and missing parts online, achieving an overall precision of 0.844 and a recall of 0.771. However, it performs poorly in detecting foreign objects, with a precision of only 0.25 and a recall of 0.5. It is clear that existing methods for defect detection using multimodal information fusion have limited defect detection capabilities and low accuracy, making it difficult to meet the high-precision inspection requirements of engineering sites. How to effectively integrate multimodal image data to achieve efficient defect detection in composite materials, completely replacing the currently indispensable manual inspection process and meeting the high safety requirements of composite material structures, is a problem that must be solved before it can be applied in practice.

[0006] AFP defect detection technology suffers from insufficient real-time feedback, hindering real-time defect interaction. Most current AFP defect detection technologies employ offline detection methods (i.e., after carbon fiber filament placement, the entire detection system is moved to the detection station), processing and analyzing detection data in stages, interrupting continuous production processes and compromising production efficiency. Recent research has revealed a method proposed in 2021 by Tang of Zhejiang University: an online defect detection method for Automatic Fiber Placement (AFP) based on laser contour point cloud data and corresponding algorithms. This method can process up to 200 laser lines per second (approximately 160,000 points), achieving an overall defect type identification accuracy of around 78%. This demonstrates that while high-frame-rate hardware sensors combined with high-precision defect detection algorithms can shorten detection time to some extent, they still cannot meet the demands of real-time detection and real-time defect interaction. To achieve real-time detection, the key technology lies in the deep integration of the AFP defect detection system with its connected CNC system, ensuring all systems operate within the same coordinate system. This guarantees that detection results (such as "gap found at X coordinate") can be immediately understood and responded to by the CNC system.

[0007] Based on existing AFP defect detection technology, its shortcomings can be identified in the following three main aspects: 1) In short-range measurement scenarios, there is a lack of high-quality multimodal information acquisition methods: First, due to the light absorption characteristics of composite materials, the detection images acquired by 2D cameras have low signal-to-noise ratios and blurred image features; line lasers have weak signals on low-reflectivity surfaces, which may result in missing point clouds. Second, the space around the AFP automatic fiber placement head is compact, and the sensor must be integrated into a very limited space to perform "in-situ" detection close to the placement point; furthermore, the distance between the AFP automatic fiber placement head and the component surface changes dynamically with the placement position. To ensure clear imaging within the measurement area, sufficient light intake and a large depth of field are required within the short-range measurement range. Finally, the geometric characteristics of composite materials contained in single-modal data are limited, which cannot meet the diverse defect requirements of actual engineering sites, easily leading to false detections and missed detections. Among them, 2D visual images can provide high-resolution surface texture features, but cannot provide 3D data; line laser projection for point cloud reconstruction can provide accurate 3D topography and height data, but cannot provide material surface texture features; therefore, there is an urgent need to design a high-quality multimodal information acquisition method for low reflectivity, large depth of field, and short ranging scenarios.

[0008] 2) Existing inspection technologies have limited defect detection capabilities, low success rates, and inaccurate defect localization: AFP defect detection technology can detect and classify 3-6 types of common defects (gap, overlap, missing, bridging, foreign matter, torsion, wrinkles), with a defect recognition rate of over 85%. However, current defect detection technologies cannot meet the high-precision and high-efficiency defect detection needs of industrial sites. To address complex defect detection, a rapid localization and identification technology based on multimodal fiber layup defects is required.

[0009] In detail, relying solely on two-dimensional or three-dimensional modal data in AFP defect detection has significant limitations. In 2D images, narrow and elongated defects such as gaps and overlaps often exhibit weak edge gradients and low texture contrast due to their extremely small width, resulting in indistinct image features, low recognition accuracy, and difficulty in achieving stable detection. On the other hand, using only 3D laser point cloud recognition technology is limited by scanning frequency and the high-speed movement of the filament placement head, leading to insufficient point cloud density in the defect area. This makes it difficult to fully capture complex geometric shapes such as wrinkles and twists. Furthermore, the diverse types of defects with relatively simple geometric features easily result in missed and false detections. Therefore, to overcome the shortcomings of single-modality data in terms of information completeness and feature representation, integrating the texture details of 2D images with the geometric information of 3D laser point clouds to form a multi-modal collaborative analysis mechanism has become a necessary approach to improve the accuracy and robustness of defect recognition.

[0010] In addition, existing methods struggle to achieve real-time defect detection in engineering settings with limited computing power due to the large data volume and high computational resource requirements of high-resolution 2D images. To balance efficiency and accuracy, lightweight networks are often introduced; however, in fields such as automatic tile detection, the scarcity of labeled data restricts the training effectiveness of supervised learning models. Furthermore, excessive lightweighting weakens feature extraction capabilities, while excessive pursuit of accuracy sacrifices speed, making the model difficult to adapt to different hardware. Therefore, there is an urgent need for a lightweight, real-time-efficient, and highly transferable deep learning network based on small sample sizes to achieve efficient and high-precision defect detection with limited resources.

[0011] 3) Insufficient Real-Time Feedback for Defect Detection: First, most existing AFP defect detection systems adopt an offline mode of "acquisition first, processing later," meaning defect information can only be analyzed after a layup or even the entire component has been laid. This significant lag in process feedback prevents real-time defect correction. Second, most existing AFP defect detection systems can only determine the number of defects, failing to achieve pixel-level precise positioning. This fuzzy positioning method makes subsequent processes unable to be precisely adjusted, and layups cannot be repaired completely. To achieve real-time detection, the key technology lies in the deep integration of the AFP defect detection system with its connected CNC system. Specifically, within a layup cycle, the 3D imaging system moves synchronously with the wire-laying robotic arm. Considering that the layup cycle may exceed the projection range of some projection devices, projection devices are set at fixed intervals, and multiple laser projection systems dynamically project the real-time position coordinates of defects through a segmented relay method. The entire process requires the coordinate systems of related systems to be unified through calibration to ensure that the detection results (such as "gap found at X coordinate") can be immediately understood and responded to by the CNC system.

[0012] It should be noted that the information disclosed in the background section above is only for understanding the background of this application, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0013] The main objective of this invention is to overcome the deficiencies in the above-mentioned background technology and provide a method and system for online detection of automatic wire laying defects in robots based on 3D vision.

[0014] To achieve the above objectives, the present invention adopts the following technical solution: A 3D vision-based method for automated online detection of defects in wire laying by a robot includes the following steps: S1. Construct and calibrate a 3D vision imaging system based on a Sham lens and a line laser, so that it moves synchronously with the wire laying robot arm, acquires two-dimensional images of the layup containing laser light stripes online, and reconstructs the corresponding three-dimensional point cloud data based on the plane equation of the line laser blade. S2. Perform fusion analysis on the two-dimensional image and three-dimensional point cloud data: Use a lightweight deep learning network based on few-sample training to perform real-time defect detection and preliminary localization of the two-dimensional image, and at the same time, locate and segment the defect area based on the normal vector and curvature information of the three-dimensional point cloud. Then, fuse two-dimensional texture features and three-dimensional geometric features through a collaborative representation model to achieve accurate identification and localization of wire laying defects. S3. The identified and located defect information is transformed into coordinates in the robot arm coordinate system, 3D vision imaging system coordinate system and multiple laser projection device coordinate systems that have been calibrated by target points. The multiple laser projection devices are then controlled to project the defect location coordinates onto the ply surface in real time in a segmented relay manner for feedback.

[0015] A 3D vision-based robotic automated wire-laying defect online detection system includes: Robotic arms are used to perform carrying and movement during the filament laying process; A 3D vision imaging subsystem, fixed to the end of the robotic arm, includes a camera with a Sham lens, a line laser, and a strobe light source, used to acquire two-dimensional images of the layup online and simultaneously reconstruct three-dimensional point cloud data; The image processing and defect analysis subsystem is communicatively connected to the 3D vision imaging subsystem and is used to perform fusion analysis of the acquired two-dimensional images and three-dimensional point cloud data using the method described above, so as to realize defect identification and localization. The multi-laser projection subsystem includes multiple laser projection devices spaced apart along the ply direction, used to receive defect coordinate information output by the image processing and defect analysis subsystem, and project the defect position onto the ply surface in real time under a unified coordinate system. The robotic arm, 3D vision imaging subsystem, and multi-laser projection subsystem are calibrated and unified under the same workpiece coordinate system.

[0016] The present invention has the following beneficial effects: This invention addresses the core bottlenecks of existing automatic wire placement defect detection technologies under complex working conditions, such as poor quality of multimodal information acquisition, inability of single-modal geometric features to cope with diverse defects, and severe lag in detection feedback. It proposes a systematic solution that brings significant benefits at the level of basic technical solutions.

[0017] First, at the data acquisition source, this invention constructs a high-precision 3D vision imaging system that integrates Sham lens imaging and monocular laser scanning. Addressing demanding conditions such as the high light absorption of carbon fiber composite surfaces, the compact internal space of the automated fiber placement head, and short detection distances, this system effectively enhances the signal-to-noise ratio of low-reflectivity surfaces through a reflective structural arrangement and the coordinated triggering of a stroboscopic light source. Simultaneously, the introduction of a Sham lens cleverly resolves the optical contradiction between short measurement distance and large depth of field. This design enables the 3D vision imaging system to stably acquire high-quality two-dimensional texture images and three-dimensional point cloud data within a compact space with relatively low laser power, laying a reliable data foundation for subsequent accurate defect detection.

[0018] In the core data processing stage, the 2D-3D dual-modal fusion technology proposed in this invention effectively overcomes the limitations of insufficient feature representation in single-modal data. On the one hand, for two-dimensional images, a lightweight convolutional neural network combined with an attention mechanism and YOLO detection architecture is used to achieve real-time classification and localization of surface defects such as missing wires and bubbles. On the other hand, for three-dimensional point clouds, a defect localization module is constructed using point cloud normal vectors and curvature information, and clustering algorithms are used to accurately segment defect regions with significant geometric features such as gaps and overlaps. Furthermore, through fusion optimization strategies such as collaborative representation models and Kalman filtering, two-dimensional texture details and three-dimensional macroscopic geometric information are deeply combined, achieving full-scale defect perception from microscopic texture to macroscopic geometry. Based on this, this invention also proposes a small-sample lightweight training strategy based on model pruning, knowledge distillation, and transfer learning. Under limited computing power, only about 500 real samples are needed to achieve a defect recognition rate of over 95%, and it supports high-speed placement detection at 80m / min, significantly improving the accuracy and efficiency of defect detection.

[0019] At the system integration and real-time feedback level, this invention achieves a high degree of synergy and integration between the robotic arm, the 3D imaging system, and multiple laser projection devices. Through hand-eye calibration and target-point-based coordinate unification technology, the robotic arm, imaging system, and projection devices are precisely unified under the same workpiece coordinate system, ensuring that the detection results can be instantly converted and understood. Based on this, this invention designs a laser projection relay mechanism based on regional segmentation and overlapping coverage: each laser projection device is arranged at a fixed interval, with their projection areas partially overlapping; the system dynamically allocates the projection device corresponding to the current layup segment according to the real-time position of the 3D imaging system. For defects within the overlapping area, the shortest distance method is used to automatically determine the projection assignment. This design enables the detection system to continuously, seamlessly, and accurately project the defect coordinates onto the layup surface in the form of visible light spots throughout the entire long-distance layup process, completely changing the traditional offline mode of "collect first, process later," and achieving true online detection and real-time feedback.

[0020] In summary, this invention forms a complete technology chain from high-quality data acquisition and multimodal feature fusion to system collaboration and real-time feedback. It not only solves the imaging problems of existing technologies in low reflectivity, large depth of field, and short distance scenarios, but also effectively overcomes the limitations of insufficient single-modal data features and serious lag in detection feedback. While ensuring high accuracy and high recognition rate, it realizes real-time defect detection and visual positioning in high-speed fiber placement, significantly improving the detection efficiency and quality control level of the automatic fiber placement process for composite materials.

[0021] Other beneficial effects of the embodiments of the present invention will be further described below. Attached Figure Description

[0022] Figure 1 This is a schematic diagram of an automated online detection system for wire laying defects, which integrates a robotic arm, a 3D imaging system, and a projection device, according to an embodiment of the present invention.

[0023] Figure 2 This is a technical roadmap for the automatic online detection method for wire placement defects according to an embodiment of the present invention.

[0024] Figure 3 This is a three-dimensional model diagram of the automatic online detection system for wire placement defects according to an embodiment of the present invention.

[0025] Figure 4 This is a schematic diagram illustrating the coarse positioning of a laser beam using the improved Steger algorithm according to an embodiment of the present invention.

[0026] Figure 5 This is a flowchart of a monocular laser scanning defect detection process according to an embodiment of the present invention.

[0027] Figure 6 This is a neural network architecture diagram for real-time lightweight composite material defect detection according to an embodiment of the present invention.

[0028] Figure 7 This is a framework diagram of a multi-method fusion convolutional neural network model compression according to an embodiment of the present invention.

[0029] Figure 8 This is a schematic diagram of a small-sample industrial defect detection model based on a twin network architecture, according to an embodiment of the present invention.

[0030] Figure 9 This is a schematic diagram of the large model fine-tuning technique based on transfer learning in an embodiment of the present invention.

[0031] Figure 10 This is a schematic diagram of multimodal joint characterization according to an embodiment of the present invention.

[0032] Figure 11 This is a schematic diagram illustrating the collaborative representation of 2D images and 3D point clouds in an embodiment of the present invention.

[0033] Figure 12 This is a schematic diagram of a composite material defect detection model based on the fusion of 2D images and 3D point clouds according to an embodiment of the present invention.

[0034] Figure 13 This is a schematic diagram illustrating the coordinate system of the robotic arm linkage according to an embodiment of the present invention.

[0035] Figure 14 This is a schematic diagram of the AX=XB calibration model in the hand-eye pose parameter identification of an embodiment of the present invention.

[0036] Figure 15 This is a schematic diagram of turntable calibration and ID recognition tracking of coding markers according to an embodiment of the present invention.

[0037] Figure 16 This is a schematic diagram of the rigid coordinate system of the robotic arm base and the external turntable in an embodiment of the present invention.

[0038] Figure 17 This is a schematic diagram of the external axis calibration of the turntable according to an embodiment of the present invention.

[0039] Figure 18 This is a schematic diagram illustrating the geometric error identification principle of an embodiment of the present invention.

[0040] Figure 19 This is a flowchart illustrating the online compensation process for orientation errors according to an embodiment of the present invention.

[0041] Figure 20 This is a flowchart of the geometric iteration calibration process according to an embodiment of the present invention.

[0042] Figure 21 This is a schematic diagram of the laser projection segmented relay mechanism according to an embodiment of the present invention.

[0043] Figure 22 This is an overall structural diagram of the automatic online detection system for wire placement defects according to an embodiment of the present invention. Detailed Implementation

[0044] The embodiments of the present invention will be described in detail below. It should be emphasized that the following description is merely exemplary and not intended to limit the scope and application of the present invention.

[0045] It should be noted that when a component is referred to as "fixed to" or "set on" another component, it can be directly on or indirectly on that other component. When a component is referred to as "connected to" another component, it can be directly connected to or indirectly connected to that other component. Furthermore, a connection can be used for fixing, coupling, or communication.

[0046] It should be understood that the terms "length", "width", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", and "outer" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing the embodiments of the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the present invention.

[0047] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of embodiments of the present invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0048] This invention aims to address the technical bottlenecks in existing automated fiber placement processes, such as poor quality of multimodal information acquisition under conditions of low reflectivity, large depth of field, and short distance measurement due to the light absorption characteristics of carbon fiber composite materials and compact space, as well as the limited features of single-modal data making it difficult to cope with diverse defects and severe lag in detection feedback. It proposes an automated online inspection solution integrating high-quality 3D imaging, dual-modal fusion recognition, and real-time feedback from multiple devices. This is achieved by constructing a reflective 3D vision imaging system based on a SAM lens and line laser, deeply integrating two-dimensional image texture features and three-dimensional point cloud geometric features, and combining an integrated calibration and segmented relay projection mechanism of a robotic arm, imaging system, and multiple laser projection devices. This enables closed-loop control of the entire process from data acquisition and defect identification to real-time positioning feedback. Specifically: 1) A high-quality multimodal fiber placement defect acquisition method is proposed for scenarios with low reflectivity, large depth of field, and short distance measurement: First, due to the light absorption characteristics of the composite material surface, the 2D image signal-to-noise ratio is low, and the line laser signal is weak. Adding a stroboscopic light source can increase the amount of light entering the camera while maintaining the same depth of field. Simultaneously, the reflective structure of the line laser and 2D camera is suitable for measuring dark objects with weak reflectivity. Second, the internal space of the AFP automatic fiber placement head is limited, requiring large depth of field measurement at short distances. Introducing a Sham lens resolves the contradiction between large depth of field and short distance measurement. Finally, single-modal data contains limited geometric features of the composite material, which cannot meet the diverse defect requirements of actual engineering sites. A high-resolution 2D camera and a line laser are selected and simultaneously triggered to form a reflective 3D vision imaging system to acquire high-quality multimodal fiber placement defects. Therefore, a 3D vision imaging system based on a Sham lens and a line laser needs to be designed and developed for scenarios with low reflectivity, large depth of field, and short distance measurement.

[0049] 2) To address the limitations of existing defect detection data, a dual-modal defect localization and rapid identification technology based on 2D-3D fusion is proposed. Firstly, given the contradiction between the limited geometric features of single-modal data and the diversity of defects, an effective fusion model for composite defect detection based on 2D images and 3D point clouds is needed. This includes: a. Effective fusion of multimodal data requires real-time processing of the modal data; therefore, the YOLO detection algorithm needs to be applied to the lightweight 2D image CNN mesh output layer to ensure real-time image detection. b. The 3D point cloud obtained from point cloud recognition has inaccurate defect module localization; by designing a defect localization module that combines point cloud normal vectors and curvature information, the localization accuracy is improved; clustering algorithms (such as DBSCAN) are used to segment the point cloud and refine the defect region. c. Given the existing 2D texture and 3D point cloud, data complementarity is used to improve the full-scale perception capability of defect detection from microscopic texture to macroscopic geometry. Kalman filtering or particle filtering algorithms are used to smooth the defect center coordinates, improving localization stability. Meanwhile, texture information in 2D images is used to verify and correct the 3D positioning results.

[0050] Secondly, current mainstream supervised learning methods in 2D image defect detection suffer from limited datasets, making it difficult to train large detection and classification networks. Therefore, a lightweight, real-time, and transferable deep training network based on small sample sizes is needed. This includes: a. Introducing a lightweight model to handle massive pixel data and adapt to different hardware computing power. b. Improving computational efficiency by reducing the model size through pruning and quantization techniques without sacrificing accuracy. c. Enabling strong cross-scene adaptability through transfer learning and model fine-tuning. These strategies enable effective defect detection of 2D pixel data, improving both efficiency and accuracy.

[0051] 3) Real-time defect localization technology integrating robotic arms, 3D imaging systems, and projection equipment to improve the real-time feedback of defect detection: First, existing AFP defect detection systems mostly adopt an offline mode of "acquisition first, processing later," resulting in a serious lag in the defect feedback mechanism, leading to the inability to correct some process defects (such as gaps, bridging, and wrinkles). By calibrating the robotic arm, 3D imaging system, and projection equipment, the coordinate system of the three is unified, ensuring that the detection results can be converted, understood, and responded to instantly. Second, current projection localization technology can only determine the number of defects, not achieve pixel-level precise localization. Introducing a laser projection system, with laser projection systems set at fixed intervals within a layup cycle, allows multiple laser projection systems to dynamically project the real-time position coordinates of defects through a segmented relay method. This enables instant conversion of detection data and precise localization of defect positions in real-time captured images via laser projection. Therefore, a real-time defect localization technology integrating robotic arms, 3D imaging systems, and projection equipment is needed to achieve real-time defect detection.

[0052] See Figures 1 to 2 This invention provides a method for online automatic wire placement defect detection using a robot based on 3D vision, comprising the following steps: Step S1: Construct and calibrate a 3D vision imaging system based on a SAM lens and a line laser, so that it moves synchronously with the wire-laying robotic arm, acquires two-dimensional images of the layup containing laser light stripes online, and reconstructs the corresponding three-dimensional point cloud data based on the plane equation of the line laser blade.

[0053] In some embodiments, the calibration process of the 3D vision imaging system in step S1 includes: SAM lens calibration: A calibration plate with coded and non-coded points arranged regularly on its surface is placed within the measurement area of ​​the imaging system and adjusted to multiple different positions; images of the calibration plate at each position are acquired by the camera, the marker points in the images are identified, and their pixel coordinates are obtained; based on the known global coordinates of the marker points in the calibration plate's own coordinate system and their corresponding image pixel coordinates, a projection geometric model that introduces a parameter describing the relative tilt of the lens plane and the camera sensor plane is established and solved to calculate the camera's intrinsic and extrinsic parameters as well as the relative tilt parameter caused by the SAM lens.

[0054] Line laser beam cutter plane calibration: After completing the Sham lens calibration, keep the calibration plate position unchanged, turn on the line laser to project the laser line onto the surface of the calibration plate, and acquire images with laser beams; using the beam center extraction algorithm, extract the sub-pixel level image coordinates of the center point of the laser beam from the images at each position; combining the calibrated camera parameters and the transformation relationship between the calibration plate coordinate system and the camera coordinate system, calculate the image coordinates of each beam center point to the three-dimensional spatial coordinates in the camera coordinate system; finally, use the least squares method to perform plane fitting on all the calculated three-dimensional spatial coordinate points, and the resulting plane equation is the beam cutter plane equation of the line laser.

[0055] In some embodiments, the light stripe center extraction algorithm employs an improved Steger algorithm, which includes the following steps: Coarse localization of the light stripe: Traverse the image row by row, perform a one-dimensional Gaussian convolution operation on the gray values ​​of each row of pixels, and preliminarily determine the integer pixel position region of the center of the laser light stripe by finding the maximum value of the convolution result.

[0056] Fine-grained region segmentation: Based on the coarse positioning results, a certain width is extended outward from the center point of the light stripe to segment out a sub-image region containing the complete light stripe.

[0057] Sub-pixel center calculation: Within the segmented sub-image region, the first-order and second-order gradients of the image in the x and y directions are calculated simultaneously using an optimized convolution kernel, thereby constructing the Hessian matrix for each pixel; the eigenvector direction corresponding to the largest eigenvalue of the Hessian matrix is ​​calculated, and sub-pixel interpolation is performed in this direction using Taylor expansion, finally obtaining the sub-pixel level precise coordinates of the laser light stripe center.

[0058] Step S2: Perform fusion analysis on the two-dimensional image and three-dimensional point cloud data: Use a lightweight deep learning network trained on few samples to perform real-time defect detection and preliminary localization on the two-dimensional image, and at the same time, locate and segment the defect area based on the normal vector and curvature information of the three-dimensional point cloud. Then, fuse the two-dimensional texture features and three-dimensional geometric features through a collaborative representation model to achieve accurate identification and localization of wire laying defects.

[0059] In some embodiments, the construction and training process of the lightweight deep learning network based on few-shot training in step S2 includes: Basic model construction: A lightweight convolutional neural network containing a multilayer perceptron module is established as the backbone network to extract multi-scale features from two-dimensional images, and attention is introduced to enhance the focus on features of defective regions.

[0060] Model compression and optimization: The backbone network is structurally pruned through sparse training and network width and depth pruning; a knowledge distillation method is used to guide the training of the pruned student network using a complex teacher network to compensate for accuracy loss; further, the model weights are quantized and converted into low-bit-width integer format to compress the model size and improve inference speed.

[0061] Few-shot transfer learning: A few-shot learning model based on a Siamese network architecture is adopted, using the compressed and optimized network as an encoder. Defect classification is achieved by comparing the similarity of the embedding vectors of query samples and support samples in the feature space; or a transfer learning method is adopted, freezing most of the parameters of the model pre-trained on a public dataset, and only using a small amount of target defect sample data to fine-tune the high-level classification layer or specific modules of the network, and designing a multi-task loss function that combines classification loss and domain adaptation loss for optimization.

[0062] In some embodiments, the process of establishing the collaborative representation model in step S2 specifically includes: defining projection functions for the two-dimensional image modality and the three-dimensional point cloud modality respectively, mapping the extracted features to a shared or associated representation space. An optimization objective function is constructed, which aims to minimize the sum of the Frobenius norms of the errors when all modal data are linearly reconstructed through other modal data and their corresponding collaborative representation matrices, while adding regularization constraints on the collaborative representation matrix to prevent overfitting. By solving the optimization objective function, the collaborative representation matrix is ​​obtained, thereby achieving effective fusion of two-dimensional image texture features and three-dimensional point cloud geometric features at the deep representation level, for subsequent joint identification and localization of defects.

[0063] Step S3: The identified and located defect information is transformed into coordinates in the robot arm coordinate system, 3D vision imaging system coordinate system and multiple laser projection device coordinate systems that have been calibrated by target points. The multiple laser projection devices are then controlled to project the defect location coordinates onto the ply surface in real time in a segmented relay manner for feedback.

[0064] In some embodiments, the calibration and error compensation process of coordinate system one in step S3 includes: Hand-eye calibration: A hand-eye calibration method based on quaternion multiplication is adopted. By controlling the robotic arm to drive the 3D vision imaging system to observe the calibration plate at a fixed position from multiple different poses, multiple sets of pose transformation matrices of the robotic arm end effector and the corresponding pose transformation matrices of the camera relative to the calibration plate are obtained. The hand-eye calibration equation is established and solved to obtain the initial values ​​of the rotation and translation transformation matrices of the 3D vision imaging system relative to the robotic arm end effector.

[0065] Synchronous calibration and compensation of geometric errors: A comprehensive system geometric error model is established, which simultaneously considers the influence of joint kinematic errors of the robotic arm, hand-eye calibration errors, and external axis errors (if present) on the overall measurement pose. A nonlinear optimization problem is constructed with the constraint that the theoretical coordinates of the same set of fixed target points in the world coordinate system should be consistent with the coordinates estimated through the system kinematic chain under multiple different observation poses. The optimal system geometric error parameters are identified through an iterative algorithm. The identified error parameters are used to correct the system kinematic model, and the robotic arm's motion commands are corrected online by compensating for joint rotation angles in the joint space, thereby achieving high-precision coordinate system alignment and pose orientation.

[0066] Unified projection system coordinates: Multiple target points with known three-dimensional coordinates in the workpiece coordinate system are laid out on the already laid carbon fiber ply plane; the target points are measured and identified by the 3D vision imaging system with completed hand-eye calibration and error compensation, and by each laser projection device; using a point cloud registration algorithm, the rigid transformation matrices from the robot arm base coordinate system to the workpiece coordinate system and from the coordinate system of each laser projection device to the workpiece coordinate system are calculated, thereby unifying all systems to the same workpiece coordinate system.

[0067] See Figure 21 In some embodiments, the segmented relay projection method in step S3 specifically includes: arranging multiple laser projection devices at fixed intervals along the laying direction, with some projection areas of adjacent projection devices overlapping; when the 3D vision imaging system moves to different layup segments and performs detection, the identified defect coordinates are converted to the workpiece coordinate system in real time, and then dynamically allocated and sent to the laser projection device corresponding to the current layup segment according to the current real-time spatial position of the 3D vision imaging system; for defect coordinates located in the overlapping area of ​​adjacent projection devices, the spatial distance from the defect coordinates to the origin of each adjacent projection device is calculated, and the one with the shortest distance is selected as the projection device responsible for projection; the laser projection device receives the defect coordinate information and accurately projects it to the corresponding position on the layup surface, thereby realizing dynamic and real-time visual feedback of the defect position throughout the entire layup process.

[0068] See Figure 1 and Figure 22A robotic automated online defect detection system for wire layup based on 3D vision includes a robotic arm, a 3D vision imaging subsystem, an image processing and defect analysis subsystem, and a multi-laser projection subsystem. The robotic arm performs carrying and movement during the wire layup process. The 3D vision imaging subsystem, fixed to the end of the robotic arm, includes a camera with a SAM lens, a line laser, and a stroboscopic light source, used to acquire two-dimensional images of the layup online and simultaneously reconstruct three-dimensional point cloud data. The image processing and defect analysis subsystem is communicatively connected to the 3D vision imaging subsystem and is used to perform fusion analysis of the acquired two-dimensional images and three-dimensional point cloud data using the online detection method to achieve defect identification and location. The multi-laser projection subsystem includes multiple laser projection devices spaced apart along the layup direction, used to receive defect coordinate information output by the image processing and defect analysis subsystem and project the defect position onto the layup surface in real time under a unified coordinate system. The robotic arm, 3D vision imaging subsystem, and multi-laser projection subsystem are calibrated and unified under the same workpiece coordinate system.

[0069] In some embodiments, the image processing and defect analysis subsystem includes an image acquisition card and defect detection and analysis software: the image acquisition card is used to receive raw image data acquired by the 3D vision imaging subsystem through a high-speed communication interface and perform preliminary processing; the defect detection and analysis software runs on a host computer and integrates a lightweight deep learning network based on few-sample training and a multimodal fusion algorithm, used to accurately identify and locate defects in the pre-processed image data, and send the results to the multi-laser projection subsystem.

[0070] See Figure 21 In some embodiments, the multi-laser projection subsystem operates in a segmented relay manner: each laser projection device is associated with a specific ply segment area, and the projection areas of adjacent devices partially overlap; the defect detection and analysis software dynamically determines the current working ply segment based on the real-time position of the robotic arm or 3D vision imaging subsystem, and allocates and sends defect coordinate information to the laser projection device corresponding to that ply segment; for defect coordinates located within the overlapping area of ​​adjacent projection devices, the defect detection and analysis software calculates the spatial distance from the defect coordinates to the origin of each adjacent projection device's own coordinates, and selects the one with the shortest distance as the projection device responsible for projection; the laser projection device receiving the information projects the defect coordinates onto the surface of the ply segment it is responsible for, thereby achieving continuous and seamless visualization of defect positions along long-distance ply lines.

[0071] This invention proposes a 3D vision-based robotic automated filament layup defect online detection method and system. By constructing a reflective 3D vision imaging system based on a SAM lens and line laser, it overcomes the bottleneck of high-quality imaging under conditions of low reflectivity, large depth of field, and short distance measurement. On this basis, it deeply integrates two-dimensional texture features and three-dimensional geometric information, combined with a lightweight few-sample learning strategy, achieving a defect recognition rate of over 95% with only 500 samples and supporting high-speed layup detection at 80m / min (to be further detailed later). Furthermore, through multi-device collaborative calibration and segmented relay projection mechanism, the defect coordinates are dynamically projected onto the layup surface in real time and seamlessly, completely solving the problem of lag in feedback from traditional offline detection, and forming an integrated closed-loop control from data acquisition, intelligent recognition to real-time positioning.

[0072] The technical principles, implementation methods, and advantages of specific embodiments of the present invention are further described below.

[0073] To address the technical bottlenecks of existing online detection methods for automated fiber placement defects in AFP systems, the core technical points of this invention include: 1. Constructing a 3D Vision Imaging System Based on a Schamm Lens and Line Laser: Constrained by the limited space within the AFP fiber placement head and the light absorption characteristics of the composite material surface, this project addresses the image acquisition requirements of small area, long depth of field, and high brightness. A high-precision 3D imaging system based on a Schamm lens and line laser is specifically designed and developed. This includes: Schamm lens calibration to resolve the contradiction between large depth of field and short distance measurement, meeting the image acquisition requirements of large depth of field within a short distance; calibration of the beam cutter plane of the monocular line laser, supplemented by an improved high-precision light stripe extraction algorithm to ensure high-precision light stripe identification; and a high-resolution monocular camera and line laser arranged in a reflective configuration, supplemented by a stroboscopic light source, forming a 3D imaging system. Through synchronous triggering and acquisition, high-quality imaging of the black carbon fiber surface is achieved under relatively low laser power conditions.

[0074] 2. Effectively Integrating 2D-3D Dual-Modal Fiber Placement Defects for Rapid Defect Localization and Identification: Current defect detection technologies cannot meet the high-precision and high-efficiency defect detection requirements of industrial sites. To address complex defect detection, a technology integrating 2D-3D dual-modal fiber placement defects is needed for rapid defect localization and identification. This includes: First, 2D images and 3D point clouds have complementary advantages in composite material defect detection. 2D images, with their high-resolution texture information, excel at identifying surface defects such as missing fibers, bubbles, and foreign objects, but have low sensitivity to subtle irregular defects such as gaps and overlaps. 3D point clouds, on the other hand, effectively characterize gaps and overlaps through geometric and depth data, but due to the sparsity of point clouds, they struggle to capture details of complex morphologies such as wrinkles and twists. To integrate the advantages of both methods, a collaborative detection framework is constructed as follows: a. 2D Image: A lightweight CNN (such as MobileNetV3) is used to extract multi-scale texture features, an attention mechanism is introduced to enhance the response of defect areas, and real-time classification and 2D localization are achieved based on the YOLO architecture; b. 3D Point Cloud: A defect localization module is constructed based on point cloud normal vectors and curvature information, the 3D coordinates of the defect center are regressed, and clustering algorithms are used to refine the defect area; c. 2D Image and 3D Point Cloud Fusion Optimization: The 2D detection results and 3D localization information are combined, and the defect coordinates are smoothed and optimized through filtering algorithms. 2D texture data is used to verify and correct the 3D output, improving localization stability and recognition robustness. This fusion strategy achieves full-scale defect perception from micro-texture to macro-geometry, effectively improving detection accuracy and system anti-interference capability. In addition, to effectively process massive 2D pixel data, a lightweight model based on small samples is required. Under limited computing power, a technical approach of model pruning, quantization, transfer learning, and model fine-tuning is adopted to achieve efficient and high-quality recognition and detection of image defects. The rapid localization and identification technology for 2D-3D dual-modal fiber placement defects described above can achieve high-speed fiber placement detection at 80m / min, and achieve a defect identification rate of over 95% with a small sample model of about 500 sheets.

[0075] 3. Real-time Defect Localization Integrated with Robotic Arm, 3D Imaging System, and Projection Equipment: Due to the significant lag in defect feedback mechanisms, some process defects cannot be corrected in a timely manner. Based on the requirement for real-time defect detection, a real-time defect localization scheme integrating a robotic arm, 3D imaging system, and projection equipment was designed, such as... Figure 1 As shown. This includes: using target points with known coordinates to unify the coordinate systems of the robotic arm, 3D imaging system, and multiple projection devices, ensuring that detection results can be converted, understood, and responded to in real time; introducing a laser projection system to accurately locate defects, setting projection devices at fixed intervals within a layup cycle, and having multiple laser projection systems dynamically project the real-time position coordinates of defects through a segmented relay method, achieving pixel-level defect location technology.

[0076] 4. Integrated Automated Online Detection of Wire Placement Defects: Based on the aforementioned core technical points, and to adapt to actual defect detection scenarios, an integrated automated online detection method for wire placement defects is adopted. The detection process includes: First, using a calibration plate with regularly arranged marker points, combined with an improved light stripe extraction algorithm, high-quality laser light stripe images are extracted to achieve Sham lens calibration and line laser beam plane calibration; then, using a calibrated 3D imaging system, multimodal pixel data is acquired, and 2D-3D dual-modal rapid location and identification technology for wire placement defects is applied to complete defect detection and identification; finally, based on the coordinate system, a robotic arm, 3D imaging system, and laser projection system, the laser projection system dynamically and in real-time feeds back defect data to the host computer via a relay mechanism, and the host computer confirms whether process repair is required. Simultaneously, the software supports outputting a defect report (including the number of defects detected and their coordinates).

[0077] Figure 2 This invention demonstrates the technical approach used to achieve automated detection and control of surface defects in the fiber placement process. Based on the core technical points of this invention, the implementation methods are described in detail below.

[0078] As a preferred embodiment, the 3D visual imaging system based on a Sham lens and a line laser includes: first, using components such as a Sham lens, a monocular line laser, and a stroboscopic light source to form a reflective 3D visual imaging system; then, calibrating the 3D visual imaging system according to the optical characteristics of the Sham lens and combined with an improved high-precision light stripe extraction algorithm; finally, acquiring multimodal pixel data using the calibrated 3D visual imaging system.

[0079] The aforementioned reflective 3D vision imaging system includes: a 3D reconstruction model based on the principle of triangulation; a line laser emitting a linear laser beam onto the surface of the object being measured; and a monocular camera with a Schahm lens located in the reflective structure of the line laser. It receives a three-dimensional laser stripe image modulated onto the surface of the object under test. Using pre-calibrated camera intrinsic parameters and light plane information in the camera coordinate system, it calculates the three-dimensional contour line in the camera coordinate system. This reflective structure is complex to install, and the application of the calculation results is not very intuitive; it is mostly used for surface measurement of dark-colored objects with weak reflectivity.

[0080] Figure 3 A 3D model of the detection device of the AFP automated wire placement defect online detection system is shown, which includes an industrial camera group 2 (using a monocular camera), a line laser 3, an illumination auxiliary device 4, and a placement surface 10.

[0081] The method for calibrating the 3D vision imaging system includes: placing a calibration plate with coded and non-coded points arranged in a regular pattern on the surface matching the measurement area at a standard measurement distance; adjusting the position of the calibration plate; and acquiring images of the calibration plate at different positions using a 2D camera to calibrate the 3D vision imaging system. The calibration parameters of the 3D vision imaging system include: 1) the camera's intrinsic and extrinsic parameters, as well as the tilt parameters of the lens plane relative to the camera sensor plane caused by the SAM lens; 2) the precise spatial positioning of the line laser beam projected by the line laser, i.e., the equation of the laser beam plane. Specifically, with the laser off, images of the calibration plate at different positions are captured to solve for the camera's intrinsic and extrinsic parameters; keeping the calibration plate position unchanged, the laser is turned on, and images of the calibration plate containing the laser beam are captured. By combining the laser beam images from all calibration plate positions, the laser beam plane calibration of the line laser is completed.

[0082] The Sham lens calibration method includes: Sham calibration is considered a generalization of the imaging law under normal conditions. It introduces parameters describing the relative tilt of the camera sensor plane and the lens plane, and establishes a new projection geometric model that accurately describes the light rays from the object point through the tilted lens to the sensor pixel. Given the global coordinates of the calibration plate in its own calibration plate coordinate system, the light source and the camera with the Sham lens are activated. The position of the calibration plate is continuously adjusted within the measurement area. Using the calibration plate images captured by the camera at various positions, combined with the corresponding marker point image coordinates and the global point coordinates, the intrinsic and extrinsic parameters of the camera and the relative tilt parameters caused by the Sham lens are solved. The specific projection process can be expressed by the following formula: in: — Coordinates of the object point in the world coordinate system / mm; — Coordinates of the projection center point in the world coordinate system (mm); —Image point coordinates / pixel; —Principal point deviation / pixel; —Lens focal length / mm; —Pixel deviation caused by lens distortion; —The rotation angle / pixel of the lens plane of the Sham lens relative to the camera sensor plane. , , These are the camera's intrinsic parameters.

[0083] and These represent the rotation and translation matrix relationships from the world coordinate system to the camera coordinate system, i.e., the camera's extrinsic parameters. The specific transformation relationships are as follows: in: —Indicates the coordinates of the object point in the camera coordinate system in mm.

[0084] The line laser scalpel plane calibration method includes: accurately locating the planar position and orientation of the laser line (scalpel) in three-dimensional space using the scalpel plane, for precise reconstruction of the 3D point cloud of the line laser stripe. First, while capturing an image of the calibration board used for camera calibration, the line laser is turned on, and the camera captures a layered image with the laser stripe. The improved Steger algorithm is then used to extract the pixel coordinates of the center points of the laser stripes at all calibration locations. Next, given the camera's extrinsic parameters, the 3D coordinates of the pixel coordinates of all center points of the laser stripe at a given calibration location are determined in the camera coordinate system. Finally, the 3D coordinates of the center points of the laser stripes at all calibration locations in the camera coordinate system are solved. Based on the least squares method, the 3D coordinates of all center points are fitted to obtain the laser plane, which is the scalpel plane of the line laser.

[0085] At a certain calibration plate position, the matrix transformation relationship between the calibration plate coordinate system and the camera coordinate system is determined through camera calibration. , Extracting the image coordinates of the center point of a laser beam. Solve for the 3D coordinates of the center point of the light stripe in the camera coordinate system. The specific expression is: in, The two-dimensional coordinates of the center point of the light stripe in the pixel coordinate system Reconstruct the 3D point coordinates in the calibration plate coordinate system; for The corresponding 3D point coordinates in the camera coordinate system.

[0086] The improved Steger algorithm includes the following: Since the core of optimizing the computational efficiency of the Steger algorithm lies in eliminating redundant operations in the image processing chain, unnecessary calculations are avoided during the computation process by introducing a pre-screening mechanism and a dynamic thresholding strategy, while increasing the robustness of the algorithm. The specific optimization strategy is as follows: First, simplify the number of convolutions. The simplified convolution expression is: in: In the formula , The first-order gradient of the image; The second-order gradient of the image; , They are direction, One-dimensional Gaussian convolution kernel in the direction, , It uses a gradient-oriented convolution kernel with the Sobel operator. Since the image has undergone high-speed filtering, it doesn't require further smoothing convolution with the Sobel operator. Secondly, it performs coarse positioning of the laser beam. For example... Figure 4 As shown, the image is traversed row by row. For positions where the grayscale value is below a set threshold, a high-speed convolution kernel is applied to calculate the convolution result at that position. The specific convolution expression is: The position with the largest convolution result is the integer pixel position at the center of the laser stripe, as shown in the specific figure. Figure 4 As shown, the improved Steger algorithm is used to achieve coarse positioning of laser beams.

[0087] The optimized algorithm flow is shown below: 1) Convert the image format; 2) Traverse the image row by row, using a 1*7 high-speed convolution kernel to locate the integer pixel position of the light stripe's center point; 3) Segment the light stripe based on the location result, with the segmentation border required to be twice the line width from the light stripe's center point; 4) Perform convolution calculations on the segmented image to obtain the first-order and second-order gradient maps; 5) Calculate the eigenvalue corresponding to the largest eigenvalue of the Hessian matrix for the integer pixel position located in 2), perform Taylor expansion, and calculate t. Then, the center point of the light stripe is... .

[0088] The method for acquiring multimodal pixel data using a calibrated 3D vision imaging system includes: First, the calibrated 3D vision imaging system moves synchronously along the fiber placement direction following the fiber placement head; then, a line laser and a stroboscopic light source are activated, the line laser projects a thin laser line onto the layup surface, and a camera with a Sham lens captures a 2D layup image with laser light stripes; finally, based on the light blade plane equation of the line laser, point cloud recognition technology is applied to reconstruct the 3D point cloud of the laser light stripes.

[0089] Figure 5 The process of defect detection using monocular laser scanning was demonstrated. As a preferred embodiment, the dual-modal fiber placement defect rapid localization and identification technology based on 2D-3D fusion includes: First, a lightweight model is built based on a small sample size to efficiently extract defect features under limited computing power. The model is then pruned and quantized, and through transfer learning and fine-tuning, the identification and detection of defects in 2D images are achieved. Next, a multimodal collaborative representation model is established, and the YOLO algorithm is applied to improve the real-time performance of 2D image detection. Clustering algorithms (such as DBSCAN) are used to segment the 3D point cloud and refine the defect region. Finally, using 2D images and 3D point clouds, wire-laying defects are quickly located. Kalman filtering or particle filtering algorithms are used to improve the stability of defect location. Simultaneously, texture information from the 2D images is applied to quickly verify and repair the 3D point cloud location results.

[0090] The method for building a lightweight, real-time, and transferable deep training network based on small samples includes: First, establishing a lightweight model based on 2D images. Specifically, a feature extraction network based on MLP is used to move feature information along the axial direction of the 2D image to obtain information flow in different directions. Spatial movement, by emphasizing the extraction of low-level features, effectively captures local correlations and obtains more detailed edge information of the 2D image. Through the structure of a multi-layer neural network, it automatically learns and adapts to these complex defect features, achieving accurate detection of multiple types of defects on composite surfaces. Figure 6 shows the real-time lightweight composite defect detection neural network architecture.

[0091] Then, the model is effectively compressed. This includes three parts: model pruning, knowledge distillation, and model quantization. Model pruning achieves significant network compression through width and depth pruning in sparse training. Knowledge distillation uses a teacher network (a complex knowledge model) to assist the student network (a simplified knowledge model) in training, compensating for the loss of model accuracy caused by extensive pruning and ensuring the stable performance of the student network after pruning. Model quantization quantizes the model's parameters into 8-bit integers, further compressing the model size and improving the model's inference speed. Figure 7 A framework for compressing convolutional neural network models by fusing multiple methods is presented.

[0092] Finally, model transfer and generalization training. a. Model fine-tuning and generalization training. For example... Figure 8 As shown, a small-sample industrial defect detection model based on a Siamese network architecture is designed. The network transforms the region of interest into a feature vector, which is then used as input to a distance metric learning sub-network. The similarity is measured by comparing the distance (similarity) between the embedded feature vector and the representation vector of each target class, i.e., by using matrices respectively. and Obtain query features Query embedding and supporting features Key embedding . and The correspondence between them can be calculated by the following method, which enables industrial defect detection under small sample conditions after calculating the posterior probability of each region of interest.

[0093] .

[0094] b. Representation transfer techniques for fine-tuning models. For example... Figure 9 As shown, during the model training phase, a model fully trained on a public large dataset (a pre-trained model optimized for industrial scenarios) is used as the pre-trained model. The structural parameters of the trained model are then transferred to a model with a small number of new target class samples. During the fine-tuning phase, the entire encoder network is frozen, and the model is fine-tuned from the intermediate layers of the multilayer perceptron using only the data distribution of the new target classes. The shallow parameters of the pre-trained model are frozen, and only high-level parameters (such as the classification layer and attention mechanism layer) are fine-tuned to reduce computation and prevent overfitting. A supervised balanced loss function is designed during the fine-tuning phase. Combined with classification loss (cross-entropy loss) ), detection loss ( We construct a multi-task optimization objective by using domain-adaptive loss (adversarial loss) and domain-adaptive loss (adversarial loss) to ensure robustness to new data distributions.

[0095] The aforementioned technique for rapid localization of wire-laying defects using 2D images and 3D point clouds includes: First, determining the multimodal feature relationships. After extracting features from the single-modal data, the features from different modalities are concatenated into a single high-dimensional feature vector using an algorithm. The expression for the joint representation is: Figure 10 Modal joint characterization is demonstrated.

[0096] Then, the multimodal cooperative representation model is determined. Cooperative representation is one of the multimodal fusion representation learning methods, responsible for mapping each modality in the multimodal representation to its respective representation space. Assume we have... There are 1 modality, and each modality uses a matrix. It means that, among them The goal of cooperative representation is to minimize the reconstruction error between different modes. A basic cooperative representation model can be represented by the following formula.

[0097] In the formula: for Norm; For the first The modality pair of the first Contribution weights for each modality; For the cooperative representation matrix; For regularization functions; This is the regularization parameter. The proposed co-representation of 2D image and 3D point cloud is as follows: Figure 11As shown in the figure: and The projection function corresponding to each mode.

[0098] Finally, 2D images and 3D point clouds are used to achieve rapid localization of wire-laying defects. A lightweight CNN (such as MobileNetV3 or EfficientNet-Lite) is used as the backbone network to extract multi-scale texture features from the 2D images. By adjusting the network depth and width, feature extraction capability and computational efficiency are balanced. An attention mechanism (such as an SE module or CBAM module) is introduced into the CNN network to enhance attention to defect region features. Simultaneously, a multi-scale fusion strategy is adopted, combining shallow texture details with deep semantic information to improve the richness of feature representation. A defect detection head is designed in the output layer of the CNN network to achieve defect classification and bounding box regression in the 2D images. The YOLO detection algorithm is used to ensure real-time performance. In the point cloud network, a defect localization module is designed to directly regress the 3D coordinates of the defect center point. By combining point cloud normal vectors and curvature information, positioning accuracy is improved. Simultaneously, clustering algorithms (such as DBSCAN) are used to segment the point cloud, further refining the defect region. The defect location is jointly optimized by combining 2D detection results and 3D positioning information. Kalman filtering or particle filtering algorithms are used to smooth the defect center coordinates, improving positioning stability. Furthermore, texture information from the 2D image is used to verify and correct the 3D positioning results.

[0099] Figure 12 A composite defect detection model based on the fusion of 2D images and 3D point clouds was demonstrated.

[0100] As a preferred embodiment, the real-time defect localization technology integrating the robotic arm, 3D imaging system, and projection equipment includes: first, applying hand-eye calibration technology to determine the relative pose relationship between the 3D vision imaging system and the end effector of the robotic arm; then, applying target points with known three-dimensional coordinates to unify the coordinate systems of the robotic arm, the 3D vision imaging system, and the laser projector; finally, the laser projection system dynamically and in real-time feeds back defect data to the host computer via a relay method, thereby realizing the detection of wire laying defects and process adjustment.

[0101] The method for determining the relative pose relationship between the 3D vision imaging system and the robotic arm end effector through hand-eye calibration includes the following: Traditional hand-eye calibration methods directly read the nominal kinematic parameters of the controller for calibration, ignoring the transmission and accumulation of the robotic arm end effector positioning error along the spatial kinematic chain. Therefore, the calibration result under the nominal kinematic parameters of the controller is selected as the initial value for overall synchronous calibration of the system's orientation error, thereby improving the calibration accuracy of the system's geometric pose parameters.

[0102] The initial values ​​of hand-eye pose and external rotation axis pose can be obtained through quaternion-based hand-eye calibration and turntable calibration based on marker tracking.

[0103] Quaternion-based hand-eye calibration methods include: Hierarchical kinematic modeling of robotic arm: Ignoring joint errors, the homogeneous pose transition matrix of adjacent joints of the robotic arm is obtained based on the MDH transformation: Among them, the four link parameters between the robotic arm links include: link torsion angle. Link length Linkage offset Three fixed structural parameters and joint rotation variables The parameters of the four links between the robotic arm links include: link torsion angle. Link length Linkage offset Three fixed structural parameters and joint rotation variables .

[0104] Figure 13 illustrates the description of the link coordinate system.

[0105] The hand-eye calibration equation is rewritten based on the quaternion multiplication facilitation, and the rotation matrix of the hand-eye relationship is solved using singular value decomposition (SVD). Substitute into the rotation matrix Solve for the translation matrix of the hand-eye relationship. This completes the initial calibration of hand and eye posture.

[0106] Hand-eye calibration equation: This section analyzes the structure of the eye in the hand shape, selecting a planar calibration target as the measurement object, and defining... , , , These are the robot arm base coordinate system, the end effector coordinate system, the camera coordinate system of the 3D vision imaging system, and the planar target coordinate system, respectively. Figure 14 As shown, this illustrates the process of hand-eye pose parameter recognition. A schematic diagram.

[0107] Hand-eye rotation matrix: The unit quaternion is used to describe the rotational changes of a rigid body in three-dimensional space, and its expression exists as follows: in, and These represent two unit quaternions describing three-dimensional rotational motion. Operators To represent quaternion multiplication, and Let each represent an antisymmetric matrix expression for the quaternion product calculated from the coordinates to the right, and let the characteristic of continuous rotation be represented by unit quaternions: , and Representing rotation matrices respectively , and The quaternion form.

[0108] Hand-eye calibration steps The calibration plate is placed at a fixed position on the measuring plane of the turntable. The robotic arm controls the camera to move to 15 different calibration positions shown in the teaching diagram. The camera captures the calibration image at each position, and at the same time, the corresponding posture of the robotic arm end effector is read from the controller. For the acquired calibration images, mark point identification is performed, and the camera's attitude relative to the calibration board at each teaching position is calculated. Therefore, for Each hand-eye alignment is defined, and there exists a total of [number] hand-eye alignments. For each unique combination, a set of relative pose matrices can be calculated, i.e.: and , ; to matrix , Substituting the above quaternion expression, the hand-eye relative pose is calculated based on the hand-eye calibration method of the above quaternion multiplication ease.

[0109] Solving the matrix transformation relationship between the robotic arm and the rotation axis of the external turntable: Turntable calibration can determine the pose of the external rotation axis in the robotic arm coordinate system. Unlike traditional methods, this invention proposes a turntable rotation axis calibration method that tracks marker points. The theoretical basis is that for any three non-collinear points performing rigid rotational motion in space, their initial positions and rotated positions can uniquely determine a spatial transformation relationship. Through this transformation relationship, the unit direction vector of the rotation axis can be determined. coordinates of a point on the axis .

[0110] Rigid body rotation transformation: like Figure 15 As shown, the 3D vision imaging system captures images of the calibration plate on the turntable plane and reconstructs the set of marker points. Keep the robotic arm stationary, and rotate the turntable by an angle. Then, the set of marker points is reconstructed. Selecting a point set Encoding markers in Using the unique encoding ID of feature points, the marker points are tracked and their location within the point set is obtained. The corresponding The matrix relationship between matching point pairs is expressed as: dot set and The rotation matrix between them is The translation vector is The number of identified coded markers is .

[0111] Equivalent axis of rotation and equivalent angle of rotation: When calibrating a turntable, the turntable carries the calibration plate and rotates around its axis; therefore, its corresponding rotation matrix has one external rotational degree of freedom. In the formula: Represents any axis that passes through the origin. Rotation angle The equivalent edge-transformation matrix; .right The expression is generalized to determine the equivalent homogeneous transformation matrix in the turntable rotation measurement process.

[0112] Equivalent homogeneous transformation While the robotic arm grips the 3D vision imaging system, the workpiece to be measured is fixed on the turntable plane and rotates around an external axis. The workpiece's rotation around this external axis can be equivalent to a homogeneous coordinate transformation process involving rotation around an axis in space that does not pass through the origin by a certain angle. Regarding the rotation matrix... To promote this, that is, to define a world coordinate system. coordinate system with robotic arm base Fixed and coincident, this corresponds to the first unrotated state (zero position) of the turntable; using a coordinate system. Spatial rotation axis that does not pass through the origin and the point above Establish the turntable coordinate system The movement of the workpiece on the turntable plane can be considered as moving around a spatial axis. The pure rotation. The coordinate space of the workpiece being measured is determined by the central axis. A fixed point on the shaft The components are marked at the fixed point of the turntable zero position. At the rotation angle of the turntable The fixed point after that is marked as According to the right-hand rule, assuming the turntable rotates counterclockwise as the positive direction, the coordinate system... Rotation angle of the turntable The workpiece coordinate space after The pose in the middle is ,and It has a corresponding rotation angle Sequence point cloud The coordinate space of the turntable at its zero position before rotation. In coordinate system The pose in the middle is ,but The corresponding sequence point cloud can be represented as: Figure 16 shows the rigid coordinate system of the robotic arm base and the external turntable.

[0113] matrix The vector representing the external axis of the turntable around which the workpiece does not pass through the origin of the world coordinate system. Rotate clockwise The equivalent homogeneous edge permutation matrix is ​​expressed as follows: Rotary table calibration steps. Figure 17 shows a schematic diagram of external axis calibration for the rotary table.

[0114] 1) Use a robotic arm to adjust the scanner to a suitable measurement pose and place the calibration plate in a fixed position on the turntable's measurement plane. Then, rotate the turntable at intervals along the positive direction of the rotation axis to obtain 12 different calibration positions of the calibration plate, and further control the scanner camera to acquire calibration images at each position.

[0115] 2) Identify and reconstruct the 3D coordinates of coded marker points in the calibration image. All reconstructed points at each calibration location constitute a point set, resulting in a total of 12 distinct point sets. The pose of a turntable rotation axis can be calculated from any two point sets at different locations. Assuming N point sets are obtained, the th... The set of points and the first Each set of points is used as a pair to calculate a rotation axis pose, and a total of N-2 rotation axis poses can be calculated.

[0116] 3) The coded markers in each pair of points are tracked and matched using their unique coded IDs. Then, the equivalent rotation axis pose of each pair of points is calculated using the equivalent rotation axis method described in this section. A total of 10 rotation axis pose calculation results can be obtained from the 12 sets of reconstructed points.

[0117] 4) Due to the existence of errors, the calculated rotation axis pose will be different for different point sets. Therefore, the average value of multiple measurement results is taken as the final calibration result of the turntable rotation axis.

[0118] External axis pose determined by turntable calibration In the camera coordinate system As described in the text, further positioning of the robotic arm's end effector is required. and hand-eye position The data is then transformed into the base coordinate system of the robotic arm for stitching. During this process, the influence of joint kinematic errors and hand-eye posture errors on the accuracy of external axis pose parameter identification is ignored. The specific expression is as follows: c. Geometric Synchronous Calibration and Orientation Error Compensation. Because the hand-eye calibration algorithm neglects the impact of robotic arm joint kinematic errors and hand-eye posture errors on the accuracy of external axis pose parameter identification, error accumulation reduces the turntable calibration accuracy. Therefore, geometric calibration of the scanning viewpoint pose is an important means to reduce system orientation errors.

[0119] Geometric error synchronous modeling and analysis Geometric calibration of the viewpoint pose in a 3D vision imaging system is an important means of reducing the system's orientation error. This invention proposes a method for simultaneous geometric calibration and topline error compensation of the viewpoint pose in a 3D vision imaging system. Based on the system's kinematic model and error model, the method uses efficient and accurate visual measurements to obtain the system's orientation error as input to the error model, and solves the system error model through numerical optimization.

[0120] Simultaneously considering the kinematic errors of the robotic arm joints, hand-eye errors, and external rotation axis errors, and taking the invariance of the relative pose of the robotic arm base and the zero-position target of the turntable as a constraint, a comprehensive calibration error model of the system is constructed. In the world coordinate system... In fact: in It consists of three attitude errors , and This causes errors in the reconstructed location of the marker points. By ignoring second-order and higher-order error terms, the above equation can be expanded as follows: Among the symbols , and These represent positional errors caused by hand-eye coordination errors, robotic arm joint errors, and external axis errors, respectively.

[0121] Error parameter optimization identification and solution Based on the established theoretical model of system geometric calibration error, and using the system orientation error obtained from visual measurement as input, the calibration error model is solved using a nonlinear numerical optimization method, thereby identifying the geometric parameter errors of the measurement system. A schematic diagram of the specific geometric error identification principle is shown below. Figure 18 As shown.

[0122] Joint space error compensation Because the control system of the robotic arm is not open, users usually cannot obtain the underlying permissions of the controller to directly modify the control parameters to improve the positioning accuracy. Therefore, this invention chooses to correct the joint rotation variable in the joint configuration space, thereby compensating for the orientation error of the spatial pose of the scanner in the measurement system, improving the accuracy of the orientation movement of the robotic arm system, and thus ensuring the orientation accuracy of any scanning viewpoint pose in the workspace.

[0123] To ensure the orientation accuracy of the 3D vision imaging system at any scanning viewpoint pose, this invention utilizes precisely identified geometric error vectors to correct the system's kinematic model parameters, and then employs a method to compensate for joint angle variables in joint space. Specifically, this involves: First, using the geometric error vectors calibrated by the proposed method to correct the system's kinematic model. The corrected model closely approximates the actual kinematic model of the measurement system, thus allowing online estimation of the orientation error of the current viewpoint pose using its forward kinematic solution. Second, compensating for the pose error in joint configuration space, the corrected joint angles corresponding to the actual kinematic model are derived and calculated, ensuring accurate access to the simulated design viewpoint pose. Finally, the corrected joint angles are written to the controller in real-time via Ethernet communication, ensuring the robotic arm orientation system moves correctly toward the desired design viewpoint pose.

[0124] The online compensation process for orientation error is as follows: Figure 19 As shown.

[0125] The geometric iteration calibration is as follows: Since the step-by-step calibration method for obtaining initial pose values ​​is computationally inefficient, the calibration accuracy is limited by the motion accuracy of the robotic arm itself. This section establishes a synchronous calibration and overall compensation method for the geometric kinematic parameters among multiple working units of the robotic arm measurement system. This method integrates robotic arm joint errors, hand-eye errors, and external axis errors into the entire system and performs calibration simultaneously, suppressing the impact of robotic arm joint errors on the overall system accuracy. By accurately identifying geometric errors to correct the system's kinematic model and by online compensation for joint angle errors during measurement, the spatial orientation accuracy of any scanner viewpoint pose within the measurement system's workspace is ensured. Figure 20 illustrates the overall iterative calibration process of the proposed geometric calibration method, used to achieve higher calibration accuracy for the scanner's spatial pose.

[0126] At this point, the matrix transformation relationship between the 3D vision imaging system and the robotic arm end effector has been determined through hand-eye calibration.

[0127] Then, the coordinate systems of the robotic arm, 3D vision imaging system, and laser projector are unified. To effectively transmit real-time defect data, laser projectors are installed at fixed intervals along the sides of the layup. Target points are laid at the four vertices of the plane containing a certain layup segment, and the coordinates of these target points in the workpiece coordinate system are known. The target points are then photographed using both the calibrated robotic arm with the 3D vision imaging system and the laser projector used for the current layup segment. Next, image point recognition is performed to determine the set of target point coordinates in both the robotic arm base coordinate system and the laser projector's own coordinate system. Finally, the ICP algorithm is used to determine two sets of matrix transformation relationships, which are as follows: in: , These represent the sets of target point coordinates captured by the AFP automated wire placement defect online detection system and the laser projector, respectively. , Respectively represent and , The set of target point coordinates in the corresponding workpiece coordinate system; therefore, , This represents the rotation and translation matrix relationship between the base coordinate system and the workpiece coordinate system of the robotic arm; , This represents the rotation and translation matrix relationship between the laser projection's own coordinate system and the workpiece's coordinate system.

[0128] The workpiece coordinate system in which the layup is located includes: the center point of the plane where the carbon fiber filaments are laid is the origin of the coordinate system, the xoy plane is the plane where the carbon fiber filaments are laid, and the Z-axis points vertically to the filament layup machine.

[0129] Finally, multiple laser projectors relay the dynamic, real-time feedback of defect locations. Since the robotic arm, 3D vision imaging system, and multiple laser projectors are all operating within the workpiece coordinate system, the 3D imaging system, as it moves to different layup segments, can dynamically transmit and convert defect data (including defect type and coordinates) to the corresponding laser projector for that layup segment in real time. The laser projector for the current layup segment then dynamically relays the defect data back to the host computer and projects the coordinates of all defects, facilitating defect repair work for the operator. This host computer is primarily used for data transmission and reception throughout the entire inspection process.

[0130] Laser projector relay mechanism, such as Figure 21 As shown.

[0131] The preset area divisions are as follows: All laser projectors were fixed at the same height at the top of the measurement space. And the specifications are completely consistent. The projection area of ​​the first laser projector is the starting point of the composite layer. Nearby, its projected area size is: (The length of the projection area is) The width of the projection area is The robotic arm is fixed at a distance from the start of the layer layup. The position. Subsequent intervals The length is fixed with one set of laser projectors; after the projection areas of each laser projector are connected, they cover all the layers.

[0132] The principle of laser projection is as follows: First, take the area covered by the robotic arm as an example. The layup (the laser projector is always positioned directly above the center axis of the layup, the layup length) Not less than the laser projection length The layup width W must not exceed the laser projection width. ( ) is a measurement cycle, with each interval length being... At that time, the plane where the ply is located Target points are laid at the four corners. Then, the coordinates of all target points in the workpiece coordinate system are known through photogrammetry. The target points within the projection area are photographed using a laser projector directly above the layup plane. Next, the matrix transformation relationship between the workpiece coordinate system and the laser projector's own coordinate system is determined using the ICP matching algorithm. Finally, the laser projector receives the coordinates of the defect points transmitted back from the 3D vision imaging system and applies the solved matrix transformation relationship between the workpiece coordinate system and the laser projector's own coordinate system to transform the coordinates of the defect points in the machine coordinate system to the workpiece coordinate system, that is, to project the projection points back onto the layup plane.

[0133] The cross-regional defect handling strategy is as follows: First, the required spacing between projectors is: The length is such that there is a common field of view between each pair of projectors (i.e., the defect point can be observed by both sets of projectors at the same time); then, for the defect point within the common field of view, the shortest distance method is used to determine its projection relationship, that is, by calculating the distance from the defect point to the origin of the coordinates of the two sets of projectors respectively, the one with the shortest distance is responsible for projecting the defect point onto the ply plane.

[0134] The system supports software-generated defect detection reports. The AFP automated wire placement defect online detection software can store detected defect data and create traceable quality reports within the software, with data records retained for up to 90 days. Operators can then generate corresponding quality reports based on the specific detection results, including the defect type, defect location coordinates, and defect quantity.

[0135] The experimental example demonstrates the construction of an AFP (Automatic Fiber Placement) online defect detection system. This system enables real-time analysis of fiber placement defects, detecting and identifying potential gaps, overlaps, and appearance defects between fiber bundles and tapes during the fiber placement process. (See also...) Figure 22 The system includes: a robotic arm 1, an industrial camera group 2 (with a SAM lens), a line laser 3, lighting auxiliary equipment 4 (LED strobe light source), an image acquisition and processing system 5, defect detection and analysis software 6, a laser projector 7, a calibration plate 8, and a positioning target 9. It can meet the detection requirements of automatic laying of unidirectional thermosetting carbon fiber composite materials within 80mm, supports high-speed laying detection at 80m / min, and can achieve a defect recognition rate of about 95% with about 500 training samples.

[0136] The robotic arm 1 carries the industrial camera 2, line laser 3, and lighting auxiliary equipment 4, and moves synchronously with the automatic fiber placement device 12. The industrial camera assembly 2, together with the line laser 3 and lighting auxiliary equipment 4, forms a 3D vision imaging system 11, fixed to the end of the robotic arm 1, used to acquire images of the laser stripes projected by the line laser 3 and 2D images of the layup layers near the stripes. A SAM lens with a fixed tilt angle (currently 12.49°) is fixed to the front of the industrial camera assembly 2.

[0137] A line laser 3 is fixed at the end of the robotic arm 1 and forms a reflective measurement structure with the industrial camera group 2. When the lighting auxiliary equipment 4 is turned on, it projects a thin laser line onto the laid carbon fiber layer.

[0138] The lighting auxiliary device 4 is also fixed at the end of the robotic arm 1 and starts synchronously with the industrial camera group 2 and the line laser 3. It is located in the middle of the line connecting the industrial camera group 2 and the line laser 3, providing a stable strobe white light source for the laser light stripe image and the layered 2D image.

[0139] Image acquisition and processing system 5, namely image acquisition card, transmits images captured by industrial camera group 2 to image acquisition card through high-speed communication gigabit network interface. Image acquisition card performs preliminary processing on the acquired layer images and feeds the information back to the next defect detection and analysis software 6.

[0140] The defect detection and analysis software 6 analyzes and processes image data and identifies defects, and then feeds the defect data back to the laser projector 7 via the host computer.

[0141] Laser projector 7 acquires the defect data transmitted from the host computer and accurately projects the coordinates of the defects that need to be located onto the laid carbon fiber layer to reflect the defect area.

[0142] Calibration board 8 is used to complete the SAM calibration of industrial camera group 2, the optical knife calibration of line laser 3, and the hand-eye calibration of robotic arm.

[0143] Target 9 is used to solve the matrix transformation relationship between the base coordinate system of the robotic arm and the workpiece's own coordinate system, as well as the matrix transformation relationship between the laser projector's own coordinate system and the workpiece's coordinate system, to facilitate the transmission and projection of defect data.

[0144] Small sample training experiment Dataset composition (500 original images): The industrial defect detection small sample dataset constructed in this experiment contains 500 original images with no duplication. No augmented images were used as the basic training samples. All images are from real industrial production scenarios.

[0145] The dataset covers six typical defects: gaps, overlaps, wrinkles, bubbles, foreign objects, and missing wires. Among them, gaps and overlaps are high-frequency core defects, accounting for a higher percentage of samples than other defects, which is consistent with the defect distribution pattern in actual production. The specific sample distribution is shown in Table 1 below: Table 1 Comparison of model accuracy before and after compression: A lightweight pruning and quantization combined compression strategy was adopted, which significantly reduced the number of model parameters and inference time while maintaining a 95% recognition rate. The comparison data is shown in Table 2 below: Table 2 Achieving a 95% recognition rate.

[0146] During the model training phase, the batch size was set to 2, and a total of 120 iterations (epochs) were executed. The Adam optimizer was selected, and the initial learning rate was set to 0.0002, along with a MultiStep decay strategy to ensure smooth model convergence.

[0147] In summary, this invention addresses the current technical bottlenecks in AFP defect detection technology by proposing a 3D vision-based online detection method and system for robotic automated fiber placement defects. This system offers several significant advantages: 1. It proposes a high-precision 3D vision imaging system integrating Sham lens imaging and monocular laser scanning, solving the high-quality imaging requirements under conditions of large depth of field, short distance, and low reflectivity during robotic online fiber placement. 2. The multimodal fiber placement defect rapid localization and identification technology based on 2D images and 3D point clouds resolves the contradiction between the singularity of modal geometric features and the diversity of defect types. While maintaining measurement accuracy, it significantly improves defect detection efficiency and broadens the types of defects that can be detected. 3. It proposes a real-time positioning technology for a robotic arm, a 3D vision imaging system, and a laser projector, meeting the measurement needs for real-time defect data detection and response, truly realizing online defect detection technology for composite materials. Through the above technical approach, high-speed fiber placement detection at 80m / min can be achieved, with a defect recognition rate of over 95% using a small sample model of approximately 500 images.

[0148] The above description, in conjunction with specific / preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various substitutions or modifications can be made to these described embodiments without departing from the inventive concept, and all such substitutions or modifications should be considered within the scope of protection of the present invention.

Claims

1. A 3D vision-based robot automatic fiber placement defect online detection method, characterized in that, The method comprises the following steps: S1, a set of 3D vision imaging systems based on Sham lens and line laser is constructed and calibrated, so as to follow the synchronous movement of the fiber laying mechanical arm, collect the two-dimensional image of the laying layer containing the laser light strip in line, and reconstruct the corresponding three-dimensional point cloud data based on the laser light knife plane equation; S2, fusion analysis is performed on the two-dimensional image and the three-dimensional point cloud data: a lightweight deep learning network based on small sample training is used for real-time defect detection and preliminary positioning of the two-dimensional image, and the normal vector and curvature information of the three-dimensional point cloud are used for defect area positioning and segmentation, and then a collaborative representation model is used to fuse the two-dimensional texture features and three-dimensional geometric features, so that the accurate identification and positioning of the fiber laying defect are realized; S3, the defect information identified and positioned is converted in the coordinate system of the mechanical arm, the 3D vision imaging system and the plurality of laser projection devices which have been calibrated by the target point, and the plurality of laser projection devices are controlled to project the defect position coordinates to the laying surface in a segmented relay manner for feedback.

2. The method of claim 1, wherein, The calibration process of the 3D vision imaging system in step S1 comprises: A calibration board with regularly arranged mark points is used, calibration board images are collected at different poses, and the internal and external parameters of the camera with Sham lens and the tilt parameters of the lens plane relative to the camera sensor plane are solved; In the case of keeping the pose of the calibration board unchanged, the line laser is turned on and the image with the laser light strip is collected, the center pixel coordinates of the laser light strip in each image are extracted, and all the light strip center points are calculated back to the three-dimensional coordinates in the camera coordinate system by combining the solved camera parameters, and finally the light knife plane equation of the line laser is obtained by fitting.

3. The method of claim 2, wherein, The improved Steger algorithm is used to extract the pixel coordinates of the laser light strip center points, which comprises: The image is traversed by rows, a one-dimensional convolution kernel is used to filter the grayscale, and the integer pixel position area of the laser light strip is preliminarily positioned; Within the preliminarily positioned area, the first and second order gradients of the image are calculated, the Hessian matrix is constructed, and the eigenvector corresponding to the maximum eigenvalue is solved, and the accurate light strip center point coordinates are obtained through sub-pixel calculation.

4. The method of claim 1, wherein, The lightweight deep learning network based on small sample training in step S2 is constructed by the following methods: A feature extraction network is constructed based on a multi-layer perceptron architecture to capture the local correlation and edge detail information of the defects in the two-dimensional image; A strategy combining model pruning and knowledge distillation is used to compress the feature extraction network, which reduces the model size while maintaining its recognition accuracy; Transfer learning technology is used to migrate the model parameters pre-trained based on public data sets to the target network, and a small amount of target new class samples are used to fine-tune the high-level parameters of the network to enhance the cross-scene adaptability of the model to the fiber laying defects.

5. The method of claim 1, wherein, The establishment of the collaborative representation model in step S2 comprises: The extracted two-dimensional image features and three-dimensional point cloud features are respectively mapped to their respective representation spaces through projection functions; An optimization model is constructed with a target function of minimizing reconstruction error between different modalities, and a collaborative representation matrix for fusing two-dimensional and three-dimensional modal features to perform joint defect identification and positioning is obtained by solving the optimization model.

6. The method of claim 1, wherein, The process of unifying the coordinate system in step S3 includes: The relative pose relationship between the 3D visual imaging system and the end of the mechanical arm is determined through hand-eye calibration; Target points with known coordinates of the workpiece coordinate system are arranged at fixed positions on the laying plane, and the target points are photographed and recognized by the 3D visual imaging system and each laser projection device respectively; The rotation and translation transformation matrices from the base coordinate system of the mechanical arm to the workpiece coordinate system and from the coordinate system of each laser projection device to the workpiece coordinate system are calculated respectively by using the iterative closest point algorithm, so as to realize the unification of all systems in the workpiece coordinate system.

7. The method of claim 1, wherein, The segmented relay projection mode in step S3 specifically includes: A plurality of laser projection devices are arranged at fixed intervals along the laying direction, and there is overlap between the projection regions of adjacent projection devices; When the 3D visual imaging system moves to different laying sections and performs detection, the recognized defect coordinates are converted to the workpiece coordinate system in real time, and according to the real-time spatial position information of the 3D visual imaging system, the laser projection device corresponding to the current laying section is dynamically assigned and sent; For defect coordinates located in the overlapping region of adjacent projection devices, the projection device responsible for projection is selected according to a preset attribution determination rule; The laser projection device receives the defect coordinate information and projects it accurately to the corresponding position on the laying surface, thereby realizing dynamic and real-time visual feedback of the defect position during the entire laying process. 8.A 3D vision-based robot automated fiber placement defect online detection system, characterized in that, It includes: A mechanical arm for carrying and moving during the fiber laying process; A 3D visual imaging subsystem fixed to the end of the mechanical arm, including a camera with a Scheimpflug lens, a line laser, and a stroboscopic light source, for online acquisition of two-dimensional images of the laying and synchronous reconstruction of three-dimensional point cloud data; An image processing and defect analysis subsystem in communication with the 3D visual imaging subsystem, for fusion analysis of the acquired two-dimensional images and three-dimensional point cloud data according to the method of any one of claims 1 to 7, to realize identification and positioning of defects; A multi-laser projection subsystem including a plurality of laser projection devices arranged at intervals along the laying direction, for receiving defect coordinate information output by the image processing and defect analysis subsystem and projecting defect positions in real time to the laying surface under a unified coordinate system; Wherein, the mechanical arm, 3D visual imaging subsystem, and multi-laser projection subsystem are unified under the same workpiece coordinate system through calibration.

9. The system of claim 8, wherein, The image processing and defect analysis subsystem includes: An image acquisition card for receiving raw image data acquired by the 3D visual imaging subsystem through a high-speed communication interface and performing preliminary processing; A defect detection and analysis software running on an upper computer, integrated with a lightweight deep learning network based on small sample training and a multi-modal fusion algorithm, for accurate identification and positioning of defects on the preliminarily processed image data, and sending the results to the multi-laser projection subsystem.

10. The system of claim 8, wherein, The multi-laser projection subsystem works in a segmented relay mode: Each laser projection device is associated with a specific layering section area, and the projection areas of adjacent devices partially overlap; The defect detection and analysis software dynamically determines the current working layering section according to the real-time position of the mechanical arm or 3D vision imaging subsystem, and distributes and sends the defect coordinate information to the laser projection device corresponding to the layering section; For defect coordinates located in the overlapping area of adjacent projection devices, the defect detection and analysis software determines the projection device responsible for projection according to a preset attribution determination rule; specifically, for defect coordinates located in the overlapping area of adjacent projection devices, the spatial distances of the defect coordinates to the coordinate origins of each adjacent projection device are respectively calculated, and the one with the shortest distance is selected as the projection device responsible for projection; The laser projection device receiving the information projects the defect coordinates to the surface of the layering section responsible for it, thereby realizing continuous and seamless visualization indication of the defect positions on the long-distance layering line.

Images