Automatic fiber placement and layer placement surface defect detection device and method

Abstract

The invention discloses an on-line detection device and method for surface defects of an automatic fiber placement laying layer, and belongs to the field of aeronautical manufacturing visual inspection, the automatic fiber placement laying layer surface defect detection device comprises a multi-line structure light source, an industrial camera and an illumination light field light source; the multi-line structure light source and the industrial camera point to the surface of the laying layer, and the multi-line structure light source is used for projecting a multi-line structure light field; the illumination light field light source is arranged above the surface of the paving layer and is used for projecting an illumination light field; the industrial camera is matched with the illumination light field light source and the multi-line structure light source to collect two-dimensional images and three-dimensional point cloud information of the surface of the laying layer. Compared with an existing method of independently using a laser contourgraph or a two-dimensional camera, the method has the advantages of simple structure, multiple types of information capable of being collected, low cost, multiple defect recognition types, high defect recognition accuracy, high applicability and the like.

Description

Technical Field

[0001] The present invention relates to the technical field of aviation manufacturing machine vision detection, and in particular to an automatic wire laying layer surface defect detection device and method. Background Art

[0002] In the field of aerospace industry, advanced composite materials have become the most important component materials. Among them, carbon fiber composite materials are the most widely used in aircraft. Compared with traditional metal materials, carbon fiber composite materials have high strength-to-weight ratio, better flexibility and adaptability to complex shapes. Carbon fiber composite materials have been widely used in the latest aircraft manufacturing.

[0003] In order to improve the manufacturing efficiency of composite parts, the Automatic Fiber Placement (AFP) technology was proposed in the 1980s. According to the type of composite materials used, the AFP technology can be divided into thermosetting fiber placement technology and thermoplastic fiber placement technology. Various defects will occur in the AFP production process. Typical defects include gaps, overlaps, missing fibers, twisting, bubbles, bridges, wrinkles, flanging, etc. These defects will have a huge negative impact on the strength, stiffness and other properties of composite parts. It is necessary to perform online detection of defects generated in the production process. The traditional detection method is to use manual detection. Manual detection takes a long time. This manual detection accounts for about 50% of the total production time, which will cause long downtime and reduce production efficiency. Manual detection relies on the experience of workers, and there will be problems of personnel fatigue during long-term detection, which further reduces the accuracy and efficiency of detection and makes it difficult to meet the needs of actual production.

[0004] At present, the commonly used online detection technology of AFP ply surface defects can be divided into three categories according to the type of information used: detection methods based on visible light two-dimensional image information, detection methods based on infrared thermal images, and detection methods based on three-dimensional point cloud information. Each method has defect types that are difficult to detect. The key problem of existing research is that due to the relatively insufficient number of defect data sets, the training effect of the neural network for online detection of AFP surface defects using single modal information is not good, resulting in each detection method having its own defects that are not suitable for detection, and each method has its own unsuitable scenarios, and it is impossible to detect all defects using only one information or method. Summary of the invention

[0005] Aiming at the difficult problem of online imaging and accurate detection of various defects on the surface of automatic wire placement plies of composite materials, the present invention proposes an automatic wire placement ply surface defect detection device and method, which can realize online detection of various types of defects on the surface of automatic wire placement plies of composite materials.

[0006] In order to achieve the above-mentioned invention object. The technical solution adopted by the present invention is: In a first aspect, the present invention provides an automatic wire laying layer surface defect detection device, comprising: a multi-line structured light source, an industrial camera and an illumination light field light source; The multi-line structured light source and the industrial camera are both directed toward the pavement surface, and the multi-line structured light source is used to project a multi-line structured light field; the illumination light field light source is arranged above the pavement surface, and is used to project an illumination light field; The industrial camera cooperates with the illumination light field light source and the multi-line structure light source to collect two-dimensional images and three-dimensional point cloud information of the pavement surface.

[0007] As a further improvement of the present invention, it also includes a connecting plate connected to the wire placing head, and the multi-line structured light source and the industrial camera are respectively mounted on the connecting plate through an adjustable rotating table.

[0008] As a further improvement of the present invention, a filter is also installed at the front end of the industrial camera.

[0009] As a further improvement of the present invention, the illumination light field light source is arranged on both sides of the multi-line structure light source beam.

[0010] In a second aspect, the present invention provides a method for detecting surface defects of an automatic wire laying layer, comprising: Acquire synchronous data of 2D images and 3D point cloud information of the ply surface; Align the collected 2D image and 3D point cloud, and then perform preprocessing; Perform data augmentation on the preprocessed 2D images and 3D point clouds, annotate the defects on the ply surface, construct a ply surface defect dataset, and divide the dataset into training set, test set, and validation set; Construct a multimodal fusion defect detection neural network with two-dimensional images and three-dimensional point clouds as input, input the data set into the multimodal fusion defect detection network, and perform training, testing and verification based on the training set, test set and validation set; the trained multimodal fusion defect detection neural network model can use two-dimensional images and three-dimensional point clouds as input to perform online detection, identification and positioning of ply surface defects.

[0011] As a further improvement of the present invention, the synchronous acquisition data of the two-dimensional image and the three-dimensional point cloud information of the pavement surface is that the multi-line structured light source and the illumination light field light source periodically project the multi-line structured light field and the illumination light field under the control of a computer, so as to realize the synchronous acquisition of the two-dimensional image and the three-dimensional point cloud information of the pavement surface by the industrial camera, specifically including: When the illumination light field is projected, the multi-line structured light field is turned off, and the industrial camera collects a two-dimensional image of the surface of the wire laying in the field of view; When the multi-line structured light field is projected, the illumination light field is turned off and the industrial camera obtains the three-dimensional point cloud information of the ply surface.

[0012] As a further improvement of the present invention, the industrial camera collects a two-dimensional image of the surface of the laid wire in the field of view, specifically including: The optical axis of the industrial camera is made to approximately satisfy the reflection angle relationship with the line structured light. Under lower exposure time and lower multi-line structured light source power, obvious line structured light patterns are collected to complete the acquisition of three-dimensional information. The angle between the illumination light field and the industrial camera does not satisfy the reflection angle, so that the illumination light field is evenly irradiated on the surface of the wire bundle to assist in the acquisition of two-dimensional information.

[0013] As a further improvement of the present invention, the synchronous acquisition of the two-dimensional image and the three-dimensional point cloud information of the pavement surface by the industrial camera is achieved by periodically switching the multi-line structured light field and the illumination light field, thereby achieving the synchronous acquisition of the two-dimensional image and the three-dimensional point cloud, including: The time taken to collect the 2D image and 3D information of the ply surface within the camera field of view is , the time for projecting the illumination light field is , the time for projecting multi-line structured light is , satisfying the relationship The projection sequence is: in one acquisition cycle, the illumination light field is first projected to acquire a two-dimensional image of the ply surface with a certain field of view size; then, multi-line structured light is projected to complete the scanning and reconstruction of the ply surface within the field of view, and this cycle is repeated.

[0014] As a further improvement of the present invention, the marking of defects on the surface of the ply includes: marking missing wires, overlaps, excessive gaps, wrinkles, bulges / bubbles, foreign objects, flanges, and bridging defects.

[0015] . According to the method for detecting surface defects of an automatic wire laying layer, the method is characterized in that the collected two-dimensional image and three-dimensional point cloud are aligned and then pre-processed, comprising the following steps: By using the feature point extraction method, feature points are extracted from the two-dimensional image and the three-dimensional point cloud respectively; the extracted feature points are described using feature descriptors, and the nearest neighbor algorithm is used to match the feature points between the two data sets, establish a corresponding relationship between the two, and achieve alignment; After alignment, the median filter method is used to denoise the two-dimensional image to remove the salt and pepper noise caused by the uneven surface of the ply. Use adaptive histogram equalization to enhance the two-dimensional image and eliminate image blur caused by uneven lighting; Perform sharpening operations on two-dimensional images; After alignment, the 3D point cloud is downsampled and locally fitted and filtered to fit the ply plane, and points far from the fitting plane are removed.

[0016] The present invention has the following beneficial effects: The present invention is based on a detection device that is fused with two-dimensional images and three-dimensional point clouds. The detection device is based on an industrial camera + an illumination light field light source + a multi-line structured light source, and periodically projects an illumination light field and a multi-line structured light field. They respectively form a two-dimensional image acquisition system and a multi-line structured light system with the industrial camera to achieve synchronous acquisition of two-dimensional images and three-dimensional point clouds, and perform noise reduction, enhancement and alignment on the acquired two-dimensional images and three-dimensional point clouds. The two-dimensional images and three-dimensional point clouds are respectively annotated to produce a data set of surface defects of plies, and a defect detection network that fuses two-dimensional images and three-dimensional point clouds is constructed and trained to achieve online detection of various types of defects on the surface of automatic wire laying plies of composite materials.

[0017] The detection method of the present invention can synchronously collect the two-dimensional image and three-dimensional information of the surface of the automatic fiber laying layer through light field switching, and can be used together for online detection of surface defects of the layer. Compared with the existing method of using a laser profiler or a two-dimensional camera alone, the detection method has the advantages of simple structure, multiple types of information that can be collected, low cost, multiple types of defect recognition, high defect recognition accuracy, and strong applicability. The present invention takes into account the optical properties of the layer surface, and adopts the oblique laser triangulation method to construct a line structured light system. An angle adjustment device is installed at the industrial camera and the multi-line structured light source respectively, and the appropriate angle can be adjusted according to different fiber laying machines. In addition, the industrial camera can choose whether to install the filter 5 according to the curing method of the laid carbon fiber, and is suitable for thermosetting and thermoplastic automatic fiber laying machines at the same time, and has strong applicability. The present invention can effectively detect and identify a variety of defects such as missing wires, overlaps, excessive gaps, wrinkles, bulges / bubbles, foreign objects, flanging, bridging, etc. by fusing the collected two-dimensional images with three-dimensional information, and improving the detection accuracy and speed, as well as the types and accuracy of defect recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1a It is a schematic diagram of a device for synchronously collecting two-dimensional images and three-dimensional information on the surface of composite material automatic fiber laying based on dual light field switching of the present invention; Figure 1b It is a schematic diagram of the device for synchronously collecting two-dimensional images and three-dimensional information of the surface of the composite material automatic fiber laying layer based on dual light field switching of the present invention being installed on a thermoplastic automatic fiber laying machine; Figure 2 A schematic diagram of the projection sequence and the type of information collected of the periodic projection illumination light field and the multi-line structured light field of the present invention; Figure 3 A two-dimensional image of the ply surface of the present invention; Figure 4a A three-dimensional point cloud top view of the ply surface of the present invention; Figure 4b A three-dimensional point cloud side view of the ply surface of the present invention; Figure 5 It is a flow chart of the automatic wire laying layer surface defect detection method based on two-dimensional and three-dimensional information fusion of the present invention; Among them: 1 is a multi-line structure light source; 2 is an adjustable rotating table; 3 is a connecting plate; 4 is an industrial camera; 5 is a filter; 6 is an illumination light field light source; 7 is a carbon fiber tow; 8 is a matrix; 9 is a laser beam; 10 is a pressure roller; 11 is a laser heater; 12 is a wire cutter; 13 is a prepreg yarn tube; 14 is a wire laying head. DETAILED DESCRIPTION

[0019] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. On the contrary, they are only examples of devices and methods consistent with some aspects of the present invention as detailed in the attached claims. The following embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention.

[0020] The present invention provides an automatic wire laying layer surface defect detection device, such as Figure 1a and Figure 1b As shown, it includes: a multi-line structured light source 1; an adjustable rotating table 2; an industrial camera 4; a filter 5; an illumination light field light source 6; and a computer.

[0021] The multi-line structured light source 1 and the industrial camera 4 are respectively mounted on a connecting plate 3 through an adjustable rotating table 2, and the connecting plate 3 is a connecting plate between the detection device and the fiber placement head 14. The angles of the multi-line structured light source 1 and the industrial camera 4 can be adjusted, and both point to the carbon fiber tow 7 on the substrate 8.

[0022] A filter 5 may be installed at the front end of the industrial camera 4 ; the filter 5 may be installed or not installed according to different materials.

[0023] Illuminating light field light sources 6 are also arranged on both sides above the carbon fiber tow 7 .

[0024] The fiber placement head 14 is provided with a prepreg bobbin 13 and a laser heater 11; a pressing roller 10 is provided at the lower end of the fiber placement head 14, and the pressing roller 10 rolls to form a carbon fiber tow 7, and a laser beam 9 of the laser heater 11 is directed toward the bottom of the pressing roller 10. A wire cutter 12 is also provided inside the fiber placement head 14.

[0025] As a specific example, the multi-line structure light source 1 of the present invention can be a multi-line laser or an industrial projector. The industrial camera 4 can be a color CMOS industrial camera. The illumination light field light source 6 can be an LED illumination light field light source. Of course, other types can be selected according to needs, and the specific device selection is not limited in this embodiment.

[0026] When collecting data, the projection time of the illumination light field light source 6 in a single cycle is calculated according to the movement speed of the fiber laying head 14 and the camera exposure time, and the periodic switching between the illumination light field light source 6 and the multi-line structure light source 1 is controlled. Line structured light is reconstructed in three dimensions based on laser triangulation. The image collected by one exposure can only reconstruct the contour of the laser line. The entire ply surface needs to be scanned in coordination with the movement of the fiber laying head 14 to reconstruct the complete ply surface. Two-dimensional images are collected under the illumination light field. The image collected by one exposure contains two-dimensional information of the ply surface within the entire field of view. Therefore, in one projection cycle, the proportion of time for projecting multi-line structured light is relatively large.

[0027] Among them, the multi-line structured light source 1 and the illumination light field light source 6 can periodically project the multi-line structured light field and the illumination light field under computer control to achieve the synchronous acquisition of the two-dimensional image and the three-dimensional point cloud of the pavement surface: 1) When the illumination light field is projected, the multi-line structured light field is turned off, and the camera collects a two-dimensional image of the surface of the wire laying in the field of view at a low exposure time; 2) When the multi-line structured light field is projected, the illumination light field is turned off. At this time, the entire system is equivalent to a multi-line structured light 3D measurement system, which can obtain 3D point cloud information of the ply surface.

[0028] The industrial camera 4 and the multi-line structured light source 1 are respectively mounted on an adjustable rotating table 2. Considering the optical properties of the surface of the wire bundle, the rotating table 2 is adjusted so that the optical axis of the industrial camera 4 approximately satisfies the reflection angle relationship with the line structured light, and can collect obvious line structured light patterns at a lower exposure time and a lower power of the multi-line structured light source 1 to complete the acquisition of three-dimensional information. The angle between the illumination light field and the industrial camera 4 does not satisfy the reflection angle, so that the illumination light field is evenly irradiated on the surface of the wire bundle to assist in the acquisition of two-dimensional information. The detection device can realize the synchronous acquisition of two-dimensional images and three-dimensional point clouds on the surface of the automatic wire laying layer of the composite material.

[0029] The width, spacing and number of the projected multi-line structured light strips can be adaptively adjusted according to the surface morphology and defect information of the processed workpiece; The industrial camera 4 can choose whether to install the filter 5 according to the curing method of the laid carbon fiber: when testing the thermoplastic automatic fiber laying machine, the filter 5 can be installed to filter out the strong light interference of the laser and other heaters of the thermoplastic fiber laying machine. It is applicable to both thermosetting and thermoplastic automatic fiber laying machines; The synchronous acquisition of 2D images and 3D point clouds can be achieved through the periodic switching of multi-line structured light fields and illumination light fields, including: The time taken to collect the 2D image and 3D information of the ply surface within the camera field of view is , the time for projecting the illumination light field is , the time for projecting multi-line structured light is , satisfying the relationship The projection sequence is: in one acquisition cycle, the illumination light field is first projected to collect a two-dimensional image of the ply surface with a certain field of view, and then multi-line structured light is projected to complete the scanning and reconstruction of the ply surface within the field of view, and this cycle is repeated; The rotating stage 2 is adjusted so that the optical axis of the industrial camera 4 approximately satisfies the reflection angle relationship with the line structured light, so that a clear and high-contrast line structured light pattern can be collected at a lower exposure time and a lower power of the multi-line structured light source 1 to complete the acquisition of three-dimensional information. The angle between the illumination light field and the industrial camera 4 is adjusted not to satisfy the reflection angle, so that the illumination light field is evenly irradiated on the surface of the tow, so as to assist in the acquisition of two-dimensional information and obtain a high-quality two-dimensional image; The width, interval and number of the projected multi-line structured light strips can be adaptively adjusted according to the surface morphology and defect information of the workpiece being processed, including: When processing a flat surface, the width, spacing and number of light strips are taken as normal values; when processing a large curved surface or when a large number of surface defects are detected, the width of the light strips is reduced, the spacing of the light strips is reduced and the number of light strips is increased to improve the measurement accuracy and spatial resolution; Only one industrial camera 4 + other components are needed to realize the synchronous acquisition of two-dimensional images and three-dimensional point clouds on the surface of automatic wire laying and plying of composite materials, and input the acquired two-dimensional images and three-dimensional point cloud information into subsequent detection algorithms.

[0030] There are two hardware selection methods for using the multi-line structured light source 1: 1) using a multi-line laser as the multi-line structured light source 1 to generate a multi-line structured light field; 2) using an industrial projector as the multi-line structured light source 1 to generate a multi-line structured light field.

[0031] Periodically project the multi-line structured light field and the illumination light field: 1) When the illumination light field is projected, the multi-line structured light field is turned off, and the industrial camera 4 collects a two-dimensional image of the surface of the laid wire in the field of view at a low exposure time; 2) When the multi-line structured light field is projected, the illumination light field is turned off. At this time, the entire system is equivalent to a multi-line structured light 3D measurement system, which can obtain 3D point cloud information of the laid surface. By switching between the two light fields, a single camera can simultaneously collect 2D images and 3D point cloud information.

[0032] The rotating stage 2 is adjusted so that the optical axis of the industrial camera 4 approximately satisfies the reflection angle relationship with the line structured light, and a clear line structured light pattern can be collected at a lower exposure time and a lower power of the multi-line structured light source 1 to complete the acquisition of three-dimensional information. The angle between the illumination light field and the industrial camera 4 is adjusted not to satisfy the reflection angle, so that the illumination light field is evenly irradiated on the surface of the tow to assist in the acquisition of two-dimensional information.

[0033] The system adaptively adjusts the width, spacing and number of the projected multi-line structured light according to the surface morphology and defect information of the workpiece being processed: when processing flat surfaces, the width, spacing and number of the light strips take conventional values; when processing large curvature surfaces or when many surface defects are detected, the light strip width is reduced, the spacing of the light strips is reduced and the number of light strips is increased to improve the measurement accuracy and spatial resolution.

[0034] The industrial camera 4 can choose whether to install the filter 5 according to the curing method of the laid carbon fiber: when detecting the thermosetting automatic fiber laying machine, since the thermosetting automatic fiber laying machine does not have the strong light interference of the laser heating head, it is not necessary to install the filter 5; when detecting the thermoplastic automatic fiber laying machine, since the thermoplastic automatic fiber laying machine has the strong light interference of the laser heating head, the filter 5 is installed to filter out the strong light interference of the laser heater 11 of the thermoplastic fiber laying machine. Therefore, the detection device of the present invention can be applied to both thermosetting and thermoplastic automatic fiber laying machines.

[0035] The present invention also provides a method for detecting surface defects of automatic wire laying layers based on fusion of two-dimensional and three-dimensional information, comprising: 1) The 2D image and 3D information of the ply surface are collected through the automatic wire laying ply surface defect detection device based on the fusion of 2D and 3D information; 2) Perform high-precision alignment on the collected 2D images and 3D point clouds, and then perform preprocessing such as denoising, contrast equalization, cropping, sharpening, and downsampling; 3) Perform data augmentation on the 2D images and 3D point clouds, annotate the defects on the ply surface, construct a ply surface defect dataset, and divide the dataset into training set, test set, and validation set; 4) Construct a multimodal fusion defect detection neural network with both 2D images and 3D point clouds as input, and input the data set into the multimodal fusion defect detection network for training, testing and verification. The trained model can use 2D images and 3D point clouds as input to realize online detection, identification and positioning of ply surface defects.

[0036] Furthermore, in one embodiment of the present invention, dual light field switching can be achieved by using either of the two methods of "multi-line laser + illumination light field light source 6" as the system light source or "industrial projector + illumination light field light source 6" as the system light source.

[0037] Exemplarily, when "multi-line structured light source 1 + illumination light field light source 6" is used as the system light source, the optical properties of the surface of the tow are taken into consideration, and the rotating table 2 is adjusted so that the industrial camera 4 and the multi-line structured light source 1 satisfy the reflection angle relationship, so that high-definition and high-contrast line structured light patterns can be collected at a lower exposure time and a lower power of the multi-line structured light source 1 to complete the acquisition of three-dimensional information. At the same time, the illumination light field light source 6 is installed on both sides of the ply surface, so that the angle relationship between the light source of the illumination light field and the industrial camera 4 deviates from the reflection relationship, ensuring that the illumination light field is evenly irradiated on the surface of the tow, so as to assist in the acquisition of two-dimensional information and obtain a high-quality two-dimensional image.

[0038] The rotating stage 2 is adjusted so that the optical axis of the industrial camera 4 approximately satisfies the reflection angle relationship with the line structured light, and a high-definition and high-contrast line structured light pattern can be collected at a lower exposure time and a lower power of the multi-line structured light source 1 to complete the acquisition of three-dimensional information. The angle between the illumination light field and the industrial camera 4 is adjusted so that the illumination light field is evenly irradiated on the surface of the tow to assist in the acquisition of two-dimensional information.

[0039] In an embodiment of the present invention, the device adaptively adjusts the width, spacing and number of light strips of the projected multi-line structured light according to the surface morphology and defect information of the processed workpiece: when processing a flat surface, the width, spacing and number of light strips are conventional values; when processing a curved surface or when a large number of surface defects are detected, the width of the light strips is reduced, the spacing of the light strips is reduced and the number of light strips is increased, thereby improving the measurement accuracy and spatial resolution; In addition, the industrial camera 4 can choose whether to install the filter 5 according to the curing method of the laid carbon fiber: when detecting the thermosetting automatic fiber laying machine, since the thermosetting automatic fiber laying machine does not have the strong light interference of the laser heating head, the filter 5 does not need to be installed; when detecting the thermoplastic automatic fiber laying machine, since the thermoplastic automatic fiber laying machine has the strong light interference of the laser heating head, the filter 5 is installed to filter out the strong light interference of the laser heater 11 of the thermoplastic fiber laying machine. Therefore, the detection device can be applied to both thermosetting and thermoplastic automatic fiber laying machines.

[0040] After collecting the 2D image and 3D point cloud of the ply surface, alignment and preprocessing are required, including the following steps: 1) Alignment method of 2D image and 3D point cloud: By using feature point extraction algorithm, feature points are extracted from 2D image and 3D point cloud respectively. Feature descriptors are used to describe the extracted feature points, and the nearest neighbor algorithm is used to match the feature points between the two data sets to establish the corresponding relationship between the two.

[0041] 2) Use the median filter method to denoise the two-dimensional image and remove the salt and pepper noise caused by the uneven surface of the ply.

[0042] 3) Use adaptive histogram equalization to enhance the two-dimensional image and eliminate image blur caused by uneven lighting; 4) Perform Unsharp Mask sharpening operation on the two-dimensional image to make the edges of the tow and the outlines of the defects clearer, which is conducive to the recognition of subsequent detection algorithms.

[0043] 5) Downsample the 3D point cloud to reduce the amount of point cloud data, perform local fitting filtering, fit the ply plane, and remove points far from the fitting plane to eliminate noise points.

[0044] Data augmentation and data set production use a combination of geometric transformation + DCGAN to enhance the data of two-dimensional images and three-dimensional point clouds, increase the sample size of the data set, annotate typical defects such as missing wires, overlaps, large gaps, wrinkles, bulges / bubbles, foreign objects, flanging, bridging, etc., construct a data set of ply surface defects, and divide the data set into training set, test set, and validation set.

[0045] Construct a deep learning neural network for defect detection whose input is two-dimensional images and three-dimensional point cloud data and whose output is the type and location of defects. In the actual production process, continuously collect data to update the data set, and train and update the model.

[0046] In order to solve the problem of online imaging and precise detection of various defects on the surface of automatic wire laying plies of composite materials, the present invention proposes an automatic wire laying ply surface defect detection device and method based on the fusion of two-dimensional and three-dimensional information. The advantage of the device is that it can synchronously collect two-dimensional and three-dimensional information of the ply surface at a low cost. The advantage of the method is that it is beneficial to both two-dimensional and three-dimensional information, and can detect and classify more types of ply surface defects. It has the advantages of many detectable defect types, high defect recognition accuracy, and strong applicability.

[0047] The measured ply samples contain four typical defects: too large gap, bulge / bubble, foreign matter (white and black), and overlap. These typical defects are selected for experimental verification because they are difficult to detect and small in size, but it does not mean that this device and method can only detect these four types of defects. In fact, this device and method can also detect other defects with larger scales and more obvious two-dimensional or three-dimensional features, such as wrinkles, torsion, missing wires, flanging, bridging, etc.

[0048] Figure 3 The two-dimensional image of the ply surface collected by the device proposed in the present invention, Figure 3In the 2D image, large gaps, bulges / bubbles, and white foreign objects can be observed. The main disadvantage of the 2D image is that the contrast of the ply surface is low and the color is dark, making it difficult to clearly observe the overlap area and black foreign objects.

[0049] Figure 4a , Figure 4b This is a schematic diagram of the three-dimensional point cloud of the ply surface collected by the device proposed in the present invention. From the point cloud image, it can be observed that there are large gaps, bulges / bubbles, foreign objects (white and black), and overlaps. The main disadvantage of the three-dimensional point cloud is that the point cloud image lacks color and intensity information, so it is difficult to classify black foreign objects, white foreign objects, and bulges / bubbles.

[0050] Figure 5 This is a flow chart of the automatic wire laying surface defect detection method based on the fusion of two-dimensional and three-dimensional information of the present invention. The proposed two-dimensional image and three-dimensional point cloud synchronous acquisition device can be used to synchronously acquire the two-dimensional image and three-dimensional point cloud of the ply surface. After aligning and preprocessing the two-dimensional image and the three-dimensional point cloud, a defect data set of the ply surface is constructed and a multimodal fusion defect detection neural network is trained. The trained network can complete the detection task of the ply surface defects. Since the two-dimensional image and the three-dimensional point cloud are input at the same time, the proposed detection method has the advantages of two-dimensional image detection and three-dimensional point cloud detection at the same time: 1) It can classify defects with similar three-dimensional features such as foreign objects of different colors, bulges / bubbles, etc.; 2) It can detect defects that are difficult to observe in two-dimensional images, such as overlaps, black foreign objects, bulges / bubbles of small size or poor lighting conditions; therefore, the detection device of the present invention periodically projects two light fields: an illumination light field and a multi-line structured light field.

[0051] The illumination light field is used to assist the industrial camera 4 in obtaining a two-dimensional image of the ply surface. The multi-line structured light field and the camera constitute a three-dimensional measurement system to achieve the synchronous acquisition of two-dimensional images and three-dimensional point cloud information. The line width, spacing, and number of the multi-line structured light can be adaptively adjusted according to the surface morphology characteristics of the processed workpiece. Considering the optical properties of the carbon fiber tow 7, the projected multi-line structured light and the optical axis of the industrial camera 4 approximately meet the reflection relationship (using the oblique laser triangulation method), so that the brightness of the structured light collected by the industrial camera 4 is improved, and the angle between the illumination light field and the industrial camera 4 does not meet the reflection angle, so that the illumination light field is evenly irradiated on the surface of the tow. An optional filter 5 can be installed in front of the camera lens to filter the strong light interference of the heating head when the thermoplastic tow is heated. The collected data is pre-processed by denoising, contrast equalization, cropping, sharpening, downsampling, etc., and data amplification is performed to construct a defect data set. Finally, the multimodal fusion defect detection neural network is used to realize online detection, identification, and positioning of defects on the ply surface.

[0052] The above are only preferred embodiments of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should be regarded as the protection scope of the present invention.

Claims

1. An automatic wire laying layer surface defect detection device, characterized in that: include: Multi-line structured light source (1), industrial camera (4) and illumination light field light source (6); The multi-line structured light source (1) and the industrial camera (4) are both directed toward the pavement surface, and the multi-line structured light source (1) is used to project a multi-line structured light field; the illumination light field light source (6) is arranged above the pavement surface and is used to project the illumination light field; The industrial camera (4) cooperates with the illumination light field light source (6) and the multi-line structured light source (1) to collect two-dimensional images and three-dimensional point cloud information of the pavement surface.

2. The automatic wire laying layer surface defect detection device according to claim 1 is characterized in that: It also includes a connecting plate (3) connected to the wire placing head (14), and the multi-line structured light source (1) and the industrial camera (4) are respectively mounted on the connecting plate (3) via an adjustable rotating table (2).

3. The automatic wire laying layer surface defect detection device according to claim 1 is characterized in that: A filter (5) is also installed at the front end of the industrial camera (4).

4. The automatic wire laying layer surface defect detection device according to claim 1 is characterized in that: The illumination light field light sources (6) are arranged on both sides of the light beam of the multi-line structure light source (1).

5. A method for detecting surface defects of automatic wire laying, characterized in that: include: Acquire synchronous data of 2D images and 3D point cloud information of the ply surface; Align the collected 2D image and 3D point cloud, and then perform preprocessing; Perform data augmentation on the preprocessed 2D images and 3D point clouds, annotate the defects on the ply surface, construct a ply surface defect dataset, and divide the dataset into training set, test set, and validation set; Construct a multimodal fusion defect detection neural network with two-dimensional images and three-dimensional point clouds as input, input the data set into the multimodal fusion defect detection network, and perform training, testing, and verification based on the training set, test set, and verification set; The trained multimodal fusion defect detection neural network model can use two-dimensional images and three-dimensional point clouds as input to perform online detection, identification and positioning of ply surface defects.

6. The method for detecting surface defects of an automatic wire laying layer according to claim 5, characterized in that: The method for acquiring synchronously collected data of the two-dimensional image and three-dimensional point cloud information of the pavement surface is that the multi-line structured light source (1) and the illumination light field light source (6) periodically project the multi-line structured light field and the illumination light field under computer control, so as to realize the synchronous collection of the two-dimensional image and three-dimensional point cloud information of the pavement surface by the industrial camera (4), specifically comprising: When the illumination light field is projected, the multi-line structured light field is turned off, and the industrial camera (4) collects a two-dimensional image of the wire laying surface in the field of view; When the multi-line structured light field is projected, the illumination light field is turned off, and the industrial camera (4) obtains three-dimensional point cloud information of the ply surface.

7. The method for detecting surface defects of an automatic wire laying layer according to claim 6, characterized in that: The industrial camera (4) collects a two-dimensional image of the surface of the laid wire in the field of view, specifically including: The optical axis of the industrial camera (4) is approximately aligned with the line structured light to satisfy the reflection angle relationship, and a clear line structured light pattern is collected at a lower exposure time and a lower power of the multi-line structured light source (1) to complete the acquisition of three-dimensional information; the angle between the illumination light field and the industrial camera (4) does not satisfy the reflection angle, so that the illumination light field is evenly irradiated on the surface of the wire bundle to assist in completing the acquisition of two-dimensional information.

8. The method for detecting surface defects of an automatic wire laying layer according to claim 6, characterized in that: The method of realizing the synchronous acquisition of the two-dimensional image and the three-dimensional point cloud information of the pavement surface by the industrial camera (4) is to realize the synchronous acquisition of the two-dimensional image and the three-dimensional point cloud by periodically switching the multi-line structured light field and the illumination light field, and includes: The time taken to collect the 2D image and 3D information of the ply surface within the camera field of view is , the time for projecting the illumination light field is , the time for projecting multi-line structured light is , satisfying the relationship The projection sequence is: in one acquisition cycle, the illumination light field is first projected to acquire a two-dimensional image of the ply surface with a certain field of view size; then, multi-line structured light is projected to complete the scanning and reconstruction of the ply surface within the field of view, and this cycle is repeated.

9. The method for detecting surface defects of an automatic wire laying layer according to claim 5, characterized in that: The defects on the surface of the ply are marked, including: missing wires, overlaps, excessive gaps, wrinkles, bulges / bubbles, foreign objects, flanges, and bridging defects.

10. The method for detecting surface defects of an automatic wire laying layer according to claim 5, characterized in that: The collected two-dimensional image and three-dimensional point cloud are aligned and then preprocessed, including the following steps: By using a feature point extraction method, feature points are extracted in the two-dimensional image and the three-dimensional point cloud respectively; The extracted feature points are described using feature descriptors, and the nearest neighbor algorithm is used to match the feature points between the two data sets to establish a corresponding relationship between the two and achieve alignment; After alignment, the median filter method is used to denoise the two-dimensional image to remove the salt and pepper noise caused by the uneven surface of the ply. Use adaptive histogram equalization to enhance the two-dimensional image and eliminate image blur caused by uneven lighting; Perform sharpening operations on two-dimensional images; After alignment, the 3D point cloud is downsampled and locally fitted and filtered to fit the ply plane, and points far from the fitting plane are removed.

Images