A preform winding surface detection method based on cross-modal data

Abstract

The present invention relates to the field of computer vision technology, and in particular to a method for detecting the surface of a preform winding molding based on cross-modal data, which mainly includes the following steps: optimizing the measurement system through system calibration, and completing the collection of point cloud data and image data; constructing a mapping relationship between point cloud data and image data; constructing a network model for detecting the surface quality of a preform winding molding, and inputting point cloud data and image data into the model to obtain defect detection results and fiber segmentation results; and realizing defect detection on the surface of the preform and calculation of the diameter and density of a single fiber of the preform according to the obtained results. The present invention comprehensively utilizes the characteristics of different modal data and combines it with a deep learning network, which solves the problems of blind spots, huge workload, long time consumption, and low precision in manual visual detection under the existing technology, and improves the automation level of three-dimensional weaving.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision, and in particular relates to a preform winding forming surface detection method based on cross-modal data. Background Art

[0002] Carbon fiber three-dimensional woven preforms, with their unique spatial mesh structure and thickness, are widely used in many fields. However, due to the flat appearance formed by the combination of carbon fiber single fiber bundles, they are prone to problems such as poor cohesion and poor shear resistance during the weaving process, resulting in frequent fuzz or yarn breakage, which increases the difficulty of weaving. In continuous production, it is crucial to ensure the stability of the appearance quality of the preform. At present, manual visual inspection is still used, but this method is labor-intensive, time-consuming and low-precision. In addition, due to the large size of the preform and the blind spots in manual inspection, it cannot meet the product inspection requirements. Therefore, the application of the preform winding forming surface quality detection method in the weaving process of carbon fiber three-dimensional woven preforms is studied to realize online monitoring, which plays a vital role in improving the automation level of three-dimensional weaving. Summary of the invention

[0003] In view of the deficiencies in the prior art, the present invention provides a preform winding forming surface detection method based on cross-modal data, which solves the problems of blind spots, huge workload, long time consumption, low precision and the like in the current traditional method of manual visual detection; the present invention is applicable to carbon fiber three-dimensional woven preforms, and completes the qualitative and quantitative analysis of the preform surface through the fusion of two-dimensional images and three-dimensional point clouds, and combines deep learning and cross-modal data, comprehensively utilizes the characteristics of different modal data, and realizes preform surface defect detection and calculation of single fiber diameter and density of preform through deep learning networks, greatly improving the automation level of three-dimensional weaving.

[0004] To achieve the above object, the present invention proposes the following technical solution, a method for detecting the surface of a preform winding molding based on cross-modal data, the method comprising the following steps:

[0005] S1, through the measurement system optimization model after system calibration, complete the collection of point cloud data and image data;

[0006] S2, constructing the mapping relationship between point cloud data and image data according to the measurement system optimization model;

[0007] S3, constructing a preform winding surface quality detection network model, inputting image data and point cloud data into the model to obtain defect detection results and fiber segmentation results;

[0008] S4. The defect detection of the preform winding forming surface and the calculation of the diameter and density of a single fiber of the preform are realized through the results of the preform winding forming surface quality detection network model.

[0009] Furthermore, step S1 specifically includes the following steps:

[0010] S101, using the nine-square grid method to capture images, that is, taking images containing a chessboard at different angles in each corner; then calibrating the camera using the Zhang Zhengyou calibration method to obtain the camera's own intrinsic parameter matrix M and extrinsic parameter matrix N, and constructing the camera coordinate system Q cam With the world coordinate system Q w The relationship between them can be obtained by simultaneously obtaining the distortion coefficient;

[0011] S102, establish camera image coordinate system Q c With the projector image coordinate system Q c '; select the corner points of the chessboard projection image on the calibration board as the bridge to establish the relationship between the two coordinate systems, and get x pp =H cp x pc , where x pp =(u pp ,v pp ,1) T Indicates the checkerboard corner points on the calibration plate in the projector image coordinate system Q c 'The coordinates under, x pc =(u pc ,v pc ,1) T It represents the corresponding corner point of the chessboard captured by the camera after being projected by the projector in the camera image coordinate system Q c The coordinates of H cp Indicates the coordinate system transformation relationship;

[0012] S103, using polynomial to transform the coordinate system relationship H cp Perform fitting to eliminate the errors caused by lens distortion of the camera and projector and obtain the corrected coordinates;

[0013] S104, repeatedly obtain the calibration plate images at different positions, and finally obtain the coordinates x of the checkerboard corner points on the calibration plate in the projector image coordinate system by the methods described in steps S102 and S103 respectively. pp , and then use Zhang Zhengyou's camera calibration method to calibrate the projector's intrinsic parameter matrix M' and extrinsic parameter matrix N';

[0014] S105, perform overall calibration optimization on the system, and use the inverse process of the measurement process to optimize the overall system parameters, that is, according to the coordinates x of the corner points of the checkerboard on the calibration plate in the projector image coordinate system pp =(u pp ,v pp ) T , the coordinate x in the camera image coordinate system pc=(u pc ,v pc ) T , and the coordinate X in the world coordinate system i =(X w ,Y w ,Z w ) T , by using the Levenberg-Marquardt nonlinear minimization method, the parameter optimization of the projector intrinsic parameter matrix M' and the camera intrinsic parameter matrix M is achieved.

[0015] Furthermore, the specific process of step S2 is: establish the correspondence between the points in the point cloud data and the pixels in the image data through the camera optimization matrix, that is, find a single point (x i ,y i ,z i ) corresponds to a single pixel of image data (u i ,v i ), and its mapping relationship is expressed as:

[0016]

[0017] Among them, T represents the camera extrinsic matrix composed of the rotation matrix and the translation matrix, and S represents the camera intrinsic matrix.

[0018] Furthermore, step S3 specifically includes the following process:

[0019] S31, establishing a preform winding forming surface quality detection data set consisting of image data and point cloud data, and further dividing the data set into a training set, a test set and a validation set;

[0020] S32, in the dataset, the file directory structure of image data is constructed according to the PASCAL VOC dataset, and the file directory structure of point cloud data is constructed according to the 3DMatch dataset;

[0021] S33. The constructed preform winding surface quality detection network model includes two branches: a preform qualitative analysis branch and a preform quantitative analysis branch; the preform qualitative analysis branch realizes the preform surface defect detection, and the preform quantitative analysis branch performs preform single fiber point cloud segmentation;

[0022] S34. Constructing the loss function of the preform winding surface quality detection network model: l = l I +l P , where l I represents the preform surface defect detection loss function, l P represents the segmentation loss function of a single fiber point cloud of the preform; l I Use the same loss function as the YOLOv5 target detection algorithm;P It consists of four items, namely the cross entropy loss function l wce 、Lovasz-softmax loss function l lovasz , heat map regression loss function l through mean square error heatmap and the offset map regression loss function l by mean absolute error offset ,Right now:

[0023] l=l wce +αl lovasz +βl heatmap +γl offset ;

[0024] Among them, α, β and γ are used to balance the loss coefficients affected by each loss term;

[0025] S35, inputting the data in the training set, test set and validation set constructed above, training and evaluating the preform winding surface quality detection network model, and obtaining output results.

[0026] More specifically, step S33 specifically includes the following steps:

[0027] S331, the preform qualitative analysis branch is constructed based on the pre-trained ResNet-50 deep residual network model, and includes an image encoding module and an image decoding module; the image encoding module captures the local features in the preform surface image through a convolution kernel with a size of 3×3 and a step size of 1, and the size of the feature map is corrected between each convolution layer through the maximum value pooling with a pooling window of 2×2 and a step size of 2 to reduce the resolution of the feature map; the obtained local features are processed by the image decoding module and output for defect detection on the preform winding forming surface;

[0028] S332, the backbone network of the preform quantitative analysis branch is constructed based on the pre-trained Minkowski-UNet34 model, and includes a point cloud encoding module, a cross-modal data fusion module and a point cloud decoding module, wherein the cross-modal data fusion module includes a voxel image fusion module and a cross-modal association module;

[0029] More specifically, step S332 specifically includes the following steps:

[0030] S3321. The voxelized features of the point cloud data obtained by the point cloud encoding module are used as the original voxel features F in the input cross-modal data fusion module. i V ; According to the mapping relationship between point cloud data and image data, the original voxel feature F i V The corresponding image feature F I ;

[0031] S3322, original voxel feature F i V The single point features extracted from the cascade point cloud data by the multi-layer perceptron MLP are combined with the image features F I Input together into the voxel image fusion module; introduce the offset to the image feature F I Sampling is performed to obtain the sampled image features Its formula is:

[0032]

[0033] where p i is the offset, Δp i,j is the lth sampling offset;

[0034] The primitive feature F i V The sampled image features are regarded as the Query, and the sampled image features are regarded as the Key and Value, and the image enhanced voxel features are obtained through the multi-head attention mechanism. Its formula is:

[0035]

[0036] Among them, W m and W m ′ represents the learnable weight, m ​​represents the index of the multi-head attention mechanism, M represents the number of self-attention heads, and L represents the total number of sampled image features;

[0037] Then enhance the voxel features of the image Characteristics of the original body Through residual connection, the final fused voxel features are output The number of channels is N×2C f , C f Represents the number of channels of voxel features, and N represents the number of point clouds input to the network;

[0038] S3323, final fused voxel features As the input of the cross-modal association module, it is mapped into the point cloud space and the final fused voxel features are The point features mapped to the point cloud space are processed by trilinear interpolation and bilinear interpolation to obtain interpolated voxel features of the same dimension, which are then cascaded to obtain the global representation and view adaptive features F M ; Introduce learning parameter W v F M The weighted processing is performed and input into the multi-layer perceptron MLP, and the obtained features are then combined with F M Perform residual connection to obtain cross-view fusion feature F i M ;

[0039] S3324, cross-view fusion feature F i M Input into the point cloud decoding module; the point cloud decoding module matches the input and output points based on the reverse distance weighted average interpolation method of k neighboring points, that is, let the feature to be solved of the interpolation point x be f (j) (x), the features of the k nearest neighbors of the point with known features are f i (j) ,i=1,2,…,k;f (j) The solution formula for (x) is expressed as:

[0040]

[0041] Among them, d is the function to measure the distance, w i Represents the weight of the point, which is inversely proportional to the distance. The larger the distance d, the smaller the weight. p represents the influence of distance on the weight.

[0042] S3325, processed cross-view fusion feature F i M The multi-layer perceptron MLP is then used to further sample the features. After sampling, it passes through one-dimensional convolution, BN layer, and ReLU activation function, and the final output result is used for the point cloud segmentation of a single fiber of the preform.

[0043] Furthermore, step S4 specifically includes the following steps:

[0044] S41, realizing preform surface defect detection by utilizing the qualitative analysis branch results in the preform winding forming surface quality detection network model;

[0045] S42, using the quantitative analysis branch result in the preform winding surface quality detection network model to achieve preform single fiber point cloud segmentation, the specific process includes: projecting the single fiber point cloud data obtained by the quantitative analysis branch onto the reference line, determining the length distribution of the point cloud data on the reference line, selecting a measurement point within the length range, selecting the point closest to the measurement point in the point cloud data as a section point, determining a two-dimensional projection section based on the direction vector of the section point and the reference line, projecting the points near the section onto the section, and obtaining richer section projection point cloud data;

[0046] S43, calculating the diameter of a single fiber of the preform based on the cross-sectional projection point cloud data;

[0047] S44, measuring the number of wefts within the width of one frame of the cross-sectional projection point cloud data, and calculating the fiber density at a fixed width and a fixed number of wefts.

[0048] More specifically, step S43 specifically includes the following steps:

[0049] S431, perform preliminary circle fitting on all input data points using an algebraic fitting algorithm to obtain a preliminary estimated circle center O i (a i ,b i ), radius R i At the same time, calculate each point P in the point cloud data j The distance d from (j=0,1,…,n) to the fitting circle i,j ,Right now:

[0050]

[0051] Among them, (x j ,y j ) is point P j The coordinates of

[0052] S432, set the number of iterations I, the initial value is 0, and determine the change in the radius of the fitting circle ΔR i Is it less than the preset convergence threshold γ? th If yes, the iteration is stopped and the current fitting result is output as the final circle. The calculation formula of the radius change is:

[0053] ΔR i =R i -R i-1

[0054] Among them, R i and R i-1 are the fitting circle radii of this and last iterations respectively;

[0055] If the radius change ΔR i If the convergence condition is not met, the preset error threshold ε is used. th Search for internal points; when point P j The distance d to the fitted circle i,j Less than or equal to the error threshold ε th When , mark them as internal points, select these internal points to form a new point cloud set As the input of the next iteration; repeat the iteration until the fitting result that meets the convergence conditions is output or the preset upper limit of the number of iterations is reached. max , the result of the last iteration is taken as the final radius, that is, the diameter of a single fiber of the preform is obtained.

[0056] By means of the above technical solution, the present invention provides a method for detecting the surface of a preform winding molding based on cross-modal data, which has at least the following beneficial effects:

[0057] Compared with manual visual inspection, which has the disadvantages of blind spots, huge workload, long time consumption and low precision, the present invention combines deep learning and cross-modal data, comprehensively utilizes the characteristics of different modal data, and realizes preform surface defect detection and calculation of single fiber diameter and density of preform through deep learning network, thereby improving the automation level of carbon fiber three-dimensional weaving preforms. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 It is a schematic diagram of the process of the present invention;

[0059] Figure 2 This is a schematic diagram of the structure of the preform winding surface quality detection network model constructed in the present invention. DETAILED DESCRIPTION

[0060] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific implementation methods, so that the implementation process of how the present application uses technical means to solve technical problems and achieve technical effects can be fully understood and implemented accordingly.

[0061] Those skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a program, so the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0062] Please refer to Figure 1-2 , showing a specific implementation of the present embodiment, the present embodiment optimizes the measurement system through system calibration, thereby constructing a mapping relationship between point cloud data and image data, and comprehensively utilizes the characteristics of different modal data through the constructed preform winding molding surface quality detection network model; the model includes a qualitative analysis branch and a quantitative analysis branch, the qualitative analysis branch realizes defect detection on the preform surface according to the input image data, and the quantitative analysis branch realizes single fiber segmentation of the preform according to the cross-modal fusion data of the input point cloud data and image data, thereby realizing the calculation of fiber diameter and density; the present invention combines deep learning with cross-modal data, and effectively improves the automation level of carbon fiber three-dimensional weaving in the technical field.

[0063] Please refer to Figure 1 This embodiment proposes a preform winding forming surface detection method based on cross-modal data, and the method includes the following steps:

[0064] S1, through the measurement system optimization model after system calibration, complete the collection of point cloud data and image data;

[0065] As a preferred implementation of step S1, the specific process includes the following steps:

[0066] S101, using the nine-square grid method to capture images, that is, taking images containing a chessboard at different angles in each corner; then calibrating the camera using the Zhang Zhengyou calibration method to obtain the camera's own intrinsic parameter matrix M and extrinsic parameter matrix N, and constructing the camera coordinate system Q cam With the world coordinate system Q w The relationship between them can be obtained by simultaneously obtaining the distortion coefficient;

[0067] In this embodiment, each corner is photographed 3 times at different angles, for a total of 27 images;

[0068] S102, establish camera image coordinate system Q c With the projector image coordinate system Q c '; select the corner points of the chessboard projection image on the calibration board as the bridge to establish the relationship between the two coordinate systems, and get x pp =H cp x pc , where x pp =(u pp ,v pp ,1) T Indicates the checkerboard corner points on the calibration plate in the projector image coordinate system Q c 'The coordinates under, x pc =(u pc ,v pc ,1) T It represents the corresponding corner point of the chessboard captured by the camera after being projected by the projector in the camera image coordinate system Q c The coordinates of H cp Indicates the coordinate system transformation relationship;

[0069] S103, using polynomial to transform the coordinate system relationship H cp Perform fitting to eliminate the errors caused by lens distortion of the camera and projector;

[0070] The polynomial used in this embodiment is as follows:

[0071]

[0072] Among them, δ u and δ v Indicates the error amount that needs to be compensated in the u-axis and v-axis directions of the pixel coordinate system, n u =(n u0 ,n u1 ,n u2,n u3 ,…,n ui ) T ,i∈Z,n v =(n v0 ,n v1 ,n v2 ,n v3 ,…,n vi ) T ,i∈Z represents the error compensation coefficient, i represents the number of selected coordinate points, m, n satisfies m+n=i;

[0073] The corrected coordinates [u p ,v p ,1] T =H cp [u c ,v c ,1] T +[δ u ,δ v ,0] T

[0074] S104, repeatedly obtain the calibration plate images at different positions, and finally obtain the coordinates x of the checkerboard corner points on the calibration plate in the projector image coordinate system by the methods described in steps S102 and S103 respectively. pp , and then use Zhang Zhengyou's camera calibration method to calibrate the projector's intrinsic parameter matrix M' and extrinsic parameter matrix N';

[0075] S105, perform overall calibration optimization on the system, and use the inverse process of the measurement process to optimize the overall system parameters, that is, according to the coordinates x of the corner points of the checkerboard on the calibration plate in the projector image coordinate system pp =(u pp ,v pp ) T , the coordinate x in the camera image coordinate system pc =(u pc ,v pc ) T , and the coordinate X in the world coordinate system i =(X w ,Y w ,Z w ) T , by using the Levenberg-Marquardt nonlinear minimization method, the parameter optimization of the projector intrinsic parameter matrix M' and the camera intrinsic parameter matrix M is achieved.

[0076] S2, constructing the mapping relationship between point cloud data and image data according to the measurement system optimization model;

[0077] As a preferred implementation of step S2, the specific process is:

[0078] The camera optimization matrix is ​​used to establish the correspondence between the points in the point cloud data and the pixels in the image data, that is, to find the correspondence between a single point (xi, yi, zi) in the point cloud data and a single pixel (u i ,v i ), and its mapping relationship is expressed as:

[0079]

[0080] Among them, T represents the camera extrinsic matrix composed of the rotation matrix and the translation matrix, and S represents the camera intrinsic matrix.

[0081] S3, constructing a preform winding surface quality detection network model, inputting image data and point cloud data into the model to obtain defect detection results and fiber segmentation results;

[0082] As a preferred implementation of step S3, the specific process includes the following steps:

[0083] S31, establishing a preform winding forming surface quality detection data set consisting of image data and point cloud data, and further dividing the data set into a training set, a test set and a verification set; in this embodiment, 3000 sets of image data and 3D data are collected, and divided into a training set, a test set and a verification set in a ratio of 6:3:1;

[0084] S32, in the dataset, the file directory structure of image data is constructed according to the PASCAL VOC dataset, and the file directory structure of point cloud data is constructed according to the 3DMatch dataset;

[0085] S33, constructing a preform winding surface quality detection network model, including a preform qualitative analysis branch and a preform quantitative analysis branch, that is, including the following steps:

[0086] S331, the preform qualitative analysis branch is constructed based on the pre-trained ResNet-50 deep residual network model, and includes an image encoding module and an image decoding module; the image encoding module captures the local features in the preform surface image through a convolution kernel with a size of 3×3 and a step size of 1, and the size of the feature map is corrected between each convolution layer through the maximum value pooling with a pooling window of 2×2 and a step size of 2 to reduce the resolution of the feature map; the obtained local features are processed by the image decoding module and output for defect detection on the preform winding forming surface;

[0087] S332, the backbone network of the preform quantitative analysis branch is constructed based on the pre-trained Minkowski-UNet34 model, and includes a point cloud encoding module, a cross-modal data fusion module and a point cloud decoding module, wherein the cross-modal data fusion module includes a voxel image fusion module and a cross-modal association module;

[0088] More specifically, Figure 2 As shown, the construction of the preform quantitative analysis branch in step S332 of this embodiment specifically includes the following steps:

[0089] S3321. The voxelized features of the point cloud data obtained by the point cloud encoding module are used as the original voxel features F in the input cross-modal data fusion module. i V ; According to the mapping relationship between point cloud data and image data, the original voxel feature F i V The corresponding image feature F I ;

[0090] S3322, original voxel feature F i V The single point features extracted from the cascade point cloud data by the multi-layer perceptron MLP are combined with the image features F I Input together into the voxel image fusion module; introduce the offset to the image feature F I Sampling is performed to obtain the sampled image features Its formula is:

[0091]

[0092] where p i is the offset, Δp i,j is the lth sampling offset;

[0093] The primitive feature F i V The sampled image features are regarded as the Query, and the sampled image features are regarded as the Key and Value, and the image enhanced voxel features are obtained through the multi-head attention mechanism. Its formula is:

[0094]

[0095] Among them, W m and W m ′ represents the learnable weight, m ​​represents the index of the multi-head attention mechanism, M represents the number of self-attention heads, and L represents the total number of sampled image features;

[0096] Then enhance the voxel features of the image With the original body element characteristics F i V Through residual connection, the final fused voxel features are output

[0097] In this embodiment, the final fused voxel feature The number of channels is N×2C f , Cf The number of channels representing voxel features, N represents the number of point clouds input into the network; extracting single point features through multi-layer perceptron MLP solves the problem that point cloud data will lose certain structural information after voxelization; and the introduction of offset and sampling processing enables voxel features to automatically find the most relevant image features for fusion, and voxel features without corresponding image features will be added with zero vectors;

[0098] S3323, final fused voxel features As the input of the cross-modal association module, it is mapped into the point cloud space and the final fused voxel features are The point features mapped to the point cloud space are processed by trilinear interpolation and bilinear interpolation. This can solve the problem of quantity mismatch, thereby obtaining interpolated voxel features of the same dimension, and then cascade to obtain the global representation and view adaptive features F M ; Introduce learning parameter W v F M The weighted processing is performed and input into the multi-layer perceptron MLP, and the obtained features are then combined with F M Perform residual connection to obtain cross-view fusion feature F i M ;

[0099] S3324, cross-view fusion feature F i M Input into the point cloud decoding module; the point cloud decoding module is used to solve the mismatch problem between the input and output points. It completes the solution of the features of the interpolated points based on the reverse distance weighted average interpolation method of k neighboring points, that is, let the feature to be solved of the interpolated point x be f (j) (x), the features of the k nearest neighbors of the point with known features are f i (j) ,i=1,2,…,k。f (j) The solution formula for (x) is expressed as:

[0100]

[0101] Among them, d is the function to measure the distance, w i Represents the weight of the point, which is inversely proportional to the distance. The larger the distance d, the smaller the weight. p represents the influence of distance on the weight.

[0102] In this embodiment, k=2, p=3;

[0103] S3325, processed cross-view fusion feature F i MThe multi-layer perceptron MLP is then used to further sample the features. After sampling, it passes through one-dimensional convolution, BN layer, and ReLU activation function. The final output combines the features of point cloud data and image data, which can be used for point cloud segmentation of a single fiber in a preform.

[0104] S34. Constructing the loss function of the preform winding surface quality detection network model: l = l I +l P , where l I represents the preform surface defect detection loss function, l P represents the segmentation loss function of a single fiber point cloud of the preform; l I The same loss function as YOLOv5 is used. P It consists of four items, namely the cross entropy loss function l wce 、Lovasz-softmax loss function l lovasz , heat map regression loss function l through mean square error heatmap and the offset map regression loss function l by mean absolute error offset ,Right now:

[0105] l=l wce +αl lovasz +βl heatmap +γl offset ;

[0106] Among them, α, β and γ are used to balance the loss coefficients affected by each loss term;

[0107] In this embodiment, since there are multiple modules inside the preform winding surface quality detection network model, which involves multiple data processing, a loss function is constructed to balance the structural information loss in the process;

[0108] S35, inputting the data in the training set, test set and validation set constructed above, training and evaluating the preform winding surface quality detection network model, and obtaining output results.

[0109] S4. Based on the results of the preform winding surface quality detection network model, the defect detection of the preform winding surface and the calculation of the diameter and density of a single fiber of the preform are realized;

[0110] As a preferred implementation of step S4, the specific process includes the following steps:

[0111] S41, the output result of the qualitative analysis branch in the preform winding surface quality detection network model can realize the preform surface defect detection;

[0112] S42. The output result of the quantitative analysis branch in the preform winding surface quality detection network model needs to first realize the preform single fiber point cloud segmentation, and then further calculate the diameter and density of the single fiber; the specific process of single fiber point cloud segmentation includes: projecting the single fiber point cloud data obtained by the quantitative analysis branch onto the reference line, determining the length distribution of the point cloud data on the reference line, selecting the measurement point within the length range, selecting the point closest to the measurement point in the point cloud data as the section point, determining the two-dimensional projection section based on the direction vector of the section point and the reference line, projecting the points near the section onto the section, and obtaining richer section projection point cloud data;

[0113] In this embodiment, a cubic bounding box with 2l as width and height and 2d as acceptance range is constructed with the cross section as the symmetry plane and the measurement point as the center. The bounding box evenly divides the cross section into two parts with equal volume. The point cloud data in the cubic bounding box is projected onto the cross section to generate a cross-section projection point cloud that can represent the fiber structure.

[0114] The process of solving the cross-section projection point cloud data in this embodiment is given below:

[0115] It is known that a point p(a, b, c) on a plane T in three-dimensional space and a normal vector of the plane T The point formal equation of the plane T is: A(xa)+B(yb)+C(zc)=0. After expansion, we get the equation:

[0116] Ax+By+Cz-(Aa+Bb+Cc)=0;

[0117] Let D = -(Aa + Bb + Cc). The general equation of the plane is:

[0118] Ax+By+Cz+D=0;

[0119] Suppose there is a point m(x0,y0,z0) outside the plane and its projection point n(x I ,y I ,z I ). Considering that the line from point m to its projection point n is perpendicular to the plane, based on the vertical constraint, we get:

[0120]

[0121] Then we can solve for x I ,y I ,z I The specific expression of:

[0122]

[0123]

[0124] The cross-section projection point cloud data can be obtained;

[0125] S43, realizing the single fiber diameter of the preform according to the cross-sectional projection point cloud data;

[0126] More specifically, step S43 estimates the diameter of a single fiber by circle fitting, which specifically includes the following steps:

[0127] S431, perform preliminary circle fitting on all input data points using an algebraic fitting algorithm to obtain a preliminary estimated circle center O i (a i ,b i ), radius R i At the same time, calculate each point P in the point cloud data j The distance d from (j=0,1,…,n) to the fitting circle i,j ,Right now:

[0128]

[0129] Among them, (x j ,y j ) is point P j The coordinates of

[0130] S432, perform multiple iterations; set the number of iterations I, the initial value is 0, and determine the change in the radius of the fitting circle ΔR i Is it less than the preset convergence threshold γ? th If yes, the iteration is stopped and the current fitting result is output as the final circle. The calculation formula of the radius change is:

[0131] ΔR i =R i -R i-1 ;

[0132] Among them, R i and R i-1 are the fitting circle radii of this and last iterations respectively;

[0133] If the radius change ΔR i If the convergence condition is not met, the preset error threshold ε is used. th Search for internal points; when point P j The distance d to the fitted circle i,j Less than or equal to the error threshold ε th When , mark them as internal points, select these internal points to form a new point cloud set As the input of the next iteration; repeat the iteration until the fitting result that meets the convergence conditions is output or the preset upper limit of the number of iterations is reached. max, the result of the last iteration is taken as the final radius, that is, the diameter of a single fiber;

[0134] In this embodiment, a preset convergence threshold γ is set during the iteration process to determine whether to perform iteration. th is 0.03 mm, and the preset error threshold ε for searching the inner point th Also 0.03mm. To prevent the fitting radius from converging too slowly, the maximum number of iterations I is set. max is 20. After multiple iterations, the radius of the fitting circle is continuously reduced, thereby achieving a more accurate estimation of the diameter of a single fiber;

[0135] S44, measuring the number of wefts within the width of one frame of the cross-sectional projection point cloud data. In this embodiment, the width is 10 cm, and the fiber density at a fixed width and fixed number of wefts is calculated.

[0136] In summary, the present invention brings the following beneficial effects:

[0137] The present invention constructs a preform winding molding surface quality detection network model by combining point cloud data and image data, and comprehensively utilizes the characteristics of different modal data; the model includes a qualitative analysis branch and a quantitative analysis branch, the qualitative analysis branch can realize defect detection on the preform surface according to the input two-dimensional image data, and the quantitative analysis branch can realize single fiber segmentation of the preform according to the cross-modal fusion data of the input three-dimensional point cloud data and two-dimensional image data, thereby realizing the calculation of fiber diameter and density; the present invention combines deep learning with cross-modal data, and effectively improves the automation level of carbon fiber three-dimensional weaving in the technical field.

[0138] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification and the features of different embodiments or examples, unless they are contradictory.

[0139] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, which can be embodied in any computer-readable medium for use by an instruction execution system, apparatus or device (such as a computer-based system, a system including a processor or other system that can fetch instructions from an instruction execution system, apparatus or device and execute instructions), or used in combination with these instruction execution systems, apparatuses or devices.

[0140] The above implementation methods have been described in detail. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A method for detecting the surface of a preform winding molding based on cross-modal data, characterized in that: The method comprises the following steps: S1. Based on the measurement system optimization model after system calibration, the acquisition of point cloud data and image data is completed; step S1 specifically includes the following steps: S101, using the nine-square grid method to capture images, that is, taking images containing a chessboard at different angles in each corner; then calibrating the camera using the Zhang Zhengyou calibration method to obtain the camera's own intrinsic parameter matrix M and extrinsic parameter matrix N, and constructing the camera coordinate system Q cam With the world coordinate system Q w The relationship between them can be obtained by simultaneously obtaining the distortion coefficient; S102, establish camera image coordinate system Q c With the projector image coordinate system Q c '; select the corner points of the chessboard projection image on the calibration board as the bridge to establish the relationship between the two coordinate systems, and get x pp =H cp x pc , where x pp =(u pp ,v pp ,1) T Indicates the checkerboard corner points on the calibration plate in the projector image coordinate system Q c 'The coordinates under, x pc =(u pc ,v pc ,1) T It represents the corresponding corner point of the chessboard captured by the camera after being projected by the projector in the camera image coordinate system Q c The coordinates of H cp Indicates the coordinate system transformation relationship; S103, using polynomial to transform the coordinate system relationship H cp Perform fitting to eliminate the errors caused by lens distortion of the camera and projector and obtain the corrected coordinates; S104, repeatedly obtain the calibration plate images at different positions, and finally obtain the coordinates x of the checkerboard corner points on the calibration plate in the projector image coordinate system by the methods described in steps S102 and S103 respectively. pp , and then use Zhang Zhengyou's camera calibration method to calibrate the projector's intrinsic parameter matrix M' and extrinsic parameter matrix N'; S105, perform overall calibration optimization on the system, and use the inverse process of the measurement process to optimize the overall system parameters, that is, according to the coordinates x of the corner points of the checkerboard on the calibration plate in the projector image coordinate system pp =(u pp ,v pp ) T , the coordinate x in the camera image coordinate system pc =(u pc ,v pc ) T , and the coordinate X in the world coordinate system i =(X w ,Y w ,Z w ) T , by using the Levenberg-Marquardt nonlinear minimization method, the parameters of the projector intrinsic parameter matrix M' and the camera intrinsic parameter matrix M are optimized; S2, constructing the mapping relationship between point cloud data and image data based on the measurement system optimization model; S3, constructing a preform winding surface quality detection network model, inputting image data and point cloud data into the model to obtain defect detection results and fiber segmentation results; S4. Based on the results of the preform winding surface quality detection network model, the defect detection of the preform winding surface and the calculation of the diameter and density of a single fiber of the preform are realized.

2. A method for detecting the surface of a preform winding body based on cross-modal data according to claim 1, characterized in that: The specific process of step S2 to construct the mapping relationship between the point cloud data and the two-dimensional image data is as follows: the corresponding relationship between the points in the point cloud data and the pixels in the image data is constructed by optimizing the camera matrix, that is, a single point (x i ,y i ,z i ) corresponds to a single pixel of image data (u i ,v i ), and its mapping relationship is expressed as: Among them, T represents the camera extrinsic matrix composed of the rotation matrix and the translation matrix, and S represents the camera intrinsic matrix.

3. The method for detecting the surface of a preform winding body based on cross-modal data according to claim 1, characterized in that: Step S3 specifically includes the following steps: S31, establishing a preform winding forming surface quality detection data set consisting of image data and point cloud data, and further dividing the data set into a training set, a test set and a validation set; In S32, the image data is constructed according to the PASCAL VOC object detection data set to build a file directory structure, and the point cloud data is constructed according to the 3DMatch convolutional neural network data set to build a file directory structure; S33. The constructed preform winding surface quality detection network model includes two branches: a preform qualitative analysis branch and a preform quantitative analysis branch; the preform qualitative analysis branch realizes the preform surface defect detection, and the preform quantitative analysis branch realizes the preform single fiber point cloud segmentation; S34. Constructing the loss function of the preform winding surface quality detection network model: l = l I +l P , where l I represents the preform surface defect detection loss function, l P represents the segmentation loss function of a single fiber point cloud of the preform; l I Use the same loss function as YOLOv5; P It consists of four items, namely the cross entropy loss function l wce 、Lovasz-softmax loss function l lovasz , heat map regression loss function l through mean square error heatmap and the offset map regression loss function l by mean absolute error offset ,Right now: l=l wce +αl lovasz +βl heatmap +γl offset ; Among them, α, β and γ are used to balance the loss coefficients affected by each loss term; S35, inputting the data in the training set, test set and validation set constructed above, training and evaluating the preform winding surface quality detection network model, and obtaining output results.

4. The method for detecting the surface of preform winding based on cross-modal data according to claim 3 is characterized in that: Step S33 specifically includes the following steps: S331, the preform qualitative analysis branch is constructed based on the pre-trained ResNet-50 deep residual network model, and includes an image encoding module and an image decoding module; the image encoding module captures the local features in the preform surface image through a convolution kernel with a size of 3×3 and a step size of 1, and the size of the feature map is corrected between each convolution layer through the maximum value pooling with a pooling window of 2×2 and a step size of 2 to reduce the resolution of the feature map; the obtained local features are processed by the image decoding module and output for defect detection on the preform winding forming surface; S332, the backbone network of the preform quantitative analysis branch is constructed based on the pre-trained Minkowski-UNet34 model, and includes a point cloud encoding module, a cross-modal data fusion module and a point cloud decoding module, wherein the cross-modal data fusion module includes a voxel image fusion module and a cross-modal association module.

5. The method for detecting the surface of preform winding based on cross-modal data according to claim 4, characterized in that: The specific process of step S332 includes the following steps: S3321. The voxel features of the point cloud data obtained by the point cloud encoding module are used as the original voxel features in the input cross-modal data fusion module. According to the mapping relationship between point cloud data and image data, the original voxel features are obtained. The corresponding image feature F I ; S3322, original voxel features The single point features extracted from the cascade point cloud data by the multi-layer perceptron MLP are combined with the image features F I Input together into the voxel image fusion module; introduce the offset to the image feature F I Sampling is performed to obtain the sampled image features Its formula is: where p i is the offset, Δp i,j is the lth sampling offset; The original element characteristics The sampled image features are regarded as the Query, and the sampled image features are regarded as the Key and Value, and the image enhanced voxel features are obtained through the multi-head attention mechanism. Its formula is expressed as: Among them, W m and W m ′ represents the learnable weight, m ​​represents the index of the multi-head attention mechanism, M represents the number of self-attention heads, and L represents the total number of sampled image features; Then enhance the voxel features of the image Characteristics of the original body Through residual connection, the final fused voxel features are output S3323, final fused voxel features As the input of the cross-modal association module, it is mapped into the point cloud space and the final fused voxel features are The point features mapped to the point cloud space are processed by trilinear interpolation and bilinear interpolation to obtain interpolated voxel features of the same dimension, which are then cascaded to obtain the global representation and view adaptive features F M ; Introduce learning parameter W v F M The weighted processing is performed and input into the multi-layer perceptron MLP, and the obtained features are then combined with F M Perform residual connection to obtain cross-view fusion features S3324, cross-view fusion features Input into the point cloud decoding module; the point cloud decoding module matches the input and output points based on the reverse distance weighted average interpolation method of k neighboring points, that is, let the feature to be solved of the interpolation point x be f (j) (x), the features of the k nearest neighbors of the known features of this point are f (j) The solution formula for (x) is expressed as: Among them, d is the function to measure the distance, w i Represents the weight of the point, which is inversely proportional to the distance. The larger the distance d, the smaller the weight. p represents the influence of distance on the weight. S3325, processed cross-view fusion features The multi-layer perceptron MLP is then used to further sample the features. After sampling, it passes through one-dimensional convolution, BN layer, and ReLU activation function, and the final output result is used for the point cloud segmentation of a single fiber of the preform.

6. The method for detecting the surface of preform winding based on cross-modal data according to claim 5, characterized in that: The final fused voxel feature The number of channels is N×2C f , C f represents the number of channels of voxel features, and N represents the number of point clouds input to the network.

7. The method for detecting the surface of preform winding based on cross-modal data according to claim 3, characterized in that: Step S4 specifically includes the following steps: S41, realizing preform surface defect detection by utilizing the qualitative analysis branch results in the preform winding forming surface quality detection network model; S42, using the quantitative analysis branch result in the preform winding surface quality detection network model to achieve preform single fiber point cloud segmentation, the specific process includes: projecting the single fiber point cloud data obtained by the quantitative analysis branch onto the reference line, determining the length distribution of the point cloud data on the reference line, selecting a measurement point within the length range, selecting the point closest to the measurement point in the point cloud data as a section point, determining a two-dimensional projection section based on the direction vector of the section point and the reference line, projecting the points near the section onto the section, and obtaining richer section projection point cloud data; S43, calculating the diameter of a single fiber of the preform based on the cross-sectional projection point cloud data; S44, measuring the number of wefts within the width of one frame of the cross-sectional projection point cloud data, and calculating the fiber density at a fixed width and a fixed number of wefts.

8. The method for detecting the surface of preform winding based on cross-modal data according to claim 7, characterized in that: In step S43, the diameter of a single fiber of the preform is calculated by a circle fitting method based on the cross-sectional projection point cloud data, and the specific process includes the following steps: S431, perform preliminary circle fitting on all input data points using an algebraic fitting algorithm to obtain a preliminary estimated circle center O i (a i ,b i ), radius R i At the same time, calculate each point P in the point cloud data j The distance d from (j=0,1,…,n) to the fitting circle i,j ,Right now: Among them, (x j ,y j ) is point P j The coordinates of S432, set the number of iterations I, the initial value is 0, and determine the change in the radius of the fitting circle ΔR i Is it less than the preset convergence threshold γ? th If yes, the iteration is stopped and the current fitting result is output as the final circle. The calculation formula of the radius change is: ΔR i =R i -R i-1 ; Among them, R i and R i-1 are the fitting circle radii of this and last iterations respectively; If the radius change ΔR i If the convergence condition is not met, the preset error threshold ε is used. th Search for internal points; when point P j The distance d to the fitted circle i,j Less than or equal to the error threshold ε th When , mark them as internal points, select these internal points to form a new point cloud set As the input of the next iteration; repeat the iteration until the fitting result that meets the convergence condition is output or the preset upper limit of the number of iterations is reached. max , the result of the last iteration is taken as the final radius, that is, the diameter of a single fiber of the preform is obtained.

Images