Defect detection method, device and system based on multi-source heterogeneous data fusion
By adopting defect detection methods of multi-source heterogeneous data fusion in furniture production, combined with federated learning and deep neural networks, the problems of inefficient efficiency and insufficient accuracy in furniture detection are solved, and higher detection accuracy and data privacy protection are achieved .
Patent Information
- Application Number
- CN202411006573.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-25
- Publication Date
- 2024-12-06
AI Technical Summary
The prior art has problems of inefficiency and insufficient detection accuracy in furniture defect detection, especially in the insufficient data volume and complex scenarios, the two-dimensional picture deep convolutional neural network cannot effectively solve the problem of defect detection.
Using defect detection method based on multi-source heterogeneous data fusion, local defect detection models are trained on each production line client through the federated learning framework, and adaptive aggregation is performed through the central server to form a global defect detection model, combining table data and points Cloud data, using feature extraction, fusion and decision-making modules for defect prediction.
While ensuring data privacy and security, it improves the efficiency and accuracy of defect detection, and gradually accumulates attention and visual semantic information through data fusion, improving the accuracy of detection results.
Smart Images

Figure CN119091283A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of defect detection technology, and in particular to a defect detection method, device and system based on multi-source heterogeneous data fusion. Background Art
[0002] In recent years, benefiting from the advancement of urbanization and consumers' increasing attention to home environment and quality of life, the output demand of the furniture market has been growing. However, during the production process of furniture, the quality of the wood used to assemble the furniture may be unqualified due to factors such as environment, time, and weather, which may lead to problems in normal use. It is also possible that the furniture is not properly calibrated or measured due to mechanical failure or human error, and assembly errors may occur. When faced with the possibility of multiple types of defects in furniture, if the inspection process only relies on manual observation to inspect each inspection point one by one, this not only requires rich experience in manual work, but also leads to inefficiency.
[0003] For defect detection, most of the deep convolutional neural networks currently used are based on two-dimensional images, such as YOLOv4, YOLOv5, SSD, etc. These models often fail to achieve satisfactory results when faced with complex and changing real-world scenarios. At the same time, in industrial production environments, due to the low probability of errors in sheet metal, each production line will face the problem of lack of training data when training its own defect detection model, resulting in low detection accuracy of the trained model.
[0004] Therefore, it is necessary to develop a new plate defect detection method to solve problems such as the lack of depth perception and light projection in two-dimensional images, and to meet the data privacy requirements of various production lines. Summary of the invention
[0005] The technical problem to be solved by the present invention is to provide a defect detection method, equipment and system based on multi-source heterogeneous data fusion, which can solve the problem of insufficient data volume on the client side of each production line to train the model and low efficiency in detecting product defects.
[0006] In order to solve the above technical problems, the present invention provides a defect detection method based on multi-source heterogeneous data fusion, including: obtaining image information of the plate to be tested; inputting the image information of the plate to be tested into a pre-trained global defect detection model to output a plate defect prediction result; the global defect detection model is trained by the local defect detection model of each production line through an adaptive aggregation method.
[0007] As an improvement of the above scheme, the steps of training the global defect detection model by the local defect detection model of each production line in an adaptive aggregation manner include: each production line client receives the global model parameters sent by the central server; each production line client preprocesses the local data set, and the local data set includes a local training set and a local verification set; each production line client trains the local defect detection model with the preprocessed local training set to generate local model parameters; each production line client predicts on the trained local defect detection model with the preprocessed local verification set to obtain a prediction score, and calculates the prediction score based on the prediction score and the preprocessed local data set. Aggregation factor; each production line client sends the local model parameters and the aggregation factor to the central server, so that the central server adaptively aggregates the local model parameters according to the aggregation factor to generate new global model parameters and form a new global defect detection model; each production line client receives the new global model parameters sent by the central server; each production line client replaces the local model parameters with the new global model parameters to form a new local defect detection model; each production line client determines whether the new local defect detection model converges, if converged, terminates the training of the local defect detection model, otherwise, retrains the local defect detection model until convergence.
[0008] As an improvement of the above-mentioned scheme, the local defect detection model includes a feature extraction module, a feature fusion module and a feature decision module. The training method of the local defect detection model includes: inputting the preprocessed local training set into the feature extraction module for feature extraction processing to obtain a context tensor; inputting the context tensor into the feature fusion module for feature fusion processing of a preset number of fusions to obtain a memory tensor; inputting the memory tensor into the feature decision module for prediction processing, and updating the local model parameters through back propagation; judging whether the current training times reaches the preset training times, and if it is judged as no, re-training, and if it is judged as yes, ending the training.
[0009] As an improvement of the above-mentioned scheme, the step of inputting the preprocessed local training set into the feature extraction module for feature extraction processing to obtain the context tensor includes: inputting the tabular data in the preprocessed local training set into the LSTM network to extract the tabular feature vector, and averaging the tabular feature vector to obtain the tabular context tensor; inputting the point cloud data in the preprocessed local training set into the Transformer encoder to extract the point cloud feature vector, and averaging the point cloud feature vector to obtain the point cloud context tensor.
[0010] As an improvement of the above scheme, the context tensor includes a table context tensor and a point cloud context tensor, and the step of inputting the context tensor into the feature fusion module for feature fusion processing with a preset number of fusions to obtain a memory tensor includes: performing tensor multiplication processing on the table context tensor and the point cloud context tensor to obtain a fused memory tensor; judging whether the current fusion number reaches the preset fusion number; if the current fusion number does not reach the preset fusion number, performing element-wise multiplication of the memory tensor and the table feature vector to generate a table hidden state tensor, and inputting the table hidden state tensor into the table attention mechanism to obtain a new table context tensor, performing element-wise multiplication of the memory tensor and the point cloud feature vector to generate a point cloud hidden state tensor, and inputting the point cloud hidden state tensor into the point cloud attention mechanism to obtain a new point cloud context tensor, and re-fusing the new table context tensor, the new point cloud context tensor and the memory tensor to obtain a new fused memory tensor until the current fusion number reaches the preset fusion number.
[0011] As an improvement of the above scheme, the step of inputting the memory tensor into the feature decision module for prediction processing and updating the local model parameters by back propagation includes: inputting the memory tensor into the feature decision module to fully connect the memory tensor into a weight matrix, and outputting the probability value of defects occurring at each detection point; comprehensively considering the probability value to obtain a defect prediction result; calculating the error through the label value; and updating the local model parameters by back propagation according to the error calculation result.
[0012] As an improvement of the above solution, the error calculation formula is:
[0013]
[0014] Among them, L is the error calculation result, n is the number of samples in the local training set, The probability that the i-th sample is defective is predicted by the local defect detection model, y i is the probability that the i-th sample in the local training set is defective.
[0015] As an improvement of the above-mentioned scheme, the steps of preprocessing the local data set by each production line client include: performing data cleaning on the tabular data in the local data set, and serializing the feature names and values of the tabular data after the data cleaning into natural language strings; and / or performing initial processing on the point cloud data in the local data set, segmenting the initially processed point cloud data based on spatial proximity, and then collecting point sets from the point cloud to form separate point cloud data.
[0016] Correspondingly, the present invention also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, wherein the processor implements the steps of the above-mentioned defect detection method based on multi-source heterogeneous data fusion when executing the computer program.
[0017] Correspondingly, the present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the above-mentioned defect detection method based on multi-source heterogeneous data fusion.
[0018] The implementation of the present invention has the following beneficial effects:
[0019] The present invention uses a federated learning framework to aggregate local defect detection models on the clients of each production line through a central server, achieving the purpose of data sharing while ensuring the privacy and security of the clients of each production line, and solving the problem of insufficient data volume on the clients of each production line to train models and low efficiency in detecting product defects.
[0020] Furthermore, the present invention adopts a data fusion method to perform repeated data fusion to obtain the final memory tensor, and gradually accumulates the focused visual and table semantic information in the process of continuous fusion, so as to make the result prediction more accurate;
[0021] In addition, the present invention introduces a dual attention mechanism in the feature fusion module, and uses different attention mechanisms for different data, so that the key features of the data can be fused in the fusion stage, and finally a higher accuracy rate can be achieved. This has a significant driving effect on defect detection in the furniture production process and is of great significance to the development of the furniture industry. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] Figure 1 It is a flow chart of an embodiment of a defect detection method based on multi-source heterogeneous data fusion of the present invention;
[0023] Figure 2 It is a flow chart of an embodiment of the present invention in which the local defect detection model of each production line trains the global defect detection model by adaptive aggregation;
[0024] Figure 3 It is a structural schematic diagram of the local defect detection model in the present invention;
[0025] Figure 4 It is a flow chart of an embodiment of the training method of the local defect detection model in the present invention. DETAILED DESCRIPTION
[0026] In order to make the objectives, technical solutions and advantages of the present invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings.
[0027] See also Figure 1 , Figure 1 The flowchart of an embodiment of the defect detection method based on multi-source heterogeneous data fusion of the present invention is shown, which includes:
[0028] S101, obtaining image information of the plate to be tested;
[0029] S102, inputting the image information of the plate to be tested into a pre-trained global defect detection model to output a plate defect prediction result;
[0030] It should be noted that the global defect detection model in the present invention is trained by adaptively aggregating the local defect detection models of each production line.
[0031] See also Figure 2 , Figure 2 The flowchart of an embodiment of the present invention in which the local defect detection model of each production line trains the global defect detection model by adaptive aggregation is shown, including:
[0032] S201, each production line client receives the global model parameters sent by the central server;
[0033] The central server initializes the global model parameters w t , and sends the global model parameters to each production line client; each production line client receives the global model parameters w sent by the central server t .
[0034] S202, each production line client pre-processes the local data set;
[0035] The present invention divides the local data set into a local training set and a local validation set, and both the local training set and the local validation set include table data and point cloud data; wherein the table data includes basic morphological data such as the length, width, height, material, and punching points of the plate; and the point cloud data is data collected by a three-dimensional laser scanner, including the three-dimensional coordinates, color, reflectivity, and other information of the plate;
[0036] Accordingly, each production line client in the present invention needs to pre-process the table data and point cloud data respectively. Specifically:
[0037] 1. Tabular Data
[0038] The steps to preprocess the tabular data include:
[0039] (1) For the table data x in the local data set tab After data cleaning, the obtained data is recorded as
[0040] Cleaning processing includes processing null values, duplicate data and other operations.
[0041] (2) Use Label-To-Text to clean the tabular data The feature names and values are serialized into natural language strings y tab :
[0042]
[0043] in,
[0044] The table data after data cleaning;
[0045] L2T stands for Label-To-Text method;
[0046] y tab A natural language string.
[0047] 2. Point cloud data
[0048] The steps for preprocessing point cloud data include:
[0049] (1) For the point cloud data x in the local dataset point After initial processing, the obtained data is recorded as
[0050] Initial processing includes operations such as noise removal, filtering, standardization, and normalization of coordinates.
[0051] (2) Point cloud data after initial processing Segmentation based on spatial proximity;
[0052] (3) Collect point sets from the point cloud to form separate Point Patches, i.e., point cloud data y point :
[0053]
[0054] in,
[0055] is the point cloud data after initial processing;
[0056] f represents the preprocessing of point cloud data;
[0057] y point For separate Point Patches.
[0058] S203, each production line client trains a local defect detection model using the preprocessed local training set to generate local model parameters;
[0059] Each production line client trains the local defect detection model through the preprocessed local training set. When the training reaches the preset number of training times, the local model parameter w can be obtained. i ;
[0060] S204, each production line client performs prediction on the trained local defect detection model using the preprocessed local validation set to obtain a prediction score, and calculates an aggregation factor based on the prediction score and the preprocessed local data set;
[0061] Specifically, the prediction score is:
[0062]
[0063] in,
[0064] p i represents the score of the local defect detection model for the i-th plate;
[0065] X i Indicates the number of defect points of the i-th plate recorded in the local verification set;
[0066] x i represents the number of defect points of the i-th plate predicted by the local defect detection model;
[0067] n p Indicates the number of plates in the local verification set;
[0068] P i represents the score of the i-th local defect detection model.
[0069] Accordingly, the aggregation factor is:
[0070]
[0071] in,
[0072] P i represents the score of the local defect detection model of the i-th production line client;
[0073] h i Indicates the number of samples of each production line client;
[0074] m represents the number of clients participating in the training;
[0075] β is the weight factor;
[0076] α i is the aggregation factor of the i-th production line client;
[0077] S205, each production line client sends the local model parameters and aggregation factors to the central server, so that the central server adaptively aggregates the local model parameters according to the aggregation factors to generate new global model parameters, and forms a new global defect detection model;
[0078] That is to say, each production line client will set the local model parameter w i and the aggregation factor α i Upload to the central server; the central server waits until it receives the local model parameters w of all production line clients i and the aggregation factor α i ; The central server has n local model parameters w i Adaptive aggregation is used for aggregation; the central server sends the new global model parameter w t+1 Distribute to clients on each production line.
[0079] It should be noted that the adaptive aggregation method determines the aggregation weight based on the sample data volume and score of each production line client:
[0080]
[0081] in,
[0082] w t The local model parameters of the local defect detection model of each production line client;
[0083] w t+1 The aggregated global model parameters;
[0084] m represents the number of clients participating in the training;
[0085] α i is the aggregation factor of the i-th production line client;
[0086] S206, each production line client receives the new global model parameters sent by the central server;
[0087] S207, each production line client replaces the local model parameters with the new global model parameters to form a new local defect detection model;
[0088] Each production line client receives the new global model parameter w t+1 Then, the global model parameter w is used t+1 Instead of the local model parameter w i .
[0089] S208, each production line client determines whether the new local defect detection model converges. If converged, the training of the local defect detection model is terminated. Otherwise, the local defect detection model is retrained until converged.
[0090] For the new local defect detection model, each production line client needs to determine whether it has converged, and if so, the training is terminated. Otherwise, steps 203 to 208 are repeated until the local defect detection model converges.
[0091] Therefore, the present invention adopts a federated learning training method, and trains by aggregating parameters of each production line to obtain a generalized model suitable for multiple production lines, thereby meeting the data privacy needs of each participant and improving the efficiency of detection.
[0092] like Figure 3 As shown, the local defect detection model in the present invention includes a feature extraction module, a feature fusion module and a feature decision module; wherein the feature extraction module includes feature extraction of table data and feature extraction of point cloud data.
[0093] See also Figure 4 , Figure 4 The flowchart of an embodiment of the training method of the local defect detection model in the present invention is shown, which includes:
[0094] S301, inputting the preprocessed local training set into a feature extraction module for feature extraction processing to obtain a context tensor;
[0095] Accordingly, in the present invention, feature extraction processing needs to be performed on the table data and the point cloud data respectively. Specifically:
[0096] 1. Tabular Data
[0097] (1) Input the preprocessed table data y in the local training set into the LSTM (Long Short-Term Memory) network to extract the table feature vector {u n}:
[0098] {u n}=LSTM(y tab )
[0099] in,
[0100] y tab is the preprocessed tabular data, i.e., a natural language string;
[0101] LSTM represents LSTM network;
[0102] {u n} is a table feature vector with a dimension of N×H1, where N is the number of detection points in the plate and H1 is the feature dimension of the detection points in the plate.
[0103] (2) For the table feature vector {u n} Perform an average operation to obtain the table context tensor u (0) :
[0104]
[0105] Where n is the number of features in the table data.
[0106] 2. Point cloud data
[0107] (1) The preprocessed point cloud data in the local training set Input Transformer encoder to extract point cloud feature vector {v n}:
[0108]
[0109] in,
[0110] y is the preprocessed point cloud data, i.e., individual Point Patches;
[0111] MPN(·) represents the preprocessed point cloud data Convert to a polygonal point cloud representation function; including operations such as segmentation, feature extraction, and dimensionality reduction of point cloud data;
[0112] Encoder represents the Transformer encoder;
[0113] {v n} is the point cloud feature vector with a dimension of N×H2, where N is the number of detection points in the plate and H2 is the feature dimension of the detection points in the plate.
[0114] (2) Perform an average operation on the point cloud feature vector to obtain the point cloud context tensor v (0) :
[0115]
[0116] Where n is the number of features of the point cloud data.
[0117] S302, inputting the context tensor into a feature fusion module for feature fusion processing for a preset number of fusions to obtain a memory tensor;
[0118] It should be noted that the context tensor includes a table context tensor and a point cloud context tensor. The step of inputting the context tensor into the feature fusion module for feature fusion processing of a preset number of fusions to obtain a memory tensor includes:
[0119] (1) Perform tensor multiplication on the table context tensor and the point cloud context tensor to obtain a fused memory tensor;
[0120] (2) Determine whether the current fusion times have reached the preset fusion times;
[0121] (3) If the current fusion number does not reach the preset fusion number, the memory tensor is element-wise multiplied by the table feature vector to generate a table hidden state tensor, and the table hidden state tensor is input into the table attention mechanism to obtain a new table context tensor; at the same time, the memory tensor is element-wise multiplied by the point cloud feature vector to generate a point cloud hidden state tensor, and the point cloud hidden state tensor is input into the point cloud attention mechanism to obtain a new point cloud context tensor; then, the new table context tensor, the new point cloud context tensor and the memory tensor are re-fused to obtain a new memory tensor after fusion, until the current fusion number reaches the preset fusion number.
[0122] That is to say, there are K+1 rounds of fusion in the present invention, i.e., k=0, 1, 2, 3, 4, ..., K;
[0123] (1) k = 0 rounds of fusion is to transform the table context tensor u (0) and the point cloud context tensor v (0) Perform tensor multiplication:
[0124]
[0125] in,
[0126] m (0) Represents the initial memory tensor, that is, the memory tensor after initial fusion, with a dimension of N×N, where N is the number of detection points;
[0127] The representation treats each coordinate as a 2-D point in a dual Cartesian space defined by the table dimension and the point cloud dimension.
[0128] (2) k = 1, 2, 3, 4, ..., K rounds of fusion are performed by transforming the table feature vector {u n} and point cloud feature vector {v n}Introduce the table attention module and the point cloud attention module to obtain the table context tensor u (k+1) and the point cloud context tensor v (k+1) , and then take the form context tensor u (k+1) , point cloud context tensor v (k+1) and the memory tensor m of the previous step (k) Perform fusion processing:
[0129]
[0130] in,
[0131] m (k)is the memory tensor of the previous step, k = 0, 1, 2, 3, 4, ..., K-1;
[0132] u (k+1) is the table context tensor;
[0133] v (k+1) Point cloud context tensor.
[0134] Furthermore, the operation process of the table attention module is to store the memory tensor m (k) With the characteristic vector of the form {u n} Perform element multiplication to obtain the hidden state tensor of the table Then the table hidden state tensor Introducing the A-SCN attention mechanism (i.e., table attention mechanism):
[0135]
[0136] in,
[0137] m (k) is the memory tensor;
[0138] {u n} is the table feature vector;
[0139] The hidden state tensor of the table representing the kth fusion process has a dimension of N×H1, where N represents the number of detection points and H1 represents the feature dimension of the detection points.
[0140] ASCN stands for A-SCN attention mechanism.
[0141] Similarly, the operation process of the point cloud attention module is to convert the memory tensor m (k) With the point cloud feature vector {v n} Perform element multiplication to obtain the point cloud hidden state tensor Then the point cloud hidden state tensor Introduce the Point-Attention mechanism (i.e. point cloud attention mechanism):
[0142]
[0143] in,
[0144] m (k) is the memory tensor;
[0145] {v n} is the point cloud feature vector;
[0146] is the point cloud hidden state tensor of the kth fusion process, with a dimension of N×H2, where N represents the number of detection points and H2 represents the feature dimension of the detection points;
[0147] PA stands for Point-Attention mechanism.
[0148] S303, inputting the memory tensor into the feature decision module for prediction processing, and updating the local model parameters by back propagation;
[0149] After completing the preset number of fusions, use the final memory tensor for prediction processing. The specific steps include:
[0150] (1) Input the memory tensor into the feature decision module to fully connect the memory tensor into a weight matrix and output the probability value of defects at each detection point;
[0151] That is to say, the final memory tensor is used for full connection operation (i.e., the full connection is an N×1 weight matrix), and the final output of the full connection layer is the probability value of defects occurring at each detection point.
[0152] (2) Comprehensively consider the probability values to obtain defect prediction results;
[0153] Comprehensive consideration refers to statistically analyzing the probability of defects occurring at each inspection point in turn.
[0154] For example, if the probability of defects occurring at all inspection points is less than 5%, the plate is considered qualified; conversely, as long as the probability of defects occurring at one inspection point is greater than 5%, the plate is considered to have defects.
[0155] (3) Error calculation based on label value;
[0156] The present invention uses the cross entropy loss function as a measure of the error between the target value and the predicted value. The error calculation formula is:
[0157]
[0158] in,
[0159] L is the error calculation result;
[0160] n is the number of samples in the local training set;
[0161] Predict the probability that the i-th sample is defective for the local defect detection model;
[0162] y i is the probability that the i-th sample in the local training set is defective.
[0163] (4) Based on the error calculation results, the local model parameters are updated through back propagation.
[0164] Specifically, the label values can be used for loss calculation and back-propagation can be performed to update the local model parameters.
[0165] S304, determining whether the current training times have reached the preset training times, if determined to be no, returning to step S301 and re-performing training, if determined to be yes, ending the training.
[0166] It should be noted that each production line client trains the local defect detection model through the preprocessed local training set. When the training reaches the preset number of training times, the local model parameter w can be obtained. i .
[0167] Therefore, the present invention combines point cloud data and tabular data, uses deep neural networks and attention mechanisms to extract and fuse their features, enhances the ability of features to express targets, and can effectively improve the accuracy of defect detection, thereby solving problems such as the lack of depth perception and light projection in 2D images.
[0168] Correspondingly, the present invention also discloses a computer device, including a memory and a processor, wherein the memory stores a computer program, wherein the processor implements the steps of the above-mentioned defect detection method based on multi-source heterogeneous data fusion when executing the computer program.
[0169] At the same time, the present invention also discloses a computer-readable storage medium on which a computer program is stored, wherein the computer program, when executed by a processor, implements the steps of the above-mentioned defect detection method based on multi-source heterogeneous data fusion.
[0170] In summary, the present invention has the following beneficial effects:
[0171] The present invention uses a federated learning framework to aggregate local defect detection models on the clients of each production line through a central server, thereby achieving data sharing while ensuring the privacy and security of the clients of each production line, and solving the problem of insufficient data on the clients of each production line to train models and low efficiency in detecting product defects.
[0172] The present invention adopts a data fusion method to perform repeated data fusion to obtain the final memory tensor. In the process of continuous fusion, the visual and table semantic information focused on are gradually accumulated, so that the result prediction is more accurate.
[0173] The present invention introduces a dual attention mechanism in the feature fusion module, and adopts different attention mechanisms for different data, so that the key features of the data can be fused in the fusion stage, and finally a higher accuracy rate can be achieved. This has a significant driving effect on defect detection in the furniture production process and is of great significance to the development of the furniture industry.
[0174] The above is a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several improvements and modifications without departing from the principle of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.
Claims
1. A defect detection method based on multi-source heterogeneous data fusion, It is characterized in that include: Obtain image information of the plate to be tested; Inputting the image information of the plate to be tested into a pre-trained global defect detection model to output a plate defect prediction result; The global defect detection model is trained by adaptively aggregating the local defect detection models of each production line.
2. The defect detection method based on multi-source heterogeneous data fusion according to claim 1, It is characterized in that The steps of training the global defect detection model by the local defect detection model of each production line in an adaptive aggregation manner include: Each production line client receives the global model parameters sent by the central server; Each production line client preprocesses a local data set, where the local data set includes a local training set and a local verification set; Each production line client trains the local defect detection model using the preprocessed local training set to generate local model parameters; Each production line client performs prediction on the trained local defect detection model using the preprocessed local validation set to obtain a prediction score, and calculates an aggregation factor based on the prediction score and the preprocessed local data set; Each production line client sends the local model parameters and aggregation factors to the central server, so that the central server adaptively aggregates the local model parameters according to the aggregation factors to generate new global model parameters and form a new global defect detection model; Each production line client receives the new global model parameters sent by the central server; Each production line client replaces the local model parameters with the new global model parameters to form a new local defect detection model; Each production line client determines whether the new local defect detection model converges. If converged, the training of the local defect detection model is terminated; otherwise, the local defect detection model is retrained until converged.
3. The defect detection method based on multi-source heterogeneous data fusion according to claim 2, It is characterized in that The local defect detection model includes a feature extraction module, a feature fusion module and a feature decision module. The training method of the local defect detection model includes: Inputting the preprocessed local training set into the feature extraction module for feature extraction processing to obtain a context tensor; Inputting the context tensor into the feature fusion module to perform feature fusion processing for a preset number of fusions to obtain a memory tensor; Inputting the memory tensor into the feature decision module for prediction processing, and updating the local model parameters by back propagation; Determine whether the current training times have reached the preset training times. If it is not, restart the training. If it is yes, end the training.
4. The defect detection method based on multi-source heterogeneous data fusion according to claim 3, It is characterized in that The step of inputting the preprocessed local training set into the feature extraction module for feature extraction processing to obtain a context tensor includes: Input the preprocessed table data in the local training set into the LSTM network to extract the table feature vector, and average the table feature vector to obtain the table context tensor; The preprocessed point cloud data in the local training set is input into the Transformer encoder to extract the point cloud feature vector, and the point cloud feature vector is averaged to obtain the point cloud context tensor.
5. The defect detection method based on multi-source heterogeneous data fusion according to claim 3, It is characterized in that The context tensor includes a table context tensor and a point cloud context tensor, and the step of inputting the context tensor into the feature fusion module for performing feature fusion processing for a preset number of fusions to obtain a memory tensor includes: Performing tensor multiplication processing on the table context tensor and the point cloud context tensor to obtain a fused memory tensor; Determine whether the current fusion times reaches the preset fusion times; If the current fusion times do not reach the preset fusion times, then Element-wise multiplication of the memory tensor and the table feature vector is performed to generate a table hidden state tensor, and the table hidden state tensor is input into the table attention mechanism to obtain a new table context tensor, Performing element-wise multiplication of the memory tensor and the point cloud feature vector to generate a point cloud hidden state tensor, and inputting the point cloud hidden state tensor into the point cloud attention mechanism to obtain a new point cloud context tensor, The new table context tensor, the new point cloud context tensor and the memory tensor are re-fused to obtain a new fused memory tensor until the current fusion times reaches the preset fusion times.
6. The defect detection method based on multi-source heterogeneous data fusion according to claim 3, It is characterized in that The step of inputting the memory tensor into the feature decision module for prediction processing and updating the local model parameters by back propagation includes: The memory tensor is input into the feature decision module to fully connect the memory tensor into a weight matrix, and the probability value of the defect occurring at each detection point is output; Comprehensively considering the probability values to obtain a defect prediction result; Error calculation is performed based on label values; According to the error calculation results, the local model parameters are updated through back propagation.
7. The defect detection method based on multi-source heterogeneous data fusion according to claim 6, It is characterized in that The error calculation formula is: Among them, L is the error calculation result, n is the number of samples in the local training set, The probability that the i-th sample is defective is predicted by the local defect detection model, y i is the probability that the i-th sample in the local training set is defective.
8. The defect detection method based on multi-source heterogeneous data fusion according to claim 2, It is characterized in that The step of preprocessing the local data set by each production line client includes: Performing data cleaning on the tabular data in the local data set, and serializing the feature names and values of the tabular data after the data cleaning into natural language strings; and / or The point cloud data in the local data set is initially processed, the initially processed point cloud data is segmented based on spatial proximity, and then point sets are collected from the point cloud to form separate point cloud data.
9. A computer device comprising a memory and a processor, wherein the memory stores a computer program. It is characterized in that When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.
10. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.
Citation Information
Patent Citations
Processing method and device for carrying out feature fusion on point cloud and image data
CN114782787A
Plate surface defect detection method and system based on image and point cloud data fusion
CN115496746A
Defect detection method based on federal learning and related equipment
CN116502950A
